1

LIMITS

• Why limits?

• Methods for upper limits

• Desirable properties

• Dealing with systematics

• Feldman-Cousins

• Recommendations

2

WHY LIMITS?

Michelson-Morley experiment death of aether

HEP experiments

CERN CLW (Jan 2000)

FNAL CLW (March 2000)

Heinrich, PHYSTAT-LHC, “Review of Banff Challenge”

3

SIMPLE PROBLEM?

Gaussian ~ exp{-0.5*(x-μ)2/σ2} No restriction on μ, σ known exactly

μ x0 + k σ BUT Poisson {μ = sε + b} s ≥ 0 ε and b with uncertainties

Not like : 2 + 3 = ?

N.B. Actual limit from experiment = Expected (median) limit

4

Bayes (needs priors e.g. const, 1/μ, 1/√μ, μ, …..)Frequentist (needs ordering rule, possible empty intervals, F-C)Likelihood (DON’T integrate your L)χ2 (σ2 =μ)χ2(σ2 = n)

Recommendation 7 from CERN CLW: “Show your L” 1) Not always practical 2) Not sufficient for frequentist methods

Methods (no systematics)

5

Bayesian posterior intervals

Upper limit Lower limit

Central interval Shortest

6

90% C.L. Upper Limits

x

x0

7

Ilya Narsky, FNAL CLW 2000

8

DESIRABLE PROPERTIES

• Coverage

• Interval length

• Behaviour when n < b

• Limit increases as σb increases

9

ΔlnL = -1/2 ruleIf L(μ) is Gaussian, following definitions of σ are

equivalent:

1) RMS of L(µ)

2) 1/√(-d2L/dµ2) 3) ln(L(μ±σ) = ln(L(μ0)) -1/2

If L(μ) is non-Gaussian, these are no longer the same

“Procedure 3) above still gives interval that contains the true value of parameter μ with 68% probability”

Heinrich: CDF note 6438 (see CDF Statistics Committee Web-page)

Barlow: Phystat05

10

COVERAGE

How often does quoted range for parameter include param’s true value?

N.B. Coverage is a property of METHOD, not of a particular exptl result

Coverage can vary with

Study coverage of different methods of Poisson parameter , from observation of number of events n

Hope for:

Nominal value

100%

)(C

11

COVERAGE

If true for all : “correct coverage”

P< for some “undercoverage”

(this is serious !)

P> for some “overcoverage” Conservative

Loss of rejection power

12

Coverage : L approach (Not frequentist)

P(n,μ) = e-μμn/n! (Joel Heinrich CDF note 6438)

-2 lnλ< 1 λ = P(n,μ)/P(n,μbest) UNDERCOVERS

13

Frequentist central intervals, NEVER undercovers

(Conservative at both ends)

14

Feldman-Cousins Unified intervals

Frequentist, so NEVER undercovers

15

Probability ordering

Frequentist, so NEVER undercovers

16

= (n-µ)2/µ Δ = 0.1 24.8% coverage?

NOT frequentist : Coverage = 0% 100%

22

17

COVERAGE

N.B. Coverage alone is not sufficient

e.g. Clifford (CERN CLW, 2000)

“Friend thinks of number

Procedure for providing interval that includes number 90% of time.”

18

COVERAGE

N.B. Coverage alone is not sufficiente.g. Clifford (CERN CLW, 2000) Friend thinks of number Procedure for providing interval that

includes number 90% of time.

90%: Interval = - to +10%: number = 102.84590135…..

19

INTERVAL LENGTH

Empty Unhappy physicists

Very short False impression of sensitivity

Too long loss of power

(2-sided intervals are more complicated because ‘shorter’ is not metric-independent: e.g. 04 or 4 9)

20

90% Classical interval for Gaussian

σ = 1 μ ≥ 0 e.g. m2(νe)

21

Behaviour when n < b

Frequentist: Empty for n < < b

Frequentist: Decreases as n decreases below b

Bayes: For n = 0, limit independent of b

Sen and Woodroofe: Limit increases as data decreases below expectation

22

FELDMAN - COUSINS

Wants to avoid empty classical intervals

Uses “L-ratio ordering principle” to resolve ambiguity about “which 90% region?”

[Neyman + Pearson say L-ratio is best for hypothesis testing]

Unified No ‘Flip-Flop’ problem

23Xobs = -2 now gives upper limit

24

25

26

Black lines Classical 90% central interval

Red dashed: Classical 90% upper limit

Flip-flop

27

28

Poisson confidence intervals. Background = 3

Standard Frequentist Feldman - Cousins

29

30

31

32

33

34

35

36

37

38

39

40

41

42

Recommendations?

CDF note 7739 (May 2005)

Decide method in advance

No valid method is ruled out

Bayes is simplest for incorporating nuisance params

Check robustness

Quote coverage

Quote sensitivity

Use same method as other similar expts

Explain method used

43

44

45Caltech Workshop, Feb 11th

46

47

48

49

50

51

52

53

54

55

Tomorrow is last day of this visit

Contact me at:

l.lyons@physics.ox.ac.uk

56

Peasant and Dog

1) Dog d has 50% probability of being 100 m. of Peasant p

2) Peasant p has 50% probability of being within 100m of Dog d

d p

x

River x =0 River x =1 km

57

Given that: a) Dog d has 50% probability of being 100 m. of Peasant,

is it true that: b) Peasant p has 50% probability of being within 100m of Dog d ?

Additional information• Rivers at zero & 1 km. Peasant cannot cross them.

• Dog can swim across river - Statement a) still true

If Dog at –101 m, Peasant cannot be within 100m of Dog

Statement b) untrue

km 1 h 0