LIMITS
57
1 LIMITS • Why limits? • Methods for upper limits • Desirable properties • Dealing with systematics • Feldman-Cousins • Recommendations
description
LIMITS. Why limits? Methods for upper limits Desirable properties Dealing with systematics Feldman-Cousins Recommendations. WHY LIMITS?. Michelson-Morley experiment death of aether HEP experiments CERN CLW (Jan 2000) FNAL CLW (March 2000) - PowerPoint PPT Presentation
Transcript of LIMITS
1
LIMITS
• Why limits?
• Methods for upper limits
• Desirable properties
• Dealing with systematics
• Feldman-Cousins
• Recommendations
2
WHY LIMITS?
Michelson-Morley experiment death of aether
HEP experiments
CERN CLW (Jan 2000)
FNAL CLW (March 2000)
Heinrich, PHYSTAT-LHC, “Review of Banff Challenge”
3
SIMPLE PROBLEM?
Gaussian ~ exp{-0.5*(x-μ)2/σ2} No restriction on μ, σ known exactly
μ x0 + k σ BUT Poisson {μ = sε + b} s ≥ 0 ε and b with uncertainties
Not like : 2 + 3 = ?
N.B. Actual limit from experiment = Expected (median) limit
4
Bayes (needs priors e.g. const, 1/μ, 1/√μ, μ, …..)Frequentist (needs ordering rule, possible empty intervals, F-C)Likelihood (DON’T integrate your L)χ2 (σ2 =μ)χ2(σ2 = n)
Recommendation 7 from CERN CLW: “Show your L” 1) Not always practical 2) Not sufficient for frequentist methods
Methods (no systematics)
5
Bayesian posterior intervals
Upper limit Lower limit
Central interval Shortest
6
90% C.L. Upper Limits
x
x0
7
Ilya Narsky, FNAL CLW 2000
8
DESIRABLE PROPERTIES
• Coverage
• Interval length
• Behaviour when n < b
• Limit increases as σb increases
9
ΔlnL = -1/2 ruleIf L(μ) is Gaussian, following definitions of σ are
equivalent:
1) RMS of L(µ)
2) 1/√(-d2L/dµ2) 3) ln(L(μ±σ) = ln(L(μ0)) -1/2
If L(μ) is non-Gaussian, these are no longer the same
“Procedure 3) above still gives interval that contains the true value of parameter μ with 68% probability”
Heinrich: CDF note 6438 (see CDF Statistics Committee Web-page)
Barlow: Phystat05
10
COVERAGE
How often does quoted range for parameter include param’s true value?
N.B. Coverage is a property of METHOD, not of a particular exptl result
Coverage can vary with
Study coverage of different methods of Poisson parameter , from observation of number of events n
Hope for:
Nominal value
100%
)(C
11
COVERAGE
If true for all : “correct coverage”
P< for some “undercoverage”
(this is serious !)
P> for some “overcoverage” Conservative
Loss of rejection power
12
Coverage : L approach (Not frequentist)
P(n,μ) = e-μμn/n! (Joel Heinrich CDF note 6438)
-2 lnλ< 1 λ = P(n,μ)/P(n,μbest) UNDERCOVERS
13
Frequentist central intervals, NEVER undercovers
(Conservative at both ends)
14
Feldman-Cousins Unified intervals
Frequentist, so NEVER undercovers
15
Probability ordering
Frequentist, so NEVER undercovers
16
= (n-µ)2/µ Δ = 0.1 24.8% coverage?
NOT frequentist : Coverage = 0% 100%
22
17
COVERAGE
N.B. Coverage alone is not sufficient
e.g. Clifford (CERN CLW, 2000)
“Friend thinks of number
Procedure for providing interval that includes number 90% of time.”
18
COVERAGE
N.B. Coverage alone is not sufficiente.g. Clifford (CERN CLW, 2000) Friend thinks of number Procedure for providing interval that
includes number 90% of time.
90%: Interval = - to +10%: number = 102.84590135…..
19
INTERVAL LENGTH
Empty Unhappy physicists
Very short False impression of sensitivity
Too long loss of power
(2-sided intervals are more complicated because ‘shorter’ is not metric-independent: e.g. 04 or 4 9)
20
90% Classical interval for Gaussian
σ = 1 μ ≥ 0 e.g. m2(νe)
21
Behaviour when n < b
Frequentist: Empty for n < < b
Frequentist: Decreases as n decreases below b
Bayes: For n = 0, limit independent of b
Sen and Woodroofe: Limit increases as data decreases below expectation
22
FELDMAN - COUSINS
Wants to avoid empty classical intervals
Uses “L-ratio ordering principle” to resolve ambiguity about “which 90% region?”
[Neyman + Pearson say L-ratio is best for hypothesis testing]
Unified No ‘Flip-Flop’ problem
23Xobs = -2 now gives upper limit
24
25
26
Black lines Classical 90% central interval
Red dashed: Classical 90% upper limit
Flip-flop
27
28
Poisson confidence intervals. Background = 3
Standard Frequentist Feldman - Cousins
29
30
31
32
33
34
35
36
37
38
39
40
41
42
Recommendations?
CDF note 7739 (May 2005)
Decide method in advance
No valid method is ruled out
Bayes is simplest for incorporating nuisance params
Check robustness
Quote coverage
Quote sensitivity
Use same method as other similar expts
Explain method used
43
44
45Caltech Workshop, Feb 11th
46
47
48
49
50
51
52
53
54
56
Peasant and Dog
1) Dog d has 50% probability of being 100 m. of Peasant p
2) Peasant p has 50% probability of being within 100m of Dog d
d p
x
River x =0 River x =1 km
57
Given that: a) Dog d has 50% probability of being 100 m. of Peasant,
is it true that: b) Peasant p has 50% probability of being within 100m of Dog d ?
Additional information• Rivers at zero & 1 km. Peasant cannot cross them.
• Dog can swim across river - Statement a) still true
If Dog at –101 m, Peasant cannot be within 100m of Dog
Statement b) untrue
km 1 h 0
