Practical Statistics for Particle PhysicistsLecture 2

Harrison B. ProsperFlorida State University

European School of High-Energy PhysicsParádfürdő, Hungary

5 – 18 June, 2013

Example 4: Higgs to γγ

background model

signal model

Likelihood – Higgs to γγ (CMS)

2

x =m γγ

fb(x |c, a)=Aexp[−(cx + ax2 )]

fs (x | m,w) = Gaussian(x, m , w)

p(x | s, m,w,c,b,a) =exp[−(s+ b)] sfs(xi |m ,w)+ bfb(xi |c,a)[ ]i=1

N

∏

fb (x | c, a)dx =1∫

fs (x | m, w)dx =1∫

Likelihood – Higgs to γγ (CMS)

3

7 TeV 8 TeV

Outline

h Lecture 1h Descriptive Statisticsh Probability & Likelihood

h Lecture 2h The Frequentist Approach h The Bayesian Approach

h Lecture 3h Analysis Example

4

The Frequentist ApproachConfidence Intervals

The Frequentist Principle

The Frequentist Principle (Neyman, 1937)

Construct statements so that a fraction f ≥ p of them are guaranteed be true over an ensemble of statements.

The fraction f is called the coverage probability and p is called the confidence level (C.L.).

Note: The confidence level is a property of the ensemble to which the statements belong. Consequently, the confidence level may change if the ensemble changes.

6

Confidence Intervals – 1

Consider an experiment that observes D events with expected signal s and no background.

Neyman devised a way to make statements of the form

with the guarantee that at least a fraction p of them will be true.

But, since we don’t know the true value of s, this property must hold whatever the true value of s.

7

Confidence Intervals – 2

8

Observation space

Para

met

er sp

ace

For each value s find a region in the observation space with probability f ≥ CL

a L a R

Confidence Intervals – 3

h Central Intervals (Neyman) Has equal probabilities on either side

h Feldman – Cousins IntervalsContains largest values of the ratios p(D | s) / p(D | D)

h Mode – Centered IntervalsContains largest probabilities p(D | s)

By construction, all these intervals satisfy the frequentist principle: coverage probability ≥ confidence level

9

Confidence Intervals – 4

10

0 2 4 6 8 10 12 14 16 18 200

5

10

15

20Intervals - Poisson Distribution

Count

Para

met

er

Central

Feldman-Cousins

Mode-Centered

D ± √D

Confidence Intervals – 5

11

Central

Feldman-Cousins

Mode-Centered

D ± √D0 2 4 6 8 10 12 14 16 18 20

0

2

4

6

8

10

CentralFeldman-CousinsModeRoot(N)

Width of Intervals

Count

Inte

rval

Wid

th

Confidence Intervals – 6

12

Central

Feldman-Cousins

Mode-Centered

D ± √D0 5 10 15 20

0

0.2

0.4

0.6

0.8

1

CentralFeldman-CousinsModeRoot(N)68.3%

Coverage Probability

Poisson Parameter

Prob

abili

ty

The Frequentist ApproachThe Profile Likelihood

Maximum Likelihood – 1

Example: Top Quark Discovery (1995), D0 ResultsD = 17 eventsB = 3.8 ± 0.6 events

whereB = Q / kδB = √Q / k

14

15

Maximum Likelihood – 2

knowns:D = 17 eventsB = 3.8 ± 0.6 background events

unknowns:b expected background counts expected signal count

Find maximum likelihood estimates (MLE):

∂ln p(17 | s,b)∂s

= ∂ ln p(17 | s,b)

∂b= 0 ⇒ s, b

s = D − B, b = B

16

Maximum Likelihood – 3

The Goodh Maximum likelihood estimates (MLE) are consistent:

RMS goes to zero as more and more data are acquiredh If an unbiased estimate for a parameter exists, the

maximum likelihood procedure will find ith Given the MLE for s, the MLE for y = g(s) is just

The Bad (according to some!)h In general, MLEs are biased

The Uglyh Correcting for bias, however, can waste data and

sometimes yield absurdities

y =γ(s)

Exercise 7: Show thisHint: consider a Taylorexpansion about the MLE

Example: The Seriously UglyThe moment generating function of a probability

distribution P(k) is the average:

For the binomial, this is

which is useful for calculating moments

e.g., M2 = (np)2 + np – np2

17

Exercise 8a: Show this

Given that k events out of n pass a set of cuts, the MLE of the event selection efficiency is

p = k / n and the obvious estimate of p2 is

k2 / n2 But

is a biased estimate of p2. The best unbiased estimate of p2 isk (k – 1) / [n (n – 1) ]

Note: for a single success in n trials, p = 1/n, but p2 = 0!

18

Exercise 8b: Show this k2 / n2 =p2 +V / n

Exercise 8c: Show this

Example: The Seriously Ugly

The Profile Likelihood – 1

19

In order to make an inference about the signal, s, the 2-parameter problem,

must be reduced to one involving s only by getting rid of all nuisance parameters, such as b.

In principle, this must be done while respecting the frequentist principle: coverage prob. ≥ confidence level.

This is very difficult to do exactly.

p(D | s, b) =(s+b)D e−(s+b)

D !(bk)Qe−bk

G(Q +1)

The Profile Likelihood – 2

20

In practice, we replace all nuisance parameters by their conditional maximum likelihood estimates (CMLE), whichyields a function called the profile likelihood, pPL(D | s).

In the top quark discovery example, we find an estimate of b as a function of s

Then, in the likelihood p(D | s, b), b is replaced with its estimate.

Since this is an approximation, the frequentist principle is not guaranteed to be satisfied exactly

b =f(s)

The Profile Likelihood – 3

21

Wilks’ Theorem (1938)

If certain conditions are met, and pmax is the value of the likelihood p(D | s, b) at its maximum, the quantity

has a density that is asymptotically χ2. Therefore, by setting y(s) = 1 and solving for s, we can compute approximate 68% confidence intervals.

This is what Minuit (now TMinuit) has been doing for 40 years!

y(s) =−2lnpPL(D |s)

pm ax

The Profile Likelihood – 4

The CMLE of b is

with s = D – Bb = B

the mode (peak) of thelikelihood

b(s) =γ+ γ2 + 4(1+ k)Qs

2(1+ k)γ=D +Q−(1+ k)s

22

The Profile Likelihood – 5

By solving

for s, we can make the statement

@ 68% C.L.

23

Exercise 9: Show this

The Frequentist ApproachHypothesis Tests

Hypothesis Tests

25

Fisher’s Approach: Null hypothesis (H0), background-only

The null hypothesis isrejected if the p-value is judged to be small enough.

Note: this can be calculated only if p(x | H0) is known

Example – Top Quark Discovery

26

p(D | H0 ) =Poisson(D |B)

Background, B = 3.8 events (ignoring uncertainty)

p-value = Poisson(D |3.8)D =17

∞

∑ =5.7 ×10−7

D =17

D is observed count

This is equivalent to 4.9 σ

Hypothesis Tests – 2

27

p(x | H0 )p(x | H1)

x

€

xα

a = p(x | H0 )dxxα

∞

∫

Alternative hypothesis

significance (or size) of test

A fixed significance α is chosen before data are analyzed.

Neyman argued forcefully that it is necessary to consider alternative hypotheses H1

Neyman’s Approach: Null hypothesis (H0) + alternative (H1)

The Neyman-Pearson Test

28

x

€

xα

In Neyman’s approach,hypothesis tests area contest betweensignificance and power, i.e., the probability to accept a true alternative.

a = p(x | H0 )dxxα

∞

∫ p = p(x |H1)dxxa

∞

∫powersignificance of test

p(x | H0 ) p(x | H1)

The Neyman-Pearson Test

29

p

a

Power curvepower vs. significance. Note: in general, no analysis is uniformly the most powerful.

a = p(x | H0 )dxxα

∞

∫ p = p(x |H1)dxxa

∞

∫powersignificance of test

Blue is the morepowerful belowthe cross-over pointand green is the more powerful after.

The Bayesian Approach

The Bayesian Approach

Definition:A method is Bayesian if 1. it is based on the degree of belief interpretation of

probability and2. it uses Bayes’ theorem

for all inferences.D observed dataθ parameter of interestω nuisance parameters π prior density

p(θ,w |D )=

p(D |θ,w)p(θ,w)p(D )

31

The Bayesian Approach – 2

Bayesian analysis is just applied probability theory.

Therefore, the method for eliminating nuisance parameters is “simply” to integrate them out:

a procedure called marginalization.

The integral is a weighted average of the likelihood.

p(θ |D )= p(θ,w |D )∫ dw

∝ p(D |θ,w)p(θ,w)dw∫

32

The Bayesian ApproachAn Example

Example – Top Quark Discovery – 1

D0 1995 Top Discovery DataD = 17 eventsB = 3.8 ± 0.6 events

Calculations1. Compute the posterior density p(s | D)

2. Compute the Bayes factor B10 = p(D | H1) / p(D | H0).

3. Compute the p-value, including the effect of background uncertainty.

34

Step 1: Construct a probability model for the observations

then put in the data

D = 17 eventsB = 3.8 ± 0.6 background events

B = Q / kδB = √Q / k

to arrive at the likelihood.

p(D | s, b) =e−(s+b)(s+ b)D

D !e−kb(kb)Q

G(Q +1)

35

Q =(B /dB)2 =40.1

k =B /dB2 =10.6

Example – Top Quark Discovery – 2

Example – Top Quark Discovery – 3

Step 2: Write down Bayes’ theorem:

and specify the prior:

It is useful to compute the following marginal likelihood:

sometimes referred to as the evidence for s.

p(s, b | D) =p(D, s, b)

p(D )=p(D |s, b)p(s, b)

p(D )

36

p (s, b) = π (b | s) π (s)

p(D | s) = p(D |s,b)p(b |s)db∫

Example – Top Quark Discovery – 4

The Prior: What do

andrepresent?

They encode what we know, or assume, about the mean background and signal in the absence of new observations.

We shall assume that s and b are non-negative.

After a century of argument, the consensus today is that there is no unique way to represent such vague information.

37

p (b | s)p (s)

38

Example – Top Quark Discovery – 5

For simplicity, we take π(b | s) = 1.

We may now eliminate b from the problem:

where,

and where we have introduced the symbol H1 to denote the background + signal hypothesis.

p(D | s, H1) = p(0

∞

∫ D |s,b) p(b|s)d(kb)

=1Q(1−x)2 Beta(x, r+1, Q)

r=0

D

∑ Poisson(D −r|s)

Exercise 10: Show this

x =

11+ k

, Beta(x, n, m )= G(n + m )G(n)G(m )

xn−1(1−x)m −1

p(17|s, H1) as a function of the expected signal s.

39

Example – Top Quark Discovery – 6

40

Example – Top Quark Discovery – 7

The posterior density

Given the marginal likelihood

we can compute

where

Assuming a flat prior for the signal π (s | H1) = 1, the posterior density is given by

from which we can compute the central interval

@ 68% C.L.

Example – Top Quark Discovery – 8

41

Exercise 11: Derive an expressionfor p(s | D, H1) assuming a gammaprior Gamma(qs, U +1) for π(s | H1)

s ∈[9.9, 18.4]

The Bayesian ApproachHypothesis Testing

43

Bayesian Hypothesis Testing – 1

Conceptually, Bayesian hypothesis testing proceeds in exactly the same way as any other Bayesian calculation: one computes the posterior density

and marginalize it with respect to all parameters except those label the hypotheses

posterior prior

p(θ,φ,H |D )=

p(D |θ,φ,H)p(θ,φ,H)p(D )

likelihood

p(H | D) = p(θ,φ,H |D )dθ dφ∫∫

and of course get your pHD!

44

Bayesian Hypothesis Testing – 2

However, just like your PhD, it is usually more convenient, and instructive, to arrive at the p(H | D) in stages.

1. Factorize the priors: p(θ, f, H) = p(θ, f | H) p(H)

2. Then, for each hypothesis, H, compute the function

3. Then, compute the probability of each hypothesis, H

( | ) ( | , , ) ( , | )p D H p D H H d dθ f p θ f θ f=∫∫

( | ) ( )( | )( | ) ( )

H

p D H Hp H Dp D H H

pp

=

45

Bayesian Hypothesis Testing – 3

It is clear, however, that to compute p(H | D), it is necessary to specify the priors p(H).

Unfortunately, consensus on these numbers is highly unlikely!

Instead of asking for the probability of an hypothesis, p(H | D), we could compare probabilities:

The ratio in the first bracket is called the Bayes factor, B10.

p(H1 | D)p(H0 | D)

=p(D |H1)p(D |H0)

⎡⎣⎢

⎤⎦⎥

p(H1)p(H0)⎡⎣⎢

⎤⎦⎥

In principle, in order to compute the number

twe need to specify a proper prior for the signal, that is, a prior that integrates to one.

For simplicity, let’s assume π (s | H1) = δ(s – 14).

This yields: p(D | H0 ) = 3.86 x 10-6

p(D | H1 ) = p(D |14, H1 ) = 9.28 x 10-2

Bayesian Hypothesis Testing – 4

46

p(D | H1) = p(D |s,H1)p(s|H1)ds

0

∞

∫

Since,p(D | H0 ) = 3.86 x 10-6

p(D | H1 ) = 9.28 x 10-2

we conclude that the hypothesis with s = 14 events is favored over that with s = 0 by 24,000 to 1.

The Bayes factor can be mapped to a measure akin to “n-sigma”

Bayesian Hypothesis Testing – 5

47

Z = 2lnB10 =4.5

Exercise 12: Compute Z for the D0 results

Hypothesis Testing – A Hybrid Approach

48

Background, B = 3.8 ± 0.6 events

p-value = p(D |H0 )D =17

∞

∑ =5.4 ×10−6

D =17

D is observed count

This is equivalent to 4.4 σwhich may be compared with the 4.5 σ obtained with B10

p(D | H0 ) =p(D |s=0, H1)

=1Q(1−x)2Beta(x, D +1, Q)

Exercise 13: Verify this calculation

Summary

Frequentist Approach 1) Models statistical uncertainties with probabilities2) Uses the likelihood3) Ideally, respects the frequentist principle4) In practice, eliminates nuisance parameters through the

approximate procedure of profiling

Bayesian Approach 5) Models all uncertainties with probabilities6) Uses likelihood and prior probabilities7) Eliminate nuisance parameters through marginalization.

49