Bayesian Case Studies, week 2

Robin J. Ryder

14 January 2013

Robin J. Ryder Bayesian Case Studies, week 2

Reminder: Poisson model, Conjugate Gamma prior

For the Poisson model Yi ∼ Poisson(λ) with a Γ(a, b) prior on λ,the posterior is

π(λ|Y ) ∼ Γ(a +n∑1

yi , b + n)

Robin J. Ryder Bayesian Case Studies, week 2

Model choice

We have an extra binary variable Zi . We would like to checkwhether Yi depends on Zi , and therefore need to choose betweentwo models:

M1 M2

Yi ∼i .i .d P(λ)λ ∼ Γ(a, b)

Yi |Zi = k ∼i .i .d P(λk)λ1 ∼ Γ(a, b)λ2 ∼ Γ(a, b)

Robin J. Ryder Bayesian Case Studies, week 2

The model index is a parameter

We now consider an extra parameter M ∈ {1, 2} which indicatesthe model index. We can put a prior on M, for example a uniformprior: P[M = k] = 1/2. Inside model k, we note the parametersθk and the prior on θk is noted πk .We are then interested in the posterior distribution

P[M = k|y ] ∝ P[M = k]

∫L(θk |y)πk(θk)dθk

Robin J. Ryder Bayesian Case Studies, week 2

Bayes factor

The evidence for or against a model given data is summarized inthe Bayes factor:

B21(y) =P[M = 2|y ]/P[M = 1|y ]

P[M = 2]/P[M = 1]]

=m2(y)

m1(y)

where

mk(y) =

∫Θk

L(θk |y)πk(θk)dθk

Robin J. Ryder Bayesian Case Studies, week 2

Bayes factor

Note that the quantity

mk(y) =

∫Θk

L(θk |y)πk(θk)dθk

corresponds to the normalizing constant of the posterior when wewrite

π(θk |y) ∝ L(θk |y)πk(θk)

Robin J. Ryder Bayesian Case Studies, week 2

Interpreting the Bayes factor

Jeffrey’s scale of evidenc states that

If log10(B21) is between 0 and 0.5, then the evidence in favorof model 2 is weak

between 0.5 and 1, it is substantial

between 1 and 2, it is strong

above 2, it is decisive

(and symmetrically for negative values)

Robin J. Ryder Bayesian Case Studies, week 2

Analytical value

Remember that ∫ ∞0

λa−1e−bλdλ =Γ(a)

ba

Thus

m1(y) =ba

Γ(a)

Γ(a +∑

yi )

(b + n)a+∑

yi

and

m2(y) =b2a

Γ(a)2

Γ(a +∑

yHi )

(b + nH)a+∑

yHi

Γ(a +∑

yFi )

(b + nF )a+∑

yFi

Robin J. Ryder Bayesian Case Studies, week 2

Analytical value

Remember that ∫ ∞0

λa−1e−bλdλ =Γ(a)

ba

Thus

m1(y) =ba

Γ(a)

Γ(a +∑

yi )

(b + n)a+∑

yi

and

m2(y) =b2a

Γ(a)2

Γ(a +∑

yHi )

(b + nH)a+∑

yHi

Γ(a +∑

yFi )

(b + nF )a+∑

yFi

Robin J. Ryder Bayesian Case Studies, week 2

Monte Carlo

Let

I =

∫h(x)g(x)dx

where g is a density. Then take x1, . . . , xN iid from g and we have

IMCN =

1

N

∑h(xi )

which converges (almost surely) to I.When implementing this, you need to check convergence!

Robin J. Ryder Bayesian Case Studies, week 2

Harmonic mean estimator

Take a sample from the posterior distribution π1(θ1|y). Note that

Eπ1

[1

L(θ1|y)|y]

=

∫1

L(θ1|y)π1(θ1|y)dθ1

=

∫1

L(θ1|y)

π1(θ1)L(θ1|y)

m1(y)dθ1

=1

m1(y)

thus giving an easy way to estimate m1(y) by Monte Carlo.However, this method is in general not advised, since theassociated estimator has infinite variance.

Robin J. Ryder Bayesian Case Studies, week 2

Importance sampling

I =

∫h(x)g(x)dx

If we wish to perform Monte Carlo but cannot easily sample fromg , we can re-write

I =

∫h(x)g(x)

γ(x)γ(x)dx

where γ is easy to sample from. Then take x1, . . . , xN iid from γand we have

IISN =1

N

∑ h(xi )g(xi )

γ(xi )

Robin J. Ryder Bayesian Case Studies, week 2