Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on...
Transcript of Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on...
![Page 1: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/1.jpg)
Markov chain Monte Carlo
— Markov chain Monte Carlo (MCMC)
— Gibbs and Metropolis–Hastings
— Slice sampling
— Practical details
Iain Murrayhttp://iainmurray.net/
![Page 2: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/2.jpg)
Reminder
Need to sample large, non-standard distributions:
P (x |D) ≈ 1
S
S∑s=1
P (x |θ), θ ∼ P (θ |D) = P (D|θ)P (θ)P (D)
![Page 3: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/3.jpg)
Importance sampling weights
w =0.00548 w =1.59e-08 w =9.65e-06 w =0.371 w =0.103
w =1.01e-08 w =0.111 w =1.92e-09 w =0.0126 w =1.1e-51
![Page 4: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/4.jpg)
Metropolis algorithm
• Perturb parameters: Q(θ′; θ), e.g. N (θ, σ2)
• Accept with probability min
(1,P ∗(θ′|D)P ∗(θ|D)
)• Otherwise keep old parameters
0 0.5 1 1.5 2 2.5 30
0.5
1
1.5
2
2.5
3
This subfigure from PRML, Bishop (2006)Detail: Metropolis, as stated, requires Q(θ′; θ) = Q(θ; θ′)
![Page 5: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/5.jpg)
>20,000 citations
![Page 6: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/6.jpg)
Target distribution
P (x) =1
Ze−E(x)
e.g. x = E(x) =∑
(i,j)∈edges
Wijxixj, xi∈{±1}
![Page 7: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/7.jpg)
Local moves
↙ ↓ ↘ Q(x′;x)
![Page 8: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/8.jpg)
Markov chain exploration
→ →
↓Goal: a Markov chain,
xt ∼ T (xt←xt−1), such that:
P (x(t)) = e−E(x(t))/Z for large t.
![Page 9: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/9.jpg)
Invariant/stationary condition
If x(t−1) is a sample from P ,
x(t) is also a sample from P .
∑x
T (x′←x)P (x) = P (x′)
![Page 10: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/10.jpg)
Ergodicity
Unique invariant distribution
if ‘forget’ starting point, x(0)
![Page 11: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/11.jpg)
Quick review
MCMC: biased random walk exploring a target dist.
Markov steps,
x(s) ∼ T(x(s)←x(s−1)
)MCMC gives approximate,
correlated samples
EP [f ] ≈1
S
S∑s=1
f(x(s))
T must leave target invariant
T must be able to get everywhere in K steps (for some K)
![Page 12: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/12.jpg)
Gibbs sampling
Pick variables in turn or randomly,
and resample P (xi|xj 6=i)
z1
z2L
l ?
Ti(x′←x) = P (x′i |xj 6=i) δ(x′j 6=i − xj 6=i)
![Page 13: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/13.jpg)
Gibbs sampling correctness
P (x) = P (xi |x\i)P (x\i)
Simulate by drawing x\i, then xi |x\i
Draw x\i: sample x, throw initial xi away
![Page 14: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/14.jpg)
Reverse operators
If T leaves P (x) stationary, define a reverse operator
R(x←x′) =T (x′←x)P (x)∑x T (x
′←x)P (x)=T (x′←x)P (x)
P (x′).
A necessary condition: there exists R such that:
T (x′←x)P (x) = R(x←x′)P (x′), ∀x, x′.
If R = T , known as detailed balance (not necessary)
![Page 15: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/15.jpg)
Balance condition
T (x′← x)P (x) = R(x← x′)P (x′)
Implies that P (x) is left invariant:∑x
T (x′←x)P (x) = P (x′)
������������������*1∑
x
R(x←x′)
![Page 16: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/16.jpg)
Metropolis–Hastings
Arbitrary proposals ∼ Q:
Q(x′;x)P (x) 6= Q(x;x′)P (x′)
0 0.5 1 1.5 2 2.5 30
0.5
1
1.5
2
2.5
3
PRML, Bishop (2006)
Satisfies detailed balance by rejecting moves:
T (x′←x) =
Q(x′;x)min
(1, P (x′)Q(x;x′)
P (x)Q(x′;x)
)x′ 6= x
. . . x′=x
![Page 17: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/17.jpg)
Metropolis–HastingsTransition operator
• Propose a move from the current state Q(x′;x), e.g. N (x, σ2)
• Accept with probability min(1, P (x′)Q(x;x′)
P (x)Q(x′;x)
)• Otherwise next state in chain is a copy of current state
Notes
• Can use P ∗ ∝ P (x); normalizer cancels in acceptance ratio
• Satisfies detailed balance (shown below)
• Q must be chosen so chain is ergodic
P (x) · T (x′←x) = P (x) ·Q(x
′; x)min
(1,
P (x′)Q(x; x′)P (x)Q(x′; x)
)= min
(P (x)Q(x
′; x), P (x
′)Q(x; x
′))
= P (x′) ·Q(x; x
′)min
(1,
P (x)Q(x′; x)P (x′)Q(x; x′)
)= P (x
′) · T (x←x
′)
![Page 18: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/18.jpg)
Matlab/Octave code for demo
function samples = dumb_metropolis(init, log_pstar, iters, sigma)
D = numel(init);
samples = zeros(D, iters);
state = init;
Lp_state = log_pstar(state);
for ss = 1:iters
% Propose
prop = state + sigma*randn(size(state));
Lp_prop = log_pstar(prop);
if log(rand) < (Lp_prop - Lp_state)
% Accept
state = prop;
Lp_state = Lp_prop;
end
samples(:, ss) = state(:);
end
![Page 19: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/19.jpg)
Step-size demo
Explore N (0, 1) with different step sizes σsigma = @(s) plot(dumb_metropolis(0, @(x)-0.5*x*x, 1e3, s));
sigma(0.1)
0 100 200 300 400 500 600 700 800 900 1000−4
−2
0
2
4
99.8% accepts
sigma(1)
0 100 200 300 400 500 600 700 800 900 1000−4
−2
0
2
4
68.4% accepts
sigma(100)
0 100 200 300 400 500 600 700 800 900 1000−4
−2
0
2
4
0.5% accepts
![Page 20: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/20.jpg)
Diffusion time
Q
P
L
Generic proposals use
Q(x′;x) = N (x, σ2)
σ large → many rejections
σ small → slow diffusion:
∼(L/σ)2 iterations required
Adapted from MacKay (2003)
![Page 21: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/21.jpg)
An MCMC strategy
Come up with good proposals Q(x′;x)
Combine transition operators:
x1 ∼ TA(·←x0)
x2 ∼ TB(·←x1)
x3 ∼ TC(·←x2)
x4 ∼ TA(·←x3)
x5 ∼ TB(·←x4)
. . .
![Page 22: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/22.jpg)
Slice sampling idea
Sample point uniformly under curve P ∗(x) ∝ P (x)
x
u
(x, u)
P̃ (x)
p(u|x) = Uniform[0, P ∗(x)]
p(x|u) ∝{1 P ∗(x) ≥ u0 otherwise
= “Uniform on the slice”
![Page 23: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/23.jpg)
Slice sampling
Unimodal conditionals
x
u
(x, u)
x
u
(x, u)
x
u
(x, u)
• bracket slice
• sample uniformly within bracket
• shrink bracket if P ∗(x) < u (off slice)
• accept first point on the slice
![Page 24: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/24.jpg)
Slice sampling
Multimodal conditionals
x
u
(x, u)
P̃ (x)
• place bracket randomly around point
• linearly step out until bracket ends are off slice
• sample on bracket, shrinking as before
Satisfies detailed balance, leaves p(x|u) invariant
![Page 25: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/25.jpg)
Slice sampling
Advantages of slice-sampling:
• Easy — only requires P ∗(x) ∝ P (x)• No rejections
• Tweak params not too important
There are more advanced versions.Neal (2003) contains many ideas.
![Page 26: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/26.jpg)
Summary
• We need approximate methods to solve sums/integrals
• Monte Carlo does not explicitly depend on dimension.
Using samples from simple Q(x) only works in low dimensions.
• Markov chain Monte Carlo (MCMC) can make local moves.
Sample from complex distributions, even in high dimensions.
• simple computations ⇒ “easy” to implement
(harder to diagnose).
![Page 27: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/27.jpg)
How should we run MCMC?
• The samples aren’t independent. Should we thin,
only keep every Kth sample?
• Arbitrary initialization means starting iterations are bad.
Should we discard a “burn-in” period?
• Maybe we should perform multiple runs?
• How do we know if we have run for long enough?
![Page 28: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/28.jpg)
Forming estimates
Approximately independent samples can be obtained by thinning.
However, all the samples can be used.Use the simple Monte Carlo estimator on MCMC samples. It is:
— consistent
— unbiased if the chain has “burned in”
The correct motivation to thin: if computing f(x(s)) is expensive
In some special circumstances strategic thinning can help.Steven N. MacEachern and Mario Peruggia, Statistics & Probability Letters, 47(1):91–98, 2000.http://dx.doi.org/10.1016/S0167-7152(99)00142-X — Thanks to Simon Lacoste-Julien for the reference.
![Page 29: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/29.jpg)
Empirical diagnostics
Rasmussen (2000)
Recommendations
For diagnostics:Standard software packages like R-CODA
For opinion on thinning, multiple runs, burn in, etc.Practical Markov chain Monte CarloCharles J. Geyer, Statistical Science. 7(4):473–483, 1992.
http://www.jstor.org/stable/2246094
![Page 30: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/30.jpg)
Consistency checks
Do I get the right answer on tiny versions
of my problem?
Can I make good inferences about synthetic data
drawn from my model?
Getting it right: joint distribution tests of posterior simulators,
John Geweke, JASA, 99(467):799–804, 2004.
Posterior Model checking: Gelman et al. Bayesian Data Analysis
textbook and papers.
![Page 31: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/31.jpg)
Getting it right
θ
y
We write MCMC code to update θ |y
Idea: also write code to sample y |θ
Both codes leave P (θ, y) invariant
Run codes alternately. Check θ’s match prior
![Page 32: Markov chain Monte Carlo - School of Informatics ... · Monte Carlo does not explicitly depend on dimension. Using samples from simple Q(x) only works in low dimensions. Markov chain](https://reader033.fdocument.org/reader033/viewer/2022052816/60aa830f828d9e73925b5ecb/html5/thumbnails/32.jpg)
Summary
Write down the probability of everything.
Condition on what you know,sample everything that you don’t.
Samples give plausible explanations:— Look at them— Average their predictions