Towards Maximum Likelihood: Learning Undirected Graphical ...

Towards Maximum Likelihood: Learning UndirectedGraphical Models using Persistent Sequential Monte

Hanchen Xiong

Institute of Computer ScienceUniversity of Innsbruck, Austria

November 26, 2014

Hanchen Xiong (UIBK) Learning UGMs with Persistent SMC November 26, 2014 1 / 22

Background

Undirected Graphical Models:

Markov Random Fields (or Markov Network)

Condtional Random Fields

Modeling:

p(x;θ) =exp (−E (x;θ))

Z(θ)(1)

Energy: E (x;θ) = −θ>φ(x) (2)

with random variables x = [x1, x2, . . . , xD ] ∈ XD where xd can take Nd

discrete values, φ(x) is a K -dimensional vector of sufficient statistics, andparameter θ ∈ RK .

Maximum Log-Likelihood Training

UGMs’ likelihood functions is concave w.r.t. θ[Koller and Friedman, 2009];

Given training data D = {x(m)}Mm=1, the derivative of average

log-likelihood L(θ|D) = 1M

∑Mm=1 log p(x(m);θ) as

∂L(θ|D)

∂θ= ED(φ(x))︸︷︷︸

−Eθ(φ(x))︸︷︷︸ψ−

M∑m=1

φ(x(m))−∑x′∈D

p(x′;θ)φ(x)′

interpretation: iteratively pulls down the energy of the data spaceoccupied by D (positive phase), but raises the energy over all dataspace XD (negative phase), until it reaches a balance (ψ+ = ψ−).

∂L(θ|D)

∂θ= ED(φ(x))︸︷︷︸

−Eθ(φ(x))︸︷︷︸ψ−

M∑m=1

φ(x(m))−∑x′∈D

p(x′;θ)φ(x)′

∂L(θ|D)

∂θ= ED(φ(x))︸︷︷︸

−Eθ(φ(x))︸︷︷︸ψ−

M∑m=1

φ(x(m))−∑x′∈D

p(x′;θ)φ(x)′

Existing Learning Methods

Approximate the second term of the gradient L:

∂L(θ|D)

∂θ= ED(φ(x))︸︷︷︸

−Eθ(φ(x))︸︷︷︸ψ−

Markov Chain Monte Carlo Maximum Likelihood(MCMCML)[Geyer, 1991]

Contrastive Divergence (CD) [Hinton, 2002]

Persistent Contrastive Divergence (PCD) [Tieleman, 2008]

Tempered Transition (TT) [Salakhutdinov, 2010]

Parallel Tempering (PT) [Desjardins et al., 2010]

CD,PCD,TT and PT can be summarized as a Robbins-Monro’sstochastic approximation procedure (SAP).[Robbins and Monro, 1951]

∂L(θ|D)

∂θ= ED(φ(x))︸︷︷︸

−Eθ(φ(x))︸︷︷︸ψ−

∂L(θ|D)

∂θ= ED(φ(x))︸︷︷︸

−Eθ(φ(x))︸︷︷︸ψ−

∂L(θ|D)

∂θ= ED(φ(x))︸︷︷︸

−Eθ(φ(x))︸︷︷︸ψ−

∂L(θ|D)

∂θ= ED(φ(x))︸︷︷︸

−Eθ(φ(x))︸︷︷︸ψ−

∂L(θ|D)

∂θ= ED(φ(x))︸︷︷︸

−Eθ(φ(x))︸︷︷︸ψ−

∂L(θ|D)

∂θ= ED(φ(x))︸︷︷︸

−Eθ(φ(x))︸︷︷︸ψ−

Robbins-Monro’s SAP

1 Training data set D = {x(1), · · · , x(M)}. Randomly initialize modelparameters θ0 and N particles {s0,1, · · · , s0,N}.

2 for t = 0 : T do // T iterations

3 for n = 1 : N do // go through all N particles

4 Sample st+1,n from st,n using transition operator Hθt ;

5 end for

6 Update: θt+1 = θt + η[ 1

M∑m=1

φ(x(m))− 1

N∑n=1

φ(st+1,n)]

7 Decrease η.

8 end for

9 When using Gibbs sampler as Hθt , SAP becomes PCD, and similarly,TT and PT can be substituted as well.

5 end for

M∑m=1

φ(x(m))− 1

N∑n=1

φ(st+1,n)]

7 Decrease η.

8 end for

5 end for

M∑m=1

φ(x(m))− 1

N∑n=1

φ(st+1,n)]

7 Decrease η.

8 end for

5 end for

M∑m=1

φ(x(m))− 1

N∑n=1

φ(st+1,n)]

7 Decrease η.

8 end for

5 end for

M∑m=1

φ(x(m))− 1

N∑n=1

φ(st+1,n)]

7 Decrease η.

8 end for

5 end for

M∑m=1

φ(x(m))− 1

N∑n=1

φ(st+1,n)]

7 Decrease η.

8 end for

5 end for

M∑m=1

φ(x(m))− 1

N∑n=1

φ(st+1,n)]

7 Decrease η.

8 end for

MCMCML

In MCMCML, a proposal distribution p(x;θ0) is set up in the sameform as a UGM, and we have

Z(θ)Z(θ0) =

∑x exp(θ>φ(x))∑x exp(θ>0 φ(x))

x exp(θ>φ(x))

exp(θ>0 φ(x))× exp(θ>0 φ(x))∑

x exp(θ>0 φ(x))

xexp(θ>φ(x))

x exp(θ>0 φ(x))

x exp((θ − θ0)>φ(x)

)p(x;θ0)

≈ 1S

∑Ss=1 w

where w (s) isw (s) = exp

((θ − θ0)>φ(x̄(s))

), (5)

MCMCML is an importance sampling approximation of the gradient.

MCMCML

In MCMCML, a proposal distribution p(x;θ0) is set up in the sameform as a UGM, and we have

Z(θ)Z(θ0) =

∑x exp(θ>φ(x))∑x exp(θ>0 φ(x))

x exp(θ>φ(x))

x exp(θ>0 φ(x))

xexp(θ>φ(x))

x exp(θ>0 φ(x))

x exp((θ − θ0)>φ(x)

)p(x;θ0)

≈ 1S

∑Ss=1 w

where w (s) isw (s) = exp

((θ − θ0)>φ(x̄(s))

), (5)

MCMCML is an importance sampling approximation of the gradient.

MCMCML

1: t ← 0, initialize the proposal distribution p(x;θ0)2: Sample {x̄(s)} from p(x;θ0)3: while ! stop criterion do4: Calculate w (s) using (5)

5: Calculate gradient ∂L̃(θt |D)∂θt

using importance samplingapproximation.

6: update θt+1 = θt + η ∂L̃(θt |D)∂θt

7: t ← t + 18: end while

MCMCML

MCMCML’s performance highly depends on initial proposaldistribution;

at time t, it is helpful to update the proposal distribution as thep(x;θt−1);

this is analogous to sequential importance sampling with resamplingat every iteration, however, the construction of sequentialdistributions is by learning;

this also looks like SAP learning schemes.

a similar connection between PCD and Sequential Monte Carlo wasfound in [Asuncion et al., 2010]

MCMCML

SAP Learning as Sequential Monte Carlo

1: Initialize p(x;θ0), t ← 0

2: Sample particles {x̄(s)0 }Ss=1 ∼ p(x;θ0)

3: while ! stop criterion do4: Assign w (s) ← 1

S ,∀s ∈ S // importance reweighting

5: // resampling is ignored because it has no effect6: switch (algorithmic choice) // MCMC transition7: case CD:8: generate a brand new particle set {x̄(s)

t+1}Ss=1 with Gibbs sampling from D

9: case PCD:10: evolve particle set {x̄(s)

t }Ss=1 to {x̄(s)t+1}

Ss=1 with one step Gibbs sampling

11: case Tempered Transition:12: evolve particle set {x̄(s)

t }Ss=1 to {x̄(s)t+1}

Ss=1 with tempered transition

13: case Parallel Tempering:14: evolve particle set {x̄(s)

t }Ss=1 to {x̄(s)t+1}

Ss=1 with parallel tempering

15: end switch16: Compute the gradient ∆θt according to (4);17: θt+1 = θt + η∆θt , t ← t + 1;18: reduce η;

19: end whileHanchen Xiong (UIBK) Learning UGMs with Persistent SMC November 26, 2014 9 / 22

A sequential Monte Carlo (SMC) algorithms can work on thecondition that sequential, intermediate distributions are wellconstructed: two successive ones should be close.

the gap between successive distributions in SAP: η × D1 learning rate η;

2 the dimensionality of x: D.

A sequential Monte Carlo (SMC) algorithms can work on thecondition that sequential, intermediate distributions are wellconstructed: two successive ones should be close.

the gap between successive distributions in SAP: η × D1 learning rate η;2 the dimensionality of x: D.

Persistent Sequential Monte Carlo

Every sequential, intermediate distributions is constructed by learning,so learning and sampling are interestingly entangled.

applying SMC philosophy future in sampling: Persistent SMC (SPMC)

{x̄(s)}S/2s=1 ∼ UD {x̄t−1,(s)}S/2s=1 ∼ pH(x;θt−1)

p0(x;θt) p1(x;θt) · · · pH(x;θt)

{x̄(s)}S/2s=1 ∼ UD {x̄t,(s)}S/2s=1 ∼ pH(x;θt)

p0(x;θt+1) p1(x;θt+1) · · · pH(x;θt+1)

{x̄t+1,(s)}S/2s=1 ∼ pH(x;θt+1)

Every sequential, intermediate distributions is constructed by learning,so learning and sampling are interestingly entangled.

applying SMC philosophy future in sampling: Persistent SMC (SPMC)

p0(x;θt+1) p1(x;θt+1) · · · pH(x;θt+1)

{x̄t+1,(s)}S/2s=1 ∼ pH(x;θt+1)

p0(x;θt+1) p1(x;θt+1) · · · pH(x;θt+1)

{x̄t+1,(s)}S/2s=1 ∼ pH(x;θt+1)

UD is a uniform distribu-tion on x, and interme-diate sequential distribu-tions are: ph(x;θt+1) ∝p(x;θt)

1−βhp(x;θt+1)βh

where 0 ≤ βH ≤ βH−1 ≤· · ·β0 = 1.

Number of Sub-Sequential Distributions

One issue arising in PSMC is the number of βh, i.e. H

By exploiting degeneration of particle set: the importance weightingfor each particle is

w (s) =ph(x̄(s);θt+1)

ph−1(x̄(s);θt+1)

= exp(E (x̄(s);θt)

)∆βhexp

(E (x̄(s);θt+1)

)−∆βh(6)

where ∆βh is the step length from βh−1 to βh, i.e. ∆βh = βh − βh−1.the ESS of a particle set as [Kong et al., 1994]

σ =(∑S

s=1 w(s))2

s=1 w(s)2∈[

ESS σ is actually a function of ∆βh.

Set a threshold on σ, at every h, and find the biggest gap by usingbidirectional search.

w (s) =ph(x̄(s);θt+1)

ph−1(x̄(s);θt+1)

)∆βhexp

(E (x̄(s);θt+1)

)−∆βh(6)

σ =(∑S

s=1 w(s))2

s=1 w(s)2∈[

w (s) =ph(x̄(s);θt+1)

ph−1(x̄(s);θt+1)

)∆βhexp

(E (x̄(s);θt+1)

)−∆βh(6)

σ =(∑S

s=1 w(s))2

s=1 w(s)2∈[

w (s) =ph(x̄(s);θt+1)

ph−1(x̄(s);θt+1)

)∆βhexp

(E (x̄(s);θt+1)

)−∆βh(6)

σ =(∑S

s=1 w(s))2

s=1 w(s)2∈[

Input: a training dataset D = {x(m)}Mm=1, learning rate η1: Initialize p(x;θ0), t ← 0

2: Sample particles {x̄(s)0 }Ss=1 ∼ p(x;θ0)

3: while ! stop criterion // root-SMC do4: h← 0, β0 ← 15: while βh < 1 // sub-SMC do6: assign importance weights {w (s)}Ss=1 to particles according to (6)7: resample particles based on {w (s)}Ss=1

8: find the step length ∆βh9: βh+1 = βh + δβ

10: h← h + 111: end while12: Compute the gradient ∆θt according to (4)13: θt+1 = θt + η∆θt

14: t ← t + 115: end while

Experiments

two experiments on two challenges:

big learning rates

high dimensional distributions

First Experiment: Small Learning Rates

a small-size Boltzmann Machine with only 10 variables is used to avoid theeffect of model complexitylearning rate ηt = 1

0 100 200 300 400 500−5

−4.5

−3.5

−2.5

−1.5

number of epochs

log−

likelih

PCD−10

PCD−1

0 100 200 300 400 5007

number of epochs

0 100 200 300 400 5007

number of epochs

Standard SMC

Figure: The performance of algorithms with the first learning rate scheme. (a):log-likelihood vs. number of epochs; (b) and (c): the number of βs in PSMC andSMC at each iteration (blue) and their mean values (red).

First Experiment: Bigger Learning Rates

learning rate ηt = 120+0.5×t

0 20 40 60 80 100−5

−4.5

−3.5

−2.5

−1.5

number of epochs

log−

likelih

PCD−10

PCD−1

0 20 40 60 80 1007

number of epochs

0 20 40 60 80 1007

number of epochs

Standard SMC

Figure: The performance of algorithms with the second learning rate scheme. (a):log-likelihood vs. number of epochs; (b) and (c): the number of βs in PSMC andSMC at each iteration (blue) and their mean values (red).

First Experiment: Large Learning Rates

learning rate ηt = 110+0.1×t

0 10 20 30 40−5

−4.5

−3.5

−2.5

−1.5

number of epochs

log−

likelih

PCD−10

0 10 20 30 407

number of epochs

0 10 20 30 407

number of epochs

Standard SMC

Figure: The performance of algorithms with the third learning rate scheme. (a):log-likelihood vs. number of epochs; (b) and (c): the number of βs in PSMC andSMC at each iteration (blue) and their mean values (red).

Second Experiment: Small Dimensionality/Scale

we used the popular restricted Boltzmann machine to model handwrittendigit images (the MNIST database).10 hidden unites

200 400 600 800 1000−230

−225

−220

−215

−210

−205

−200

−195

number of epochs

testin

200 400 600 800 1000−230

−225

−220

−215

−210

−205

−200

−195

number of epochs

PCD−100 PCD−1 PT TT PSMC SMC

0 200 400 600 800 100060

number of epochs

0 200 400 600 800 100050

number of epochs

Standard SMC

(c)Hanchen Xiong (UIBK) Learning UGMs with Persistent SMC November 26, 2014 19 / 22

Second Experiment: Large Dimensionality/Scale

500 hidden unites

200 400 600 800 1000−200

−195

−190

−185

−180

−175

−170

−165

−160

number of epochs

testin

200 400 600 800 1000−200

−195

−190

−185

−180

−175

−170

−165

−160

number of epochs

PCD−200 PCD−1 PT TT PSMC

0 200 400 600 800 10000

number of epochs

0 200 400 600 800 10006

number of epochs

sStandard SMC

Conclusion

a new interpretation of learning undirected graphical models:sequential Monte Carlo (SMC)

reveal two challenges in learning: large learning rate, highdimensionality;

deeper application of SMC in learning −→ Persistent SMC;

yield higher likelihood than state-of-the-art algorithms in challengingcases.

Conclusion

Asuncion, A. U., Liu, Q., Ihler, A. T., and Smyth, P. (2010).Particle filtered MCMC-MLE with connections to contrastivedivergence.In ICML.

Desjardins, G., Courville, A., Bengio, Y., Vincent, P., and Delalleau,O. (2010).Tempered Markov Chain Monte Carlo for training of restrictedBoltzmann machines.In AISTATS.

Geyer, C. J. (1991).Markov Chain Monte Carlo Maximum Likelihood.Compuuting Science and Statistics:Proceedings of the 23rdSymposium on the Interface.

Hinton, G. E. (2002).Training products of experts by minimizing contrastive divergence.Neural Computation, 14(8):1771–1800.

Koller, D. and Friedman, N. (2009).Probabilistic Graphical Models: Principles and Techniques.MIT Press.

Kong, A., Liu, J. S., and Wong, W. H. (1994).Sequential Imputations and Bayesian Missing Data Problems.Journal of the American Statistical Association, 89(425):278–288.

Robbins, H. and Monro, S. (1951).A Stochastic Approximation Method.Ann.Math.Stat., 22:400–407.

Salakhutdinov, R. (2010).Learning in markov random fields using tempered transitions.In NIPS.

Tieleman, T. (2008).Training Restricted Boltzmann Machines using Approximations to theLikelihood Gradient.In ICML, pages 1064–1071.

Towards Maximum Likelihood: Learning Undirected Graphical ...

Documents

Transcript of Towards Maximum Likelihood: Learning Undirected Graphical ...

Graphical abstract 2

Learning the Structure of Mixed Graphical Modelslee715/slides/MixedGM_Proxnewton_icml2014.pdf · Learning the Structure of Mixed Graphical Models Jason Lee with Trevor Hastie, Michael

5 Bayesian inference for extremes - Newcastle Universitynlf8/shortcourse/part5.pdf · The principle of maximum likelihood estimation is to adopt the model with greatest likelihood;

Bayesian Graphical Models

4a. Inference in Graphical ModelsMachine Learning for Computer Vision Summary • Undirected Graphical Models represent conditional independence more intuitively using graph separation

Maximaum likelihood timing estimation

AS2019 11 26 MaximumLikelihoodpetersen/Teaching/Stat2019/Week2/AS2019_11… · The principle of maximum likelihood for parameter estimation consist of maximising the likelihood of

Pitfalls in likelihood land

Gaussian Graphical Models and Graphical Lassoyc5/ele538b_sparsity/lectures/... · 2018-11-07 · [1]”Sparse inverse covariance estimation with the graphical lasso,” J. Friedman,

Lecture 11: Maximum likelihood IV. (nonlinear least square ...

Gaussian Graphical Models and Graphical Lasso · 2018-11-07 · Gaussian Graphical Models and Graphical Lasso Yuxin Chen Princeton University, Fall 2018. Multivariate Gaussians Consider

Fehlspezi kation, Quasi-Likelihood und Sch …semwiso.userweb.mwn.de/schaetzentesten2-ss10/skript/ST2...Kapitel 6 Fehlspezi kation, Quasi-Likelihood und Sch atzgleichungen Bisher haben

SLAC-PUB-15589 A MAXIMUM LIKELIHOOD APPROACH TO …

Maximum Likelihood EstimationMaximum likelihood estimator (MLE) The maximum likelihood estimator (MLE) is the value b which maximises L( ;x). The MLE also maximises l( ;x) because

Formulation of model likelihood functions

Typography - continue K1066BI – Graphical Design Teppo Räisänen Teppo.raisanen@oamk.fi.

Probabilistic Graphical Models - Caltech Computingcourses.cms.caltech.edu/cs155/slides/cs155-02... · Probabilistic Graphical Models Lecture 2 –Bayesian Networks Representation

Lab 13: Pseudo-likelihoodjernite/ir14/Lab13.pdf · Lab 13: Pseudo-likelihood Yacine Jernite December 11, 2014 Yacine Jernite Lab 13: Pseudo-likelihood

CHAPTER 5: MAXIMUM LIKELIHOOD ESTIMATION

Workshop on Empirical Likelihood Methods in Survival Analysis