Approximate Likelihoods - Statistical Inference, Learning ... · Approximate Bayesian Computation...

Approximate LikelihoodsStatistical Inference Learning and Models for Big Data

Nancy ReidUniversity of Toronto

December 16 2015

Models and likelihoodbull Model for the probability distribution of y given x

bull Density f (y | x) with respect to eg Lebesgue measure

bull Parameters for the density f (y | x θ) θ = (θ1 θd )

bull Data y = (y1 yn) sometimes independent

bull Likelihood function L(θ y) prop f (y θ) (y1 yn)

bull log-likelihood function `(θ y) = log L(θ y) + c

bull often θ = (ψ λ)

bull θ could have very large dimension d gt n

bull θ could have infinite dimension in principleE(y | x) = θ(x) lsquosmoothrsquo

Approximate Likelihoods IST 2015 2

Why likelihood

bull makes probability modelling central `(θ y) = log f (y θ)

bull emphasizes the inverse problem of reasoning y rarr θ

bull converts a lsquopriorrsquo probability to a posterior π(θ)rarr π(θ | y)

bull provides a conventional set of summary quantitiesmaximum likelihood estimator score function

bull provides summary statistics with known limiting distribution

bull these define approximate pivotal quantities based onnormal distribution

bull basis for comparison of models using AIC or BIC


why likelihood



Important summaries

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0

logminuslikelihood function

θθ

logminus

likel

ihoo

d

θθθθ

θθ minusminus θθ

192 w2

bull maximum likelihood estimatorθ = arg supθ log L(θ y)

= arg supθ`(θ y)

bull observed Fisher information

j(θ) = minus part2`(θ)

partθ2

∣∣∣∣θ

bull efficient score function`prime(θ) = part`(θ y)partθ

`prime(θ) = 0 assuming enough regularity

bull `prime(θ y) =sumn

i=1(partpartθ) log fYi (yi θ) y1 yn independent


why likelihood



Limit theorems and approximate pivots

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2

bull (θ minus θ)j12(θ)Lminusrarr N(0 I)

bull 2`(θ)minus `(θ) Lminusrarr χ2d

bull under the model f (y θ)plus regularity conditions

bull d = 1 approximate pivots

re(θ) = (θ minus θ)j12(θ)sim N(01)

r(θ) = plusmnradic

[2`(θ)minus `(θ)] sim N(01) d ltlt n


approximate pivots

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2

re(θ) = (θ minus θ)j12(θ)

r(θ) = plusmnradic

[2`(θ)minus `(θ)]


Complex likelihood functionsgeneralized linear mixed models

GLM yij | ui sim expyijηij minus b(ηij) + c(yij)

linear predictor ηij = xTijβ + zT

ij ui j=1ni i=1m

random effects ui sim Nk (0Σ)

log-likelihood

`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk

expyTi Ziui minus 1T

i b(Xiβ + Ziui)minus12

uTi Σminus1uidui

)Ormerod amp Wand 2012


complex likelihood functionsmultivariate extremes example wind speed at d locations

vector observations (X1i Xdi) i = 1 n

component-wise maxima Z1 Zd Zj = max(Xj1 Xjn)

Zj are transformed (centered and scaled)

joint distribution function

Pr(Z1 le z1 Zd le zd ) = expminusV (z1 zd )

V (middot) can be parameterized via Gaussian process models

likelihood need the joint derivatives of V (middot)

combinatorial explosion Davison et al 2012


Approximate likelihood functions

bull simplify the likelihoodbull composite likelihoodbull variational approximationbull Laplace approximation to integrals

bull simulatebull Markov chain Monte Carlobull approximate Bayesian computation

bull change the mode of inferencebull indirect inferencebull quasi-likelihood


Composite likelihoodbull also called pseudo-likelihood Besag 1975bull reduce high-dimensional dependencies by ignoring them

bull for example replace f (yi1 yik θ) by

pairwise marginalprodjltj prime

f2(yij yij prime θ) or

conditionalprod

j

fc(yij | yN (ij) θ)

bull Composite likelihood function

CL(θ y) propnprod

i=1

prodjltj prime

f2(yij yij prime θ)

bull Composite ML estimates are consistent asymptoticallynormal not fully efficient Lindsay 1988 Varin R Firth 2011


Example spatial extremes Davison et al 2012 amp Huser 2015

Pr(Z1 le z1 Zd le zd ) = expminusV (z1 zd θ)

bull pairwise composite likelihood used to avoid combinatorialexplosion of derivatives

bull model choice using ldquoCLICrdquo an analogue of AICminus2 log(CL) + tr(Jminus1K )

bull Davison et al 2012 applied this to annual maximum rainfallat several stations near Zurich


Example Ising model

f (y θ) = exp(sum

(jk)isinE

θjkyjyk )1

Z (θ)j k = 1 K

bull neighbourhood contributions

f (yj | y(minusj) θ) =exp(2yj

sumk 6=j θjkyk )

exp(2yjsum

k 6=j θjkyk ) + 1= exp `j(θ y)

bull penalized Composite Likelihood functionbased on sample y (1) y (n)

CL(θ) =nsum

i=1

Ksumj=1

`j(θ y (i))minussumjltk

Pλ(|θjk |)

Xue et al 2012 Ravikumar et al 2010


Variational Approximation Ormerod amp Wand 2012

GLMM log-likelihood function

`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)

+ k one-dimensional integralsequiv `(βΣ microΛ)

summi=1 EusimN(microi Λi )

(yT

i Ziu minus 1Ti b(Xiβ + Ziu)minus 1

2 uTΣminus1u minus logφΛi (u minus microi ))


variational approximations Ormerod amp Wand 2012

`(βΣ) ge `(βΣ microΛ)

bull lower bound should be ldquocloserdquo to `(βΣ) Kullback-Leibler

bull usually approximates posterior π(θ | y) asympprod

qj(θ)

bull VL approx L(θ y) by a simpler function of θ egprod

qj(θ)

bull CL approx f (y θ) by a simpler function of y egprod

f (yj θ)

Robin 2012 Zhang amp Schneider 2012 JMLR V22 Grosse 2015 ICML







Approximate Bayesian Computation Marin et al 2010

bull likelihood is not computable butwe can simulate from the model

bull simulate θ from prior density π(middot)

bull simulate data y prime from f (middot θ)

bull if y prime = y then θ is an observation from posterior π(middot | y)

bull actually s(y prime) = s(y) for some set of statistics

bull actually ρs(y prime) s(y) lt ε for some distance function ρ(middot)

Fearnhead amp Prangle 2011

bull many variations using different MCMC methods to selectcandidate values θ


Indirect inference Smith 2008 Shalizi 2013

bull likelihood is not computable butwe can simulate from the true model

yt = Gt (ytminus1 xt εt θ) θ isin Rd

bull fit a simpler (wrong) model eg AR(1)

yt sim f (yt | ytminus1 xt θprime) θprime isin Rp

bull find the MLE θprime in the simpler model

bull choose θ simulate ylowast from true modelbull compute θprimelowast from simulated data

bull lsquogoodrsquo values of θ give θprime = θprimelowast actually close







This thematic program emphasizes both applied and theoretical aspects of statistical inference learning and models in big data The opening conference will serve as an introduction to the program concentrating on overview lectures and background preparation Workshops throughout the program will highlight cross-cutting themes such as learning and visualization as well as focus themes for applications in the social physical and life sciences It is expected that all activities will be webcast using the FieldsLive system to permit wide participation Allied activities planned include workshops at PIMS in April and May and CRM in May and August

JANUARY 12 ndash 23 2015

Opening Conference and Boot Camp

Organizing Committee Nancy Reid (Chair) Sallie Keller Lisa Lix Bin Yu


Workshop on Big Data and Statistical Machine Learning

Organizing committee Ruslan Salakhutdinov (Chair) Dale Schuurmans Yoshua Bengio Hugh Chipman Bin Yu

FEBRUARY 9 ndash 13 2015

Workshop on Optimization and Matrix Methods in Big Data

Organizing Committee Stephen Vavasis (Chair) Anima Anandkumar Petros Drineas Michael Friedlander Nancy Reid Martin Wainwright


Workshop on Visualization for Big Data Strategies and Principles

Organizing Committee Nancy Reid (Chair) Susan Holmes Snehelata HuzurbazarHadley Wickham Leland Wilkinson

MARCH 23 ndash 27 2015

Workshop on Big Data in Health Policy

Organizing Committee Lisa Lix (Chair) Constantine Gatsonis Sharon-Lise Normand

APRIL 13 ndash 17 2015

Workshop on Big Data for Social Policy

Organizing Committee Sallie Keller (Chair) Robert Groves Mary Thompson JUNE 13 ndash 14 2015

Closing Conference

Organizing Committee Nancy Reid (Chair) Sallie Keller Lisa Lix Hugh Chipman Ruslan Salakhutdinov Yoshua Bengio Richard Lockhart to be held at AARMS of Dalhousie University

Yoshua Bengio (Montreacuteal)

Hugh Chipman (Acadia)

Sallie Keller (Virginia Tech)

Lisa Lix (Manitoba)

Richard Lockhart (Simon Fraser)

Nancy Reid (Toronto)

Ruslan Salakhutdinov (Toronto)

ORGANIZING COMMITTEE

INTERNATIONAL ADVISORY COMMITTEE

Constantine Gatsonis (Brown)Susan Holmes (Stanford)Snehelata Huzurbazar (Wyoming)Nicolai Meinshausen (ETH Zurich)Dale Schuurmans (Alberta)Robert Tibshirani (Stanford)Bin Yu (UC Berkeley)

PROGRAM

JANUARY TO APRIL 2015

Large Scale Machine Learning

Instructor Ruslan Salakhutdinov (University of Toronto)


Topics in Inference for Big Data

Instructors Nancy Reid (University of Toronto) Mu Zhu (University of Waterloo)

GRADUATE COURSES

B I G DATA

THEMATIC PROGRAM ON STATISTICAL INFERENCE LEARNING AND MODELS FOR

For more information allied activities off-site and registration please visitwwwfieldsutorontocaprogramsscientific14-15bigdata

Image Credits Sheelagh Carpendale amp InnoVis

JANUARY - JUNE 2015

Six-month thematicprogram

Organized by CanadianStatistical SciencesInstitute

Hosted by FieldsInstitute for Research inMathematical Sciences

Program of workshopsbull Two week Opening Conference and Bootcamp

bull One week workshops at the Fields Institutebull Statistical Machine Learningbull Optimization and Matrix Methodsbull Visualization Strategies and Principlesbull Big Data in Health Policybull Big Data for Social Policy

All talks available at FieldsLive

bull One week workshops across Canadabull Networks Web mining and Cyber-securitybull Statistical Theory for Large-scale Databull Challenges in Environmental Science

bull Postdoctoral Fellows Courses Distinguished LectureSeries


Statistical Machine LearningRestricted Boltzmann machine

f (v h η) prop 1Z (η)

exp(αT v + βT h + vT Ωh) η = (α βΩ)

Mu Zhu U Waterloo


restricted Boltzmann machine



bull with a single binary top node h model for h given v islogistic regression

logP(h = 1 | v)P(h = 0 | v) = α + vTω

bull with several binary top nodes model for ht given hminust andv is also logistic regression

bull with odds ratio depending only on v

bull stack these in layers with top nodes for one layerbecoming bottom nodes for the next



bull estimating parameters becomes an optimization problemas well as a computational problembull natural gradient ascent η larr η + εiminus1(η)nablaη`(η v h)

bull Gaussian graphical model approximationto force sparse inverse Grosse 2015 ICML

bull example B Frey Infinite Genome Project


BrendanFreyTheInfiniteGenomesProject

Optimizationbull regularized maximum likelihood

maxθ`(θ y)minus Pλ(θ)

bull lasso penalty Pλ(θ) = λ||θ||1 is convex relaxation of λ||θ0||

bull many interesting penalties are non-convex

bull optimization routines may not find global optimum

bull Wainwright this may not matter if optimization error issmaller than statistical error


optimizationdistinction between statistical error θ minus θ and

optimization error θt minus θ

Loh amp Wainwright 2015 JMLR 2014 arxiv


Some common lsquostatisticalrsquo themesbull data carpentry ndash making data useable for analysisbull data visualization ndash extremely important communication

toolbull dimension reduction and regularization ndash geometry

topology algebra and analysisbull design of data collection ndash bigger isnrsquot necessarily betterbull networks ndash a prominent example of new types of data

not a rectangular array

bull optimization ndash statistics mathematics and computersciencebull model selection and inferencebull reproducibility and replicabilitybull training


Canadian Statistical Sciences InstituteThe purpose of CANSSI is to advance research in the statistical sciences inCanada by attracting new researchers to the field increasing the points ofcontact among researchers nationally and internationally and developingscientific collaborations with other disciplines and organizations

State Space Models for Fisheries Science



Marked Point Processes and Wildfire Modeling



Modern Spectrum Methods in Time Series Analysis



Computer and Physical Models in Earth Atmospheric andOcean Sciences



Statistical Inference for Complex Surveys



Copula Dependence Modeling theory and applications



New in 2016bull Joint Analysis of Neuro-imaging Databull Rare DNA Variants and Human Complex Traits


Models and likelihoodbull Model for the probability distribution of y given x

bull Density f (y | x) with respect to eg Lebesgue measure

bull Parameters for the density f (y | x θ) θ = (θ1 θd )

bull Data y = (y1 yn) sometimes independent

bull Likelihood function L(θ y) prop f (y θ) (y1 yn)

bull log-likelihood function `(θ y) = log L(θ y) + c

bull often θ = (ψ λ)

bull θ could have very large dimension d gt n

bull θ could have infinite dimension in principleE(y | x) = θ(x) lsquosmoothrsquo


Why likelihood









why likelihood



Important summaries

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2


= arg supθ`(θ y)



partθ2

∣∣∣∣θ






why likelihood




16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2






r(θ) = plusmnradic



approximate pivots

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2


r(θ) = plusmnradic

[2`(θ)minus `(θ)]





ij ui j=1ni i=1m


log-likelihood

`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui






















conditionalprod

j



CL(θ y) propnprod

i=1

prodjltj prime










Example Ising model

f (y θ) = exp(sum

(jk)isinE

θjkyjyk )1

Z (θ)j k = 1 K



sumk 6=j θjkyk )

exp(2yjsum



CL(θ) =nsum

i=1

Ksumj=1


Pλ(|θjk |)





`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)



(yT








qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω

















































Why likelihood









why likelihood



Important summaries

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2


= arg supθ`(θ y)



partθ2

∣∣∣∣θ






why likelihood




16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2






r(θ) = plusmnradic



approximate pivots

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2


r(θ) = plusmnradic

[2`(θ)minus `(θ)]





ij ui j=1ni i=1m


log-likelihood

`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui






















conditionalprod

j



CL(θ y) propnprod

i=1

prodjltj prime










Example Ising model

f (y θ) = exp(sum

(jk)isinE

θjkyjyk )1

Z (θ)j k = 1 K



sumk 6=j θjkyk )

exp(2yjsum



CL(θ) =nsum

i=1

Ksumj=1


Pλ(|θjk |)





`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)



(yT








qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω

















































why likelihood



Important summaries

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2


= arg supθ`(θ y)



partθ2

∣∣∣∣θ






why likelihood




16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2






r(θ) = plusmnradic



approximate pivots

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2


r(θ) = plusmnradic

[2`(θ)minus `(θ)]





ij ui j=1ni i=1m


log-likelihood

`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui






















conditionalprod

j



CL(θ y) propnprod

i=1

prodjltj prime










Example Ising model

f (y θ) = exp(sum

(jk)isinE

θjkyjyk )1

Z (θ)j k = 1 K



sumk 6=j θjkyk )

exp(2yjsum



CL(θ) =nsum

i=1

Ksumj=1


Pλ(|θjk |)





`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)



(yT








qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω

















































Important summaries

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2


= arg supθ`(θ y)



partθ2

∣∣∣∣θ






why likelihood




16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2






r(θ) = plusmnradic



approximate pivots

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2


r(θ) = plusmnradic

[2`(θ)minus `(θ)]





ij ui j=1ni i=1m


log-likelihood

`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui






















conditionalprod

j



CL(θ y) propnprod

i=1

prodjltj prime










Example Ising model

f (y θ) = exp(sum

(jk)isinE

θjkyjyk )1

Z (θ)j k = 1 K



sumk 6=j θjkyk )

exp(2yjsum



CL(θ) =nsum

i=1

Ksumj=1


Pλ(|θjk |)





`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)



(yT








qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω

















































why likelihood




16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2






r(θ) = plusmnradic



approximate pivots

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2


r(θ) = plusmnradic

[2`(θ)minus `(θ)]





ij ui j=1ni i=1m


log-likelihood

`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui






















conditionalprod

j



CL(θ y) propnprod

i=1

prodjltj prime










Example Ising model

f (y θ) = exp(sum

(jk)isinE

θjkyjyk )1

Z (θ)j k = 1 K



sumk 6=j θjkyk )

exp(2yjsum



CL(θ) =nsum

i=1

Ksumj=1


Pλ(|θjk |)





`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)



(yT








qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω


















































16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2






r(θ) = plusmnradic



approximate pivots

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2


r(θ) = plusmnradic

[2`(θ)minus `(θ)]





ij ui j=1ni i=1m


log-likelihood

`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui






















conditionalprod

j



CL(θ y) propnprod

i=1

prodjltj prime










Example Ising model

f (y θ) = exp(sum

(jk)isinE

θjkyjyk )1

Z (θ)j k = 1 K



sumk 6=j θjkyk )

exp(2yjsum



CL(θ) =nsum

i=1

Ksumj=1


Pλ(|θjk |)





`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)



(yT








qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω

















































approximate pivots

16 17 18 19 20 21 22 23

minus4

minus3

minus2

minus1

0


θθ

logminus

likel

ihoo

d

θθθθ


192 w2


r(θ) = plusmnradic

[2`(θ)minus `(θ)]





ij ui j=1ni i=1m


log-likelihood

`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui






















conditionalprod

j



CL(θ y) propnprod

i=1

prodjltj prime










Example Ising model

f (y θ) = exp(sum

(jk)isinE

θjkyjyk )1

Z (θ)j k = 1 K



sumk 6=j θjkyk )

exp(2yjsum



CL(θ) =nsum

i=1

Ksumj=1


Pλ(|θjk |)





`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)



(yT








qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω




















































ij ui j=1ni i=1m


log-likelihood

`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui






















conditionalprod

j



CL(θ y) propnprod

i=1

prodjltj prime










Example Ising model

f (y θ) = exp(sum

(jk)isinE

θjkyjyk )1

Z (θ)j k = 1 K



sumk 6=j θjkyk )

exp(2yjsum



CL(θ) =nsum

i=1

Ksumj=1


Pλ(|θjk |)





`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)



(yT








qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω




































































conditionalprod

j



CL(θ y) propnprod

i=1

prodjltj prime










Example Ising model

f (y θ) = exp(sum

(jk)isinE

θjkyjyk )1

Z (θ)j k = 1 K



sumk 6=j θjkyk )

exp(2yjsum



CL(θ) =nsum

i=1

Ksumj=1


Pλ(|θjk |)





`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)



(yT








qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω























































Example Ising model

f (y θ) = exp(sum

(jk)isinE

θjkyjyk )1

Z (θ)j k = 1 K



sumk 6=j θjkyk )

exp(2yjsum



CL(θ) =nsum

i=1

Ksumj=1


Pλ(|θjk |)





`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)



(yT








qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω



















































`(βΣ) =msum

i=1

(yT

i Xiβ minus12

log |Σ|

+ logintRk



uTi Σminus1uidui

)

variational approx

`(βΣ) gemsum

i=1

(yT

i Xiβ minus12

log |Σ|)



(yT








qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω





















































qj(θ)


qj(θ)


f (yj θ)



















































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω

































































































Closing Conference





Lisa Lix (Manitoba)







PROGRAM







GRADUATE COURSES

B I G DATA




JANUARY - JUNE 2015













Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω


























































Mu Zhu U Waterloo






logP(h = 1 | v)P(h = 0 | v) = α + vTω





















































logP(h = 1 | v)P(h = 0 | v) = α + vTω

















































Approximate Likelihoods - Statistical Inference, Learning ... · Approximate Bayesian Computation...

Documents

Transcript of Approximate Likelihoods - Statistical Inference, Learning ... · Approximate Bayesian Computation...