Bayesian methods for parameter estimation and model comparison · PDF file Bayesian methods...

Bayesian methods for parameter estimation and
model comparison
Carson C Chow, LBM, NIDDK, NIH
Monday, April 26, 2010

Task: Fit a model (ODE, PDE,...) to data - estimate parameters

t
y

dy
dt = f(t; θ)
t
y

dy
dt = f(t; θ)
t
y
y(t) = g(t|y(0), θ)

dy
dt = f(t; θ)
t
y y
t
y(t) = g(t|y(0), θ)

dy
dt = f(t; θ)
t
y y
t
y = at + b
y(t) = g(t|y(0), θ)

Questions:

Questions: What algorithm to use?

Questions: What algorithm to use?
How good is the fit?

Questions:
Is there a better model?
What algorithm to use?

Questions:
Sensitivity?

Questions:
Answer: Use Bayesian inference and MCMC
Sensitivity?

Questions:
Sensitivity?
Answer: Invert vs. infer

Bayesian inference
Frequentist: Probability is a frequency (of a random variable)

Bayesian inference
Bayesian: Probability is a measure of uncertainty

Bayesian inference
Bayesian: Probability is a measure of uncertainty
Jaynes: Probability is extended logic

Bayesian inference
Models and parameters have probabilities
Anything can be assigned a probability
Makes everything straightforward

D = Data θ = Parameters
Parameter estimation y(θ,t): solution
i.e. Maximum likelihood estimation
θ that maximizes likelihood
∑
i
(Di − yi(θ))2e.g. Minimize mean square error
P (D|θ) ∝ exp (
∑
i
− (Di − yi(θ)) 2
2σ2
)

Statistics
But in frequentist statistics models and parameters cannot have probabilities
Confidence intervals of a parameter are with respect to sampling errors
“Natural” thing to do is to find the probability distribution for a model or parameters

Bayes theorem
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
P (θ|D) = P (D|θ)P (θ) P (D)
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
Prior P (θ|D) = P (D|θ)P (θ)
P (D)
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
Prior
Likelihood
P (θ|D) = P (D|θ)P (θ) P (D)
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
Prior
Likelihood
P (θ|D) = P (D|θ)P (θ) P (D)
Evidence
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
PriorPosterior
Likelihood
P (θ|D) = P (D|θ)P (θ) P (D)
Evidence
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
Bayesian inference reduces statistics to one equation
PriorPosterior
Likelihood
P (θ|D) = P (D|θ)P (θ) P (D)
Evidence
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

P (θ|D) = P (D|θ)P (θ) P (D)
P (θ|D) ∝ P (D|θ)P (θ)
P (D) = ∫
P (D|θ)P (θ)dθ
∝ exp (
∑
i
− (Di − yi(θ)) 2
2σ2
) y(θ,t): solution

P (θ|D) = P (D|θ)P (θ) P (D)
P (θ|D) ∝ P (D|θ)P (θ)
P (D) = ∫
P (D|θ)P (θ)dθ
∝ exp (
∑
i
− (Di − yi(θ)) 2
2σ2
) y(θ,t): solution
Probability of parameter given the data

0 20 40 60 80 100−2
0
2
4
6
8
10
12
0 20 40 60 80 100−2
0
2
4
6
8
10
12
t
y
Model as exponential decay with Gaussian noise
Posterior probability
Likelihood
Model
Data
P(y|b, t) ∝ ∏ t
e−(yt−10exp(−bt)) 2/2
P (b|y) ∝ P (y|b, t)P (b)
Example

0 0.025 0.05 0.075 0.10
0.5
1
1.5
2
b 0 20 40 60 80 100−2
0
2
4
6
8
10
12
0 20 40 60 80 100−2
0
2
4
6
8
10
12
t
y
Likelihood
true b=0.05
Model
Data
Prior Posterior
P(y|b, t) ∝ ∏ t
e−(yt−10exp(−bt)) 2/2
P (b|y) ∝ P (y|b, t)P (b)
Example

0 20 40 60 80 100−2
0
2
4
6
8
10
12
0 20 40 60 80 100−2
0
2
4
6
8
10
12
t
y
0 0.025 0.05 0.075 0.10
0.2
0.4
0.6
0.8
1
b
Likelihood
true b=0.05
Model
Data
P(y|b, t) ∝ ∏ t
e−(yt−10exp(−bt)) 2/2
P (b|y) ∝ P (y|b, t)P (b)
Example

Sensitivity analysis
− [
∂2
∂θ2 lnP (D|θ)
]
max
Deviation around M.L. Fisher information
I(θ) = −E [
∂2
∂θ2 lnP (D|θ)
]

Sensitivity analysis

Bayesian methods for parameter estimation and model comparison · PDF file Bayesian methods...

Documents

Transcript of Bayesian methods for parameter estimation and model comparison · PDF file Bayesian methods...

Faidon Panagiotopoulos - Bayesian Network collision

BAYESIAN namrata/EE527_Spring08/l4.pdfآ Sequential Bayesian Idea Suppose that we have observed x 1

Bayesian Models for Astronomy

Some DIC slides - Cambridge University .DIC slides 2006 Bayesian model comparison using DIC â€¢ Natural

Tolerance levels for the Approximate Bayesian levels for the Approximate Bayesian Computation ... composed with a covariance ... Tolerance levels for the Approximate Bayesian Computation

Bayesian Updating: Continuous Priors 1/01/2017 ¢ Example of Bayesian updating so far. Three types of

Cosmology Hubble parameter Density Parameter Deceleration Parameter Density Parameter خ© o) G H c t

Bayesian analysis of genetic population structure using BAPS: 2008-08-16¢ Bayesian analysis of genetic

1 Motivation. - UW Faculty Web 1 Motivation. â€¢ Bayesian discrete choice models â€¢ Bayesian approach

Bayesian perspective on QCD global analysis ... 1/17 Bayesian perspective on QCD global analysis Nobuo

Stochastic Volatility Models: Bayesian Framework

Distributed Bayesian Learning with Stochastic Natural ... fileparameter server: • parameter x worker: • x i = x • updates to x i’ • returns Δx i = x i’ - x i y 1i y 2i

Precipitation Estimation Greece

Symbol TIMING estimation

Bayesian Parameter Inference in State-Space Models using ... ... Bayesian Parameter Inference in State-Space

Bab 6 PENAKSIRAN PARAMETER

Bayesian Methods in Positioning Applications

Regularization parameter estimation for underdetermined ... rosie/mypapers/ Regularization parameter

Parameter estimation class 5

Bayesian Interpretations of Regularization - mit. 9.520/spring09/Classes/class15-bayes.pdf · PDF fileThe Plan Bayesian estimation basics Bayesian interpretation of ERM Bayesian interpretation