Bayesian methods for parameter estimation and model comparison Bayesian methods for parameter...

Bayesian methods for parameter estimation and
model comparison
Carson C Chow, LBM, NIDDK, NIH
Monday, April 26, 2010

Task: Fit a model (ODE, PDE,...) to data - estimate parameters

t
y

dy
dt = f(t; θ)
t
y

dy
dt = f(t; θ)
t
y
y(t) = g(t|y(0), θ)

dy
dt = f(t; θ)
t
y y
t
y(t) = g(t|y(0), θ)

dy
dt = f(t; θ)
t
y y
t
y = at + b
y(t) = g(t|y(0), θ)

Questions:

Questions: What algorithm to use?

Questions: What algorithm to use?
How good is the fit?

Questions:
Is there a better model?
What algorithm to use?

Questions:
Sensitivity?

Questions:
Answer: Use Bayesian inference and MCMC
Sensitivity?

Questions:
Sensitivity?
Answer: Invert vs. infer

Bayesian inference
Frequentist: Probability is a frequency (of a random variable)

Bayesian inference
Bayesian: Probability is a measure of uncertainty

Bayesian inference
Bayesian: Probability is a measure of uncertainty
Jaynes: Probability is extended logic

Bayesian inference
Models and parameters have probabilities
Anything can be assigned a probability
Makes everything straightforward

D = Data θ = Parameters
Parameter estimation y(θ,t): solution
i.e. Maximum likelihood estimation
θ that maximizes likelihood
∑
i
(Di − yi(θ))2e.g. Minimize mean square error
P (D|θ) ∝ exp (
∑
i
− (Di − yi(θ)) 2
2σ2
)

Statistics
But in frequentist statistics models and parameters cannot have probabilities
Confidence intervals of a parameter are with respect to sampling errors
“Natural” thing to do is to find the probability distribution for a model or parameters

Bayes theorem
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
P (θ|D) = P (D|θ)P (θ) P (D)
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
Prior P (θ|D) = P (D|θ)P (θ)
P (D)
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
Prior
Likelihood
P (θ|D) = P (D|θ)P (θ) P (D)
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
Prior
Likelihood
P (θ|D) = P (D|θ)P (θ) P (D)
Evidence
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
PriorPosterior
Likelihood
P (θ|D) = P (D|θ)P (θ) P (D)
Evidence
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

Bayes theorem
Bayesian inference reduces statistics to one equation
PriorPosterior
Likelihood
P (θ|D) = P (D|θ)P (θ) P (D)
Evidence
P (θ, D) = P (θ|D)P (D) = P (D|θ)P (θ)

P (θ|D) = P (D|θ)P (θ) P (D)
P (θ|D) ∝ P (D|θ)P (θ)
P (D) = ∫
P (D|θ)P (θ)dθ
∝ exp (
∑
i
− (Di − yi(θ)) 2
2σ2
) y(θ,t): solution

P (θ|D) = P (D|θ)P (θ) P (D)
P (θ|D) ∝ P (D|θ)P (θ)
P (D) = ∫
P (D|θ)P (θ)dθ
∝ exp (
∑
i
− (Di − yi(θ)) 2
2σ2
) y(θ,t): solution
Probability of parameter given the data

0 20 40 60 80 100−2
0
2
4
6
8
10
12
0 20 40 60 80 100−2
0
2
4
6
8
10
12
t
y
Model as exponential decay with Gaussian noise
Posterior probability
Likelihood
Model
Data
P(y|b, t) ∝ ∏ t
e−(yt−10exp(−bt)) 2/2
P (b|y) ∝ P (y|b, t)P (b)
Example

0 0.025 0.05 0.075 0.10
0.5
1
1.5
2
b 0 20 40 60 80 100−2
0
2
4
6
8
10
12
0 20 40 60 80 100−2
0
2
4
6
8
10
12
t
y
Likelihood
true b=0.05
Model
Data
Prior Posterior
P(y|b, t) ∝ ∏ t
e−(yt−10exp(−bt)) 2/2
P (b|y) ∝ P (y|b, t)P (b)
Example

0 20 40 60 80 100−2
0
2
4
6
8
10
12
0 20 40 60 80 100−2
0
2
4
6
8
10
12
t
y
0 0.025 0.05 0.075 0.10
0.2
0.4
0.6
0.8
1
b
Likelihood
true b=0.05
Model
Data
P(y|b, t) ∝ ∏ t
e−(yt−10exp(−bt)) 2/2
P (b|y) ∝ P (y|b, t)P (b)
Example

Sensitivity analysis
− [
∂2
∂θ2 lnP (D|θ)
]
max
Deviation around M.L. Fisher information
I(θ) = −E [
∂2
∂θ2 lnP (D|θ)
]

Sensitivity analysis

Bayesian methods for parameter estimation and model comparison Bayesian methods for parameter...

Documents

Transcript of Bayesian methods for parameter estimation and model comparison Bayesian methods for parameter...

Bayesian large-scale multiple regression with summary ... xiangzhu/JSM_ Bayesian large-scale

Introduction to Bayesian Statistics - 3 milotti/Didattica/...¢ Introduction to Bayesian Statistics

IRT Parameter Estimation - GitHub Pages 2021. 5. 13.¢ MLE in IRT MLE in IRT Therefore,maximize f 3

A New Scoring Function for Bayesian Network Structure ... ... 2. Bayesian Networks (BNs) and Structure

Regularization parameter estimation for underdetermined ... rosie/mypapers/ Regularization parameter

Parameter Estimation for a Stochastic Volatility Model ... ewald/004-Parameter... Stochastic volatility

Parametric Density Estimation: Bayesian Estimation. Na¯ve ... rita/uml_course/add_mat/PDE_Bayesian_NB.pdf

Parameter estimation class 5

Sparse Nonlinear Regression: Parameter Estimation and ... computational and statistical theory for nonconvex

Modeling the Thrust Regulator of a Liquid Rocket Engine Rocket Engine. ... Principles Parameter Estimation

Dynare Bayesian Estimation - Wouter den Bayesian Estimation Wouter J. Den Haan London School of Economics c 2011 by Wouter J. Den Haan August 19, 2011

Regularization Parameter Estimation for rosie/mypapers/NotesMathPaper_v8.pdf¢ Regularization Parameter

Regularization Parameter Estimation for rosie/mypresentations/ Regularization Parameter Estimation

Distributed Bayesian Learning with Stochastic Natural ...?parameter server: â€¢ parameter x worker: â€¢ x i = x â€¢ updates to x iâ€™ â€¢ returns ”x i = x

Introduction to (Bayesian) Inference 2019-01-22¢ Introduction to (Bayesian) Inference Frank Schorfheide

Idiot's guide to... General Linear Model fMRI Elliot Freeman, ICN. fMRI model, Linear Time Series, Design Matrices, Parameter estimation,

Cosmology Hubble parameter Density Parameter Deceleration Parameter Density Parameter خ© o) G H c t

Copula Parameter Estimation by ML and MD Estimators or IFM method) ... Gumbel (¸=1.1,1.2,,2.9) Total of 10 estimators. Gregor Wei â€“ Copula Parameter Estimation: A Simulation

MODELLING AND PARAMETER ESTIMATION IN PET Vesa Oikonen Turku PET Centre 2004-06-03

Bayesian analysis of genetic population structure using ... Bayesian analysis of genetic population