Gaussian Processes For Regression, Classification, and Prediction.

Author
hectorlynch 
Category
Documents

view
226 
download
0
Embed Size (px)
Transcript of Gaussian Processes For Regression, Classification, and Prediction.
Nonparametric Bayesian Models
Gaussian Processes For Regression, Classification, and Prediction
How Do We Deal With Many Parameters, Little Data?1. Regularizatione.g., smoothing, L1 penalty, drop out in neural nets, large K for Knearest neighbor2. Standard Bayesian approachspecify probability of data given weights, P(DW)specify weight priors given hyperparameter , P(W)find posterior over weights given data, P(WD, )With little data, strong weight prior constrains inference3. Gaussian processesplace a prior over functions directly rather than over model parameters, WProblems That GPs SolveOverfittingIn problems with limited data, the data themselves arent sufficient to constrain the model predictionsUncertainty in predictionWant to not only predict test values, but also estimate uncertainty in the predictionsProvide a convenient language for expressing domain knowledgePrior is specified over function spaceIntuitions are likely to be stronger in function space than parameter spaceE.g., smoothness constraints on function, trends over time, etc.Intuition
Figures from Rasmussen & Williams (2006)Gaussian Processes for Machine LearningPrior: all smooth functions are allowedGrey area: plausible value of f(x)
Observe 2 data points > only allowed functions are those that are consistent with the data
End up with a posterior over functions. Kind of like throwing out all functions that dont go through the data points.Posterior uncertainty shrinks near observations. Why? smoothness of functions points near the observations need to have values similar to those of the observations
4Important IdeasNonparametric modelcomplexity of model posterior can grow as more data are addedcompare to polynomial regressionPrior determined by one or more smoothness parametersWe will learn these parametersfrom the data
Gaussian Distributions: A ReminderFigures stolen from Rasmussen NIPS 2006 tutorial
Gaussian Distributions: A ReminderFigure stolen from Rasmussen NIPS 2006 tutorialMarginals of multivariate Gaussians are Gaussian
Gaussian Distributions: A ReminderConditionals of multivariate Gaussians are Gaussian
Stochastic ProcessGeneralization of a multivariate distribution to countably infinite number of dimensionsCollection of random variables G(x) indexed by xE.g., Dirichlet processx: parameters of mixture componentx: possible word or topic in topic modelsJoint probability over any subset of values of x is DirichletP(G(x1), G(x2), , G(xk)) ~ Dirichlet(.)
Gaussian ProcessInfinite dimensional Gaussian distribution, F(x)Draws from Gaussian Process are functionsf ~ GP(.)
xfnotationswitchGaussian ProcessJoint probability over any subset of x is GaussianP(f(x1), f(x2), , f(xk)) ~ Gaussian(.)Consider relation between 2 points on functionThe value of one point constrains the value of the second point
xf
x1x2Gaussian ProcessHow do points covary as a function of their distance?
xfx1x2x
fx1x2
Gaussian ProcessA multivariate Gaussian distribution is defined by a mean vector and a covariance matrix.A GP is fully defined by a mean function and a covariance functionExample
squared exponentialcovariance
l: length scaleInferenceGP specifies a prior over functions, F(x)Suppose we have a set of observations:D = {(x1,y1), (x2, y2), (x3, y3), , (xn, yn)}Standard Bayesian approachp(FD) ~ p(DF) p(F)If data are noise freeP(DF) = 0 if any P(DF) = 1 otherwise
Graphical Model Depictionof Gaussian Process Inference
square boxes are observedRed = training dataBlue = test data15GP Prior
where
GP Posterior Predictive Distribution
Posterior makes a prediction for each value along with the _Gaussian_ uncertainty16Observation NoiseWhat ifCovariance function becomes
Posterior predictive distribution becomes
DemoAndreas Geigers demo(Mike: run in firefox)A TwoDimensional Input Space
What About The Length Scale?
Note: uncertainty grows rapidly with distance from data points with short length scale20FullBlown SquaredExponential Covariance Function Has Three ParametersHow do we determine parameters?Prior knowledgeMaximum likelihood Bayesian inferencePotentially many other free parameterse.g., mean function
sigma_f determines range of y values, e.g., all values in [1, +1] vs. all values in [100, +100]
Bayesian inference is nice because you can impose prior on parameters, but its typically very expensive21Many Forms For Covariance Function
From GPML toolboxMany Forms For Likelihood FunctionWhat is likelihood function?noise free case we discussed
noisy observation case we discussed
With these likelihoods, inference is exactGaussian process is conjugate prior of GaussianEquivalent to covariance function incorporating noise
Customizing Likelihood Function To ProblemSuppose that instead of real valued observations y, observations are binary {0, 1}, e.g., class labels
Many Approximate Inference TechniquesFor GPsChoice depends on likelihood function
From GPML toolboxWhat Do GPs Buy Us?Choice of covariance and mean functions seem a good way to incorporate prior knowledgee.g., periodic covariance functionAs with all Bayesian methods, greatest benefit is when problem is data limitedPredictions with error barsAn elegant way of doing global optimizationNext class