Nonlinear Regression - Simon Fraser University

Nonlinear RegressionChapter 2 of Bates and Watts

Friday, June 12, 2009

So far we’ve considered linear models

Here the expectation surface is a plane spanning a subspace of the observation space.

Our expectation surface has a flat shape .

Our model has a linear shape

When things are flat projections are easy to do and understand.

nonlinear least squares

Gauss-Newton

Geometry

Matlab Functions

The N observations are modeled by

Where might be:

Yn = f (Xn ,θ) + Zn

f (Xn ,θ) Y0e−αX + β /α

θ1Xθ2 + X

11+ e−β0 −β1X

We will define the expectation function

And the observation process

And again assume

η(θ) = f (Xn ,θ)

Yn = η(θ) + Zn

E(Z ) = 0 var(Z ) = E(Z 'Z ) = σ 2I

Lipoprotein problem from Bates and Watts

% of the original tracer observations by time (days)

Consider the 1 compartment model:

Although we could transform the data into the model:

We will ignore this linearization and use the nonlinear model as a simple example

η(θ) = f (Xn ,θ)η(θ) = θ1e

−θ2X

log(Yn ) = log(θ1) −θ2Xn + errorn

The response surface is defined by changing values of the single parameter in

Our expectation surface is then a one dimensional manifold

η(θ) = 100e−θ2X

Lipoprotein observation and response surface

Again we wish to minimize the residual vector

But there may be more than 1 location where the angle between the residual vector and the tangent to the expectation surface are orthogonal

The problem is even worse in the 2 dimensional model where initial conditions are unknown

η(θ) = f (Xn ,θ)η(θ) = θ1e

−θ2X

Lipoprotein observation and 2 parameter response surface

lines show the response surface where theta 1 is fixed but altering theta2dot connecting spreading lines making the kinks show changes in theta1 with fixed theta 2

steps:

1. Find the point on the expectation surface closest to

2. Find corresponding to this point

η = YY

Use a linear approximation to the expectation surface to iteratively improve an initial guess for

We will use a linear Taylor approximation to the expectation surface, and then use linear regression methods.

We will need to keep updating the Taylor expansion and keep updating our estimate.

Gauss-Newton Method

θ (0)

The expectation function

For a single observation the Taylor expansion for the p dimensional parameter vector:

Including all observations we get

η(θ) = θ1e−θ2X

η(θ) ≈ η(θ (0) )+ ∂η(θ)∂θk θ=θ (0 )

θk −θ(0)⎡⎣ ⎤⎦

η(θ) ≈ η(θ (0) )+V (0) (θ −θ (0) )

V (0) =

∂η(X1,θ)∂θ1

... ∂η(X1,θ)∂θ p

... ... ...∂η(Xn ,θ)

∂θ1... ∂η(Xn ,θ)

∂θ p

⎢⎢⎢⎢⎢⎢

⎥⎥⎥⎥⎥⎥θ=θ (0 )

The model η(θ) = θ1e−θ2X

V (0) = e−θ2X −θ1Xe−θ2X⎡

⎣⎤⎦ θ=θ (0 )

The fit to the observations on the response surface and the Gauss Newton path

The first step jump really far away from the region where the linear Taylor approximation is valid.

We can improve the Gauss Newton algorithm by enforcing the condition

This means that at each step we have to adjust the step size ∂ so that it doesn’t take us to a worse location in the response surface

SSE(θ (i+1) ) < SSE(θ (i ) )

We adjust the algorithm by halving ∂ if the SSE condition is not met and trying again.

1. Approximate the expectation surface by an expectation plane at the current value

2. generate a residual vector

3. Project the residual onto the tangent plane to get new value of expectation surface

4. Map the move to through the linear approximation to get a step

5. move to the point on the actual expectation surface

Geometryη(θ (0) )η(θ (0) )

z = y −η(θ (0) )

η(θ (1) )

η(θ (1) )δ

η(θ (0) + δ )Friday, June 12, 2009

Gauss Newton Convergence

In Matlab code I told it when to stop

Ideal convergence is based on the angle of the residual vector

Inference

Inference is based on a linear approximation to the expectation surface

then we project a disk onto the approximated expectation plane

Then transform the values back to the parameter space to get an inference region

Other IntervalsMarginal intervals:

Use the same linear approximation and apply the linear marginal inference regions

Inference bands for expected response

Replace in linear case with and replace the matrix by and the derivative vector with the corresponding derivative matrix entry

x0β f (x0 ,θ)X V

1. Why is it so wide near time 0?

2. Why is it so narrow near time 10?

3. Why is it so wide generally everywhere?

4. why is it wavy?Friday, June 12, 2009

Using 3 or 12 observationsFriday, June 12, 2009

[beta,r,J,COVB,mse] = nlinfit(X,y,fun,beta0)

X is the same as our X

y is a vector of observations

fun is a function handle for the nonlinear function, the function takes inputs (beta,X) and gives Yhat as an output

beta0 is the starting point for the iterative estimates

Matlab NLS function

beta is NLS point estimate

r is a residual vector

J is the Jacobian of the function fun evaluated at each observation: this is our V

COVB is the estimated covariance matrix for parameters

mse is an estimate of the error variance term: our s2

Matlab NLS function

Marginal Confidence intervals for parameters

CI_NLS = nlparci(beta,resids,'jacobian',J)

beta is the parameter estimate output from nlinfit

resids is the residual vector output from nlinfit

J is the Jacobian output from nlinfit

Confidence interval for a new value or x

[ypred,delta] = nlpredci(fun,x,beta,resid,'jacobian',J)

beta is the parameter estimate output from nlinfit

resids is the residual vector output from nlinfit

J is the Jacobian output from nlinfit

ypred ± delta is the CI

Or do it all using a GUI

nlintool(X,Y,@lipomodel_Xlast,theta)

Give it your data, the model and a starting point and it will give you point estimates for parameters

confidence bounds for single points, the response function, and future observations

bounds - simultaneous are for all points simultaneously, non-simultaneous are for individual points

The downside to the Matlab built-in functions:

They do not produce joint interval estimate ellipses for parameters.

We must use the QR decomposition for this.

Simplest method:

Combine NLS and ODE solvers, Just use a solver built into fun

Nonlinear Least Squares for ODEs

= γ V −V3 / 3+ R( )dRdt

= −1γ

βR +α −V( )Friday, June 12, 2009

= γ V −V3 / 3+ R( )dRdt

= −1γ

βR +α −V( )Friday, June 12, 2009

Inputs to fun must be ß and X but in our case X is time. X must a vector with the same length as data Y

The output of fun must be the fit to the data, (the expectation surface at the current parameter point), we have to make it a vector.

function [yfit] = NLS_FhN(pars,time) odefn = @fhnfunode;odeopts = odeset('RelTol',1e-13);[junk,path] = ode45(odefn,time(1:401),pars(4:5),odeopts,pars(1:3)); yfit=[path(:,1);path(:,2)];

To run this program:clearload(strcat('/Volumes/iamdavecampbell/MCMC many times/',... '1000 random data sets/FhN MCMC data set_38.mat'))time=[time;time]; % stack timeY=[Ydata(:,1);Ydata(:,2)]; % stack databeta0=[.5,.5,1,-.5,.5];fun=@NLS_FhN [beta,r,J,COVB,mse] = nlinfit(time,Y,fun,beta0);

To run this program:

[Ypred,delta] = nlpredci(fun,time,beta,r,'jacobian',J)n=length(Ydata(:,1));CI_Y=repmat(Ypred,1,3)+[-delta,zeros(2*n,1),delta]; plot(time(1:n),CI_Y(1:n,:),'b',time(1:n),Ydata(:,1),'.b',... time(1:n),CI_Y(n+1:2*n,:),'k',time(1:n),Ydata(:,2),'.k')

CI for the expectation functionFriday, June 12, 2009

To run this program:CI_NLS = nlparci(beta,r,'jacobian',J)[CI_NLS(:,1),beta',CI_NLS(:,2)]

>> [CI_NLS(:,1),beta',CI_NLS(:,2)]

0.1718 0.1958 0.2199 0.0599 0.1682 0.2764 2.9338 2.9744 3.0150 -1.1795 -1.0844 -0.9893 0.9081 0.9896 1.0712

nlinfit uses a numerical Jacobian to produce the Taylor approximation to the response surface.

There are other functions that let you include the formula for the Jacobian: lsqnonlin

Nonlinear Regression - Simon Fraser University

Documents

Transcript of Nonlinear Regression - Simon Fraser University

Simon N. Wood

Fluid Statics - SFU.ca - Simon Fraser Universitymbahrami/ENSC 283/Notes/Fluid Statics.docx · Web viewM. Bahrami Fluid Mechanics (S 09)Fluid statics 13 Fluid Statics When the fluid

Simon Critchley Interview with the Athen's Voice

EG4321/EG7040 NonlinearControl Dr. MattTurner · Books Nonlinear Control “Nonlinear Systems”, H. Khalil. The classic nonlinear control textbook. Well-written and comprehensive.

3.2 Rossby Adjustment: Weakly Nonlinear Behavior. Personal... · 2007. 4. 20. · 3.2 Rossby Adjustment: Weakly Nonlinear Behavior. The nonlinear terms neglected in Gill’s solution

Calculating cyclotomic polynomials - Simon Fraser University · algorithms were used to calculate cyclotomic polynomials of large height and length. In particular we ﬁnd cyclotomic

SIMON CRITCHLEY · SIMON CRITCHLEY «Ένα βιβλΙο πο!) ftπορεl νασας σώσει τη ζωή!» Slmon May, Nezυ Humanist

Nonlinear Regression, Nonlinear Least Squares, and ...change a nonlinear relationship into a linear one (in these data, replacing population by its cube-root linearizes the plot, as

Simon Castle on FM

Εταιρικό Προφίλ της Simon Technologies

Nonlinear Resistors

abratkiewicz simon, Expo, 2014

Georg Simon Ohm (1789-1854)

Simon RABARIJOELY

Sulfur sources and magmatic sulfide mineralization in the Fraser … · Sulfur sources and magmatic sulfide mineralization in the Fraser Zone Alex Walker, Katy Evans, Christopher

KVL Example Resistor Voltage Divider - Simon Fraser …glennc/e220/e220l4b.pdf• Conductance equation for I is equivalent of voltage divider eqn ... • If know the voltages then

Abstract. - SFU.ca - Simon Fraser Universitymohar/Papers/Spec.pdfdiameter, mean distance, and bandwidth-type parameters of a graph. Some new results and generalizations are added.

Simon Ides

Simon Castle on FM_2

Fitting ARIMA p,d,q models to data - Simon Fraser Universitypeople.stat.sfu.ca/~lockhart/richard/804/06_1/lectures/Model... · Fitting ARIMA(p,d,q) models to data Fitting Ipart easy: