Intro Reg Mod

Click here to load reader

  • date post

    18-Aug-2015
  • Category

    Documents

  • view

    215
  • download

    0

Embed Size (px)

description

summary

Transcript of Intro Reg Mod

Introduction to Regression Modeling,A SummaryBovasAbraham&JohannesLedolter2. SimpleLinearRegression2.1TheModely= + = 0 + 1x + Withnpairsofobservations(xi, yi)wherei=1, 2, . . . , n, wecancharacterizetheseobser-vationsasyi= 0 + 1xi + i2.1.1ImportantAssumptions1. xiaretakenasconstants,notrandomvariables,foralli = 1, 2, . . . , n2. E(i) = 0foralli = 1, 2, . . . , ni= E(yi) = 0 + 1xiforalli = 1, 2, . . . , n3. V (i) = 2foralli = 1, 2, . . . , n,i.e.,allobservationshavethesameprecision4. Cov(i, j) = 0 for all i = j, i.e., dierent errors i and j, and hence dierent responsesyiandyjareindependent2.1.2ObjectivesoftheAnalysis1. Canweestablisharelationshipbetweenyandx?2. Canwepredictyfromx?Towhatextentcanwepredictyfromx?3. Canwecontrolybyusingx?2.2EstimationofParameters2.2.1MaximumLikelihoodEstimationMaximumlikelihoodestimationselectstheestimatesoftheparameterssuchthatthelikeli-hoodfunctionismaximized. Aprobabilitydistributionforymustbespeciedifonewantstousethisapproach.Assumei N(0, 2). Thisimpliesthatyi N(0+ 1xi, 2). Thepdf fortheithresponseyiisp(yi|0, 1, 2) =122exp_122(yi01xi)2_andthejointpdfofy1, y2, . . . , ynisp(y1, . . . , yn|0, 1, 2) =_122_nexp_122n

i=1(yi01xi)2_1Treating this as a function of the parameters leads us to the likelihood function L(0, 1, 2|y1, . . . , yn),anditslogarithml(0, 1, 2|y1, . . . , yn) = n2ln (2) nln 122n

i=1(yi01xi)2TheMLEsof0, 1, 2maximizel(0, 1, 2).Maximizing the log-likelihood function wrt 0 and 1 is equivalent to minimizing S(0, 1) =

ni=1(yi01xi)2,whichisreferredtoasthemethodofleastsquares.2.2.2LeastSquaresEstimationOnewantstoobtainalinei= 0 +1xithatisclosesttothepoints(xi, yi). Theerrorsi= yii= yi01xishouldbeassmallaspossible. Oneapproachtoachievethisistominimizethefunctionbelowwrt0and1.S(0, 1) =n

i=12i=n

i=1(yii)2=n

i=1(yi01xi)2Solvingfortherst-orderconditionsleadstothenormalequations:n0 + (n

i=1xi)1=n

i=1yi(n

i=1xi)0 + (n

i=1x2i)1=n

i=1xiyiThesolutionstothenormalequationsarecalledtheLSEsof0and1.1=

ni=1(xi x)(yi y)

ni=1(xi x)2=sxysxx0= y 1 x2.3FittedValues,Residuals,andtheEstimateof2Theexpression yi= i=0 +1xiiscalledthettedvaluethatcorrespondstotheithobservationwithxiasthevaluefortheexplanatoryvariable. Thedierencebetweentheobservedvalueyiandthettedvalue i, yi i=ei, isreferredtoastheresidual. Itistheverticaldistancebetweentheobservationyiandtheestimatedline ievaluatedatxi.2.3.1ConsequencesoftheLeastSquaresFitLSEsset thederivativeof S(0, 1) equal tozero. Theequations, evaluatedat theleastsquaresestimates,S(0, 1)0= 2n

i=1[yi(0+ 1xi)] = 0 andS(0, 1)1= 2n

i=1[yi(0+ 1xi)]xi= 0implycertainrestrictions:2i.

ni=1ei= 0(F.O.C.wrt0)ii.

ni=1eixi= 0(F.O.C.wrt1)iii.

ni=1 iei= 0(fromresultsin(i)and(ii))iv. ( x, y)isapointontheline =0 +1xv. S(0, 1) =

ni=1e2iistheminimumofS(0, 1)2.3.2Estimationof2Maximizationofthelog-likelihoodfunctionwrt2leadstotheMLE 2=S(0, 1)n=

ni=1e2inThenumeratoriscalledtheresidualsumofsquares;itistheminimumofS(0, 1).TheLSEof2,alsocalledthemeansquareerror(MSE),iss2=S(0, 1)n 2=

ni=1e2in 2Theresidualsumofsquares,S(0, 1) =

ni=1e2i,consistsofnsquaredresiduals. However,theminimizationof S(0, 1)hasintroducedtwoconstraintsamongthesenresiduals; see(i)and(ii)givenpreviously. Hence, onlyn 2residualsareneededforthecomputation.Theremainingtworesidualscanalwaysbecalculatedfrom

ni=1ei=

ni=1eixi=0. Onesaysthattheresidualsumofsquareshasn 2degreesoffreedom.2.4PropertiesofLSEsLetuswritetheLSEof1inslightlymoreconvenientform:1=

ni=1(xi x)(yi y)

ni=1(xi x)2=

ni=1(xi x)yi y

ni=1(xi x)

ni=1(xi x)2=

ni=1(xi x)yi

ni=1(xi x)2=n

i=1ciyiwhereci=(xi x)

ni=1(xi x)2=(xi x)sxxTheconstantscihaveseveralinterestingproperties:i.

ni=1ci=

ni=1(xi x)/sxx= 0ii.

ni=1cixi=

ni=1xi(xi x)/sxx= 1iii.

ni=1c2i=

ni=1(xi x)2/s2xx= 1/sxxTheseresultscanbeusedtoderivetheexpectedvaluesandthevariancesoftheLSEs.32.4.1ExpectedValuesofLSEs1. E(1) = 1. So1isanunbiasedestimatorof1.E(1) =_n

i=1ciyi_=n

i=1ciE(yi) =n

i=1ci(0 + 1xi) = 0n

i=1ci + 1n

i=1cixi= 12. E(0) = 0. So0isalsounbiasedfor0.E(0) = E( y 1 x) = E( y) xE(1) = E(0 + 1 x) 1 x = 0 + 1 x 1 x = 03. E( 0) = E(0 +1x0) = 0 + 1x0= 0. Hence, 0isunbiasedfor0.4. E(s2) = 2canalsobeshown(SeeChapter4).2.4.2VariancesofLSEs1. V (1) = V (

ni=1ciyi) =

ni=1c2iV (yi) =

ni=1ci2= 2/sxx2. Firstshowthat0= y 1 x =n

i=1_yin_ xn

i=1(xi x)yisxx=n

i=1kiyiwhereki=1n x(xi x)sxxThen,V (0) =

ni=1k2i2= 2_1n+ x2sxx_3. ForthevarianceofV ( 0),wewrite 0=0 +1x0= y 1 x +1x0= y +1(x0 x)=n

i=1_yin+ (x0 x)(xi x)yisxx_=n

i=1diyiwheredi=1n+(x0 x)(xi x)sxxThenV ( 0) =

ni=1d2i2= 2_1n+(x0 x)2sxx_2.5InferencesabouttheRegressionParametersTheuncertaintyintheestimatescanbeexpressedthroughcondenceintervals,andforthisoneneedstomakeassumptionsaboutthedistributionoftheerrors. Assumeyi= 0 + 1xi + iwherei N(0, 2)42.5.1Inferenceabout11=

ni=1ciyiisalinearcombinationofyi N(i, 2). Therefore1 N_1,2sxx_or,afterstandardization,11/sxx N(0, 1)2is unknown and must be estimated. Substituting the factor with estimated error variances2,T=11s/sxx=11/sxx_(n 2)s22(n 2) tn2sinceZ=11/sxx N(0, 1) and U=(n 2)s22 2n2The estimated standard deviation of1, s/sxx, is also referred to as the standarderrorandtellsusaboutthevariabilityofthesamplingdistributionof1.Let t(1/2; n2) denote the 100(1/2) percentile of a t distribution with n2 degreesof freedom. Sincethetdistributionissymmetric, the100(/2)percentilet(/2; n 2)=t(1 /2; n 2). ThenthesamplingdistributionofTimpliesthatP_t_1 2; n 2_ 0. Wecallthematrixpositivesemideniteifthequadraticformy

Ay 0forallyandy

Ay = 0forsomevectory = 0.Wecall thesymmetricmatrixAnegativedeniteifforall vectorsy =0thequadraticform y

Ay < 0. We call the matrix negative semidenite if the quadratic form y

Ay 0 forallyandy

Ay = 0forsomevectory = 0.Orthogonal Matrix AsquarematrixAiscalledorthogonalifAA

= I. SinceA

= A1istheinverseof A, itfollowsthatA

A=I. Hence, anorthogonal matrixsatisesAA

=A

A = I. Therowsofanorthogonalmatrixaremutuallyorthogonalandthelengthoftherowsisone. ThesamecanbesaidaboutthecolumnsofA. Furthermore,thedeterminant|A| = 1. Thisfollowssince |AA

| = |A||A

| = |A|2= |I| = 1.Trace of a Square Matrix The trace of a square mm matrix A is dened as the sum ofitsdiagonalelements;thatis,tr(A) =

mi=1aii. Thedenitionimpliesthattr(A) = tr(A

),tr(A + B) =tr(A) + tr(B). Providedthat thematrices C, D, andEareconformable,tr(CDE) = tr(ECD) = tr(DEC). Conformablemeansthatthedimensionsofthematricesaresuchthatalltheseproductsaredened.IdempotentMatrix AsquarematrixAisidempotentifAA=A. Thedeterminantofanidempotentmatrixiseither0or1. Therankof anidempotentmatrixisequal tothesumofitsdiagonalelements.Vector Space A vector space Vis the set of all vectors that is closed under addition andmultiplication by a scalar and that contains the null vector 0. A set of linearly independentvectors in Rn, x1, x2, . . . , xnis said to span a vector space of dimension n; any other memberofthisvectorspacecanbegeneratedthroughlinearcombinationsofthesevectors. Wecallthissetabasisofthevectorspace. Basisvectorsarenotunique. Sometimesitisusefultoworkwithorthonormalbasisvectors. Thatis, vectorsu1, u2, . . . , unthatareorthogonal(u

iuj= 0fori = j)andhavelengthone(u

iui= 1). Suchorthonormalbasisvectorsalways12exist,andtheGram-Schmidtorthogonalizationcanconstructthemfromagivensetoflinearlyindependentbasisvectorsx1, x2, . . . , xn.SubspaceofRnConsiderk0areknownspeciedweights. ThiscriterionisequivalenttotheoneforGLS,withV1adiagonalmatrixhavingdiagonalelementswi. TheWLSestimatorisgivenbyWLS=_n

i=1wixix

i_1_n

i=1wixiyi_withvarianceV (WLS) = 2_n

i=1wixix

i_1265. SpecicationIssuesinRegressionModels5.1ElementarySpecialCases5.1.1One-sampleProblemSupposey1, . . . , ynobservationsaretakenunderuniformconditionsforastableprocesswithmeanlevel0.yi= 0 + iwith E(yi) = 0InthiscaseE(y) = Xwherey =__y1...yn__; X=__1...1__; = 05.1.2Two-sampleProblemSupposey1, . . . , ymaretakenunderonesetof conditions(standardprocess), whereastheremaining nm observations ym+1, . . . , ynare taken under a dierent set of conditions (newprocess). Let1denotethemeanof thestandardprocessand2denotethemeanof thenewprocess. Thenyi=_1 + ifor i = 1, . . . , m2 + ifor i = m + 1, . . . , nThiscanalsobewrittenasyi= 1xi1 + 2xi2 + iwherexi1=1ifi=1, . . . , mand0ifi=m + 1, . . . , nandxi2=0ifi=1, . . . , mand1ifi = m + 1, . . . , n. Inmatrixform,y = X +whereX=__1 0......1 0 0 1......0 1__and =__12__Ourinterestistotestwhetherthetwoprocesseshavethesamemean,i.e.,1= 2.Equivalently, wecanwrite2=1 + and=2 1, thedierenceof theprocessmeans. Thenyi= 1 + xi2 + iory = X +where,inthiscase, = (1, )

. Ourhypothesisis= 21= 0.275.1.3PolynomialModelsIfoneassumesthataprocessisquadraticinanexplanatoryvariable,yi= 0 + 1xi + 2x2i+ iorE(y) = XwhereX=__1 x1x21.........1 xnx2n__; =__012__5.2SystemsofStraightLinesSupposeyiisexpectedtochangelinearlywithacontinuousexplanatoryvariabletiandisobservedundertwodierentprocessesfori = 1, . . . , mandi = m + 1, . . . , n.Casea) Theprocesshasequaleectatalllevelsofti:E(yi) = 0 + 1ti + 2xi=_0 + 1tifor i = 1, . . . , m(0 + 2) + 1tifor i = m + 1, . . . , nThemodelrepresentstwoparallellineswhere2representsthechangeduetoprocess.Caseb) Theprocesshasaneectthatchangeswiththelevelofti:E(yi) = 0 + 1ti + 2xi + 3tixi=_0 + 1tifor i = 1, . . . , m(0 + 2) + (1 + 3)tifor i = m + 1, . . . , nGraphically, this model has 2 straight lines with dierent slopes and dierent intercepts. Totestwhethertheprocesshasanyeect, wetestthehypothesis2=3=0. Atestofjust3= 0testwhethe