Lecture 19 Multiple (Linear) Regression - Statistical Science Lecture 19 Multiple (Linear)...

download Lecture 19 Multiple (Linear) Regression - Statistical Science Lecture 19 Multiple (Linear) Regression

of 30

  • date post

    20-Oct-2019
  • Category

    Documents

  • view

    3
  • download

    0

Embed Size (px)

Transcript of Lecture 19 Multiple (Linear) Regression - Statistical Science Lecture 19 Multiple (Linear)...

  • Lecture 19 Multiple (Linear) Regression

    Thais Paiva STA 111 - Summer 2013 Term II

    August 1, 2013

    1 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • Lecture Plan

    1 Multiple regression

    2 OLS estimates of β and α

    3 Interpretation

    2 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • Linear regression

    A study on depression:

    The response variable is Depression, which is the score on a self-report depression inventory

    Predictors:

    Simplicity is the score that indicates a subjects need to see the world in black and white Fatalism is the score that indicates the belief in the ability to control ones own destiny.

    Depression is thought to be related to simplicity and fatalism

    3 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • Linear regression

    Patient Depression Simplicity Fatalism 1 0.42 0.76 0.11 2 0.52 0.73 1.00 3 0.71 0.62 0.04 4 0.66 0.84 0.42 5 0.54 0.48 0.81 6 0.34 0.41 1.23 7 0.42 0.85 0.30 8 1.08 1.50 1.20 9 0.36 0.31 0.66

    10 0.92 1.41 0.85 11 0.33 0.43 0.42 12 0.41 0.53 0.07 13 0.83 1.17 0.30 14 0.65 0.42 1.09 15 0.80 0.76 1.13

    4 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • Depression data

    0.0 0.5 1.0 1.5 2.0 2.5 3.0

    0. 0

    0. 5

    1. 0

    1. 5

    2. 0

    2. 5

    3. 0

    0.0

    0.5

    1.0

    1.5

    2.0

    2.5

    Simplicity

    Fa ta

    lis m

    D ep

    re ss

    io n

    ● ●

    ● ●

    ● ●

    ●●

    ● ●

    ●●

    ● ●

    ● ●

    ● ●

    ●●

    ● ●

    ●● ●

    ● ●

    ● ●

    ● ●

    5 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • Depression data

    0.0 0.5 1.0 1.5 2.0 2.5 3.0

    0. 0

    0. 5

    1. 0

    1. 5

    2. 0

    2. 5

    3. 0

    0.0

    0.5

    1.0

    1.5

    2.0

    2.5

    Simplicity

    Fa ta

    lis m

    D ep

    re ss

    io n

    ● ●

    ● ●

    ● ●

    ●●

    ● ●

    ●●

    ● ●

    ● ●

    ● ●

    ●●

    ● ●

    ●● ●

    ● ●

    ● ●

    ● ●

    6 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • Depression data - residuals

    0.0 0.5 1.0 1.5 2.0 2.5 3.0

    0. 0

    0. 5

    1. 0

    1. 5

    2. 0

    2. 5

    3. 0

    0.0

    0.5

    1.0

    1.5

    2.0

    2.5

    Simplicity

    Fa ta

    lis m

    D ep

    re ss

    io n

    ● ●

    ● ●

    ● ●

    ●●

    ● ●

    ●●

    ● ●

    ● ●

    ● ●

    ●●

    ● ●

    ●● ●

    ● ●

    ● ●

    ● ●

    7 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • Assumptions for multiple linear regression

    Yi = α + β1X1i + β2X2i + . . .+ βpXpi + εi

    Just as with simple linear regression, the following have to hold:

    1 Constant variance (also called homoscedasticity)

    V (εi ) = σ 2 for all i = 1, . . . , n, for some σ2

    2 Linearity

    3 Independence

    εi ⊥ εj for all i , j = 1, . . . , n, i 6= j

    8 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • Interpretation of the β’s

    Yi = α + β1X1i + β2X2i + . . .+ βpXpi + εi

    βj is the average effect on Y of increasing Xj by one unit, with all Xk 6=j held constant

    This is sometimes referred to as the effect of Xj after “controlling for” Xk 6=j

    So βsimplicity is the average effect of simplicity on depression after controlling for fatalism

    9 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • Always plot residuals

    ● ●

    ● ●

    ● ●

    ● ●

    ● ●

    ● ●

    ● ●

    ● ●

    ●●

    0.5 1.0 1.5 2.0 2.5 3.0

    − 0.

    5 0.

    0 0.

    5 1.

    0

    simplicity

    ε̂

    ● ●

    ●●

    ● ●

    ● ●

    ● ●

    ● ●

    ● ●

    ●●

    ● ●

    0.0 0.5 1.0 1.5 2.0

    − 0.

    5 0.

    0 0.

    5 1.

    0 fatalism

    ε̂

    10 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • Histogram of residuals

    ε̂

    F re

    qu en

    cy

    −0.5 0.0 0.5 1.0

    0 5

    10 15

    11 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • OLS estimates of α, β1, . . . , βp

    (This is only really reasonable to write down if p = 2)

    Yi = α + β1X1i + β2X2i + εi

    β̂1 = sY (rX1Y − rX1X2 rX2Y )

    sX1(1 − r 2X1X2)

    β̂2 = sY (rX2Y − rX1X2 rX1Y )

    sX2(1 − r 2X1X2)

    α̂ = Ȳ − β̂1X̄1 − β̂2X̄2, where

    rAB =

    ∑n i=1(Ai − Ā)(Bi − B̄)√∑n

    i=1(Ai − Ā)2 √∑n

    i=1(Bi − B̄)2 for some A and B and

    S2A = 1

    n − 1

    n∑ i=1

    (Ai − Ā)2 for some A

    12 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • It is easier if you know matrix algebra

    Y = Xβ + ε,

    where

    Y =

     y1 y2 ...

    yn

     , X = 

    1 x11 . . . x1p 1 x21 . . . x2p ...

    ... . . .

    ... 1 x21 . . . xnp

     , β =  α β1 ... βp

     , ε =  ε1 ε2 ... εn

    

    13 / 30 Thais Paiva STA 111 - Summer 2013 Term II Lecture 19, 08/01/2013

  • It is easier if you know matrix algebra

    It turns out that the error sum of squares can be written as

    ε̂ = (Y − Xβ)T (Y − Xβ) ∂ε̂

    ∂β = 2XT (Y