Logistic Regression Part One

STA2101/442 F 2012

See last slide for copyright information

Logistic Regression

For a binary dependent variable: 1=Yes, 0=No

Pr{Y = 1|X = x} = !

Least Squares vs. Logistic Regression

Linear regression model for the log odds of the event Y=1

log

!!

1! !

"= "0 + "1x1 + . . .+ "p!1xp!1

Note ! is a conditional probability.

Equivalent Statements

log

!!

1! !

"= "0 + "1x1 + . . .+ "p!1xp!1

!

1! != e!0+!1x1+...+!p!1xp!1

= e!0e!1x1 · · · e!p!1xp!1 ,

! =e!0+!1x1+...+!p!1xp!1

1 + e!0+!1x1+...+!p!1xp!1.

•  A distinctly non-linear function •  Non-linear in the betas •  So logistic regression is an example of

non-linear regression.

E(Y |x) = ! =e!0+!1x1+...+!p!1xp!1

1 + e!0+!1x1+...+!p!1xp!1

F (x) = ex

1+ex is called the logistic distribution.

•  Could use any cumulative distribution function:

•  CDF of the standard normal used to be popular

•  Called probit analysis •  Can be closely approximated with a

logistic regression.

! = F ("0 + "1x1 + . . .+ "p!1xp!1)

In terms of log odds, logistic regression is like regular

regression

log

!!

1! !

"= "0 + "1x1 + . . .+ "p!1xp!1

In terms of plain odds,

•  (Exponential function of) the logistic regression coefficients are odds ratios

•  For example, “Among 50 year old men, the odds of being dead before age 60 are three times as great for smokers.”

Logistic regression

•  X=1 means smoker, X=0 means non-smoker

•  Y=1 means dead, Y=0 means alive

•  Log odds of death =

•  Odds of death =

Cancer Therapy Example

x is severity of disease

For any given disease severity x,

In general,

•  When xk is increased by one unit and all other independent variables are held constant, the odds of Y=1 are multiplied by

•  That is, is an odds ratio --- the ratio of the odds of Y=1 when xk is increased by one unit, to the odds of Y=1 when everything is left alone.

•  As in ordinary regression, we speak of “controlling” for the other variables.

The conditional probability of Y=1

This formula can be used to calculate a predicted P(Y=1|x). Just replace betas by their estimates

It can also be used to calculate the probability of getting the sample data values we actually did observe, as a function of the betas.

! =e!0+!1x1+...+!p!1xp!1

1 + e!0+!1x1+...+!p!1xp!1

=ex

"i!

1 + ex"i!

Likelihood Function

!(!) =n!

i=1

P (Yi = yi|xi) =n!

i=1

"yi(1! ")1!yi

=n!

i=1

"ex

!i!

1 + ex!i!

#yi"1! ex

!i!

1 + ex!i!

#1!yi

=n!

i=1

"ex

!i!

1 + ex!i!

#yi $1

1 + ex!i!

%1!yi

=n!

i=1

eyix!i!

1 + ex!i!

=e!n

i=1 yix!i!

&ni=1

'1 + ex

!i!(

Maximum likelihood estimation •  Likelihood = Conditional probability of getting

the data values we did observe, •  As a function of the betas •  Maximize the (log) likelihood with respect to

betas. •  Maximize numerically (“Iteratively re-weighted

least squares”) •  Likelihood ratio, Wald tests as usual •  Divide regression coefficients by estimated

standard errors to get Z-tests of H0: βj=0. •  These Z-tests are like the t-tests in ordinary

regression.

Copyright Information

This slide show was prepared by Jerry Brunner, Department of Statistics, University of Toronto. It is licensed under a Creative Commons Attribution - ShareAlike 3.0 Unported License. Use any part of it as you like and share the result freely. These Powerpoint slides will be available from the course website: http://www.utstat.toronto.edu/brunner/oldclass/appliedf12