Post on 16-Aug-2020
Bootstrap Confidence IntervalsST552 Lecture 13
Charlotte Wickham2019-02-11
1
Motivation
The inferences we’ve covered so far relied on our assumption ofNormal errors:
ε ∼ N(0, σ2In×n)
For example, we’ve seen under this assumption, the least squaresestimates are also Normally distributed:
β ∼ N(β, σ2
(XT X
)−1)
If the errors aren’t truly Normally distributed, whatdistribution do the estimates have?
2
Warm-up: Your Turn
Imagine the errors are in fact t3 distributed?
With your neighbour: design a simulation to understand thedistribution of the least squares estimates.
3
4
Example: 1. Fix n, fix X
2
4
6
8
10
5 10 15 20
x
Res
pons
e
5
Example: 1. Fix β, find y
0
4
8
12
5 10 15 20
x
Res
pons
e
6
Example: 2. Simulate errors, find y
0
4
8
12
5 10 15 20
x
Res
pons
e
7
Example: 3. Find least squares line
0
4
8
12
5 10 15 20
x
Res
pons
e
8
Example: 4. Repeat #2. and #3. many times
0
4
8
12
5 10 15 20
x
Res
pons
e
9
Example: Examine distribution of estimates
0.0
0.1
0.2
0.3
0.4
−5 0 5 10
Estimate of intercept
dens
ity
0
1
2
3
4
5
0.00 0.25 0.50 0.75
Estimate of slope
dens
ity
10
Example: Compared to theory
0.0
0.1
0.2
0.3
0.4
−5 0 5 10
Estimate of intercept
dens
ity
0
1
2
3
4
5
0.00 0.25 0.50 0.75
Estimate of slope
dens
ity
11
When the errors aren’t Normal: CLT
Think of our estimates like linear combinations of the errors. I.e. asort of average of i.i.d random variables.Some version of the Central Limit Theorem will apply.
For large samples, even when the errors aren’t Normal,
β∼N(β, σ2(XT X )−1)
12
Summary so far
If we knew the error distribution and true parameters we could usesimulation to understand the sampling distribution the leastsquares estimates.
Simulation can also be used to demonstrate the CLT at work inregression.
13
Bootstrap confidence intervals
In practice, with data in front of us, we don’t know the distributionof the errors (nor the true parameter values).
The bootstrap is one approach to estimate the samplingdistribution of β, by using the simulation idea, and substituting inour best guesses for the things we don’t know.
14
Bootstrapping regression
(Model based resampling)
0. Fit model and find estimates, β, and residuals, ei
1. Fix X ,2. For k = 1, . . . ,B
i. Generate errors, ε∗i sampled with replacement from ei
ii. Construct y , using the model, y = y + ε∗
iii. Use least squares to find β∗(k)
3. Examine the distribution of β∗ and compare to β
One confidence interval for βj is the 2.5% and 97.5% quantiles ofthe distribution of β∗
j .(Known as the Percentile method, there are other (better?)methods).
15
Example: Faraway Galapagos Islands
(I’ll illustrate with simple linear regression, Faraway does multiplecase in 3.6)
−200
0
200
400
0 500 1000 1500
Elevation
Spe
cies
Observed data
16
Bootstrap: 1. Find β, y , and ei .
−200
0
200
400
0 500 1000 1500
Elevation
Spe
cies
Observed data
17
Bootstrap: Using fixed X , ˆbeta from observed data
−200
0
200
400
0 500 1000 1500
Elevation
Spe
cies
Observed data
18
Bootstrap: 2. Resample residuals to construct bootstrappedresponse
−200
0
200
400
0 500 1000 1500
Elevation
Spe
cies
Bootstrapped data
19
Bootstrap: 3. Fit regression model to bootstrapped response
−200
0
200
400
0 500 1000 1500
Elevation
Spe
cies
Bootstrapped data
20
Bootstrap: 3. Repeat #2. and #3. many times
−200
0
200
400
0 500 1000 1500
Elevation
Spe
cies
21
Examine distribution of estimates
0.000
0.005
0.010
0.015
0.020
−50 0 50
Bootstrap estimates of intercept
dens
ity
0
5
10
0.10 0.15 0.20 0.25 0.30 0.35
Bootstrap estimates of slope
dens
ity
22
High level: bootstrap idea
We don’t know the distribution of the errors, but our best guess isprobably the empirical c.d.f on the residuals.
Sampling from a random variable with a c.d.f. defined as theempirical c.d.f. of the residuals, boils down to sampling withreplacement from residuals.
23
Limitations
We might rely on bootstrap confidence intervals when we areworried about the assumption of Normal errors. But, there arelimitations.
• We still rely on the assumption that the errors areindependent and identically distributed.
• Generally scaled residuals are used (residuals don’t have thesame variance, more later)
• An alternative bootstrap resamples the (yi , xi1, . . . , xip)vectors, i.e. resamples the rows of the data, a.k.a resamplingcases bootstrap.
24