Mle Testing

44
Basic Concepts The Trilogy Application: Test for Heteroskedasticity ML Based Hypothesis Testing Walter Sosa-Escudero Econ 507. Econometric Analysis. Spring 2009 April 21, 2009 Walter Sosa-Escudero ML Based Hypothesis Testing

description

Econometria

Transcript of Mle Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

ML Based Hypothesis Testing

Walter Sosa-Escudero

Econ 507. Econometric Analysis. Spring 2009

April 21, 2009

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Basic Concepts

Z ∼ f(y; θ), θ ∈ Θ ⊆ <K .

θ is a vector of K parameters.

Θ is the parameter space: set of values θ can take.

A hypothesis is a statement about θ. The goal is to learnsomething about the validity of the hypothesis, based on a sample.

If H0 : θ ∈ Θ0, the hypothesis is nested if Θ0 ⊆ Θ. In general,they are restriction hypotheses (eg., β = 0, Θ = <,Θ0 = {0}).

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

A test statistic is a random variable that has one particulardistribution when H0 is true and some other one when it isfalse.

The alternative hypothesis, HA, is a subset of all the values θcan take when H0 is false.Example: K = 1, Θ = <, H0 : θ = 0, HA : θ > 0.

The rejection region is the set of all value the test statisticcan take for which the null is rejected.

A test is a test statistic combined with a rejection region.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

The probability that the test rejects H0 when it is true (type-Ierror) is the level of the test. It depends strictly on the nullhypothesis.

The power of a test is the probability that the test rejects H0.When H0 is false, this probability depends on HA.

The type-II error is one minus the power (do not reject whenH0 is false).

In general there is a trade-off between type I and II errors: atest with level 0 (always accepts) tends to have zero power(never reject).

The classical approach (Neyman-Pearson) proposes to set alevel exogeneously and find test that maximize power for somerelevant alternative hypothesis.

A test is consistent if the power tends to one when H0 is falseand the sample size tends to infinity.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Non-linear Hypothesis

H0 : h(θ) = 0, h(θ) : <K → <r,

h(θ) is a vector of r restrictions on the K parameters.

In this case Θ = <k, and

Θ0 ={θ ∈ Θ | h(θ) = 0

},⇒ Θ0 ⊆ Θ

so the hypothesis is nested.

Let D(θ), be an (r × k) matrix with (i, j) element:

D(θ)ij =∂hi(θ)∂θj

and hi(θ) denotes the i−th element of h(θ), i = 1, . . . , r. We willassume ρ(D(θ)) = r

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Example: θ = (θ1, θ2, θ3)′. Consider H0 : h(θ) = 0, with

h(θ) =[θ1 − θ2

θ2θ3 − 1

]This is equivalent to H0 : θ1 = θ2 , θ2θ3 = 1. There are r = 2restrictions on K parameters.

In this case:

D(θ) =[

1 −1 00 θ3 θ2

]

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

The nested testing problem implies two estimators.

The unrestricted MLE, θ is defined as

θ ≡ argmaxθ∈Θ

l(θ; z)

and under suitable regularity conditions, the FOC’s are:

s(θ; z) = 0

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Similarly, the restricted MLE, θR is defined as

θ ≡ argmaxθ∈Θ0

l(θ; z)

The lagrangean function for the restricted maximization problem is

LG(θ, λ) = l(θ; z)− λ h(θ)

and the FOC’s are:

s(θR; z)−D(θR)′λ = 0h(θR) = 0

}

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Three Tests

Assume Vn is a consistent estimator of J . Consider the followingtest statistics for H0 : h(θ0) = 0.

1 Likelihood Ratio:

LR = n 2

[l(θ; z)n− l(θR; z)

n

]2 Wald:

W = n h(θ)′[D(θ)Vn(θ)−1D(θ)′

]−1h(θ)

3 Score (or Lagrange Multiplier):

LM = n

(s(θRn

)′V −1n (θR)

(s(θRn

)

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Result: Under all the regularity assumptions, if Zi ∼ f(z; θ0),i = 1, 2, . . . , n, under H0 : h(θ0) = 0, LR, W and LM converge indistribution to χ2(r) and they diverge to infinity underHA : h(θ0) = δ, for any constant δ 6= 0.

The three tests are asymptotically equivalent.

The three tests are consistent.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Intuition: θ is always consistent for θ0. θR is consistent for θ0 onlyif H0 holds. Then, under H0:

θ and θR are consistent, then l(θ)− l(θR) ∼= 0 (LR).

h(θ) ∼= 0 (W)

The ‘shadow price’ of imposing the restriction is zero: λ ∼= 0.From the first FOC:

s(θR; z)−D(θR)′λ = 0

Since ρ(H(θ)) = r, then s(θR) ∼= 0 (score or LM).

The idea is to reject H0 when LR,W or LM are large.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

A graphical representation

Consider the following simplification

θ ∈ < (K = 1)

H0 : θ = θ0 (a simple case).

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Asymptotic Distributions under H0

The plan:

1 We will start with the Wald test since it depends on θ: wehave already proved its asymptotic properties, in particular

√n(θ − θ0) d→ N(0, J−1)

2 LR and LM depend on θR. We only know it is consistent.Then first we need to establish its asymptotic behavior.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Wald Test

W = n h(θ)′[D(θ)V −1

n D(θ)′]−1

h(θ)

Take a mean value expansion of h(θ) around θ0:

h(θ) = h(θ0) +D(θ)(θ − θ0

)where θ is a mean value between θ and θ0. Again, consistencyguarantees the exactness of the approximation.Under H0 : h(θ0) = 0

√n h(θ) = D(θ)

√n(θ − θ0

)

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

By asymptotic normality of θ

√n h(θ) d→ N

(0, D(θ0)J−1D(θ0)′

)since D(θ) and D(θ)

p→ D(θ0), Vnp→ J and θ

p→ θ0, by Slutzky’stheorem: √

n h(θ) d→ N(

0, D(θ)V −1n D(θ)′

)Then, taking the normed quadratic form:

W = n h(θ)′[D(θ)V −1

n D(θ)′]−1

h(θ) d−→ χ2(r)

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

AN of the restrited MLE

To prove the other two cases, we need two additional results:

1 √n(θR − θ0) d→ N(0, V (θ0)−1)

, where V (θ0)−1 ≡ J−1 − J−1D′[DJ−1D′]−1DJ−1,D ≡ D(θ0)

2

1√nλ ' −R(θ0)−1DJ−1 s(θ0)√

n

d→ N(0, R(θ0)−1),

with R ≡ DJ−1D′.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Proof: from the FOC’s

1√ns(θR; z)− 1√

nD(θR)′λ = 0

√n h(θR) = 0

(1)

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Take a mean value expansion around θ0 for both equations:

s(θR; z) = s(θ0; z) +H(θ)(θR − θ0)1√ns(θR; z) =

1√ns(θ0; z) +

√nH(θ)n

(θR − θ0)

=1√ns(θ0; z)−

√n J (θR − θ0) (Asymptotically, Why?) (2)

√n h(θR) =

√n h(θ0) +

√n D(θ)(θR − θ0)

=√n D(θ0)(θR − θ0) (Asymptotically, Why?) (3)

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Now replace (2) and (3) in (1) and use the fact that under H0

D(θR)p→ D(θ0).

1√ns(θ0; z)−

√n J (θR − θ0)− 1√

nD′λ = 0

√n D(θR − θ0) = 0

Solving the system (do it as homework):

√n (θR − θ0) = A

1√ns(θ0; z)

with A ≡ J−1 − J−1D′R−1DJ−1

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Note that by our asymptotic normality result:

1√ns(θ0; z) =

√ns(θ0; z)n

=√n

∑ni=1 s(θ0, yi)

n

d→ N(0, J)

Then using Slutzky’s theorem,

√n(θR − θ0) = A

1√ns(θ0; z) d→ N(0, A J A′)

which is the desired result.

We leave as homework to prove the similar result for λ.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

LM/Score

LM =1ns(θR)′V (θR)−1s(θR)

From the FOC of the restricted MLE problem:

1√ns(θR; z)− 1√

nD(θR)′λ = 0

Replacing:

λ′√nD(θR)V (θR)−1D(θR)′

λ√n' λ′√

nR(θ0)

λ√n

d→ χ2(r),

since λ/√n

d→ N(0, R(θ)−1), and it is a normed quadratic form.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

LR Test

LR = 2[l(θ; z)− l(θR; z)

]Consider a second order mean value expansion of (θR) around θ,under H0:

Supress dependence on z momentarily to simplify notation

l(θR) = l(θ) + s(θ)′(θR − θ

)+

12

(θR − θ

)′H(θ)

(θR − θ

)l(θR)− l(θ) =

12

(θR − θ

)′H(θ)

(θR − θ

)(Why?)

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Replacing and dividing and multiplying by n:

LR = −√n(θR − θ

)′( 1nH(θ)

)√n(θR − θ

)'

√n(θR − θ

)′I(θ0)

√n(θR − θ

)(Why?)

Recall

√n (θ − θ0) ' J−1 s(θ0; z)√

n

√n (θR − θ0) '

(J−1 − J−1D′R−1DJ−1

) s(θ0; z)√n

Substracting both sides:

√n(θR − θ

)' −J−1D′R−1DJ−1 s(θ0)√

n= −J−1D′

λ√n

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Replacing above:

LR ' λ′√nDJ−1JJ−1D′

λ′√n

' λ′√nDJ−1D′

λ′√n

' LMd→ χ2(r)

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Consistency of LR, LM and W

Now HA : h(θ0) 6= 0. Assume HA : h(θ0) = δ 6= 0.

W = n h(θ)′[D(θ)V −1

n D(θ)′]−1

h(θ)

→ n δ′[D(θ0)V −1

n D(θ0)′]−1

δ →∞

Note that under HA, plim s(θR; z)/n 6= 0 (Why?). Then:

LM =1ns(θR; z)′V (θR)−1s(θR; z)

= ns(θR; z)′

nV (θR)−1 s(θR; z)′

n→∞

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

We can use a similar argument with LR

LR = 2[l(θ; z)− l(θR; z)

]= n 2

[l(θ; z)n− l(θR; z)

n

]︸ ︷︷ ︸

6=0

→∞

(Why: recall the consistency proof...)

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Consistency

Under H0, W, LM and LR have asymptotic χ2(r) distribution.

For a level α, the acceptance region is [0, zα], where zα is the1− α quantile of χ2(r), a finite number.

Hence, under our HA, W, LM and LR →∞, hence lie outsidethe acceptance region wpt1: they are consistent.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

An application: A test for heteroskedasticity

Recall that the Breusch/Pagan test for heteroskedasticity is basedon the following steps:

1 Estimate by OLS, and save squared residuals e2i .

2 Regresss e2i on the Zik variables, k = 2, . . . , p and get (ESS).

The test statistic is:

12ESS ∼ χ2(p− 1) ∼ χ2(p)

under H0, asymptotically. We reject if it is too large.

This is an LM based test. The goal is to derive it from basicprinciples (likelihood).

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Setup:yi = x′iβ + ui

1 xi is a non-stochastic vector of K explanatory variables,including an intercept.

2 ui ∼ N(0, σ2i ) (possible heteroskedasticity)

3 σ2i = h(α1 + α2z2i + α3z3i + . . .+ αpzpi).

4 h(.) is any positive, twice differentiable function.

5 zi is a vector of p non-random variables.

Homoskedasticity: H0 : α2 = α3 = · · · = αp = 0

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Preliminary results I: LM tests for subvectors

In this setupθ = (β, α1, α2, . . . , αp)′.

The null of homskedasticity, H0 : α2, . . . , αp = 0 involves asubvector of θ, say, all the other parameters are free.

The LM test can be simplified in this situation

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Remember that the null h(θ) = 0 can be tested based on:

LM = n s(θR)′ V −1n s(θR)

where Vn is consistent for J and θR is the restricted MLE.Consider now the case:

θ = (θ1 θ2)′, H0 : θ2 = θ20, θ2 is r × 1.

Note that:

H0 :[

0 Ir] [ θ1

θ2

]− θ2o︸ ︷︷ ︸

h(θ)

= 0

with D(θ) = [0 Ir].

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Similarly

s(θ)[s1(θ)s1(θ)

], Vn(θ) =

[V11 V12

V21 V22

], V −1

n (θ) ≡[V 11 V 12

V 21 V 22

]

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Result (Theil, 1971, pp. 18): V 22 =[V22 − V21V

−111 V12

]−1.

In particular, if V21 = 0 =⇒ V 22 = V −122 .

FOC of the restricted MLE problem: in this case, θR = (θ1, θ2o) soas:

s1(θ1, θ2o) = 0s2(θ1, θ2o) = λ

θ2 = θ2o

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Reeplacing and simplifying in the LM formula:

LM = n s2(θ1; θ2o)′ V 22(θ) s2(θ1; θ2o)

If V21 = 0

LM = n s2(θ1; θ2o)′ V22(θ) s2(θ1; θ2o)

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Preliminary results II: some OLS algebra

Y = Xβ + u, Xn×K including intercept.

ESS =∑

(Yi− Y )2 =∑

Y 2i −NY 2 =

∑Y 2i − (1/N)(

∑Yi)2

In matrix form:

SCE = Y ′Y − (i′Y )2/N, con i = (1, 1, . . . , 1)′

But:

Y ′Y = Y ′X(X ′X)−1X ′︸ ︷︷ ︸(Xβ′)

X(X ′X)−1X ′Y︸ ︷︷ ︸Xβ

= Y ′X(X ′X)−1X ′Y

replacing: SCE = Y ′X(X ′X)−1X ′Y − (i′Y )2/N

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

The Breusch-Pagan Test

Let

ei, OLS residuals.

σ2 = (1/N)∑e2i , MLE of σ2 under H0.

gt ≡ e2i /σ

2.

Test (Breusch-Pagan): Under H0 :

LM =12SCEg,z ∼ χ2(p− 1)

ESSg,z = ESS of regressing gi on zi.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Proof: yi ∼ N(x′iβ, σ2i ), σ

2i = h(z′iα)

l(β, α) = −12N ln(2π)− 1

2

∑lnσ2

i −12

∑(1/σ2

i )(yi − x′iβ

)2Note

s(β; x, z) =∑ 1

σ2i

(yi − x′iβ)xi

and

s(α; x, z) = −12

∑ h′ihizi −

12

∑ h′ih2i

zi

with hi ≡ h(z′iα), h′i ≡ ∂h(z′iα)/∂α.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Let β0 and α0 denote the true values under the null. Also notethat α0 = (α1, 0, . . . , 0)′.

From the previous result, it is easy to check that J is blockdiagonal, that is

Jα,β = −E[∂2l

∂α∂β

]= 0

when evaluated at the true values under the null. Then, since our

H0 only involves the components of α, according to our previousresult, a test can be based on

LM =1nsα(θR; z, x)′J−1

αα sα(θR; z, x)

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Let β denote the MLE of β under the null. What it β in this case?.

Under H0 of homoskedasticity, let σ2 denote the MLE ofV (ui) = h(α1). By the invariance property, the MLE of α1 (α1) isdefined implicitely by σh(α1).

It is easy to verify

sα(θR) =12

[h′(α1)σ2

]∑zi

(e2i

σ2− 1)

and

Jαα =∂l(θ; x, z)∂αα′

∣∣∣∣θ=θR

=12

[h′(α1)σ2

]∑ziz′i

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Replacing and simplifying:

LM =12

(∑zifi

)′ (∑ziz′i

)−1 (∑zifi

)with fi ≡ e2

i /σ2 − 1 = gi − 1. In matrix terms:

LM =12f ′Z(Z ′Z)−1Z ′f =

12f ′PZf

with f = g − i, i is a vector of n ones.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Note

i′g =∑e2i /σ

2 = n.

PZi = i (Why?).

Then

LM = 1/2 f ′PZf= 1/2 (g − i)′PZ(g − i)= 1/2

[g′PZg − i′PZg − g′PZi+ i′PZi

]= 1/2

[g′PZg − n

]= 1/2

[g′PZg − (i′g)2/n

]= 1/2 ESSg,z

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

On the small sample performance

Key result: under all the assumptions andH0 : LMbp ∼ χ2(p− 1), asymptotically.

This result is used to determine the acceptance region, for anexogenously set level α.

How reliable is this this result?

A key assumption is normality.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

Evans (1992): a Monte Carlo exploration

Setup:

yi = x′iβ + ui

xi is generated using a log-normal and a uniform distribution

V (ui) = σ2(1 + λzi).

2000 replications for alternative disributions for ui.

n = 64, α = 0.05.

Tests: Breusch-Pagan (BPTrue), Modified BP (BPasym),Koenker (BPmod), White.

Walter Sosa-Escudero ML Based Hypothesis Testing

Basic ConceptsThe Trilogy

Application: Test for Heteroskedasticity

The size is severly affected by non-normalities.

Walter Sosa-Escudero ML Based Hypothesis Testing