3. The F Test for Comparing Reduced vs. Full Models · Now back to determining the distribution of...

3. The F Test for ComparingReduced vs. Full Models

Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 1 / 43

Assume the Gauss-Markov Model with normal errors:

y = Xβ + ε, ε ∼ N(0, σ2I).

Suppose C(X0) ⊂ C(X) and we wish to test

H0 : E(y) ∈ C(X0) vs. HA : E(y) ∈ C(X) \ C(X0).


The “reduced” model corresponds to the null hypothesis andsays that E(y) ∈ C(X0), a specified subspace of C(X).

The “full” model says that E(y) can be anywhere in C(X).

For example, suppose

X0 =

111111

and X =

1 0 01 0 00 1 00 1 00 0 10 0 1

.


In this case, the reduced model says that all 6 observations havethe same mean.

The full model says that there are three groups of twoobservations. Within each group, observations have the samemean. The three group means may be different from oneanother.


For this example, let µ1, µ2, and µ3 be the elements of β in thefull model, i.e., β = [µ1, µ2, µ3]

′. Then, for the full model,

E(y) = Xβ =

1 0 01 0 00 1 00 1 00 0 10 0 1

µ1

µ2

µ3

=

µ1

µ1

µ2

µ2

µ3

µ3

, and

H0 : E(y) ∈ C(X0) vs. HA : E(y) ∈ C(X) \ C(X0)

is equivalent to

H0 : µ1 = µ2 = µ3 vs. HA : µi 6= µj, for some i 6= j.


For the general case, consider the test statistic

F =y′(PX − PX0)y/ [rank(X)− rank(X0)]

y′(I − PX)y/ [n− rank(X)].

To show that this statistic has an F distribution, we will use thefollowing fact:

PX0PX = PXPX0 = PX0 .


There are many ways to see that this fact is true. First,

C(X0) ⊂ C(X) =⇒ Each column of X0 ∈ C(X)

=⇒ PXX0 = X0.

Thus,

PXPX0 = PXX0(X′0X0)−X′0 = X0(X′0X0)

−X′0= PX0 .

This implies that

(PXPX0)′ = P′X0

=⇒ P′X0P′X = P′X0

=⇒ PX0PX = PX0 . 2


Alternatively,

∀ a ∈ Rn, PX0a ∈ C(X0) ⊂ C(X).

Thus, ∀ a ∈ Rn, PXPX0a = PX0a.

This implies PXPX0 = PX0.

Transposing both sides of this equality and using symmetry ofprojection matrices yields

PX0PX = PX0 . 2


Alternatively, C(X0) ⊂ C(X) =⇒ XB = X0 for some B becauseevery column of X0 must be in C(X).

Thus,

PX0PX = X0(X′0X0)−X′0PX = X0(X′0X0)

−(XB)′PX

= X0(X′0X0)−B′X′PX = X0(X′0X0)

−B′X′

= X0(X′0X0)−(XB)′ = X0(X′0X0)

−X′0 = PX0 .

PXPX0 = PXX0(X′0X0)−X′0 = PXXB(X′0X0)

−X′0

= XB(X′0X0)−X′0 = X0(X′0X0)

−X′0 = PX0 .

2Copyright c©2018 Dan Nettleton (Iowa State University) 3. Statistics 510 9 / 43

Note that PX − PX0 is a symmetric and idempotent matrix:

(PX − PX0)′ = P′X − P′X0

= PX − PX0 .

(PX − PX0)(PX − PX0) = PXPX − PXPX0 − PX0PX + PX0PX0

= PX − PX0 − PX0 + PX0

= PX − PX0 .


Now back to determining the distribution of


y′(I − PX)y/ [n− rank(X)].

An important first step is to note that

F =y′(

PX−PX0σ2

)y/ [rank(X)− rank(X0)]

y′( I−PX

σ2

)y/ [n− rank(X)]

.

Now we can show that the numerator is a chi-square randomvariable divided by its degrees of freedom, independent of thedenominator, which is a central chi-square random variabledivided by its degrees of freedom. Once we show all thesethings, we will have established that the statistic F has an Fdistribution (see Slide Set 1 Slide 35).


We will use the following result from Slide Set 1.

Suppose Σ is an n× n positive definite matrix.

Suppose A is an n× n symmetric matrix of rank m such thatAΣ is idempotent (i.e., AΣAΣ = AΣ).

Then y ∼ N(µ,Σ) =⇒ y′Ay ∼ χ2m(µ

′Aµ/2).

For the numerator of our F statistic, we have

µ = Xβ, Σ = σ2I, A =

(PX − PX0

σ2

), and


m = rank(A) = rank(

PX − PX0

σ2

)= rank(PX − PX0)

= tr(PX − PX0) = tr(PX)− tr(PX0)

= rank(PX)− rank(PX0) = rank(X)− rank(X0).

(Multiplying by a nonzero constant does not affect the rank of amatrix. Rank is the same as trace for idempotent matrices.Trace of a difference is the same as the difference of traces. Therank of a projection matrix is equal to the rank of the matrixwhose column space is projected onto.)


To verify that Σ is positive definite, note that for any a ∈ Rn \ {0},

a′Σa = a′(σ2I)

a = σ2a′a = σ2n∑

i=1

a2i > 0.

To verify that AΣ is idempotent, we have

AΣ =

(PX − PX0

σ2

)(σ2I) = PX − PX0 .

Thus, we have

y′(

PX − PX0

σ2

)y ∼ χ2

rank(X)−rank(X0)

(12β′X′

(PX − PX0

σ2

)Xβ

).


Denominator Distribution

From the result on Slide Set 2 Slide 19, we have

y′(

I − PX

σ2

)y ∼ χ2

n−rank(X).

(This fact can be proved using the result on Slide Set 1 Slide 31,following the same type of argument used to show thedistribution of the numerator.)


Independence of Numerator and Denominator

By Slide Set 1 Slide 38,

y′(

PX−PX0σ2

)y is independent of y′

( I−PXσ2

)y because

(PX − PX0

σ2

)(σ2I)

(I − PX

σ2

)=

1σ2 (PX − PXPX − PX0 + PX0PX)

=1σ2 (PX − PX − PX0 + PX0) = 0.


Thus, it follows that


y′(I − PX)y/ [n− rank(X)]

∼ Frank(X)−rank(X0),n−rank(X)

(β′X′(PX − PX0)Xβ

2σ2

).

If H0 is true, i.e. if E(y) = Xβ ∈ C(X0), then the noncentralityparameter is 0 because

(PX − PX0)Xβ = PXXβ − PX0Xβ

= Xβ − Xβ = 0.


In general, the noncentrality parameter quantifies how far themean of y is from C(X0) because

β′X′(PX − PX0)Xβ

= β′X′(PX − PX0)′(PX − PX0)Xβ

= || (PX − PX0)Xβ ||2= || PXXβ − PX0Xβ ||2

= || Xβ − PX0Xβ ||2= || E(y)− PX0E(y) ||2 .


Note that

y′(PX − PX0)y = y′ [(I − PX0)− (I − PX)] y

= y′(I − PX0)y− y′(I − PX)y

= SSEREDUCED − SSEFULL.

Also rank(X)−rank(X0)

= [n− rank(X0)]− [n− rank(X)]

= DFEREDUCED − DFEFULL,

where DFE = Degrees of Freedom for Error.


Thus, the F statistic has the familiar form

(SSEREDUCED − SSEFULL)/(DFEREDUCED − DFEFULL)

SSEFULL/DFEFULL.


Equivalence of F Tests

It turns out that this reduced vs. full model F test is equivalent tothe F test for testing

H0 : Cβ = d vs. HA : Cβ 6= d

with an appropriately chosen C and d.

The equivalence of these tests is proved in STAT 611.


Example: F Test for Lack of Linear Fit

Suppose a balanced, completely randomized design is used toassign 1, 2, or 3 units of fertilizer to a total of 9 plots of land.

The yield harvested from each plot is recorded as the response.


Let yij denote the yield from the jth plot that received i units offertilizer (i, j = 1, 2, 3).

Suppose all yields are independent and yij ∼ N(µi, σ2) for all

i, j = 1, 2, 3.

If y =

y11

y12

y13

y21

y22

y23

y31

y32

y33

, then E(y) =

µ1

µ1

µ1

µ2

µ2

µ2

µ3

µ3

µ3

.


Suppose we wish to determine whether there is a linearrelationship between the amount of fertilizer applied to a plot andthe expected value of a plot’s yield.

In other words, suppose we wish to know if there exists realnumbers β1 and β2 such that

µi = β1 + β2(i) for all i = 1, 2, 3.


Consider testing H0 : E(y) ∈ C(X0) vs. HA : E(y) ∈ C(X) \ C(X0),where

X0 =

1 11 11 11 21 21 21 31 31 3

and X =

1 0 01 0 01 0 00 1 00 1 00 1 00 0 10 0 10 0 1

.


Note H0 : E(y) ∈ C(X0)⇐⇒ ∃ β ∈ R2 3 E(y) = X0β

⇐⇒ ∃[β1

β2

]∈ R2 3

µ1

µ1

µ1

µ2

µ2

µ2

µ3

µ3

µ3

=

1 11 11 11 21 21 21 31 31 3

[β1

β2

]=

β1 + β2(1)β1 + β2(1)β1 + β2(1)β1 + β2(2)β1 + β2(2)β1 + β2(2)β1 + β2(3)β1 + β2(3)β1 + β2(3)

⇐⇒ µi = β1 + β2(i) for all i = 1, 2, 3.


Note E(y) ∈ C(X)⇐⇒ ∃ β ∈ R3 3 E(y) = Xβ

⇐⇒ ∃

β1

β2

β3

∈ R3 3

µ1

µ1

µ1

µ2

µ2

µ2

µ3

µ3

µ3

=

1 0 01 0 01 0 00 1 00 1 00 1 00 0 10 0 10 0 1

β1

β2

β3

=

β1

β1

β1

β2

β2

β2

β3

β3

β3

.

This condition clearly holds with βi = µi for all i = 1, 2, 3.


The alternative hypothesis

HA : E(y) ∈ C(X) \ C(X0)

is equivalent to

HA : There do not exist β1, β2 ∈ R such that

µi = β1 + β2(i) ∀ i = 1, 2, 3.


Because the lack of fit test is a reduced vs. full model F test, wecan also obtain this test by testing

H0 : Cβ = d vs. HA : Cβ 6= d

for appropriate C and d.

β =

µ1

µ2

µ3

C =? d =?


R Code and Output

> x=rep(1:3,each=3)> x[1] 1 1 1 2 2 2 3 3 3>> y=c(11,13,9,18,22,23,19,24,22)>> plot(x,y,pch=16,col=4,xlim=c(.5,3.5),+ xlab="Fertilizer Amount",+ ylab="Yield",axes=F,cex.lab=1.5)> axis(1,labels=1:3,at=1:3)> axis(2)> box()


●

●

●

●

●

●

●

●

●

Fertilizer Amount

Yie

ld

1 2 3

1015

20


> X0=model.matrix(˜x)> X0

(Intercept) x1 1 12 1 13 1 14 1 25 1 26 1 27 1 38 1 39 1 3


> X=model.matrix(˜0+factor(x))> X

factor(x)1 factor(x)2 factor(x)31 1 0 02 1 0 03 1 0 04 0 1 05 0 1 06 0 1 07 0 0 18 0 0 19 0 0 1


> proj=function(x){+ x%*%ginv(t(x)%*%x)%*%t(x)+ }>> library(MASS)> PX0=proj(X0)> PX=proj(X)


> Fstat=(t(y)%*%(PX-PX0)%*%y/1)/+ (t(y)%*%(diag(rep(1,9))-PX)%*%y/(9-3))> Fstat

[,1][1,] 7.538462>> pvalue=1-pf(Fstat,1,6)> pvalue

[,1][1,] 0.03348515


> reduced=lm(y˜x)> full=lm(y˜0+factor(x))>> rvsf=function(reduced,full)+ {+ sser=deviance(reduced)+ ssef=deviance(full)+ dfer=reduced$df+ dfef=full$df+ dfn=dfer-dfef+ Fstat=(sser-ssef)/dfn/+ (ssef/dfef)+ pvalue=1-pf(Fstat,dfn,dfef)+ list(Fstat=Fstat,dfn=dfn,dfd=dfef,+ pvalue=pvalue)+ }


> rvsf(reduced,full)$Fstat[1] 7.538462

$dfn[1] 1

$dfd[1] 6

$pvalue[1] 0.03348515


> anova(reduced,full)Analysis of Variance Table

Model 1: y ˜ xModel 2: y ˜ 0 + factor(x)

Res.Df RSS Df Sum of Sq F Pr(>F)1 7 78.2222 6 34.667 1 43.556 7.5385 0.03349 *---Signif. codes: 0 *** 0.001 ** 0.01 * 0.05 . 0.1 1


> test=function(lmout,C,d=0){+ b=coef(lmout)+ V=vcov(lmout)+ dfn=nrow(C)+ dfd=lmout$df+ Cb.d=C%*%b-d+ Fstat=drop(

t(Cb.d)%*%solve(C%*%V%*%t(C))%*%Cb.d/dfn)+ pvalue=1-pf(Fstat,dfn,dfd)+ list(Fstat=Fstat,pvalue=pvalue)+ }> test(full,matrix(c(1,-2,1),nrow=1))$Fstat[1] 7.538462$pvalue[1] 0.03348515


SAS Code and Output

data d;input x y;cards;

1 111 131 92 182 222 233 193 243 22;run;


proc glm;class x;model y=x;contrast ’Lack of Linear Fit’ x 1 -2 1;

run;


The SAS System

The GLM Procedure

Dependent Variable: y

Sum ofSource DF Squares Mean Square F Value Pr > F

Model 2 214.2222222 107.1111111 18.54 0.0027

Error 6 34.6666667 5.7777778

Corrected Total 8 248.8888889

R-Square Coeff Var Root MSE y Mean

0.860714 13.43684 2.403701 17.88889

Source DF Type I SS Mean Square F Value Pr > F

x 2 214.2222222 107.1111111 18.54 0.0027

Source DF Type III SS Mean Square F Value Pr > F

x 2 214.2222222 107.1111111 18.54 0.0027


Contrast DF Contrast SS Mean Square F Value Pr > F

Lack of Linear Fit 1 43.55555556 43.55555556 7.54 0.0335


3. The F Test for Comparing Reduced vs. Full Models · Now back to determining the distribution of...

Documents

Transcript of 3. The F Test for Comparing Reduced vs. Full Models · Now back to determining the distribution of...