Week 14 Comparing k(>2) Populationspersonal.psu.edu/acq/401/course.info/week14.pdf · Week 14...

OutlineComparing k> 2 Means: The ANOVA F Test

Comparing k> 2 Proportions: The χ2 TestThe Kruskal-Wallis Test

Multiple Comparisons and Simultaneous CIsTwo-Factor Designs

Week 14Comparing k(> 2) Populations

Week 14 Comparing k(> 2) Populations




Week 14 Objectives

Methods associated with testing for the equality of k(> 2) means orproportions are presented. Post-testing concepts and analysis areintroduced. In particular:

1 The principle of analysis of variance is used to derive the F testfor comparing k(> 2) means.

2 The χ2 test for comparing k(> 2) proportions is presented in itscontingency table form and its relation to the F test is explained.

3 The rank-based Kruskal-Wallis test is presented.

4 The concepts of multiple comparisons and simultaneous CIs areintroduced and methods for their implementation are presented.





1 Comparing k> 2 Means: The ANOVA F Test

2 Comparing k> 2 Proportions: The χ2 Test

3 The Kruskal-Wallis Test

4 Multiple Comparisons and Simultaneous CIs

The Experiment-Wise Error Rate

Bonferroni Multiple Comparisons and Simultaneous CIs

Tukey’s Multiple Comparisons and Simultaneous CIs

5 Two-Factor Designs

Data and Hypotheses

ANOVA methodology





The Statistical Model and Hypothesis

Xi1,Xi2, . . . ,Xini , i = 1, . . . , k , are independent samples.

Xij = µi + εij , Var(εij) = σ2i .

Write µi = µ+ αi , where µ = k−1∑ni=1 µi , and αi = µi − µ

is the effect of population i (of factor level i).

Of interest is the hypothesis of no effects (all αi = 0), or

H0 : µ1 = µ2 = · · · = µk vs Ha : H0 is false.

The sample mean and sample variance from population iare: X i , S2

i , i = 1, . . . , k .





The basic idea

As in regression, variability is represented by the so-calledsums of squares, or SS, and the ANOVA approachdecomposes the total variability into components.For the comparison of k means, one of the components iscalled between groups variability and the other is calledwithin groups variability.If the between groups variability is large compared to thewithin groups variability, the hypothesis of equal means isrejected.





Assume homoscedasticity, i.e. Var(εij) = σ2, for alli = 1, . . . , k , set N = n1 + · · ·+ nk , and define:

B X ·· =1N

k∑i=1

ni∑j=1

Xij =1N

k∑i=1

niX i The overall mean

• X ·· is the sample mean of all data.

B SST =k∑

i=1

ni∑j=1

(Xij − X ··)2 Total Sum of Squares

• SST is the numerator of sample variance of all data.





B SSE =k∑

i=1

ni∑j=1

(Xij − X i·)2 Error Sum of Squares

• SSE represents the within group variability

B SSTr =k∑

i=1

ni(X i· − X ··)2 Treatment Sum of Squares

• SSTr represents the between group variability

• It can be shown that

SST = SSTr + SSE.





Each SS has its own DF (recall N = n1 + · · ·+ nk ):

DFSST = N − 1, DFSSE = N − k , DFSSTr = k − 1

Dividing each SS by its DF we obtain the mean squares:

MSE =SSE

N − k, MSTr =

SSTrk − 1

.

MSE is the k -sample version of the 2-sample pooled variance:

MSE = S2p =

(n1 − 1)S21 + · · ·+ (nk − 1)S2

kn1 + · · ·+ nk − k

Thus, MSE is an unbiased estimator of the common errorvariance σ2.





These calculations are summarized in the ANOVA table:

Source df SS MS F

Treatment k − 1 SSTr MSTr=SSTrk − 1

FH0 =MSTrMSE

Error N − k SSE MSE=SSE

N − k

Total N − 1 SSTUnder H0, and the assumption of normality

FH0 ∼ Fk−1,N−k .

Without normality the above holds approximately if allsample sizes are ≥ 30.





If H0 does not hold, the statistic FH0 tends to take largervalues.Thus, H0 is rejected at level α if

FH0 > Fk−1,N−k ,α,

where Fk−1,N−k ,α denotes the (1 − α)100-th percentile ofthe F distribution with k − 1 and N − k degrees of freedom.





ExampleThe following data resulted from comparing the degree ofsoiling in fabric treated with three different mixtures ofmethacrylic acid.

Mix 1: 0.56 1.12 0.90 1.07 0.94

Mix 2: 0.72 0.69 0.87 0.78 0.91

Mix 3: 0.62 1.08 1.07 0.99 0.93

Test H0 : µ1 = µ2 = µ3 vs Ha : H0 is false, at α = 0.1.





Example (Continued)Solution. Because of the small sample sizes, we need toassume that the three populations are normal andhomoscedastic. With this data, X 1 = 0.918, X 2 = 0.794,X 3 = 0.938 and X ·· = 0.8833. Also, the three sample variancesare S2

1 = 0.04822, S22 = 0.00893 and S2

3 = 0.03537. Thus,

SSTr = 5(0.918 − 0.8833)2 + 5(0.794 − 0.8833)2

+5(0.938 − 0.8833)2 = 0.06085

SSE = 4S21 + 4S2

2 + 4S23 = 0.37008.

It follows that, SST = 0.06085 + 0.37008 = 0.43093,MSTr = 0.06085/(3 − 1) = 0.030425,





Example (Continued)

MSE = 0.37008/(15 − 3) = 0.03084 andFH0 = MSTr/MSE = 0.9865. The above calculation aresummarized in the following ANOVA table

Source df SS MS FTreatment k − 1 = 2 .0608 .0304 .99Error N − k = 12 .3701 .0308Total N − 1 = 14 .4309

The rejection rule specifies that H0 be rejected if F > F2,12,0.1.Since F2,12,0.1 = 2.807 (found in R by qf(0.9,2,12)), H0 is notrejected.





F and χ2 Tests

• F and χ2 distributions are connected by the following result:

If Y ∼ Fν1,ν2 , with ν2 ”large”, then ν1Y ·∼ χ2ν1

.

For example, under H0,

QH0 = (k − 1)FH0 =

∑ki=1 ni(X i· − X )2

S2p

·∼ χ2k−1.

• The χ2 statistic for testing the equality of k proportions isessentially the QH0 statistic adjusted to Bernoulli data.





Computational Form of QH0

• To adjust QH0 to Bernoulli data, replace X i by p̂i , X by thepooled proportion of “successes” p̂, and S2

p by p̂(1 − p̂).

Thus the test statistic for H0 : p1 = · · · = pk is:

QH0 =

∑ki=1 ni(p̂i − p̂)2

p̂(1 − p̂),

and H0 is rejected at level α if QH0 > χ2k−1,α.

χ2k−1,α is found in R by qchisq(1-α, k-1).

The p-value corresponding to QH0 is found in R by1-pchisq(QH0 , k-1).





The Contingency Table Form of QH0

An alternative form of the test statistic QH0 , called thecontingency table form, is more common:

QH0 =k∑

i=1

2∑`=1

(Oì − Eì)2

Eì, where

O1i = ni p̂i , O2i = ni(1 − p̂i

), E1i = ni p̂, E2i = ni

(1 − p̂

).

The algebraic equivalence of the two forms of QH0 isshown on page 357 of the book.The contingency table form is not recommended for handcalculations.





Example (Pilot response time for different panel designs)

The sample sizes, ni , and number of times, O1i , that theresponse times were below 3 seconds for the four designs areas follows: n1 = 45, O11 = 29; n2 = 50, O12 = 42; n3 = 55,O13 = 28; n4 = 50, O14 = 24. Perform the test at α = 0.05.

Solution. We will use the computational form of QH0 . Here,each p̂i = O1i/ni , so p̂1 = 0.6444, p̂2 = 0.84, p̂3 = 0.5091 andp̂4 = 0.48. Also,

p̂ =O11 + O12 + O13 + O14

n1 + n2 + n3 + n4= 0.615,

so that S2p = p̂(1 − p̂) = 0.2368. Thus,





Example (Example continued)

QH0 =45(0.6444 − 0.615)2

0.2368+

50(0.84 − 0.615)2

0.2368

+55(0.5091 − 0.615)2

0.2368+

50(0.48 − 0.615)2

0.2368= 17.307.

Since 17.307 > χ23(0.05) = 7.815, H0 is rejected.





• The extension of the two-sample rank sum test procedure tok samples, i.e., for testing

HF0 : F1 = · · · = Fk , (4.1)

is called Kruskal-Wallis test.

• Like the rank-sum test, the Kruskal-Wallis test uses the(mid-)ranks of the observations:

Combine the observations, Xi1, . . . ,Xini , i = 1, . . . , k , fromthe k samples into an overall set of N = n1 + · · ·+ nkobservations, and arrange them from smallest to largest.





Let Rij denote the (mid-)rank of observation Xij , and set

R i = n−1i

ni∑j=1

Rij , S2KW =

1N − 1

k∑i=1

ni∑j=1

(Rij −

N + 12

)2

.

Note that S2KW is the sample variance of the collection of all

ranks.

The Kruskal-Wallis TS is

KWk =1

S2KW

k∑i=1

ni

(R i −

N + 12

)2

.





If there are no ties a simpler expression holds:

KWk =12

N(N + 1)

k∑i=1

ni

(R i −

N + 12

)2

.

If the k populations have continuous distributions, theexact null distribution of KWk is known.For hand implementation, we require ni > 8, for all i , anduse the approximate null distribution

KWk·∼ χ2

k−1.

Thus, the RR at level α is KWk > χ2k−1,α, and the p-value is

computed in R by 1-pchisq(KWk , k-1).





Example (Rank procedures for the flammability data)A flammability test was performed on six pieces from each ofthree types of fabric used in children’s clothing. The responsevariable is the length of the burn mark. The data is given inFlammability.txt. Test the hypothesis that there is no differencein flammability among the three materials, at α = 0.05.

Solution The ranks, and the three rank averages are

Ranks R iMaterial 1 1 18 8 15.5 10 12 10.75Material 2 4 3 7 5 6 9 5.67Material 3 2 17 15.5 14 13 11 12.08





Example (Example continued)

The sample variance of the combined ranks is S2KW = 28.4706.

Thus,

KWk =1

28.47066

[(10.75 − 18 + 1

2

)2

+

(5.67 − 18 + 1

2

)2

+

(12.08 − 18 + 1

2

)2]= 4.823.

The R command 1-pchisq(4.823,2) yields a p-value of 0.08965,so that H0 cannot be rejected at level 0.05.





The Experiment-Wise Error RateBonferroni Multiple Comparisons and Simultaneous CIsTukey’s Multiple Comparisons and Simultaneous CIs









Data and Hypotheses

ANOVA methodology






Why not do all pairwise comparisons

• H0 : µ1 = · · · = µk is equivalent to H i,j0 : µi = µj , for all

(k2

)pairs i , j . Thus we can reject H0 if at least one of the H i,j

0 isrejected. So why do we need the ANOVA F test?

ANSWER: Because we do not know at what level α̃ to test eachH i,j

0 in order to for the test for H0 to have level exactly α. Indeed:

α = P(type I error) = P(H0 is rejected when true)

= P(at least one of the H i,j0 is rejected when all true)

= 1 − P(none of the H i,j0 is rejected when all true)

= 1 − (1 − α̃)(k2)






• The problem is that the last equality requires independence.Using Bonferroni’s inequality we have:

P(type I error) = P(at least one H i,j0 is rejected when all true)

≤∑

all pairs i,j

P(H i,j0 is rejected when true)

=

(k2

)α̃

• Choosing α̃ = α/(k

2

)results in a conservative test, i.e.,

P(type I error) ≤ α.






• Pairwise testing and pairwise CIs become useful ifH0 : µ1 = µ2 = · · · = µk is rejected.

• The equivalence of H0 : µ1 = · · · = µk with H i,j0 : µi = µj , for

all pairs i , j leads to the concepts of multiple comparisons andsimultaneous CIs.






When H0 : µ1 = µ2 = · · · = µk is rejected, it is not clearwhich of the µi are significantly different.It would seem that this question can be addressed quitesimply by making all pairwise comparisons.

For example, if k = 5 and H0 is rejected we can make CIsfor all 10 pairwise differences,

µ1 − µ2, . . . , µ1 − µ5, µ2 − µ3, . . . , µ2 − µ5, . . . , µ4 − µ5,

and if a CI does not include 0, the corresponding differenceis declared significantly different from 0.Alternatively, we can test each of the 10 null hypothesesH i,j

0 : µi = µj , for 1 ≤ i < j ≤ 5, against the two-sidedalternative.

With some fine tuning this idea works.






The fine-tuning is needed to assure that the overall, orexperiment-wise error rate does not exceed α.

The word “experiment” in the term “experiment-wise errorrate” refers to all pairwise comparisons.

The experiment-wise error rate is defined as the probability of atleast one pair of means being declared different when all meansare equal.

The experiment-wise error rate is nothing but the type Ierror rate for the procedure that rejectsH0 : µ1 = µ2 = · · · = µk if at least one of the H i,j

0 is rejected,or if at least one of the

(k2

)CI for the pairwise differences

does not include 0.

Fine-tuning is needed because the experiment-wise error rate isdifferent from the level of significance used for testing each H i,j

0 .






To appreciate how much the experiment-wise error ratecan differ from the level of significance used for each H i,j

0 ,consider k = 5 and suppose the 10 tests are performedindependently. (They are not really!)Then the probability that no H i,j

0 is rejected when in fact H0is true, is (1 − α)10, and thus the experiment-wise errorrate is 1 − (1 − α)10. If α = 0.05, this is

1 − (1 − 0.05)10 = 0.401.

In spite of the unrealistic independence assumption, theabove calculation gives an fairly close approximation to thetrue experiment-wise error rate.






Confidence intervals which control the experiment-wiseerror rate at a desired level α will be called (1 − α)100%simultaneous confidence intervals.

Thus, 10 CI are called 90% simultaneous confidenceintervals if 90% of the time they all contain the true value ofthe respective difference or, equivalently, 10% of the time atleast one of them does not contain the true value of therespective difference.

The term multiple comparisons is used to refer to a testprocedure that controls the experiment-wise error rate.

Multiple comparisons can be done through simultaneousCI, or via hypothesis testing.






We will see two methods of constructing simultaneous CIs.Bonferroni CIs: These use Bonferroni’s inequality to give alower bound on the level of the simultaneous CIs (or anupper bound on the experiment-wise error rate).Tukey CIs: The level of these simultaneous CIs is exactwhen sampling from normal homoscedastic populations.They can also be used as a good approximation with largesamples from any distribution.

Tukey’s method can also be applied when sampling fromskewed homoscedastic populations and have large samplesizes, as well as on the ranks with smaller sample sizes.














Data and Hypotheses

ANOVA methodology






Bonferroni’s CIs achieve the desired level by adjusting thelevel of the traditional CIs.

Exact adjustment is not possible due to the dependence ofthe CIs. But Bonferroni’s inequality guarantees that theintervals are conservative (i.e., the exact level of 90%Bonferroni CIs is higher than 90%).

Bonferroni’s inequality asserts that, the simultaneouscoverage of m CIs, each of which has confidence level1 − α, is at least 1 − mα.

Equivalently, if each of m pairwise tests are performed atlevel α, the experiment-wise error rate is no greater thanmα.






For example, if we are comparing k means (or proportions)there are

(k2

)pairwise comparisons to be made. For

k = 3,4,5 we have 3, 6, and 10 pairwise comparisonsrespectively. Assuming the pairwise 95% CIs areindependently constructed, their simultaneous coverageand the corresponding Bonferroni lower bound, is

(0.857,0.85), (0.735,0.7), (0.599,0.5).

The above was computed by the R command:

a=0.05; (1-a)**3; 1-3*a; (1-a)**6; 1-6*a; (1-a)**10; 1-10*a)






• The above discussion leads to the following procedure forconstructing (1 − α)100% Bonferroni simultaneous CIs andmultiple comparisons:

For each of the m contrast construct a (1 − α/m)100% CI.This set of m CIs are the (1 − α)100% Bonferronisimultaneous CIs for the m contrasts.

Bonferroni multiple comparisons at level α: If any of them (1 − α)100% Bonferroni simultaneous CIs does notcontain zero, the corresponding contrast is declaredsignificantly different from zero at experiment-wise level α.Bonferroni multiple comparisons can also be conductedwith pairwise testing, without CIs. Each of the m pairwisetests is conducted at level α/m. Those rejected aredeclared significantly different at experiment-wise level α.






ExampleTest, at α = 0.05, the null hypothesis that the four paneldesigns have no effect on whether or not the pilot reaction timeis below 3 seconds (H0 : p1 = p2 = p3 = p4 vs Ha : H0 is false)using Bonferroni multiple comparisons.Solution: We will construct 95% Bonferroni simultaneous CIsfor the contrasts

p1 − p2, p1 − p3, p1 − p4, p2 − p3, p2 − p4, p3 − p4.

Because there are m = 6 contrasts, we construct(1 − 0.05/6)100% = 99.17% CIs for each of the abovecontrasts. The data are: n1 = 45, O11 = 29; n2 = 50, O12 = 42;n3 = 55, O13 = 28; n4 = 50, O14 = 24.






Example (Continued)The CIs for each contrast were constructed according to

p̂i − p̂j ± zαB/2

√p̂i(1 − p̂i)

ni+

p̂j(1 − p̂j)

nj

where αB = 1 − 0.9917. For example, the following R commandscompute the CI for p1 − p2:n1= 45; n2=50; p1=29/n1; p2=42/n2p1-p2 -qnorm(1-(1-.9917)/2)*sqrt(p1*(1-p1)/n1+p2*(1-p2)/n2)p1-p2 +qnorm(1-(1-.9917)/2)*sqrt(p1*(1-p1)/n1+p2*(1-p2)/n2)The second and third commands give -0.4283797, 0.03726856,respectively.The command prop.test(c(29,42), c(45,50), conf.level = 0.9917,correct=F) gives the same result.






Example (Continued)The resulting CIs can be presented in a table:

Contrast 99.17% CI Contains zero?p1 − p2 (-0.428, 0.0373) Yesp1 − p3 (-0.124, 0.394) Yesp1 − p4 (-0.101, 0.429) Yesp2 − p3 (0.106, 0.555) Nop2 − p4 (0.129, 0.591) Nop3 − p4 (-0.229, 0.287) Yes

Only p2 − p3, and p2 − p4 are significantly different from zero, atexperiment-wise level α = 0.05, from p3 and p4. Thus,H0 : p1 = · · · = p4 was rejected due to significant differencesbetween panel design 2 and panel designs 3 and 4.






• An organized way of presenting the multiple comparisonsoutcome consists ofa) List the sample proportions (or sample means) in increasingorder, andb) underline pairs that are not significantly different.Then pairs that are not underlined are significantly different.

In the previous example, p̂4 = 0.48, p̂3 = 0.51, p̂1 = 0.64and p̂2 = 0.84. Thus, the multiple comparisons outcomecan be presented as

p̂4 p̂3 p̂1 p̂20.48 0.51 0.64 0.84






ExampleThe data in GradesTeachMeth.txt contain exams scores for studentsexposed to three different teaching methods (Exercises 5 and 6,Section 10.3). After reading the data into gr, the commandsattach(gr); kruskal.test(score ∼ method) return a p-value of 0.021;thus H0 : F1 = F2 = F3 is rejected at α = 0.05. Use the Bonferronimultiple comparisons procedure, based on the rank-sum test, toidentify which methods differ significantly at α = 0.05.

Solution: We will perform multiple comparisons through testing.Because the desired experiment-wise error rate is 0.05, we willconduct each of the m = 3 pair-wise comparisons (A vs B, A vs C,and B vs C) at level 0.05/3 = 0.0167. If the p-value of one of thesecomparisons is smaller than 0.0167, the corresponding methods aredeclared significantly different.






Example (Continued)The command wilcox.test(score[1:16] ∼ method[1:16]) yields ap-value of 0.1049 for the A vs B comparison. The p-values for theother comparisons are obtained similarly. The results from all treerank-sum tests are summarized in the following table:

Comparison p-value Less than 0.0167?A vs B 0.1049 NoA vs C 0.0136 YesB vs C 0.1415 No

Thus, methods A and C are significantly different at experiment-wiseerror rate α = 0.05, but methods A and B, as well as methods B andC are not significantly different.














Data and Hypotheses

ANOVA methodology






This procedure requires normality (or all ni ≥ 30) andhomoscedasticity.

They are based on the studentized range distributionwhich is characterized by two degrees of freedom (df).

The numerator df equals k (the number of populations).

The denominator df equals the df for the SSE, i.e., N − k ,with N = n1 + · · ·+ nk .

Let Qα,k,N−k denote the 100(1 − α) percentile of thestudentized range distribution with k and N − k df.

Table A.7 gives Qα,k,ν for α = 0.1 and 0.05, and a range ofdf. The R command for Qα,k,ν is qtukey(1 − α, k , ν).






Tukey’s (1 − α)100% Simultaneous CIs:

X i − X j ± Qα,k ,N−k

√MSE

2

(1ni

+1nj

)Multiple comparisons at level α: If above interval doesnot contain zero, µi and muj are declared significantlydifferent at level α.






R Commands for Tukey’s Simultaneous CIs

Import the iron concentration data from http://personal.psu.edu/acq/401/Data/FeData.txtin the data frame fe. Then do:

out=aov(fe$conc ∼ fe$ind)

TukeyHSD(out) # lm instead of aov will NOT work here

plot(TukeyHSD(out))

orTukeyHSD(out,conf.level=0.99) # for 99% simultaneous CIs

plot(TukeyHSD(out,conf.level=0.99))


http://personal.psu.edu/acq/401/Data/FeData.txt

http://personal.psu.edu/acq/401/Data/FeData.txt




Data and HypothesesANOVA methodology









Data and Hypotheses

ANOVA methodology






• Data: Xijk , i = 1, . . . ,a, j = 1, . . . ,b, k = 1, . . . ,n.

• Mean-plus-error expression: Xijk = µij + εijk

• Mean decomposition: µijk = µ+ αi + βj + γij

• Hypotheses of interest:

HAB0 : γ11 = · · · = γab = 0

HA0 : α1 = · · · = αa = 0

HB0 : β1 = · · · = βb = 0














Data and Hypotheses

ANOVA methodology






• Total sum of squares:

SST =a∑

i=1

b∑j=1

n∑k=1

(Xijk − X ···)2

• Decomposition: SST = SSA + SSB + SSAB + SSE

• Mean squares

MSA =SSAa − 1

, MSB =SSBb − 1

,

MSAB =SSAB

(a − 1)(b − 1), MSE =

SSEab(n − 1)






• F-Statistics

F ABH0

=MSABMSE

∼ F(a−1)(b−1),ab(n−1)

F AH0

=MSAMSE

∼ F(a−1),ab(n−1)

F BH0

=MSBMSE

∼ F(b−1),ab(n−1)


Week 14 Comparing k(>2) Populationspersonal.psu.edu/acq/401/course.info/week14.pdf · Week 14...

Documents

Transcript of Week 14 Comparing k(>2) Populationspersonal.psu.edu/acq/401/course.info/week14.pdf · Week 14...