Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of...

23
Multiple Comparisons October 16th & 18th, 2007 Reading: Chapter 7 HH Multiple Comparisons – p. 1/2

Transcript of Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of...

Page 1: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Multiple ComparisonsOctober 16th & 18th, 2007

Reading: Chapter 7 HH

Multiple Comparisons – p. 1/23

Page 2: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Simultaneous Inferences

Individual tests of hypotheses or confidence intervalsSuppose you try to control the Type I error of K

hypothesis tests at level α.

Pr(at least one Type I error) = 1 - (1 − α)K . Thisleads to an unacceptable error threshold.

Multiple comparison procedure: control thefamily-wise error rate (FWE):

FWE = Pr(reject at least one true hypothesis underany configuration of true and false hypotheses)

FDR = false discovery rate (the expected proportion offalsely rejected hypotheses).

Multiple Comparisons – p. 2/23

Page 3: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Common Multiple Comparisons Procedures

Bonferroni method

Tukey procedure

Dunnett procedure

Scheffe and Extended Tukey: simultaneouslycomparing all possible contrasts.

Multiple Comparisons – p. 3/23

Page 4: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Bonferroni Method

Bonferroni inequality: P (⋃

Ri) ≤∑

P (Ri).

Perform m related tests and conduct each test at levelαm

: FWE ≤ α.

Conservative multiple comparison procedure

Useful in situations when the statistics associated withthe m inferences have nonidentical probabilitydistributions.

Multiple Comparisons – p. 4/23

Page 5: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Tukey Procedure

Examines all pairwise comparisons by making use ofthe information about the joint distribution of thestatistics used in the inferences.

Less conservative than Bonferroni.

If there are a groups, there will be(

a2

)

pairwise tests.

Confidence intervals are constructed using criticalvalues from the Studentized range distribution.Intervals based on the Studentized range statistic,Tukey “Honest Significant Differences” method.

See ptukey and qtukey functions in R.

Multiple Comparisons – p. 5/23

Page 6: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

More on Tukey’s method

Confidence level exact when sample sizes are equalacross the a groups. If the sample sizes are unequal,confidence intervals are conservative.

R adjusts for slightly unbalanced design – seecomments in help(TukeyHSD) .

In R, use either TukeyHSD or simint withtype=”Tukey” option.

Multiple Comparisons – p. 6/23

Page 7: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Tukey Graphical Output

Female mice diet example

−20 −10 0 10 20

R.R50−NP

R.R50−N.R50

NP−N.R50

R.R50−N.R40

NP−N.R40

N.R50−N.R40

R.R50−N.N85

NP−N.N85

N.R50−N.N85

N.R40−N.N85

R.R50−lopro

NP−lopro

N.R50−lopro

N.R40−lopro

N.N85−lopro

95% family−wise confidence level

Differences in mean levels of DIET

Multiple Comparisons – p. 7/23

Page 8: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Dunnett’s Procedure

Compare the mean of one population with each of themeans of the remaining populations (e.g., compare acontrol to different treatments).

Uses the percentiles of a marginal distribution of amultivariate t distribution.

Multiple Comparisons – p. 8/23

Page 9: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Example of Dunnett’s Procedure

Random sample of 50 men matched for poundsoverweight was randomly separated into 5 equalgroups.

Each group was given exactly one of the weight lossagents: A, B, C, D, or E .

After a fixed period of time, each man’s weight losswas recorded.

Multiple Comparisons – p. 9/23

Page 10: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Weight Loss Boxplots

group

Wei

ght L

oss

A B C D E

8

9

10

11

12

13

Multiple Comparisons – p. 10/23

Page 11: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Dunnett Output

Dunnett contrasts

95 % one−sided confidence intervals

0.0 0.5 1.0 1.5 2.0 2.5 3.0

groupE−groupD

groupC−groupD

groupB−groupD

groupA−groupD

(

(

(

(

Multiple Comparisons – p. 11/23

Page 12: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Scheffe’s Method

Simultaneously compare all possible contrasts.

Uses a percentile of an F distribution to constructsimultaneous confidence intervals

a∑

j=1

cj yj ±√

(a − 1)F0.05,a−1,N−a s

a∑

j=1

c2

j

nj

s = σ

N =∑a

j=1nj

Results specify the constrast’s significant differencefrom zero.

Multiple Comparisons – p. 12/23

Page 13: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Turkey Data

Six turkeys were randomly assigned to each of 5 dietgroups and fed for the same length of time.

Control diet

A1: control + amount 1 of additive A

A2: control + amount 2 of additive A

B1: control + amount 1 of additive B

B2: control + amount 2 of additive B

Multiple Comparisons – p. 13/23

Page 14: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Turkey Boxplots

diet

Wei

ght G

ain

control A1 A2 B1 B2

4

6

8

10

Multiple Comparisons – p. 14/23

Page 15: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Scheffe simultaneous 95% CI

control vs treatment: (-4.28, -2.58)

A vs B: (-2.71, -1.19)

amount: (-2.69, -1.17)

A vs B by amount: (-0.312, 1.21)

Multiple Comparisons – p. 15/23

Page 16: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Contrast Analysis

treatment vs control: averaged over the 4 treatments,turkeys receiving a dietary additive gain significantlymore weight than ones not receiving an additive.

additive: turkeys receiving additive B gain significantlymore weight than turkeys receiving additive A.

amount: turkeys receiving amount 2 gain significantlymore weight than turkeys receiving amount 1.

interaction between additive and amount: the extentof increased weight gain as a result of receivingamount 2 rather than amount 1 is not significantlydifferent for the two additives

Multiple Comparisons – p. 16/23

Page 17: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Extended Tukey Procedure

Can modify Tukey procedure to cover the family of allpossible contrasts when the sample sizes are equalacross groups.

a∑

j=1

cj yj ± qα

2

s√n

a∑

j=1

|cj |

Wider CI compared to Scheffe’s method.

Appropriate for situations with a small number of morecomplicated contrasts.

Multiple Comparisons – p. 17/23

Page 18: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Graphical Displays

The standard tabular and graphical outputs do notconvey some aspects of a multiple comparisonanalysis.

For example, in the Tukey pairwise comparison, thestandard output just shows the CI for the difference.The mean of each group being compared is obscured.

The standard displays do not show the relativedistances between adjacent sorted sample means.

Multiple Comparisons – p. 18/23

Page 19: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

MMC Plot

Mean-Mean Multiple Comparisons displays

Horizontal axis shows the contrast value (e.g., for acomparison between two groups, it would show thedifference between the two sample means).

Vertical axis shows the sample mean of eachsubgroup. This allows visualization of the relativedistances between the sample means of the differentsubgroups.

Multiple Comparisons – p. 19/23

Page 20: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Example with Turkey data

Pairwise confidence intervals from Tukey procedure:R code

> turkeyci <- simint(wt.gain˜diet, data=turkey,type=’’Tukey’’)

> plot(turkeyci)

Multiple Comparisons – p. 20/23

Page 21: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Pairwise confidence intervals for turkey data

Tukey contrasts

95 % two−sided confidence intervals

0 2 4 6

dietB2−dietB1

dietB2−dietA2

dietB1−dietA2

dietB2−dietA1

dietB1−dietA1

dietA2−dietA1

dietB2−dietcontrol

dietB1−dietcontrol

dietA2−dietcontrol

dietA1−dietcontrol

( )

( )

( )

( )

( )

( )

( )

( )

( )

( )

Multiple Comparisons – p. 21/23

Page 22: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

MMC plot with Turkey data

R code

> tmp0 <- t(simint(wt.gain˜diet, data=turkey,type=’’Tukey’’)$cmatrix

> tmp1 <- simint.mmc(wt.gain˜diet, data=Turkey,method=’’Tukey’’,whichf=’’diet’’, lmat=tmp0,lmat.rows=2:6)

> plot(tmp1)

Multiple Comparisons – p. 22/23

Page 23: Multiple Comparisons - StatSci · PDF fileExamines all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons

Pairwise confidence intervals for turkey data

multiple comparisons of means of wt.gain

contrast value

−5 0 5

simultaneous 95% confidence limits, Tukey method

mean

wt.gain

level

contrast

3.78333333333333control

5.5 A1

6.98333333333333 A27 B1

9.38333333333333 B2

A1

A2B1

B2

A1

A2B1

B2

control

dietA1−dietcontrol

dietA2−dietcontroldietB1−dietcontrol

dietB2−dietcontrol

dietA2−dietA1dietB1−dietA1

dietB2−dietA1

dietB1−dietA2

dietB2−dietA2dietB2−dietB1

Multiple Comparisons – p. 23/23