Multiple ComparisonsOctober 16th & 18th, 2007
Reading: Chapter 7 HH
Multiple Comparisons – p. 1/23
Simultaneous Inferences
Individual tests of hypotheses or confidence intervalsSuppose you try to control the Type I error of K
hypothesis tests at level α.
Pr(at least one Type I error) = 1 - (1 − α)K . Thisleads to an unacceptable error threshold.
Multiple comparison procedure: control thefamily-wise error rate (FWE):
FWE = Pr(reject at least one true hypothesis underany configuration of true and false hypotheses)
FDR = false discovery rate (the expected proportion offalsely rejected hypotheses).
Multiple Comparisons – p. 2/23
Common Multiple Comparisons Procedures
Bonferroni method
Tukey procedure
Dunnett procedure
Scheffe and Extended Tukey: simultaneouslycomparing all possible contrasts.
Multiple Comparisons – p. 3/23
Bonferroni Method
Bonferroni inequality: P (⋃
Ri) ≤∑
P (Ri).
Perform m related tests and conduct each test at levelαm
: FWE ≤ α.
Conservative multiple comparison procedure
Useful in situations when the statistics associated withthe m inferences have nonidentical probabilitydistributions.
Multiple Comparisons – p. 4/23
Tukey Procedure
Examines all pairwise comparisons by making use ofthe information about the joint distribution of thestatistics used in the inferences.
Less conservative than Bonferroni.
If there are a groups, there will be(
a2
)
pairwise tests.
Confidence intervals are constructed using criticalvalues from the Studentized range distribution.Intervals based on the Studentized range statistic,Tukey “Honest Significant Differences” method.
See ptukey and qtukey functions in R.
Multiple Comparisons – p. 5/23
More on Tukey’s method
Confidence level exact when sample sizes are equalacross the a groups. If the sample sizes are unequal,confidence intervals are conservative.
R adjusts for slightly unbalanced design – seecomments in help(TukeyHSD) .
In R, use either TukeyHSD or simint withtype=”Tukey” option.
Multiple Comparisons – p. 6/23
Tukey Graphical Output
Female mice diet example
−20 −10 0 10 20
R.R50−NP
R.R50−N.R50
NP−N.R50
R.R50−N.R40
NP−N.R40
N.R50−N.R40
R.R50−N.N85
NP−N.N85
N.R50−N.N85
N.R40−N.N85
R.R50−lopro
NP−lopro
N.R50−lopro
N.R40−lopro
N.N85−lopro
95% family−wise confidence level
Differences in mean levels of DIET
Multiple Comparisons – p. 7/23
Dunnett’s Procedure
Compare the mean of one population with each of themeans of the remaining populations (e.g., compare acontrol to different treatments).
Uses the percentiles of a marginal distribution of amultivariate t distribution.
Multiple Comparisons – p. 8/23
Example of Dunnett’s Procedure
Random sample of 50 men matched for poundsoverweight was randomly separated into 5 equalgroups.
Each group was given exactly one of the weight lossagents: A, B, C, D, or E .
After a fixed period of time, each man’s weight losswas recorded.
Multiple Comparisons – p. 9/23
Weight Loss Boxplots
group
Wei
ght L
oss
A B C D E
8
9
10
11
12
13
Multiple Comparisons – p. 10/23
Dunnett Output
Dunnett contrasts
95 % one−sided confidence intervals
0.0 0.5 1.0 1.5 2.0 2.5 3.0
groupE−groupD
groupC−groupD
groupB−groupD
groupA−groupD
(
(
(
(
Multiple Comparisons – p. 11/23
Scheffe’s Method
Simultaneously compare all possible contrasts.
Uses a percentile of an F distribution to constructsimultaneous confidence intervals
a∑
j=1
cj yj ±√
(a − 1)F0.05,a−1,N−a s
√
√
√
√
a∑
j=1
c2
j
nj
s = σ
N =∑a
j=1nj
Results specify the constrast’s significant differencefrom zero.
Multiple Comparisons – p. 12/23
Turkey Data
Six turkeys were randomly assigned to each of 5 dietgroups and fed for the same length of time.
Control diet
A1: control + amount 1 of additive A
A2: control + amount 2 of additive A
B1: control + amount 1 of additive B
B2: control + amount 2 of additive B
Multiple Comparisons – p. 13/23
Turkey Boxplots
diet
Wei
ght G
ain
control A1 A2 B1 B2
4
6
8
10
Multiple Comparisons – p. 14/23
Scheffe simultaneous 95% CI
control vs treatment: (-4.28, -2.58)
A vs B: (-2.71, -1.19)
amount: (-2.69, -1.17)
A vs B by amount: (-0.312, 1.21)
Multiple Comparisons – p. 15/23
Contrast Analysis
treatment vs control: averaged over the 4 treatments,turkeys receiving a dietary additive gain significantlymore weight than ones not receiving an additive.
additive: turkeys receiving additive B gain significantlymore weight than turkeys receiving additive A.
amount: turkeys receiving amount 2 gain significantlymore weight than turkeys receiving amount 1.
interaction between additive and amount: the extentof increased weight gain as a result of receivingamount 2 rather than amount 1 is not significantlydifferent for the two additives
Multiple Comparisons – p. 16/23
Extended Tukey Procedure
Can modify Tukey procedure to cover the family of allpossible contrasts when the sample sizes are equalacross groups.
a∑
j=1
cj yj ± qα
2
s√n
a∑
j=1
|cj |
Wider CI compared to Scheffe’s method.
Appropriate for situations with a small number of morecomplicated contrasts.
Multiple Comparisons – p. 17/23
Graphical Displays
The standard tabular and graphical outputs do notconvey some aspects of a multiple comparisonanalysis.
For example, in the Tukey pairwise comparison, thestandard output just shows the CI for the difference.The mean of each group being compared is obscured.
The standard displays do not show the relativedistances between adjacent sorted sample means.
Multiple Comparisons – p. 18/23
MMC Plot
Mean-Mean Multiple Comparisons displays
Horizontal axis shows the contrast value (e.g., for acomparison between two groups, it would show thedifference between the two sample means).
Vertical axis shows the sample mean of eachsubgroup. This allows visualization of the relativedistances between the sample means of the differentsubgroups.
Multiple Comparisons – p. 19/23
Example with Turkey data
Pairwise confidence intervals from Tukey procedure:R code
> turkeyci <- simint(wt.gain˜diet, data=turkey,type=’’Tukey’’)
> plot(turkeyci)
Multiple Comparisons – p. 20/23
Pairwise confidence intervals for turkey data
Tukey contrasts
95 % two−sided confidence intervals
0 2 4 6
dietB2−dietB1
dietB2−dietA2
dietB1−dietA2
dietB2−dietA1
dietB1−dietA1
dietA2−dietA1
dietB2−dietcontrol
dietB1−dietcontrol
dietA2−dietcontrol
dietA1−dietcontrol
( )
( )
( )
( )
( )
( )
( )
( )
( )
( )
Multiple Comparisons – p. 21/23
MMC plot with Turkey data
R code
> tmp0 <- t(simint(wt.gain˜diet, data=turkey,type=’’Tukey’’)$cmatrix
> tmp1 <- simint.mmc(wt.gain˜diet, data=Turkey,method=’’Tukey’’,whichf=’’diet’’, lmat=tmp0,lmat.rows=2:6)
> plot(tmp1)
Multiple Comparisons – p. 22/23
Pairwise confidence intervals for turkey data
multiple comparisons of means of wt.gain
contrast value
−5 0 5
simultaneous 95% confidence limits, Tukey method
mean
wt.gain
level
contrast
3.78333333333333control
5.5 A1
6.98333333333333 A27 B1
9.38333333333333 B2
A1
A2B1
B2
A1
A2B1
B2
control
dietA1−dietcontrol
dietA2−dietcontroldietB1−dietcontrol
dietB2−dietcontrol
dietA2−dietA1dietB1−dietA1
dietB2−dietA1
dietB1−dietA2
dietB2−dietA2dietB2−dietB1
Multiple Comparisons – p. 23/23
Top Related