Multiple Comparisons - StatSci all pairwise comparisons by making use of ... control A1 A2 B1 B2 4...
Embed Size (px)
Transcript of Multiple Comparisons - StatSci all pairwise comparisons by making use of ... control A1 A2 B1 B2 4...
Multiple ComparisonsOctober 16th & 18th, 2007
Reading: Chapter 7 HH
Multiple Comparisons p. 1/23
Individual tests of hypotheses or confidence intervalsSuppose you try to control the Type I error of Khypothesis tests at level .
Pr(at least one Type I error) = 1 - (1 )K . Thisleads to an unacceptable error threshold.
Multiple comparison procedure: control thefamily-wise error rate (FWE):
FWE = Pr(reject at least one true hypothesis underany configuration of true and false hypotheses)
FDR = false discovery rate (the expected proportion offalsely rejected hypotheses).
Multiple Comparisons p. 2/23
Common Multiple Comparisons Procedures
Scheffe and Extended Tukey: simultaneouslycomparing all possible contrasts.
Multiple Comparisons p. 3/23
Bonferroni inequality: P (
Perform m related tests and conduct each test at levelm
: FWE .Conservative multiple comparison procedure
Useful in situations when the statistics associated withthe m inferences have nonidentical probabilitydistributions.
Multiple Comparisons p. 4/23
Examines all pairwise comparisons by making use ofthe information about the joint distribution of thestatistics used in the inferences.
Less conservative than Bonferroni.
If there are a groups, there will be(
Confidence intervals are constructed using criticalvalues from the Studentized range distribution.Intervals based on the Studentized range statistic,Tukey Honest Significant Differences method.
See ptukey and qtukey functions in R.
Multiple Comparisons p. 5/23
More on Tukeys method
Confidence level exact when sample sizes are equalacross the a groups. If the sample sizes are unequal,confidence intervals are conservative.
R adjusts for slightly unbalanced design seecomments in help(TukeyHSD).
In R, use either TukeyHSD or simint withtype=Tukey option.
Multiple Comparisons p. 6/23
Tukey Graphical Output
Female mice diet example
20 10 0 10 20
95% familywise confidence level
Differences in mean levels of DIET
Multiple Comparisons p. 7/23
Compare the mean of one population with each of themeans of the remaining populations (e.g., compare acontrol to different treatments).
Uses the percentiles of a marginal distribution of amultivariate t distribution.
Multiple Comparisons p. 8/23
Example of Dunnetts Procedure
Random sample of 50 men matched for poundsoverweight was randomly separated into 5 equalgroups.
Each group was given exactly one of the weight lossagents: A, B, C, D, or E.
After a fixed period of time, each mans weight losswas recorded.
Multiple Comparisons p. 9/23
Weight Loss Boxplots
A B C D E
Multiple Comparisons p. 10/23
95 % onesided confidence intervals
0.0 0.5 1.0 1.5 2.0 2.5 3.0
Multiple Comparisons p. 11/23
Simultaneously compare all possible contrasts.
Uses a percentile of an F distribution to constructsimultaneous confidence intervals
(a 1)F0.05,a1,Na s
Results specify the constrasts significant differencefrom zero.
Multiple Comparisons p. 12/23
Six turkeys were randomly assigned to each of 5 dietgroups and fed for the same length of time.
A1: control + amount 1 of additive A
A2: control + amount 2 of additive A
B1: control + amount 1 of additive B
B2: control + amount 2 of additive B
Multiple Comparisons p. 13/23
control A1 A2 B1 B2
Multiple Comparisons p. 14/23
Scheffe simultaneous 95% CI
control vs treatment: (-4.28, -2.58)
A vs B: (-2.71, -1.19)
amount: (-2.69, -1.17)
A vs B by amount: (-0.312, 1.21)
Multiple Comparisons p. 15/23
treatment vs control: averaged over the 4 treatments,turkeys receiving a dietary additive gain significantlymore weight than ones not receiving an additive.
additive: turkeys receiving additive B gain significantlymore weight than turkeys receiving additive A.
amount: turkeys receiving amount 2 gain significantlymore weight than turkeys receiving amount 1.
interaction between additive and amount: the extentof increased weight gain as a result of receivingamount 2 rather than amount 1 is not significantlydifferent for the two additives
Multiple Comparisons p. 16/23
Extended Tukey Procedure
Can modify Tukey procedure to cover the family of allpossible contrasts when the sample sizes are equalacross groups.
cj yj q
Wider CI compared to Scheffes method.
Appropriate for situations with a small number of morecomplicated contrasts.
Multiple Comparisons p. 17/23
The standard tabular and graphical outputs do notconvey some aspects of a multiple comparisonanalysis.
For example, in the Tukey pairwise comparison, thestandard output just shows the CI for the difference.The mean of each group being compared is obscured.
The standard displays do not show the relativedistances between adjacent sorted sample means.
Multiple Comparisons p. 18/23
Mean-Mean Multiple Comparisons displays
Horizontal axis shows the contrast value (e.g., for acomparison between two groups, it would show thedifference between the two sample means).
Vertical axis shows the sample mean of eachsubgroup. This allows visualization of the relativedistances between the sample means of the differentsubgroups.
Multiple Comparisons p. 19/23
Example with Turkey data
Pairwise confidence intervals from Tukey procedure:R code
> turkeyci plot(turkeyci)
Multiple Comparisons p. 20/23
Pairwise confidence intervals for turkey data
95 % twosided confidence intervals
0 2 4 6
Multiple Comparisons p. 21/23
MMC plot with Turkey data
> tmp0 tmp1 plot(tmp1)
Multiple Comparisons p. 22/23
Pairwise confidence intervals for turkey data
multiple comparisons of means of wt.gain
5 0 5
simultaneous 95% confidence limits, Tukey method
6.98333333333333 A27 B1
Multiple Comparisons p. 23/23
Simultaneous InferencesCommon Multiple Comparisons ProceduresBonferroni MethodTukey ProcedureMore on Tukey's methodTukey Graphical OutputDunnett's ProcedureExample of Dunnett's ProcedureWeight Loss BoxplotsDunnett OutputScheffe's MethodTurkey DataTurkey BoxplotsScheffe simultaneous 95% CIContrast AnalysisExtended Tukey ProcedureGraphical DisplaysMMC PlotExample with Turkey dataPairwise confidence intervals for turkey dataMMC plot with Turkey dataPairwise confidence intervals for turkey data