# Multiple Comparisons - StatSci all pairwise comparisons by making use of ... control A1 A2 B1 B2 4...

date post

20-Mar-2018Category

## Documents

view

226download

2

Embed Size (px)

### Transcript of Multiple Comparisons - StatSci all pairwise comparisons by making use of ... control A1 A2 B1 B2 4...

Multiple ComparisonsOctober 16th & 18th, 2007

Reading: Chapter 7 HH

Multiple Comparisons p. 1/23

Simultaneous Inferences

Individual tests of hypotheses or confidence intervalsSuppose you try to control the Type I error of Khypothesis tests at level .

Pr(at least one Type I error) = 1 - (1 )K . Thisleads to an unacceptable error threshold.

Multiple comparison procedure: control thefamily-wise error rate (FWE):

FWE = Pr(reject at least one true hypothesis underany configuration of true and false hypotheses)

FDR = false discovery rate (the expected proportion offalsely rejected hypotheses).

Multiple Comparisons p. 2/23

Common Multiple Comparisons Procedures

Bonferroni method

Tukey procedure

Dunnett procedure

Scheffe and Extended Tukey: simultaneouslycomparing all possible contrasts.

Multiple Comparisons p. 3/23

Bonferroni Method

Bonferroni inequality: P (

Ri)

P (Ri).

Perform m related tests and conduct each test at levelm

: FWE .Conservative multiple comparison procedure

Useful in situations when the statistics associated withthe m inferences have nonidentical probabilitydistributions.

Multiple Comparisons p. 4/23

Tukey Procedure

Examines all pairwise comparisons by making use ofthe information about the joint distribution of thestatistics used in the inferences.

Less conservative than Bonferroni.

If there are a groups, there will be(

a2

)

pairwise tests.

Confidence intervals are constructed using criticalvalues from the Studentized range distribution.Intervals based on the Studentized range statistic,Tukey Honest Significant Differences method.

See ptukey and qtukey functions in R.

Multiple Comparisons p. 5/23

More on Tukeys method

Confidence level exact when sample sizes are equalacross the a groups. If the sample sizes are unequal,confidence intervals are conservative.

R adjusts for slightly unbalanced design seecomments in help(TukeyHSD).

In R, use either TukeyHSD or simint withtype=Tukey option.

Multiple Comparisons p. 6/23

Tukey Graphical Output

Female mice diet example

20 10 0 10 20

R.R50NP

R.R50N.R50

NPN.R50

R.R50N.R40

NPN.R40

N.R50N.R40

R.R50N.N85

NPN.N85

N.R50N.N85

N.R40N.N85

R.R50lopro

NPlopro

N.R50lopro

N.R40lopro

N.N85lopro

95% familywise confidence level

Differences in mean levels of DIET

Multiple Comparisons p. 7/23

Dunnetts Procedure

Compare the mean of one population with each of themeans of the remaining populations (e.g., compare acontrol to different treatments).

Uses the percentiles of a marginal distribution of amultivariate t distribution.

Multiple Comparisons p. 8/23

Example of Dunnetts Procedure

Random sample of 50 men matched for poundsoverweight was randomly separated into 5 equalgroups.

Each group was given exactly one of the weight lossagents: A, B, C, D, or E.

After a fixed period of time, each mans weight losswas recorded.

Multiple Comparisons p. 9/23

Weight Loss Boxplots

group

Wei

ght L

oss

A B C D E

8

9

10

11

12

13

Multiple Comparisons p. 10/23

Dunnett Output

Dunnett contrasts

95 % onesided confidence intervals

0.0 0.5 1.0 1.5 2.0 2.5 3.0

groupEgroupD

groupCgroupD

groupBgroupD

groupAgroupD

(

(

(

(

Multiple Comparisons p. 11/23

Scheffes Method

Simultaneously compare all possible contrasts.

Uses a percentile of an F distribution to constructsimultaneous confidence intervals

a

j=1

cj yj

(a 1)F0.05,a1,Na s

a

j=1

c2j

nj

s =

N =a

j=1 nj

Results specify the constrasts significant differencefrom zero.

Multiple Comparisons p. 12/23

Turkey Data

Six turkeys were randomly assigned to each of 5 dietgroups and fed for the same length of time.

Control diet

A1: control + amount 1 of additive A

A2: control + amount 2 of additive A

B1: control + amount 1 of additive B

B2: control + amount 2 of additive B

Multiple Comparisons p. 13/23

Turkey Boxplots

diet

Wei

ght G

ain

control A1 A2 B1 B2

4

6

8

10

Multiple Comparisons p. 14/23

Scheffe simultaneous 95% CI

control vs treatment: (-4.28, -2.58)

A vs B: (-2.71, -1.19)

amount: (-2.69, -1.17)

A vs B by amount: (-0.312, 1.21)

Multiple Comparisons p. 15/23

Contrast Analysis

treatment vs control: averaged over the 4 treatments,turkeys receiving a dietary additive gain significantlymore weight than ones not receiving an additive.

additive: turkeys receiving additive B gain significantlymore weight than turkeys receiving additive A.

amount: turkeys receiving amount 2 gain significantlymore weight than turkeys receiving amount 1.

interaction between additive and amount: the extentof increased weight gain as a result of receivingamount 2 rather than amount 1 is not significantlydifferent for the two additives

Multiple Comparisons p. 16/23

Extended Tukey Procedure

Can modify Tukey procedure to cover the family of allpossible contrasts when the sample sizes are equalacross groups.

a

j=1

cj yj q

2

sn

a

j=1

|cj |

Wider CI compared to Scheffes method.

Appropriate for situations with a small number of morecomplicated contrasts.

Multiple Comparisons p. 17/23

Graphical Displays

The standard tabular and graphical outputs do notconvey some aspects of a multiple comparisonanalysis.

For example, in the Tukey pairwise comparison, thestandard output just shows the CI for the difference.The mean of each group being compared is obscured.

The standard displays do not show the relativedistances between adjacent sorted sample means.

Multiple Comparisons p. 18/23

MMC Plot

Mean-Mean Multiple Comparisons displays

Horizontal axis shows the contrast value (e.g., for acomparison between two groups, it would show thedifference between the two sample means).

Vertical axis shows the sample mean of eachsubgroup. This allows visualization of the relativedistances between the sample means of the differentsubgroups.

Multiple Comparisons p. 19/23

Example with Turkey data

Pairwise confidence intervals from Tukey procedure:R code

> turkeyci plot(turkeyci)

Multiple Comparisons p. 20/23

Pairwise confidence intervals for turkey data

Tukey contrasts

95 % twosided confidence intervals

0 2 4 6

dietB2dietB1

dietB2dietA2

dietB1dietA2

dietB2dietA1

dietB1dietA1

dietA2dietA1

dietB2dietcontrol

dietB1dietcontrol

dietA2dietcontrol

dietA1dietcontrol

( )

( )

( )

( )

( )

( )

( )

( )

( )

( )

Multiple Comparisons p. 21/23

MMC plot with Turkey data

R code

> tmp0 tmp1 plot(tmp1)

Multiple Comparisons p. 22/23

Pairwise confidence intervals for turkey data

multiple comparisons of means of wt.gain

contrast value

5 0 5

simultaneous 95% confidence limits, Tukey method

mean

wt.gain

level

contrast

3.78333333333333control

5.5 A1

6.98333333333333 A27 B1

9.38333333333333 B2

A1

A2B1

B2

A1

A2B1

B2

control

dietA1dietcontrol

dietA2dietcontroldietB1dietcontrol

dietB2dietcontrol

dietA2dietA1dietB1dietA1

dietB2dietA1

dietB1dietA2

dietB2dietA2dietB2dietB1

Multiple Comparisons p. 23/23

Simultaneous InferencesCommon Multiple Comparisons ProceduresBonferroni MethodTukey ProcedureMore on Tukey's methodTukey Graphical OutputDunnett's ProcedureExample of Dunnett's ProcedureWeight Loss BoxplotsDunnett OutputScheffe's MethodTurkey DataTurkey BoxplotsScheffe simultaneous 95% CIContrast AnalysisExtended Tukey ProcedureGraphical DisplaysMMC PlotExample with Turkey dataPairwise confidence intervals for turkey dataMMC plot with Turkey dataPairwise confidence intervals for turkey data