Multiple Comparisons - StatSci all pairwise comparisons by making use of ... control A1 A2 B1 B2 4...

download Multiple Comparisons - StatSci  all pairwise comparisons by making use of ... control A1 A2 B1 B2 4 6 8 10 Multiple Comparisons ... Multiple Comparisons ...

of 23

  • date post

    20-Mar-2018
  • Category

    Documents

  • view

    215
  • download

    2

Embed Size (px)

Transcript of Multiple Comparisons - StatSci all pairwise comparisons by making use of ... control A1 A2 B1 B2 4...

  • Multiple ComparisonsOctober 16th & 18th, 2007

    Reading: Chapter 7 HH

    Multiple Comparisons p. 1/23

  • Simultaneous Inferences

    Individual tests of hypotheses or confidence intervalsSuppose you try to control the Type I error of Khypothesis tests at level .

    Pr(at least one Type I error) = 1 - (1 )K . Thisleads to an unacceptable error threshold.

    Multiple comparison procedure: control thefamily-wise error rate (FWE):

    FWE = Pr(reject at least one true hypothesis underany configuration of true and false hypotheses)

    FDR = false discovery rate (the expected proportion offalsely rejected hypotheses).

    Multiple Comparisons p. 2/23

  • Common Multiple Comparisons Procedures

    Bonferroni method

    Tukey procedure

    Dunnett procedure

    Scheffe and Extended Tukey: simultaneouslycomparing all possible contrasts.

    Multiple Comparisons p. 3/23

  • Bonferroni Method

    Bonferroni inequality: P (

    Ri)

    P (Ri).

    Perform m related tests and conduct each test at levelm

    : FWE .Conservative multiple comparison procedure

    Useful in situations when the statistics associated withthe m inferences have nonidentical probabilitydistributions.

    Multiple Comparisons p. 4/23

  • Tukey Procedure

    Examines all pairwise comparisons by making use ofthe information about the joint distribution of thestatistics used in the inferences.

    Less conservative than Bonferroni.

    If there are a groups, there will be(

    a2

    )

    pairwise tests.

    Confidence intervals are constructed using criticalvalues from the Studentized range distribution.Intervals based on the Studentized range statistic,Tukey Honest Significant Differences method.

    See ptukey and qtukey functions in R.

    Multiple Comparisons p. 5/23

  • More on Tukeys method

    Confidence level exact when sample sizes are equalacross the a groups. If the sample sizes are unequal,confidence intervals are conservative.

    R adjusts for slightly unbalanced design seecomments in help(TukeyHSD).

    In R, use either TukeyHSD or simint withtype=Tukey option.

    Multiple Comparisons p. 6/23

  • Tukey Graphical Output

    Female mice diet example

    20 10 0 10 20

    R.R50NP

    R.R50N.R50

    NPN.R50

    R.R50N.R40

    NPN.R40

    N.R50N.R40

    R.R50N.N85

    NPN.N85

    N.R50N.N85

    N.R40N.N85

    R.R50lopro

    NPlopro

    N.R50lopro

    N.R40lopro

    N.N85lopro

    95% familywise confidence level

    Differences in mean levels of DIET

    Multiple Comparisons p. 7/23

  • Dunnetts Procedure

    Compare the mean of one population with each of themeans of the remaining populations (e.g., compare acontrol to different treatments).

    Uses the percentiles of a marginal distribution of amultivariate t distribution.

    Multiple Comparisons p. 8/23

  • Example of Dunnetts Procedure

    Random sample of 50 men matched for poundsoverweight was randomly separated into 5 equalgroups.

    Each group was given exactly one of the weight lossagents: A, B, C, D, or E.

    After a fixed period of time, each mans weight losswas recorded.

    Multiple Comparisons p. 9/23

  • Weight Loss Boxplots

    group

    Wei

    ght L

    oss

    A B C D E

    8

    9

    10

    11

    12

    13

    Multiple Comparisons p. 10/23

  • Dunnett Output

    Dunnett contrasts

    95 % onesided confidence intervals

    0.0 0.5 1.0 1.5 2.0 2.5 3.0

    groupEgroupD

    groupCgroupD

    groupBgroupD

    groupAgroupD

    (

    (

    (

    (

    Multiple Comparisons p. 11/23

  • Scheffes Method

    Simultaneously compare all possible contrasts.

    Uses a percentile of an F distribution to constructsimultaneous confidence intervals

    a

    j=1

    cj yj

    (a 1)F0.05,a1,Na s

    a

    j=1

    c2j

    nj

    s =

    N =a

    j=1 nj

    Results specify the constrasts significant differencefrom zero.

    Multiple Comparisons p. 12/23

  • Turkey Data

    Six turkeys were randomly assigned to each of 5 dietgroups and fed for the same length of time.

    Control diet

    A1: control + amount 1 of additive A

    A2: control + amount 2 of additive A

    B1: control + amount 1 of additive B

    B2: control + amount 2 of additive B

    Multiple Comparisons p. 13/23

  • Turkey Boxplots

    diet

    Wei

    ght G

    ain

    control A1 A2 B1 B2

    4

    6

    8

    10

    Multiple Comparisons p. 14/23

  • Scheffe simultaneous 95% CI

    control vs treatment: (-4.28, -2.58)

    A vs B: (-2.71, -1.19)

    amount: (-2.69, -1.17)

    A vs B by amount: (-0.312, 1.21)

    Multiple Comparisons p. 15/23

  • Contrast Analysis

    treatment vs control: averaged over the 4 treatments,turkeys receiving a dietary additive gain significantlymore weight than ones not receiving an additive.

    additive: turkeys receiving additive B gain significantlymore weight than turkeys receiving additive A.

    amount: turkeys receiving amount 2 gain significantlymore weight than turkeys receiving amount 1.

    interaction between additive and amount: the extentof increased weight gain as a result of receivingamount 2 rather than amount 1 is not significantlydifferent for the two additives

    Multiple Comparisons p. 16/23

  • Extended Tukey Procedure

    Can modify Tukey procedure to cover the family of allpossible contrasts when the sample sizes are equalacross groups.

    a

    j=1

    cj yj q

    2

    sn

    a

    j=1

    |cj |

    Wider CI compared to Scheffes method.

    Appropriate for situations with a small number of morecomplicated contrasts.

    Multiple Comparisons p. 17/23

  • Graphical Displays

    The standard tabular and graphical outputs do notconvey some aspects of a multiple comparisonanalysis.

    For example, in the Tukey pairwise comparison, thestandard output just shows the CI for the difference.The mean of each group being compared is obscured.

    The standard displays do not show the relativedistances between adjacent sorted sample means.

    Multiple Comparisons p. 18/23

  • MMC Plot

    Mean-Mean Multiple Comparisons displays

    Horizontal axis shows the contrast value (e.g., for acomparison between two groups, it would show thedifference between the two sample means).

    Vertical axis shows the sample mean of eachsubgroup. This allows visualization of the relativedistances between the sample means of the differentsubgroups.

    Multiple Comparisons p. 19/23

  • Example with Turkey data

    Pairwise confidence intervals from Tukey procedure:R code

    > turkeyci plot(turkeyci)

    Multiple Comparisons p. 20/23

  • Pairwise confidence intervals for turkey data

    Tukey contrasts

    95 % twosided confidence intervals

    0 2 4 6

    dietB2dietB1

    dietB2dietA2

    dietB1dietA2

    dietB2dietA1

    dietB1dietA1

    dietA2dietA1

    dietB2dietcontrol

    dietB1dietcontrol

    dietA2dietcontrol

    dietA1dietcontrol

    ( )

    ( )

    ( )

    ( )

    ( )

    ( )

    ( )

    ( )

    ( )

    ( )

    Multiple Comparisons p. 21/23

  • MMC plot with Turkey data

    R code

    > tmp0 tmp1 plot(tmp1)

    Multiple Comparisons p. 22/23

  • Pairwise confidence intervals for turkey data

    multiple comparisons of means of wt.gain

    contrast value

    5 0 5

    simultaneous 95% confidence limits, Tukey method

    mean

    wt.gain

    level

    contrast

    3.78333333333333control

    5.5 A1

    6.98333333333333 A27 B1

    9.38333333333333 B2

    A1

    A2B1

    B2

    A1

    A2B1

    B2

    control

    dietA1dietcontrol

    dietA2dietcontroldietB1dietcontrol

    dietB2dietcontrol

    dietA2dietA1dietB1dietA1

    dietB2dietA1

    dietB1dietA2

    dietB2dietA2dietB2dietB1

    Multiple Comparisons p. 23/23

    Simultaneous InferencesCommon Multiple Comparisons ProceduresBonferroni MethodTukey ProcedureMore on Tukey's methodTukey Graphical OutputDunnett's ProcedureExample of Dunnett's ProcedureWeight Loss BoxplotsDunnett OutputScheffe's MethodTurkey DataTurkey BoxplotsScheffe simultaneous 95% CIContrast AnalysisExtended Tukey ProcedureGraphical DisplaysMMC PlotExample with Turkey dataPairwise confidence intervals for turkey dataMMC plot with Turkey dataPairwise confidence intervals for turkey data