1

Statistical Power

2

First:Effect Size

• The size of the distance between two means in standardized units (not inferential).

• A measure of the impact of an intervention based on the distributions of the samples in your study.

• We use two measures of effect size– Cohen’s d– Eta Squared (η2)

3

Effect Size: Cohen’s d• Cohen’s d equals the difference in the means divided by the

average of the standard deviations.• It describes the distance between the means in units of pooled

standard deviation (remember z scores?).• It is a standardized measure of the impact of a statistically

significant intervention (independent variable).

X2X1

4

Effect Size: Eta Squared (η2)• Eta squared is the ratio of between group variance (impact of the

intervention) to total variance. • As the between group variance becomes a larger portion of the total

variance (more intervention impact), eta squared gets closer to 1.• Eta squared is a standardized measure of the explanatory power of

the independent variable.

Between group variance

Total variance

Within group variance sum of squares betweensum of squares totalη2 =

5

Effect Size

Label d r r2 or η2

Extremely large effect 2.0 .707 .500

Very large effect 1.5 .600 .360

Large effect 0.8 .371 .138

Medium effect 0.5 .243 .059

Small effect 0.2 .100 .010

• Although all the measures of effect size represent the impact of the independent variable in somewhat different ways, they are equivalent.

• For an expanded version see the Effect Size table on the website

6

Effect Size

• In the real world how much difference did the independent variable make?

• Effect size is not based on inference. It is based on observed measures.

X2X1

Sample Mean

Sample Distribution

Sampling Distribution Mean for a Given Group Size

Sampling Distribution for a Given Group Size

Now:Inferential Mistakes

Avoiding Type II Errors

.05

Sampling Distribution of the Mean Theoretical distribution based on randomly selected groups of a given size. Means in this area

would appear randomly less than 5% of the time.

Avoiding Type II Errors

.05

Sometimes a mean score would not be identified as significant because it is likely to appear randomly more than 5% of the time.

Sometimes that mean score represents a real world change in what is being measured but the difference isn’t enough to be significant.

Type II Error

Avoiding Type II Errors

.05

Now a larger group size moves the point at which the alpha level appears and something that wasn’t significant becomes so.

Using larger groups reduces type II errors. .05

Avoiding Type II Errors

.05

Great. Using larger group sizes helps reduce type II errors.

But, the cost is that developing large samples is difficult.

We need to figure out how big the sample size needs to be to reasonably reduce type II errors but still keep the group as small as possible.

12

Power

• Power is defined as the probability of finding significance if it exists (avoiding type II errors).

• Eighty percent (.80) is accepted as a reasonable target power.

• If non-random change occurs it has an 80% probability of being observed.

13

Power

• There are 2 ways to use power calculations.• First, they can be used to figure out

appropriate sample sizes for a study.• Second, they can be used to evaluate the use of

a specific sample size after a study has been completed.

14

Power and Effect Size

• If a study shows larger effect sizes, smaller sample sizes will still be expected to show significance.

• Conversely, smaller effect sizes would require larger sample sizes.

• Fortunately, all of this can be read offof a table.

X2X1

15

Power and Effect SizeChoosing Sample Sizes When Designing a StudyUsing Power to Explain Results

16

While we are here …

• Remember we have talked about inferential errors when something appears significant but it really wasn’t?

• Type I errors

Avoiding Type I Errors

.05

Every 100 times a mean appear in the .05 area, 5 of them would have occurred randomly.

That means we would identify something as not random (significant) 5 times out of 100 and be wrong.

Type I Error

Avoiding Type I Errors

.01

Solution:Move the alpha level to .01

That means we would identify something as not random (significant) 1 time out of 100 and be wrong.

Less chance of a Type I Error

Avoiding Type I Errors

.05

But this is social science and there is no good reason to make it this difficult to demonstrate significance. .05 is a reasonable alpha level.

Avoiding Type I Errors

.05

With larger group sizes a given point moves to a smaller probability of appearing.

Using larger groups reduces type I errors.

.05

21

Sample Sizes

• We know that using larger sample sizes is statistically powerful but life isn’t that simple.

• Whenever possible use Power Analysis to help you be more confident you will find something if it is there.

• When things don’t appear to be significant at least now Power Analysis gives you something else to talk about to suggest what might be done to improve the quality of your data.

22

Examples1. Most of the studies in your lit review are

showing medium effects around 0.4 Cohen’s d. You want to be 90% sure you find non-random effects if they are there. Approximately how big does your sample need to be?

2. In your study you showed mean differences of 0.4 Cohen’s d but groups were not significantly different. Your sample size was 30. What was the probability of finding significant differences if they were there?

Analysis

Inferential Statistics

• Assumptions– Dependent variable is an interval measure of one

characteristic of a group.– Tests are based on knowing or assuming the

distribution of a population.– Statistics demonstrate if comparison samples are

from the same population.

Testing Group DifferencesIndependent Dependent Test

1 (1 group) 1 measure 2 times (intervention in between) Paired t-test

1 (2 groups) 1 measure (usually after the intervention) Independent samples t-test

1 (1 group) 1 measure 3 or more times (usually two after the intervention)

Repeated measures ANOVA

1 (2 or more groups) 1 measure (usually after the intervention) Single factor

ANOVA

Non-Parametric

1 group 2 non-interval measures Chi-square

EZA

EZA

EZA

EZA

EZA

Post Hoc TestsTest

ANOVA Similar group sizes Tukey’s

ANOVA Dissimilar group sizes Scheffe’s

EZA

Practical Significance(not inferential)

Test

Pooled Standard Deviation Cohen’s d

Ratio of variances Eta Squared EZA

Testing Group Differences(Things We Haven’t Done)

Independent Dependent Test

2 (2 or more groups) 1 measure Factorial (2-way ANOVA) Shows interaction

between groups2 (2 or more groups)

1 measure 2 or more times

ANCOVA (Analysis of Co-Variance) Allows for control of instance of dependent measure

2 or more variables

2 or more measures MANOVA (Multiple Analysis of Variance)

2 or more variables

2 or more measures 2 or more times

MANCOVA (Multiple Analysis of Co-Variance) Allows for control of instance of dependent measure.

Non-Parametric2 (2 or more groups) Non-interval Kruskal-Wallis (ranked sum)

EZA?

Correlational Statistics

• Assumptions– The relationship among the measures of two

characteristics is linear.– Compared measures come from individuals in the

same population– Correlations are not causal

How Do Variables Relate?Comparison Test

Association of 2 or more interval measures Pearson’s r

Association of 2 measures at least one of which is not interval (ranked comparisons)

Spearman’s ρ (rho)

Measure of internal consistency Cronbach’s alpha

Prediction based on association of 2 measures Linear Regression

Things We Haven’t DoneAssociation of 3 or more measures at least one of which is not interval (ranked comparisons)

Kendall’s τ (tau)

Prediction based on 3 or more associations Multiple Regression

Finding relationships among items in a set of items (data reduction)

Factor Analysis

EZA

EZA?

EZA