Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of...

24
Brief historical interlude Brief historical interlude •Karl Pearson (b. London1857; d. London 1936) •Considered the founder of mathematical statistics •Developed the chi-square test (published July 1900) •Coined the term “standard deviation” •Developed the product- moment correlation •Now you know who to blame

Transcript of Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of...

Page 1: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Brief historical interludeBrief historical interlude

•Karl Pearson (b. London1857; d. London 1936)•Considered the founder of mathematical statistics •Developed the chi-square test (published July 1900)•Coined the term “standard deviation”•Developed the product-moment correlation•Now you know who to blame

Page 2: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Chi-square test of significanceChi-square test of significance

•Χ2 tests how cases are distributed across a variable

One variable—how its distribution compares to a

second, given distribution.

Two variables—tests whether the two variables are

related (statistically) to each other•Popular for crosstabs

H0: every i.v. category has the same distribution

across the d.v. as the total—i.e., the i.v. doesn’t

matter (the two variables are unrelated).

Kayser
in career dv -- i.e. law, pol, etc columns: each PID % should equa the Total row's.l
Page 3: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Chi-square (cont.)Chi-square (cont.)

•The idea is that we calculate a statistic (i.e., we use a particular formula to calculate a number), where we know that the statistic has a certain distribution (in this case what’s called a chi-square distribution) that will occur simply by chance variation (in a sample). I.e., this statistic has a known (not normal) sampling distribution.

•Because we know the distribution of this statistic, we can tell whether the result we get (the number that we calculate) is larger than one would expect by chance.

•If it is, we conclude (in the case of two variables) that they are in fact related.

Page 4: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Chi-square (cont.)Chi-square (cont.)

•Here’s the formula:

•As you can see, once we figure out what frequencies we’re talking about, it’s not a very complicated formula. Nevertheless, SPSS will do the calculations for us.

(SPSS is really good at multiplying, dividing, stuff like that.)

frequency expected

)frequency expectedfrequency observed( 22

Page 5: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Chi-square (cont.)Chi-square (cont.)

•Here’s the distribution:

•Pearson’s great insight was to figure out that if there is in fact no relationship between two variables, if you draw repeated samples and calculate the formula on the last slide, you’ll get this kind of distribution simply because of chance variation.

•What we do in practice is to draw one sample and make the calculation. If the number we get is large enough, it tells us that we almost certainly didn’t get this number by chance. I.e., there really is a relation-ship between these variables in the population.

Page 6: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Chi-square (cont.)Chi-square (cont.)

•The distribution is what you would expect by chance. It’s very likely you would get a small value and very unlikely that you would get larger and larger values. (The x-axis isn’t labeled, but distribution starts at zero.)

•If you look back at the formula, you can see that the larger the difference between what you expected (given no relationship) and what you observed, the larger will be the chi-square number.

•If it’s large enough, it tells you that you almost certainly didn’t get this number by chance. I.e., there really is a relationship between these variables in the population.

Page 7: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Example: Does an observed frequency Example: Does an observed frequency distributiondistributionmatch the population distribution?match the population distribution?

•A study of grand juries in one county compared the demographic characteristics of jurors with the general population, to see if the jury panels were representative. The investigators wanted to know whether the jurors were selected at random from the population of this county. (This is an example of comparing one distribution with a second distribution. In this case, the second distribution is that of the population of the county on some characteristic.)

•The observed data are given on the next page.

Page 8: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Example continued….Example continued….

Age

County-wide population

Number of jurors

21 to 40 42% 5 (7.6%)

41 to 50 23% 9 (13.6%)

51-60 16% 19 (28.8%)

61 and over 19% 33 (50.0%)

Total 100% 66 (100.0%)

Observed data

Page 9: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Example continued….Example continued….

Age Observed Expected

21 to 40 5 0.42*66=27.7

41 to 50 9 0.23*66=15.2

51-60 19 0.16*66=10.6

61 and over 33 0.19*66=12.5

Total 66 66

Expected data

Page 10: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Example continued….Example continued….

As noted, the test statistic is:

Given our data,

frequency expected

)frequency expectedfrequency observed( 22

615.12

)5.1233(

6.10

)6.1019(

2.15

)2.159(

7.27

)7.275( 22222

Page 11: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

The chi-square tableThe chi-square table

To use the table, we need what is called the degrees of freedom.

In this case, the degree of freedom is the number of categories minus 1, or 4-1=3.

Just like the t-table, we look up 3 on the table, and then look for the test statistic and report the bounds.

Page 12: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Example concludedExample concluded

The df are (4-1)=3, and the p-value is therefore roughly 0. (Read values across the top as the area to the right of the critical value.)

So, with simple random sample, we conclude that it is almost impossible for a jury to differ this much from the county age distribution. The inference is that grand juries are not selected at random.

Page 13: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Now with a picture…Now with a picture…

Chi-sq critical values(area to the right of crit val.)

Page 14: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Testing independence with chi-Testing independence with chi-squaresquare

In a certain town, there are about 1 million voters. An SRS of 10,000 was chosen to study the relationship between gender and participation.

Are gender and voting independent?

We can answer this with a chi-square test.

First, we need the expected values for each cell.

Men Women Total

Voted 2792 3591 6383

Didn’t vote 1486 2131 3617

Total 4278 5722 10000

Page 15: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Expected valuesExpected values

The expected value for each cell is simply:

For men who voted, this is:

Thus, we have the following:

total

alcolumn tot totalrow E

6.273010000

42786383

E

Page 16: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Observed and expected valuesObserved and expected values

Observed Expected Difference

Men Women Men Women Men Women

Vote 2792 3591 2730.6 3652.4 61.4 -61.4

Didn’t vote 1486 2131 1547.4 2069.6 -61.4 61.4

We calculate the test statistic in the same way as before:

frequency expected

)frequency expectedfrequency observed( 22

Page 17: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

The test statisticThe test statistic

7.66.2069

)4.61(

4.1547

)4.61(

4.3652

)4.61(

6.2730

)4.61( 22222

The degrees of freedom are:

1

1)-(21)-(2

)1columns ()1 row (

-#-#d

Page 18: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

The p-valueThe p-value

The p-value is therefore around 1%.

Based on the p-value, we reject the null hypothesis and conclude that voting and gender are not independent. Or, in other words, men and women (in the population) don’t vote the same way.

Page 19: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Caveats, complicationsCaveats, complications

•Caveat: low frequencies can befuddle chi-square. When the expected frequency in a cell is below about 10, the values aren’t quite what they should be.

•One “solution” is to recode categories so the expected frequencies aren’t so small.

•There are some other “corrections” that one can make (but we won’t go into).

Page 20: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Testing caveat #1Testing caveat #1

There is nothing special about 5% or 1%. There is nothing special about 5% or 1%.

If our significance level is 5%, what is the difference between a p-value of 4.9% and a p-value of 5.1%?

One is statistically significant, and one is not.

But does that make sense?

One solution (not often used): report the p-value, not just the conclusion.

Page 21: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Testing caveat #2Testing caveat #2

Data snoopingData snooping

What does a significance level of 5% mean?

There is a 5% chance of rejecting the null hypothesis when it is actually true.

If our significance level is 5%, how many results would be “statistically significant” just by chance if we ran 100 tests?

We would expect 5 to be “statistically significant,”

and 1 to be “highly significant.”

Page 22: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Testing caveat #2 continued….Testing caveat #2 continued….

So what can we do?

1. One can state how many tests were run before statistically significant results turned up.

2. If possible, one can test one’s conclusions on an independent set of data.

3. Again, there are some statistical procedures that can help—basically playing off the idea that (at the 5% level) one should get 5% of the results significant merely by chance.

Page 23: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Testing caveat #3Testing caveat #3

Was the result important?Was the result important?

•What is the magnitude of the difference? In the example above, 65.3% of men voted, 62.8% of women voted). Chi-square doesn’t tell you that.

•Even the level of significance doesn’t tell you that. (Yes, it’s related, but it depends on the number of cases, so you can’t simply say that a difference significant at, say, the 1% level is really big.)

This leads directly to the next topic: measures of assoc.

Page 24: Brief historical interlude Karl Pearson (b. London1857; d. London 1936) Considered the founder of mathematical statistics Developed the chi-square test.

Testing caveat #3 conclusionTesting caveat #3 conclusion

The moral of the story is:

A statistically significant difference may not be important.

And…

An important difference may not be statistically significant.