Post on 12-Jan-2016
Chi Square
χ2
Parametric Statistics
• Everything we have done so far assumes that data are representative of a probability distribution (normal curve).
• We are making inferences about the parameters (statistics of the population) of the distribution.
• That is why this is called parametric statistics.
Non-Parametric Statistics
• If the data are not assumed to be part of a probability distribution, then there is no distribution to which to make inferences.
• The most frequent reason this happens is when data are not interval.
• To make inferences, some characteristic of the data must approximate a probability distribution.
• These are call non-parametric statistics.
Variables
• Nominal– Named groups (mode)
• Ordinal– Ordered named groups (median)
• Interval (Ratio)– Continuous scales (mean)
A Problem
• With nominal or ordinal data we can’t compare group means.– average category or average of low/medium/high
• But, there ought to be some way to know if the the distributions of responses in nominal or ordinal categories could have occurred by chance.
• The most common way to do this is with a χ2 (chi square).
Reading Selection by Source
Book Magazine Online
Contingency Tables (Cross Tabs)Test of Independence
• Are these numbers independent? If they aren’t then it means that one variable influences the other.
Women
Men
24
1922
42
30
36
Book Magazine Online
Contingency Tables (Cross Tabs)Test of Independence
• If gender isn’t influenced by reading preference then the proportions for each category of reading preference should be the same.
Women
Men
24
1922
42
30
36
Book Magazine Online
Contingency Tables (Cross Tabs)Test of Independence
• If reading preference isn’t influenced by gender then the proportions for each category gender should be the same.
Women
Men
24
1922
42
30
36
Book Magazine Online
Contingency Tables (Cross Tabs)Test of Independence
• If neither is influenced by the other then the proportions should be the same throughout the model.
Women
Men
24
1922
42
30
36
Book Magazine Online
• Is the the distribution of reading preference different by gender?
• Is the distribution of gender different by reading preference?
• Are these two variables related?
• (Null: the two variables are not related)
Chi SquareTests of Independence
• Given the Observed Frequencies there ought to be some way to imagine what the most likely number would be in each cell if the numbers were independent.
• This is kind of super averaging the counts in related cells and predicting what should be in the cell.
Contingency TablesTest of Independence
Women
Men
24
1922
42
30
36?
? ?
? ?
?
Book Magazine Online
Chi SquareTests of Independence
• Given the Observed Frequencies determine what would be in each cell if the variance in the rows and columns were accounted for—looking at row and column proportions simultaneously.
• This is done by:Row total x Column total/ Total
• This new set of values is called the Expected Frequencies
Contingency TablesTest of Independence
Women
Men
24
1922
42
30
3627.12
18.88 22.57
32.43 42.45
29.55
Book Magazine Online
Contingency TablesTest of Independence
Women
Men
24
1922
42
30
3627.12
18.88 22.57
32.43 42.45
29.55
Book Magazine Online
Column Total = 46
Row Total= 102
Sample Total = 173
(102 x 46)/173 = 27.12
Now What? (Computing a Chi Square)
• The gathered data are the Observed (Actual) Values.
• Expected Values—a computation of what should be in each cell based on the existing sample distribution.
• First the computer builds a model that represents the expected frequencies for each cell.
• Then the differences between the observed frequencies the expected frequencies are computed. (O – E)
Now What? (Computing a Chi Square)
• Because (O – E) might be negative each difference is squared.
• Since we want to know when the differences are comparatively big or small the real number difference has to be turned into a ratio.
• Now it is time to add all of these up: the sum of squared differences—chi square
• Last, given the df, the computed sum of squares is compared to a distribution of possible sum of squares.
(O – E)2
(O – E)2
E
(O – E)2
EΣ
5% of the area
Chi DistributionThis curve represents the probability of getting a given chi square.
(The sum of each of the squared differences divided by the expected frequency.)
There is one of these for every possible degrees of freedom df = (number of rows - 1) x (number of columns - 1)
• Sometimes the difference between the actual values and the expected values is so small that they it can be attributed to chance variation.
• If that is true we say that the variables are independent.
• Sometimes the difference between the actual values and the expected values is so large that it is worth talking about why those differences appeared. The difference is so large it is unlikely to have happened by chance. (p <.05)
Contingency TablesTest of Independence
Women
Men
24
1922
42
30
3627.12
18.88 22.57
32.43 42.45
29.55
chi square = 1.85
Book Magazine Online
5% of the area
Chi DistributionThis curve represents the probability of getting a given chi square.,
(The sum of all the differences squared divided by the total number of data points.)
There is one of these for every possible degrees of freedom (rows-1) x (columns-1)
1.85
2 df
Contingency TablesTest of Independence
Women
Men
24
1922
42
30
3627.12
18.88 22.57
32.43 42.45
29.55
chi square = 1.85p = 0.397
Book Magazine Online
χ2 Cautions
• When expected values drop below 5, the estimator has too much influence on the statistic.
• In other words, don’t do χ2 with small samples.
• Avoid over interpreting the results.
Caution: chi square only tells if the total difference was likely to occur by chance—not individual differences.
You can only say IF the variables are related—not how.
Women
Men
24
1922
42
30
3627.12
18.88 22.57
32.43 42.45
29.55
chi square = 1.85p = 0.397
Book Magazine Online
Using Excel to Compute χ2
The Chi Square Calculation
• You will never see the distribution, only the chi calculation and the p value.
• The chi table example is on the webpage
Reported Skill Level
Minimally Skilled Moderately Skilled Accomplished
Male Faculty 7 56 22
Female Faculty 48 153 79
Male Students 4 15 17
Female Students 2 15 4
χ2 (6) = 13.22, p = .04
Actual countsDegrees of Freedom
Chi Square valuep value
Table 1
Faculty and Student Self-Perception of Technology Competence by Gender
Chi Square(Goodness of Fit)
• Special case when the expected frequencies are predetermined.
• Are there important differences between what we are seeing and some assumed norm (the predetermined values)?
χ2 Goodness of Fit
• Counts in categories
• Compares actual counts to norms.
• Tests to see if the differences between the two are so large they are unlikely to have occurred randomly.
χ2 Goodness of Fit
χ2 Goodness of Fit
Exp
ecte
d
Act
ual
Act
ual
Act
ual
Exp
ecte
d
Exp
ecte
d
Chi Square = 8.09 p = .0175
χ2 Goodness of Fit
In Excel
=CHITEST(actual_range,expected_range)
Examples of Goodness of Fit χ2
• Technically all χ2 are Goodness of fit.
• Uses might be:– When no difference is predicted in categories.– Comparison of two iterations of the same group
to see if change has occurred.– Comparison to known distribution (i.e., z-
scores)
Chi Square
• Non-parametric comparisons are weak.
• They serve as a motivation for parametric analysis.
• Sometimes they are all that is possible.
Regression Analysis
Regression
• Correlation: Strength of association between two variables. (The amount of shared variance).
• Linear Regression: Defining a functional relationship between variables so that one can be predicted from the other.
Regression
• Linear Regression is done by figuring out how to put a line through the data where all of the points are as close to the line as possible (least squares).
Regression
• What you end up with is a formula for predicting one variable from another but this isn’t perfect because not all of the points will fall on the line.
Regression
• The added up amount that the actual points vary from the prediction line is called the residual.
• The inverse of the residual is the amount of variance in the model.
Regression
• Regression is a model for predicting how much variance in one variable (dependent) is explained by another (predictor or independent).
Regression
• If the residual is low then the variance explained will be high. The model captures most of the variance.
Regression
• If the residual is high then the variance explained will be low. The model doesn’t capture much of the variance.
Regression—Why Do This
• Linear regression has predictive power.• The strength of that power is related to the
residual—the amount of the variance explained by the prediction in the model.
• It might be worth finding out how to predict blood pressure from age. Linear regression would allow us to build a model to do this and tell us how much of the variance in the model would be explained by the prediction—not very much.
Excel
• From the Course Site: Content Scores
Multiple Regression
• Think about variance as a way to understand how confident you can be that you have a good model for prediction.
• It might be valuable to look at a whole bunch of variables at the same time and see how much variance they explain.
Multiple Regression
• Example: what combination of variables would do a good job of predicting life expectancy?
• Read the literature to make a good guess.
• Look at whole bunch of people whose age at death you know.
• Gather data on all the variables that you think might impact how long someone lives.
Multiple Regression
• Now build a model:– Look at age at death (dependent variable)– Look at one of the variables that might predict
death and see how much variance is explained in the model.
– Now add another variable into the model and see how much variance is explained now. (In a way it is like adding the variances together.)
– Keep going.
Multiple Regression
• At some point you can stop and say that if you have all of these data on predictor variables then you can do a good job of predicting the dependent variable. You confidence about this would be based on the amount of variance the model explains—in other words how good the model is in predicting the dependent variable.
Multiple RegressionThis is not easy to do.
• The calculations aren’t that difficult. EZAnalyze or even Excel can do this.
• The problem is that there are a bunch of underlying assumptions that have to be true.– Linear relationships– Large sample sizes (30 per variable)– Interval data– Recoded dummy variables if categorical– Strength of importance in the model (Beta)
Multiple RegressionThis is not easy to do.
• Multiple regression is a powerful tool.
• Don’t expect to be able to figure this out by yourself the first time.
• Find someone to help.
• Be prepared for it not to do what you expected.
Back to Linear Regression
• It is easy to do. If you can do a correlation you can do regression (you can set up a model to predict one variable from another).
• How would you use this in your classroom or school?
Interrater Reliability
Why Do This?
• The Right Thing To Do
• Garbage In—Garbage Out
• Fairness, Accuracy, Consistency, and the Avoidance of Bias (NCATE Standard 2)
Consistency
• Assessments are consistent when they produce dependable results or results that would remain constant on repeated trials. Essentially, in approaching consistency, the standards are requiring that the assessments and results be trustworthy.
(Assessing the Assessments, NCATE)
Measurement Design Issues
• A reminder: assessments can be both consistent and wrong.
• What do you want to know?• Assessment elements that are connected to
the standards.• Instrument response categories that define
the range that is possible within an element.• Response categories that match the analysis
you want to conduct.
Response Categories
• Simply providing a descriptor of the response categories is not enough(e.g., beginning, developing, advanced).
• Rubrics– Descriptors of each criterion at each
response level.
Rubrics
• Rubrics contain a large amount of text.
• Train to develop measurement precision.– Develop common understanding of terms
• Train for consistent instrument use.
• Train to eliminate rater bias.
Rater Effects
• Confirmatory Bias: Halo/Pitchfork– Without knowing the individual you anticipate
performance.
• Contrast/Carry Over Effects– Having seen a previous performance you assume
this observation will be similar.
• Leniency; Severity; Central Tendency Effect– Rating is done with preconceived notions of how
the group should be rated.
Rater Effects
• Similar-to-Me– Rating is not done from the rubric but from a
comparison to you.
• First Impression/Cosmetics– Pretty is better.
• Rater Drift– You lose interest or you start getting tough.
Interrater Reliability
• Analyzing ordinal data.– What is the correlation among rankings of
common items? (2 raters: Spearman;3+ raters: Spearman-Brown correction)
– What is the correlation among raters of common items? (Intraclass Correlation)
– What percent of raters are ±1 of the “normed” score? (Percent Agreement)
Percent AgreementPercent of ratings ± 1 of mode of responses17 Raters
Designing Training
• Select element rubrics for training.
• Select work samples that represent different levels of ability.
• Faculty norming of elements as pilot before training.
• Data gathering during training.
• Look for qualitative and quantitative analysis of experience.
Interrater Reliability
• It isn’t difficult to get this to work but it takes time.
• If you do it you will get better data: that means better quality and more defensible.