Statistics and Hypothesis Testing

Michael Ash

Lecture 5

But first, let’s finish some material from last time.

Summary of Main Points

I We will never know population parameters µY or σ2

Y, but we

will use data from samples to compute estimates Y and s 2

Y

and statistical theory to judge the quality of the estimatesfor addressing real-world questions.

I Statistical knowledge is couched around hypothesis testing:will the sample data permit us to accept a maintainedhypothesis or to reject it in favor of an alternative?

I We will use the sample mean to test hypotheses about thepopulation mean. What are some conequences of the focus onthe mean?

What is an estimator?

An estimator is a method of guessing a population parameter, forexample, the population mean µY , using a sample of data. Theestimate is the specific numerical guess that an estimator yields.Estimators of a parameter are indicated by inserting aˆ . Forexample, guesses about the population mean are labeled µ̂Y

What would be a good way to guess µY ?

One method of guessing the population mean is to take a sampleand compute the sample mean: Y =

∑n

i=1

1

nYi . Another way

would be to take whatever observation happens to come first onthe list, Y1 (or last Yn).

Why is Y a good estimator of µY ?

Unbiased We showed last time that E (Y ) = µY (“on average, thesample mean is equal to the population mean”), which isthe definition of unbiasedness. (Note, however, thatunbiasedness also holds for using the first observation Y1

as the estimator of µY : “On average, the first observationis equal to the population mean.”)

Consistent Y becomes closer and closer to (a better and betterestimate of) µY as the sample size grows. (Note, by theway, that Y1 does not become a better estimate of µY asthe sample size grows.)

Most efficient Y has varianceσ

2

Y

n, which turns out to be the lowest

possible variance among unbiased estimators of µY . (Notethat Y1 has variance σ2

Y, which is terrible by comparison.)

The book demonstrates that Y , which equally weights

each observations in∑n

i=1

1

nYi , has lower variance than

alternative weightings of the data. Alternative weightedaverages of all the data are better (lower variance) thanY1 but not as good as Y

Y is the least squares estimator of µY

Suppose that the data Y1,Y2, . . . ,Yn are spread along the numberline, and you can make one guess, m, about where to put anestimate of µY .

Y

Y

i

i

The criterion for judging the guess will be to maken∑

i=1

(Yi − m)2

as small as possible. (Translation: square the gap between eachobservations Yi and the guess and add up the sum of squaredgaps.) If the guess m is too high, then the small values of Yi willmake the sum of squared gaps get big. If the guess m is too low,then the big values of Yi will make the sum of squares gaps getbig. If m is just right, then the sum of squared gaps will be assmall as possible.

Y is the least squares estimator of µY

It turns out that Y , the sample mean, is the best guess (the guessthat makes the sum of squared gaps as small as possible).Choosing m = Y makes the following expression as small aspossible.

n∑

i=1

(Yi − m)2

We will use this method (keeping the sum of squared gaps as lowas possible) for defining the best guess again soon.

Random SamplingThe consequences of non-random sampling

1. Non-random samplesI Convenience samples and how hard it is to avoid them

I The Literary Gazette, Landon, and Roosevelt.

I Nonresponse biasI Attrition biasI Purposive sampling (e.g., for qualitative research)

2. High quality surveysI Current Population Survey

Hypothesis Testing

With statistical methods, we can test hypotheses about populationparameters, e.g., the population mean. For example: Does thepopulation mean of hour earnings equal $20 per hour?

I Define the null hypothesis

H0 : E (Y ) = µY ,0

H0 : E (hourly earnings) = $20 per hour

I Define the alternative hypothesis.

H1 : E (Y ) 6= µY ,0

H1 : E (hourly earnings) 6= $20 per hour

I Gather a sample of data and compute the actual sample mean

Hypothesis Testing

1. Gather a sample of data and compute the actual sample mean

2. If the null hypothesis were true, would the r.v. the samplemean likely be as big (or small) as the actual sample mean?

PrH0

[∣∣Y − µY ,0

∣∣ >∣∣∣Y act − µY ,0

∣∣∣]

(There is only one random variable in the precedingmathematical phrase. Can you find it?)

2.1 If so (the probability is large), “accept the null hypothesis”(which does not mean that the null hypothesis is true, simplythat the data do not reject it).

2.2 If not (the probability is small), “reject the null hypothesis” infavor of the alternative.

Important Notes on Hypothesis Testing

I Summary of the hypothesis-testing approach1. The null hypothesis is a hypothesis about the population mean.2. The null hypothesis (and the size of the sample) implies a

distribution of the sample mean.3. An actual sample of real-world data gives an actual value of

the sample mean.4. The test of the null hypothesis asks if the actual value of

the sample mean is likely under the implied distribution of thesample mean (likely if the null hypothesis is true).

I We learn about the population mean. For example, if we learnthat E (hourly earnings) > $20 per hour, this does not meanthat every worker earns more than $20 per hour! Do notconfuse the mean with the entire distribution.

I Do not confuse statistical significance and practicalsignificance. With a large enough sample, you can distinguishan hypothesized mean of $20 per hour from an hypothesizedmean of $20.07 per hour. Does anyone care? More on thislater.

The p-Value

p-value ≡ PrH0

[∣∣Y − µY ,0

∣∣ >∣∣∣Y act − µY ,0

∣∣∣]

This phrase expresses how likely the observed, actual sample meanY

actwould be to deviate from the null-hypothesized population

mean µY ,0 if the null hypothesis were true.Why can it deviate at all (if the null hypothesis is true)? Samplingvariation.But if the actual sample mean deviates “too much” from thenull-hypothesized population mean, then sampling variationbecomes an unlikely reason for the difference.

Defining “too much.”

We know that under the null hypothesis, the sample mean is arandom variable distributed in a particular way: N(µY ,0, σ

2

Y).

Because this is a normal distribution, we know exactly theprobability that the sample mean is more than any specifieddistance away from the hypothesized mean (if the hypothesizedmean is accurate).For example, it is less than 5 percent likely that the sample meanwill be more than 2 (really 1.96) standard deviations away from thehypothesized mean (if the hypothesized mean is accurate).

How likely is the observed value?

In words, the p-value is how likely the random variable Y is toexceed the observed actual Y

actif the null hypothesis is true.

As p falls, we become increasingly sure that the null hypothesis isnot true. (It’s really unlikely that we could have a sample meanthis big (small) if the null were true. We do have a sample meanthis big (small). Ergo, the null hypothesis is not true.)Convert to a standard normal problem

Convert to a standard normal problem

p-value = PrH0

[∣∣∣∣∣Y − µY ,0

σY

∣∣∣∣∣ >

∣∣∣∣∣Y

act − µY ,0

σY

∣∣∣∣∣

]

= PrH0

[|Z | >

∣∣∣∣∣Y

act − µY ,0

σY

∣∣∣∣∣

]

= PrH0

[Z < −

∣∣∣∣∣Y

act − µY ,0

σY

∣∣∣∣∣

]+ Pr

H0

[Z >

∣∣∣∣∣Y

act − µY ,0

σY

∣∣∣∣∣

]

= PrH0

[Z < −

∣∣∣∣∣Y

act − µY ,0

σY

∣∣∣∣∣

]+ Pr

H0

[Z < −

∣∣∣∣∣Y

act − µY ,0

σY

∣∣∣∣∣

]

= ΦH0

(−∣∣∣∣∣Y

act − µY ,0

σY

∣∣∣∣∣

)+ ΦH0

(−∣∣∣∣∣Y

act − µY ,0

σY

∣∣∣∣∣

)

= 2ΦH0

(−∣∣∣∣∣Y

act − µY ,0

σY

∣∣∣∣∣

)

Sample Variance, Sample Standard Deviation, Standard

Error

Why?

I The sample variance and sample standard deviation areinteresting in their own right as a description of the spread inthe data. Is income equally or unequally distributed? Dowinters vary from year to year?

I As we can estimate the population mean using the samplemean, we can estimate the population variance and standarddeviation using the sample variance and standard deviation.

I The sample variance and standard deviation of the underlyingdata are needed to estimate the variance and standarddeviation of the sample mean. (We need the latter as ameasure of how precisely the sample mean estimate thepopulation mean.)

Sample Variance

I An unbiased and consistent estimator of population variance

s2

Y≡ 1

n − 1

n∑

i=1

(Yi − Y

)2

I Definition of s2

Yis almost: compute the average squared

deviation of each observation from the population mean. But:

I The population mean µY is unknown; so instead we use Y ,

the sample meanI Because we had to use the sample data to compute the sample

mean Y , we used up one degree of freedom. So when we

compute the average we divide by n − 1 instead of n.

I If we used n instead of n − 1, we would slightly underestimatethe population variance. Note that the difference becomessmall as the sample-size grows (Dividing by n or by n − 1 isnot very different when n is very large.)

Sample Standard Deviation

The sample standard deviation is simply the square root of thesample variance and has the advantage of being in the same unitsas Y and Y . It is an unbiased and consistent estimator of thepopulation standard deviation.

sY =√

s2

Y=

√√√√ 1

n − 1

n∑

i=1

(Yi − Y

)2

Standard Error of Y

Recall that the standard deviation of Y is denoted σY

and equalsσY /

√n.

The standard error of Y is an estimator of the standard deviationof Y using nothing but the sample data:

standard error of Y = SE (Y ) = σ̂Y

= sY /√

n

It is ok to substitute sY for σY because sY is a consistentestimator of σY .

Summary: Why standard error?

I Can be computed entirely from the sample data.

I Expresses the expected spread in sample means if multiplesamples were taken from the population.

I Measures the precision of the sample mean as an estimate ofthe population mean.

I Increases with the spread in the (sample) data, sY , anddecreases with the sample size, n.

Practical Hypothesis Testing

Form a t-Statistic

t ≡ Yact − µY ,0

SE (Y )

I µY ,0 comes from the null hypothesis

I Y and SE (Y ) come from the data.

t is approximately distributed N(0, 1) when the sample n is large.We will use the normal approximation for computing the p-value.

p-value = 2Φ(− |t|)

The Prespecified Significance Level

A standard in the social sciences is that a p-value below 0.05 isappropriate grounds for rejecting the null hypothesis. Thiscorresponds to a t-statistics more than 1.96 (almost exactly 2)standard deviations away from zero.Reject H0 if p < 0.05is equivalent toReject H0 if |t| > 1.96

Size of a test: probability of erring in rejection

By pure sampling chance, one time out of twenty, or 5 percent ofthe time, the data will reject the null hypothesis even though thenull hypothesis is true. For more sensitive matters than mere socialscience, a higher standard (lower p-value) may be required.

An example

Is the mean wage among recent college graduates $20 per hour?

H0 : E (Y ) = µY ,0

H0 : E (hourly earnings) = $20 per hour

In a sample of n = 200 recent college graduates, the sampleaverage wage, Y

act= $22.64. STOP RIGHT HERE. Why doesn’t

this prove immediately that the mean wage among recent collegegraduates is $22.64, obviously above $20?

Average wage example, continued

The sample standard deviation is sY = $18.14.

I Compute the standard error of Y

SE (Y ) = sY /√

n

= 18.14/√

200

= 1.28

I Compute the t-statistic

t =Y

act − µY ,0

SE (Y )

=22.64 − 20

1.28= 2.06

Now how likely is it that our sample would generate a t-statistic of2.06 when we would expect a t-statistic of zero under the nullhypothesis?

Average wage example, continued

I Compute the p-value

p = 2Φ(−|t|)= 2Φ(−2.06)

= 2(0.0197)

= 0.039, or 3.9 percent

Conclusion: it is fairly unlikely that the mean earnings amongrecent college graduates is $20 per hour given that our sample of200 had mean earnings of $22.64 (and standard deviation of$18.14). If we were using the conventional significance level of 5percent, we would reject the null hypothesis.

See Key Concept 3.5

Statistical analogy to U.S. law (outdated)

I Formulation and testing of the null hypothesis is equivalent topresumption of innocence.

I Rejecting the null hypothesis based on sample data isequivalent to finding that the evidence indicates guilt.

I Four things can happen in court: a guilty person can be foundguilty; a guilty person can be found not guilty; an innocentperson can be found guilty; and an innocent person can befound not guilty. Some of these are errors.

The p-value is the probability that the test rejects the nullhypothesis even though the null hypothesis is true. Choosing acritical p-value of 0.05 means that we will accept conviction of aninnocent 5 percent of the time. (If that’s upsetting, then it’s goodto know that “guilt beyond a reasonable doubt” is typically held torequire more than 95 percent certainty.)

Type I and Type II errors

Null Hypothesis is reallyTrue False

Null Not Rejected Correct acceptance Type II error (β)Hypothesis

is Rejected Type I error (α) Correct rejection (1 − β)

size power

We set α, the size of the test or the probability of Type I error, bychoosing a critical p-value (typically 5 percent). However, themore stringent we are about not rejecting when the null hypothesisis true, the more likely we are to commit a Type II error, failing toreject when the null hypothesis is false and ought to be rejected.We would ideally like the size (probability of false rejection) to below and the power (probability of correction rejection) to be high.But we consider false rejection a more serious problem; so wespecify size and live with the resulting power.

From Hypothesis Testing to Confidence Interval

We never learn the true value of µY , the population mean of Y

(the object inside the box).But from sample data we can specify a range, the confidence

interval that is 95 percent (or any other pre-specified percent, theconfidence level) likely to include the population mean.Thought experiment: using hypothesis testing, we couldexhaustively test the null hypothesis for all possible values of µY ,0

and then keep, for the confidence interval, all of those that are notrejected by the data. Since the true value of µY will be rejectedless than 5 percent of the time, it is 95 percent likely to be on thelist.

From Hypothesis Testing to Confidence Interval

Practical approach: to construct a 95-percent confidence intervalfor µY from the sample data. All values of µY ,0 within 1.96standard errors of Y will not be rejected.A 95 percent confidence interval for µY is

Y − 1.96SE (Y ) ≤ µY ≤ Y + 1.96SE (Y )

(Reminder: 1.96 corresponds to the 95 percent confidence intervalaccording to the standard normal distribution.)

Less and more cautious confidence intervals

I A slightly narrower interval is slightly less likely to contain thetrue value of µY :

Y − 1.64SE (Y ) ≤ µY ≤ Y + 1.64SE (Y )

will contain the true value with 90 percent confidence.

I A slightly wider interval is slightly more likely to contain thetrue value of µY :

Y − 2.58SE (Y ) ≤ µY ≤ Y + 2.58SE (Y )

will contain the true value with 99 percent confidence.

See Key Concept 3.7

Example: 95 percent CI

In a sample of n = 200 recent college graduates, the sampleaverage wage, Y

act= $22.64. The sample standard deviation is

sY = $18.14. Recall we computed that SE (Y ) = 1.28The 95-percent confidence interval is

Y − 1.96SE (Y ) ≤ µY ≤ Y + 1.96SE (Y )

22.64 − 1.96(1.28) ≤ µY ≤ 22.64 + 1.96(1.28)

22.64 − 2.51 ≤ µY ≤ 22.64 + 2.51

20.13 ≤ µY ≤ 25.15

It is 95 percent likely that the mean falls between $20.13 and$25.15 per hour. (Note that this range does not include $20,which we had earlier rejected, with a p of 5 percent.)

Example: 99 percent CI

The 99-percent confidence interval is

Y − 2.58SE (Y ) ≤ µY ≤ Y + 2.58SE (Y )

22.64 − 2.58(1.28) ≤ µY ≤ 22.64 + 2.58(1.28)

22.64 − 3.30 ≤ µY ≤ 22.64 + 3.30

19.34 ≤ µY ≤ 25.94

It is 99 percent likely that the mean falls between $19.34 and$25.94 per hour. (Note that this range does include $20, a nullhypothesis with a p of 3.9 percent.)

Next time we do something from the real world!