Theory of tests

Theory of testsThomas INGICCO

1 – Principle of tests (hypotheses)2 – A first example with R3 – Probabilities (law of random)4 – Statistic and its law (test variable)5 – Risk α and errors6 – p-value7 – Summary

E. Munch, The scream

Theory of tests

Example 1 Let’s consider dices. I roll dices 4 times and obtain 4 times different

results -> I deduce that the dices are not fake.So I am doubting now and want to check. So I roll the dices once

again 4 times and I obtain 4 times the same result -> I change my opinion and conclude that the dices are fake.

How do I know the truth? I only have 8 rolls so a small sample for an infinite population… -> I need a statistical test.

Theory of tests

Example 1 Let’s consider dices. I roll dices 4 times and obtain 4 times different

results -> I deduce that the dices are not fake.So I am doubting now and want to check. So I roll the dices once

again 4 times and I obtain 4 times the same result -> I change my opinion and conclude that the dices are fake.

How do I know the truth? I only have 8 rolls so a small sample for an infinite population… -> I need a statistical test.

Example 2I study two collections of lithic tools, one from northern Luzon, the

other one from Mindoro.I sample 20 tools in each population. The mean length of the 20

artefacts in the first population is 110 mm. In the second population it is 130mm.

May I deduce that the lithic tools from Mindoro are longer and then that the prehistoric group are: 1) Different, 2) Have different abilities to knap, 3)Produce tools with different interests ?

If I do so, what is my risk to be wrong? -> I need a statistical test

Theory of tests

Investigation or inventory : All the operations aiming at collecting in a systematic way, data or informations related to a group of individual or elements.

What do we test and why do we test?


Individual : Element = Statistical unit = Basal unit = what we sample




Population : Universe = Statistic assemblage = All the individuals





Sample : Part of the population that we observe


Probability







Probability Statistic










Observation: Measured value for each sampled individual

The principle of test

First step :

• We define the hypotheses: H0 and H1.H0 = Null hypothesisH1 = Alternative hypothesis

Most of the time, H0 defines an absence of structure in the data.

Example 1: Chi2 TestH0 = Independance between two qualitative variablesH1 = Dependance

Example 2: Comparison of a mean to a theoretical meanH0 = Means are equalH1 = Means are differentH1 = The observed mean is bigger than the theoretical meanH1 = The observed mean is smaller than the theoretical mean

Example 3: Comparison of a mean to a theoretical meanH0 = The difference between the means is of 5cmH1 = The difference between the means is not of 5cm


Nul hypothesis H0 versus Alternative hypothesis H1

We start by considering that H0 is True.

We test H0: 1- If the test determines that H0 is False, then I accept H1 (we reject H0)2- If my test does not determine that H0 is False, I cannot reject this hypothesis (we keep H0)

-> A theory is replaced by another one only if it can be rejected.


Second step :

• Once the hypotheses H0 and H1 defined, we determine the variable of the test (the statistic) we need to reject H0.

It exists two categories of tests:

- Parametric tests: The distribution of probability of the measured variable in the targeted population follows a law. The analysed data can be modelised according to a known law. Then these tests will focus on the parameters of this law (mean, variance,…).

- Non-parametric tests (or distribution free test): The distribution of the measured variable does not matter. These tests do not focus on the distribution parameters of the data. They are used for small samples (>10) and/or for variables that do not follow a normal law.


Second step :

• Once the hypotheses H0 and H1 defined, we determine the variable of the test (the statistic) we need to reject H0.

In the case of Chi2, the statistic is:

In the case of the comparison of a mean to a theoretical mean, the statistic is:

𝜒 ²=∑ 𝑖 , 𝑗 (𝑂𝑖𝑗−𝐸 𝑖𝑗) ²𝐸𝑖𝑗

𝑇=𝑋 −𝜇0

√�̂� ² /𝑛


Why to calculate a statistic?

-> Because we know the law of probability of this statistic under the hypothesis H0, meaning if H0 is true.

This is this law of probability that would permit us to determine if « there are some chances » that we obtained these data if H0 is true, or in the opposite, if « there are very few chances » that we obtained these data if H0 is true. In this second case, we will deduce that according to our data, H0 is most probably False.

Let’s have an example !

Concretly, how to realize a test in R?

1- Determination of the hypotheses H0 and H1 which permits to clarify the scientific question.

2- Choice of the test, search for the function in R.

3- Use the help for the test, so you know how to make the test and how it works. In other words, you understand what you do, which is not the case in Excel.

Example 1:Question: Is the high of a ceramic correlated with the width of its openning?

How to make a test

Write the following instructions in R window.

Take the time to understand the instructions. Do not forget to use the help in R and to comment the

meaning of each instruction.

How to make a test

Ceram <- read.table("K:/Cours/Philippines/Statistics-210/Data/Ceramics.txt",header=TRUE)

str(Ceram)

plot(Ceram$W.mouth,Ceram$H.rim)




How to make a test


str(Ceram)


par(mfrow=c(1, 2))hist(Ceram$W.mouth)hist(Ceram$H.rim)




How to make a test


str(Ceram)



apropos("test")help.search("test")?cor.test




How to make a test


str(Ceram)



apropos("test")help.search("test")?cor.testcor.test(Ceram$W.mouth,Ceram$H.rim)

Understand the result of a test

The P-value – a question of probabilities

I flip 50 times a coin. Is the coin fake?

We consider a statistic:X= the number of times Heads is chosen in my experiment.

H0= the coin is not fake.H1= the coin is fake.

If H0 is True, what should be the value of X? Meaning how many times Heads should be obtained in 50 flips?

Is it 25?


I flip 50 times a coin. Is the coin fake?

We consider a statistic:X= the number of times Heads is chosen in my experiment.

H0= the coin is not fake.H1= the coin is fake.

If H0 is True, what should be the value of X? Meaning how many times Heads should be obtained in 50 flips?

Is it 25?

-> NO. In 50 flips, I may obtain between 0 and 50 Heads, but, if the coin is not fake, the probability (the chance) to obtain 0 Heads or 50 Heads is low, while the probability to obtain 23, 24, 25, 26, 27 Heads is stronger.


Write the following instructions in R window. We are going to estimate the law of probability of X from 1000 experiments. One experiment correspond to 50 flips.



Take the tim to understand the instructions. Do not forget to use

the help of R and to take notes on the meaning of each

instruction.

# statfun is a function that makes an experiment (50 flips of a coin) and returns the number of Heads obtained.

coin <- c("T", "H")statfun <- function(i){simu <- sample(coin, 50, replace = TRUE, prob = c(0.5, 0.5))# nbHeads <- table(simu)[1]nbHeads <- table(simu)[2]return(nbHeads)}res4 <- sapply(1:1000, statfun)res4


table(res4)names(table(res4))as.numeric(names(table(res4)))Head <- as.numeric(names(table(res4)))Probability <- table(res4)/1000plot(x, y, type="h")

x <- 0:50y <- dbinom(x, 50, 0.5)plot(x, y, type = "h")





instruction.

Binomial lawB(N,p)B(50,0.5)

Number of selectionProbability of success

We will note that this law depends on two parameters.The probabilities of each possible event from this plot are true only if p=0.5 and N=50. Meaning that the probability to obtain Heads for every flip is of 0.5, so 50%, and the number of flips is 50.

p=50 is True only if the coin is not fake.


k: each possible value within the discrete variable X.

f(k): frequency associated to each value = probability associated to k.

F(k): sum of the probabilities f(k) on the left or right of k regarding our interest.

F(k): it is the probability that X is upper or lower/equal to a value k.F(k)upper = P(X>k)F(30) upper = P(X>30)=0.118

cumsum(Probability)sum(Probability[which(rownames(Probability)=="31"):length(Probability)])

k: each possible value within the continuous variable X.

f(k): probability distribution of X = probability density associated to k.

F(k): area under the curve f(k) on the left or right of k regarding our interest.

F(k): it is the probability that X is upper or lower/equal to a value k.F(k)upper = P(X>k)F(xk)lower - F(xj)lower = P(xk<xi≤xk)

Laws of probability

Binomial law

dbinom : probability f(k) of the variable Xpbinom : function of repartition of F(k) of the variable Xqbinom : give the value k of the variable X for a given value of F(k)rbinom : generates random values for the variable X considering

probabilitiesNormal law

dnorm : probability f(k) of the variable Xpnorm : function of repartition of F(k) of the variable Xqnorm : give the value k of the variable X for a given value of F(k)rnorm : generates random values for the variable X considering

probabilitiesChi2 law

dchisq : probability f(k) of the variable Xpchisq : function of repartition of F(k) of the variable Xqchisq: give the value k of the variable X for a given value of F(k) rchisq: generates random values for the variable X considering

probabilities

Laws of probability

Binomial law

dbinom : probability f(k) of the variable Xpbinom : function of repartition of F(k) of the variable Xqbinom : give the value k of the variable X for a given value of F(k)rbinom : generates random values for the variable X considering

probabilities

Laws of probability

Binomial law

X: random variableP(X=k): function of repartition the variable X, so f(k): Combination of order k of the n elementsp: probability of the event 1 black ball for one selection, so n: number of selection with replacement K: number of black balls

𝑃 ( 𝑋=𝑘 )=𝐶𝑛❑𝑘×𝑝𝑘×(1−𝑝)𝑛−𝑘

Laws of probability

Binomial law



The binomial law is grounded on two exclusive elements, « black » and « white » for example when starting a chess game; or « boy » and « girl »; or « yes » or « no » when you date a girl.Each of these events is associated to a probability of appearance.The binomial law gives the probability corresponding to many apparitions.

You have 50 balls in a bag: 10 black, 40 red.The probability to have a random black ball after 1 selection is:

If the selection is randomly done, then the probability is p=10/50=0.2

𝑝=𝑃 (1𝑏𝑙𝑎𝑐𝑘 /1𝑠𝑒𝑙𝑒𝑐𝑡𝑖𝑜𝑛)

Laws of probability

Binomial law



Let’s calculate the probability to have 2 black balls when selecting 4 balls with replacement (knowing that the probability to get one black is of 0.2):p=0.2; n=4; k=2choose(n,k)*p^k*(1-p)^(n-k)

Laws of probability

Binomial law



Let’s calculate the probability to have 2 black balls when selecting 4 balls with replacement (knowing that the probability to get one black is of 0.2):p=0.2; n=4; k=2choose(n,k)*p^k*(1-p)^(n-k)dbinom(k, n, p)

Laws of probability

Binomial law



Let’s calculate the probability to have 2 black balls when selecting 4 balls with replacement (knowing that the probability to get one black is of 0.2):p=0.2; n=4; k=2choose(n,k)*p^k*(1-p)^(n-k)dbinom(k, n, p)The probability P(X=2) is then of 15.36%

Laws of probability

Binomial law



Let’s draw the function of repartition (of probability) of the law B(n=4, p=0.2):p=0.2; n=4; k=0:4plot(k, dbinom(k, n, p), pch=16, cex=2, xlim=range(0, 5), ylim=range(0, 0.5), xlab=« Number of black balls k", ylab="Probability f(k)", cex.lab=1.5, cex.axis=1.5, bty="l")

table(res4)names(table(res4))as.numeric(names(table(res4)))x <- as.numeric(names(table(res4)))y <- table(res4)/1000plot(x, y, type="h")

x <- 0:50y <- dbinom(x, 50, 0.5)plot(x, y, type = "h")





instruction.

Binomial lawB(N,p)B(50,0.5)B(Number of selection, Probability of success)

We will note that this law depends on two parameters.The probabilities of each possible event from this plot are true only if p=0.5 and N=50. Meaning that the probability to obtain Heads for every flip is of 0.5, so 50%, and the number of flips is 50.

p=50 is True only if the coin is not fake.


Laws of probability

Binomial law

Let’s calculate the probability to have 2 black balls when selecting 4 balls with replacement (knowing that the probability to get one black is of 0.2).

The question is to find the probability P(X ≥ 2).

We may then calculate the sum P(X = 2) + P(X = 3) + P(X = 4)

p=0.2; n=4dbinom(2,n,p)+dbinom(3,n,p)+dbinom(4,n,p)

We may also exclude the probability P(X = 0) to have 0 black ball and the probability P(X = 1) to have 1 black ball. The whole function of repartition being equal to 1:

p=0.2; n=41-dbinom(0,n,p)-dbinom(1,n,p)

Or we can directly calculate P(X ≥ 2), meaning F(1)upper=P(X>1)pbinom(1,n,p, lower.tail=FALSE)


Laws of probability

Binomial law

When we have 4 balls we replacement, to how many black balls corresponds the probability p=0.1808:

p=0.2; n=4qbinom(0.1808,n,p, lower.tail=FALSE)

When we have 4 balls we replacement, to how many black balls at least, corresponds the probability p=0.15:

qbinom(0.15,n,p, lower.tail=FALSE)qbinom(0.5,n,p)

p=0.2; n=4; k=0:4plot(k, dbinom(k, n, p), pch=16, cex=2, xlim=range(0, 5), ylim=range(0, 0.5), xlab="Number of black balls k", ylab="Probability f(k)", cex.lab=1.5, cex.axis=1.5, bty="l")abline(h=0.15,lwd=1,col="black")abline(h=0.5,lwd=1,col="red")


Laws of probability

Binomial law

When we have 4 balls we replacement, to how many black balls corresponds the probability p=0.1808:

p=0.2; n=4qbinom(0.1808,n,p, lower.tail=FALSE)

When we have 4 balls we replacement, to how many black balls at least, corresponds the probability p=0.15:

qbinom(0.15,n,p, lower.tail=FALSE)qbinom(0.5,n,p)

p=0.2; n=4; k=0:20plot(k,dbinom(k,n,p), type="l", xlab=« Number of black balls k", ylab="Probability f(k)", cex.lab=1.5, cex.axis=1.5)

p=0.2; n=20lines(k,dbinom(k,n,p),lwd=3)


Laws of probability

Normal law

f(x): Probability densityx: Continuous quantitative variable (): Mean of the variable xσ: Standrad deviation of the variable x

𝑓 (𝑥 )= 1√2𝜋 𝜎2

𝑒− (𝑥−𝜇)2

2𝜎 2

Laws of probability

Normal law

1) The normal is not normal in the way the other laws are abnormal.2) The normal is very common but far to be the only law of probability

The normal distribution is immensely useful because of the central limit theorem, which states that, under mild conditions, the mean of many random variables independently drawn from the same distribution is distributed approximately normally.


𝑓 (𝑥 )= 1√2𝜋 𝜎2

𝑒− (𝑥−𝜇)2

2𝜎 2

Laws of probability

Normal law

Let’s consider the example of flakes size per layer in an archaeological site, following a law N(70,10) and caluclate the density of probabilit of the value 60mm. It is f(60):x=60; mu=70; sigma=101/(sqrt(2*pi)*sigma)*exp(-((x - mu)^2/(2*sigma^2)))


𝑓 (𝑥 )= 1√2𝜋 𝜎2

𝑒− (𝑥−𝜇)2

2𝜎 2

Laws of probability

Normal law

Let’s consider the example of flakes size per layer in an archaeological site, following a law N(70,10) and caluclate the density of probability of the value 60cm. It is f(60):x=60; mu=70; sigma=101/(sqrt(2*pi)*sigma)*exp(-((x - mu)^2/(2*sigma^2)))

With R:dnorm(x,mu,sigma)


𝑓 (𝑥 )= 1√2𝜋 𝜎2

𝑒− (𝑥−𝜇)2

2𝜎 2

Laws of probability

Normal law

Let’s calculate the probability to select a 60mm long flake at least. The aim is to determine the function of repartition F(60)upper=P(x ≥ 60):x=60; mu=70; sigma=10pnorm(x,mu,sigma, lower.tail=FALSE)

And the probability to select a 60mm long flake at the most. The aim is to determine the function of repartition F(60)lower=P(x ≤ 60)pnorm(x,mu,sigma)


𝑓 (𝑥 )= 1√2𝜋 𝜎2

𝑒− (𝑥−𝜇)2

2𝜎 2

Laws of probability

Normal law

Let’s calculate the threshold of sizes corresponding to 50% of the smallest flakes. The aim is to determine the value of size xi that remains 50% of the area on the left (so lower) of the curve, so F(xi)left=P(x ≤ xi)=0.5:F=0.5; mu=70; sigma=10qnorm(F,mu,sigma, lower.tail=TRUE)

Let’s do the same with a threshold of 2.5% of the longest flakes, so F(xi)right=P(x > xi)=0.025: F=0.025; mu=70; sigma=10qnorm(F,mu,sigma, lower.tail=FALSE)


𝑓 (𝑥 )= 1√2𝜋 𝜎2

𝑒− (𝑥−𝜇)2

2𝜎 2

Laws of probability

Normal law

Let’s calculate the probability to select a flake between μ-1.96σ and μ+1.96σ . The aim is to determine the function of repartition P(μ-1.96σ < x ≤ μ+1.96σ ) =P(x > μ-1.96σ )-P(x > μ+1.96σ ) :mu=70; sigma=10pnorm(mu-1.96*sigma,mu,sigma, lower.tail=FALSE)- pnorm(mu+1.96*sigma,mu,sigma, lower.tail=FALSE)

Or the other way around, P(μ-1.96σ < x ≤ μ+1.96σ ) =P(x ≤ μ+1.96σ )-P(x ≤ μ-1.96σ ) :mu=70; sigma=10pnorm(mu+1.96*sigma,mu,sigma)- pnorm(mu-1.96*sigma,mu,sigma)


𝑓 (𝑥 )= 1√2𝜋 𝜎2

𝑒− (𝑥−𝜇)2

2𝜎 2

Laws of probability

Normal law

Remember, the 2.5% of the longest flakes mesure 89.6mm, so:mu+1.96*sigma

Indeed, it corresponds to the value μ+1.96σ


𝑓 (𝑥 )= 1√2𝜋 𝜎2

𝑒− (𝑥−𝜇)2

2𝜎 2

Laws of probability

Chi2 law

This law is used in many statistic tests.

f(x): Probability densityx: Continuous variable (): Number of df (): Gamma function

𝑓 (𝑥 )= 1

2𝜈2 Γ (𝜈 /2)

×𝑥𝜈2 −1×𝑒

−𝑥2

Laws of probability

Chi2 law

Degree of freedom df:

If Then

There are two Random variable but only one Degree of freedom


𝑓 (𝑥 )= 1

2𝜈2 Γ (𝜈 /2)

×𝑥𝜈2 −1×𝑒

−𝑥2

Laws of probability

Chi2 law


𝑓 (𝑥 )= 1

2𝜈2 Γ (𝜈 /2)

×𝑥𝜈2 −1×𝑒

−𝑥2

Let’s calculate the probability density for the law 4 for the value x=2:x=2;df=41/(2^(df/2)*gamma(df/2)) *x^(df/2-1)*exp(-x/2)

In R:x=2; df=4dchisq(x,df)

Laws of probability

Chi2 law


𝑓 (𝑥 )= 1

2𝜈2 Γ (𝜈 /2)

×𝑥𝜈2 −1×𝑒

−𝑥2

Let’s calculate the probability that x≤2 on the law 4. The aim is to determine the function of repartition F(2)upper =P(x ≤2):x=2; df=4pchisq(x,df)

Let’s calculate the probability x>2 . so F(2)lower =P(x >2):x=2; df=4pchisq(x,df, lower.tail=FALSE)

Let’s calculate the threshold corresponding to 2.5% of the highest values. So determining so F(xi)lower =P(x >xi)=0.025:F=0.025; df=4qchisq(F,df, lower.tail=FALSE)

Laws of probability

Chi2 law


𝑓 (𝑥 )= 1

2𝜈2 Γ (𝜈 /2)

×𝑥𝜈2 −1×𝑒

−𝑥2

Shape of the density curve:

Light linex=seq(-0.1,10,0.01);df=1plot(x, dchisq(x,df),type="l", ylim=range(0,1), xlab="Continuous variable x", ylab="Density f(x)", cex.lab=1.5, cex.axis=1.5)Thick linedf=2lines(x, dchisq(x,df),lwd=3)Dotted linedf=4lines(x, dchisq(x,df), lty="39",lwd=2)Long dotted linedf=8lines(x, dchisq(x,df), lty="83",lwd=2)

Laws of probability

These laws are probability laws of statistic. They are different from the probability law of a measured variable.

The measured variable fluctuates according to the individuals while the statistic variable fluctuates accoring to the sample.

For example, for a Student t test, if you want to compare means, your sample must follow a normal law, while the statistic of the test follows a Student law.

What is a law of probability?


The aim of a statistical test is to conclude on a population while the observations are only available for a sample of this population.

The value of the test varies based on the samples according to a probability distribution.

The test forsees this fluctuation of the value of the test in only one case: the one efined by the H0 hypothesis.

The aim of the test is then to determine if the value obtained is probable according to the statistic distribution under H0. And for this it gives a p-value.

When the probability distribution under H0 is known, the p-value is nothing else than a function of repartition of the law defined by the sample.



Let’s consider the example of flakes size in one layer of an archaeological site.Question: Is the size of the obsidian flakes longer than the one of the andesite flakes that is around 165mm.

A random sample of 50 obsidian flakes is selected that shows a mean of 165mm and a standard deviation of 7mm.

The aim of the test is to conclude on the population of obsidian flakes for this archaeological site although we sampled only a small part of this site.

H0: The mean obsidian flake size is equal to andesite ones.H1: The mean obsidian flake size is strictly longer.

The formula is then M=m=168mmWith M: calculated statistics

m: mean size in the sample

According to the test, M is a random variable which under H0 follows a normal law of N(165,7/√50 = 0.99)

The function of repartition beyond M=168mm on the normal law N(165,0.99) is:pnorm(168,165,0.99, lower.tail=FALSE)



The probability that M is above 168mm on N(165,0.99) is p=0.0012 what is low.It means that if the random sample of 50 obsidian flakes comes from a population of = 165mm, the probability that m ≥ 168mm by sampling is p.As we are gonna to see, we can then reject H0 with this p-value.In conclusion, the sample does not permit to explain to distance from m to 165mm: m is significatively longer than = 165mm. It is then probable that is this archaeological site, the >


H0 H1

H0 Right decision1-α

Type I errorα

H1 Type II errorβ

Right decision1-β

Usually, we choose α= 5%, 1% or 0.1%. But those values do not have any special meaning.

The 4 possible situations for a test of hypotheses:

Reality

Decision




H0 H1

H0 Right decision1-α

Type I errorα

H1 Type II errorβ

Right decision1-β


-> Do we have to minimize the risk α or the risk β ?α means that we reject H0 while H0 is true.β means that we reject H1 while H1 is true.

Reality

Decision




H0: Innocent H1: Guilty

H0: Innocent Right decision1-α

Type I errorα

H1: Guilty Type II errorβ

Right decision1-β



Reality

Decision






Type I errorΑ = Weak

H1: Guilty Type II errorβ

Right decision1-β



Reality

Decision






Type I errorα

H1: Guilty Type II errorΒ = Weak

Right decision1-β



Reality

Decision






Type I errorΑ =Middle

H1: Guilty Type II errorΒ =Middle

Right decision1-β



Reality

Decision






Type I errorΑ =Middle

H1: Guilty Type II errorΒ =Middle

Right decision1-β


-> Do we have to minimize the risk α or the risk β ?α means that we reject H0 while H0 is true.β means that we reject H1 while H1 is true.Minimizing the error α, increases the error β and then reduces the power of the test.

Reality

Decision





In the case of a bilateral test, the technic consists in looking on upper and lower the cacluclated X, to pick up the smallest value and to multiply by 2:

In the case of the chi2, the p-value is the probability, if H0 is true, to obtain a value of the statistic higher or equal to the observed value.


So what is the interest of the p-value?-> to interpret the test and to conclude:

The risk α is the p-value. Or the risk α is the threshold to reject H0.

If p-value ≤ α, then I accept H1 (reject H0)If p-value > α, then I accept H0

In the first case, the p-value of the test is significative, in the second case it is not.

By default, α is set to 5%


Two hypotheses to define:H0 is the nul hypothesis: the one we want to testH1 is the alternative hypothesis (in the case we reject H0, H1 is considered as true)

The test depends of its statistic and its law of probability.

There are two types of error which are opposite one to the other one:Type I error: conclude that H0 is false while in reality it is trueType II error: conclude that H0 is true while in reality it is false

We always know the risk α as we define it.If we know H1 with precision, then we can calculate β and the power of the test which is equal to 1- β.

The P-value permits us to take our decision:If it is inferior or equal to α, then I reject H0 and accept H1If it is superior to α, then I accept H0

Summary and conclusion

Theory of tests

Documents

Transcript of Theory of tests