PsyLab Advanced Course: Introduction to simulation

PsyLab Advanced Course: Introduction tosimulation

Karl Bang ChristensenMike Horton

Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018

Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation

http://biostat.ku.dk/~kach/PsyLab2018


Overview

Two examples

Item fit statistics

INFIT and OUTFIT are often reported - but evaluated using ’rulesof thumb’ (e.g. values larger than 1.3 indicating misfit) - this hasbeen critisized

Can we trust, e.g., the item χ2 in RUMM ?

Impact of LD

does local dependence impact the planned analysis

Solution

Simulation

Use simulated data sets to learn about the critical values (what toexpect if Rasch model fits)



Item fit statistics

Let Xiv denote the response to item i by subject v with the valueθv on the latent trait. Standard evaluation of individual item fituses

Ziv =Xiv − E (Xiv |θv = θ̂v )√

V (Xiv |θv = θ̂v )

to compute fit statistics like

OUTFITi =1

N

N∑v=1

Z 2iv

with no established null distribution.

History lesson

early Rasch literature incorrectly claims χ2 distribution1

1Wright, Stone (1979). Mesa Press, Chicago, USA.Karl Bang Christensen Mike Horton Svend Kreinerhttp://biostat.ku.dk/~kach/PsyLab2018PsyLab Advanced Course: Introduction to simulation


Item fit statistics

Re-writingN∑

v=1

Z 2iv =

k∑r=0

∑v :Rv=r

Z 2i |r︸︷︷︸

iid

’known’ null distribution2

History lesson

This fact has been forgotten/abandoned3

2Christensen, Kreiner (2013). Rasch Models in Health (chap. 5, pp.83-104), Wiley https://doi.org/10.1002/9781118574454.ch5

3Wright, Panchapakesan (1969). Educational and PsychologicalMeasurement, 29, 23-48. http://doi.org/10.1177/001316446902900102


https://doi.org/10.1002/9781118574454.ch5

http://doi.org/10.1177/001316446902900102


Item fit statistics

It has been suggested that the Wilson-Hilferty cube-roottransformation

ti = (OUTFIT1/3i − 1)

3

V (OUTFITi )+

V (OUTFITi )

3

has an approximate t distribution

History lesson

It seems unlikely that anyone has looked at

Wilson, Hilferty (1931). The Distribution of Chi-Square. InProceedings of the National Academy of Sciences of theUnited States of America (Vol. 17, pp. 684-688).https://doi.org/10.1073/pnas.17.12.684


https://doi.org/10.1073/pnas.17.12.684


Item fit statistics

It has been suggested that

FitResidi =f (log(N · OUTFITi )− log(f ))√

V (N · OUTFITi )

with f = (Nk − N − k + 1)/k , has a symmetrical distribution withmean zero and variance one.



Item fit statistics

RUMM

groups respondents in G ’class intervals’ based on the estimatedperson locations θ̂v . ANOVA item fit statistic or item χ2 fitstatistic

χ2(Xi ) =∑g

(∑v∈Vg

Xvi −∑

v∈VgE (Xvi )

)2∑v∈Vg

V (Xvi )

where Vg denotes the set of respondents in class interval g .

problem

still relies on θ̂1, θ̂2, . . .



AMTS data example - individual item fit

item chisq df p

ADDRESS 1.73 1 0.1889

AGE 2.74 1 0.0976

COUNTBAC 0.98 1 0.3226

DOB 0.60 1 0.4382

FIRSTWW 2.25 1 0.1337

MONARCH 3.39 1 0.0656

MONTH 12.99 1 0.0003

NAME 2.74 1 0.0976

TIME 2.21 1 0.1369

YEAR 6.25 1 0.0124

can we trust the p-values ? How can we address multiple testing ?



AMTS data example - evaluating item fit

The null distribution (what would we expect if the data fit themodel)

11:12 Wednesday, September 19, 2018 111:12 Wednesday, September 19, 2018 1

0 5 10

Item fit statistic

0

10

20

30

40

50

Pe

rce

nt

Theoretical distribution

unlikely that χ2MONTH is above 3.84



AMTS data example - simulation

The steps

1 Simulate data set based on estimated item parameters andestimated person locations

2 This yields a data set where we know the Rasch model fits

3 Estimate item parameters and estimated person locations

4 Compute χ2MONTH

are repeated, say, 1000 times

Summarizing

Now we know what to expect if the Rasch model fits - and we cananswer the question: ’how often would you see large values just bychance ?’



21:31 Monday, January 22, 2018 121:31 Monday, January 22, 2018 1

0 5 10

Item fit statistic

0

10

20

30

40

50

Pe

rce

nt

empiricaltheoretical

Empirical and theoretical distribution

0 5 10

Item fit statistic

0

10

20

30

40

50

Pe

rce

nt

4.5095% percentile8.3%rejection rate

empiricaltheoretical

Empirical and theoretical distribution



Perspectives in using simulation

Appropriate critical value for item χ2

Appropriate critical value for maxi=1,..,10(χ2i )

Tabulate critical values

Implement parametric bootstrap



Appropriate critical value for maxi=1,..,10(χ2i )

11:12 Wednesday, September 19, 2018 111:12 Wednesday, September 19, 2018 1

95th: 10.52 99th: 14.61

0 5 10 15 20

Empirical distribution for maximum value of the chisq item fit statistic

0

5

10

15

20

Pe

rce

nt



PsyLab Advanced Course: Introduction to simulation

Documents

Transcript of PsyLab Advanced Course: Introduction to simulation