1 Sample Size and Power for CFA: A Monte Carlo Approach Myers, N.D., Ahn, S., & Jin, Y....

1

Sample Size and Power for CFA: A Sample Size and Power for CFA: A Monte Carlo ApproachMonte Carlo Approach

Myers, N.D., Ahn, S., & Jin, Y.Myers, N.D., Ahn, S., & Jin, [email protected]@miami.edu

2

I. PurposeI. Purpose Demonstrate Demonstrate how Monte Carlo methods can be usedhow Monte Carlo methods can be used to to

make empirically-based decisions:make empirically-based decisions:1.1. NN for a fixed level of for a fixed level of ππ, and, (i.e., , and, (i.e., N?N?))2.2. ππ for a fixed for a fixed NN ( (ππ??))under a under a CFACFA model with model-data conditions model with model-data conditions

commonly encountered in sport & exercisecommonly encountered in sport & exercise

Generality of Generality of MC methodsMC methods has allowed for two types of has allowed for two types of applications in statisticsapplications in statistics11

1.1. MC studies of MC studies of statistical methods statistical methods 2.2. MC methods in MC methods in data analysis data analysis

3

I. PurposeI. Purpose Why is the additional complexity worthwhile?Why is the additional complexity worthwhile?

Rules of thumb Rules of thumb are known to be of limited utilityare known to be of limited utility22 NN ≥ 200 ≥ 200 N/p ≥ N/p ≥ 1010 N/q ≥ N/q ≥ 55 construct reliability and adequate construct reliability and adequate NN

Adequate Adequate NN for CFA depends on many factors that typically vary for CFA depends on many factors that typically vary across any two studies across any two studies that use real data and inexact theoretical that use real data and inexact theoretical modelsmodels

These factors can be These factors can be directly modeled directly modeled using MC methodsusing MC methods

4

I. PurposeI. Purpose Informal review of RQES (and others)Informal review of RQES (and others)

Some observed trends in practice: Some observed trends in practice: single population modelssingle population models many many pp and multiple and multiple ordinal dataordinal data a sig a sig 22 value and value and ≤ .05 ≤ .05 variable variable andand(parameters of interest, (parameters of interest, ii)) few few vc > 1 vc > 1 and/or off-diagonal and/or off-diagonal ≠ 0≠ 0 A priori plan for A priori plan for NN, for a desired level of , for a desired level of ? ? An estimate of An estimate of , for a fixed , for a fixed NN?? ....must have ....must have “knowns”“knowns”

Model the messiness of practice… Model the messiness of practice… N? N?

II. A Conceptual DemonstrationII. A Conceptual Demonstration

Coaching EfficacyCoaching Efficacy: coach’s belief in his/her ability to influence the : coach’s belief in his/her ability to influence the learning and performance of his/her athleteslearning and performance of his/her athletes44

Psych: Psych: Bandura Bandura (1997)(1997)55; Educ: ; Educ: Denham & Michael Denham & Michael (1981)(1981)66

Coaching Effectiveness: Coaching Effectiveness: Horn Horn (2002)(2002)77

5


Coaching Efficacy Scale II-HSTCoaching Efficacy Scale II-HST88

Motivation(M)

m1 m2 m3 m4

GameStrategy (GS)

g1 g2 g3 g4

Technique(T)

t1 t2 t3 t4

CharacterBuilding (CB)

c1 c2 c3

PhysicalConditioning (PC)

p2 p3p1

m1* m2* m3* m4* g1* g2* g3* g4* t1* t2* t3* t4* c1* c2* c3* p1* p2* p3*

6

7

III. Methods/ResultsIII. Methods/Results

A.A. Design Stage: Conceptualize experiment - Design Stage: Conceptualize experiment - “Intro” “Intro” 99

1.1. Research Question(s)Research Question(s) SmallestSmallest NN necessary to achieve at least .80 necessary to achieve at least .80 for each for each i

Given a particular Given a particular N N , what is the , what is the estimate for each estimate for each i i

2. 2. Derive Derive theoretical model & population model

3. 3. Design ExperimentDesign Experiment Model mis-specification (Model mis-specification (MODEL POPULATION MODEL POPULATION versus versus MODELMODEL)) N N = 800, 200, 300, 400, 500; = 800, 200, 300, 400, 500; NNRR = 10,000= 10,000 Ordinal dataOrdinal data

4.4. Choose values of population parametersChoose values of population parameters range of parameter estimates range of parameter estimates

5.5. Choosing Software Choosing Software Mplus 5.2 (code is available by request)Mplus 5.2 (code is available by request)10,1110,11

8


B.B. Generating Data Stage: Performing experimentGenerating Data Stage: Performing experiment

““Data collection”Data collection”6.6. Executing the simulations Executing the simulations Same code varied Same code varied NN

7.7. File storageFile storage Saved where input file is located (stuff happens…)Saved where input file is located (stuff happens…)

8. 8. Trouble shootingTrouble shooting Data Data generation generation problem (e.g., no obs in a particular category)problem (e.g., no obs in a particular category) Data Data analyticanalytic problem (e.g., non-convergent, improper solution) problem (e.g., non-convergent, improper solution)

9


C.C. Interpreting Results Stage: Findings from experimentInterpreting Results Stage: Findings from experiment““Results/Discussion”Results/Discussion”

9.9. Summarizing ResultsSummarizing Results

Theoretical Model fit to data generated from Population ModelTheoretical Model fit to data generated from Population Model Q1 (N?): Q1 (N?): A relatively small A relatively small N N (200) (200) provides ample provides ample π (> 98%)π (> 98%) Q2 (Q2 (?): ?): common common NN (300, 400, 500) is similar to Q1 (300, 400, 500) is similar to Q1 (π > 99%)(π > 99%) Problematic Problematic bias values (> |10%|) bias values (> |10%|) and and coverage values (< 91%)coverage values (< 91%), for a , for a

few parameters few parameters (~20%) (~20%) are observed for all sample sizesare observed for all sample sizes

Population Model fit to data generated from Population ModelPopulation Model fit to data generated from Population ModelTo what degree might the “relatively minor” model mis-specifications be To what degree might the “relatively minor” model mis-specifications be

responsible for these problems? responsible for these problems? Q1 (N?): Q1 (N?): and and Q2 (Q2 (?) ?) similar to findings from Theoretical Model except similar to findings from Theoretical Model except

no problems with no problems with bias values bias values and and coverage valuescoverage values

10

IV. ImportanceIV. Importance

1.1. MC methods MC methods have long been used to advance statistical theory. have long been used to advance statistical theory. There have been several recent calls to use Monte Carlo methods There have been several recent calls to use Monte Carlo methods as a tool to improve applications of quantitative methods in as a tool to improve applications of quantitative methods in substantive research.substantive research.

2.2. Mplus code Mplus code is available and can easily be alteredis available and can easily be altered

3.3. CES II – HSTCES II – HST: : N N ≥ 200 for the theoretical model ≥ 200 for the theoretical model N N ≥ 300 for the population model≥ 300 for the population model

4.4. a level of misfit a level of misfit that may be regarded as trivial in practice may that may be regarded as trivial in practice may have troubling effects on parameters of conceptual interesthave troubling effects on parameters of conceptual interest1.1. encourage sustained efforts toward generating closer approximations of encourage sustained efforts toward generating closer approximations of

population modelspopulation models

Thank you. Thank you. Ms is in press at RQES.Ms is in press at RQES.

11

1.1. Gentle, J. E. (2003). Gentle, J. E. (2003). Random number generation and Monte Carlo methods Random number generation and Monte Carlo methods (2(2ndnd ed.) ed.).. New New York: Springer.York: Springer.

2.2. Marsh, H. W., Hau, K.-T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The Marsh, H. W., Hau, K.-T., Balla, J. R., & Grayson, D. (1998). Is more ever too much? The number of indicators per factor in confirmatory factor analysis. number of indicators per factor in confirmatory factor analysis. Multivariate Behavioral Multivariate Behavioral Research, 33,Research, 33, 181-220. 181-220.

3.3. MacCallum, R. (2003). Working with imperfect models. MacCallum, R. (2003). Working with imperfect models. Multivariate Behavioral Research, 38,Multivariate Behavioral Research, 38, 113-139.113-139.

4.4. Feltz, D.L., Chase, M.A., Moritz, S.E., & Sullivan, P.J. (1999). A conceptual model of coaching Feltz, D.L., Chase, M.A., Moritz, S.E., & Sullivan, P.J. (1999). A conceptual model of coaching efficacy:efficacy:

Preliminary investigation and instrument development. Preliminary investigation and instrument development. Journal of Educational Journal of Educational Psychology, 91,Psychology, 91, 765-776. 765-776.

5.5. Bandura, A. (1997). Bandura, A. (1997). Self-efficacy: The exercise of control. Self-efficacy: The exercise of control. New York: Freeman.New York: Freeman.6.6. Denham, C.H., & Michael, J.J. (1981). Teacher sense of efficacy: A definition and a model for Denham, C.H., & Michael, J.J. (1981). Teacher sense of efficacy: A definition and a model for

further research. further research. Educational Research Quarterly, 5,Educational Research Quarterly, 5, 39-63. 39-63. 7.7. Horn, T.S. (2002). Coaching effectiveness in the sport domain. In T.S. Horn (Ed.), Horn, T.S. (2002). Coaching effectiveness in the sport domain. In T.S. Horn (Ed.), Advances in Advances in

sport psychologysport psychology (2nd ed., pp. 309-354). Champaign, IL: Human Kinetics. (2nd ed., pp. 309-354). Champaign, IL: Human Kinetics.8.8. Myers, N. D., Feltz, D. L., Chase, M. A., Reckase, M. D., & Hancock, G. R. (2008). The Myers, N. D., Feltz, D. L., Chase, M. A., Reckase, M. D., & Hancock, G. R. (2008). The

Coaching Efficacy Scale II - High School Teams. Coaching Efficacy Scale II - High School Teams. Educational and Psychological Educational and Psychological Measurement, 68,Measurement, 68, 1059-1076. 1059-1076.

9. 9. Paxton, P., Curran, P. J., Bollen, K. A., Kirby, J., & Chen, F. (2001). Monte Carlo experiments: Paxton, P., Curran, P. J., Bollen, K. A., Kirby, J., & Chen, F. (2001). Monte Carlo experiments: Design and implementation. Design and implementation. Structural Equation Modeling, 8Structural Equation Modeling, 8 , 287-312., 287-312.

10.10. Muthén, L. K., & Muthén, B. O. (2002). How to use a Monte Carlo study to decide on sample Muthén, L. K., & Muthén, B. O. (2002). How to use a Monte Carlo study to decide on sample size and determine power. size and determine power. Structural Equation Modeling, 9Structural Equation Modeling, 9 , 599-620., 599-620.

11.11. Muthén, L.K. and Muthén, B.O. (1998-2010). Muthén, L.K. and Muthén, B.O. (1998-2010). Mplus User’s GuideMplus User’s Guide (6th ed.). Los Angeles, CA: (6th ed.). Los Angeles, CA:

Muthén & Muthén.Muthén & Muthén.

Motivation(M)

m1 m2 m3 m4

GameStrategy (GS)

g1 g2 g3 g4

Technique(T)

t1 t2 t3 t4


c1 c2 c3


p2 p3p1


12

2 (78) 115, .004; RMSEA=.024; CFI=.992, TLI=.997p 2 (76) 91, .116; RMSEA=.016; CFI=.997, TLI=.999p

.26

.14


Coaching Efficacy Scale II-HSTCoaching Efficacy Scale II-HST 88

Motivation(M)

m1 m2 m3 m4

GameStrategy (GS)

g1 g2 g3 g4

Technique(T)

t1 t2 t3 t4


c1 c2 c3


p2 p3p1


16

Motivation(M)

m1 m2 m3 m4

GameStrategy (GS)

g1 g2 g3 g4

Technique(T)

t1 t2 t3 t4


c1 c2 c3


p2 p3p1


17

.26

.14

1 Sample Size and Power for CFA: A Monte Carlo Approach Myers, N.D., Ahn, S., & Jin, Y....

Documents

Transcript of 1 Sample Size and Power for CFA: A Monte Carlo Approach Myers, N.D., Ahn, S., & Jin, Y....