Factor Analysis - University of...

24
Factor Analysis The Measurement Model

Transcript of Factor Analysis - University of...

Page 1: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

FactorAnalysis

TheMeasurementModel

Page 2: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

FactorAnalysis:TheMeasurementModel

D1 D8D7D6

D5D4D3D2

F1 F2

Di = !Fi + ei

Page 3: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

Examplewith2factorsand8observedvariables

!

""""""""""#

Di,1

Di,2

Di,3

Di,4

Di,5

Di,6

Di,7

Di,8

$

%%%%%%%%%%&

=

!

""""""""""#

!11 !12

!21 !22

!31 !32

!41 !42

!51 !52

!61 !62

!71 !27

!81 !82

$

%%%%%%%%%%&

'Fi,1

Fi,2

(+

!

""""""""""#

ei,1

ei,2

ei,3

ei,4

ei,5

ei,6

ei,7

ei,8

$

%%%%%%%%%%&

Di = !Fi + ei

Di,1 = !11Fi,1 + !12Fi,2 + ei,1

Di,2 = !21Fi,1 + !22Fi,2 + ei,2 etc.

Thelambdavaluesarecalledfactorloadings.

Page 4: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

Terminology

•  Thelambdavaluesarecalledfactorloadings.•  F1andF2aresometimescalledcommonfactors,becausetheyinfluencealltheobservedvariables.

•  e1,…,e8sometimescalleduniquefactors,becauseeachoneinfluencesonlyasingleobservedvariable.

Di,1 = !11Fi,1 + !12Fi,2 + ei,1

Di,2 = !21Fi,1 + !22Fi,2 + ei,2 etc.

Page 5: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

FactorAnalysiscanbe

•  Exploratory:Thegoalistodescribeandsummarizethedatabyexplainingalargenumberofobservedvariablesintermsofasmallernumberoflatentvariables(factors).Thefactorsarethereasontheobservablevariableshavethecorrelationstheydo.

•  Confirmatory:Statisticalestimationandtestingasusual

Page 6: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

PartOne:Unconstrained(Exploratory)FactorAnalysis

Page 7: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

F and e independent multivariate normal

V (D) = ! = "#"! + $

D = !F + e

Main interest is in the number of factors and the factor loadings !.

V (F) = !V (e) = " (usually diagonal)

Page 8: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

ARe‐parameterization

! = "#"! + $

Write ! = SS! (square root matrix), so

!"!! = !SS!!!

= (!S)I(S!!!)= (!S)I(!S)!

= !2I!!2

Page 9: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

Parametersarenotidentifiable

•  Phicouldbeanysymmetricpositivedefinitematrixandthiswouldwork.

•  Infinitelymany(Lambda,Phi)pairsgivethesameSigma,andhencethesamedistributionofthedata.

•  Phiiscompletelyarbitrary(soisLambda)•  Thisshowsthattheparametersofthegeneralmeasurementmodelarenotidentifiablewithoutsomerestrictionsonthepossiblevaluesoftheparametermatrices.

•  Noticethatthegeneralunrestrictedmodelcouldbeveryclosetothetruth.Buttheparameterscannotbeestimatedsuccessfully,period.

! = "#"! + $ !"!! = !2I!!2

Page 10: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

Restrictthemodel

•  SetPhi=theidentity,soV(F)=I•  Justifythisonthegroundsofsimplicity.

•  Saythefactorsare“orthogonal”(atrightangles,uncorrelated).

!"!! = !2I!!2

Page 11: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

Standardizetheobservedvariables

•  Forj=1,…,kandindependentlyfori=1,…,n

• 

•  Assumeeachobservedvariablehasvarianceoneaswellasmeanzero.

•  Sigmaisnowacorrelationmatrix.

Zij =Dij !Dj

sj

Page 12: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

RevisedExploratoryFactorAnalysisModel

V (F) = IV (e) = ! (usually diagonal)

F and e independent multivariate normal

V (D) = ! = ""! + #

Z = !F + e

Page 13: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

Meaningofthefactorloadings

•  λijisthecorrelationbetweenvariableiandfactorj.

•  Squareofλijisthereliabilityofvariableiasameasureoffactorj.

Corr(X6, F2) = Cov(X6, F2) = E(X6F2)= E ((!61F1 + !62F2)F2)= !61E(F1F2) + !62E(F 2

2 )= !61E(F1)E(F2) + !62V ar(F2)= !62

Page 14: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

Communality

•  istheproportionofvarianceinvariableithatcomesfromthecommonfactors.

•  Itiscalledthecommunalityofvariablei.

•  Thecommunalitycannotexceedone.

•  Peculiar?

V ar(Xi) =p!

j=1

!ijFj + ei

=p!

j=1

!2ijV ar(Fj) + V ar(ei)

=p!

j=1

!2ij + "i

p!

j=1

!2ij

V ar(!i) = 1!!p

j=1 "2ij

Page 15: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

Ifwecouldestimatethefactorloadings

•  Wecouldestimatethecorrelationofeachobservablevariablewitheachfactor.

•  Wecouldeasilyestimatereliabilities.•  Wecouldestimatehowmuchofthevarianceineachobservablevariablecomesfromeachfactor.

•  Thiscouldrevealwhattheunderlyingfactorsare,andwhattheymean.

•  Numberofcommonfactorscanbeveryimportanttoo.

Page 16: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

Examples•  Amajorstudyofhowpeopledescribeobjects(using7‐pointscalesfromUglytoBeautiful,StrongtoWeak,FasttoSlowetc.revealed3factorsofconnotativemeaning:–  Evaluation–  Potency– Activity

•  Factoranalysisofalargecollectionofpersonalityscalesrevealed2majorfactors:– Neuroticism–  Extraversion

•  Yetanotherstudyledto16personalityfactors,thebasisofthewidelyused16pftest.

Page 17: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

RotationMatrices

• Have a co-ordinate system in terms of!i ,!j orthonormal vectors

• Roatate the axies through an angle !.

!

i

i!

j! j

i! = i cos ! + j sin !

j! = !i sin ! + j cos !

Page 18: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

i! = (cos !)i + (sin !)jj! = (! sin !)i + (cos !)j

!i!

j!

"=

!cos ! sin !

! sin ! cos !

" !ij

"= R

!ij

"

RR! =!

cos ! sin !! sin ! cos !

" !cos ! ! sin !sin ! cos !

"

=!

cos2 ! + sin2 ! ! cos ! sin ! + sin ! cos !! sin ! cos ! + cos ! sin ! sin2 ! + cos2 !

"

=!

1 00 1

"= I

Thetransposerotatedtheaxiesbackthroughanangleofminustheta.

Page 19: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

InGeneral

•  ApxpmatrixRsatisfyingR‐inverse=R‐transposeiscalledanorthogonalmatrix.

•  Geometrically,pre‐multiplicationbyanorthogonalmatrixcorrespondstoarotationinp‐dimensionalspace.

•  IfyouthinkofasetoffactorsFasasetofaxies(underlyingdimensions),thenRFisarotationofthefactors.

•  Callitanorthoganalrotation,becausethefactorsremainuncorrelated(atrightangles).

Page 20: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

AnotherSourceofnon‐identifiability

! = ""! + #= "RR!"! + #= ("R)(R!"!) + #= ("R)("R)! + #= "2"!

2 + #

InfinitelymanyrotationmatricesproducethesameSigma.

Page 21: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

NewModel

Z = !2F + e= (!R)F + e= !(RF) + e= !F! + e

F! is a set of rotated factors.

Page 22: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

ASolution

•  Placesomerestrictionsonthefactorloadings,sothattheonlyrotationmatrixthatpreservestherestrictionsistheidentitymatrix.Forexample,λij=0forj>i

•  Thereareothersetsofrestrictionsthatwork.•  Generally,theyresultinasetoffactorloadingsthatareimpossibletointerpret.Don’tworryaboutit.

•  Estimatetheloadingsbymaximumlikelihood.Othermethodsarepossiblebutusedmuchlessthaninthepast.

•  All(orthoganal)rotationsresultinthesamevalueofthelikelihoodfunction(themaximumisnotunique).

•  Rotatethefactors(thatis,multiplytheloadingsbyarotationmatrix)soastoachieveasimplepatternthatiseasytointerpret.

Page 23: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

Rotatethefactorsolution

•  Rotatethefactorstoachieveasimplepatternthatiseasytointerpret.

•  Therearevariouscriteria.Theyarealliterative,takinganumberofstepstoapproachsomecriterion.

•  Themostpopularrotationmethodisvarimaxrotation.•  Varimaxrotationtriestomaximizethe(squared)loadingofeachobservablevariablewithjustoneunderlyingfactor.

•  Sotypicallyeachvariablehasabigloadingon(correlationwith)oneofthefactors,andsmallloadingsontherest.

•  Lookattheloadingsanddecidewhatthefactorsmean(namethefactors).

Page 24: Factor Analysis - University of Torontoutstat.toronto.edu/.../431s11/handouts/FactorAnalysis1.pdf · 2011-03-08 · factor analysis, it’s the observed variables that arise from

AWarning

•  Whenanon‐statisticianclaimstohavedonea“factoranalysis,”askwhatkind.

•  Usuallyitwasaprincipalcomponentsanalysis.•  Principalcomponentsarelinearcombinationsoftheobservedvariables.Theycomefromtheobservedvariablesbydirectcalculation.

•  Intruefactoranalysis,it’stheobservedvariablesthatarisefromthefactors.

•  Soprincipalcomponentsanalysisiskindoflikebackwardsfactoranalysis,thoughthespiritissimilar.

•  Mostfactoranalysis(SAS,SPSS,etc.)doesprincipalcomponentsanalysisbydefault.