Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Click here to load reader
Transcript of Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Likelihood Ratio Procedure in GFLDA (1)
Hung-Shin Lee3/23/2009
• Log Likelihood:
Likelihood Ratio Procedure in GFLDA (1)
−−=
C
kkN Ndp 1
1
)2log(2
}){},{,(log Σμx π ),( dNg
⎧
( )∑=
−− ++−−C
kkkkkkk
Tkkkn
1
11 ||log)trace()()(21 ΣΣΣμμΣμμ
• Hypotheses:⎩⎨⎧
edunrestrict are s':equal are s':
edunrestrict:1
0
k
kk H
Hμμ
Σ
1Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Likelihood Ratio Procedure in GFLDA (2)
• Likelihood :
Likelihood Ratio Procedure in GFLDA (2)
}){}{(max
}){,,(max
}){}{(max
}){},{,(max 1100
N
kN
HN
kkN
H
p
p
p
pLR
Σμx
Σμx
Σμx
Σμx
′′=
′′=
• In logarithmic domain,
}){},{,(max}){},{,(max 1111
kkHkkHpp ΣμxΣμx
In logarithmic domain,
}){},{,(logmax}){,,(logmaxlog 1110
iiN
HiN
HppLR ΣμxΣμx ′′−=
2Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH0 (1)
• Estimate μk in H0 ,
Hypothesis H0 (1)
∂∂=
∂∂ k
N
k
kkN pp 11 }){,,(log}){},{,(log
μΣμx
μΣμx
∑∑
−=
−
=−=∂
⎟⎠
⎞⎜⎝
⎛ −−−∂=
C
kkk
C
kkk
Tkk
nn
11
1
0)()()(
21
μμΣμμΣμμ
∑∑∑∑
∑
−−
−−−
=
⎟⎞
⎜⎛≈⇒=⇒
∂C
kkk
C
kk
C
kk
C
kkk
kkkk
nnnn 11
111
1
ˆ
)(
μΣΣμμΣμΣ
μμμ
∑∑∑∑====
⎟⎠
⎜⎝
⇒⇒k
kkkk
kkk
kkk
kkk nnnn1111
μμμμ
weighted mean
3
g
Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH0 (2)
• Estimate Σk in H0 ,
Hypothesis H0 (2)
k
kN
k
kkN pp
ΣΣμx
ΣΣμx
∂∂=
∂∂ }){,ˆ,(log}){},{,(log 11
( )C
kkkkkk
Tkkn ΣΣΣμμΣμμ
∂
⎟⎠
⎞⎜⎝
⎛ ++−−−∂=
−−∑ ||log)trace()ˆ()ˆ(21
1
11
( )kkkkkT
kkkk
k
n ΣΣΣΣΣμμμμΣ
Σ
=+−−−−−=
∂
−−−−− 0)ˆ)(ˆ(21 11111
kkkT
kkk ΣBΣμμμμΣ +=+−−=⇒ )ˆ)(ˆ(ˆ2
4Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Appendix A - Derivatives of MatricesAppendix A Derivatives of Matrices
( )11
2 )()(trace −∂ ASAASA TTsymmetric) :(A ( )1
211
22
12
))(()(2
)()(trace
−−−
=∂
∂
ASAASAASAASA
ASAASA
TTT2
y )(
=∂
∂ AxxxAx
121 )(2 −+ ASAAS T
1
1||log
−
−
∂
=∂
∂
A
AAA
T
111
111
)trace( −
−−
∂
−=∂
∂
XA
AxxAAxAx
T
TT
11)trace( −−−=∂
∂ AXAA
XA T
5Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH0 (3)
• The ML value of the data is
Hypothesis H0 (3)
( )∑ −− ++
=C
T
kN
HN
H
dN
pp
11
11*
|ˆ|log)ˆtrace()ˆ()ˆ(1)(
})ˆ{,ˆ,(log)(log00
ΣΣΣμμΣμμ
Σμxx
( )∑−
=
⎟⎞
⎜⎛ +−+−
++−−−=
Ckkk
Tk
kkkkkk
TkkndNg
1
1
11
)ˆ()()ˆ(
||log)trace()()(2
),(
μμΣBμμ
ΣΣΣμμΣμμ
∑=
−
⎟⎟⎟⎟
⎠⎜⎜⎜⎜
⎝ +++−=
C
kkk
kkkkndNg1
1
||log))trace((
21),(
ΣBΣΣB
∑=
+−−=
⎠⎝C
kkkk
kk
nNddNg1
||log21
2),( ΣB
6
k 1
Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Appendix BAppendix B
( )))trace(()ˆ()()ˆ(1 11nC
T +++∑ −− ΣΣBμμΣBμμ( )
( ))()ˆ)(ˆ()(trace1
))trace(()()()(2
11
1
n
n
C
kkkT
kkkkk
kkkkkkkkk
++−−+=
++−+−
∑
∑−−
=
ΣΣBμμμμΣB
ΣΣBμμΣBμμ
( )
( )))ˆ)(ˆ(()(trace21
)())(()(trace2
1
1
n
n
C
kT
kkkkk
kkkkkkkkk
+−−+=
+++
∑
∑−
=
ΣμμμμΣB
ΣΣBμμμμΣB
( )
( ))()(trace21
)))((()(2
1
1
nC
kkkkk
kkkkkkk
++= ∑
∑−
=
ΣBΣB
μμμμ
( )2
trace212
1
1
NdnC
kddk
k
== ∑ ×
=
I
7
22 1k=
Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH1 (1)
• Estimate μk in H1 ,
Hypothesis H1 (1)
C
kkkk
Tkkk
kkN n
pμμΣμμ
Σμx⎟⎠
⎞⎜⎝
⎛ −−−∂=∂ =
−∑ )()(21
}){},{,(log 1
1
1
kkkk
kk
n μμΣμμ
=−=
∂=
∂− 0)(1
kk μμ =⇒ ˆ
sample mean
8Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH1 (2)
• Estimate Σk in H1 ,
Hypothesis H1 (2)
k
kkN
k
kkN pp
ΣΣμx
ΣΣμx
∂∂=
∂∂ }){},{,(log}){},ˆ{,(log 11
( )C
kkkkkkk
Tkkkn ΣΣΣμμΣμμ ⎟
⎠
⎞⎜⎝
⎛ ++−−−∂=
−−∑ ||log)trace()()(21
1
11
( )kkkkk
k
n ΣΣΣΣ
Σ
=+−−=
∂
−−− 021 111( )
kk
kkkkk
ΣΣ =⇒ ˆ2
9
sample covaraince
Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH1 (3)
• The ML value of the data is
Hypothesis H1 (3)
( )∑
==C
T
kkN
HkkN
HN
H
d
ppp
11
111*
||l)()()(1)(
}){},{,(log})ˆ{},ˆ{,(log)(log111
ΣμxΣμxx
( )
( )∑
∑=
−−
+
++−−−=
Ck
kkkkkkT
kkk
dndNg
ndNg1
11
||log1)(
||log)trace()()(21),(
Σ
ΣΣΣμμΣμμ
( )
∑
∑=
−−=
+−=
C
kk
kkk
nNddNg
dndNg1
||log1)(
||log2
),(
Σ
Σ
∑=
=k
kkndNg1
||log22
),( Σ
10Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Likelihood Ratio Procedure in GFLDA (3)
• Log-likelihood Ratio between H0 and H1
Likelihood Ratio Procedure in GFLDA (3)
⎞⎛
′′−=
C
iiN
HiN
HHH
Nd
ppLR 11,
1
}){},{,(logmax}){,,(logmaxlog10
10ΣμxΣμx
∑=
⎞⎛
⎟⎠
⎞⎜⎝
⎛ +−−=
C
C
kkkk
Nd
nNddNg1
1
||log21
2),( ΣB
∑∑
∑=
⎟⎞
⎜⎛
⎟⎠
⎞⎜⎝
⎛ −−−
CC
kkkn
NddNg
1
1
11
||log21
2),( Σ
∑∑=
−
=
+−=⎟⎠
⎞⎜⎝
⎛ −+−=k
kkkkk
kkk nn1
1
1||log
21|)|log||(log
21 BΣIΣΣB
11Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Likelihood Ratio Procedure in GFLDA (4)
• Objective Function:
Likelihood Ratio Procedure in GFLDA (4)
∑∑=
−
=
− −−+−=+−C
k
Tkkkk
C
kkkk nn
1
1
1
1 |)ˆ)(ˆ(|log21||log
21 μμμμΣIBΣI
( )∑=
− −−+−=C
kkk
Tkkn
1
1 )ˆ()ˆ(1log21 μμΣμμ
yx vectors):,(
• After a linear projection: xyxyIyx
TT 1)det(vectors):,(
+=+
( )∑C
11 ( )∑=
−−+−=k
kkT
kknJ1
TT1-TTT )ˆ()()ˆ(1log21)( μΘμΘΘΣΘμΘμΘΘ
12Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Likelihood Ratio Procedure in GFLDA (5)
• Optimization:
Likelihood Ratio Procedure in GFLDA (5)
( )∑=
−−+−=C
kkk
TkknJ
1
TT1-TTT )ˆ()()ˆ(1log21)( μΘμΘΘΣΘμΘμΘΘ
∑ −
−−
++−−=
∂∂ C
kkkkkkn
J1
11
)~~t (1
~)~~(21)(
BΣΣΘBBΣΘΣ
ΘΘ
⎪
⎪⎨
⎧
=−−=
ΘBΘBμμμμB
kk
Tkkk
T~)ˆ)(ˆ(
∑= +∂ k kk1
1 )trace(12 BΣΘ ⎪⎩ = ΘΣΘΣ kk
T~
13Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis