Post on 13-Nov-2014
Likelihood Ratio Procedure in GFLDA (1)
Hung-Shin Lee3/23/2009
• Log Likelihood:
Likelihood Ratio Procedure in GFLDA (1)
−−=
C
kkN Ndp 1
1
)2log(2
}){},{,(log Σμx π ),( dNg
⎧
( )∑=
−− ++−−C
kkkkkkk
Tkkkn
1
11 ||log)trace()()(21 ΣΣΣμμΣμμ
• Hypotheses:⎩⎨⎧
edunrestrict are s':equal are s':
edunrestrict:1
0
k
kk H
Hμμ
Σ
1Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Likelihood Ratio Procedure in GFLDA (2)
• Likelihood :
Likelihood Ratio Procedure in GFLDA (2)
}){}{(max
}){,,(max
}){}{(max
}){},{,(max 1100
N
kN
HN
kkN
H
p
p
p
pLR
Σμx
Σμx
Σμx
Σμx
′′=
′′=
• In logarithmic domain,
}){},{,(max}){},{,(max 1111
kkHkkHpp ΣμxΣμx
In logarithmic domain,
}){},{,(logmax}){,,(logmaxlog 1110
iiN
HiN
HppLR ΣμxΣμx ′′−=
2Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH0 (1)
• Estimate μk in H0 ,
Hypothesis H0 (1)
∂∂=
∂∂ k
N
k
kkN pp 11 }){,,(log}){},{,(log
μΣμx
μΣμx
∑∑
−=
−
=−=∂
⎟⎠
⎞⎜⎝
⎛ −−−∂=
C
kkk
C
kkk
Tkk
nn
11
1
0)()()(
21
μμΣμμΣμμ
∑∑∑∑
∑
−−
−−−
=
⎟⎞
⎜⎛≈⇒=⇒
∂C
kkk
C
kk
C
kk
C
kkk
kkkk
nnnn 11
111
1
ˆ
)(
μΣΣμμΣμΣ
μμμ
∑∑∑∑====
⎟⎠
⎜⎝
⇒⇒k
kkkk
kkk
kkk
kkk nnnn1111
μμμμ
weighted mean
3
g
Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH0 (2)
• Estimate Σk in H0 ,
Hypothesis H0 (2)
k
kN
k
kkN pp
ΣΣμx
ΣΣμx
∂∂=
∂∂ }){,ˆ,(log}){},{,(log 11
( )C
kkkkkk
Tkkn ΣΣΣμμΣμμ
∂
⎟⎠
⎞⎜⎝
⎛ ++−−−∂=
−−∑ ||log)trace()ˆ()ˆ(21
1
11
( )kkkkkT
kkkk
k
n ΣΣΣΣΣμμμμΣ
Σ
=+−−−−−=
∂
−−−−− 0)ˆ)(ˆ(21 11111
kkkT
kkk ΣBΣμμμμΣ +=+−−=⇒ )ˆ)(ˆ(ˆ2
4Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Appendix A - Derivatives of MatricesAppendix A Derivatives of Matrices
( )11
2 )()(trace −∂ ASAASA TTsymmetric) :(A ( )1
211
22
12
))(()(2
)()(trace
−−−
=∂
∂
ASAASAASAASA
ASAASA
TTT2
y )(
=∂
∂ AxxxAx
121 )(2 −+ ASAAS T
1
1||log
−
−
∂
=∂
∂
A
AAA
T
111
111
)trace( −
−−
∂
−=∂
∂
XA
AxxAAxAx
T
TT
11)trace( −−−=∂
∂ AXAA
XA T
5Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH0 (3)
• The ML value of the data is
Hypothesis H0 (3)
( )∑ −− ++
=C
T
kN
HN
H
dN
pp
11
11*
|ˆ|log)ˆtrace()ˆ()ˆ(1)(
})ˆ{,ˆ,(log)(log00
ΣΣΣμμΣμμ
Σμxx
( )∑−
=
⎟⎞
⎜⎛ +−+−
++−−−=
Ckkk
Tk
kkkkkk
TkkndNg
1
1
11
)ˆ()()ˆ(
||log)trace()()(2
),(
μμΣBμμ
ΣΣΣμμΣμμ
∑=
−
⎟⎟⎟⎟
⎠⎜⎜⎜⎜
⎝ +++−=
C
kkk
kkkkndNg1
1
||log))trace((
21),(
ΣBΣΣB
∑=
+−−=
⎠⎝C
kkkk
kk
nNddNg1
||log21
2),( ΣB
6
k 1
Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Appendix BAppendix B
( )))trace(()ˆ()()ˆ(1 11nC
T +++∑ −− ΣΣBμμΣBμμ( )
( ))()ˆ)(ˆ()(trace1
))trace(()()()(2
11
1
n
n
C
kkkT
kkkkk
kkkkkkkkk
++−−+=
++−+−
∑
∑−−
=
ΣΣBμμμμΣB
ΣΣBμμΣBμμ
( )
( )))ˆ)(ˆ(()(trace21
)())(()(trace2
1
1
n
n
C
kT
kkkkk
kkkkkkkkk
+−−+=
+++
∑
∑−
=
ΣμμμμΣB
ΣΣBμμμμΣB
( )
( ))()(trace21
)))((()(2
1
1
nC
kkkkk
kkkkkkk
++= ∑
∑−
=
ΣBΣB
μμμμ
( )2
trace212
1
1
NdnC
kddk
k
== ∑ ×
=
I
7
22 1k=
Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH1 (1)
• Estimate μk in H1 ,
Hypothesis H1 (1)
C
kkkk
Tkkk
kkN n
pμμΣμμ
Σμx⎟⎠
⎞⎜⎝
⎛ −−−∂=∂ =
−∑ )()(21
}){},{,(log 1
1
1
kkkk
kk
n μμΣμμ
=−=
∂=
∂− 0)(1
kk μμ =⇒ ˆ
sample mean
8Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH1 (2)
• Estimate Σk in H1 ,
Hypothesis H1 (2)
k
kkN
k
kkN pp
ΣΣμx
ΣΣμx
∂∂=
∂∂ }){},{,(log}){},ˆ{,(log 11
( )C
kkkkkkk
Tkkkn ΣΣΣμμΣμμ ⎟
⎠
⎞⎜⎝
⎛ ++−−−∂=
−−∑ ||log)trace()()(21
1
11
( )kkkkk
k
n ΣΣΣΣ
Σ
=+−−=
∂
−−− 021 111( )
kk
kkkkk
ΣΣ =⇒ ˆ2
9
sample covaraince
Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
HypothesisH1 (3)
• The ML value of the data is
Hypothesis H1 (3)
( )∑
==C
T
kkN
HkkN
HN
H
d
ppp
11
111*
||l)()()(1)(
}){},{,(log})ˆ{},ˆ{,(log)(log111
ΣμxΣμxx
( )
( )∑
∑=
−−
+
++−−−=
Ck
kkkkkkT
kkk
dndNg
ndNg1
11
||log1)(
||log)trace()()(21),(
Σ
ΣΣΣμμΣμμ
( )
∑
∑=
−−=
+−=
C
kk
kkk
nNddNg
dndNg1
||log1)(
||log2
),(
Σ
Σ
∑=
=k
kkndNg1
||log22
),( Σ
10Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Likelihood Ratio Procedure in GFLDA (3)
• Log-likelihood Ratio between H0 and H1
Likelihood Ratio Procedure in GFLDA (3)
⎞⎛
′′−=
C
iiN
HiN
HHH
Nd
ppLR 11,
1
}){},{,(logmax}){,,(logmaxlog10
10ΣμxΣμx
∑=
⎞⎛
⎟⎠
⎞⎜⎝
⎛ +−−=
C
C
kkkk
Nd
nNddNg1
1
||log21
2),( ΣB
∑∑
∑=
⎟⎞
⎜⎛
⎟⎠
⎞⎜⎝
⎛ −−−
CC
kkkn
NddNg
1
1
11
||log21
2),( Σ
∑∑=
−
=
+−=⎟⎠
⎞⎜⎝
⎛ −+−=k
kkkkk
kkk nn1
1
1||log
21|)|log||(log
21 BΣIΣΣB
11Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Likelihood Ratio Procedure in GFLDA (4)
• Objective Function:
Likelihood Ratio Procedure in GFLDA (4)
∑∑=
−
=
− −−+−=+−C
k
Tkkkk
C
kkkk nn
1
1
1
1 |)ˆ)(ˆ(|log21||log
21 μμμμΣIBΣI
( )∑=
− −−+−=C
kkk
Tkkn
1
1 )ˆ()ˆ(1log21 μμΣμμ
yx vectors):,(
• After a linear projection: xyxyIyx
TT 1)det(vectors):,(
+=+
( )∑C
11 ( )∑=
−−+−=k
kkT
kknJ1
TT1-TTT )ˆ()()ˆ(1log21)( μΘμΘΘΣΘμΘμΘΘ
12Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis
Likelihood Ratio Procedure in GFLDA (5)
• Optimization:
Likelihood Ratio Procedure in GFLDA (5)
( )∑=
−−+−=C
kkk
TkknJ
1
TT1-TTT )ˆ()()ˆ(1log21)( μΘμΘΘΣΘμΘμΘΘ
∑ −
−−
++−−=
∂∂ C
kkkkkkn
J1
11
)~~t (1
~)~~(21)(
BΣΣΘBBΣΘΣ
ΘΘ
⎪
⎪⎨
⎧
=−−=
ΘBΘBμμμμB
kk
Tkkk
T~)ˆ)(ˆ(
∑= +∂ k kk1
1 )trace(12 BΣΘ ⎪⎩ = ΘΣΘΣ kk
T~
13Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis