Likelihood Ratio Procedure in GFLDA (1)

Hung-Shin Lee3/23/2009

• Log Likelihood:

Likelihood Ratio Procedure in GFLDA (1)

−−=

C

kkN Ndp 1

1

)2log(2

}){},{,(log Σμx π ),( dNg

⎧

( )∑=

−− ++−−C

kkkkkkk

Tkkkn

1

11 ||log)trace()()(21 ΣΣΣμμΣμμ

• Hypotheses:⎩⎨⎧

edunrestrict are s':equal are s':

edunrestrict:1

0

k

kk H

Hμμ

Σ

1Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

Likelihood Ratio Procedure in GFLDA (2)

• Likelihood :

Likelihood Ratio Procedure in GFLDA (2)

}){}{(max

}){,,(max

}){}{(max

}){},{,(max 1100

N

kN

HN

kkN

H

p

p

p

pLR

Σμx

Σμx

Σμx

Σμx

′′=

′′=

• In logarithmic domain,

}){},{,(max}){},{,(max 1111

kkHkkHpp ΣμxΣμx

In logarithmic domain,

}){},{,(logmax}){,,(logmaxlog 1110

iiN

HiN

HppLR ΣμxΣμx ′′−=

2Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

HypothesisH0 (1)

• Estimate μk in H0 ,

Hypothesis H0 (1)

∂∂=

∂∂ k

N

k

kkN pp 11 }){,,(log}){},{,(log

μΣμx

μΣμx

∑∑

−=

−

=−=∂

⎟⎠

⎞⎜⎝

⎛ −−−∂=

C

kkk

C

kkk

Tkk

nn

11

1

0)()()(

21

μμΣμμΣμμ

∑∑∑∑

∑

−−

−−−

=

⎟⎞

⎜⎛≈⇒=⇒

∂C

kkk

C

kk

C

kk

C

kkk

kkkk

nnnn 11

111

1

ˆ

)(

μΣΣμμΣμΣ

μμμ

∑∑∑∑====

⎟⎠

⎜⎝

⇒⇒k

kkkk

kkk

kkk

kkk nnnn1111

μμμμ

weighted mean

3

g

Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

HypothesisH0 (2)

• Estimate Σk in H0 ,

Hypothesis H0 (2)

k

kN

k

kkN pp

ΣΣμx

ΣΣμx

∂∂=

∂∂ }){,ˆ,(log}){},{,(log 11

( )C

kkkkkk

Tkkn ΣΣΣμμΣμμ

∂

⎟⎠

⎞⎜⎝

⎛ ++−−−∂=

−−∑ ||log)trace()ˆ()ˆ(21

1

11

( )kkkkkT

kkkk

k

n ΣΣΣΣΣμμμμΣ

Σ

=+−−−−−=

∂

−−−−− 0)ˆ)(ˆ(21 11111

kkkT

kkk ΣBΣμμμμΣ +=+−−=⇒ )ˆ)(ˆ(ˆ2

4Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

Appendix A - Derivatives of MatricesAppendix A Derivatives of Matrices

( )11

2 )()(trace −∂ ASAASA TTsymmetric) :(A ( )1

211

22

12

))(()(2

)()(trace

−−−

=∂

∂

ASAASAASAASA

ASAASA

TTT2

y )(

=∂

∂ AxxxAx

121 )(2 −+ ASAAS T

1

1||log

−

−

∂

=∂

∂

A

AAA

T

111

111

)trace( −

−−

∂

−=∂

∂

XA

AxxAAxAx

T

TT

11)trace( −−−=∂

∂ AXAA

XA T

5Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

HypothesisH0 (3)

• The ML value of the data is

Hypothesis H0 (3)

( )∑ −− ++

=C

T

kN

HN

H

dN

pp

11

11*

|ˆ|log)ˆtrace()ˆ()ˆ(1)(

})ˆ{,ˆ,(log)(log00

ΣΣΣμμΣμμ

Σμxx

( )∑−

=

⎟⎞

⎜⎛ +−+−

++−−−=

Ckkk

Tk

kkkkkk

TkkndNg

1

1

11

)ˆ()()ˆ(

||log)trace()()(2

),(

μμΣBμμ

ΣΣΣμμΣμμ

∑=

−

⎟⎟⎟⎟

⎠⎜⎜⎜⎜

⎝ +++−=

C

kkk

kkkkndNg1

1

||log))trace((

21),(

ΣBΣΣB

∑=

+−−=

⎠⎝C

kkkk

kk

nNddNg1

||log21

2),( ΣB

6

k 1

Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

Appendix BAppendix B

( )))trace(()ˆ()()ˆ(1 11nC

T +++∑ −− ΣΣBμμΣBμμ( )

( ))()ˆ)(ˆ()(trace1

))trace(()()()(2

11

1

n

n

C

kkkT

kkkkk

kkkkkkkkk

++−−+=

++−+−

∑

∑−−

=

ΣΣBμμμμΣB

ΣΣBμμΣBμμ

( )

( )))ˆ)(ˆ(()(trace21

)())(()(trace2

1

1

n

n

C

kT

kkkkk

kkkkkkkkk

+−−+=

+++

∑

∑−

=

ΣμμμμΣB

ΣΣBμμμμΣB

( )

( ))()(trace21

)))((()(2

1

1

nC

kkkkk

kkkkkkk

++= ∑

∑−

=

ΣBΣB

μμμμ

( )2

trace212

1

1

NdnC

kddk

k

== ∑ ×

=

I

7

22 1k=

Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

HypothesisH1 (1)

• Estimate μk in H1 ,

Hypothesis H1 (1)

C

kkkk

Tkkk

kkN n

pμμΣμμ

Σμx⎟⎠

⎞⎜⎝

⎛ −−−∂=∂ =

−∑ )()(21

}){},{,(log 1

1

1

kkkk

kk

n μμΣμμ

=−=

∂=

∂− 0)(1

kk μμ =⇒ ˆ

sample mean

8Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

HypothesisH1 (2)

• Estimate Σk in H1 ,

Hypothesis H1 (2)

k

kkN

k

kkN pp

ΣΣμx

ΣΣμx

∂∂=

∂∂ }){},{,(log}){},ˆ{,(log 11

( )C

kkkkkkk

Tkkkn ΣΣΣμμΣμμ ⎟

⎠

⎞⎜⎝

⎛ ++−−−∂=

−−∑ ||log)trace()()(21

1

11

( )kkkkk

k

n ΣΣΣΣ

Σ

=+−−=

∂

−−− 021 111( )

kk

kkkkk

ΣΣ =⇒ ˆ2

9

sample covaraince

Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

HypothesisH1 (3)

• The ML value of the data is

Hypothesis H1 (3)

( )∑

==C

T

kkN

HkkN

HN

H

d

ppp

11

111*

||l)()()(1)(

}){},{,(log})ˆ{},ˆ{,(log)(log111

ΣμxΣμxx

( )

( )∑

∑=

−−

+

++−−−=

Ck

kkkkkkT

kkk

dndNg

ndNg1

11

||log1)(

||log)trace()()(21),(

Σ

ΣΣΣμμΣμμ

( )

∑

∑=

−−=

+−=

C

kk

kkk

nNddNg

dndNg1

||log1)(

||log2

),(

Σ

Σ

∑=

=k

kkndNg1

||log22

),( Σ

10Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

Likelihood Ratio Procedure in GFLDA (3)

• Log-likelihood Ratio between H0 and H1

Likelihood Ratio Procedure in GFLDA (3)

⎞⎛

′′−=

C

iiN

HiN

HHH

Nd

ppLR 11,

1

}){},{,(logmax}){,,(logmaxlog10

10ΣμxΣμx

∑=

⎞⎛

⎟⎠

⎞⎜⎝

⎛ +−−=

C

C

kkkk

Nd

nNddNg1

1

||log21

2),( ΣB

∑∑

∑=

⎟⎞

⎜⎛

⎟⎠

⎞⎜⎝

⎛ −−−

CC

kkkn

NddNg

1

1

11

||log21

2),( Σ

∑∑=

−

=

+−=⎟⎠

⎞⎜⎝

⎛ −+−=k

kkkkk

kkk nn1

1

1||log

21|)|log||(log

21 BΣIΣΣB

11Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

Likelihood Ratio Procedure in GFLDA (4)

• Objective Function:

Likelihood Ratio Procedure in GFLDA (4)

∑∑=

−

=

− −−+−=+−C

k

Tkkkk

C

kkkk nn

1

1

1

1 |)ˆ)(ˆ(|log21||log

21 μμμμΣIBΣI

( )∑=

− −−+−=C

kkk

Tkkn

1

1 )ˆ()ˆ(1log21 μμΣμμ

yx vectors):,(

• After a linear projection: xyxyIyx

TT 1)det(vectors):,(

+=+

( )∑C

11 ( )∑=

−−+−=k

kkT

kknJ1

TT1-TTT )ˆ()()ˆ(1log21)( μΘμΘΘΣΘμΘμΘΘ

12Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis

Likelihood Ratio Procedure in GFLDA (5)

• Optimization:

Likelihood Ratio Procedure in GFLDA (5)

( )∑=

−−+−=C

kkk

TkknJ

1

TT1-TTT )ˆ()()ˆ(1log21)( μΘμΘΘΣΘμΘμΘΘ

∑ −

−−

++−−=

∂∂ C

kkkkkkn

J1

11

)~~t (1

~)~~(21)(

BΣΣΘBBΣΘΣ

ΘΘ

⎪

⎪⎨

⎧

=−−=

ΘBΘBμμμμB

kk

Tkkk

T~)ˆ)(ˆ(

∑= +∂ k kk1

1 )trace(12 BΣΘ ⎪⎩ = ΘΣΘΣ kk

T~

13Likelihood Ratio Procedure in Generalized Fisher’s Linear Discriminant Analysis