A Note on Sparse Stochastic Inference for Latent Dirichlet Allocation

Click here to load reader

  • date post

    28-Nov-2014
  • Category

    Technology

  • view

    384
  • download

    2

Embed Size (px)

description

 

Transcript of A Note on Sparse Stochastic Inference for Latent Dirichlet Allocation

  • 1. A derivation of Eqs. (4) and (5) of Sparse stochastic inference for latent Dirichlet allocation Tomonari MASADA @ Nagasaki University May 23, 2013 The evidence can be written as follows: p(w|, ) = z k p(k|) d p(d|)p(zd|d)p(w|zd, ) dd . (1) By integrating out, we have p(w|, ) = z k p(k|) d p(zd|)p(w|zd, ) d . (2) By applying Jensens inequality, we have a lower bound of the evidence as follows: log p(w|, ) = log z k p(k|) d p(zd|)p(w|zd, )d = log z d q(zd) k q(k) k p(k|) d p(zd|)p(w|zd, ) d q(zd) k q(k) d z d q(zd) k q(k) log k p(k|) d p(zd|)p(w|zd, ) d q(zd) k q(k) d = k q(k) log p(k|)dk + d zd q(zd) log p(zd|) + k q(k) d zd q(zd) log p(w|zd, )d + H(q) = k q(k) log p(k|)dk + d zd q(zd) log p(zd|) + k q(k) d zd q(zd) Nd i=1 log zdiwdi d + H(q) , (3) where H(q) = k q(k) log q(k)dk + d zd q(zd) log q(zd). Let L denote the lower bound in Eq. (3). By picking up the terms related to zd from L, we dene Lzd as follows: Lzd = zd q(zd) log p(zd|) + k q(k) zd q(zd) Nd i=1 log zdiwdi d zd q(zd) log q(zd) . (4) We obtain a functional derivative of Lzd with respect to q(zd) as follows: Lzd q(zd) = lim 0 zd {q(zd) + (zd zd)} log p(zd|) zd q(zd) log p(zd|) + lim 0 k q(k) zd {q(zd) + (zd zd)} Nd i=1 log zdiwdi zd q(zd) Nd i=1 log zdiwdi d lim 0 zd {q(zd) + (zd zd)} log{q(zd) + (zd zd)} zd q(zd) log q(zd) , (5) 1
  • 2. where lim 0 zd {q(zd) + (zd zd)} log{q(zd) + (zd zd)} zd q(zd) log q(zd) = lim 0 zd q(zd) log q(zd)+ (zdzd) q(zd) + zd (zd zd) log{q(zd) + (zd zd)} = lim 0 zd q(zd) (zdzd) q(zd) + O( 2 ) + lim 0 zd (zd zd) log{q(zd) + (zd zd)} = zd (zd zd) + zd (zd zd) log q(zd) = 1 + log q(zd) . (6) Therefore, Lzd q(zd) = log p(zd|) + k q(k) Nd i=1 log zdiwdi d 1 log q(zd) (7) By solving Lzd q(zd) = 0, we obtain q(zd) p(zd|) exp k q(k) Nd i=1 log zdiwdi d = p(zd|) Nd i=1 exp k q(k) log zdiwdi d (K) (K + Nd) k ( + i Izdi=k) () i exp Eq log zdiwdi (8) We assume that q(k) = ( w kw) w (kw) w kw1 kw . By picking up the terms related to from L, we dene L as follows: L = k q(k) log p(k|)dk + k q(k) d zd q(zd) Nd i=1 log zdiwdi d k q(k) log q(k)dk . (9) Each term in Eq. (9) can be rewritten as below. q(k) log p(k|)dk = log (W) w log () + w ( 1) (kw) ( w kw) (10) q(k) log q(k)dk = log ( w kw) w log (kw) + w (kw 1) (kw) ( w kw) (11) k q(k) d zd q(zd) Nd i=1 log zdiwdi d = k q(k) d zd q(zd) Nd i=1 k w I(zdi = k, wdi = w) log kw d = k q(k) k w log kw d zd q(zd) Nd i=1 I(zdi = k, wdi = w) d = k w (kw) ( w kw) d zd q(zd) Nd i=1 I(zdi = k, wdi = w) (12) 2
  • 3. Therefore, L kw = kw + d zd q(zd) Nd i=1 I(zdi = k, wdi = w) (kw) kw + d zd q(zd) Nd i=1 I(zdi = k, wdi = w) ( w kw) . (13) By solving L kw = 0, we obtain kw = + d zd q(zd) Nd i=1 I(zdi = k, wdi = w) . (14) 3