Posterior Matching Variants and Fixed-Point Eliminationofersha/allerton09seminar.pdf · 2010. 1....

31
Posterior Matching Variants and Fixed-Point Elimination Ofer Shayevitz, UCSD Allerton 2009 Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 1/14

Transcript of Posterior Matching Variants and Fixed-Point Eliminationofersha/allerton09seminar.pdf · 2010. 1....

  • Posterior Matching Variants andFixed-Point Elimination

    Ofer Shayevitz, UCSD

    Allerton 2009

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 1/14

  • Feedback Communication Setting

    ⊆ (0,1)

    Message point Θ0 ∼ Unif(0, 1)

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 2/14

  • Feedback Communication Setting

    ⊆ (0,1)

    Message point Θ0 ∼ Unif(0, 1)

    Transmission scheme achieves a rate R if

    PΘ0|Y n(∆n|Yn) → 1 , |∆n(Y

    n))| = O(

    2−nR)

    Possibly under input constraints limn→∞

    n−1n

    k=1

    η(Xk) ≤ Γ a.s.

    Implies standard achievability

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 2/14

  • The Posterior Matching (PM) Scheme[Shayevitz & Feder ’07,’08]

    For any input/channel pair (PX , PY |X)

    Xn+1 = F−1X ◦ FΘ0|Y n(Θ0|Y

    n)

    Simple and sequential, achieves R < I(X;Y ) under general

    conditions, within input constraints encapsulated in PX

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 3/14

  • The Posterior Matching (PM) Scheme[Shayevitz & Feder ’07,’08]

    For any input/channel pair (PX , PY |X)

    Xn+1 = F−1X ◦ FΘ0|Y n(Θ0|Y

    n)

    Simple and sequential, achieves R < I(X;Y ) under general

    conditions, within input constraints encapsulated in PX

    Specifically, holds for (minus issues immediately discussed)

    DMC

    PXY with a sufficiently smooth p.d.f.

    Horstein & Schalkwijk-Kailath are special cases

    Stochastic control formulation [Coleman ’09]

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 3/14

  • Normalized Channel and Recursive Representation

    PM scheme recursive representation [Shayevitz & Feder ’08]

    Θ1 = Θ0 , Θn+1 = FΘ|Y (Θn|Yn)

    FΘ|Y (·|·) is called the PM kernel

    Posterior c.d.f. evolves by function composition with PM kernel

    FΘ0|Y n(·|Yn) = FΘ|Y (·|Yn) ◦ FΘ0|Y n−1(·|Y

    n−1)

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 4/14

  • A Necessary Condition for Achievability

    The PM kernel has a fixed-point at θf if

    FΘ|Y (θf |y) = θf , ∀ y ∈ Y

    Claim: If the PM kernel has a fixed point, then the PM scheme isnot ergodic and cannot achieve any positive rate

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 5/14

  • A Necessary Condition for Achievability

    The PM kernel has a fixed-point at θf if

    FΘ|Y (θf |y) = θf , ∀ y ∈ Y

    Claim: If the PM kernel has a fixed point, then the PM scheme isnot ergodic and cannot achieve any positive rate

    For a DMC, maximal number of fixed-points is |X | − 2

    No fixed point for a binary input alphabet

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 5/14

  • A Necessary Condition for Achievability

    The PM kernel has a fixed-point at θf if

    FΘ|Y (θf |y) = θf , ∀ y ∈ Y

    Claim: If the PM kernel has a fixed point, then the PM scheme isnot ergodic and cannot achieve any positive rate

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 5/14

  • 3 × 2 Example

    PM kernel comprised of two quasi-linear functions

    Suppose there is a fixed point, i.e., FΘ|Y (θf |0) = FΘ|Y (θf |1) = θf

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 6/14

  • 3 × 2 Example

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 7/14

  • 3 × 2 Example

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 7/14

  • 3 × 2 Example

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 7/14

  • 3 × 2 Example

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 7/14

  • 3 × 2 Example

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 7/14

  • 3 × 2 Example – What can we do?

    Two invariant intervals

    Idea I:

    Decode two messages

    Resolve (e.g. via repetition)

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 8/14

  • 3 × 2 Example – What can we do?

    Two invariant intervals

    Idea I:

    Decode two messages

    Resolve (e.g. via repetition)

    Problems:

    Rate per invariant interval can be < I(X;Y )

    Non-ergodic chain, input constraints not necessarily satisfied

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 8/14

  • 3 × 2 Example – What can we do?

    Two invariant intervals

    Idea II:

    Map message to ”best“ interval

    R ≥ I(X;Y )

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 9/14

  • 3 × 2 Example – What can we do?

    Two invariant intervals

    Idea II:

    Map message to ”best“ interval

    R ≥ I(X;Y )

    Equivalent to PM with a different (binary) input distribution

    Related to the observation that when |X | > |Y|, using |Y| input

    symbols suffices to achieve the unconstrained capacity [Shannon ’57]

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 9/14

  • 3 × 2 Example – What can we do?

    Two invariant intervals

    Idea II:

    Map message to ”best“ interval

    R ≥ I(X;Y )

    Equivalent to PM with a different (binary) input distribution

    Related to the observation that when |X | > |Y|, using |Y| input

    symbols suffices to achieve the unconstrained capacity [Shannon ’57]

    Problem: Input constraints not satisfied

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 9/14

  • 3 × 2 Example – How Common?

    Choose PXY uniformly over the simplex

    Let FΘ|Y be the corresponding PM kernel

    How likely is a fixed-point for FΘ|Y ?

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 10/14

  • 3 × 2 Example – How Common?

    Choose PXY uniformly over the simplex

    Let FΘ|Y be the corresponding PM kernel

    How likely is a fixed-point for FΘ|Y ?

    Claim: P(

    FΘ|Y has a fixed point)

    = 13

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 10/14

  • 3 × 2 Example – How Common?

    Proof:

    Define δk = PX|Y (k|1) − PX|Y (k|0), k = 0, 1, 2∑

    δk = 0

    Suppose |δ0| ≤ |δ2| ≤ |δ1|, hence δ1 = −(δ0 + δ2)

    The c.d.f’s must intersect

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 11/14

  • 3 × 2 Example – How Common?

    Proof:

    Define δk = PX|Y (k|1) − PX|Y (k|0), k = 0, 1, 2∑

    δk = 0

    Suppose |δ0| ≤ |δ2| ≤ |δ1|, hence δ1 = −(δ0 + δ2)

    The c.d.f’s must intersect

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 11/14

  • 3 × 2 Example – How Common?

    Proof:

    Define δk = PX|Y (k|1) − PX|Y (k|0), k = 0, 1, 2∑

    δk = 0

    Suppose |δ0| ≤ |δ2| ≤ |δ1|, hence δ1 = −(δ0 + δ2)

    The c.d.f’s must intersect

    E(FΘ|Y (θ|Y )) = FΘ(θ) = θ

    Intersection is a fixed point

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 11/14

  • 3 × 2 Example – How Common?

    Proof:

    Define δk = PX|Y (k|1) − PX|Y (k|0), k = 0, 1, 2∑

    δk = 0

    Suppose |δ0| ≤ |δ2| ≤ |δ1|, hence δ1 = −(δ0 + δ2)

    The c.d.f’s must intersect

    E(FΘ|Y (θ|Y )) = FΘ(θ) = θ

    Intersection is a fixed point

    Can always permute/relabel the input so that δk satisfies this

    Fixed point obtained also for mirror case

    2 out of 6 permutations ⇒ fixed point

    Result follows from symmetry

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 11/14

  • A Systematic Solution

    Note the glass is 23 full..

    Relabeling the input s.t. δ0 ≥ δ1 ≥ δ2

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 12/14

  • A Systematic Solution

    Note the glass is 23 full..

    Relabeling the input s.t. δ0 ≥ δ1 ≥ δ2 ⇒ no fixed points!

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 12/14

  • A Systematic Solution

    Note the glass is 23 full..

    Relabeling the input s.t. δ0 ≥ δ1 ≥ δ2 ⇒ no fixed points!

    For any DMC with I(X;Y ) > 0

    ∃ y1, y2 ∈ Y s.t. PX|Y (·|y1) 6= PX|Y (·|y2).

    Find an input permutation σ sorting the δk, for which

    Fσ(X)|Y (·|y1) < Fσ(X)|Y (·|y2)

    Consider PM for the equivalent input/channel pair (Pσ(X), PY |σ(X))

    No fixed points, achieves I(X ; Y ) within input constraints

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 12/14

  • The General Case

    Beyond permutations: A bijective function µ : (0,1) 7→ (0,1) is called

    uniformity preserving if

    Θ ∼ Unif(0,1) ⇒ µ(Θ) ∼ Unif(0,1)

    The µ-variant of the PM scheme is

    Xn+1 = F−1X ◦ µ ◦ FΘ0|Y n(Θ0|Y

    n)

    µ-variant kernel µ ◦ Fµ−1(Θ)|Y (·|y) ◦ µ−1

    µ eliminates fixed points, I(X;Y ) achieved under input constraints

    For DMC, µ was a permutation of intervals that correspond to thediscrete input alphabet

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 13/14

  • Summary

    With fixed points, PM fails to achieve capacity

    PM variants can eliminate fixed points, achieving capacity

    Many variants achieving capacity, some “closer” to having a

    fixed-point than others

    What is the “best” variant, minimizing error probability for a giveninput/channel pair?

    Ofer Shayevitz, UCSD Posterior Matching Variants and Fixed-Point Elimination – p. 14/14

    s {Feedback Communication Setting}s {Feedback Communication Setting}

    s {The Posterior Matching (PM) Scheme {iny [Shayevitz & Feder'07,'08]}}s {The Posterior Matching (PM)Scheme {iny [Shayevitz & Feder '07,'08]}}

    s {Normalized Channel and Recursive Representation}s {A Necessary Condition for Achievability}s {A Necessary Condition for Achievability}s {A Necessary Condition for Achievability}

    s {$3imes 2$ Example}s {$3imes 2$ Example}s {$3imes 2$ Example}s {$3imes 2$ Example}s {$3imes 2$ Example}s {$3imes 2$ Example}

    s {$3imes 2$ Example -- What can we do?}s {$3imes 2$ Example -- What can we do?}

    s {$3imes 2$ Example -- What can we do?}s {$3imes 2$ Example -- What can we do?}s {$3imes 2$ Example -- What can we do?}

    s {$3imes 2$ Example -- How Common?}s {$3imes 2$ Example -- How Common?}

    s {$3imes 2$ Example -- How Common?}s {$3imes 2$ Example -- How Common?}s {$3imes 2$ Example -- How Common?}s {$3imes 2$ Example -- How Common?}

    s {A Systematic Solution}s {A Systematic Solution}s {A Systematic Solution}

    s {The General Case}s {Summary}