Lecture 21∗

Agenda for the lecture

• Feinstein’s maximal code construction for BSC

21.1 Feinstein’s maximal code construction

In the previous lecture, we showed that for a BSC with crossover probability δ < 1/4,

1− h(2δ) ≤ Cε(W ) ≤ 1− h(δ),

for every 0 < ε < 1. The achievability bound was derived using a greedy construction due

to Gilbert and Varshamov. One may suspect that this construction is suboptimal since we

were insisting on decoding sets Dm to be Hamming balls. Such structured codes, while

great for implementation, may not be of optimal rates.

In this lecture, we present another construction due to Feinstein which yields an optimal

rate code. The construction is still greedy. However, instead of seeking decoding sets that

are Hamming balls, we strive to match the minimal requirements for a decoding set. We

recursively select a subset Dm from the output sequences such that Dm can constitute

a decoding set. Once a Dm is selected, we repeat the process and look for Dm+2 in the

complement of D1, ..., Dm. Note that the only property required of the selected subset

∗ c©Himanshu Tyagi. Feel free to use with acknowledgement.

1

Dm = S at each step is that there exists x ∈ X n such that Wn(S|x) ≥ 1− ε. Furthermore,

we need each selected S to be of the least cardinality possible to maximize the number of

subsets D1, ..., DM we select. Once such an S and x are found, we fix e(m) = x. Formally,

the process is described as follows:

Step 1. Initialize R = Yn.

Step 2. Find the subset S ⊆ R of the least cardinality such that

∃x ∈ X n satisfying Wn(S|x) ≥ 1− ε.

Choose S as the next decoding set Dm with the corresponding codeword xm = x.

Step 3. Update R → R \Dm.

Step 4. Repeat steps 2 and 3 while you can find a set S in Step 2.

Suppose the process above identifies the decoding sets {D1, ..., DM}. Then, the final R

must be disjoint from Dm’s and together R, D1, ..., Dm form a partition of Yn. In order to

estimate the number of decoding sets M that this process can select, we need to address

the following questions:

Q1. How many sequences are selected by the decoding sets?

Q2. What is the cardinality of each decoding set?

To address the first question, we need to first understand the condition for the process

above to terminate. The process terminates when the remaining set of output sequences

R satisfies

Wn(R|x) < 1− ε, ∀x ∈ X n.

Note that ifD1, ..., DM are the disjoint decoding sets selected up to this point, {D1, ..., DM ,R}

2

forms a partition of Yn. Therefore, by the condition above

Wn(∪Mm=1Dm|x) > ε, ∀x ∈ X n.

Taking average over x on the left-side, we obtain

>1

2n

∑x

Wn(∪Mm=1Dm|x) > ε.

Next, we observe that the average probability on the left-side above corresponds to a i.i.d.

rvs. Indeed, for each y

1

2n

∑x

Wn(y|x) =∑x

Pn(x)n∏i=1

W (yi|xi)

=n∏i=1

∑xi

P(xi)W (yi|xi)

=n∏i=1

PW (yi), (1)

where P is Bernoulli(0.5). It follows that for any set B ⊆ {0, 1}n

1

2n

∑x

Wn(B|x) = (PW )n(B).

Note that for a BSC,

PW (y) =1

2W (y|0) +

1

2W (y|1) =

1

2.

Thus,

(PW )n(B) =B

2n.

Combining the observations above

∣∣∪Mm=1Dm

∣∣ ≥ ε · 2n. (2)

3

This answers Q1.

Moving to Q2, note that each decoding set Dm has the property that

1− ε ≤Wn(Dm|xm) = PnZ(Dm ⊕ xm),

where PZ denotes the Bernoulli(δ) distribution. We saw earlier in fixed-length source

coding that for i.i.d. Zn, there exists a set A such that PnZ(A) ≥ 1−ε and log |A| . nH(Z).

Before the process stops, there exists a sequence x such that the residual set R satisfies

Wn(R|x) ≥ 1− ε. It is tempting to suggest that we can choose Dm = A⊕ x and xm = x

so that

PnZ(Dm ⊕ xm) = PnZ(A) ≥ 1− ε.

However, in the greedy construction above Dm must be chosen from the residual set R,

and A⊕ xm may not be a subset of R. It will suffice to show that for any set A such that

PnZ(A) ≥ 1− ε, there exists a subset B of A such that PnZ(B) ≥ 1− ε and log |B| . nH(Z).

As we discussed in the class, this seems not so easy to prove.

Instead, we modify the greedy process above to ensure that every decoding set is selected

to be of cardinality which is not too large. From the previous lecture, recall that for

ρn = nδ +√

nδ(1−δ)η , the Hamming ball Bρn(0) satisfies

PnZ(Bρn(0)) ≥ 1− η,

and

log |Bρn(0)| ≤ nh(ρnn

)+ log n . nh(δ).

Our modified code construction is as follows:

Step 1. Initialize R = Yn.

4

Step 2. If there exists a sequence x ∈ X n such that

Wn(R∩Bρn(x)|x) ≥ 1− ε,

Choose the next decoding set Dm = R ∩ Bρn(x) with the corresponding codeword

xm = x.

Step 3. Update R → R \Dm.

Step 4. Repeat steps 2 and 3 while you can find a set S in Step 2.

We now revisit the two questions raised earlier for this modified construction. We begin

with Q2, the obstacle in the previous construction. Note that each decoding set Dm is a

subset of Bρn(x). Therefore,

|Dm| ≤ |Bρn(x)| = |Bρn(0)| ≤ n · 2nh(ρnn ), (3)

whereby the sizes of our decoding sets are under control.

Returning to Q1, the process terminates when for all x

Wn(Bρn(x) ∩R|x) < 1− ε,

which yields

1− η ≤ PnZ(Bρn(0)

= Wn(Bρn(x)|x)

= Wn(Bρn(x) ∩R|x) +Wn(Bρn(x) ∩Rc|x)

< 1− ε+Wn(Bρn(x) ∩Rc|x)

≤ 1− ε+Wn(Rc|x), ∀x ∈ X n.

5

Therefore,

Wn(Rc|x) ≥ ε− η ∀x ∈ X n.

Proceeding as we did to obtain (2), we have

|Rc| =M∑m=1

|Dm| ≥ (ε− η)2n. (4)

On combining (4) with (3), we obtain

1

nlogM ≥ 1− h

(δ +

√δ(1− δ)nη

)− 1

nlog

1

ε− η.

Therefore, the right-side above is also a lower bound for (1/n) logM(n, ε), which yields

Cε(W ) ≥ 1− h(δ).

We have established the following result.

Theorem 21.1 (Capacity of a BSC). Consider a BSC W with crossover probability 0 ≤

δ ≤ 1. Then, for every 0 < ε < 1,

Cε(W ) = 1− h(δ) = C(W ).

Note that we have established a strong converse – the ε-capacity Cε(W ) remains the

same for each ε ∈ (0, 1).

6