102 CHAPTER 5. CONDITIONAL PROBABILITY AND EXPECTATION

Exercises

1. Let f and g be discrete measurements on a probability space (,B,m). Suppose in particularthat

g() =

i

ai1Gi().

Define a new random variable E(g|f) by

E(g|f)() = Em(.|f())(g) =

i

aim(Gi|f()).

This random variable is called the conditional expectation of g given f . Prove that

Em(g) = Em[E(g|f)].

5.5 Projections

One of the proofs of the Radon-Nikodym theorem is based on the projection theorem of Hilbertspace theory. As this proof is more intuitive than others (at least to the author) and as the pro-jection theorem is of interest in its own right, this section is devoted to developing the propertiesof projections and the projection theorem. Good references for the theory of Hilbert spaces andprojections are Halmos [31] and Luenberger [45].

Recall from Chapter 3 that a Hilbert space is a complete inner product space as defined inExample 2.5.10, which is in turn a normed linear space as Example 2.5.8. In fact, all of the resultsconsidered in this section will be applied in the next section to a very particular Hilbert space, thespace L2(m) of all square integrable functions with respect to a probability measure m. Since thisspace is Polish, it is a Hilbert space.

Let L be a Hilbert space with inner product (f, g) and norm ||f || = (f, f)1/2. For the case whereL = L2(m), the inner product is given by

(f, g) = Em(fg).

A (measurable) subset M of L will be called a subspace of L if f , g M implies that af + bg Mfor all scalar multipliers a, b. We will be primarily interested in closed subspaces of L. Recall fromLemma 3.2.4 that a closed subset of a Polish space is also Polish with the same metric, and henceM is a complete space with the same norm.

A fundamentally important property of linear spaces with inner products is the parallelogramlaw

||f + g||2 + ||f g||2 = 2||f ||2 + 2||g||2. (5.7)

This is proved simply by expanding the left-hand side as

(f, f) + 2(f, g) + (g, g) + (f, f) 2(f, g) + (g, g).

A simple application of this formula and the completeness of a subspace yield the following result:given a fixed element f in a Hilbert space and given a closed subspace of the Hilbert space, we canalways find a unique member of the subspace that provides the stpproximation to f in the sense ofminimizing the norm of the difference.

5.5. PROJECTIONS 103

Lemma 5.5.1 Fix an element f in a Hilbert space L and let M be a closed subspace of L. Thenthere is a unique element f M such that

||f f || = infgM

||f g||, (5.8)

where f and g are considered identical if ||f g|| = 0. Thus, in particular, the infimum is actuallya minimum.

Proof: Let denote the infimum of (5.8) and choose a sequence of fn M so that ||f fn|| .From the parallelogram law

||fn fm||2 = 2||fn f ||2 + 2||fm f ||2 4||fn + fm

2 f ||2.

Since M is a subspace, (fn + fm)/2 M and hence the latter norm squared is bound below by 2.This means, however, that

limn,m

||fn fm||2 22 + 22 42 = 0,

and hence fn is a Cauchy sequence. Since M is complete, it must have a limit, say f . From thetriangle inequality

||f f || ||f fn|| + ||fn f || n ,

which proves that there is an f in M with the desired property. Suppose next that there are twosuch functions, e.g., f and g. Again invoking (5.7)

||f g||2 2||f f ||2 + 2||g f ||2 4|| f + g2

f ||2.

Since (f + g)/2 M , the latter norm can be no smaller than , and hence

||f g||2 22 + 22 42 = 0,

and therefore ||f g|| = 0 and the two are identical. 2Two elements f and g of L will be called orthogonal and we write f g if (f, g) = 0. If M L,

we say that f L is orthogonal to M if f is orthogonal to every g M . The collection of all f Lwhich are orthogonal to M is called the orthogonal complement of M .

Lemma 5.5.2 Given the assumptions of Lemma 5.5.1, f will yield the infimum if and only iff f M ; that is, f is chosen so that the error f f is orthogonal to the subspace M .

Proof: First we prove necessity. Suppose that f f is not orthogonal to M . We will show that fcannot then yield the infimum of Lemma 5.5.1. By assumption there is a g M such that g is notorthogonal to f f . We can assume that ||g|| = 1 (otherwise just normalize g by dividing by itsnorm). Define a = (f f , g) 6= 0. Now set h = f + ag. Clearly h M and we now show that it is abetter estimate of f than f is. To see this write

||f h||2 = ||f f ag||2

= ||f f ||2 + a2||g||2 2a(f f , g)= ||f f ||2 a2 < ||f f ||2

104 CHAPTER 5. CONDITIONAL PROBABILITY AND EXPECTATION

as claimed. Thus orthogonality is necessary for the minimum norm estimate. To see that it issufficient, suppose that f M . Let g be any other element in M . Then

||f g||2 = ||f f + f g||2

= ||f f ||2 + ||f g||2 2(f f , f g).

Since f g M , the inner product is 0 by assumption, proving that ||f g|| is strictly greater than||f f || if ||f g|| is not identically 0. 2

Combining the two lemmas we have the following famous result.

Theorem 5.5.1 (The Projection Theorem)Suppose that M is a closed subspace of a Hilbert space L and that f L. Then there is a unique

f M with the properties thatf f M

and||f f || = inf

gM||f g||.

The resulting f is called the projection of f onto M and will be denoted PM (f).

Exercises

1. Show that projections satisfy the following properties:

(a) If f M , then PM (f) = f .(b) For any scalar a PM (af) = aPM (f).(c) For any f , g L and scalars a, b:

PM (af + bg) = aPM (f) + bPM (g).

2. Consider the special case of L2(m) of all square-integrable functions with respect to a prob-ability measure m. Let P be the projection with respect to a closed subspace M . Prove thefollowing:

(a) If f is a constant, say c, then P(f) = c.(b) If f 0 with probability one, then P(f) 0.

3. Suppose that g is a discrete measurement and hence is in L2(m). Let M be the space of allsquare-integrable (g)-measurable functions. Assume that M is complete. Given a discretemeasurement f , let E(f |g) denote the conditional expectation of exercise 5.4.1. Show thatE(f |g) = PM (f), the projection of f onto M .

4. Suppose that H M are closed subspaces of a Hilbert space L. Show that for any f L,

||f PM (f)|| ||f PH(f)||.

5.6. THE RADON-NIKODYM THEOREM 105

5.6 The Radon-Nikodym Theorem

This section is devoted to the statement and proof of the Radon-Nikodym theorem, one of thefundamental results of measure and integration theory and a key result in the proof of the existenceof conditional probability measures.

A measure m is said to be absolutely continuous with respect to another measure P on the samemeasurable space and we write m

106 CHAPTER 5. CONDITIONAL PROBABILITY AND EXPECTATION

The first term on the right is 0 since g M and f r(Emf)/(Emr) has zero expectation withrespect to m and hence is in M. Thus

fgdq =Emf

Emr

rgdq =

Emf

Emr

Emr||r||2||r||2 = Emf

which proves (5.11). Observe that if we set f = 1F for any event F , then

Em1F = m(F ) =

F

gdq,

a formula which resembles the desired formula (5.9) except that the integral is with respect to qinstead of m. From the definition of q 0 m(F ) 2q(F ) and hence

0

F

gdq

F

2dq, all events F.

From Lemma 4.4.7 this implies that g [0, 2] q-a.e.. Since m