Problem Solving

ECE 313

April 30, 2017

Problem Solving

Correlation and Covariance

ECE 313 | Section B 2/51

Problem Solving

Definition

Let X and Y be random variables with finite second moments.

the correlation: E[XY ]

ECE 313 | Section B 3/51

Problem Solving

Definition

Let X and Y be random variables with finite second moments.

the correlation: E[XY ]the covariance: Cov(X,Y ) = E[(X − E[X])(Y − E[Y ])]

ECE 313 | Section B 4/51

Problem Solving

Definition

Let X and Y be random variables with finite second moments.

the correlation: E[XY ]the covariance: Cov(X,Y ) = E[(X − E[X])(Y − E[Y ])]

the correlation coefficient: ρX,Y = Cov(X,Y )√Var(X)Var(Y )

= Cov(X,Y )σXσY

.

ECE 313 | Section B 5/51

Problem Solving

Covariance generalizes variance

Var(X) = Cov(X,X).

ECE 313 | Section B 6/51

Problem Solving

Covariance generalizes variance

Var(X) = Cov(X,X).

Shortcut for computing variance:

Var(X) = E[X(X − E[X])] = E[X2]− E[X]2.

ECE 313 | Section B 7/51

Problem Solving

Covariance generalizes variance

Var(X) = Cov(X,X).

Shortcut for computing variance:

Var(X) = E[X(X − E[X])] = E[X2]− E[X]2.

Similar shortcuts exist for computing covariances:

Cov(X,Y ) = E[X(Y−E[Y ])] = E[(X−E[X])Y ] = E[XY ]−E[X]E[Y ].

In particular, if either X or Y has mean zero, then E[XY ] = Cov(X,Y ).

ECE 313 | Section B 8/51

Problem Solving

Correlated

If Cov(X,Y ) = 0, X and Y are uncorrelated.

If Cov(X,Y ) > 0, X and Y are positively correlated.

If Cov(X,Y ) < 0, X and Y are negatively correlated.

ρX,Y is a scaled version of Cov(X,Y ).

ECE 313 | Section B 9/51

Problem Solving

Uncorrelated vs. Independent

If Cov(X,Y ) = 0, X and Y are uncorrelated.

Cov(X,Y ) = 0 is equivalent to E[XY ] = E[X]E[Y ].

ECE 313 | Section B 10/51

Problem Solving

Uncorrelated vs. Independent

If Cov(X,Y ) = 0, X and Y are uncorrelated.

Cov(X,Y ) = 0 is equivalent to E[XY ] = E[X]E[Y ].

Does independence of X and Y imply E[XY ] = E[X]E[Y ]?

ECE 313 | Section B 11/51

Problem Solving

Uncorrelated vs. Independent

If Cov(X,Y ) = 0, X and Y are uncorrelated.

Cov(X,Y ) = 0 is equivalent to E[XY ] = E[X]E[Y ].

Does independence of X and Y imply E[XY ] = E[X]E[Y ]?

Yes, independence V uncorrelated.

ECE 313 | Section B 12/51

Problem Solving

Uncorrelated vs. Independent

Does uncorrelated V independence?

ECE 313 | Section B 13/51

Problem Solving

Uncorrelated vs. Independent

Does uncorrelated V independence?

No, independence is a much stronger condition.

Independence requires a larger number of equations to hold, namelyFXY (u, v) = FX(u)FY (v) for every real value of u and v. Uncorrelatedonly requires a single equation to hold.

ECE 313 | Section B 14/51

Problem Solving

Uncorrelated vs. Independent

Correlation is a pairwise property.

A set of random variables being uncorrelated is the same as being pairwiseuncorrelated.

ECE 313 | Section B 15/51

Problem Solving

Uncorrelated vs. Independent

Correlation is a pairwise property.

A set of random variables being uncorrelated is the same as being pairwiseuncorrelated.

In contrast, mutual independence of 3 or more random variables is astronger property than pairwise independence.

ECE 313 | Section B 16/51

Problem Solving

Play with Covariance

ECE 313 | Section B 17/51

Problem Solving

Linearity

Covariance is linear in each of its two arguments:

Cov(X + Y, U + V ) = Cov(X,U) + Cov(X,V ) + Cov(Y,U) + Cov(Y, V )Cov(aX + b, cY + d) = acCov(X,Y ).

for constants a, b, c, d.

Recall thatVar(aX + b) = a2Var(X).

ECE 313 | Section B 18/51

Problem Solving

Variance of sum of r.v.

The variance of the sum of uncorrelated random variables is equal to thesum of the variances of the random variables. For example, if X and Yare uncorrelated,

Var(X + Y ) = Var(X) + Var(Y ),

ECE 313 | Section B 19/51

Problem Solving

Variance of sum of r.v.

The variance of the sum of uncorrelated random variables is equal to thesum of the variances of the random variables. For example, if X and Yare uncorrelated,

Var(X+Y ) = Cov(X+Y,X+Y ) = Cov(X,X)+Cov(Y, Y )+2Cov(X,Y )

= Var(X) + Var(Y ).

ECE 313 | Section B 20/51

Problem Solving

Variance of sum of r.v.

Consider the sum Sn = X1 + · · ·+Xn, such that X1, · · · , Xn are uncor-related (so Cov(Xi, Xj) = 0 if i 6= j) with E[Xi] = µ and Var(Xi) = σ2

for 1 ≤ i ≤ n.

Find E(Sn) and Var(Sn).

ECE 313 | Section B 21/51

Problem Solving

Variance of sum of r.v.

E[Sn] = nµ (1)

ECE 313 | Section B 22/51

Problem Solving

and

Var(Sn) = Cov(Sn, Sn) = Cov

n∑i=1

Xi,n∑j=1

Xj

=

n∑i=1

n∑j=1

Cov(Xi, Xj)

=n∑i=1

Cov(Xi, Xi) +∑i,j:i 6=j

Cov(Xi, Xj)

=n∑i=1

Var(Xi) + 0 = nσ2. (2)

ECE 313 | Section B 23/51

Problem Solving

Practice!

Simplify the following expressions:(a) Cov(8X+3, 5Y −2), (b) Cov(10X−5,−3X+15), (c) Cov(X+2,10X-3Y), (d) ρ10X,Y+4.

ECE 313 | Section B 24/51

Problem Solving

Correlation coefficient

ECE 313 | Section B 25/51

Problem Solving

Correlation coefficient

ρX,Y = Cov(X,Y )√Var(X)Var(Y )

= Cov(X,Y )σXσY

.

ρX,Y is a scaled version of Cov(X,Y ).

The situation is similar to the standardized version of X and Y .

FindCov

(X − E[X]

σX,Y − E[Y ]

σY

).

ECE 313 | Section B 26/51

Problem Solving

Correlation coefficient

Cov(X − E[X]

σX,Y − E[Y ]

σY

)= Cov

(X

σX,Y

σY

)= Cov(X,Y )

σXσY= ρX,Y .

ECE 313 | Section B 27/51

Problem Solving

Correlation coefficient

FindρaX+b,cY+d

ECE 313 | Section B 28/51

Problem Solving

Correlation coefficient

ρaX+b,cY+d = ρX,Y for a, c > 0.

ECE 313 | Section B 29/51

Problem Solving

Correlation coefficient

• |ρX,Y | ≤ 1,

• ρX,Y = 1 if and only if Y = aX + b for some a, b with a > 0, and

• ρX,Y = −1 if and only if Y = aX + b for some a, b with a < 0.

ECE 313 | Section B 30/51

Problem Solving

Unbiased Estimator

Suppose X1, . . . , Xn are independent and identically distributed randomvariables, with mean µ and variance σ2. We estimate µ and σ2 by thesample mean and sample variancedefined as follows:

X = 1n

n∑k=1

Xk σ2 = 1n− 1

n∑k=1

(Xk − X)2.

Note the perhaps unexpected appearance of n− 1 in the sample variance.Of course, we should have n ≥ 2 to estimate the variance (assuming wedon’t know the mean) so it is not surprising that the formula is not definedif n = 1. An estimator is called unbiased if the mean of the estimator isequal to the parameter that is being estimated.

ECE 313 | Section B 31/51

Problem Solving

Q1: Why don’t we use ML parameter estimation for µ and σ2?Q2: Why is σ2 undefined for n = 1?

(a) Is the sample mean an unbiased estimator of µ?(b) Find the mean square error, E[(µ− X)2], for estimation of the meanby the sample mean.(c) Is the sample variance an unbiased estimator of σ2?

ECE 313 | Section B 32/51

Problem Solving

Minimum mean square error estimation

ECE 313 | Section B 33/51

Problem Solving

Constant estimator

Let Y be a random variable with some known distribution. Suppose Y isnot observed but that we wish to estimate Y .

• We use a constant δ to estimate Y .

ECE 313 | Section B 34/51

Problem Solving

Constant estimator

Let Y be a random variable with some known distribution. Suppose Y isnot observed but that we wish to estimate Y .

• We use a constant δ to estimate Y .

• The mean square error (MSE) for estimating Y by δ is defined by

E[(Y − δ)2].

Q. How do we find a δ that minimizes E[(Y − δ)2]? What is the resultingMSE?

ECE 313 | Section B 35/51

Problem Solving

Unconstrained Estimator

We want to estimate Y . We have an observation X. The joint distributionis fX,Y .

• We use the estimator g(X) for some function g.

• The resulting mean square error (MSE) is E[(Y − g(X))2]

Q. What is the function g that minimizes the MSE? What is the resultingMSE?

ECE 313 | Section B 36/51

Problem Solving

Unconstrained Estimator

Suppose you observe X = 10. What do you know about Y ?

ECE 313 | Section B 37/51

Problem Solving

Unconstrained Estimator

Suppose you observe X = 10. What do you know about Y ?

You can derive the conditional pdf of Y given X = 10, denoted byfY |X(v|10).

Which value of Y should you pick?

ECE 313 | Section B 38/51

Problem Solving

Unconstrained EstimatorSuppose you observe X = 10. What do you know about Y ?

You can derive the conditional pdf of Y given X = 10, denoted byfY |X(v|10).

Which value of Y should you pick?

Based on the fact, discussed above, that the minimum MSE constantestimator for a random variable is its mean, it makes sense to estimate Yby the conditional mean:

E[Y |X = 10] =∫ ∞−∞

vfY |X(v|10)dv.

ECE 313 | Section B 39/51

Problem Solving

Unconstrained Estimator

In general, we can show that

g∗(u) = E[Y |X = u] =∫ ∞−∞

vfY |X(v|u)dv.

The minimum MSE is

MSE = E[Y 2]− E[(E[Y |X])2], (3)= Var(Y )− Var(Y |X) (4)

ECE 313 | Section B 40/51

Problem Solving

Linear estimator

• In practice it is not always possible to compute g∗(u).

• The conditional density fY |X(v|u) may not be available or might bedifficult to compute.

• Worse, there might not even be a good way to decide what jointpdf fX,Y to use in the first place.

A reasonable alternative is to consider linear estimators of Y given X.

ECE 313 | Section B 41/51

Problem Solving

Linear estimator

• We use a linear estimator L(X) = aX + b. We only need to find aand b.

• The resulting mean square error (MSE) is E[(Y − (aX + b))2].

Q. What are the a and b that minimize the MSE? What is the resultingMSE?

ECE 313 | Section B 42/51

Problem Solving

Linear estimator

The minimum MSE linear estimator is given by L∗(X) = E[Y |X], where

E[Y |X] = µY +(Cov(Y,X)

Var(X)

)(X − µX)

= µY + σY ρX,Y

(X − µXσX

).

minimum MSE for linear estimation = σ2Y−

(Cov(X,Y ))2

Var(X) = σ2Y (1−ρ2

X,Y ).

ECE 313 | Section B 43/51

Problem Solving

If X and Y are standard, then E[Y |X] = ρX,YX and the MSE is 1−ρ2X,Y .

ECE 313 | Section B 44/51

Problem Solving

Three estimators

• Constant

• Unconstrained

• Linear

Which is the best?

ECE 313 | Section B 45/51

Problem Solving

Three estimators

Constant (estimator) ⊂ Linear ⊂ Unconstrained

E[(Y − g∗(X))2]︸ ︷︷ ︸MSE for g∗(X)=E[Y |X]

≤ σ2Y (1− ρ2

X,Y )︸ ︷︷ ︸MSE for L∗(X)=E[Y |X]

≤ σ2Y︸︷︷︸

MSE for δ∗=E[Y ].

(5)

ECE 313 | Section B 46/51

Problem Solving

Three estimators

All three estimators are linear as functions of the variable to be estimated:

E[aY + bZ + c] = aE[Y ] + bE[Z] + c

E[aY + bZ + c|X] = aE[Y |X] + bE[Z|X] + c

E[aY + bZ + c|X] = aE[Y |X] + bE[Z|X] + c

ECE 313 | Section B 47/51

Problem Solving

Noisy observation

Let X = Y +N , where Y has the exponential distribution with parameterλ, and N is Gaussian with mean 0 and variance σ2

N . Suppose the variablesY and N are independent, and the parameters λ and σ2

N are known andstrictly positive. (Recall that E[Y ] = 1

λ and Var(Y ) = σ2Y = 1

λ2 .)(a) Find E[Y |X], the MSE linear estimator of Y given X, and also findthe resulting MSE.(b) Find an unconstrained estimator of Y yielding a strictly smaller MSEthan E[Y |X] does.

ECE 313 | Section B 48/51

Problem Solving

Uniform distribution

Suppose (X,Y ) is uniformly distributed over the triangular region withvertices at (−1, 0), (0, 1), and (1, 1), shown in Figure 1. (a) Find and sketch

1

u

v

!1 10

Figure 1: Support of fX,Y .

ECE 313 | Section B 49/51

Problem Solving

the minimum MSE estimator of Y given X = u, g∗(u) = E[Y |X = u], forall u such that it is well defined, and find the resulting minimum MSE forusing g∗(X) = E[Y |X] to estimate Y. (b) Find and sketch the functionE[Y |X = u], used for minimum MSE linear estimation of Y from X, andfind the resulting MSE for using E[Y |X] to estimate Y.

ECE 313 | Section B 50/51

Questions?