Stat 411 – Homework 07 Solutions

1. The distribution N(θ, 1) is a regular one-parameter exponential family problemwith K(x) = x. Therefore, T =

∑ni=1Xi is a complete sufficient statistic for θ and,

consequently, the MVUE of θ is X = T/n. It is easy to check that η̂ = X2 − 1/n

is an unbiased estimator of η = θ2. Since it’s a function of the complete sufficientstatistic, by the Lehmann–Scheffe theorem, it must be the MVUE.

2. Problem 7.5.3 from HMC6–7.

(a) The beta distribution with PDF fθ(x) = θxθ−1, with x ∈ (0, 1) and θ > 0, isa regular exponential family. That is,

fθ(x) = exp{log θ + (θ − 1) log x}.

Since K(x) = log x here, a complete sufficient statistic, based on X1, . . . , Xniid∼

fθ(x), is T0 =∑n

i=1 logXi. So too is T = exp{T0/n} = (X1X2 · · ·Xn)1/n, thegeometric mean, since the function t 7→ et/n is one-to-one.

(b) The log-likelihood function `(θ) =∑n

i=1 log fθ(Xi) = n log θ+(θ−1)∑n

i=1 logXi.Differentiating and setting equal to zero gives the likelihood equation:

n

θ+

n∑i=1

logXi = 0.

Therefore, the MLE is θ̂ = −n/∑n

i=1 logXi = −1/ log T , a function of thegeometric mean T from part (a).

3. The moment-generating function MX(t) of X is given by

MX(t) = Eθ(etX) =

∫etxeθx+S(x)+q(θ) dx

=

∫eq(θ)−q(θ+t)e(θ+t)x+S(x)+q(θ+t) dx

= eq(θ)−q(θ+t)∫e(θ+t)x+S(x)+q(θ+t) dx

= eq(θ)−q(θ+t),

where the last inequality is because the integrand is the exponential family PDFwith parameter “θ + t” instead of “θ.”

4. Problem 7.5.13 in HMC6 = Problem 7.5.12 in HMC7.

(a) The log-likelihood looks like

`(θ) =n∑i=1

[Xi log θ + log(1− θ)

]= (log θ)

n∑i=1

Xi + n log(1− θ).

1

Differentiating and setting equal to zero gives the equation

1

θ

n∑i=1

Xi −n

1− θ= 0.

The solution to this is the MLE: θ̂ = T/(n+ T ), where T =∑n

i=1Xi.

(b) The distribution is a member of the regular exponential family:

fθ(x) = θx(1− θ) = exp{(log θ)x+ log(1− θ)}.

Therefore, by Theorem 7.5.2, T =∑n

i=1Xi is a complete sufficient statistic.

(c) To find the MVUE of θ, we need a function g(T ) that’s unbiased; by theLehmann–Scheffe theorem, this g(T ) must be the MVUE. To start, recall thatT (a sum of independent geometric RVs) has a negative binomial distribution.1

That is, the PMF of T is

fT,θ(t) =

(n+ t− 1

n− 1

)θt(1− θ)n, t = 0, 1, 2, . . .

Since the MLE θ̂ = T/(n + T ) is a reasonable choice, let’s first try g(T ) =T/(n+ T ). The expected value looks like:

Eθ[T/(n+ T )] =∞∑t=0

t

n+ t

(n+ t− 1

n− 1

)θt(1− θ)n.

There’s some potential for cancellation if we instead take g(T ) = T/(n+T−1).In this case,

Eθ[T/(n+ T − 1)] =∞∑t=0

t

n+ t− 1

(n+ t− 1

n− 1

)θt(1− θ)n

=∞∑t=0

t

n+ t− 1

(n+ t− 1)!

t!(n− 1)!θt(1− θ)n

= θ

∞∑t=1

(n+ t− 2)!

(t− 1)!(n− 1)!θt−1(1− θ)n

= θ

∞∑u=0

(n+ u− 1

n− 1

)θu(1− θ)n

= θ.

The next-to-last line follows from a change of variable: u = t− 1. Therefore,T/(n+ T − 1) is an unbiased estimator of θ; moreover, since it’s a function ofthe complete sufficient statistic, it must be the MVUE.

1Each geometric RV Xi counts the number of trials until the first success; T =∑n

i=1 Xi counts thenumber of trials until the n-th success.

2

5. (Graduate only)

(a) The estimator m̂θ = 1n

∑ni=1K(Xi) is unbiased for mθ = Eθ[K(X1)] by defini-

tion. Since it’s a function of the complete sufficient statistic T =∑n

i=1K(Xi)it must be the MVUE according to Lehmann–Scheffe.

(b) Here I prove that m̂θ is the MVUE of mθ by showing that its variance is equalto the Cramer–Rao lower bound.

• Observe that Vθ(m̂θ) = Vθ(T/n) = Vθ(T )/n2. By Theorem 7.5.1(3),

Vθ(T ) = n{p′′(θ)q′(θ)− p′(θ)q′′(θ)}/p′(θ)3,

which implies

Vθ(m̂θ) =p′′(θ)q′(θ)− p′(θ)q′′(θ)

np′(θ)3.

• For the Cramer–Rao lower bound, I first need the Fisher information. Thesecond derivative of log fθ(x) is

(∂2/∂θ2) log fθ(x) = p′′(θ)K(x) + q′′(θ).

Therefore,

I(θ) = −Eθ[ ∂2∂θ2

log fθ(X1)]

= −p′′(θ)Eθ[K(X1)]− q′′(θ) =p′′(θ)q′(θ)− p′(θ)q′′(θ)

p′(θ),

where the last equality uses the result in Theorem 7.5.1(2) to expressEθ[K(X1)] in terms of p and q. Since the estimand is mθ, not just θ, theCramer–Rao lower bound for estimating m(θ) involves the derivative ofm(θ). Since mθ = Eθ[K(X1)] = −q′(θ)/p′(θ) from Theorem 7.5.1(2), itsderivative with respect to θ is

m′θ =p′′(θ)q′(θ)− p′(θ)q′′(θ)

p′(θ)2.

Therefore, the Cramer–Rao lower bound is

LB =(m′θ)

2

nI(θ)=p′′(θ)q′(θ)− p′(θ)q′′(θ)

np′(θ)3.

Since the Cramer–Rao lower bound is the same as Vθ(m̂θ), there’s no otherestimator with smaller variance, i.e., it’s the MVUE.

This result in this exercise showed that the Cramer–Rao lower bound is at-tained in an exponential family problem. There is a partial converse to thisresult which says, roughly, if the Cramer–Rao lower bound is attained, then theproblem must be an exponential family type—very interesting! See Lehmannand Casella (1998, p. 121) for the precise statement of this result.

3