Statistical Theory 550.630

Solution to HW4

PROBLEM 1. We are interested in finding the maximum likelihood estimator for the parameters in thegross-error model

F = {fθ = (1− ε)ϕ(·;µ, ν) + εϕ(·;µ, cν) : θ = [ε, µ, ν, c] ∈ (0, 1)× (−∞,∞)× (0,∞)× (1,∞)}. (1)

Suppose we have a sample X1, ..., Xniid∼ fθ ∈ F . Let’s look at its log-likelihood

l(θ) = log

n∏i=1

fθ(xi) =

n∑i=1

log(fθ(xi)) =

n∑i=1

log((1− ε)ϕ(xi;µ, ν) + εϕ(xi;µ, cν)

=

n∑i=1

log

((1− ε)√

2πνe−(xi−µ)2/2ν +

ε√2πcν

e−(xi−µ)2/2cν). (2)

Usually the procedure to find the MLE would be to look for extremal points of this function in the interior ofits domain (by setting the gradient to zero) and also analyzing the behavior of the function at the boundary.We will do the second first. We observe that it is easy to drive one of the terms in the above sum to infinityby setting the mean equal one of the observed points, and letting the variance go to zero. So let µ̂ = x1 andlet νk = 1/k. Then clearly we have for any ε and c

limk→∞

log(f(x1; ε, µ̂, νk, c)) = limk→∞

log((1− ε)ϕ(x1; µ̂, νk) + εϕ(x1; µ̂, cνk)) =

= limk→∞

log

(1− ε√2πνk

+ε√

2πcνk

)=∞ (3)

To analyze the behavior of the remaining terms we also set ck = k. Thus the variance of the secondcomponent stays equal to one. We have for i = 2, ..., n

limk→∞

log(f(xi; ε, µ̂, νk, ck)) = limk→∞

log((1− ε)ϕ(x1; µ̂, νk) + εϕ(x1; µ̂, ckνk))

= limk→∞

log

((1− ε)√

2πνke−(xi−x1)

2/2νk +ε√2πe−(xi−x1)

2/2

)> lim

k→∞log

(ε√2πe−(xi−x1)

2/2

)= const > 0, (4)

where the inequality follows from the monotonicity of the logarithm function. Now we can put the resultsof (3) and (4) together to obtain

limk→∞

l(ε, µ̂, νk, ck) = limk→∞

[log(f(x1; ε, µ̂, νk, ck)) +

n∑i=2

log f(xi; ε, µ̂, νk, ck)

]=∞+ n× const =∞. (5)

Thus we have obtained a subsequence in the domain of the parameters for which the log-likelihood valuesincrease unboundedly. Hence, the MLE does not exist.

PROBLEM 2. Let Xiiid∼ Exp(λi), i = 1, ..., 5;λi = e−a−ib, θ = [a, b] ∈ R2. Denote the family of distributions

of this sample as Q. We observe that for general λi > 0 the sample comes from an exponential familyof exponentially distributed random variables. However, the additional constraints turn it into a curvedexponential family.

We first show that the family of distributions P of Xiiid∼ Exp(λi), i = 1, ..., 5;λi > 0 for i = 1, ..., n is

exponential. The form of the joint density is

p(x, λ) =

n∏i=1

λie−λixi = e−

∑5i=1 λixi+

∑5i=1 log(λi). (6)

1

Therefore, P is an exponential family with a sufficient statistic T (X) = [−x1, ..,−x5]T . The normalization

factor A(λ) = −∑5i=1 log(λi) is finite when each λi > 0 so the natural parameter space is E = (R+)5 (which

is clearly open). We have A′(λ) = −[1/λ1, ..., 1/λ5], and hence A′′(λ) is a diagonal matrix with diagonalentries 1/λ21, ..., 1/λ

25. Since each λi > 0, these entries are positive and A′′(λ) is positive definite on E .

Therefore, we have justified that A is strictly convex on E . By Theorem 1.6.4 we conclude that P is of rank5. This fact, together with the openness of E and the continuity of the density, is sufficient to show that theMLE for λ exists with probability 1 (by Theorem 2.3.2).

Now let’s consider the form of the density of Q

p(x; a, b) =

5∏i=1

e−a−ibe−e−a−ibxi , a, b ∈ R. (7)

If we define the function c : R2 → E , such that

c(a, b) = [e−a−b, e−a−2b, e−a−3b, e−a−4b, e−a−5b]T , (8)

we can rewrite p(x; a, b) as

p(x, a, b) = ec(a,b)TT (x)−A(c(a,b)). (9)

We further decompose c into c(a, b) = exp(g(a, b)), where g : R2 → R5 is

g(a, b) = [−a− b,−a− 2b,−a,−3b,−a− 4b,−a− 5b]T . (10)

Clearly g is a linear map so it maps closed sets into closed sets, i.e. g(R2) is closed. On the other hand theexponential function is invertible with continuous inverse when considered as a function from R5 to E , soit also maps closed sets of R5 into closed sets of E . Therefore, c(R2) = exp(g(R2)) is closed in E . Thus byTheorem 2.3.3, the MLE for Q also exists with probability 1, and satisfies

c′T (a, b)(t0 −A′(c(a, b))) = 0 (11)

⇒

−e−a−b −e−a−b−e−a−2b −2e−a−2b

−e−a−3b −3e−a−3b

−e−a−4b −4e−a−4b

−e−a−5b −5e−a−5b

T

−x1 + ea+b

−x2 + ea+2b

−x3 + ea+3b

−x4 + ea+4b

−x5 + ea+5b

= 0

⇒e−a

∑ni=1 xie

−ib = 5,

e−a∑ni=1 ixie

−ib = 15;

⇒

∑ni=1 xi(3− i)e−ib = 0,

a = log(∑ni=1 xie

−ib/5).

We observe that left hand side of the first equation in the system above is a polynomial in e−b so it can beexplicitly solved for b and consequently the second equation can be solved for a.

2