Lecture 12 -- Another way to find the Best Estimator

1. (Regular) Exponential Family

The density function of a regular exponential family is:

Example. Poisson(θ)

Example. Normal. (both unknown).

2. Theorem (Exponential family & sufficient Statistic). Let be a random sample from the regular exponential family.

Then

is sufficient for

Example. Poisson(θ)

Let be a random sample from Poisson(θ)

Then

is sufficient for

Example. Normal. (both unknown).

Let be a random sample from

Then

is sufficient for

Exercise.

Apply the general exponential family result to all the standard families discussed above such as binomial, Poisson, normal, exponential, gamma.

A Non-Exponential Family Example.

Discrete uniform.

is a positive integer.

Another Non-exponential Example.

iid

Universal Cases.

are iid with density .

• The original data are always sufficient for .

(They are trivial statistics, since they do not lead any data reduction)

• Order statistics are always sufficient for .

( The dimension of order statistics is , the same as the dimension of the data. Still this is a nontrivial reduction as ! different values of data corresponds to one value of . )

3. Theorem (Rao-Blackwell)

Let be a random sample from the population with pdf . Let be a sufficient statistic for θ, and be any unbiased estimator of θ.

Let , then

(1) is an unbiased estimator of

(2) is a function of T,

(3) for every , and for some unless with probability 1 .

Rao-Blackwell theorem tells us that in searching for an unbiased estimator with the smallest possible variance (i.e., the best estimator, also called the uniformly minimum variance unbiased estimator – UMVUE, which is also referred to as simply the MVUE), we can restrict our search to only unbiased functions of the sufficient statistic T(X).

Proof: Make use of the following equations:

Note: The fact that is a sufficient statistic for θ will ensure that is a function of only the sample and in particular, is independent of θ.

4. Transformation of Sufficient Statistics

1. If is sufficient for and a mathematical function of some other statistic, then is also sufficient.

2. If is sufficient for , and with being one-to-one, then is also sufficient.

Remark: When one statistic is a function of the other statistic and vise verse, then they carry exactly the same amount of information.

Examples:

• If is sufficient, so is .

• If ( are sufficient, so is .

• If is sufficient, so is is sufficient, and so is .

Examples of non-sufficiency.

Ex. iid Poisson(). is not sufficient.

Ex. iid pmf . is not sufficient.

5. Minimal Sufficient Statistics

It is seen that different sufficient statistics are possible. Which one is the "best"? Naturally, the one with the maximum reduction.

• For , is a better sufficient statistic for than

Definition:

is a minimal sufficient statistic if, given any other sufficient statistic, there is a function such that .

Equivalently, is minimal sufficient if, given any other sufficient statistic whenever and are two data values such that , then .

Partition Interpretation for Minimal Sufficient Statistics:

• Any sufficient statistic introduces a partition on the sample space.

• The partition of a minimal sufficient statistic is the coarsest.

• Minimal sufficient statistic has the smallest dimension among possible sufficient statistics. Often the dimension is equal to the number of free parameters (exceptions do exist).

Theorem (How to check minimal sufficiency).

A statistic T is minimal sufficient if the following property holds: For any two sample points x and y does not depend on (i.e. is a constant function of ) if and only if

6. Exponential Families & Minimal Sufficient Statistic:

For a random sample from the regular exponential family with probability density ], where is k dimensional, the statistic

is minimal sufficient for .

Example. Poisson(θ)

Let be a random sample from Poisson(θ)

Then

is minimal sufficient for

Example. Normal. (both unknown).

Let be a random sample from

Then

Is minimal sufficient for

Remarks:

• Minimal sufficient statistic is not unique. Any two are in one-to-one correspondence, so are equivalent.

7. Complete Statistics

Let a parametric family be given. Let be a statistic. Induced family of distributions .

A statistic is complete for the family or equivalently, the induced family is called complete, if for all implies that with probability 1.

Example. Poisson(θ)

Let be a random sample from Poisson(θ)

Then

is minimal sufficient for Now we show that T is also complete.

We know that

Consider any function . We have

Because setting requires all the coefficient

to be zero, which implies

Example. Let be iid from . Show is a complete statistic. (*Please read our text book for more examples – but the following result on the regular exponential family is the most important.)

8. Exponential Families & Complete Statistics

Theorem. Let be iid observations from the regular exponential family, with the pdf ],

and Then

is complete if the parameter space contains an open set in .

(This is only a sufficient condition, not a necessary condition)

Example. ∼

Example. Poisson(θ);

Example. Normal. (both unknown).

Example. ∼ , is not complete.

Example. ∼ is complete.

Example. {Bin(2,), p = 1/2, p = 1/4} is not complete.

Example. The family {Bin(2,p), 0 < p < 1} is complete.

Properties of the Complete Statistics

(i) If is complete and , then is also complete.

(ii) If a statistic T is complete and sufficient, then any minimal sufficient statistic is complete.

(iii) Trivial (constant) statistics are complete for any family.

9. Theorem (Lehmann-Scheffe). (Complete Sufficient Statistic and the Best Estimator)

If T is complete and sufficient, then is the Best Estimator (also called UMVUE or MVUE) of its expectation.

Example. Poisson(θ)

Let be a random sample from Poisson(θ)

Then

is complete sufficient for Since

is an unbiased estimator of θ – by the Lehmann-Scheffe theorem we know that U is a best estimator (UMVUE/MVUE) for θ.

Example. Let , be a random sample from the normal population where is assumed known. Please derive

(a)

The maximum likelihood estimator for.

(b)

Is the above MLE forunbiased?

(c)

Is the MLE a best estimator (UMVUE) for?

Solution:

(a) The likelihood function is

The log likelihood function is

Solving

We obtain the MLE for

(b) Since

It is straight-forward to verify that the MLE

is an unbiased estimator for .

(c) Now we calculate the Cramer-Lower bound for the variance of an unbiased estimator for

Thus the Cramer-Rao lower bound is:

Therefore we claim that the MLE is an efficient estimator for . Since the regularity conditions for the Cramer-Rao lower bound Theorem to be true holds here, we declare the MLE is also a best estimator (UMVUE) for .

(c) Alternatively, we can derive that the MLE is the best estimator directly (rather than using the efficient estimator is also a best estimator argument) as follows: The population pdf is:

So it is a regular exponential family, where the red part is and the green part is .

Thus is a complete & sufficient statistic (CSS) for .

It is easy to see that the MLE for is a function of the complete & sufficient statistic as follows:

In addition, we know that the MLE is an unbiased estimator for as we have done in part (b):

Since the MLE is an unbiased estimator and a function of the CSS, therefore we claim that is the best estimator (UMVUE) for by the Lehmann-Scheffe Theorem.

10. Theorem (Basu)

A complete sufficient statistic T for the parameter θ is independent of any ancillary statistic – that is, a statistic whose distribution does not depend on θ

Example. Consider a random sample of size n from a normal distribution .

Consider the MLEs

It is easy to verify that is a complete sufficient statistic for for fixed values of . Also:

which does not depend on . It follows from the Basu Theorem that the two MLEs are independent to each other.

Homework 6:

Problems in our textbook:

5.1, 5.3, 5.5, 5.6, 5.16, 5.21, 5.22, 5.29, 5.30

6.3, 6.6, 6.17, 6.22, 6.30, 7.49, 7.50, 7.52 (a&b), 7.59, 7.60

Read our textbook:

Chapter 5: Sections 5.1, 5.2, 5.3 & 5.4

Chapter 6: Sections 6.1 & 6.2

Chapter 7: Sections 7.3.3 and 7.5 (Miscellanea): 7.5.1 & 7.5.3

9

m

2

s

2

2

ˆ

()

ˆ

i

X

XX

n

m

s

ì

=

ï

Þ

í

-

=

ï

î

å

...

2

1

,,~(,)

iid

n

XXN

ms

K