Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3:...

21
Lesson 3: Cramer-Rao inequality Umberto Triacca Universit` a dell’Aquila [email protected] Umberto Triacca Lesson 3: Cramer-Rao inequality

Transcript of Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3:...

Page 1: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Lesson 3: Cramer-Rao inequality

Umberto Triacca

Universita dell’[email protected]

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 2: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Cramer-Rao inequality

Theorem 3.

Let x = (x1, ..., xn) be a random sample of nobservations from the distribution with p.d.f. f (x ; θ)depending on a real parameter θ.Let T (x) be an unbiasedestimator of θ.Then, subject to certain regularity conditions onf (x ; θ),the variance of T (x) satisfies the inequality

Var[T (x)] ≥ 1

E

[(δlnf (x;θ)

δθ

)2]

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 3: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Cramer-Rao inequality

Theorem 3.Let x = (x1, ..., xn) be a random sample of nobservations from the distribution with p.d.f. f (x ; θ)depending on a real parameter θ.

Let T (x) be an unbiasedestimator of θ.Then, subject to certain regularity conditions onf (x ; θ),the variance of T (x) satisfies the inequality

Var[T (x)] ≥ 1

E

[(δlnf (x;θ)

δθ

)2]

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 4: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Cramer-Rao inequality

Theorem 3.Let x = (x1, ..., xn) be a random sample of nobservations from the distribution with p.d.f. f (x ; θ)depending on a real parameter θ.Let T (x) be an unbiasedestimator of θ.

Then, subject to certain regularity conditions onf (x ; θ),the variance of T (x) satisfies the inequality

Var[T (x)] ≥ 1

E

[(δlnf (x;θ)

δθ

)2]

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 5: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Cramer-Rao inequality

Theorem 3.Let x = (x1, ..., xn) be a random sample of nobservations from the distribution with p.d.f. f (x ; θ)depending on a real parameter θ.Let T (x) be an unbiasedestimator of θ.Then, subject to certain regularity conditions onf (x ; θ),

the variance of T (x) satisfies the inequality

Var[T (x)] ≥ 1

E

[(δlnf (x;θ)

δθ

)2]

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 6: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Cramer-Rao inequality

Theorem 3.Let x = (x1, ..., xn) be a random sample of nobservations from the distribution with p.d.f. f (x ; θ)depending on a real parameter θ.Let T (x) be an unbiasedestimator of θ.Then, subject to certain regularity conditions onf (x ; θ),the variance of T (x) satisfies the inequality

Var[T (x)] ≥ 1

E

[(δlnf (x;θ)

δθ

)2]

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 7: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Cramer-Rao inequality

Theorem 3.Let x = (x1, ..., xn) be a random sample of nobservations from the distribution with p.d.f. f (x ; θ)depending on a real parameter θ.Let T (x) be an unbiasedestimator of θ.Then, subject to certain regularity conditions onf (x ; θ),the variance of T (x) satisfies the inequality

Var[T (x)] ≥ 1

E

[(δlnf (x;θ)

δθ

)2]

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 8: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Proof

T (x) is an unbiased estimator of θ,

so

E [T (x)] =

∫T (x)f (x; θ)dx = θ (1)

Differentiating both sides of equation (1) with respect to θ,and interchanging the order of integration and differentiation,gives ∫

T (x)δf (x; θ)

δθdx = 1 (2)

or ∫T (x)

δlnf (x; θ)

δθf (x; θ)dx = 1 (3)

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 9: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Proof

T (x) is an unbiased estimator of θ,so

E [T (x)] =

∫T (x)f (x; θ)dx = θ (1)

Differentiating both sides of equation (1) with respect to θ,and interchanging the order of integration and differentiation,gives ∫

T (x)δf (x; θ)

δθdx = 1 (2)

or ∫T (x)

δlnf (x; θ)

δθf (x; θ)dx = 1 (3)

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 10: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Proof

T (x) is an unbiased estimator of θ,so

E [T (x)] =

∫T (x)f (x; θ)dx = θ (1)

Differentiating both sides of equation (1) with respect to θ,and interchanging the order of integration and differentiation,gives ∫

T (x)δf (x; θ)

δθdx = 1 (2)

or ∫T (x)

δlnf (x; θ)

δθf (x; θ)dx = 1 (3)

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 11: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Proof

T (x) is an unbiased estimator of θ,so

E [T (x)] =

∫T (x)f (x; θ)dx = θ (1)

Differentiating both sides of equation (1) with respect to θ,and interchanging the order of integration and differentiation,gives ∫

T (x)δf (x; θ)

δθdx = 1 (2)

or ∫T (x)

δlnf (x; θ)

δθf (x; θ)dx = 1 (3)

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 12: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Proof

Because∫T (x)

δlnf (x; θ)

δθf (x; θ)dx = E

[T (x)

δlnf (x; θ)

δθ

](4)

by (3) it follows that

E

[T (x)

δlnf (x; θ)

δθ

]= 1

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 13: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Proof

Because∫T (x)

δlnf (x; θ)

δθf (x; θ)dx = E

[T (x)

δlnf (x; θ)

δθ

](4)

by (3) it follows that

E

[T (x)

δlnf (x; θ)

δθ

]= 1

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 14: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Proof

On the other hand, since

E

[δlnf (x; θ)

δθ

]= 0

we have that

E

[T (x)

δlnf (x; θ)

δθ

]= Cov

[T (x),

δlnf (x; θ)

δθ

]Hence

Cov

[T (x),

δlnf (x; θ)

δθ

]= 1

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 15: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Proof

On the other hand, since

E

[δlnf (x; θ)

δθ

]= 0

we have that

E

[T (x)

δlnf (x; θ)

δθ

]= Cov

[T (x),

δlnf (x; θ)

δθ

]Hence

Cov

[T (x),

δlnf (x; θ)

δθ

]= 1

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 16: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Proof

On the other hand, since

E

[δlnf (x; θ)

δθ

]= 0

we have that

E

[T (x)

δlnf (x; θ)

δθ

]= Cov

[T (x),

δlnf (x; θ)

δθ

]

Hence

Cov

[T (x),

δlnf (x; θ)

δθ

]= 1

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 17: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Proof

On the other hand, since

E

[δlnf (x; θ)

δθ

]= 0

we have that

E

[T (x)

δlnf (x; θ)

δθ

]= Cov

[T (x),

δlnf (x; θ)

δθ

]Hence

Cov

[T (x),

δlnf (x; θ)

δθ

]= 1

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 18: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Proof

Since the squared covariance cannot exceed the product of thetwo variances, we have

1 =

(Cov

[T (x),

δlnf (x; θ)

δθ

])2

≤ Var [T (x)] Var

[δlnf (x; θ)

δθ

]or

1 =

(Cov

[T (x),

δlnf (x; θ)

δθ

])2

≤ Var [T (x)]E

[(δlnf (x; θ)

δθ

)2]

It follows that

Var[T (x)] ≥ 1

E

[(δlnf (x;θ)

δθ

)2]

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 19: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Efficency

Definition 6. An unbiased estimator is efficient if its varianceis the lower bound of the inequality:

Var[T (x)] =1

E

[(δlnf (x;θ)

δθ

)2]

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 20: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Multidimensional Cramer-Rao inequality

The Cramer-Rao ineqality (Theorem 3) can be generalized toa vector valued parameter θ

The generalization of the Cramer-Rao inequality states that,again subject to regularity conditions, the variance-covariancematrix of the unbiased estimator T (x), the k × k matrixVar(T (x)) is such that Var(T (x)) − I−1(θ; x) is positivesemi-definite.

Umberto Triacca Lesson 3: Cramer-Rao inequality

Page 21: Lesson 3: Cramer-Rao inequalitywpage.unina.it/fdiiorio/SIdE/Triacca/Lezione3.pdf · Lesson 3: Cramer-Rao inequality Umberto Triacca Universit a dell’Aquila umberto.triacca@gmail.com

Multidimensional Cramer-Rao inequality

The Cramer-Rao ineqality (Theorem 3) can be generalized toa vector valued parameter θ

The generalization of the Cramer-Rao inequality states that,again subject to regularity conditions, the variance-covariancematrix of the unbiased estimator T (x), the k × k matrixVar(T (x)) is such that Var(T (x)) − I−1(θ; x) is positivesemi-definite.

Umberto Triacca Lesson 3: Cramer-Rao inequality