myWiki/raw/papers/hbs-cramerrao-bound-notes.md

The Cramer-Rao Lower Bound – Derivation and Examples

Source: HBS Research Computing Services Training Material URL: https://www.hbs.edu/research-computing-services/Shared%20Documents/Training/cramerrao.pdf

Content Summary

This document provides a step-by-step derivation and examples of the Cramer-Rao Lower Bound (CRLB) using the normal and binomial distributions. It covers the following concepts:

• The Score: Derivative of the log-likelihood function, viewed as a random variable.
• Expectation of the Score: Proven to be 0.
• Fisher Information: Expectation of the square of the score (or variance of the score), representing the "information" the data provides about the parameter.
• Cramer-Rao Bound: The minimum possible variance for any unbiased estimator is 1/I, where I is the Fisher Information.
• Alternative Expression for Fisher Information: I(\theta) = -E[\frac{\partial^2}{\partial \theta^2} \log f(x|\theta)], connecting information to the curvature of the log-likelihood.
• Observed vs. Expected Information: Expected information uses the true parameter and expectation over all data; observed information uses the estimated parameter and actual data.
• Information Matrix: Extension to multiple parameters.
• Connection to Maximum Likelihood Estimation (MLE): MLE is asymptotically efficient, meaning its variance reaches the CRLB as sample size grows.

Examples Detailed

1. Normal Distribution:
   • Score: g(\mu) = \frac{n}{\sigma^2}(\bar{x} - \mu)
   • Fisher Information: I = \frac{n}{\sigma^2}
   • CRLB: \frac{\sigma^2}{n}, which matches the variance of the sample mean \bar{x}.
2. Binomial Distribution:
   • Score: g(\pi) = \frac{k}{\pi} - \frac{n-k}{1-\pi}
   • Fisher Information: I = \frac{n}{\pi(1-\pi)}
   • CRLB: \frac{\pi(1-\pi)}{n}, matching the variance of the sample proportion k/n.