26 lines
1.9 KiB
Markdown
26 lines
1.9 KiB
Markdown
# The Cramer-Rao Lower Bound – Derivation and Examples
|
||
|
||
**Source:** HBS Research Computing Services Training Material
|
||
**URL:** https://www.hbs.edu/research-computing-services/Shared%20Documents/Training/cramerrao.pdf
|
||
|
||
## Content Summary
|
||
This document provides a step-by-step derivation and examples of the Cramer-Rao Lower Bound (CRLB) using the normal and binomial distributions. It covers the following concepts:
|
||
- **The Score:** Derivative of the log-likelihood function, viewed as a random variable.
|
||
- **Expectation of the Score:** Proven to be 0.
|
||
- **Fisher Information:** Expectation of the square of the score (or variance of the score), representing the "information" the data provides about the parameter.
|
||
- **Cramer-Rao Bound:** The minimum possible variance for any unbiased estimator is $1/I$, where $I$ is the Fisher Information.
|
||
- **Alternative Expression for Fisher Information:** $I(\theta) = -E[\frac{\partial^2}{\partial \theta^2} \log f(x|\theta)]$, connecting information to the curvature of the log-likelihood.
|
||
- **Observed vs. Expected Information:** Expected information uses the true parameter and expectation over all data; observed information uses the estimated parameter and actual data.
|
||
- **Information Matrix:** Extension to multiple parameters.
|
||
- **Connection to Maximum Likelihood Estimation (MLE):** MLE is asymptotically efficient, meaning its variance reaches the CRLB as sample size grows.
|
||
|
||
## Examples Detailed
|
||
1. **Normal Distribution:**
|
||
- Score: $g(\mu) = \frac{n}{\sigma^2}(\bar{x} - \mu)$
|
||
- Fisher Information: $I = \frac{n}{\sigma^2}$
|
||
- CRLB: $\frac{\sigma^2}{n}$, which matches the variance of the sample mean $\bar{x}$.
|
||
2. **Binomial Distribution:**
|
||
- Score: $g(\pi) = \frac{k}{\pi} - \frac{n-k}{1-\pi}$
|
||
- Fisher Information: $I = \frac{n}{\pi(1-\pi)}$
|
||
- CRLB: $\frac{\pi(1-\pi)}{n}$, matching the variance of the sample proportion $k/n$.
|