# The Cramer-Rao Lower Bound – Derivation and Examples

**Source:** HBS Research Computing Services Training Material
**URL:** https://www.hbs.edu/research-computing-services/Shared%20Documents/Training/cramerrao.pdf

## Content Summary
This document provides a step-by-step derivation and examples of the Cramer-Rao Lower Bound (CRLB) using the normal and binomial distributions. It covers the following concepts:
- **The Score:** Derivative of the log-likelihood function, viewed as a random variable.
- **Expectation of the Score:** Proven to be 0.
- **Fisher Information:** Expectation of the square of the score (or variance of the score), representing the "information" the data provides about the parameter.
- **Cramer-Rao Bound:** The minimum possible variance for any unbiased estimator is $1/I$, where $I$ is the Fisher Information.
- **Alternative Expression for Fisher Information:** $I(\theta) = -E[\frac{\partial^2}{\partial \theta^2} \log f(x|\theta)]$, connecting information to the curvature of the log-likelihood.
- **Observed vs. Expected Information:** Expected information uses the true parameter and expectation over all data; observed information uses the estimated parameter and actual data.
- **Information Matrix:** Extension to multiple parameters.
- **Connection to Maximum Likelihood Estimation (MLE):** MLE is asymptotically efficient, meaning its variance reaches the CRLB as sample size grows.

## Examples Detailed
1. **Normal Distribution:** 
   - Score: $g(\mu) = \frac{n}{\sigma^2}(\bar{x} - \mu)$
   - Fisher Information: $I = \frac{n}{\sigma^2}$
   - CRLB: $\frac{\sigma^2}{n}$, which matches the variance of the sample mean $\bar{x}$.
2. **Binomial Distribution:**
   - Score: $g(\pi) = \frac{k}{\pi} - \frac{n-k}{1-\pi}$
   - Fisher Information: $I = \frac{n}{\pi(1-\pi)}$
   - CRLB: $\frac{\pi(1-\pi)}{n}$, matching the variance of the sample proportion $k/n$.
