20260420:first commit

raw/papers/hbs-cramerrao-bound-notes.md

@@ -0,0 +1,25 @@
+# The Cramer-Rao Lower Bound – Derivation and Examples
+
+**Source:** HBS Research Computing Services Training Material
+**URL:** https://www.hbs.edu/research-computing-services/Shared%20Documents/Training/cramerrao.pdf
+
+## Content Summary
+This document provides a step-by-step derivation and examples of the Cramer-Rao Lower Bound (CRLB) using the normal and binomial distributions. It covers the following concepts:
+- **The Score:** Derivative of the log-likelihood function, viewed as a random variable.
+- **Expectation of the Score:** Proven to be 0.
+- **Fisher Information:** Expectation of the square of the score (or variance of the score), representing the "information" the data provides about the parameter.
+- **Cramer-Rao Bound:** The minimum possible variance for any unbiased estimator is $1/I$, where $I$ is the Fisher Information.
+- **Alternative Expression for Fisher Information:** $I(\theta) = -E[\frac{\partial^2}{\partial \theta^2} \log f(x|\theta)]$, connecting information to the curvature of the log-likelihood.
+- **Observed vs. Expected Information:** Expected information uses the true parameter and expectation over all data; observed information uses the estimated parameter and actual data.
+- **Information Matrix:** Extension to multiple parameters.
+- **Connection to Maximum Likelihood Estimation (MLE):** MLE is asymptotically efficient, meaning its variance reaches the CRLB as sample size grows.
+
+## Examples Detailed
+1. **Normal Distribution:** 
+   - Score: $g(\mu) = \frac{n}{\sigma^2}(\bar{x} - \mu)$
+   - Fisher Information: $I = \frac{n}{\sigma^2}$
+   - CRLB: $\frac{\sigma^2}{n}$, which matches the variance of the sample mean $\bar{x}$.
+2. **Binomial Distribution:**
+   - Score: $g(\pi) = \frac{k}{\pi} - \frac{n-k}{1-\pi}$
+   - Fisher Information: $I = \frac{n}{\pi(1-\pi)}$
+   - CRLB: $\frac{\pi(1-\pi)}{n}$, matching the variance of the sample proportion $k/n$.