20260420:first commit

concepts/cramer-rao-lower-bound.md

@@ -0,0 +1,77 @@
+---
+title: Cramér-Rao Lower Bound (CRLB)
+created: 2026-04-17
+updated: 2026-04-17
+type: concept
+tags: [machine-learning, benchmark]
+sources: [raw/papers/hbs-cramerrao-bound-notes.md]
+---
+
+# Cramér-Rao Lower Bound (CRLB)
+
+## Definition
+The Cramér-Rao Lower Bound (CRLB) states that for **any unbiased estimator** of a population parameter $\theta$, the lowest possible variance is the reciprocal of the Fisher Information $I(\theta)$:
+$$\text{Var}(\hat{\theta}) \geq \frac{1}{I(\theta)}$$
+
+It represents a fundamental limit in statistical estimation: no matter how clever your estimation method is, you cannot beat this bound.
+
+## Key Concepts
+
+### 1. The Score Function
+The score $g(\theta; \mathbf{x})$ is the derivative of the log-likelihood with respect to the parameter:
+$$g(\theta; \mathbf{x}) = \frac{\partial}{\partial \theta} \log f(\mathbf{x} \mid \theta)$$
+- It measures the "force" the data exerts on the parameter estimate.
+- **Crucial property:** $\mathbb{E}[g(\theta; \mathbf{x})] = 0$ (under regularity conditions).
+
+### 2. Fisher Information
+Fisher Information $I(\theta)$ is the variance of the score function:
+$$I(\theta) = \text{Var}(g(\theta; \mathbf{x})) = \mathbb{E}\left[ \left( \frac{\partial}{\partial \theta} \log f(\mathbf{x} \mid \theta) \right)^2 \right]$$
+
+**Alternative expression (via curvature):**
+$$I(\theta) = -\mathbb{E}\left[ \frac{\partial^2}{\partial \theta^2} \log f(\mathbf{x} \mid \theta) \right]$$
+This connects information directly to the curvature of the log-likelihood function. A sharper peak (higher curvature) means higher information and a tighter bound.
+
+**Properties:**
+- $I(\theta)$ is proportional to sample size $n$ ($I_n = n \cdot I_1$).
+- Higher variance in the data means lower information per data point.
+
+### 3. Observed vs. Expected Information
+- **Expected Information:** Uses the true parameter and expectation over all possible data. Formula-based.
+- **Observed Information:** Uses the actual observed data and the estimated parameter $\hat{\theta}$. Computed from the Hessian of the log-likelihood at $\hat{\theta}$.
+- In practice (especially in MLE), standard errors are calculated using the observed information.
+
+## Classic Examples
+
+### Normal Distribution (Mean Estimation)
+- **Parameter:** $\mu$
+- **Score:** $g(\mu) = \frac{n}{\sigma^2}(\bar{x} - \mu)$
+- **Fisher Information:** $I = \frac{n}{\sigma^2}$
+- **CRLB:** $\frac{\sigma^2}{n}$
+- **Conclusion:** The sample mean $\bar{x}$ is the "best" unbiased estimator, as its variance exactly hits the bound.
+
+### Binomial Distribution (Proportion Estimation)
+- **Parameter:** $\pi$
+- **Score:** $g(\pi) = \frac{k}{\pi} - \frac{n-k}{1-\pi}$
+- **Fisher Information:** $I = \frac{n}{\pi(1-\pi)}$
+- **CRLB:** $\frac{\pi(1-\pi)}{n}$
+- **Conclusion:** The sample proportion $\hat{\pi} = k/n$ is the optimal unbiased estimator.
+
+## Connection to Maximum Likelihood Estimation (MLE)
+- MLE is **consistent** and **asymptotically efficient**.
+- As sample size $n \to \infty$, the variance of the MLE approaches the CRLB: $\text{Var}(\hat{\theta}_{\text{MLE}}) \approx 1/I(\theta)$.
+- This is why standard errors reported by MLE software are calculated as $1/\sqrt{I_{\text{observed}}}$.
+
+## Role in Computerized Adaptive Testing (CAT)
+In CAT, the CRLB dictates the theoretical limit of measurement precision. 
+- Each question contributes a certain amount of Fisher Information $I_i(\theta)$.
+- The test continues until the accumulated information $I(\theta) = \sum I_i(\theta)$ is large enough that $1/I(\theta)$ (the minimum possible variance) is below a predefined threshold.
+- **选题策略 (Item Selection):** Choosing the item with the maximum $I_i(\theta)$ at the current ability estimate $\hat{\theta}$ is equivalent to driving the CRLB down as fast as possible.
+
+## Multidimensional Extension (Information Matrix)
+For a vector of parameters $\boldsymbol{\theta}$, the Fisher Information becomes a matrix $\mathbf{I}(\boldsymbol{\theta})$. The CRLB states that the covariance matrix of any unbiased estimator satisfies:
+$$\text{Cov}(\hat{\boldsymbol{\theta}}) \succeq \mathbf{I}(\boldsymbol{\theta})^{-1}$$
+(where $\succeq$ denotes positive semi-definiteness).
+
+## 相关概念
+- [[computerized-adaptive-testing]] — CAT 的核心目标是最小化能力估计方差，CRLB 提供了理论下界，选题策略本质上是在最大化 Fisher 信息以快速逼近该下界。
+- [[eml-universal-operator]] — EML 树的梯度优化依赖于对参数空间的曲率估计，与 CRLB 中 Fisher 信息作为对数似然曲率的数学本质相通。