20260706:新增一些文章

raw/papers/tapered-language-models.md

@@ -0,0 +1,60 @@
+---
+title: "Tapered Language Models (Raw)"
+source: https://arxiv.org/abs/2606.23670
+authors: Reza Bayat, Ali Behrouz, Aaron Courville
+institutions: Mila, Cornell University, Université de Montréal, CIFAR
+arxiv: 2606.23670v1
+category: cs.LG
+date: June 22, 2026
+---
+
+# Tapered Language Models
+
+## Abstract
+
+Modern LMs share a common chassis: a stack of identical layers with parameters allocated uniformly across depth. Evidence suggests layers contribute non-uniformly—later layers refine rather than transform the residual stream. We ask: should parameter capacity reflect this asymmetry? Under fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity; reverse allocation hurts. We introduce Tapered Language Models (TLMs), where a parameter-bearing component is monotonically tapered across depth. MLPs are the natural site: they dominate parameter count and expose width as a clean axis. Across three scales and four architectures, tapering MLP width via cosine schedule consistently improves perplexity and downstream benchmarks at no additional cost.
+
+## 1. Introduction
+
+Uniform layer width is an inherited default from Vaswani et al. (2017). As models scaled, this uniformity remained unexamined.
+
+## 2. Related Work
+
+- Mixture-of-Experts (MoE): conditional parameter allocation
+- LayerDrop, stochastic depth: layer-wise computation pruning
+- Depth-wise scaling: changing number of layers
+
+## 3. Tapered Language Models
+
+**Principle**: Under fixed total parameter budget, monotonically decrease parameter allocation from early to late layers.
+
+**Why MLP width**: 
+- MLP accounts for majority of parameters in Transformer, Gated Attention, Mamba, Titans
+- Width (d_ff) is a single clean axis of variation
+- Token-mixing modules vary across architectures, making uniform comparison difficult
+
+**Cosine schedule**: w_ℓ = w_max · (cos(πℓ/(2L)))^p, where p controls steepness
+
+## 4. Experiments
+
+**Controlled study** (440M Transformer):
+- Uniform: 16.28 perplexity
+- Cosine taper (1.50→0.50 × d_ff): 14.44 perplexity
+- Cosine dominates linear at every taper range
+- U-shape: optimal at 1.50→0.50, extreme tapers underperform
+
+**Architecture sweep** (440M):
+- Transformer ✓
+- Gated Attention ✓ 
+- Hope-attention ✓
+- Titans ✓
+
+**Scale sweep**: 440M, 1B, 3B — consistent improvement
+
+**Downstream**: HellaSwag, ARC-E, PIQA, WinoGrande — taper improves all
+
+## 5. Analysis
+
+- Early layers benefit more from MLP capacity (transformation)
+- Late layers benefit less (refinement of residual stream)
+- Cosine > Linear > Step-wise > Uniform