myWiki/raw/papers/tapered-language-models.md

title, source, authors, institutions, arxiv, category, date

title	source	authors	institutions	arxiv	category	date
Tapered Language Models (Raw)	https://arxiv.org/abs/2606.23670	Reza Bayat, Ali Behrouz, Aaron Courville	Mila, Cornell University, Université de Montréal, CIFAR	2606.23670v1	cs.LG	June 22, 2026

Tapered Language Models

Abstract

Modern LMs share a common chassis: a stack of identical layers with parameters allocated uniformly across depth. Evidence suggests layers contribute non-uniformly—later layers refine rather than transform the residual stream. We ask: should parameter capacity reflect this asymmetry? Under fixed budget, allocating more capacity to earlier layers and less to later layers improves perplexity; reverse allocation hurts. We introduce Tapered Language Models (TLMs), where a parameter-bearing component is monotonically tapered across depth. MLPs are the natural site: they dominate parameter count and expose width as a clean axis. Across three scales and four architectures, tapering MLP width via cosine schedule consistently improves perplexity and downstream benchmarks at no additional cost.

1. Introduction

Uniform layer width is an inherited default from Vaswani et al. (2017). As models scaled, this uniformity remained unexamined.

2. Related Work

• Mixture-of-Experts (MoE): conditional parameter allocation
• LayerDrop, stochastic depth: layer-wise computation pruning
• Depth-wise scaling: changing number of layers

3. Tapered Language Models

Principle: Under fixed total parameter budget, monotonically decrease parameter allocation from early to late layers.

Why MLP width:

• MLP accounts for majority of parameters in Transformer, Gated Attention, Mamba, Titans
• Width (d_ff) is a single clean axis of variation
• Token-mixing modules vary across architectures, making uniform comparison difficult

Cosine schedule: w_ℓ = w_max · (cos(πℓ/(2L)))^p, where p controls steepness

4. Experiments

Controlled study (440M Transformer):

• Uniform: 16.28 perplexity
• Cosine taper (1.50→0.50 × d_ff): 14.44 perplexity
• Cosine dominates linear at every taper range
• U-shape: optimal at 1.50→0.50, extreme tapers underperform

Architecture sweep (440M):

• Transformer ✓
• Gated Attention ✓
• Hope-attention ✓
• Titans ✓

Scale sweep: 440M, 1B, 3B — consistent improvement

Downstream: HellaSwag, ARC-E, PIQA, WinoGrande — taper improves all

5. Analysis

• Early layers benefit more from MLP capacity (transformation)
• Late layers benefit less (refinement of residual stream)
• Cosine > Linear > Step-wise > Uniform