20260625:很多新内容

raw/papers/engram-conditional-memory-2026.md

@@ -0,0 +1,32 @@
+---
+title: "Engram: Conditional Memory via Scalable Lookup (Raw Archive)"
+created: 2026-06-25
+updated: 2026-06-25
+type: raw
+tags: ["conditional-memory", "sparsity", "ngram", "mixture-of-experts"]
+source: "https://arxiv.org/abs/2601.07372"
+---
+
+# Engram: Conditional Memory via Scalable Lookup — Raw Archive
+
+## Metadata
+
+- **Title**: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models
+- **Authors**: Xin Cheng, Wangding Zeng, Damai Dai, Qinyu Chen, Bingxuan Wang, Zhenda Xie, Kezhao Huang, Xingkai Yu, Zhewen Hao, Yukun Li, Han Zhang, Huishuai Zhang, Dongyan Zhao, Wenfeng Liang
+- **Affiliations**: Peking University, DeepSeek-AI
+- **arXiv**: 2601.07372
+- **Date**: 2026-01-12
+- **Categories**: cs.CL, cs.AI
+- **Code**: https://github.com/deepseek-ai/Engram
+
+## Abstract
+
+While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup, forcing them to inefficiently simulate retrieval through computation. To address this, we introduce conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic N-gram embedding for O(1) lookup. By formulating the Sparsity Allocation problem, we uncover a U-shaped scaling law that optimizes the trade-off between neural computation (MoE) and static memory (Engram). Guided by this law, we scale Engram to 27B parameters, achieving superior performance over a strictly iso-parameter and iso-FLOPs MoE baseline. Most notably, while the memory module is expected to aid knowledge retrieval (e.g., MMLU +3.4; CMMLU +4.0), we observe even larger gains in general reasoning (e.g., BBH +5.0; ARC-Challenge +3.7) and code/math domains (HumanEval +3.0; MATH +2.4). Mechanistic analyses reveal that Engram relieves the backbone's early layers from static reconstruction, effectively deepening the network for complex reasoning. Furthermore, by delegating local dependencies to lookups, it frees up attention capacity for global context, substantially boosting long-context retrieval (e.g., Multi-Query NIAH: 84.2 to 97.0).
+
+## Key Contributions
+
+1. Conditional memory as a new sparsity axis complementary to MoE
+2. Engram module: modernized N-gram embedding with multi-head hashing, context-aware gating, depthwise convolution
+3. Sparsity Allocation problem and U-shaped scaling law
+4. Infrastructure-aware design: deterministic addressing enables host memory prefetching
+5. Empirical validation at 27B-40B scale with comprehensive ablation