20260429:一些新东西

raw/papers/deepseek-ai-deepseek-v4-2026.md

@@ -0,0 +1,62 @@
+# DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence
+
+> **Source**: Hugging Face (technical report)
+> **Authors**: DeepSeek-AI
+> **Date**: 2026
+> **Link**: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf
+> **Models**: DeepSeek-V4-Pro (1.6T/49B activated), DeepSeek-V4-Flash (284B/13B activated)
+
+## Abstract
+
+We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models — DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) — both supporting a context length of one million tokens.
+
+## Key Upgrades over DeepSeek-V3
+
+1. **Hybrid attention architecture**: Compressed Sparse Attention (CSA) + Heavily Compressed Attention (HCA) for long-context efficiency
+2. **Manifold-Constrained Hyper-Connections (mHC)**: Upgrades conventional residual connections for stability and expressivity
+3. **Muon optimizer**: Faster convergence and greater training stability
+
+## Architecture Summary
+
+- Retains DeepSeekMoE framework (fine-grained + shared experts) and Multi-Token Prediction (MTP)
+- Hybrid CSA/HCA: CSA compresses KV cache along sequence dimension then applies sparse attention; HCA applies aggressive compression with dense attention
+- mHC constrains residual mapping to doubly stochastic matrices (Birkhoff polytope) via Sinkhorn-Knopp algorithm
+- Muon with hybrid Newton-Schulz orthogonalization for most modules; AdamW for embeddings, heads, biases, RMSNorm
+
+## Infrastructure Highlights
+
+- Fine-grained communication-computation overlap in Expert Parallelism (1.5-1.73x speedup)
+- MegaMoE2 mega-kernel (open-sourced)
+- TileLang DSL with Z3 SMT solver integration
+- Batch-invariant and deterministic kernel libraries
+- FP4 quantization-aware training for MoE experts
+- Inference: heterogeneous KV cache with on-disk storage
+
+## Pre-Training
+
+- DeepSeek-V4-Flash: 32T tokens; DeepSeek-V4-Pro: 33T tokens
+- Both natively support 1M-length contexts after pre-training
+
+## Post-Training Pipeline
+
+Two-stage paradigm:
+1. **Specialist Training**: Independent expert models trained per domain (math, coding, agent, instruction following) via SFT + RL (GRPO)
+2. **On-Policy Distillation (OPD)**: Multi-teacher reverse-KL distillation merging expert capabilities into unified model
+
+## Key Evaluation Results
+
+- **Knowledge (SimpleQA, MMLU-Pro, HLE, GPQA)**: Significantly outperforms open-source models; closing gap with Gemini-3.1-Pro
+- **Reasoning**: Superior to GPT-5.2, Gemini-3.0-Pro; trails GPT-5.4/Gemini-3.1-Pro by ~3-6 months
+- **Agent**: On par with Kimi-K2.6, GLM-5.1; outperforms Claude Sonnet 4.5 in internal eval
+- **Long-Context**: Surpasses Gemini-3.1-Pro on academic benchmarks at 1M tokens
+- **Chinese Writing**: 62.7% win rate vs Gemini-3.1-Pro
+
+## Efficiency (1M-token context vs DeepSeek-V3.2)
+
+- DeepSeek-V4-Pro: 27% FLOPs, 10% KV cache
+- DeepSeek-V4-Flash: 10% FLOPs, 7% KV cache
+
+---
+
+*Format: Raw paper archive. See [[deepseek-v4-million-token-context]] for the wiki page.*
+*Last Updated: 2026-04-27*