myWiki/raw/papers/deepseek-ai-deepseek-v4-2026.md

DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

Source: Hugging Face (technical report) Authors: DeepSeek-AI Date: 2026 Link: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/DeepSeek_V4.pdf Models: DeepSeek-V4-Pro (1.6T/49B activated), DeepSeek-V4-Flash (284B/13B activated)

Abstract

We present a preview version of DeepSeek-V4 series, including two strong Mixture-of-Experts (MoE) language models — DeepSeek-V4-Pro with 1.6T parameters (49B activated) and DeepSeek-V4-Flash with 284B parameters (13B activated) — both supporting a context length of one million tokens.

Key Upgrades over DeepSeek-V3

1. Hybrid attention architecture: Compressed Sparse Attention (CSA) + Heavily Compressed Attention (HCA) for long-context efficiency
2. Manifold-Constrained Hyper-Connections (mHC): Upgrades conventional residual connections for stability and expressivity
3. Muon optimizer: Faster convergence and greater training stability

Architecture Summary

• Retains DeepSeekMoE framework (fine-grained + shared experts) and Multi-Token Prediction (MTP)
• Hybrid CSA/HCA: CSA compresses KV cache along sequence dimension then applies sparse attention; HCA applies aggressive compression with dense attention
• mHC constrains residual mapping to doubly stochastic matrices (Birkhoff polytope) via Sinkhorn-Knopp algorithm
• Muon with hybrid Newton-Schulz orthogonalization for most modules; AdamW for embeddings, heads, biases, RMSNorm

Infrastructure Highlights

• Fine-grained communication-computation overlap in Expert Parallelism (1.5-1.73x speedup)
• MegaMoE2 mega-kernel (open-sourced)
• TileLang DSL with Z3 SMT solver integration
• Batch-invariant and deterministic kernel libraries
• FP4 quantization-aware training for MoE experts
• Inference: heterogeneous KV cache with on-disk storage

Pre-Training

• DeepSeek-V4-Flash: 32T tokens; DeepSeek-V4-Pro: 33T tokens
• Both natively support 1M-length contexts after pre-training

Post-Training Pipeline

Two-stage paradigm:

1. Specialist Training: Independent expert models trained per domain (math, coding, agent, instruction following) via SFT + RL (GRPO)
2. On-Policy Distillation (OPD): Multi-teacher reverse-KL distillation merging expert capabilities into unified model

Key Evaluation Results

• Knowledge (SimpleQA, MMLU-Pro, HLE, GPQA): Significantly outperforms open-source models; closing gap with Gemini-3.1-Pro
• Reasoning: Superior to GPT-5.2, Gemini-3.0-Pro; trails GPT-5.4/Gemini-3.1-Pro by ~3-6 months
• Agent: On par with Kimi-K2.6, GLM-5.1; outperforms Claude Sonnet 4.5 in internal eval
• Long-Context: Surpasses Gemini-3.1-Pro on academic benchmarks at 1M tokens
• Chinese Writing: 62.7% win rate vs Gemini-3.1-Pro

Efficiency (1M-token context vs DeepSeek-V3.2)

• DeepSeek-V4-Pro: 27% FLOPs, 10% KV cache
• DeepSeek-V4-Flash: 10% FLOPs, 7% KV cache

---

Format: Raw paper archive. See deepseek-v4-million-token-context for the wiki page. Last Updated: 2026-04-27