DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence

A new family of large language models from DeepSeek supports context windows of one million tokens while sharply reducing the computational cost of processing long inputs, according to a technical paper posted to arXiv [1]. The DeepSeek-V4 series comprises two Mixture-of-Experts models: DeepSeek-V4-Pro, with 1.6 trillion total parameters and 49 billion activated parameters, and DeepSeek-V4-Flash, with 284 billion total parameters and 13 billion activated parameters [1]. Both models were pre-trained on more than 32 trillion diverse and high-quality tokens and underwent a comprehensive post-training pipeline [1]. The model checkpoints have been released on Hugging Face [1]. The architecture introduces a hybrid attention mechanism that combines Compressed Sparse Attention and Heavily Compressed Attention to improve long-context efficiency [1]. The models also use Manifold-Constrained Hyper-Connections, which enhance conventional residual connections, and the Muon optimizer for faster convergence and greater training stability [1]. In the one-million-token context setting, DeepSeek-V4-Pro requires only 27 percent of the single-token inference FLOPs and 10 percent of the KV cache compared with its predecessor, DeepSeek-V3.2 [1]. The paper states that this efficiency enables the team to routinely support one-million-token contexts, making long-horizon tasks and further test-time scaling more feasible [1]. A maximum reasoning effort mode, DeepSeek-V4-Pro-Max, redefines the state-of-the-art for open models, outperforming its predecessors in core tasks [1]. Large language models are a type of machine learning model designed for natural language processing tasks such as language generation, trained with self-supervised learning on vast amounts of text [3]. The DeepSeek-V4 series extends this paradigm by pushing context length and inference efficiency simultaneously [1][3].