Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 2 (February 2025) of the DeepSeek-V3 Technical Report).
What it changed
It showed that an openly released model could reach the level of leading closed models such as GPT-4o, with a final training run that its authors put at about 2.8 million GPU hours, or about $5.6 million at rented prices.
The problem before it
Open models, such as Llama 3.1, were improving but still trailed the best closed models. Training a model at that level was very expensive.
Mixture-of-experts models save compute, because each token uses only a few of many “expert” sub-networks. But they bring their own problems. The work must be spread evenly across experts. The usual fix, an extra “auxiliary” loss that rewards balance, can hurt quality. And when experts sit on many machines, sending tokens between them is slow.
DeepSeek’s earlier models, DeepSeekMoE and DeepSeek-V2, had already introduced the two architectures that V3 keeps.
The idea
The mechanism. DeepSeek-V3 is a Transformer with 671 billion parameters, of which 37 billion are used for each token. It combines known parts with several new ones:
- Multi-head Latent Attention (MLA), from DeepSeek-V2. Keys and values are compressed into one small vector per token, so the memory needed during generation (the KV cache) is much smaller. Position is added separately with rotary position embedding.
- DeepSeekMoE, with fine-grained experts. Each mixture-of-experts layer has 1 shared expert that every token uses and 256 routed experts, of which 8 are chosen for each token.
- Load balancing without an auxiliary loss. Each expert gets a bias number that is added to its score only when choosing experts. After each training step, the bias goes down for overloaded experts and up for underused ones. A very small extra loss stops extreme imbalance inside a single sequence. No tokens were dropped in training or use.
- Multi-token prediction. Besides the next token, the model also predicts the token after it, through an extra module. The authors found this improved results. The extra module can also be used for speculative decoding to speed up generation.
- FP8 training. Most heavy calculations use 8-bit floating-point numbers, extending mixed-precision training. Values are scaled in small tiles and blocks to avoid errors from outliers. Compared with 16-bit training, the loss differed by less than 0.25%.
- Engineering for speed. A pipeline schedule called DualPipe overlaps calculation with communication between machines, and each token is sent to at most 4 machines.
It was trained on 14.8 trillion tokens, with more maths, code and languages other than English and Chinese than before. The context was then extended from 4K to 32K and then 128K tokens. After that came supervised fine-tuning and reinforcement learning with GRPO. The fine-tuning data includes reasoning examples generated by a DeepSeek-R1 model, adjusted to keep answers from getting too long.
The notation. The output of a mixture-of-experts layer is h'_t = u_t + Σ_shared FFN_i(u_t) + Σ_routed g_(i,t) FFN_i(u_t), the input plus the shared experts plus the chosen routed experts, each weighted by its gate value g. An expert is chosen if its score s_(i,t) + b_i is among the top 8. The bias b_i affects only the choice, not the weight g.
What it showed
Reported in the paper. It is a technical report from DeepSeek, not a peer-reviewed paper. The comparisons were run by DeepSeek.
- Cost. Pretraining took 2.664 million H800 GPU hours, about 3.7 days per trillion tokens on 2,048 GPUs, in under two months. With context extension and post-training, the total was 2.788 million GPU hours, or $5.576 million at an assumed $2 per GPU hour. The authors say this covers only the final training run, not earlier research and experiments.
- Stability. There were no loss spikes that could not be recovered from, and no rollbacks, in the whole run.
- Knowledge. 88.5 on MMLU, 75.9 on MMLU-Pro and 59.1 on GPQA-Diamond, above all other open models. GPT-4o scored 72.6 on MMLU-Pro and 49.9 on GPQA-Diamond. Claude 3.5 Sonnet scored 78.0 and 65.0.
- Maths. 90.2 on MATH-500 and 39.2 on AIME 2024, against 74.6 and 9.3 for GPT-4o. The authors call this the best result for models without long chain-of-thought reasoning.
- Code. On SWE-bench Verified it resolved 42.0% of problems, behind Claude 3.5 Sonnet (50.8%) and ahead of GPT-4o (38.8%). It led on the competition-style coding tests LiveCodeBench and Codeforces.
- Facts. It trailed GPT-4o and Claude 3.5 Sonnet on English factual questions (SimpleQA), but led on the Chinese version. The authors say they put more training into Chinese knowledge.
- Open-ended answers. 85.5 on Arena-Hard and 70.0 on AlpacaEval 2.0, the highest of the models compared.
- Long context. It found facts placed anywhere in inputs of up to 128K tokens.
Limits, and what came later
The limits come from the paper:
- To run efficiently, the recommended deployment unit is large, which may be a burden for small teams.
- Generation speed, though more than twice that of DeepSeek-V2, could still be improved.
- Distilling from a reasoning model improved results but made answers much longer, so the authors had to balance accuracy against length.
- The authors warn that optimising for a fixed set of benchmarks can give a misleading picture of a model’s ability.
- The cost figure assumes rented GPUs at $2 an hour and covers only the final training run, not earlier research and experiments.
Later work built on it directly. DeepSeek-R1 was trained from the DeepSeek-V3 base model. DeepSeek-V3.2 continued the same line, and DeepSeek-V4 continued DeepSeek’s mixture-of-experts design.
What to know first
How to read it
The report is 53 pages. The main text is 35.
- Read section 1 and Table 1 for the overview and the cost.
- Read section 2: MLA, the mixture-of-experts layer with its load balancing, and multi-token prediction, with Figures 2 and 3.
- Skim section 3 unless the engineering interests you. Section 3.3 explains FP8 training.
- Read sections 4.1 and 4.2 for the data and model settings.
- Read section 5.3 with Tables 6 and 7, then section 6 for the limits.
Expect about 3 hours for the parts above (an estimate).
Links