NLP Lineage

10 October 2017 · Efficiency and infrastructure

Mixed precision

Mixed Precision Training

P. Micikevicius, et al. (10 authors)

Date
10 October 2017
What the date marks
arXiv submission
Kind
Training method
Organisation
NVIDIA / Baidu
Access
Not applicable
Other dates
ICLR 2018
Threads
In the guided tour
Chapter 9. Making it run anywhere
Learning phases
8. Training from scratch

A milestone: 2 later milestones descend directly from it, and its descendants reach 5 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Training ran in 32-bit floating point throughout.

The problem

Memory and bandwidth, not arithmetic, were the binding constraints on model size.

What it introduced

Keep weights in 16-bit for the forward and backward pass, with loss scaling and a 32-bit master copy for stability.

Why it matters

It roughly halved the memory cost of training, which is part of why billion-parameter models became affordable.

What modern models inherit

Reduced precision as the default, now extended to bf16 and fp8.

What came next

Megatron, ZeRO and every large training run since depend on it.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

No documented ancestors are recorded for Mixed precision without weak links.

No documented ancestors are recorded for Mixed precision without weak links.

No recorded ancestors are recorded for Mixed precision.

No recorded ancestors are recorded for Mixed precision.

After: what it led to

Mixed precision has 13 documented descendants across 6 chapters, reaching forward to Qwen3.8 in 2026. Showing its direct descendants.

Mixed precision has 13 documented descendants across 6 chapters, reaching forward to Qwen3.8 in 2026. Showing all of them.

Mixed precision has 19 recorded descendants, including indirect ones, across 6 chapters, reaching forward to Qwen3.8 in 2026. Showing its direct descendants.

Mixed precision has 19 recorded descendants, including indirect ones, across 6 chapters, reaching forward to Qwen3.8 in 2026. Showing all of them.

  1. 2017Mixed precisionThis milestone.
  2. 2019Megatron-LMwas influenced by Mixed precisionChapter 4. Pretrain once, use everywhere
  3. 2022MT-NLGbuilt directly on Megatron-LMChapter 5. Scale becomes a science
  4. 2022OPT-175Bwas influenced by Megatron-LMChapter 8. The weights get out
  5. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  6. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  7. 2023QLoRAbuilt directly on LLM.int8()Chapter 9. Making it run anywhere
  8. 2024DeepSeek-V3was influenced by Mixed precisionChapter 8. The weights get out
  9. 2025DeepSeek-R1built directly on DeepSeek-V3Chapter 12. Thinking longer
  10. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  11. 2025DeepSeek-V3.2built directly on DeepSeek-V3Chapter 10. Reading more than you remember
  12. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  13. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  14. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  1. 2017Mixed precisionThis milestone.
  2. 2019Megatron-LMwas influenced by Mixed precisionChapter 4. Pretrain once, use everywhere
  3. 2022MT-NLGbuilt directly on Megatron-LMChapter 5. Scale becomes a science
  4. 2022OPT-175Bwas influenced by Megatron-LMChapter 8. The weights get out
  5. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  6. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  7. 2023llama.cppwas influenced by GPTQ (indirect influence)Chapter 9. Making it run anywhere
  8. 2023QLoRAbuilt directly on LLM.int8()Chapter 9. Making it run anywhere
  9. 2024DeepSeek-V3was influenced by Mixed precisionChapter 8. The weights get out
  10. 2025DeepSeek-R1built directly on DeepSeek-V3Chapter 12. Thinking longer
  11. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  12. 2025Kimi K2was influenced by DeepSeek-V3 (indirect influence)Chapter 8. The weights get out
  13. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  14. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  15. 2025DeepSeek-V3.2built directly on DeepSeek-V3Chapter 10. Reading more than you remember
  16. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  17. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  18. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  19. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  20. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out

Direct links and their sources

Where it came from

None recorded in the sources used.

What it led to

Influenced

Terms