Mixed precision training
Training with 16-bit numbers for most calculations, which is faster and uses less memory, while keeping a 32-bit copy of the weights so that small updates are not lost.
Introduced by
Mixed precision, 2017
Used by
- Sparse Transformer, 2019
- Chinchilla, 2022
- DeepSeek-V3, 2024