Before it
Data-parallel training replicated optimizer state, gradients and weights on every GPU.
The problem
That replication wastes most of the cluster's memory on identical copies.
What it introduced
Partition optimizer state, gradients and parameters across devices instead of duplicating them.
Why it matters
It removed the memory ceiling on data-parallel training and became the open-source default via DeepSpeed.
What modern models inherit
Sharded training state, now assumed in any large run.
What came next
Used for MT-NLG, BLOOM and much of the open training ecosystem.