Before it
Models were trained by copying the whole model onto each GPU.
The problem
Past a few billion parameters, the model no longer fits on one device at all.
What it introduced
Split individual layers across GPUs with carefully placed communication, reaching 8.3B parameters on 512 GPUs.
Why it matters
The engineering that made the scaling era physically possible. Almost every large model since uses this or its descendants.
What modern models inherit
Tensor parallelism, still part of every frontier training stack.
What came next
ZeRO complemented it; together they trained MT-NLG, PaLM and beyond.