Before it
GPT-3's 175B was the largest dense model publicly described.
The problem
Could dense training be pushed further, and what does it take in engineering terms?
What it introduced
530B dense parameters trained with combined tensor, pipeline and data parallelism.
Why it matters
The high-water mark of pure dense scaling, and the clearest evidence that this path was hitting diminishing returns.
What modern models inherit
Mostly a cautionary datapoint about scaling parameters without scaling data.
What came next
Chinchilla redirected the field toward data; sparsity took the parameter-count crown.