Scale and data
42 milestones sit on this thread. For 25 of them it is the main thread. The others also belong to another thread, which is named under them.
2017
- Sparse MoEOutrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerArchitectural idea
2019
- GPT-2Language Models are Unsupervised Multitask LearnersMain thread: Language modellingModel
- WebTextWebTextMain thread: Language modellingDataset
- Megatron-LMMegatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismInfrastructure
- ZeRO / DeepSpeedZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsInfrastructure
- C4C4: Colossal Clean Crawled CorpusMain thread: Language modellingDataset
2020
- Scaling lawsScaling Laws for Neural Language ModelsMain thread: Language modellingResearch paper
- GPT-3Language Models are Few-Shot LearnersMain thread: Language modellingModel
- GShardGShard: Scaling Giant Models with Conditional ComputationInfrastructure
- MMLUMeasuring Massive Multitask Language UnderstandingDataset
- The PileThe Pile: An 800GB Dataset of Diverse TextMain thread: Language modellingDataset
2021
- Switch TransformerSwitch Transformers: Scaling to Trillion Parameter ModelsModel
- GopherScaling Language Models: Methods, Analysis and Insights from Training GopherMain thread: Language modellingModel
2022
- MT-NLGMegatron-Turing NLG 530BModel
- ChinchillaTraining Compute-Optimal Large Language ModelsMain thread: Language modellingResearch paper
- PaLMPaLM: Scaling Language Modeling with PathwaysMain thread: Language modellingModel
- OPT-175BOPT: Open Pre-trained Transformer Language ModelsModel
- Emergent abilitiesEmergent Abilities of Large Language ModelsResearch paper
- BLOOMBLOOM: A 176B-Parameter Open-Access Multilingual Language ModelMain thread: Language modellingModel
- Flan-T5 / Flan-PaLMScaling Instruction-Finetuned Language ModelsTraining method
- Self-InstructSelf-Instruct: Aligning Language Models with Self-Generated InstructionsTraining method
2023
- LLaMALLaMA: Open and Efficient Foundation Language ModelsMain thread: Language modellingModel
- GPT-4GPT-4 Technical ReportModel
- PythiaPythia: A Suite for Analyzing Large Language Models Across Training and ScalingMain thread: Language modellingModel
- Emergence mirageAre Emergent Abilities of Large Language Models a Mirage?Research paper
- FalconFalcon / RefinedWeb modelsMain thread: Language modellingModel
- Process supervisionLet's Verify Step by StepResearch paper
- RefinedWebThe RefinedWeb Dataset for Falcon LLMMain thread: Language modellingDataset
- phi-1Textbooks Are All You NeedMain thread: Language modellingModel
- Gemini 1.0Gemini: A Family of Highly Capable Multimodal ModelsModel
- Mixtral 8x7BMixtral of ExpertsMain thread: Language modellingModel
2024
- DeepSeek LLMDeepSeek LLM: Scaling Open-Source Language Models with LongtermismResearch paper
- DolmaDolma: an Open Corpus of Three Trillion TokensDataset
- Gemini 1.5Gemini 1.5Model
- Chatbot ArenaChatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceBenchmark
- Llama 3Llama 3Model
- Phi-3Phi-3 Technical ReportModel
- FineWebThe FineWeb Datasets: Decanting the Web for the Finest Text Data at ScaleDataset
- Llama 3.1 405BThe Llama 3 Herd of ModelsModel
- Test-time computeScaling LLM Test-Time Compute OptimallyResearch paper
- DeepSeek-V3DeepSeek-V3 Technical ReportModel
2025
- GPT-4.5GPT-4.5Main thread: Language modellingModel