Research thread · 1991 to 2026

Efficiency and infrastructure

44 milestones sit on this thread. For 19 of them it is the main thread. The others also belong to another thread, which is named under them.

1991

Residual connectionsDeep Residual Learning for Image RecognitionArchitectural idea

Sparse MoEOutrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerMain thread: Scale and dataArchitectural idea
FAISSBillion-scale similarity search with GPUsMain thread: Representation and retrievalInfrastructure
Mixed precisionMixed Precision TrainingTraining method

Transformer-XLTransformer-XL: Attentive Language Models Beyond a Fixed-Length ContextMain thread: Language modellingArchitectural idea
Sparse TransformerGenerating Long Sequences with Sparse TransformersMain thread: Language modellingArchitectural idea
Megatron-LMMegatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismMain thread: Scale and dataInfrastructure
ZeRO / DeepSpeedZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsMain thread: Scale and dataInfrastructure
Multi-query attentionFast Transformer Decoding: One Write-Head is All You NeedMain thread: Language modellingArchitectural idea

LongformerLongformer: The Long-Document TransformerMain thread: Language modellingArchitectural idea
GShardGShard: Scaling Giant Models with Conditional ComputationMain thread: Scale and dataInfrastructure
BigBirdBig Bird: Transformers for Longer SequencesMain thread: Language modellingArchitectural idea

Switch TransformerSwitch Transformers: Scaling to Trillion Parameter ModelsMain thread: Scale and dataModel
LoRALoRA: Low-Rank Adaptation of Large Language ModelsMain thread: Language modellingTraining method
ALiBiTrain Short, Test Long: Attention with Linear BiasesMain thread: Language modellingArchitectural idea

MT-NLGMegatron-Turing NLG 530BMain thread: Scale and dataModel
FlashAttentionFlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessMain thread: Language modellingTechnique
LLM.int8()LLM.int8(): 8-bit Matrix Multiplication for Transformers at ScaleTechnique
GPTQGPTQ: Accurate Post-Training Quantization for Generative Pre-trained TransformersTechnique
Speculative decodingFast Inference from Transformers via Speculative DecodingTechnique

llama.cppllama.cppInfrastructure
100K context100K context windowsProduct
Grouped-query attentionGQA: Training Generalized Multi-Query Transformer ModelsMain thread: Language modellingArchitectural idea
QLoRAQLoRA: Efficient Finetuning of Quantized LLMsTechnique
AWQAWQ: Activation-aware Weight QuantizationTechnique
Position interpolationExtending Context Window of Large Language Models via Positional InterpolationTechnique
Lost in the middleLost in the Middle: How Language Models Use Long ContextsMain thread: Representation and retrievalResearch paper
vLLMEfficient Memory Management for LLM Serving with PagedAttentionInfrastructure
Mistral 7BMistral 7BMain thread: Language modellingModel
MambaMamba: Linear-Time Sequence Modeling with Selective State SpacesMain thread: Language modellingArchitectural idea

DeepSeekMoEDeepSeekMoE: Towards Ultimate Expert SpecializationMain thread: Language modellingArchitectural idea
OLMoOLMo: Accelerating the Science of Language ModelsMain thread: Language modellingModel
Gemini 1.5Gemini 1.5Main thread: Scale and dataModel
DeepSeek-V2 / MLADeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelModel
FlashAttention-3FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisionInfrastructure
DeepSeek-V3DeepSeek-V3 Technical ReportMain thread: Scale and dataModel

Llama 4Llama 4Model
o4-miniOpenAI o4-miniModel
DeepSeek-V3.2DeepSeek-V3.2 with DeepSeek Sparse AttentionMain thread: Language modellingModel