NLP Lineage

200 milestones, 1948 to 2026

Milestones

Papers, models, methods, datasets and events, in date order. Open one to see what it changed, its source, and what came before and after it.

1948

1990

1991

1992

1997

  • LSTMLong Short-Term MemoryResearch paper

2003

  • Neural LMA Neural Probabilistic Language ModelResearch paper

2010

  • RNN LMRecurrent neural network based language modelResearch paper

2011

2013

  • word2vecEfficient Estimation of Word Representations in Vector SpaceTechnique

2014

  • GRU / encoder-decoderLearning Phrase Representations using RNN Encoder-DecoderResearch paper
  • AttentionNeural Machine Translation by Jointly Learning to Align and TranslateArchitectural idea
  • Seq2seqSequence to Sequence Learning with Neural NetworksArchitectural idea
  • GloVeGloVe: Global Vectors for Word RepresentationTechnique
  • AdamAdam: A Method for Stochastic OptimizationTraining method

2015

2016

  • Layer normLayer NormalizationTechnique
  • GNMTGoogle's Neural Machine Translation SystemTechnical report

2017

  • Sparse MoEOutrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts LayerArchitectural idea
  • FAISSBillion-scale similarity search with GPUsInfrastructure
  • ConvS2SConvolutional Sequence to Sequence LearningArchitectural idea
  • RLHF originDeep Reinforcement Learning from Human PreferencesTraining method
  • TransformerAttention Is All You NeedArchitectural idea
  • CoVeLearned in Translation: Contextualized Word VectorsResearch paper
  • Mixed precisionMixed Precision TrainingTraining method

2018

  • ULMFiTUniversal Language Model Fine-tuning for Text ClassificationTraining method
  • ELMoDeep Contextualized Word RepresentationsModel
  • GPT-1Improving Language Understanding by Generative Pre-TrainingModel
  • SentencePieceSentencePiece: A simple and language independent subword tokenizerInfrastructure
  • BERTBERT: Pre-training of Deep Bidirectional TransformersModel

2019

  • Transformer-XLTransformer-XL: Attentive Language Models Beyond a Fixed-Length ContextArchitectural idea
  • GPT-2Language Models are Unsupervised Multitask LearnersModel
  • WebTextWebTextDataset
  • Sparse TransformerGenerating Long Sequences with Sparse TransformersArchitectural idea
  • RoBERTaRoBERTa: A Robustly Optimized BERT Pretraining ApproachModel
  • Sentence-BERTSentence-BERT: Sentence Embeddings using Siamese BERT-NetworksResearch paper
  • Megatron-LMMegatron-LM: Training Multi-Billion Parameter Language Models Using Model ParallelismInfrastructure
  • RLHF for LMsFine-Tuning Language Models from Human PreferencesTraining method
  • ZeRO / DeepSpeedZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsInfrastructure
  • C4C4: Colossal Clean Crawled CorpusDataset
  • T5Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerModel
  • BARTBART: Denoising Sequence-to-Sequence Pre-trainingModel
  • Multi-query attentionFast Transformer Decoding: One Write-Head is All You NeedArchitectural idea

2020

  • Scaling lawsScaling Laws for Neural Language ModelsResearch paper
  • Dense retrievalDense Passage Retrieval for Open-Domain Question AnsweringResearch paper
  • LongformerLongformer: The Long-Document TransformerArchitectural idea
  • RAGRetrieval-Augmented Generation for Knowledge-Intensive NLP TasksResearch paper
  • GPT-3Language Models are Few-Shot LearnersModel
  • GShardGShard: Scaling Giant Models with Conditional ComputationInfrastructure
  • BigBirdBig Bird: Transformers for Longer SequencesArchitectural idea
  • RLHF for summarisationLearning to Summarize from Human FeedbackTraining method
  • MMLUMeasuring Massive Multitask Language UnderstandingDataset
  • Vision TransformerAn Image is Worth 16x16 WordsModel
  • The PileThe Pile: An 800GB Dataset of Diverse TextDataset

2021

  • Switch TransformerSwitch Transformers: Scaling to Trillion Parameter ModelsModel
  • CLIPLearning Transferable Visual Models From Natural Language SupervisionModel
  • RoPERoFormer: Enhanced Transformer with Rotary Position EmbeddingArchitectural idea
  • LoRALoRA: Low-Rank Adaptation of Large Language ModelsTraining method
  • CopilotGitHub CopilotProduct
  • Codex / HumanEvalEvaluating Large Language Models Trained on CodeModel
  • ALiBiTrain Short, Test Long: Attention with Linear BiasesArchitectural idea
  • FLANFinetuned Language Models Are Zero-Shot LearnersTraining method
  • T0Multitask Prompted Training Enables Zero-Shot Task GeneralizationModel
  • HHH assistantA General Language Assistant as a Laboratory for AlignmentResearch paper
  • GopherScaling Language Models: Methods, Analysis and Insights from Training GopherModel
  • RETROImproving Language Models by Retrieving from Trillions of TokensResearch paper
  • WebGPTWebGPT: Browser-assisted question-answering with human feedbackResearch paper

2022

  • LaMDALaMDA: Language Models for Dialog ApplicationsModel
  • Chain of thoughtChain-of-Thought Prompting Elicits Reasoning in Large Language ModelsTechnique
  • MT-NLGMegatron-Turing NLG 530BModel
  • AlphaCodeCompetition-Level Code Generation with AlphaCodeModel
  • InstructGPTTraining language models to follow instructions with human feedbackTraining method
  • Self-consistencySelf-Consistency Improves Chain of Thought ReasoningTechnique
  • STaRSTaR: Bootstrapping Reasoning With ReasoningTraining method
  • ChinchillaTraining Compute-Optimal Large Language ModelsResearch paper
  • PaLMPaLM: Scaling Language Modeling with PathwaysModel
  • HH-RLHFTraining a Helpful and Harmless Assistant with RLHFTraining method
  • FlamingoFlamingo: a Visual Language Model for Few-Shot LearningModel
  • OPT-175BOPT: Open Pre-trained Transformer Language ModelsModel
  • Zero-shot CoTLarge Language Models are Zero-Shot ReasonersTechnique
  • FlashAttentionFlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTechnique
  • Emergent abilitiesEmergent Abilities of Large Language ModelsResearch paper
  • BLOOMBLOOM: A 176B-Parameter Open-Access Multilingual Language ModelModel
  • LLM.int8()LLM.int8(): 8-bit Matrix Multiplication for Transformers at ScaleTechnique
  • WhisperRobust Speech Recognition via Large-Scale Weak SupervisionModel
  • ReActReAct: Synergizing Reasoning and Acting in Language ModelsTechnique
  • Flan-T5 / Flan-PaLMScaling Instruction-Finetuned Language ModelsTraining method
  • GPTQGPTQ: Accurate Post-Training Quantization for Generative Pre-trained TransformersTechnique
  • ChatGPTChatGPTProduct
  • GPT-3.5GPT-3.5 seriesModel
  • Speculative decodingFast Inference from Transformers via Speculative DecodingTechnique
  • Constitutional AIConstitutional AI: Harmlessness from AI FeedbackTraining method
  • Self-InstructSelf-Instruct: Aligning Language Models with Self-Generated InstructionsTraining method

2023

  • ToolformerToolformer: Language Models Can Teach Themselves to Use ToolsResearch paper
  • LLaMALLaMA: Open and Efficient Foundation Language ModelsModel
  • llama.cppllama.cppInfrastructure
  • AlpacaStanford AlpacaModel
  • ClaudeClaudeProduct
  • GPT-4GPT-4 Technical ReportModel
  • ReflexionReflexion: Language Agents with Verbal Reinforcement LearningResearch paper
  • AutoGPTAutoGPTInfrastructure
  • PythiaPythia: A Suite for Analyzing Large Language Models Across Training and ScalingModel
  • LLaVAVisual Instruction TuningModel
  • Emergence mirageAre Emergent Abilities of Large Language Models a Mirage?Research paper
  • 100K context100K context windowsProduct
  • Tree of thoughtsTree of Thoughts: Deliberate Problem Solving with Large Language ModelsTechnique
  • Grouped-query attentionGQA: Training Generalized Multi-Query Transformer ModelsArchitectural idea
  • QLoRAQLoRA: Efficient Finetuning of Quantized LLMsTechnique
  • FalconFalcon / RefinedWeb modelsModel
  • DPODirect Preference OptimizationTraining method
  • Process supervisionLet's Verify Step by StepResearch paper
  • AWQAWQ: Activation-aware Weight QuantizationTechnique
  • RefinedWebThe RefinedWeb Dataset for Falcon LLMDataset
  • Function callingFunction calling in the OpenAI APIProduct
  • phi-1Textbooks Are All You NeedModel
  • Position interpolationExtending Context Window of Large Language Models via Positional InterpolationTechnique
  • Lost in the middleLost in the Middle: How Language Models Use Long ContextsResearch paper
  • Llama 2Llama 2: Open Foundation and Fine-Tuned Chat ModelsModel
  • vLLMEfficient Memory Management for LLM Serving with PagedAttentionInfrastructure
  • GPT-4VGPT-4V(ision) System CardModel
  • Mistral 7BMistral 7BModel
  • SWE-benchSWE-bench: Can Language Models Resolve Real-World GitHub Issues?Dataset
  • MambaMamba: Linear-Time Sequence Modeling with Selective State SpacesArchitectural idea
  • Gemini 1.0Gemini: A Family of Highly Capable Multimodal ModelsModel
  • Mixtral 8x7BMixtral of ExpertsModel

2024

  • DeepSeek LLMDeepSeek LLM: Scaling Open-Source Language Models with LongtermismResearch paper
  • DeepSeekMoEDeepSeekMoE: Towards Ultimate Expert SpecializationArchitectural idea
  • DolmaDolma: an Open Corpus of Three Trillion TokensDataset
  • OLMoOLMo: Accelerating the Science of Language ModelsModel
  • GRPODeepSeekMath: Pushing the Limits of Mathematical ReasoningTraining method
  • Gemini 1.5Gemini 1.5Model
  • GemmaGemma: Open Models Based on Gemini Research and TechnologyModel
  • Claude 3The Claude 3 model familyModel
  • Chatbot ArenaChatbot Arena: An Open Platform for Evaluating LLMs by Human PreferenceBenchmark
  • DevinDevinProduct
  • Llama 3Llama 3Model
  • Phi-3Phi-3 Technical ReportModel
  • DeepSeek-V2 / MLADeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language ModelModel
  • SWE-agentSWE-agent: Agent-Computer Interfaces Enable Automated Software EngineeringResearch paper
  • GPT-4oGPT-4oModel
  • Claude 3.5 SonnetClaude 3.5 SonnetModel
  • FineWebThe FineWeb Datasets: Decanting the Web for the Finest Text Data at ScaleDataset
  • FlashAttention-3FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisionInfrastructure
  • Qwen2Qwen2 Technical ReportModel
  • Llama 3.1 405BThe Llama 3 Herd of ModelsModel
  • Test-time computeScaling LLM Test-Time Compute OptimallyResearch paper
  • o1OpenAI o1Model
  • Computer useClaude computer useProduct
  • MCPModel Context ProtocolInfrastructure
  • Gemini 2.0Gemini 2.0Model
  • o3 announcedOpenAI o3 announcementModel
  • DeepSeek-V3DeepSeek-V3 Technical ReportModel

2025

2026