Language modelling
68 milestones sit on this thread. For 68 of them it is the main thread. The others also belong to another thread, which is named under them.
1948
- Information theoryA Mathematical Theory of CommunicationResearch paper
1990
- Statistical MTA Statistical Approach to Machine TranslationResearch paper
- Recurrent networksFinding Structure in TimeResearch paper
1992
- Class-based n-gramsClass-Based n-gram Models of Natural LanguageResearch paper
1997
- LSTMLong Short-Term MemoryResearch paper
2003
- Neural LMA Neural Probabilistic Language ModelResearch paper
2010
- RNN LMRecurrent neural network based language modelResearch paper
2011
- NLP from scratchNatural Language Processing (Almost) from ScratchResearch paper
2014
- GRU / encoder-decoderLearning Phrase Representations using RNN Encoder-DecoderResearch paper
- AttentionNeural Machine Translation by Jointly Learning to Align and TranslateArchitectural idea
- Seq2seqSequence to Sequence Learning with Neural NetworksArchitectural idea
2015
- Luong attentionEffective Approaches to Attention-based Neural Machine TranslationResearch paper
- BPE tokenizationNeural Machine Translation of Rare Words with Subword UnitsTechnique
- LM pretrainingSemi-supervised Sequence LearningTraining method
2016
- Layer normLayer NormalizationTechnique
- GNMTGoogle's Neural Machine Translation SystemTechnical report
2017
- ConvS2SConvolutional Sequence to Sequence LearningArchitectural idea
- TransformerAttention Is All You NeedArchitectural idea
- CoVeLearned in Translation: Contextualized Word VectorsResearch paper
2018
- ULMFiTUniversal Language Model Fine-tuning for Text ClassificationTraining method
- ELMoDeep Contextualized Word RepresentationsModel
- GPT-1Improving Language Understanding by Generative Pre-TrainingModel
- SentencePieceSentencePiece: A simple and language independent subword tokenizerInfrastructure
- BERTBERT: Pre-training of Deep Bidirectional TransformersModel
2019
- Transformer-XLTransformer-XL: Attentive Language Models Beyond a Fixed-Length ContextArchitectural idea
- GPT-2Language Models are Unsupervised Multitask LearnersModel
- WebTextWebTextDataset
- Sparse TransformerGenerating Long Sequences with Sparse TransformersArchitectural idea
- RoBERTaRoBERTa: A Robustly Optimized BERT Pretraining ApproachModel
- C4C4: Colossal Clean Crawled CorpusDataset
- T5Exploring the Limits of Transfer Learning with a Unified Text-to-Text TransformerModel
- BARTBART: Denoising Sequence-to-Sequence Pre-trainingModel
- Multi-query attentionFast Transformer Decoding: One Write-Head is All You NeedArchitectural idea
2020
- Scaling lawsScaling Laws for Neural Language ModelsResearch paper
- LongformerLongformer: The Long-Document TransformerArchitectural idea
- RAGRetrieval-Augmented Generation for Knowledge-Intensive NLP TasksResearch paper
- GPT-3Language Models are Few-Shot LearnersModel
- BigBirdBig Bird: Transformers for Longer SequencesArchitectural idea
- Vision TransformerAn Image is Worth 16x16 WordsModel
- The PileThe Pile: An 800GB Dataset of Diverse TextDataset
2021
- RoPERoFormer: Enhanced Transformer with Rotary Position EmbeddingArchitectural idea
- LoRALoRA: Low-Rank Adaptation of Large Language ModelsTraining method
- Codex / HumanEvalEvaluating Large Language Models Trained on CodeModel
- ALiBiTrain Short, Test Long: Attention with Linear BiasesArchitectural idea
- GopherScaling Language Models: Methods, Analysis and Insights from Training GopherModel
- RETROImproving Language Models by Retrieving from Trillions of TokensResearch paper
2022
- AlphaCodeCompetition-Level Code Generation with AlphaCodeModel
- ChinchillaTraining Compute-Optimal Large Language ModelsResearch paper
- PaLMPaLM: Scaling Language Modeling with PathwaysModel
- FlashAttentionFlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessTechnique
- BLOOMBLOOM: A 176B-Parameter Open-Access Multilingual Language ModelModel
- WhisperRobust Speech Recognition via Large-Scale Weak SupervisionModel
2023
- LLaMALLaMA: Open and Efficient Foundation Language ModelsModel
- PythiaPythia: A Suite for Analyzing Large Language Models Across Training and ScalingModel
- Grouped-query attentionGQA: Training Generalized Multi-Query Transformer ModelsArchitectural idea
- FalconFalcon / RefinedWeb modelsModel
- RefinedWebThe RefinedWeb Dataset for Falcon LLMDataset
- phi-1Textbooks Are All You NeedModel
- Llama 2Llama 2: Open Foundation and Fine-Tuned Chat ModelsModel
- Mistral 7BMistral 7BModel
- MambaMamba: Linear-Time Sequence Modeling with Selective State SpacesArchitectural idea
- Mixtral 8x7BMixtral of ExpertsModel
2024
- DeepSeekMoEDeepSeekMoE: Towards Ultimate Expert SpecializationArchitectural idea
- OLMoOLMo: Accelerating the Science of Language ModelsModel
2025
- DeepSeek-R1DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningModel
- GPT-4.5GPT-4.5Model
- Kimi K2Kimi K2Model
- DeepSeek-V3.2DeepSeek-V3.2 with DeepSeek Sparse AttentionModel