Language model
A model that gives a probability to each possible next word or token, given the text so far. Repeating this one token at a time lets it score a whole text or write new text.
Introduced by
Information theory, 1948
Used by
- Neural LM, 2003
- LM pretraining, 2015
- GPT-1, 2018
- GPT-2, 2019
- Scaling laws, 2020
- GPT-3, 2020
- Chinchilla, 2022
- LLaMA, 2023