Language Models are Unsupervised Multitask Learners
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever
- Date
- 14 February 2019
- What the date marks
- Technical report
- Kind
- Model
- Organisation
- OpenAI
- Access
- MIT
- Other dates
- 2019-11-05 full 1.5B release
- Learning phases
- 2. How LLMs are trained; 8. Training from scratch
A turning point: 6 later milestones descend directly from it, and its descendants reach 10 of the 10 threads. How levels are decided
Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from cdn.openai.com (OpenAI technical report, February 2019).
What it changed
It showed that a large language model trained only to predict the next word on varied web text starts to do tasks it was never trained for, such as answering questions and translating, when the task is written into its input. This is called zero-shot task transfer.
The problem before it
The best systems of 2018 were pretrained on plain text and then fine-tuned on a labelled dataset for each task (GPT-1, BERT). Each new task still needed its own labelled data and its own training run.
The authors describe these systems as narrow experts. They can fail when the data changes slightly. Collecting a labelled dataset for every task does not scale.
The idea
The mechanism. Train one language model on a very large and varied text collection, and give it enough capacity. The authors’ bet is that natural text already contains many demonstrations of tasks. For example, web pages contain sentences followed by their translations, and questions followed by answers. To predict such text well, the model must learn to do these tasks.
So a task becomes a text prompt. To translate, give the model a few lines like english sentence = french sentence, then a new English sentence and =, and let it continue. To summarise, add TL;DR: after an article. No weights change.
The work had three parts:
- WebText, a new dataset. The authors collected the pages linked from Reddit posts that received at least 3 karma, a rough signal that people found them useful. This gave 45 million links and, after cleaning, about 8 million documents, or 40 GB of text. Wikipedia pages were removed, because many test sets come from Wikipedia.
- Byte-level BPE. BPE builds a vocabulary of frequent pieces of words. The authors run it on bytes, so any text can be represented and no word is ever “unknown”. They stop merges across character types, such as letters with punctuation. The vocabulary has 50,257 tokens.
- A larger Transformer decoder. It follows GPT-1, with small changes: layer normalisation moves to the input of each block, and the context grows from 512 to 1,024 tokens. The largest of four models, called GPT-2, has 1.5 billion parameters, more than ten times GPT-1.
The notation. The model estimates p(x) = Π_i p(s_i | s_1, ..., s_(i-1)), the probability of a text as a product of next-token probabilities. A task is framed as p(output | input, task), with the task given as text.
What it showed
Reported in the paper. It is a technical report from OpenAI, not a peer-reviewed paper. All results are zero-shot: no training or fine-tuning on the test task.
- Language modelling. GPT-2 set new best results on 7 of 8 standard language-modelling datasets without training on any of them. It was still clearly worse on the One Billion Word benchmark.
- LAMBADA, which needs long context to guess a sentence’s last word. Perplexity fell from the earlier best of 99.8 to 8.6. Accuracy rose from 19% to 52.66%, and to 63.24% with a simple filter on the answers.
- Children’s Book Test. 93.3% on common nouns and 89.1% on named entities, both new best results.
- Reading comprehension (CoQA). 55 F1. This matched or beat 3 of 4 baseline systems that were trained on more than 127,000 examples. The best supervised system was near the human score of 89 F1.
- Summarisation. With
TL;DR:, the summaries scored only a little above choosing 3 random sentences from the article. Without the TL;DR: hint, the score fell by 6.4 points, so the hint does steer the model.
- Translation. 5 BLEU for English to French and 11.5 BLEU for French to English. WebText contains only about 10 MB of French.
- Factual questions. It answered 4.1% of Natural Questions exactly. On the 1% of questions where it was most confident, it was right 63.1% of the time.
- Size. Performance improved steadily with model size on every task, and even the largest model still underfit WebText.
Limits, and what came later
The limits come from the report:
- The authors say zero-shot GPT-2 is “still far from use-able” for practical tasks. Summarisation and translation are rudimentary.
- In reading comprehension, it often uses simple shortcuts, such as answering a “who” question with a name from the document.
- Some test text appears in the training data. Test sets overlapped with WebText by 1% to 6% of their 8-word sequences, 3.2% on average.
- It reads in one direction, and the authors leave open whether fine-tuning can match bidirectional models like BERT.
The next steps built directly on it. GPT-3 scaled the same architecture much further. Scaling laws turned its “bigger is better” trend into a measured law. RLHF for LMs fine-tuned GPT-2 with human preferences. CLIP used a GPT-style text encoder.
What to know first
- The Transformer decoder and GPT-1.
- What a language model does, and what perplexity measures: how surprised the model is by the real text. Lower is better.
- BPE tokenization.
- The idea of zero-shot use: doing a task with no examples of it in training.
How to read it
The report is 24 pages, and about half is example outputs.
- Read sections 1 and 2, especially 2.1 on WebText and 2.2 on byte-level BPE.
- Look at Figure 1, which shows each task improving with model size.
- Read section 3, task by task. Each part is short. Start with 3.1 and Table 3.
- Read section 4 on overlap between training and test data.
- Read section 6 (discussion) for the authors’ own limits.
Skim the sample texts at the end. Expect about 1.5 hours for the steps above (an estimate).
Links
- The report: cdn.openai.com
- Code: openai/gpt-2. The report describes it as preliminary code for using the small model. The repository is archived on GitHub.
- Video: none is linked here yet.