Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from cdn.openai.com (OpenAI paper, June 2018, marked "Preprint. Work in progress.").
What it changed
It showed that one Transformer, first trained to predict the next word on a large set of books and then fine-tuned with almost no changes, beats models built specially for each task on 9 of 12 language understanding tasks.
The problem before it
Labelled data for each task is scarce, but plain text is plentiful. The most common way to use plain text was pretrained word embeddings, which carry information about single words only.
The authors name two open questions about going further. First, it was unclear which training goal teaches the most useful knowledge: language modelling, translation and other goals each won on different tasks. Second, there was no agreed way to move that knowledge into a new task. Methods changed the model’s design for each task, added complex training schemes, or added extra training goals.
The closest earlier work, LM pretraining and ULMFiT, pretrained a language model and then fine-tuned it, but only for text classification. Both used LSTMs, which the authors say restricts how far back the model can look. Other work, such as ELMo, used a pretrained model’s internal values as extra features for a separate model, which adds many new parameters for each task.
The idea
The mechanism. Train in two stages, with one model:
- Pretraining. Train a Transformer decoder as a language model on BooksCorpus, over 7,000 unpublished books. The authors chose books because they have long stretches of connected text, so the model must learn to use long context.
- Fine-tuning. For each task, add one linear output layer and train the whole model on the task’s labelled data. During this stage, keep the language-modelling goal as a second, smaller goal.
The new part is how the input is shaped. Many tasks have structured inputs, such as two sentences, or a document, a question and several answers. Instead of changing the model, the authors turn each input into one sequence of tokens, with start, delimiter and end tokens:
- Entailment: the premise, a delimiter, then the hypothesis.
- Similarity: both orders of the two sentences, processed separately and added together, because the pair has no natural order.
- Multiple choice: the document and question, a delimiter, then one answer. Each answer is scored separately, and a softmax compares them.
So the only new parameters for a task are the output layer and the vectors for the delimiter tokens. This is pretraining and fine-tuning with one unchanged model.
The model has 12 layers, 768-value states and 12 attention heads, uses learned position vectors, and has a byte pair encoding vocabulary built with 40,000 merges. It was pretrained for 100 passes over the data on sequences of 512 tokens.
The notation. Pretraining maximises L1(U) = Σ_i log P(u_i | u_(i-k), ..., u_(i-1)), the log-probability of each token given the k tokens before it. Fine-tuning predicts a label from the last layer’s output for the last token: P(y | x_1, ..., x_m) = softmax(h_l^m W_y). The full fine-tuning goal is L3 = L2 + λ · L1, where L2 is the task goal and λ was 0.5.
What it showed
Reported in the paper. The paper is marked as a preprint, a work in progress. All comparisons are with the best published results at the time.
- Overall. New best results on 9 of the 12 datasets studied, often beating ensembles of several models.
- Commonsense reasoning (Story Cloze). 86.5% accuracy, 8.9 points above the earlier best of 77.6%.
- Reading comprehension (RACE, school exam questions). 59.0% accuracy, 5.7 points above the earlier best of 53.3%.
- Entailment. 82.1% on MultiNLI (matched), 1.5 points above the earlier best. It was better on 4 of 5 entailment datasets. On RTE, a small dataset of 2,490 examples, it scored 56.0%, below the earlier best of 61.7%.
- Grammar judgements (CoLA). 45.4, against an earlier best of 35.0 (Matthews correlation, where 0 is chance).
- GLUE, a benchmark that combines several language understanding tasks. The model scored 72.8 overall, against an earlier best of 68.9 in the paper’s table, a gain of 3.9 points. The abstract and introduction instead describe an absolute improvement of 5.5% on GLUE. Both numbers are given here.
- Pretraining matters most. Without pretraining, the average score over 8 tasks fell from 74.7 to 59.9.
- The Transformer matters. An LSTM in the same setup averaged 69.1, 5.6 points lower.
- The second goal is mixed. Without the language-modelling goal during fine-tuning, the average was 75.0, slightly higher than 74.7. It helped on the large entailment datasets and on QQP, but not on the small datasets.
- Zero-shot. The authors also used the pretrained language model directly on four tasks, with simple scoring rules and no fine-tuning. Its scores rose steadily during pretraining, and varied less than those of an LSTM.
Limits, and what came later
The limits come from the paper:
- It was weaker on the small RTE dataset. The authors suggest training on several tasks at once might help, but did not try it.
- The language-modelling goal during fine-tuning helped large datasets and did not help small ones.
Two more limits follow from the design. The paper does not discuss them. The model reads only from left to right, and each task still needs labelled data and its own fine-tuning run.
Later work built on it directly. BERT contrasted itself with GPT’s left-to-right language model and pretrained in both directions. GPT-2 scaled up the same architecture and dropped fine-tuning. WebText was built to scale up GPT-style pretraining.
What to know first
How to read it
The paper is 12 pages, including 4 pages of references.
- Read section 1 for the two open questions.
- Read section 3 with Figure 1. The right side of the figure shows how each task’s input is shaped. This is the key idea.
- Read section 4.2 with Tables 2, 3 and 4 for the results.
- Read section 5, especially the ablations in Table 5.
Expect about 1 hour for the parts above (an estimate).
Links
- The paper: cdn.openai.com
- Code and model: openai/finetune-transformer-lm. The paper does not name it; the repository describes itself as the code and model for this paper. It is archived on GitHub.
- Video: none is linked here yet.