Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 4 (September 2023) of the paper published in the Journal of Machine Learning Research (2020)).
What it changed
It turned every language task into the same form: text goes in, and text comes out. With one model, one training objective and one decoding method for all tasks, the authors could compare the parts of transfer learning fairly. Then they scaled up the best combination, called T5, to set new best results on 18 of 24 tasks.
The problem before it
By 2019, pretraining on plain text and then fine-tuning on a task was the standard method (BERT, GPT-2). But new work changed many things at once: the training objective, the model design, the pretraining data and the fine-tuning method.
Each task also had its own output format. A classifier needs a label, translation needs a sentence, and a similarity task needs a number. So a model like BERT needed a different output layer for each kind of task, and could not generate text such as translations or summaries.
With so much changing at once, it was hard to say which choices really helped.
The idea
The mechanism. The paper has two parts: a common format, and a large controlled study.
Text to text. Every task is written as an input text with a short task prefix, and the model learns to write the answer as text.
- Translation: input
translate English to German: That is good., output Das ist gut.
- Classification: the output is a label word, such as
entailment or not acceptable.
- Similarity scores from 1 to 5: the score is rounded to the nearest 0.2 and written as text, such as
3.8.
So the same model, loss, training method and decoding method work for every task.
The model. An encoder-decoder Transformer, close to the original design. Two small changes: layer normalisation is applied before each part of a block, and position is given by a learned number for each range of distances between two tokens, added to the attention scores.
The pretraining objective. The model learns to fill gaps, as in BERT. In the baseline, 15% of the tokens are dropped. Each run of dropped tokens is replaced by one special marker, and the model outputs only the missing pieces, each after its marker. For example, Thank you <X> me to your party <Y> week. becomes the target <X> for inviting <Y> last <Z>. Short targets make training faster.
The data. C4, about 750 GB of cleaned English web text, built for this paper.
The study. Start from a baseline model of 220 million parameters, pretrained on about 34 billion tokens. Then change one thing at a time and measure the effect on GLUE, SuperGLUE, SQuAD, CNN/Daily Mail summarisation and three translation tasks.
The notation. Training always uses the same loss: the cross-entropy of the target text, one token at a time. There are no task-specific equations.
What it showed
Reported in the paper. It was published in the peer-reviewed Journal of Machine Learning Research in 2020. The study results are on validation sets; the final results are on test sets, except SQuAD.
Findings from the study:
- Pretraining matters. Without it, the baseline scored 66.22 on GLUE instead of 83.28. English to French translation, which has a very large training set, barely changed.
- Encoder-decoder was best. It beat a decoder-only language model and a “prefix” language model of similar cost. Sharing the weights between encoder and decoder cost little.
- Filling gaps beat next-word prediction as a pretraining objective. The many variants of gap-filling all performed about the same, so the authors suggest choosing the one that trains fastest.
- Data: see C4. Cleaning the web text helped on every task, and repeating a small dataset many times hurt.
- Fine-tuning all weights was best. Methods that train only a few added weights, such as adapter layers, did worse.
- Training on many tasks at once was worse on most tasks than pretraining then fine-tuning. But pretraining on a mix of tasks, and then fine-tuning, came close to it.
- Scale. With 4 times the compute, a larger model trained for fewer steps usually beat a smaller model trained for longer.
The final models were trained on about 1 trillion tokens, with five sizes: from 60 million to 11 billion parameters. (The paper calls the second-largest “3B”, but gives its size as about 2.8 billion.)
- New best results on 18 of 24 tasks. GLUE: 90.3 average. SQuAD: 91.26 exact match, against an earlier best of 90.1.
- SuperGLUE rose from an earlier best of 84.6 to 88.9, close to the human score of 89.8.
- No new best result in translation. The authors suggest the English-only pretraining data is one reason.
- The study’s choices mattered beyond scale. A 220-million-parameter T5 beat the baseline trained on the same 1 trillion tokens, on every task.
Limits, and what came later
The limits come from the paper:
- One change at a time. The study cannot detect combined effects. For example, an objective might work best only in a larger model.
- One run per test. Most experiments ran once. The authors measured the spread over 10 runs of the baseline and assumed it held for the others. Small tasks such as CB and COPA varied a lot from run to run.
- English only. Pretraining used only English text, and the vocabulary covers a fixed set of languages.
- Cost. Large models are expensive to fine-tune and run. The authors call for cheaper methods, such as distillation, parameter sharing and conditional computation.
- Inefficient pretraining. They suspect gap-filling is a slow way to give a model general knowledge, since it needed 1 trillion tokens.
- Humans still lead on some tasks, such as COPA and WSC, where people score 100%.
T5 became a base for later work. Switch Transformer used the T5 setup to test a mixture-of-experts model. T0 and Flan-T5 fine-tuned T5 models on many tasks written as prompts or instructions. FLAN also fine-tuned on many tasks written as text instructions.
What to know first
- The Transformer encoder-decoder.
- Pretraining then fine-tuning, and BERT’s gap-filling objective (BERT).
- What benchmarks like GLUE and SuperGLUE measure: average scores over a fixed set of language-understanding tasks.
How to read it
The paper is 67 pages, with 44 pages of main text. It is long but easy to follow, because each experiment has the same structure.
- Read sections 1 and 2 with Figure 1: the text-to-text format, the model and the tasks.
- Read section 3.1 for the baseline, and Figure 2 for the objective.
- Read section 4.1, “Takeaways”. It summarises the whole study in two pages.
- Then read the parts of section 3 you care about. Each has a table; the flow chart in Figure 5 sums up the objectives.
- Read section 3.7 with Table 14 for the final models and results.
Expect about 3 hours for the steps above (an estimate).
Links