Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 2 (May 2019)).
What it changed
It pretrained a Transformer to use the words on both sides of every word at once, and then fine-tuned that one model to set new best results on 11 language tasks. After BERT, pretraining and then fine-tuning became the default way to build language understanding systems.
The problem before it
Pretraining a language model on plain text, and then fine-tuning it on a labelled task, already worked (LM pretraining, ULMFiT, GPT-1). But these models read in one direction. A left-to-right model predicts each word from the words before it, so each word’s representation only sees its left side.
The authors argue this is a serious limit. In question answering, for example, the meaning of a word often depends on the words after it. ELMo joined a left-to-right and a right-to-left model, but each half was trained alone, so no single layer saw both sides together.
A normal language model cannot simply look both ways. If every word could see every other word, the model could read the word it is meant to predict.
The idea
The mechanism. BERT is a Transformer encoder. Every position attends to every other position, in both directions. Two pretraining tasks make this possible without letting the model see the answer:
- Masked language model (MLM). In each input, 15% of the word pieces are chosen at random. The model must predict the original word piece at each chosen position from the rest of the text. This is like a fill-in-the-blank test, called a cloze task. Of the chosen positions, 80% are replaced by a special
[MASK] token, 10% by a random word piece, and 10% stay unchanged. The mix exists because [MASK] never appears during fine-tuning.
- Next sentence prediction (NSP). The input is two text spans, A and B. Half of the time B really follows A; half of the time it is a random span. The model predicts which case it is. This teaches it about relations between sentences, which tasks like question answering need.
The input always starts with a [CLS] token, and a [SEP] token separates the two spans. Each token’s input is the sum of three vectors: the word piece, the span it belongs to (A or B), and its position.
Fine-tuning. For each task, add one small output layer and train all the weights on the labelled data. For classification, the output layer reads the final vector of [CLS]. For question answering, it learns a start vector and an end vector, and scores each position as the start or end of the answer.
The sizes. BERT-base has 12 layers, a hidden size of 768 and 12 attention heads: 110 million parameters, the same size as GPT-1. BERT-large has 24 layers, a hidden size of 1,024 and 16 heads: 340 million parameters.
What it showed
Reported in the paper. We read version 2 of the arXiv paper. The models were pretrained on BooksCorpus (800 million words) and English Wikipedia (2,500 million words).
- GLUE, a set of language understanding tasks. BERT-large scored 80.5 on the official leaderboard, 7.7 points above the earlier best. GPT-1 scored 72.8.
- MultiNLI, which asks whether one sentence follows from another. Accuracy rose to 86.7%, 4.6 points above the earlier best.
- SQuAD v1.1, question answering on Wikipedia. Test F1 rose to 93.2, 1.5 points above the earlier best. SQuAD v2.0, where some questions have no answer: test F1 rose to 83.1, 5.1 points above.
- Both directions matter. With the same data and settings, a left-to-right version did worse on every task tested. For example, on SQuAD it scored 77.8 F1, against 87.9 for the masked model. Both were trained without next sentence prediction, so only the direction differs.
- Bigger is better, even on small tasks. Larger models improved accuracy on every task tested, even MRPC, which has only 3,600 labelled examples.
- Cost. Pretraining BERT-large took 4 days on 16 Cloud TPUs (64 TPU chips). Fine-tuning is cheap: every result in the paper can be reproduced in at most 1 hour on one Cloud TPU.
Limits, and what came later
The limits come from the paper:
- The
[MASK] token appears in pretraining but never in fine-tuning. The 80, 10, 10 mix reduces this mismatch but does not remove it.
- Fine-tuning BERT-large was sometimes unstable on small datasets. The authors ran several random restarts and kept the best.
- BERT is an encoder. It reads text and represents it, but it is not built to generate text word by word.
Later work built on it. RoBERTa retrained BERT with a better recipe. Sentence-BERT and dense retrieval use BERT to turn text into vectors for search. T5 studied masked objectives like BERT’s in a larger comparison.
What to know first
- The Transformer, and especially self-attention.
- The pretraining and fine-tuning recipe from LM pretraining and GPT-1.
- What a language model does: predict a word from its context.
- Word pieces: rare words are split into smaller, common pieces. See BPE tokenization.
How to read it
The paper is 16 pages including the appendices.
- Read section 1 for the argument about one-direction models.
- Read section 3 with Figures 1 and 2. Section 3.1 explains the two pretraining tasks.
- Look at Tables 1 and 2 for the main results.
- Read section 5.1 with Table 5. It tests whether each part matters.
- Read appendix A.1 for the exact masking procedure.
Expect about 1.5 hours for the steps above (an estimate).
Links