Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 4 (September 2023) of the paper published in the Journal of Machine Learning Research (2020)). C4 was introduced in the T5 paper. This summary covers the parts about C4, mainly sections 2.2 and 3.4.
What it changed
It gave researchers a large, cleaned, public dataset of English web text, about 750 GB, and measured how the choice of pretraining data affects results. C4 became a standard pretraining corpus and a model for how later web datasets were built.
The problem before it
Pretraining needs a lot of plain text. The web is the largest source. Common Crawl, a public web archive, adds about 20 TB of extracted text every month.
But most of that text is not natural language. It is menus, error messages, repeated boilerplate, code, placeholder text, and offensive content. Earlier pretraining datasets were often small, limited to one domain such as news or Wikipedia, or not public. New datasets were usually introduced as a side part of a new model, so they were rarely compared with each other.
The idea
The mechanism. Start from one month of Common Crawl (April 2019), and keep only text that looks like clean English prose. The paper lists simple rules, called heuristics:
- Keep only lines that end in a full stop, exclamation mark, question mark or closing quotation mark.
- Drop pages with fewer than 3 sentences. Keep only lines with at least 5 words.
- Drop any page that contains a word from a public list of bad words.
- Drop lines that mention “Javascript”, and pages with “lorem ipsum” placeholder text.
- Drop any page with a curly bracket
{, as a simple test for program code.
- Remove citation marks such as
[1], and lines with cookie and privacy-policy notices.
- Remove duplicates: keep only one copy of any 3-sentence span that appears more than once.
- Keep only pages that a language detector classifies as English with a probability of at least 0.99.
The result is the Colossal Clean Crawled Corpus, C4. It was released in TensorFlow Datasets, together with the other variants the paper tests.
The test. The authors pretrained the same model on several datasets and compared the fine-tuned results: C4, C4 without the filters, news-like C4, WebText-like C4, Wikipedia, and Wikipedia with a books corpus. They also tested what happens when a smaller dataset is repeated many times during pretraining.
The notation. There are no equations. The paper’s baseline pretrains on 2^35 tokens, about 34 billion.
What it showed
Reported in the paper. It was published in the peer-reviewed Journal of Machine Learning Research in 2020. The scores are on standard benchmarks; higher is better.
- Cleaning matters. The unfiltered version (6.1 TB) was the worst in every task. For example, its GLUE score was 81.46, against 83.28 for C4 (745 GB).
- In-domain data helps some tasks. Wikipedia with the books corpus scored 73.24 on SuperGLUE, against 71.36 for C4. Most of this gain came from one task, MultiRC, whose texts come mostly from fiction. There, the exact-match score rose from 25.78 to 50.93.
- Narrow data is small data. Filtering by Reddit links, as WebText does, gave a dataset about 40 times smaller than C4, even though it started from 12 times more Common Crawl data.
- Repeating data hurts. Using only the first 2^23 tokens of C4, repeated 4,096 times, lowered GLUE from 83.28 to 76.34. The training loss also fell, which suggests the model memorised the data. Repeating the data 64 times had only a small effect.
The authors conclude that a large and varied dataset like C4 is the better choice for general language understanding.
Limits, and what came later
The limits come from the paper and from the rules themselves:
- C4 is English only, and built from a single month of the web.
- The rules are simple heuristics. The bad-word rule removes a whole page if any listed word appears, whatever the context. The curly-bracket rule removes every page with code.
- For some tasks, a smaller dataset from the task’s own domain beat C4.
C4 was the pretraining data for T5. Later web corpora set out to improve on C4-style filtering, including The Pile, RefinedWeb and Dolma.
What to know first
- What pretraining is, and why it needs a lot of text. See LM pretraining and BERT.
- What a benchmark score like GLUE measures: performance on a fixed set of test tasks.
- WebText, the Reddit-filtered dataset used for GPT-2, which the paper compares with.
How to read it
The T5 paper is 67 pages. For C4, read only these parts:
- The abstract.
- Section 2.2, about two pages. It describes how C4 was built.
- Section 3.4.1 with Table 8, which compares the datasets.
- Section 3.4.2 with Table 9, about repeating data.
- The “Data sets” paragraph in section 4.1 (“Takeaways”), the paper’s conclusion about C4.
Expect about 45 minutes for these parts (an estimate). The rest of the paper is the T5 study.
Links
- The paper: arXiv 1910.10683
- The dataset: C4 in TensorFlow Datasets, as named in the paper.
- Code: the paper releases its code. Its link is not repeated here because this summary covers only the dataset.
- Video: none is linked here yet.