Drafted with Claude from the paper on 15 October 2026; reviewed by Zeeshan Arif. Read from arxiv.org (arXiv version 1 (February 2021); the paper was later published at ICML 2021).
What it changed
It trained an image encoder and a text encoder together on 400 million image and caption pairs from the internet, so that images and text share one space, and an image can then be classified into any categories that are written as text, with no task-specific training.
The problem before it
The best computer vision systems of the time were trained to predict a fixed set of categories, such as the 1,000 classes of ImageNet. Every new kind of visual concept needed new labelled data.
In language, pretraining on raw web text had removed much of this need. GPT-2 and GPT-3 could do tasks with little or no task-specific data. Learning images from the text that comes with them had been tried, but results were weak. The best earlier zero-shot result on ImageNet, from Visual N-Grams in 2017, was 11.5% accuracy. Those studies also trained on at most a few hundred thousand images, while the strongest vision models trained on millions to billions.
The idea
The mechanism. The authors first tried to train a model to write each image’s caption, word by word. This learned slowly: 3 times slower, at recognising ImageNet classes, than a simpler model that predicts which words appear in the caption. So they changed the task. CLIP (Contrastive Language-Image Pre-training) only has to pick which caption in a batch belongs to which image. That is 4 times more efficient again.
Training works like this. Take a batch of N images and their N captions. Turn each image into a vector with an image encoder, and each caption into a vector with a text encoder. Train both encoders so that the N real pairs have similar vectors, and the N² - N wrong pairs have different vectors. The batch had 32,768 pairs.
To classify an image with no training on the task, write each category as a sentence, such as “A photo of a dog.”, and turn each sentence into a vector. The category whose vector is closest to the image’s vector wins. In effect, the text encoder writes a new classifier from the category names.
The other parts:
- The data. A new dataset of 400 million image and text pairs from public internet sources, built by searching for 500,000 queries. The authors call it WIT (WebImageText).
- The image encoder. Either a modified ResNet or a Vision Transformer. Both were trained from scratch.
- The text encoder. A Transformer with the changes made in GPT-2: 63 million parameters, 12 layers, and a byte pair encoding vocabulary of 49,152 tokens. The text is cut at 76 tokens.
- Prompts. Writing categories as “A photo of a {label}.” instead of the bare label raised ImageNet accuracy by 1.3%. Averaging over 80 different prompts added 3.5% more.
The largest ResNet took 18 days to train on 592 V100 GPUs. The largest Vision Transformer took 12 days on 256 V100 GPUs.
The notation. For a batch, I_e and T_e are the normalised image and text vectors. The scores are logits = I_e T_eᵀ · exp(t), an N × N table of similarities, where t is a learned temperature. The loss is the average of two cross-entropy losses: one picks the right caption for each image (along the rows), and the other picks the right image for each caption (along the columns).
What it showed
Reported in the paper. We read version 1 of the arXiv paper, a preprint. The paper was later published at ICML 2021. Unless the paper says otherwise, “CLIP” means the best model, ViT-L/14 at 336-pixel resolution.
- ImageNet, zero-shot. 76.2% accuracy, against 11.5% for Visual N-Grams. This matches the original ResNet-50, which was trained on the 1.28 million ImageNet training images that CLIP never used.
- Against a supervised baseline. On 27 datasets, zero-shot CLIP beat a logistic regression classifier trained on ResNet-50 features on 16. It was far ahead on Stanford Cars (+28.9 points) and Food101 (+22.5), and far behind on satellite images (EuroSAT, -37.1) and on judging distance to a car (KITTI, -34.0).
- Against few-shot learning. Zero-shot CLIP matched a classifier trained on CLIP’s own features with 4 labelled examples per class, averaged over 20 datasets.
- As a feature extractor (a linear classifier trained on CLIP’s image vectors). The best CLIP model beat the best existing model, Noisy Student EfficientNet-L2, on 21 of 27 datasets. The average gain was 2.6% on a standard set of 12 datasets and 5% on the 27.
- Robustness. On 7 datasets where the images differ from ImageNet’s, such as sketches and unusual photos, zero-shot CLIP closed up to 75% of the usual gap between ImageNet accuracy and accuracy on the new images. A ResNet-101 with the same ImageNet accuracy scored 2.7% on ImageNet-A, and CLIP scored 77.1%.
- Adapting to ImageNet removes most of that. Training a classifier on CLIP’s features with ImageNet’s labels raised ImageNet accuracy by 9.2 points, to 85.4%, but average accuracy on the 7 other datasets fell slightly.
- Scale. Over a 44-fold range of compute for 5 ResNet models, average zero-shot error fell smoothly, along a straight line on a log-log plot. Results on single datasets were much noisier.
- Overlap with the test sets. Of 35 test datasets, 9 had no detected overlap with the training data. The median overlap was 2.2%. Overlap raised overall accuracy by at most 0.6%, on Birdsnap.
- People and pets. On 37 cat and dog breeds, people with no examples scored 53.7%, and 75.7% with one example of each breed. Zero-shot CLIP scored 93.5%.
Limits, and what came later
The limits come from the paper:
- Zero-shot CLIP is often well below the best task-specific models. The authors estimate it would need about 1,000 times more compute to reach them, which they call infeasible with current hardware.
- It is weak at fine-grained types, such as car models and flower species, at counting objects, and at tasks unlikely to be in its data. On some tasks it is near chance.
- It generalises poorly to data truly unlike its training data. It scored only 88% on handwritten digits (MNIST), below logistic regression on raw pixels.
- It can only choose among the categories it is given. It cannot write a new description, as a captioning model can.
- It needs a very large amount of data: 12.8 billion images were seen in training.
- The authors checked full validation sets many times during development, which is unrealistic for true zero-shot use. Their set of 27 datasets is, in their words, “co-adapted” with CLIP.
- It learned social biases from unfiltered internet data. In one test with 10,000 face images, 4.9% were wrongly put into non-human categories. The rate was about 14% for images of Black people and under 8% for all other groups. Adding the category “child” to the choices greatly reduced how often people under 20 were put into crime-related and non-human categories.
- The authors also discuss its use for surveillance.
Later work built on it directly. Flamingo used a contrastively trained vision encoder. LLaVA used a CLIP vision encoder to connect images to a language model.
What to know first
- The Transformer and the Vision Transformer, which splits an image into patches and treats them like tokens.
- What image classification is, and what ImageNet measures.
- The GPT-2 idea of zero-shot use: doing a task with no examples of it in training.
- Cosine similarity, and the idea of putting two kinds of data in one vector space.
How to read it
The paper is 48 pages, but the main text is 27, and the rest is references and appendices.
- Read section 1, then sections 2.2 and 2.3 with Figures 2 and 3. Figure 3 is the whole training method in 12 lines of code.
- Read section 3.1.2 for zero-shot classification, and 3.1.4 for prompts.
- Look at Figures 5 and 13 for the main results.
- Read section 6 for the authors’ own limits, and section 7.1 on bias.
Expect about 2 hours for the parts above (an estimate). Skim the rest of section 3 if time is short.
Links