NLP Lineage

22 October 2020 · Language modelling

Vision Transformer

An Image is Worth 16x16 Words

A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al. (12 authors)

Date
22 October 2020
What the date marks
arXiv submission
Kind
Model
Organisation
Google Research
Access
Apache 2.0
Other dates
ICLR 2021
Threads
In the guided tour
Chapter 11. Seeing and hearing
Learning phases
1. Transformer architecture

A milestone: 3 later milestones descend directly from it, and its descendants reach 9 of the 10 threads. How levels are decided

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Vision belonged to convolutional networks; transformers were for text.

The problem

Two different architectures meant no shared representation between modalities.

What it introduced

Cut an image into patches, treat each as a token, and feed them to a plain Transformer encoder.

Why it matters

One architecture for pixels and words is the precondition for every multimodal model that followed.

What modern models inherit

Patches as tokens, the standard way images enter an LLM.

What came next

CLIP, Flamingo, LLaVA and the native multimodal frontier models.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

Vision Transformer has 20 documented ancestors across 4 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 6 turning points among them.

Vision Transformer has 20 documented ancestors across 4 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

Vision Transformer has 21 recorded ancestors, including indirect ones, across 4 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 6 turning points among them.

Vision Transformer has 21 recorded ancestors, including indirect ones, across 4 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2020Vision TransformerThis milestone.
  2. 2018BERT Turning pointVision Transformer was influenced by thisChapter 4. Pretrain once, use everywhere
  3. 2018GPT-1BERT was influenced by thisChapter 4. Pretrain once, use everywhere
  4. 2018ELMoBERT was influenced by thisChapter 4. Pretrain once, use everywhere
  5. 2018ULMFiTBERT was influenced by thisChapter 4. Pretrain once, use everywhere
  6. 2017CoVeELMo was influenced by thisChapter 3. Attention, then only attention
  7. 2017Transformer Turning pointVision Transformer built directly on thisChapter 3. Attention, then only attention
  8. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  9. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  10. 2015LM pretraining Turning pointBERT was influenced by thisChapter 2. Words become vectors
  11. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  12. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  13. 2014GloVeELMo built directly on thisChapter 2. Words become vectors
  14. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  15. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  16. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  17. 2013word2vecELMo was influenced by thisChapter 2. Words become vectors
  18. 2011NLP from scratchELMo was influenced by thisChapter 1. Language as probability
  19. 2003Neural LMNLP from scratch was influenced by this
    word2vec was influenced by this
    Chapter 1. Language as probability
  20. 1997LSTM Turning pointSeq2seq built directly on this
    ELMo built directly on this
    Chapter 1. Language as probability
  21. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2020Vision TransformerThis milestone.
  2. 2018BERT Turning pointVision Transformer was influenced by thisChapter 4. Pretrain once, use everywhere
  3. 2018GPT-1BERT was influenced by thisChapter 4. Pretrain once, use everywhere
  4. 2018ELMoBERT was influenced by thisChapter 4. Pretrain once, use everywhere
  5. 2018ULMFiTBERT was influenced by thisChapter 4. Pretrain once, use everywhere
  6. 2017CoVeELMo was influenced by thisChapter 3. Attention, then only attention
  7. 2017Transformer Turning pointVision Transformer built directly on thisChapter 3. Attention, then only attention
  8. 2017ConvS2STransformer was influenced by this (indirect influence)Chapter 3. Attention, then only attention
  9. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  10. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  11. 2015LM pretraining Turning pointBERT was influenced by thisChapter 2. Words become vectors
  12. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  13. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  14. 2014GloVeELMo built directly on thisChapter 2. Words become vectors
  15. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  16. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  17. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  18. 2013word2vecELMo was influenced by thisChapter 2. Words become vectors
  19. 2011NLP from scratchELMo was influenced by thisChapter 1. Language as probability
  20. 2003Neural LMNLP from scratch was influenced by this
    word2vec was influenced by this
    Chapter 1. Language as probability
  21. 1997LSTM Turning pointSeq2seq built directly on this
    ELMo built directly on this
    Chapter 1. Language as probability
  22. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

Vision Transformer has 13 documented descendants across 5 chapters, reaching forward to Gemini 3.1 in 2026. Showing its direct descendants.

Vision Transformer has 13 documented descendants across 5 chapters, reaching forward to Gemini 3.1 in 2026. Showing all of them.

Vision Transformer has 39 recorded descendants, including indirect ones, across 6 chapters, reaching forward to Qwen3.8 in 2026. Showing its direct descendants.

Vision Transformer has 39 recorded descendants, including indirect ones, across 6 chapters, reaching forward to Qwen3.8 in 2026. Showing all of them.

  1. 2020Vision TransformerThis milestone.
  2. 2021CLIPbuilt directly on Vision TransformerChapter 11. Seeing and hearing
  3. 2022Flamingobuilt directly on Vision TransformerChapter 11. Seeing and hearing
  4. 2023LLaVAbuilt directly on CLIPChapter 11. Seeing and hearing
  5. 2023Gemini 1.0was influenced by Vision TransformerChapter 11. Seeing and hearing
  6. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  7. 2024Gemmabuilt directly on Gemini 1.0Chapter 8. The weights get out
  8. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  9. 2025Gemini 2.5was influenced by Gemini 1.5Chapter 12. Thinking longer
  10. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  11. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  12. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  13. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  14. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  1. 2020Vision TransformerThis milestone.
  2. 2021CLIPbuilt directly on Vision TransformerChapter 11. Seeing and hearing
  3. 2022Flamingobuilt directly on Vision TransformerChapter 11. Seeing and hearing
  4. 2023LLaVAbuilt directly on CLIP
    was influenced by Flamingo (indirect influence)
    Chapter 11. Seeing and hearing
  5. 2023GPT-4Vwas influenced by CLIP (indirect influence)Chapter 11. Seeing and hearing
  6. 2023Gemini 1.0was influenced by Vision TransformerChapter 11. Seeing and hearing
  7. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  8. 2024Gemmabuilt directly on Gemini 1.0Chapter 8. The weights get out
  9. 2024GPT-4obuilt directly on GPT-4VChapter 11. Seeing and hearing
  10. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  11. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  12. 2024DeepSeek-V3was influenced by GPT-4oChapter 8. The weights get out
  13. 2025DeepSeek-R1built directly on DeepSeek-V3Chapter 12. Thinking longer
  14. 2025Claude Codewas influenced by Computer useChapter 13. Models that act
  15. 2025GPT-4.5built directly on GPT-4oChapter 14. Where the threads meet now
  16. 2025Gemini 2.5was influenced by Gemini 1.5Chapter 12. Thinking longer
  17. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  18. 2025Kimi K2was influenced by DeepSeek-V3 (indirect influence)Chapter 8. The weights get out
  19. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  20. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  21. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  22. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  23. 2025DeepSeek-V3.2built directly on DeepSeek-V3Chapter 10. Reading more than you remember
  24. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  25. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  26. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  27. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  28. 2026GPT-5.3-Codexwas influenced by Computer use (indirect influence)Chapter 13. Models that act
  29. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  30. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  31. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  32. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  33. 2026Project Glasswingwas influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  34. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  35. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  36. 2026Fable 5 / Mythos 5was influenced by Project GlasswingChapter 14. Where the threads meet now
  37. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  38. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  39. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  40. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out

Direct links and their sources

Where it came from

Built directly on

Influenced by

What it led to

Built on directly by

Influenced