NLP Lineage

19 August 2018 · Language modelling

SentencePiece

SentencePiece: A simple and language independent subword tokenizer

T. Kudo, J. Richardson

Date
19 August 2018
What the date marks
arXiv submission
Kind
Infrastructure
Organisation
Google
Access
Apache 2.0
Other dates
EMNLP 2018 demo
Threads
In the guided tour
Chapter 4. Pretrain once, use everywhere
Learning phases
1. Transformer architecture

A milestone: 1 later milestone descends directly from it, and its descendants reach 10 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Subword tokenizers assumed text was already split into words by whitespace.

The problem

That assumption fails for Japanese, Chinese, Thai and any language without spaces.

What it introduced

Treat raw text, spaces included, as the input and learn subwords directly from it, reversibly.

Why it matters

It made tokenization language-agnostic, which is why multilingual models are practical.

What modern models inherit

The tokenizer library much of the field still uses.

What came next

Used by T5, Llama and most non-English-first models.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

No documented ancestors are recorded for SentencePiece without weak links.

No documented ancestors are recorded for SentencePiece without weak links.

No recorded ancestors are recorded for SentencePiece.

No recorded ancestors are recorded for SentencePiece.

After: what it led to

SentencePiece has 66 documented descendants across 11 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 5 turning points among them.

SentencePiece has 66 documented descendants across 11 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

SentencePiece has 78 recorded descendants, including indirect ones, across 11 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 5 turning points among them.

SentencePiece has 78 recorded descendants, including indirect ones, across 11 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

  1. 2018SentencePieceThis milestone.
  2. 2019T5 Turning pointbuilt directly on SentencePieceChapter 4. Pretrain once, use everywhere
  3. 2021Switch Transformerbuilt directly on T5Chapter 5. Scale becomes a science
  4. 2021FLANwas influenced by T5Chapter 6. Teaching models what people want
  5. 2021T0built directly on T5Chapter 6. Teaching models what people want
  6. 2022InstructGPT Turning pointwas influenced by FLANChapter 6. Teaching models what people want
  7. 2022Flan-T5 / Flan-PaLMbuilt directly on T5Chapter 6. Teaching models what people want
  8. 2022ChatGPTbuilt directly on InstructGPTChapter 7. The assistant goes public
  9. 2022GPT-3.5 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  10. 2022Self-Instructwas influenced by FLANChapter 6. Teaching models what people want
  11. 2023Alpacabuilt directly on Self-InstructChapter 8. The weights get out
  12. 2023GPT-4 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  13. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  14. 2023LLaVAbuilt directly on GPT-4Chapter 11. Seeing and hearing
  15. 2023DPObuilt directly on InstructGPTChapter 6. Teaching models what people want
  16. 2023phi-1was influenced by GPT-3.5Chapter 8. The weights get out
  17. 2023Llama 2 Turning pointwas influenced by InstructGPTChapter 8. The weights get out
  18. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  19. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  20. 2023Mixtral 8x7Bbuilt directly on Switch TransformerChapter 8. The weights get out
  21. 2024DeepSeek LLMwas influenced by Llama 2Chapter 8. The weights get out
  22. 2024DeepSeekMoEwas influenced by Switch TransformerChapter 8. The weights get out
  23. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  24. 2024GRPOwas influenced by InstructGPTChapter 12. Thinking longer
  25. 2024Gemmawas influenced by Llama 2Chapter 8. The weights get out
  26. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  27. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  28. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  29. 2024Llama 3built directly on Llama 2Chapter 8. The weights get out
  30. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  31. 2024DeepSeek-V2 / MLAbuilt directly on DeepSeekMoEChapter 9. Making it run anywhere
  32. 2024SWE-agentbuilt directly on SWE-benchChapter 13. Models that act
  33. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  34. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  35. 2024FineWebwas influenced by phi-1Chapter 8. The weights get out
  36. 2024Llama 3.1 405Bbuilt directly on DPOChapter 8. The weights get out
  37. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  38. 2024DeepSeek-V3built directly on DeepSeekMoEChapter 8. The weights get out
  39. 2025DeepSeek-R1built directly on GRPO
    built directly on DeepSeek-V3    Chapter 12. Thinking longer
  40. 2025Claude Codewas influenced by Computer useChapter 13. Models that act
  41. 2025Claude 3.7built directly on Claude 3.5 SonnetChapter 12. Thinking longer
  42. 2025GPT-4.5built directly on GPT-4oChapter 14. Where the threads meet now
  43. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  44. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  45. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  46. 2025Kimi K2was influenced by DeepSeekMoEChapter 8. The weights get out
  47. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  48. 2025DeepSeek-V3.2was influenced by DeepSeek-V2 / MLA
    built directly on DeepSeek-V3    Chapter 10. Reading more than you remember
  49. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  50. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  51. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  52. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  53. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  54. 2026GPT-5.3-Codexbuilt directly on GPT-5.2Chapter 13. Models that act
  55. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  56. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  57. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  58. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  59. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  60. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  61. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  62. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  63. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  64. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  65. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  66. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  67. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now
  1. 2018SentencePieceThis milestone.
  2. 2019T5 Turning pointbuilt directly on SentencePieceChapter 4. Pretrain once, use everywhere
  3. 2021Switch Transformerbuilt directly on T5Chapter 5. Scale becomes a science
  4. 2021FLANwas influenced by T5Chapter 6. Teaching models what people want
  5. 2021T0built directly on T5Chapter 6. Teaching models what people want
  6. 2022InstructGPT Turning pointwas influenced by FLANChapter 6. Teaching models what people want
  7. 2022Flan-T5 / Flan-PaLMbuilt directly on T5Chapter 6. Teaching models what people want
  8. 2022ChatGPTbuilt directly on InstructGPTChapter 7. The assistant goes public
  9. 2022GPT-3.5 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  10. 2022Self-Instructwas influenced by FLANChapter 6. Teaching models what people want
  11. 2023Alpacabuilt directly on Self-InstructChapter 8. The weights get out
  12. 2023GPT-4 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  13. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  14. 2023LLaVAbuilt directly on GPT-4Chapter 11. Seeing and hearing
  15. 2023DPObuilt directly on InstructGPTChapter 6. Teaching models what people want
  16. 2023phi-1was influenced by Self-Instruct (indirect influence)Chapter 8. The weights get out
  17. 2023Llama 2 Turning pointwas influenced by InstructGPTChapter 8. The weights get out
  18. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  19. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  20. 2023Mixtral 8x7Bbuilt directly on Switch TransformerChapter 8. The weights get out
  21. 2024DeepSeek LLMwas influenced by Llama 2Chapter 8. The weights get out
  22. 2024DeepSeekMoEwas influenced by Switch TransformerChapter 8. The weights get out
  23. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  24. 2024GRPOwas influenced by InstructGPTChapter 12. Thinking longer
  25. 2024Gemmawas influenced by Llama 2Chapter 8. The weights get out
  26. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  27. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  28. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  29. 2024Llama 3built directly on Llama 2Chapter 8. The weights get out
  30. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  31. 2024DeepSeek-V2 / MLAbuilt directly on DeepSeekMoEChapter 9. Making it run anywhere
  32. 2024SWE-agentbuilt directly on SWE-benchChapter 13. Models that act
  33. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  34. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  35. 2024FineWebwas influenced by phi-1Chapter 8. The weights get out
  36. 2024Llama 3.1 405Bbuilt directly on DPOChapter 8. The weights get out
  37. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  38. 2024Gemini 2.0was influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  39. 2024DeepSeek-V3built directly on DeepSeekMoEChapter 8. The weights get out
  40. 2025DeepSeek-R1built directly on GRPO
    built directly on DeepSeek-V3    Chapter 12. Thinking longer
  41. 2025Claude Codewas influenced by Computer use
    was influenced by SWE-agent (indirect influence)    Chapter 13. Models that act
  42. 2025Claude 3.7built directly on Claude 3.5 SonnetChapter 12. Thinking longer
  43. 2025GPT-4.5built directly on GPT-4oChapter 14. Where the threads meet now
  44. 2025Gemini 2.5built directly on Gemini 2.0Chapter 12. Thinking longer
  45. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  46. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  47. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  48. 2025Kimi K2was influenced by DeepSeekMoEChapter 8. The weights get out
  49. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  50. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  51. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  52. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  53. 2025DeepSeek-V3.2was influenced by DeepSeek-V2 / MLA
    built directly on DeepSeek-V3    Chapter 10. Reading more than you remember
  54. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  55. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  56. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  57. 2025Gemini 3built directly on Gemini 2.5Chapter 14. Where the threads meet now
  58. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  59. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  60. 2026GPT-5.3-Codexwas influenced by Computer use (indirect influence)Chapter 13. Models that act
  61. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  62. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3
    was influenced by IMO gold    Chapter 12. Thinking longer
  63. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  64. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  65. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  66. 2026Project Glasswingwas influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  67. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  68. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  69. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  70. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  71. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  72. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  73. 2026Fable 5 / Mythos 5was influenced by Project GlasswingChapter 14. Where the threads meet now
  74. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  75. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  76. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  77. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  78. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  79. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now

Direct links and their sources

Where it came from

None recorded in the sources used.

What it led to

Built on directly by