NLP Lineage

May 2017 · Language modelling

ConvS2S

Convolutional Sequence to Sequence Learning

J. Gehring, M. Auli, D. Grangier, D. Yarats, Y. Dauphin

Date
May 2017
What the date marks
arXiv submission
Kind
Architectural idea
Organisation
Facebook AI Research
Access
MIT
Other dates
ICML 2017
Threads
In the guided tour
Chapter 3. Attention, then only attention
Learning phases
1. Transformer architecture

Notable: no later milestone in the data is documented as descending from it. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Sequence models were recurrent, and recurrence forced sequential computation.

The problem

Training could not be parallelised across positions, so it was slow.

What it introduced

Replace recurrence with stacked convolutions plus attention at every layer.

Why it matters

The road not taken. It showed removing recurrence was possible months before the Transformer did it differently.

What modern models inherit

Mostly historical: it marks that parallelism, not attention alone, was the prize.

What came next

The Transformer cited it and then overtook it comprehensively.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

ConvS2S has 6 documented ancestors across 3 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 3 turning points among them.

ConvS2S has 6 documented ancestors across 3 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

ConvS2S has 6 recorded ancestors, including indirect ones, across 3 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 3 turning points among them.

ConvS2S has 6 recorded ancestors, including indirect ones, across 3 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2017ConvS2SThis milestone.
  2. 2015Residual connectionsConvS2S built directly on thisChapter 3. Attention, then only attention
  3. 2014Seq2seq Turning pointAttention built directly on thisChapter 2. Words become vectors
  4. 2014Attention Turning pointConvS2S built directly on thisChapter 3. Attention, then only attention
  5. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  6. 1997LSTM Turning pointSeq2seq built directly on this
    GRU / encoder-decoder built directly on this    Chapter 1. Language as probability
  7. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2017ConvS2SThis milestone.
  2. 2015Residual connectionsConvS2S built directly on thisChapter 3. Attention, then only attention
  3. 2014Seq2seq Turning pointAttention built directly on thisChapter 2. Words become vectors
  4. 2014Attention Turning pointConvS2S built directly on thisChapter 3. Attention, then only attention
  5. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  6. 1997LSTM Turning pointSeq2seq built directly on this
    GRU / encoder-decoder built directly on this    Chapter 1. Language as probability
  7. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

No documented descendants are recorded for ConvS2S without weak links.

No documented descendants are recorded for ConvS2S without weak links.

ConvS2S has 159 recorded descendants, including indirect ones, across 12 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 19 turning points among them.

ConvS2S has 159 recorded descendants, including indirect ones, across 12 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

  1. 2017ConvS2SThis milestone.
  2. 2017Transformer Turning pointwas influenced by ConvS2S (indirect influence)Chapter 3. Attention, then only attention
  3. 2018GPT-1built directly on TransformerChapter 4. Pretrain once, use everywhere
  4. 2018BERT Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  5. 2019Transformer-XLbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  6. 2019GPT-2 Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  7. 2019WebTextwas influenced by GPT-1Chapter 4. Pretrain once, use everywhere
  8. 2019Sparse Transformer Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  9. 2019RoBERTabuilt directly on BERTChapter 4. Pretrain once, use everywhere
  10. 2019Sentence-BERTbuilt directly on BERTChapter 10. Reading more than you remember
  11. 2019Megatron-LMbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  12. 2019RLHF for LMsbuilt directly on GPT-2Chapter 6. Teaching models what people want
  13. 2019T5 Turning pointbuilt directly on TransformerChapter 4. Pretrain once, use everywhere
  14. 2019C4 Turning pointwas influenced by WebText (indirect influence)Chapter 4. Pretrain once, use everywhere
  15. 2019Multi-query attentionbuilt directly on TransformerChapter 9. Making it run anywhere
  16. 2020Scaling laws Turning pointbuilt directly on TransformerChapter 5. Scale becomes a science
  17. 2020Longformerbuilt directly on TransformerChapter 10. Reading more than you remember
  18. 2020Dense retrievalbuilt directly on BERTChapter 10. Reading more than you remember
  19. 2020RAGbuilt directly on Dense retrievalChapter 10. Reading more than you remember
  20. 2020GPT-3 Turning pointwas influenced by Scaling laws
    built directly on GPT-2
    built directly on Sparse Transformer    Chapter 5. Scale becomes a science
  21. 2020BigBirdbuilt directly on TransformerChapter 10. Reading more than you remember
  22. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  23. 2020Vision Transformerbuilt directly on TransformerChapter 11. Seeing and hearing
  24. 2020The Pilewas influenced by GPT-3Chapter 5. Scale becomes a science
  25. 2021Switch Transformerbuilt directly on T5Chapter 5. Scale becomes a science
  26. 2021CLIPbuilt directly on GPT-2
    built directly on Vision Transformer    Chapter 11. Seeing and hearing
  27. 2021RoPE Turning pointbuilt directly on TransformerChapter 10. Reading more than you remember
  28. 2021LoRAbuilt directly on TransformerChapter 9. Making it run anywhere
  29. 2021Copilotwas influenced by Codex / HumanEvalChapter 7. The assistant goes public
  30. 2021Codex / HumanEvalbuilt directly on GPT-3Chapter 13. Models that act
  31. 2021ALiBibuilt directly on TransformerChapter 10. Reading more than you remember
  32. 2021FLANwas influenced by T5Chapter 6. Teaching models what people want
  33. 2021T0built directly on T5Chapter 6. Teaching models what people want
  34. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  35. 2021RETROwas influenced by Dense retrievalChapter 10. Reading more than you remember
  36. 2021WebGPTbuilt directly on GPT-3Chapter 13. Models that act
  37. 2022MT-NLGwas influenced by Scaling laws
    built directly on Megatron-LM    Chapter 5. Scale becomes a science
  38. 2022Chain of thought Turning pointwas influenced by GPT-3Chapter 12. Thinking longer
  39. 2022InstructGPT Turning pointbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  40. 2022Self-consistencybuilt directly on Chain of thoughtChapter 12. Thinking longer
  41. 2022STaRwas influenced by Chain of thoughtChapter 12. Thinking longer
  42. 2022Chinchilla Turning pointbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  43. 2022PaLMwas influenced by Chain of thoughtChapter 5. Scale becomes a science
  44. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  45. 2022Flamingobuilt directly on Vision TransformerChapter 11. Seeing and hearing
  46. 2022OPT-175Bwas influenced by Megatron-LMChapter 8. The weights get out
  47. 2022Zero-shot CoTwas influenced by GPT-3Chapter 12. Thinking longer
  48. 2022FlashAttentionbuilt directly on TransformerChapter 9. Making it run anywhere
  49. 2022Emergent abilitiesbuilt directly on Scaling lawsChapter 5. Scale becomes a science
  50. 2022BLOOMbuilt directly on ALiBiChapter 8. The weights get out
  51. 2022LLM.int8()was influenced by OPT-175BChapter 9. Making it run anywhere
  52. 2022Whisperbuilt directly on TransformerChapter 11. Seeing and hearing
  53. 2022ReActbuilt directly on Chain of thought
    was influenced by WebGPT    Chapter 13. Models that act
  54. 2022Flan-T5 / Flan-PaLMbuilt directly on T5Chapter 6. Teaching models what people want
  55. 2022GPTQwas influenced by LLM.int8()Chapter 9. Making it run anywhere
  56. 2022Speculative decodingwas influenced by TransformerChapter 9. Making it run anywhere
  57. 2022GPT-3.5 Turning pointbuilt directly on GPT-3Chapter 7. The assistant goes public
  58. 2022ChatGPTbuilt directly on InstructGPT
    built directly on GPT-3.5    Chapter 7. The assistant goes public
  59. 2022Constitutional AIwas influenced by Chain of thoughtChapter 6. Teaching models what people want
  60. 2022Self-Instructbuilt directly on GPT-3
    was influenced by FLAN    Chapter 6. Teaching models what people want
  61. 2023Toolformerbuilt directly on GPT-3Chapter 13. Models that act
  62. 2023LLaMA Turning pointbuilt directly on RoPEChapter 8. The weights get out
  63. 2023llama.cppwas influenced by LLaMAChapter 9. Making it run anywhere
  64. 2023Alpacabuilt directly on LLaMAChapter 8. The weights get out
  65. 2023Claudebuilt directly on Constitutional AI
    was influenced by HH-RLHF    Chapter 7. The assistant goes public
  66. 2023GPT-4 Turning pointbuilt directly on Scaling lawsChapter 7. The assistant goes public
  67. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  68. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  69. 2023Pythiawas influenced by Scaling lawsChapter 8. The weights get out
  70. 2023LLaVAbuilt directly on GPT-4
    built directly on LLaMA
    built directly on CLIP    Chapter 11. Seeing and hearing
  71. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  72. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  73. 2023Tree of thoughtsbuilt directly on Chain of thoughtChapter 12. Thinking longer
  74. 2023Grouped-query attention Turning pointbuilt directly on Multi-query attentionChapter 9. Making it run anywhere
  75. 2023QLoRAbuilt directly on LoRAChapter 9. Making it run anywhere
  76. 2023Falconbuilt directly on RefinedWebChapter 8. The weights get out
  77. 2023DPObuilt directly on RLHF for summarisation
    built directly on InstructGPT    Chapter 6. Teaching models what people want
  78. 2023Process supervisionbuilt directly on Chain of thoughtChapter 12. Thinking longer
  79. 2023RefinedWebwas influenced by C4Chapter 8. The weights get out
  80. 2023Function callingwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  81. 2023phi-1was influenced by GPT-3.5
    was influenced by Self-Instruct (indirect influence)    Chapter 8. The weights get out
  82. 2023Position interpolationbuilt directly on RoPEChapter 10. Reading more than you remember
  83. 2023Lost in the middlewas influenced by RAGChapter 10. Reading more than you remember
  84. 2023Llama 2 Turning pointbuilt directly on LLaMA
    built directly on Grouped-query attention    Chapter 8. The weights get out
  85. 2023vLLMwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  86. 2023GPT-4Vbuilt directly on GPT-4
    was influenced by CLIP (indirect influence)    Chapter 11. Seeing and hearing
  87. 2023Mistral 7Bwas influenced by Sparse Transformer
    was influenced by Longformer
    was influenced by FlashAttention    Chapter 8. The weights get out
  88. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  89. 2023Mambawas influenced by FlashAttentionChapter 9. Making it run anywhere
  90. 2023Gemini 1.0was influenced by Vision TransformerChapter 11. Seeing and hearing
  91. 2023Mixtral 8x7Bbuilt directly on Switch Transformer
    built directly on Mistral 7B    Chapter 8. The weights get out
  92. 2024DeepSeek LLMbuilt directly on Scaling lawsChapter 8. The weights get out
  93. 2024DeepSeekMoEwas influenced by Switch TransformerChapter 8. The weights get out
  94. 2024Dolmawas influenced by C4
    was influenced by The Pile    Chapter 8. The weights get out
  95. 2024OLMowas influenced by PythiaChapter 8. The weights get out
  96. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  97. 2024Gemini 1.5built directly on Gemini 1.0Chapter 10. Reading more than you remember
  98. 2024Gemmabuilt directly on Gemini 1.0
    was influenced by Mistral 7B    Chapter 8. The weights get out
  99. 2024Claude 3was influenced by GPT-4Chapter 14. Where the threads meet now
  100. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  101. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  102. 2024Llama 3built directly on Grouped-query attention
    was influenced by Chinchilla    Chapter 8. The weights get out
  103. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  104. 2024DeepSeek-V2 / MLAbuilt directly on RoPE
    was influenced by Multi-query attention    Chapter 9. Making it run anywhere
  105. 2024SWE-agentbuilt directly on SWE-benchChapter 13. Models that act
  106. 2024GPT-4owas influenced by Whisper (indirect influence)Chapter 11. Seeing and hearing
  107. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  108. 2024FineWebwas influenced by The PileChapter 8. The weights get out
  109. 2024FlashAttention-3built directly on FlashAttentionChapter 9. Making it run anywhere
  110. 2024Qwen2built directly on RoPEChapter 8. The weights get out
  111. 2024Llama 3.1 405Bwas influenced by Chinchilla
    built directly on Grouped-query attention
    was influenced by Position interpolation    Chapter 8. The weights get out
  112. 2024Test-time computewas influenced by Process supervision
    was influenced by Tree of thoughts (indirect influence)
    was influenced by STaR (indirect influence)    Chapter 12. Thinking longer
  113. 2024o1 Turning pointwas influenced by Chain of thoughtChapter 12. Thinking longer
  114. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  115. 2024MCPwas influenced by Toolformer (indirect influence)Chapter 13. Models that act
  116. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  117. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  118. 2024DeepSeek-V3built directly on DeepSeek-V2 / MLA
    was influenced by GPT-4o    Chapter 8. The weights get out
  119. 2025DeepSeek-R1was influenced by Chain of thought
    built directly on GRPO
    built directly on DeepSeek-V3    Chapter 12. Thinking longer
  120. 2025Deep researchwas influenced by WebGPT (indirect influence)Chapter 13. Models that act
  121. 2025Claude 3.7was influenced by o1
    built directly on Claude 3.5 Sonnet    Chapter 12. Thinking longer
  122. 2025Claude Codewas influenced by Computer use
    was influenced by SWE-agent (indirect influence)
    was influenced by MCP    Chapter 13. Models that act
  123. 2025GPT-4.5was influenced by Scaling lawsChapter 14. Where the threads meet now
  124. 2025Gemini 2.5was influenced by Gemini 1.5Chapter 12. Thinking longer
  125. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  126. 2025o3 releasedbuilt directly on o1
    was influenced by Function calling    Chapter 12. Thinking longer
  127. 2025Qwen3built directly on Qwen2Chapter 8. The weights get out
  128. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  129. 2025Kimi K2was influenced by DeepSeekMoE
    was influenced by DeepSeek-V3 (indirect influence)
    was influenced by GRPO    Chapter 8. The weights get out
  130. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  131. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  132. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  133. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  134. 2025DeepSeek-V3.2was influenced by Sparse TransformerChapter 10. Reading more than you remember
  135. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  136. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  137. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  138. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  139. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  140. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  141. 2026GPT-5.3-Codexwas influenced by Computer use (indirect influence)Chapter 13. Models that act
  142. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  143. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  144. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  145. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  146. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  147. 2026Project Glasswingwas influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  148. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  149. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  150. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  151. 2026DeepSeek-V4built directly on DeepSeek-V3.2Chapter 8. The weights get out
  152. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  153. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  154. 2026Fable 5 / Mythos 5was influenced by Project GlasswingChapter 14. Where the threads meet now
  155. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  156. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  157. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  158. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  159. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  160. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now

Direct links and their sources

Where it came from

Built directly on

What it led to

Influenced