NLP Lineage

6 November 2019 · Language modelling

Multi-query attention

Fast Transformer Decoding: One Write-Head is All You Need

N. Shazeer

Date
6 November 2019
What the date marks
arXiv submission
Kind
Architectural idea
Organisation
Google
Access
Not applicable
Threads
In the guided tour
Chapter 9. Making it run anywhere
Learning phases
1. Transformer architecture; 6. Local models and deployment

A milestone: 3 later milestones descend directly from it, and its descendants reach 9 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Every attention head kept its own keys and values, all cached during generation.

The problem

The KV cache dominates memory during inference and throttles throughput.

What it introduced

Share one set of keys and values across all heads, shrinking the cache dramatically.

Why it matters

The start of the inference-efficiency line that runs through GQA to DeepSeek's latent attention.

What modern models inherit

KV cache design, the main memory constraint when you serve a model.

What came next

Grouped-query attention in 2023 found the middle ground that Llama 2 and Mistral adopted.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

Multi-query attention has 10 documented ancestors across 3 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 4 turning points among them.

Multi-query attention has 10 documented ancestors across 3 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

Multi-query attention has 11 recorded ancestors, including indirect ones, across 3 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 4 turning points among them.

Multi-query attention has 11 recorded ancestors, including indirect ones, across 3 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2019Multi-query attentionThis milestone.
  2. 2017Transformer Turning pointMulti-query attention built directly on thisChapter 3. Attention, then only attention
  3. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  4. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  5. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  6. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  7. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  8. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  9. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  10. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  11. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2019Multi-query attentionThis milestone.
  2. 2017Transformer Turning pointMulti-query attention built directly on thisChapter 3. Attention, then only attention
  3. 2017ConvS2STransformer was influenced by this (indirect influence)Chapter 3. Attention, then only attention
  4. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  5. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  6. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  7. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  8. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  9. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  10. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  11. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  12. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

Multi-query attention has 24 documented descendants across 4 chapters, reaching forward to Qwen3.8 in 2026. Showing its direct descendants and the 2 turning points among them.

Multi-query attention has 24 documented descendants across 4 chapters, reaching forward to Qwen3.8 in 2026. Showing all of them.

Multi-query attention has 27 recorded descendants, including indirect ones, across 4 chapters, reaching forward to Qwen3.8 in 2026. Showing its direct descendants and the 2 turning points among them.

Multi-query attention has 27 recorded descendants, including indirect ones, across 4 chapters, reaching forward to Qwen3.8 in 2026. Showing all of them.

  1. 2019Multi-query attentionThis milestone.
  2. 2023Grouped-query attention Turning pointbuilt directly on Multi-query attentionChapter 9. Making it run anywhere
  3. 2023Llama 2 Turning pointbuilt directly on Grouped-query attentionChapter 8. The weights get out
  4. 2023vLLMwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  5. 2023Mistral 7Bbuilt directly on Grouped-query attentionChapter 8. The weights get out
  6. 2023Mixtral 8x7Bbuilt directly on Mistral 7BChapter 8. The weights get out
  7. 2024DeepSeek LLMwas influenced by Llama 2Chapter 8. The weights get out
  8. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  9. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  10. 2024Gemmawas influenced by Llama 2
    was influenced by Mistral 7B    Chapter 8. The weights get out
  11. 2024Llama 3built directly on Grouped-query attentionChapter 8. The weights get out
  12. 2024DeepSeek-V2 / MLAwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  13. 2024Qwen2built directly on Grouped-query attentionChapter 8. The weights get out
  14. 2024Llama 3.1 405Bbuilt directly on Grouped-query attentionChapter 8. The weights get out
  15. 2024DeepSeek-V3built directly on DeepSeek-V2 / MLAChapter 8. The weights get out
  16. 2025DeepSeek-R1built directly on DeepSeek-V3Chapter 12. Thinking longer
  17. 2025Llama 4succeeded Llama 3.1 405BChapter 8. The weights get out
  18. 2025Qwen3built directly on Qwen2Chapter 8. The weights get out
  19. 2025Kimi K2was influenced by GRPOChapter 8. The weights get out
  20. 2025DeepSeek-V3.2was influenced by DeepSeek-V2 / MLAChapter 10. Reading more than you remember
  21. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  22. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  23. 2026DeepSeek-V4built directly on DeepSeek-V3.2
    was influenced by DeepSeek-V3    Chapter 8. The weights get out
  24. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  25. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  1. 2019Multi-query attentionThis milestone.
  2. 2023Grouped-query attention Turning pointbuilt directly on Multi-query attentionChapter 9. Making it run anywhere
  3. 2023Llama 2 Turning pointbuilt directly on Grouped-query attentionChapter 8. The weights get out
  4. 2023vLLMwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  5. 2023Mistral 7Bbuilt directly on Grouped-query attentionChapter 8. The weights get out
  6. 2023Mixtral 8x7Bbuilt directly on Mistral 7BChapter 8. The weights get out
  7. 2024DeepSeek LLMwas influenced by Llama 2Chapter 8. The weights get out
  8. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  9. 2024GRPObuilt directly on DeepSeek LLMChapter 12. Thinking longer
  10. 2024Gemmawas influenced by Llama 2
    was influenced by Mistral 7B    Chapter 8. The weights get out
  11. 2024Llama 3built directly on Grouped-query attentionChapter 8. The weights get out
  12. 2024DeepSeek-V2 / MLAwas influenced by Multi-query attentionChapter 9. Making it run anywhere
  13. 2024Qwen2built directly on Grouped-query attentionChapter 8. The weights get out
  14. 2024Llama 3.1 405Bbuilt directly on Grouped-query attentionChapter 8. The weights get out
  15. 2024DeepSeek-V3built directly on DeepSeek-V2 / MLAChapter 8. The weights get out
  16. 2025DeepSeek-R1built directly on DeepSeek-V3Chapter 12. Thinking longer
  17. 2025Llama 4succeeded Llama 3.1 405BChapter 8. The weights get out
  18. 2025Qwen3built directly on Qwen2Chapter 8. The weights get out
  19. 2025Kimi K2was influenced by DeepSeek-V3 (indirect influence)Chapter 8. The weights get out
  20. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  21. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  22. 2025DeepSeek-V3.2was influenced by DeepSeek-V2 / MLAChapter 10. Reading more than you remember
  23. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  24. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  25. 2026DeepSeek-V4built directly on DeepSeek-V3.2
    was influenced by DeepSeek-V3    Chapter 8. The weights get out
  26. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  27. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  28. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out

Direct links and their sources

Where it came from

Built directly on

What it led to

Built on directly by

Influenced

Terms