NLP Lineage

27 September 2023 · Language modelling

Mistral 7B

Mistral 7B

A. Jiang, A. Sablayrolles, A. Mensch, et al.

Date
27 September 2023
What the date marks
Model release
Kind
Model
Organisation
Mistral AI
Access
Apache 2.0
Other dates
paper 2023-10-10
Threads
In the guided tour
Chapter 8. The weights get out
Learning phases
1. Transformer architecture; 2. How LLMs are trained; 8. Training from scratch

A milestone: 2 later milestones descend directly from it, and its descendants reach 3 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Open models came from large US labs under restrictive or custom licences.

The problem

Could a small model with good engineering beat a much larger one, under a fully permissive licence?

What it introduced

7B with grouped-query and sliding-window attention, beating Llama 2 13B on every benchmark, Apache 2.0.

Why it matters

It reset expectations about what a small model can do and made Apache-licensed weights competitive.

What modern models inherit

The efficient small open model as a serious deployment target.

What came next

Mixtral's open MoE, and a European frontier lab.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

Mistral 7B has 16 documented ancestors across 6 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 6 turning points among them.

Mistral 7B has 16 documented ancestors across 6 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

Mistral 7B has 18 recorded ancestors, including indirect ones, across 6 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 6 turning points among them.

Mistral 7B has 18 recorded ancestors, including indirect ones, across 6 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2023Mistral 7BThis milestone.
  2. 2023Grouped-query attention Turning pointMistral 7B built directly on thisChapter 9. Making it run anywhere
  3. 2022FlashAttentionMistral 7B was influenced by thisChapter 9. Making it run anywhere
  4. 2020BigBirdFlashAttention succeeded thisChapter 10. Reading more than you remember
  5. 2020LongformerMistral 7B was influenced by thisChapter 10. Reading more than you remember
  6. 2019Multi-query attentionGrouped-query attention built directly on thisChapter 9. Making it run anywhere
  7. 2019Sparse Transformer Turning pointMistral 7B was influenced by thisChapter 4. Pretrain once, use everywhere
  8. 2017Transformer Turning pointSparse Transformer built directly on this
    Longformer built directly on this
    FlashAttention built directly on this    Chapter 3. Attention, then only attention
  9. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  10. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  11. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  12. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  13. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  14. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  15. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  16. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  17. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2023Mistral 7BThis milestone.
  2. 2023Grouped-query attention Turning pointMistral 7B built directly on thisChapter 9. Making it run anywhere
  3. 2022FlashAttentionMistral 7B was influenced by thisChapter 9. Making it run anywhere
  4. 2020BigBirdFlashAttention succeeded thisChapter 10. Reading more than you remember
  5. 2020LongformerMistral 7B was influenced by thisChapter 10. Reading more than you remember
  6. 2019Multi-query attentionGrouped-query attention built directly on thisChapter 9. Making it run anywhere
  7. 2019Sparse Transformer Turning pointMistral 7B was influenced by thisChapter 4. Pretrain once, use everywhere
  8. 2019Transformer-XLSparse Transformer was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  9. 2017Transformer Turning pointSparse Transformer built directly on this
    Longformer built directly on this
    FlashAttention built directly on this    Chapter 3. Attention, then only attention
  10. 2017ConvS2STransformer was influenced by this (indirect influence)Chapter 3. Attention, then only attention
  11. 2016Layer normTransformer built directly on thisChapter 3. Attention, then only attention
  12. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  13. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  14. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  15. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  16. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  17. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  18. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  19. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

Mistral 7B has 2 documented descendants across 1 chapter, reaching forward to Gemma in 2024. Showing all of them.

Mistral 7B has 2 documented descendants across 1 chapter, reaching forward to Gemma in 2024. Showing all of them.

Mistral 7B has 4 recorded descendants, including indirect ones, across 1 chapter, reaching forward to Muse Spark in 2026. Showing its direct descendants.

Mistral 7B has 4 recorded descendants, including indirect ones, across 1 chapter, reaching forward to Muse Spark in 2026. Showing all of them.

  1. 2023Mistral 7BThis milestone.
  2. 2023Mixtral 8x7Bbuilt directly on Mistral 7BChapter 8. The weights get out
  3. 2024Gemmawas influenced by Mistral 7BChapter 8. The weights get out
  1. 2023Mistral 7BThis milestone.
  2. 2023Mixtral 8x7Bbuilt directly on Mistral 7BChapter 8. The weights get out
  3. 2024Gemmawas influenced by Mistral 7BChapter 8. The weights get out
  4. 2025Llama 4was influenced by Mixtral 8x7B (indirect influence)Chapter 8. The weights get out
  5. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out

Direct links and their sources

Where it came from

Built directly on

Influenced by

Developed alongside

What it led to

Built on directly by

Influenced

Terms