NLP Lineage

1991 · Efficiency and infrastructure

Mixture of experts

Adaptive Mixtures of Local Experts

R. Jacobs, M. Jordan, S. Nowlan, G. Hinton

Date
1991
What the date marks
Journal
Kind
Research paper
Organisation
U. Massachusetts / U. Toronto
Access
Not applicable
Threads
In the guided tour
Chapter 1. Language as probability

A milestone: 1 later milestone descends directly from it, and its descendants reach 9 of the 10 threads. How levels are decided

Read the primary source (direct.mit.edu)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

One network had to learn every part of a task with the same parameters.

The problem

Different regions of a problem want different specialists, and one dense model spends all its capacity on all of it.

What it introduced

Several expert subnetworks plus a gating network that decides which expert handles which input.

Why it matters

This is the ancestor of every sparse mixture-of-experts model, including Mixtral, DeepSeek and Llama 4.

What modern models inherit

Conditional computation: activate a fraction of the parameters per token.

What came next

Dormant for 25 years until Shazeer scaled it in 2017.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

No documented ancestors are recorded for Mixture of experts without weak links.

No documented ancestors are recorded for Mixture of experts without weak links.

No recorded ancestors are recorded for Mixture of experts.

No recorded ancestors are recorded for Mixture of experts.

After: what it led to

Mixture of experts has 31 documented descendants across 9 chapters, reaching forward to Qwen3.8 in 2026. Showing its direct descendants and the 2 turning points among them.

Mixture of experts has 31 documented descendants across 9 chapters, reaching forward to Qwen3.8 in 2026. Showing all of them.

Mixture of experts has 34 recorded descendants, including indirect ones, across 9 chapters, reaching forward to Qwen3.8 in 2026. Showing its direct descendants and the 2 turning points among them.

Mixture of experts has 34 recorded descendants, including indirect ones, across 9 chapters, reaching forward to Qwen3.8 in 2026. Showing all of them.

  1. 1991Mixture of expertsThis milestone.
  2. 2017Sparse MoE Turning pointbuilt directly on Mixture of expertsChapter 3. Attention, then only attention
  3. 2020GShard Turning pointbuilt directly on Sparse MoEChapter 5. Scale becomes a science
  4. 2021Switch Transformerbuilt directly on Sparse MoEChapter 5. Scale becomes a science
  5. 2022PaLMwas influenced by GShardChapter 5. Scale becomes a science
  6. 2022Emergent abilitieswas influenced by PaLMChapter 5. Scale becomes a science
  7. 2022Flan-T5 / Flan-PaLMbuilt directly on PaLMChapter 6. Teaching models what people want
  8. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  9. 2023Gemini 1.0was influenced by GShardChapter 11. Seeing and hearing
  10. 2023Mixtral 8x7Bwas influenced by Sparse MoEChapter 8. The weights get out
  11. 2024DeepSeekMoEwas influenced by Sparse MoEChapter 8. The weights get out
  12. 2024Gemini 1.5was influenced by GShardChapter 10. Reading more than you remember
  13. 2024Gemmabuilt directly on Gemini 1.0Chapter 8. The weights get out
  14. 2024DeepSeek-V2 / MLAbuilt directly on DeepSeekMoEChapter 9. Making it run anywhere
  15. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  16. 2024DeepSeek-V3built directly on DeepSeekMoEChapter 8. The weights get out
  17. 2025DeepSeek-R1built directly on DeepSeek-V3Chapter 12. Thinking longer
  18. 2025Gemini 2.5was influenced by Gemini 1.5Chapter 12. Thinking longer
  19. 2025Llama 4was influenced by Switch TransformerChapter 8. The weights get out
  20. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  21. 2025Kimi K2was influenced by DeepSeekMoEChapter 8. The weights get out
  22. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  23. 2025DeepSeek-V3.2was influenced by DeepSeek-V2 / MLA
    built directly on DeepSeek-V3    Chapter 10. Reading more than you remember
  24. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  25. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  26. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  27. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  28. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  29. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  30. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  31. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  32. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  1. 1991Mixture of expertsThis milestone.
  2. 2017Sparse MoE Turning pointbuilt directly on Mixture of expertsChapter 3. Attention, then only attention
  3. 2020GShard Turning pointbuilt directly on Sparse MoEChapter 5. Scale becomes a science
  4. 2021Switch Transformerbuilt directly on Sparse MoEChapter 5. Scale becomes a science
  5. 2022PaLMwas influenced by GShardChapter 5. Scale becomes a science
  6. 2022Emergent abilitieswas influenced by PaLMChapter 5. Scale becomes a science
  7. 2022Flan-T5 / Flan-PaLMbuilt directly on PaLMChapter 6. Teaching models what people want
  8. 2023Emergence miragesucceeded Emergent abilitiesChapter 5. Scale becomes a science
  9. 2023Gemini 1.0was influenced by GShardChapter 11. Seeing and hearing
  10. 2023Mixtral 8x7Bwas influenced by Sparse MoEChapter 8. The weights get out
  11. 2024DeepSeekMoEwas influenced by Sparse MoEChapter 8. The weights get out
  12. 2024Gemini 1.5was influenced by GShardChapter 10. Reading more than you remember
  13. 2024Gemmabuilt directly on Gemini 1.0Chapter 8. The weights get out
  14. 2024DeepSeek-V2 / MLAbuilt directly on DeepSeekMoEChapter 9. Making it run anywhere
  15. 2024Gemini 2.0built directly on Gemini 1.5Chapter 14. Where the threads meet now
  16. 2024DeepSeek-V3built directly on DeepSeekMoEChapter 8. The weights get out
  17. 2025DeepSeek-R1built directly on DeepSeek-V3Chapter 12. Thinking longer
  18. 2025Gemini 2.5was influenced by Gemini 1.5Chapter 12. Thinking longer
  19. 2025Llama 4was influenced by Switch Transformer
    was influenced by Mixtral 8x7B (indirect influence)    Chapter 8. The weights get out
  20. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  21. 2025Kimi K2was influenced by DeepSeekMoEChapter 8. The weights get out
  22. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  23. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  24. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  25. 2025DeepSeek-V3.2was influenced by DeepSeek-V2 / MLA
    built directly on DeepSeek-V3    Chapter 10. Reading more than you remember
  26. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  27. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  28. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  29. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  30. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  31. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  32. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  33. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  34. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  35. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out

Direct links and their sources

Where it came from

None recorded in the sources used.

What it led to

Built on directly by

Terms