NLP Lineage

Glossary

Expert capacity

In a mixture-of-experts layer, the most tokens one expert may take in a batch. It keeps the work balanced between experts; tokens sent to a full expert skip the layer.

Introduced by

GShard, 2020