Sparse mixture of experts

A mixture of experts in which a gating network sends each token to only a few of many experts. The model can have a very large number of weights while each token uses only a small part of them.