Multi-head attention
Several attention operations, called heads, run side by side on smaller projections of the same vectors. Each head can attend to a different kind of relation, and their outputs are joined.
Introduced by
Transformer, 2017
Used by
- BERT, 2018
- Sparse Transformer, 2019
- Grouped-query attention, 2023