Layer normalisation
A step that rescales the numbers in each example's vector to a standard range, using only that example. It keeps the values in a deep network stable during training.
Introduced by
Layer norm, 2016
Used by
- Transformer, 2017
- GPT-2, 2019
- T5, 2019