Residual connection
A shortcut that adds a layer's input to its output. The layer only has to learn a correction to its input, which makes very deep networks much easier to train.
Introduced by
Residual connections, 2015
Used by
- Transformer, 2017
- Sparse Transformer, 2019