Attention

A way for a model to look back at all parts of its input and weight each part by how relevant it is at the current step. The output is the weighted average of those parts.