Before it
Every word had one vector regardless of the sentence it appeared in.
The problem
Bank in a river sentence and bank in a finance sentence got identical representations.
What it introduced
Take the encoder from a trained translation model and use its hidden states as context-sensitive word vectors.
Why it matters
The first workable contextual embedding, a few months before ELMo made the idea famous.
What modern models inherit
The principle that a word's representation should depend on its context.
What came next
ELMo did the same thing from a language model instead of a translation model, and did it better.