Before it
Neural language models used a fixed window of previous words.
The problem
A fixed window cannot carry context from arbitrarily far back.
What it introduced
Use a recurrent net as the language model, so context is unbounded in principle. Roughly halved perplexity against backoff n-grams.
Why it matters
It proved neural language models beat the statistical state of the art on a real task, not just in theory.
What modern models inherit
The neural language model as a serious competitor rather than a curiosity.
What came next
The same author's word2vec, and the seq2seq line.