Before it
Each NLP task had its own pipeline of hand-engineered features.
The problem
Feature engineering did not transfer between tasks and capped accuracy.
What it introduced
One network, trained on unlabelled text, producing representations that serve many tasks at once.
Why it matters
An early statement of the idea that became pretraining: learn general representations first, specialise later.
What modern models inherit
Unsupervised pretraining followed by task-specific use.
What came next
Word2vec, then ELMo and BERT, made the idea work at scale.