Pretraining and fine-tuning

A two-step recipe. First, train a model on a large amount of plain text with no labels, for example by predicting the next word (pretraining). Then train it a little more on a smaller labelled dataset for the task you want (fine-tuning).