Reinforcement learning from human feedback (RLHF)

A way to train a model towards what people prefer. People compare pairs of outputs, a reward model learns to predict their choices, and the main model is then trained with reinforcement learning to get high scores from the reward model.