NLP Lineage

18 September 2019 · Alignment

RLHF for LMs

Fine-Tuning Language Models from Human Preferences

D. Ziegler, N. Stiennon, J. Wu, T. Brown, A. Radford, D. Amodei, P. Christiano, G. Irving

Date
18 September 2019
What the date marks
arXiv submission
Kind
Training method
Organisation
OpenAI
Access
MIT
Threads
In the guided tour
Chapter 6. Teaching models what people want
Learning phases
2. How LLMs are trained

A milestone: 1 later milestone descends directly from it, and its descendants reach 10 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Preference-based RL had been shown on robotics and games, not language.

The problem

Language generation has no reward signal, and likelihood is a poor proxy for what people want.

What it introduced

Apply the reward-model-plus-RL recipe to fine-tuning a language model, on GPT-2.

Why it matters

The bridge between the 2017 RLHF idea and the assistants of 2022. Everything after is refinement and scale.

What modern models inherit

The three-stage pipeline: pretrain, learn a reward model, optimise the policy.

What came next

Summarisation from human feedback in 2020, then InstructGPT in 2022.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

RLHF for LMs has 16 documented ancestors across 5 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 6 turning points among them.

RLHF for LMs has 16 documented ancestors across 5 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

RLHF for LMs has 24 recorded ancestors, including indirect ones, across 5 chapters, reaching back to Recurrent networks in 1990. Showing its direct parents and the 6 turning points among them.

RLHF for LMs has 24 recorded ancestors, including indirect ones, across 5 chapters, reaching back to Recurrent networks in 1990. Showing all of them.

  1. 2019RLHF for LMsThis milestone.
  2. 2019GPT-2 Turning pointRLHF for LMs built directly on thisChapter 4. Pretrain once, use everywhere
  3. 2019WebTextGPT-2 was influenced by thisChapter 4. Pretrain once, use everywhere
  4. 2018GPT-1GPT-2 built directly on thisChapter 4. Pretrain once, use everywhere
  5. 2017RLHF originRLHF for LMs built directly on thisChapter 6. Teaching models what people want
  6. 2017Transformer Turning pointGPT-2 built directly on thisChapter 3. Attention, then only attention
  7. 2016Layer normGPT-2 built directly on thisChapter 3. Attention, then only attention
  8. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  9. 2015LM pretraining Turning pointGPT-1 was influenced by thisChapter 2. Words become vectors
  10. 2015BPE tokenizationGPT-2 built directly on thisChapter 2. Words become vectors
  11. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  12. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  13. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  14. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  15. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  16. 1997LSTM Turning pointSeq2seq built directly on thisChapter 1. Language as probability
  17. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability
  1. 2019RLHF for LMsThis milestone.
  2. 2019GPT-2 Turning pointRLHF for LMs built directly on thisChapter 4. Pretrain once, use everywhere
  3. 2019WebTextGPT-2 was influenced by thisChapter 4. Pretrain once, use everywhere
  4. 2018GPT-1GPT-2 built directly on thisChapter 4. Pretrain once, use everywhere
  5. 2018ELMoGPT-1 was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  6. 2018ULMFiTGPT-1 was influenced by this (indirect influence)Chapter 4. Pretrain once, use everywhere
  7. 2017CoVeELMo was influenced by thisChapter 3. Attention, then only attention
  8. 2017RLHF originRLHF for LMs built directly on thisChapter 6. Teaching models what people want
  9. 2017Transformer Turning pointGPT-2 built directly on thisChapter 3. Attention, then only attention
  10. 2017ConvS2STransformer was influenced by this (indirect influence)Chapter 3. Attention, then only attention
  11. 2016Layer normGPT-2 built directly on thisChapter 3. Attention, then only attention
  12. 2015Residual connectionsTransformer was influenced by thisChapter 3. Attention, then only attention
  13. 2015LM pretraining Turning pointGPT-1 was influenced by thisChapter 2. Words become vectors
  14. 2015BPE tokenizationGPT-2 built directly on thisChapter 2. Words become vectors
  15. 2015Luong attentionTransformer was influenced by thisChapter 3. Attention, then only attention
  16. 2014AdamTransformer was influenced by thisChapter 2. Words become vectors
  17. 2014GloVeELMo built directly on thisChapter 2. Words become vectors
  18. 2014Seq2seq Turning pointTransformer built directly on thisChapter 2. Words become vectors
  19. 2014Attention Turning pointTransformer built directly on thisChapter 3. Attention, then only attention
  20. 2014GRU / encoder-decoderAttention built directly on thisChapter 2. Words become vectors
  21. 2013word2vecELMo was influenced by thisChapter 2. Words become vectors
  22. 2011NLP from scratchELMo was influenced by thisChapter 1. Language as probability
  23. 2003Neural LMNLP from scratch was influenced by this
    word2vec was influenced by this    Chapter 1. Language as probability
  24. 1997LSTM Turning pointSeq2seq built directly on this
    ELMo built directly on this    Chapter 1. Language as probability
  25. 1990Recurrent networksLSTM was influenced by thisChapter 1. Language as probability

After: what it led to

RLHF for LMs has 67 documented descendants across 8 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 5 turning points among them.

RLHF for LMs has 67 documented descendants across 8 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

RLHF for LMs has 85 recorded descendants, including indirect ones, across 8 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 5 turning points among them.

RLHF for LMs has 85 recorded descendants, including indirect ones, across 8 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

  1. 2019RLHF for LMsThis milestone.
  2. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  3. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  4. 2021WebGPTbuilt directly on RLHF for summarisationChapter 13. Models that act
  5. 2022InstructGPT Turning pointbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  6. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  7. 2022ReActwas influenced by WebGPTChapter 13. Models that act
  8. 2022ChatGPTbuilt directly on InstructGPTChapter 7. The assistant goes public
  9. 2022GPT-3.5 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  10. 2022Constitutional AIbuilt directly on HH-RLHFChapter 6. Teaching models what people want
  11. 2023Toolformerwas influenced by WebGPTChapter 13. Models that act
  12. 2023Alpacawas influenced by GPT-3.5Chapter 8. The weights get out
  13. 2023Claudewas influenced by HH-RLHFChapter 7. The assistant goes public
  14. 2023GPT-4 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  15. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  16. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  17. 2023LLaVAbuilt directly on GPT-4Chapter 11. Seeing and hearing
  18. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  19. 2023DPObuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  20. 2023phi-1was influenced by GPT-3.5Chapter 8. The weights get out
  21. 2023Llama 2 Turning pointwas influenced by InstructGPTChapter 8. The weights get out
  22. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  23. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  24. 2024DeepSeek LLMwas influenced by Llama 2Chapter 8. The weights get out
  25. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  26. 2024GRPOwas influenced by InstructGPTChapter 12. Thinking longer
  27. 2024Gemmawas influenced by Llama 2Chapter 8. The weights get out
  28. 2024Claude 3was influenced by GPT-4
    built directly on Claude
    built directly on Constitutional AI    Chapter 14. Where the threads meet now
  29. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  30. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  31. 2024Llama 3built directly on Llama 2Chapter 8. The weights get out
  32. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  33. 2024SWE-agentwas influenced by ReActChapter 13. Models that act
  34. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  35. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  36. 2024FineWebwas influenced by phi-1Chapter 8. The weights get out
  37. 2024Llama 3.1 405Bbuilt directly on DPOChapter 8. The weights get out
  38. 2024Computer usewas influenced by GPT-4VChapter 13. Models that act
  39. 2024DeepSeek-V3built directly on GRPO
    was influenced by Llama 3.1 405B    Chapter 8. The weights get out
  40. 2025DeepSeek-R1built directly on GRPOChapter 12. Thinking longer
  41. 2025Claude Codewas influenced by Computer useChapter 13. Models that act
  42. 2025Claude 3.7built directly on Claude 3.5 SonnetChapter 12. Thinking longer
  43. 2025GPT-4.5built directly on GPT-4oChapter 14. Where the threads meet now
  44. 2025Llama 4succeeded Llama 3.1 405BChapter 8. The weights get out
  45. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  46. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  47. 2025Kimi K2was influenced by GRPOChapter 8. The weights get out
  48. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  49. 2025DeepSeek-V3.2built directly on DeepSeek-V3Chapter 10. Reading more than you remember
  50. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  51. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  52. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  53. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  54. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  55. 2026GPT-5.3-Codexbuilt directly on GPT-5.2Chapter 13. Models that act
  56. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  57. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  58. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  59. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  60. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  61. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  62. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  63. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  64. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  65. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  66. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  67. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  68. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now
  1. 2019RLHF for LMsThis milestone.
  2. 2020RLHF for summarisation Turning pointbuilt directly on RLHF for LMsChapter 6. Teaching models what people want
  3. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  4. 2021WebGPTbuilt directly on RLHF for summarisationChapter 13. Models that act
  5. 2022InstructGPT Turning pointbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  6. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  7. 2022ReActwas influenced by WebGPTChapter 13. Models that act
  8. 2022ChatGPTbuilt directly on InstructGPTChapter 7. The assistant goes public
  9. 2022GPT-3.5 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  10. 2022Constitutional AIbuilt directly on HH-RLHFChapter 6. Teaching models what people want
  11. 2023Toolformerwas influenced by WebGPTChapter 13. Models that act
  12. 2023Alpacawas influenced by GPT-3.5Chapter 8. The weights get out
  13. 2023Claudewas influenced by HH-RLHFChapter 7. The assistant goes public
  14. 2023GPT-4 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  15. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  16. 2023AutoGPTwas influenced by GPT-4
    was influenced by ReAct (indirect influence)    Chapter 13. Models that act
  17. 2023LLaVAbuilt directly on GPT-4Chapter 11. Seeing and hearing
  18. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  19. 2023DPObuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  20. 2023Function callingwas influenced by Toolformer (indirect influence)
    was influenced by ReAct (indirect influence)    Chapter 13. Models that act
  21. 2023phi-1was influenced by GPT-3.5Chapter 8. The weights get out
  22. 2023Lost in the middlewas influenced by 100K context (indirect influence)Chapter 10. Reading more than you remember
  23. 2023Llama 2 Turning pointwas influenced by InstructGPTChapter 8. The weights get out
  24. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  25. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  26. 2024DeepSeek LLMwas influenced by Llama 2Chapter 8. The weights get out
  27. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  28. 2024GRPOwas influenced by InstructGPTChapter 12. Thinking longer
  29. 2024Gemini 1.5was influenced by Lost in the middle (indirect influence)Chapter 10. Reading more than you remember
  30. 2024Gemmawas influenced by Llama 2Chapter 8. The weights get out
  31. 2024Claude 3was influenced by GPT-4
    built directly on Claude
    built directly on Constitutional AI    Chapter 14. Where the threads meet now
  32. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  33. 2024Devinwas influenced by GPT-4
    was influenced by ReAct (indirect influence)    Chapter 13. Models that act
  34. 2024Llama 3built directly on Llama 2Chapter 8. The weights get out
  35. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  36. 2024SWE-agentwas influenced by ReActChapter 13. Models that act
  37. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  38. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  39. 2024FineWebwas influenced by phi-1Chapter 8. The weights get out
  40. 2024Llama 3.1 405Bbuilt directly on DPOChapter 8. The weights get out
  41. 2024Computer usewas influenced by Function calling (indirect influence)
    was influenced by GPT-4V    Chapter 13. Models that act
  42. 2024MCPwas influenced by Toolformer (indirect influence)
    was influenced by ReAct (indirect influence)    Chapter 13. Models that act
  43. 2024Gemini 2.0was influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  44. 2024DeepSeek-V3built directly on GRPO
    was influenced by Llama 3.1 405B    Chapter 8. The weights get out
  45. 2025DeepSeek-R1built directly on GRPOChapter 12. Thinking longer
  46. 2025Deep researchwas influenced by WebGPT (indirect influence)Chapter 13. Models that act
  47. 2025Claude Codewas influenced by SWE-agent (indirect influence)
    was influenced by MCP    Chapter 13. Models that act
  48. 2025Claude 3.7built directly on Claude 3.5 SonnetChapter 12. Thinking longer
  49. 2025GPT-4.5built directly on GPT-4oChapter 14. Where the threads meet now
  50. 2025Gemini 2.5built directly on Gemini 2.0
    was influenced by Gemini 1.5    Chapter 12. Thinking longer
  51. 2025Llama 4succeeded Llama 3.1 405BChapter 8. The weights get out
  52. 2025o3 releasedwas influenced by Function callingChapter 12. Thinking longer
  53. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  54. 2025Claude 4succeeded Claude 3
    was influenced by SWE-bench    Chapter 13. Models that act
  55. 2025Kimi K2was influenced by GRPOChapter 8. The weights get out
  56. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  57. 2025gpt-osswas influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  58. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  59. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  60. 2025DeepSeek-V3.2built directly on DeepSeek-V3Chapter 10. Reading more than you remember
  61. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  62. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  63. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  64. 2025Gemini 3was influenced by Gemini 1.5Chapter 14. Where the threads meet now
  65. 2025Claude Opus 4.5was influenced by SWE-benchChapter 13. Models that act
  66. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  67. 2026GPT-5.3-Codexwas influenced by Computer use (indirect influence)Chapter 13. Models that act
  68. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  69. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3Chapter 12. Thinking longer
  70. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  71. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  72. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  73. 2026Project Glasswingwas influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  74. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  75. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  76. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  77. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  78. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  79. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  80. 2026Fable 5 / Mythos 5was influenced by Project GlasswingChapter 14. Where the threads meet now
  81. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  82. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  83. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  84. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  85. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  86. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now

Direct links and their sources

Where it came from

Built directly on

What it led to

Built on directly by

Terms