NLP Lineage

12 June 2017 · Alignment

RLHF origin

Deep Reinforcement Learning from Human Preferences

P. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, D. Amodei

Date
12 June 2017
What the date marks
arXiv submission
Kind
Training method
Organisation
OpenAI / DeepMind
Access
Not applicable
Other dates
NeurIPS 2017
Threads
In the guided tour
Chapter 6. Teaching models what people want
Learning phases
2. How LLMs are trained

A milestone: 3 later milestones descend directly from it, and its descendants reach 10 of the 10 threads. How levels are decided

Read the primary source (arxiv.org)

Full summary in progress. Until then, this page shows the milestone's record from the research data.

What it changed

Before it

Reinforcement learning needed a hand-written reward function.

The problem

Many goals are easy to judge and hard to specify. You know good behaviour when you see it.

What it introduced

Learn a reward model from human comparisons between pairs of behaviours, then optimise against it.

Why it matters

The whole alignment stack starts here. InstructGPT, ChatGPT and Claude are this method applied to language.

What modern models inherit

Reward models learned from preference comparisons.

What came next

Ziegler applied it to language models in 2019, then summarisation, then instruction following.

Before and after

The milestones it descends from, and the milestones that descend from it. Work developed alongside it is not counted. Weak links are left out unless you include them.

Before: where it came from

No documented ancestors are recorded for RLHF origin without weak links.

No documented ancestors are recorded for RLHF origin without weak links.

No recorded ancestors are recorded for RLHF origin.

No recorded ancestors are recorded for RLHF origin.

After: what it led to

RLHF origin has 68 documented descendants across 8 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 5 turning points among them.

RLHF origin has 68 documented descendants across 8 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

RLHF origin has 88 recorded descendants, including indirect ones, across 8 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing its direct descendants and the 6 turning points among them.

RLHF origin has 88 recorded descendants, including indirect ones, across 8 chapters, reaching forward to Claude Opus 5.5 in 2026. Showing all of them.

  1. 2017RLHF originThis milestone.
  2. 2019RLHF for LMsbuilt directly on RLHF originChapter 6. Teaching models what people want
  3. 2020RLHF for summarisation Turning pointbuilt directly on RLHF originChapter 6. Teaching models what people want
  4. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  5. 2021WebGPTbuilt directly on RLHF for summarisationChapter 13. Models that act
  6. 2022InstructGPT Turning pointbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  7. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  8. 2022ReActwas influenced by WebGPTChapter 13. Models that act
  9. 2022ChatGPTbuilt directly on InstructGPTChapter 7. The assistant goes public
  10. 2022GPT-3.5 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  11. 2022Constitutional AIwas influenced by RLHF originChapter 6. Teaching models what people want
  12. 2023Toolformerwas influenced by WebGPTChapter 13. Models that act
  13. 2023Alpacawas influenced by GPT-3.5Chapter 8. The weights get out
  14. 2023Claudebuilt directly on Constitutional AIChapter 7. The assistant goes public
  15. 2023GPT-4 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  16. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  17. 2023AutoGPTwas influenced by GPT-4Chapter 13. Models that act
  18. 2023LLaVAbuilt directly on GPT-4Chapter 11. Seeing and hearing
  19. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  20. 2023DPObuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  21. 2023phi-1was influenced by GPT-3.5Chapter 8. The weights get out
  22. 2023Llama 2 Turning pointwas influenced by InstructGPTChapter 8. The weights get out
  23. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  24. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  25. 2024DeepSeek LLMwas influenced by Llama 2Chapter 8. The weights get out
  26. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  27. 2024GRPOwas influenced by InstructGPTChapter 12. Thinking longer
  28. 2024Gemmawas influenced by Llama 2Chapter 8. The weights get out
  29. 2024Claude 3built directly on Constitutional AIChapter 14. Where the threads meet now
  30. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  31. 2024Devinwas influenced by GPT-4Chapter 13. Models that act
  32. 2024Llama 3built directly on Llama 2Chapter 8. The weights get out
  33. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  34. 2024SWE-agentwas influenced by ReActChapter 13. Models that act
  35. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  36. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  37. 2024FineWebwas influenced by phi-1Chapter 8. The weights get out
  38. 2024Llama 3.1 405Bbuilt directly on DPOChapter 8. The weights get out
  39. 2024Computer usebuilt directly on Claude 3.5 SonnetChapter 13. Models that act
  40. 2024DeepSeek-V3built directly on GRPO
    was influenced by Llama 3.1 405B    Chapter 8. The weights get out
  41. 2025DeepSeek-R1built directly on GRPOChapter 12. Thinking longer
  42. 2025Claude 3.7built directly on Claude 3.5 SonnetChapter 12. Thinking longer
  43. 2025Claude Codewas influenced by Computer use
    was influenced by Claude 3.7    Chapter 13. Models that act
  44. 2025GPT-4.5built directly on GPT-4oChapter 14. Where the threads meet now
  45. 2025Llama 4succeeded Llama 3.1 405BChapter 8. The weights get out
  46. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  47. 2025Claude 4succeeded Claude 3Chapter 13. Models that act
  48. 2025Kimi K2was influenced by GRPOChapter 8. The weights get out
  49. 2025GPT-5succeeded GPT-4oChapter 14. Where the threads meet now
  50. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  51. 2025DeepSeek-V3.2built directly on DeepSeek-V3Chapter 10. Reading more than you remember
  52. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  53. 2025Claude Opus 4.5built directly on Claude Sonnet 4.5
    was influenced by SWE-bench    Chapter 13. Models that act
  54. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  55. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  56. 2026GPT-5.3-Codexbuilt directly on GPT-5.2Chapter 13. Models that act
  57. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  58. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  59. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  60. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  61. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  62. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  63. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  64. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  65. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  66. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  67. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  68. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  69. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now
  1. 2017RLHF originThis milestone.
  2. 2019RLHF for LMsbuilt directly on RLHF originChapter 6. Teaching models what people want
  3. 2020RLHF for summarisation Turning pointbuilt directly on RLHF originChapter 6. Teaching models what people want
  4. 2021HHH assistantwas influenced by RLHF for summarisationChapter 6. Teaching models what people want
  5. 2021WebGPTbuilt directly on RLHF for summarisationChapter 13. Models that act
  6. 2022InstructGPT Turning pointbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  7. 2022HH-RLHFbuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  8. 2022ReActwas influenced by WebGPTChapter 13. Models that act
  9. 2022ChatGPTbuilt directly on InstructGPTChapter 7. The assistant goes public
  10. 2022GPT-3.5 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  11. 2022Constitutional AIwas influenced by RLHF originChapter 6. Teaching models what people want
  12. 2023Toolformerwas influenced by WebGPTChapter 13. Models that act
  13. 2023Alpacawas influenced by GPT-3.5Chapter 8. The weights get out
  14. 2023Claudebuilt directly on Constitutional AIChapter 7. The assistant goes public
  15. 2023GPT-4 Turning pointbuilt directly on InstructGPTChapter 7. The assistant goes public
  16. 2023Reflexionbuilt directly on ReActChapter 13. Models that act
  17. 2023AutoGPTwas influenced by GPT-4
    was influenced by ReAct (indirect influence)    Chapter 13. Models that act
  18. 2023LLaVAbuilt directly on GPT-4Chapter 11. Seeing and hearing
  19. 2023100K contextbuilt directly on ClaudeChapter 10. Reading more than you remember
  20. 2023DPObuilt directly on RLHF for summarisationChapter 6. Teaching models what people want
  21. 2023Function callingwas influenced by Toolformer (indirect influence)
    was influenced by ReAct (indirect influence)    Chapter 13. Models that act
  22. 2023phi-1was influenced by GPT-3.5Chapter 8. The weights get out
  23. 2023Lost in the middlewas influenced by 100K context (indirect influence)Chapter 10. Reading more than you remember
  24. 2023Llama 2 Turning pointwas influenced by InstructGPTChapter 8. The weights get out
  25. 2023GPT-4Vbuilt directly on GPT-4Chapter 11. Seeing and hearing
  26. 2023SWE-benchwas influenced by GPT-4Chapter 13. Models that act
  27. 2024DeepSeek LLMwas influenced by Llama 2Chapter 8. The weights get out
  28. 2024OLMowas influenced by Llama 2Chapter 8. The weights get out
  29. 2024GRPOwas influenced by InstructGPTChapter 12. Thinking longer
  30. 2024Gemini 1.5was influenced by Lost in the middle (indirect influence)Chapter 10. Reading more than you remember
  31. 2024Gemmawas influenced by Llama 2Chapter 8. The weights get out
  32. 2024Claude 3built directly on Constitutional AIChapter 14. Where the threads meet now
  33. 2024Chatbot Arenawas influenced by ChatGPTChapter 14. Where the threads meet now
  34. 2024Devinwas influenced by GPT-4
    was influenced by ReAct (indirect influence)    Chapter 13. Models that act
  35. 2024Llama 3built directly on Llama 2Chapter 8. The weights get out
  36. 2024Phi-3built directly on phi-1Chapter 8. The weights get out
  37. 2024SWE-agentwas influenced by ReActChapter 13. Models that act
  38. 2024GPT-4obuilt directly on GPT-4Chapter 11. Seeing and hearing
  39. 2024Claude 3.5 Sonnetbuilt directly on Claude 3Chapter 14. Where the threads meet now
  40. 2024FineWebwas influenced by phi-1Chapter 8. The weights get out
  41. 2024Llama 3.1 405Bbuilt directly on DPOChapter 8. The weights get out
  42. 2024o1 Turning pointwas influenced by RLHF origin (indirect influence)Chapter 12. Thinking longer
  43. 2024Computer usebuilt directly on Claude 3.5 SonnetChapter 13. Models that act
  44. 2024MCPwas influenced by Toolformer (indirect influence)
    was influenced by ReAct (indirect influence)    Chapter 13. Models that act
  45. 2024Gemini 2.0was influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  46. 2024o3 announcedbuilt directly on o1Chapter 12. Thinking longer
  47. 2024DeepSeek-V3built directly on GRPO
    was influenced by Llama 3.1 405B    Chapter 8. The weights get out
  48. 2025DeepSeek-R1was influenced by o1Chapter 12. Thinking longer
  49. 2025Deep researchwas influenced by o3 released
    was influenced by WebGPT (indirect influence)    Chapter 13. Models that act
  50. 2025Claude 3.7was influenced by o1Chapter 12. Thinking longer
  51. 2025Claude Codewas influenced by Claude 3.7Chapter 13. Models that act
  52. 2025GPT-4.5built directly on GPT-4oChapter 14. Where the threads meet now
  53. 2025Gemini 2.5was influenced by o1Chapter 12. Thinking longer
  54. 2025Llama 4succeeded Llama 3.1 405BChapter 8. The weights get out
  55. 2025o3 releasedbuilt directly on o1Chapter 12. Thinking longer
  56. 2025Qwen3was influenced by DeepSeek-R1Chapter 8. The weights get out
  57. 2025Claude 4built directly on Claude 3.7
    succeeded Claude 3    Chapter 13. Models that act
  58. 2025Kimi K2was influenced by GRPOChapter 8. The weights get out
  59. 2025IMO goldbuilt directly on Gemini 2.5Chapter 12. Thinking longer
  60. 2025gpt-ossbuilt directly on o3 released
    was influenced by DeepSeek-R1 (indirect influence)    Chapter 8. The weights get out
  61. 2025GPT-5built directly on o3 releasedChapter 14. Where the threads meet now
  62. 2025GLM-4.5was influenced by DeepSeek-R1 (indirect influence)Chapter 8. The weights get out
  63. 2025Claude Sonnet 4.5built directly on Claude 4Chapter 13. Models that act
  64. 2025DeepSeek-V3.2built directly on DeepSeek-V3Chapter 10. Reading more than you remember
  65. 2025IMO-ProofBenchwas influenced by IMO goldChapter 12. Thinking longer
  66. 2025GPT-5.1built directly on GPT-5Chapter 14. Where the threads meet now
  67. 2025Gemini 3built directly on Gemini 2.5Chapter 14. Where the threads meet now
  68. 2025Claude Opus 4.5built directly on Claude Sonnet 4.5
    was influenced by SWE-bench    Chapter 13. Models that act
  69. 2025GPT-5.2built directly on GPT-5.1Chapter 14. Where the threads meet now
  70. 2026GPT-5.3-Codexwas influenced by Computer use (indirect influence)Chapter 13. Models that act
  71. 2026Claude Opus 4.6built directly on Claude Opus 4.5Chapter 14. Where the threads meet now
  72. 2026Gemini 3 Deep Thinkbuilt directly on Gemini 3
    was influenced by IMO gold    Chapter 12. Thinking longer
  73. 2026Qwen3.5built directly on Qwen3Chapter 8. The weights get out
  74. 2026Gemini 3.1built directly on Gemini 3Chapter 14. Where the threads meet now
  75. 2026GPT-5.4built directly on GPT-5.3-CodexChapter 14. Where the threads meet now
  76. 2026Project Glasswingwas influenced by Computer use (indirect influence)Chapter 14. Where the threads meet now
  77. 2026Muse Sparksucceeded Llama 4Chapter 8. The weights get out
  78. 2026Claude Opus 4.7built directly on Claude Opus 4.6Chapter 14. Where the threads meet now
  79. 2026GPT-5.5built directly on GPT-5.4Chapter 14. Where the threads meet now
  80. 2026DeepSeek-V4was influenced by DeepSeek-V3Chapter 8. The weights get out
  81. 2026Claude Opus 4.8built directly on Claude Opus 4.7Chapter 14. Where the threads meet now
  82. 2026Claude Sonnet 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  83. 2026Fable 5 / Mythos 5was influenced by Project GlasswingChapter 14. Where the threads meet now
  84. 2026GLM-5.2built directly on GLM-4.5Chapter 8. The weights get out
  85. 2026GPT-5.6 Solbuilt directly on GPT-5.5Chapter 14. Where the threads meet now
  86. 2026Kimi K3built directly on Kimi K2Chapter 8. The weights get out
  87. 2026Claude Opus 5succeeded Claude Opus 4.8Chapter 14. Where the threads meet now
  88. 2026Qwen3.8built directly on Qwen3.5Chapter 8. The weights get out
  89. 2026Claude Opus 5.5built directly on Claude Opus 5Chapter 14. Where the threads meet now

Direct links and their sources

Where it came from

None recorded in the sources used.

What it led to

Built on directly by

Influenced

Terms