RLHF

RLHF, short for Reinforcement Learning from Human Feedback, is a method for training AI systems on human judgments instead of a hand-written objective. People compare a model’s outputs. A separate reward model learns to predict which output they would prefer, and the original model is then optimized by reinforcement learning to score highly on it.

How it works

In the form that made it famous, OpenAI’s InstructGPT of 2022, the procedure has three steps:

1. Demonstrations. Human labellers write examples of the desired behavior, and a pretrained language model is fine-tuned on them.
2. Comparisons. Labellers rank several of the model’s answers to the same prompt, and a reward model is trained to predict which answer they would prefer.
3. Optimization. The model is trained to maximize the reward model’s score with the reinforcement learning algorithm PPO (Proximal Policy Optimization, John Schulman et al., 2017). A penalty keeps it close to its starting point.

Where the term comes from

Learning from human preference judgments has older roots in robotics and reinforcement learning, but the technique in its modern form goes back to Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg and Dario Amodei in “Deep reinforcement learning from human preferences“, posted in June 2017 and published at NIPS 2017. Non-expert humans compared pairs of short segments of an agent’s behavior in Atari games and simulated robot locomotion. Feedback on less than one percent of the agent’s interactions, and about an hour of human time, sufficed to teach new behaviors. In 2019 Daniel Ziegler and colleagues applied the approach to language models.

The breakthrough came with “Training language models to follow instructions with human feedback” by Long Ouyang and 19 co-authors (March 2022). About 40 contractors supplied the data, and labellers preferred the 1.3-billion-parameter InstructGPT to the 175-billion-parameter GPT-3. ChatGPT, released on November 30, 2022, was trained with “the same methods as InstructGPT”. In 2023 Stephen Casper and 31 co-authors described RLHF as the main technique for fine-tuning state-of-the-art large language models.

Variants

• RLAIF and Constitutional AI. In December 2022 Yuntao Bai and colleagues at Anthropic replaced human labels of harmful outputs with an AI model’s judgments. A written list of principles guided those judgments; the authors called the reinforcement step “RL from AI Feedback”.
• DPO. Direct Preference Optimization (Rafael Rafailov et al., May 2023) turns the RLHF objective into a simple classification loss on preference data, without a reinforcement learning loop.
• RLVR. Reinforcement learning with verifiable rewards scores answers that can be checked automatically, such as maths solutions or code. The name was introduced in the Tülu 3 report (Nathan Lambert et al., November 2024). The International AI Safety Report 2026 names it, beside RLHF, as one of the two reinforcement learning methods of post-training, used for tasks that require factual correctness.

Limits and criticism

• Reward hacking. A reward model is a proxy, and optimizing a proxy too hard degrades true performance. This is Goodhart’s law, measured for RLHF by Leo Gao, John Schulman and Jacob Hilton in 2022. Verifiable rewards are not immune: in March 2025 OpenAI reported a reasoning model that, in training on coding tasks, rewrote a verification function to always return true.
• Sycophancy. Mrinank Sharma and colleagues (2023) found that humans and preference models sometimes prefer well-written answers that agree with the user to correct ones. In April 2025 OpenAI rolled back a GPT-4o update after changes, including a new reward signal from users’ thumbs-up and thumbs-down ratings, made the model sycophantic.
• Whose preferences. The InstructGPT authors noted that their procedure aligns a model to a specific group of people, mostly their labellers and researchers, not to human values in general.
• Surface or substance. The LIMA study (Chunting Zhou et al., 2023) proposed a “Superficial Alignment Hypothesis”: knowledge and capabilities come from pretraining, and alignment mainly teaches format. Other work shows narrow fine-tuning changing behavior broadly. The Shoggoth meme turns this debate into a picture.

Why it matters now

The International AI Safety Report 2026 observes that capability gains increasingly come from techniques applied after a model’s initial training. RLHF and its successors therefore decide not only what models can do but how they treat the people who use them.

Related

Shoggoth · A Thumb on the Scale (essay) · Paperclip Maximizer · Sharp Left Turn · Scaling Laws · I’m Upping My P(doom) — when the machine sings of its own undoing (essay on the song) · Reward Hacking

Sources: Christiano et al.: Deep reinforcement learning from human preferences (arXiv) · NeurIPS proceedings entry · Ziegler et al.: Fine-Tuning Language Models from Human Preferences · Ouyang et al.: Training language models to follow instructions with human feedback · Schulman et al.: Proximal Policy Optimization Algorithms · OpenAI: Introducing ChatGPT · Casper et al.: Open Problems and Fundamental Limitations of RLHF · Bai et al.: Constitutional AI · Anthropic: Constitutional AI · Rafailov et al.: Direct Preference Optimization · Lambert et al.: Tülu 3 · International AI Safety Report 2026 · Gao et al.: Scaling Laws for Reward Model Overoptimization · OpenAI: Detecting misbehavior in frontier reasoning models · Sharma et al.: Towards Understanding Sycophancy in Language Models · OpenAI: Sycophancy in GPT-4o · OpenAI: Expanding on what we missed with sycophancy · Zhou et al.: LIMA · Betley et al.: Emergent Misalignment