ChatGPT’s remarkable ability to be helpful, harmless, and honest is not an accident — it is the result of a specific technique called Reinforcement Learning from Human Feedback (RLHF). This approach transformed a powerful but unaligned base model into an AI assistant that millions trust daily.
The Alignment Problem
Base language models are trained to predict the next token — not to be helpful, truthful, or safe. A raw model asked “How do I make a bomb?” will provide detailed instructions because its training objective is simply to complete the text. Alignment is the process of tuning a model’s behavior to align with human values and intentions.
The InstructGPT Paper (2022)
OpenAI’s InstructGPT paper laid out a three-stage RLHF pipeline that became the blueprint for ChatGPT:
Stage 1: Supervised Fine-Tuning (SFT)
Human labelers write ideal responses to a diverse set of prompts. The base model is fine-tuned on these demonstration pairs. This gives the model an initial sense of what good responses look like — concise, accurate, and helpful.
Stage 2: Reward Modeling
For a given prompt, the SFT model generates multiple responses. Human labelers rank these responses from best to worst. A reward model — typically another Transformer — is trained to predict these human preferences. The reward model learns: “Given this prompt and this response, how good would a human judge this to be?”
Stage 3: Reinforcement Learning with PPO
Using Proximal Policy Optimization (PPO), the policy (language model) generates responses, the reward model scores them, and the policy is updated to maximize expected reward. A KL-divergence penalty prevents the policy from deviating too far from the SFT model — maintaining coherence while improving alignment.
Why RLHF Works
RLHF works because human preferences capture nuances that simple metrics cannot:
- Humans prefer responses that are accurate over those that sound confident but are wrong
- Humans prefer responses that refuse harmful requests over those that comply with them
- Humans prefer concise answers that get to the point
- Humans prefer responses that acknowledge uncertainty rather than fabricating information
These preferences are difficult to encode as a loss function but easy for humans to express as rankings.
Limitations and Criticisms
- Reward Hacking: Models may learn to exploit the reward model’s weaknesses rather than genuinely improving
- Human Bias: The preferences of a small group of labelers (often from specific cultural backgrounds) may not represent diverse global values
- Cost: Human labeling is expensive and slow — tens of thousands of comparisons are needed
- Scalability: As models improve, humans may struggle to reliably judge their outputs, creating a “superhuman evaluation” problem
Beyond RLHF: DPO and RLAIF
Newer approaches address RLHF’s limitations:
- Direct Preference Optimization (DPO): Eliminates the separate reward model, directly optimizing the policy from preference data — simpler and more stable than RLHF
- RLAIF (RL from AI Feedback): Uses another AI model (like GPT-4) to provide feedback instead of humans, dramatically reducing costs and enabling faster iteration