What is RLHF and why is it used to align language models?

RLHF (Reinforcement Learning from Human Feedback) aligns a pretrained/instruction-tuned model with human preferences. It has three stages: (1) supervised fine-tuning on demonstration data to get a reasonable base; (2) train a reward model by having humans rank multiple model responses to the same prompt, teaching a model to predict which output people prefer; (3) optimize the LLM against that reward model using RL, typically PPO, with a KL-divergence penalty that keeps the policy from drifting too far from the SFT model and degrading language quality. RLHF is what makes models helpful, harmless, and honest, reducing toxic or off-task outputs, because raw next-token prediction optimizes for likelihood, not human usefulness. Simpler preference-optimization methods like DPO now often replace the explicit reward-model + PPO loop, directly optimizing on preference pairs while achieving similar alignment.