What is model alignment, and how do RLHF and Constitutional AI approach it?

Alignment means getting a model to pursue what its developers and users actually intend — being helpful, honest, and harmless — rather than just optimizing its training objective. RLHF (Reinforcement Learning from Human Feedback) collects human preference comparisons between model outputs, trains a reward model on them, then fine-tunes the model to score higher, steering behavior toward preferred responses. Constitutional AI (Anthropic) reduces reliance on human labels for harmlessness by giving the model a written set of principles (a 'constitution') and having it critique and revise its own outputs against them, using AI feedback (RLAIF) for part of the loop. Both improve behavior but are imperfect: reward models can be gamed, preferences encode the labelers' values, and neither guarantees alignment on novel situations — an active research area, not a solved problem.