Deep Dives
DPO: How Direct Preference Optimization Replaced RLHF
Direct Preference Optimization (DPO), introduced in a 2023 NeurIPS paper by Rafailov et al., aligns language models directly on preference pairs without training a separate reward model or running reinforcement learning. It replaces RLHF's fragile four-model PPO pipeline with a single supervised loss governed mainly by one parameter, beta, and works best stacked after SFT on subjective tasks — not on problems with a single correct answer.
