Direct preference optimization – Definition & Meaning

Definition of direct preference optimization

: a method for aligning a language model with pairs of preferred and rejected responses by directly optimizing the model on those preferences, without first training a separate explicit reward model and then running reinforcement learning
- The team used direct preference optimization on pairs in which reviewers chose the clearer of two answers.
- DPO simplified the post-training pipeline, but its result still depended on the quality and coverage of the preference data.

Origin & history

first known use2023

Rafael Rafailov and collaborators introduced Direct Preference Optimization in a 2023 paper titled Your Language Model is Secretly a Reward Model. The method rewrites the preference-learning objective so that a policy can be trained with a classification-style loss relative to a reference model.

Test yourself

Which of these is the meaning of direct preference optimization?

Related terms

RLHF reinforcement learning from human feedback: a training method in which people rate or compare… reinforcement learning a way of training in which a model tries actions, receives rewards or penalties, and gradually… reward model a separate model trained to score how good a response is, used to guide reinforcement learning… fine-tuning further training of an existing model on a smaller, focused dataset so it gets better at a… post-training everything done to a model after pre-training to make it useful and safe, such as instruction… alignment the work of making an AI system's goals and behavior match what its designers and users…

Cite this entry

"Direct preference optimization." AI Dictionary, Dadgogo, https://dadgogo.com/dictionary/direct-preference-optimization/. Accessed 8 Oct. 2026.