acceptodds
Under review as a conference paper at ICLR 2027

Reinforced DPO: Single-Stage Preference Optimization Provably Equivalent to RLHF

11 views

Abstract

Direct preference optimization (DPO) aligns language models in a single training stage, offering a simpler and more stable alternative to the two-stage pipeline of reinforcement learning from human feedback (RLHF). Yet despite being derived from the RLHF objective, DPO fails to reproduce the desirable properties of RLHF: it learns only from offline preference data without on-policy learning, and it does not effectively regularize the policy toward the reference model. In this paper, we first prove that DPO provably deviates from RLHF in both respects. We then trace this inequivalence to a gap in DPO's derivation: the change of variables from rewards to Q-functions silently drops the reward-boundedness constraint that makes RLHF's KL-regularized RL stage well posed. Reinstating this constraint, which takes the form of a Bellman constraint on Q-functions, yields Reinforced Direct Preference Optimization (ReDPO). We prove that ReDPO is equivalent to RLHF and thus inherits its bound on the divergence from the reference policy. Crucially, the Bellman constraint induces a value flow that propagates both the successor Q-values and the reference log-probability into the current Q-value, carrying out KL-regularized RL through dynamic programming. Experiments show that ReDPO clearly outperforms DPO and its variants, while keeping the policy closer to the reference model and mitigating the forgetting of general capabilities. To our knowledge, ReDPO is the first single-stage preference optimization method that is provably equivalent to RLHF.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Learning Theory, Reinforcement Learning from Human Feedback

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fKNGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on the relationship between DPO and RLHF. It proposes ReDPO, which combines preference optimization with a temporal-difference penalty on on-policy rollouts. It establishes equivalence to RLHF for a constrained formulation and reports improvements over DPO on AlpacaEval 2 and Arena-Hard using Llama-3.2-3B and Qwen2.5-7B.

Critical strengths
  • The paper addresses an important question about the relationship between DPO and RLHF.

  • The evaluation includes both preference optimization and RLHF baselines, along with general-capability benchmarks.

Critical weaknesses
  • Theorem 1 does not cover the practical training objective. It requires bounded implicit rewards at every state-action pair in Eq. (7), whereas Eq. (8) applies a finite squared penalty on sampled rollouts and pushes rewards toward zero without enforcing the stated bounds. The equivalence to RLHF and the KL guarantee are therefore not established for the actual algorithm.

  • The supplied implementation does not follow Eq. (8). The equation stops gradients only through the next-state value, but online_rein_trainer.py also detaches the current-state value. This changes the gradient of the penalty, leaving a mismatch between the described algorithm and the implementation used to support it.

  • The distinction from Inverse Preference Learning (Hejna and Sadigh, NeurIPS 2023) is insufficiently addressed. IPL already learns from preferences using implicit rewards defined through Q-functions and applies an L2 penalty to those rewards, without a separate reward model. This closely related work is not discussed, making the additional methodological contribution of ReDPO unclear.

  • The experiments do not isolate the contribution of the proposed value propagation. There is no comparison with the same penalty applied to offline data, or with DPO plus an explicit KL penalty on the same online rollouts. These controls are needed to explain which part of the method produces the gains.

  • The reward setup for PPO and GRPO is not clearly specified. Appendix D names ArmoRM for constructing preference pairs, but does not explain which reward model is used during RL or whether it is trained on the same preferences. This missing detail makes the comparison with RLHF difficult to assess.

Questions for authors

Please see the weaknesses above.

Reviewer_3vZgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper attributes discrepancies between DPO and bounded-reward RLHF to reward constraints omitted during reparameterization. It proposes ReDPO, proves equivalence for an ideal Bellman-constrained formulation, and implements a squared TD penalty on on-policy rollouts. Experiments on two instruction-tuned models report improved preference alignment, lower KL divergence, and reduced forgetting.

Critical strengths
  1. The relationship between DPO and RLHF, particularly policy drift and forgetting, is relevant to language-model alignment.

  2. Translating reward bounds into Bellman inequality constraints clarifies the correspondence between reward and Q-function classes under idealized assumptions.

  3. ReDPO improves Qwen's AlpacaEval LC from 41.31 to 48.44 and Arena-Hard SC from 48.2 to 54.5 over DPO, with improvements also observed on Llama.

Critical weaknesses
  1. Theorem 1 explicitly assumes a solution to Equation (7); using an approximate implementation does not refute that theorem. However, Section 4.3 states that Equation (8) enforces the Bellman constraint and keeps implicit rewards bounded, without establishing this connection. A one-step example illustrates the difficulty. Take reward bounds [−1,1][-1,1], one preference w≻lw\succ l, reference probabilities p,1−pp,1-p, and

    Q(w)=βlog⁡p,Q(l)=βlog⁡(1−p)−M.Q(w)=\beta\log p,\qquad Q(l)=\beta\log(1-p)-M.

    Then rQ(w)=0r_Q(w)=0, rQ(l)=−Mr_Q(l)=-M, and πQ(l)=O(e−M/β)\pi_Q(l)=O(e^{-M/\beta}). For fixed α,β>0\alpha,\beta>0, the expected penalty M2πQ(l)M^2\pi_Q(l) vanishes and the preference term log⁡σ(M)\log\sigma(M) approaches zero, the scalar objective's global supremum. Yet the reward bound is violated and KL approaches log⁡(1/p)\log(1/p), which can exceed 2/β2/\beta for sufficiently small pp. Thus, near-optimal objective values do not certify the theorem's feasibility or KL bound. This construction concerns expected loss values under fresh on-policy sampling; it does not prove that Algorithm 1's sampled updates converge to this sequence. The authors should establish an approximate guarantee under explicit coverage assumptions, or limit the guarantee to Equation (7) and empirically assess constraint violations.

  2. “Difference 2: Loss of Reference Regularization” is too categorical. DPO retains reference dependence through policy/reference log-ratios and can recover the KL-regularized RLHF optimum under matched function classes, realizable preferences, sufficient coverage, and exact optimization. Proposition 2 establishes the absence of a uniform KL bound for unrestricted DPO in a separable empirical construction, not the absence of implicit reference regularization; its comparison uses different reward/policy constraints. Su et al. [2] analyze the reference policy as a prior (Equations 11–12), target distributions, and sampling-related corrections (Section 4), but are not discussed. The authors should distinguish weaker control under particular assumptions from a general “loss” of regularization. The omitted DPO-COV equivalence result [1, Proposition 2] also requires comparison before asserting the broad “first single-stage” claim.

  3. The theory defines πQθ=softmax⁡(Qθ/β)\pi_{Q_\theta}=\operatorname{softmax}(Q_\theta/\beta) but identifies QθQ_\theta with logits. If these are raw logits zθz_\theta, β=0.02\beta=0.02 multiplies pairwise log-odds by 50 relative to temperature 1, potentially changing sampling entropy and the rollout distribution. Such changes could contribute to the gains independently of the proposed TD mechanism. If Qθ=βzθQ_\theta=\beta z_\theta, this temperature change disappears, but the TD penalty scale also needs clarification. The authors should specify Q/logit scaling and rollout/evaluation temperatures, vary β\beta and rollout temperature independently, and compare training with and without the TD penalty at fixed temperature. Evaluation decoding settings should be matched across baselines, with rollout temperatures additionally controlled for online baselines. Report win rates, KL, entropy, and response length to quantify the contribution of temperature.

  4. The experiments do not isolate on-policy sampling, successor values, reference anchoring, or stop-gradient, or adequately assess sensitivity to α\alpha and rollout count. Figures 1–3 track outcomes rather than directly testing value flow; TD-residual/implicit-reward distributions, bound-violation rates, and Q/V evolution along trajectories would provide stronger evidence. More importantly, fresh rollouts and TD training introduce additional computation beyond offline DPO. Matching two rollouts per prompt with PPO/GRPO does not match total generation, policy/reference/reward/critic processing, and optimization costs. The paper should report GPU-hours or FLOPs and performance-versus-compute curves, reallocating ReDPO's budget to additional SFT on available supervised/preferred responses, further DPO optimization, or PPO/GRPO training with appropriately tuned rollout/update allocations. Comparisons should use the same initialization and data-access conditions and account for any reward-model training or labeling costs. Without these controls, it is unclear whether ReDPO's advantage reflects the TD mechanism or additional computation. Multi-seed uncertainty is also needed for the claimed elimination of the alignment tax.

    Validation is required using state-of-the-art models (such as Qwen 3.5), larger model sizes, and a more diverse range of data. The set of baseline algorithms should also be expanded to include a broader variety of approaches. Furthermore, given that the method is theoretically equivalent to RLHF, its effectiveness should be evaluated within the online RLHF paradigm. The current version still relies on DprefD^{pref} which is identical to the offline setting used in DPO, meaning that the point "Difference 1: No On-Policy Learning" remains applicable.

Questions for authors
  1. Under what coverage and implementation assumptions does the sampled TD penalty control the pointwise reward bounds or KL divergence required by Theorem 1? If no such guarantee is available, can the authors explicitly limit the theorem to Equation (7) and report constraint violations for Algorithm 1?

  2. Can the authors distinguish implicit reference regularization from the lack of a uniform KL bound, reconcile Difference 2 with the prior and target-distribution analysis in [2], and clarify novelty relative to DPO-COV [1]?

  3. Is QθQ_\theta raw or β\beta-scaled logits, and what rollout/evaluation temperatures are used? After separating β\beta from temperature, how much improvement remains under matched decoding settings and with versus without the TD penalty at fixed rollout temperature?

  4. At the same total GPU-hour/FLOP budget, how does ReDPO compare with additional SFT, further DPO optimization, and PPO/GRPO with tuned rollout/update allocations? Please report the cost breakdown and performance-versus-compute curves, alongside component ablations, α\alpha/rollout-count sensitivity, and TD-residual, reward-violation, and Q/V visualizations with uncertainty estimates.

[1] Chen, Z., Li, J., Yu, P., and Huang, H. Provably Mitigating Corruption, Overoptimization, and Verbosity Simultaneously in Offline and Online RLHF/DPO Alignment. arXiv:2510.05526

[2] Su, X., et al. Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms. arXiv:2502.03095

Reviewer_N1xGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies DPO's relationship to RLHF with bounded rewards. It expresses reward bounds as constraints on an implicit Q-function and proves that solving the constrained preference objective recovers a corresponding RLHF solution. The practical algorithm adds a squared Bellman-residual penalty on fresh policy rollouts. Experiments on two language models report better preference scores, lower divergence from the reference, and better retention of general capabilities.

Critical strengths

S1. The constrained formulation preserves the intended reward class. The Bellman constraints provide a concrete way to express bounded reward fitting in Q-space. The proof connects a constrained optimizer to its corresponding RLHF reward solution and KL bound. This is a useful specialization of the broader reward-policy correspondence.

S2. The practical regularizer has positive empirical evidence. ReDPO improves over DPO on all reported preference metrics and has the best result in six of eight win-rate columns. The comparisons include length- and style-controlled scores, common preference data, several baselines, and matched rollout counts for the online methods. Lower KL accompanies better retained capability scores. These results support practical value even though they do not verify the claimed equivalence.

Critical weaknesses

W1. The squared penalty changes the optimization problem, even when the reward bounds hold. Theorem 1 concerns the hard constraints in Equation 7. Equation 8 penalizes reward magnitude inside the feasible interval as well. In a one-step, two-action example with a uniform reference, bounds [-1, 1], and α=β=1\alpha=\beta=1, the expected practical update can converge to feasible rewards (0.239, -0.5). Its preferred-action probability is 0.677, while the bounded-reward RLHF solution gives 0.881. Both actions are sampled. The gap therefore concerns the fitted objective as well as coverage. The exact theorem survives, but the practical method does not inherit its equivalence merely by keeping residuals bounded. The claims in Sections 4.3 and 6.2 need this distinction.

W2. The practical construction needs a closer comparison with IPL. Inverse Preference Learning (Hejna and Sadigh, 2023) already combines preference fitting through inverse Bellman rewards with L2 regularization of those rewards. It also gives a conditional convergence result for the corresponding regularized reward fit. This directly relevant work is not cited. ReDPO's hard-bounded formulation and on-policy LLM implementation remain differences, but the broad claim to be the first such method obscures the narrower contribution. This is a positioning issue about the main construction, not a request to reproduce a robotics baseline.

Questions for authors

Q1. Is Equation 8 intended to recover the bounded-reward optimum or a different regularized reward fit? Please separate its claim from Theorem 1, or provide an argument that addresses both feasibility and the changed optimum.

Q2. What is the precise addition over IPL for the practical objective and its guarantee? Please distinguish the hard-bounded result from the on-policy implementation. Both questions can be addressed from existing analysis.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.