- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper focuses on the relationship between DPO and RLHF. It proposes ReDPO, which combines preference optimization with a temporal-difference penalty on on-policy rollouts. It establishes equivalence to RLHF for a constrained formulation and reports improvements over DPO on AlpacaEval 2 and Arena-Hard using Llama-3.2-3B and Qwen2.5-7B.
Critical strengths
-
The paper addresses an important question about the relationship between DPO and RLHF.
-
The evaluation includes both preference optimization and RLHF baselines, along with general-capability benchmarks.
Critical weaknesses
-
Theorem 1 does not cover the practical training objective. It requires bounded implicit rewards at every state-action pair in Eq. (7), whereas Eq. (8) applies a finite squared penalty on sampled rollouts and pushes rewards toward zero without enforcing the stated bounds. The equivalence to RLHF and the KL guarantee are therefore not established for the actual algorithm.
-
The supplied implementation does not follow Eq. (8). The equation stops gradients only through the next-state value, but online_rein_trainer.py also detaches the current-state value. This changes the gradient of the penalty, leaving a mismatch between the described algorithm and the implementation used to support it.
-
The distinction from Inverse Preference Learning (Hejna and Sadigh, NeurIPS 2023) is insufficiently addressed. IPL already learns from preferences using implicit rewards defined through Q-functions and applies an L2 penalty to those rewards, without a separate reward model. This closely related work is not discussed, making the additional methodological contribution of ReDPO unclear.
-
The experiments do not isolate the contribution of the proposed value propagation. There is no comparison with the same penalty applied to offline data, or with DPO plus an explicit KL penalty on the same online rollouts. These controls are needed to explain which part of the method produces the gains.
-
The reward setup for PPO and GRPO is not clearly specified. Appendix D names ArmoRM for constructing preference pairs, but does not explain which reward model is used during RL or whether it is trained on the same preferences. This missing detail makes the comparison with RLHF difficult to assess.
Questions for authors
Please see the weaknesses above.