- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposes Nash rejection sampling (Nash-RS) for Nash learning from human feedback (NLHF) that accommodates potentially non-transitive preferences without assuming a Bradley–Terry reward model. Theoretically, it shows that the KL-regularized two-player max–min preference game can be reduced to a single concave maximization problem by analytically proving that the opponent's optimal response takes the form of a Gibbs distribution. The gradient of the resulting objective can then be expressed as a standard policy gradient with an implicit scalar reward. The author(s) next proposes to use rejection sampling to approximate the Gibbs opponent, allowing standard on-policy optimization algorithms such as PPO to be applied without training a separate adversarial policy. They further establish linear convergence guarantees to the Nash equilibrium. Experiments on UltraFeedback and PKU-SafeRLHF show that Nash-RS outperforms the several baselines algorithms in pairwise preference comparisons.
Critical strengths
I see three main strengths of this paper: (i) a novel rejection-sampling procedure for implementing Nash learning from human feedback, together with its integration into PPO; (ii) linear convergence guarantees for the policy update; and (iii) promising empirical results demonstrating the proposal's competitive performance against existing state-of-the-art algorithms.
Critical weaknesses
The theoretical results are mathematically elegant. However, the paper does not adequately clarify what is genuinely new over existing NLHF theory. For instance, EGPO already exploits the Gibbs fixed-point characterization of the regularized Nash equilibrium, raising questions about the novelty of the Gibbs best-response characterization derived in this paper. As another example, EGPO already establishes linear last-iterate convergence to the same regularized Nash equilibrium, as the author(s) acknowledges. Although the paper establishes linear convergence for a different update rule, it is not sufficiently clear what theoretical advantages it offers. In fact, it seems EGPO's convergence guarantee has a more favorable dependence on the regularization parameter as it approaches zero.
For the experiments, some of the improvements are quite small. On UltraFeedback, Nash-RS achieves an average win rate of , compared with for Reward-PPO. Its direct win probability against Reward-PPO is only . Similarly, the pairwise win rate against EGPO are on UltraFeedback and on PKU-SafeRLHF.
Moreover, the experiments do not adequately isolate the contribution of rejection sampling. It is true that the author(s) compare against EGPO and Nash-MD, but these algorithms differ in several other aspects beyond their opponent-sampling strategies. For instance, Nash-RS uses PPO, whereas EGPO uses IPO in the implementation.
Finally, I would encourage the authors to include comparisons with DPO-based methods, such as rDPO and Dr. DPO, which are omitted from the experiments.
Questions for authors
Please refer to Weaknesses.
2 comments
| Opponent (same PPO pipeline) | avg. win rate | worst-case win rate | Nash-RS wins head-to-head | vs initial |
|---|---|---|---|---|
| Reference samples (no rejection) | 0.503 ± 0.005 | 0.483 | 0.517 | 0.510 |
| Current-policy samples (PPO self-play) | 0.516 ± 0.005 | 0.496 | 0.504 | 0.522 |
| Gibbs / rejection sampling (Nash-RS) | 0.521 ± 0.005 | 0.504 | — | 0.526 |
With PPO held fixed, removing rejection lowers the average win rate from 0.521 to 0.503, and Nash-RS wins their direct comparison at 0.517. The gain over PPO self-play is smaller: 0.521 versus 0.516, with a head-to-head win rate of 0.504. These results support an additional contribution from the Gibbs-opponent construction in this run.
The PPO self-play variant also substantially outperforms the TRL self-play baseline, which scores 0.489 in the same evaluation population. This supports the reviewer’s concern that the training pipeline contributes to the cross-method differences. We will therefore avoid attributing the entire advantage over the original baselines to rejection sampling.
Computational cost. This ablation matches training steps and opponent counts, not compute. The reference-opponent variant requires 8.2K preference-model calls and 6.95 sampling GPU-hours, compared with 52K calls and 13.1 GPU-hours for Nash-RS. The measured improvement therefore comes with additional sampling cost.
W4. I would encourage the authors to include comparisons with DPO-based methods, such as rDPO and Dr. DPO, which are omitted from the experiments.
A. We added DPO, rDPO (Chowdhury et al., 2024) and Dr. DPO (Wu et al., 2024) on Qwen2.5-3B / UltraFeedback.
Setup,
- All three methods use full-parameter training, β = 0.1, learning rate 5e-7, 2 epochs and batch size 32. They differ only in the loss:
- rDPO uses label-noise rate ε = 0.1;
- Dr. DPO uses β′ = 1.
- They are evaluated in the same population, with the same protocol and test prompts.
| Method | avg. win rate (13-policy population) | Nash-RS wins head-to-head |
|---|---|---|
| DPO | 0.494 ± 0.003 | 0.525 |
| rDPO | 0.491 ± 0.003 | 0.528 |
| Dr. DPO | 0.488 ± 0.004 | 0.531 |
| Nash-RS | 0.521 ± 0.005 | — |
With the same prompts and the same preference oracle, Nash-RS beats each of them head-to-head (0.525–0.531).
- These methods are offline and learn from one fixed pair per prompt. Nash-RS instead queries the oracle on fresh on-policy samples throughout training.
- Because of time limits, we used a single standard hyperparameter setting for each method and did not tune them further.
Conceptual difference. DPO-type methods assume a Bradley–Terry-style link between preferences and an implicit reward, with added robustness to label noise (rDPO) or to pairwise-data perturbations (Dr. DPO). Nash-RS targets the Nash equilibrium of a general, possibly intransitive, preference model.
The revision adds these baselines to the new experiments appendix (they are evaluated in a different, 13-policy population, so we keep them out of Table 2) and discusses them in the related-work appendix (Appendix G).
We thank the reviewer for the questions and comments.
A. The main theoretical contribution is linear convergence for a different type of algorithms. As it it stated, both NashRS and EGPO provide linear convergence for NLHF, but EGPO establishes linear convergence for IPO-type algorithms, and Nash-RS establishes linear convergence for RL-type algorithms. These are different optimization frameworks, and EGPO’s guarantee does not directly apply to our update. Similar to RL/DPO for RLHF, guarantee of RL/IPO for NLHF are both important for NLHF theory.
As for the use of Gibbs, the only similarity is that both NashRS and EGPO use the Gibbs map. However, their role differs. EGPO uses a damped predictor within an extragradient update, whereas Nash-RS uses the full Gibbs best response as the opponent defining the gradient of the reduced scalar objective. Rejection sampling makes this opponent accessible for implicit-reward RL updates. Our contribution is this connection between the Gibbs response, scalar-reward policy optimization, and sampling.
A. We ran two new sets of experiments for this concern; the table are provided in our response to Reviewer k2oa, Q3.
Context for the effect sizes. Win rates in the 0.52–0.53 range are typical for this evaluation protocol: all policies are close to the same reference policy with KL regularization at τ = 0.5, and the comparison is pairwise against strong baselines. On PKU-SafeRLHF, the independent Beaver cost model gives the same ranking as the training oracle (k2oa-3).
A. Thanks for the question. First, we would like to clarify the role of rejection sampling in Nash-RS. Unlike standard RS that was used for selecting training-data, our rejection sampling step are used to provide an unbiased implicit reward in our RL formulation. Then, “PPO+RS” are coupled in the designed of algorithms.
For ablation, an “IPO + RS” variant would not be well-defined. When using IPO, we don’t have an implicit reward, thus we don’t need to use RS to estimate the reward. On the other hand, an “PPO+non-RS” would be well-defined, but not theoretical grounded. It is because it will not provided an unbiased estimation of implicated reward. Nevertheless, the non-RS variant is an informative control for the reviewer’s question: does the Gibbs-opponent construction provide an additional benefit when the PPO pipeline is held fixed?
We ran a controlled ablation (Qwen2.5-3B / UltraFeedback). We only change the opponent distribution.