acceptodds
Under review as a conference paper at ICLR 2027

Efficient Exploration for Iterative Nash Preference Optimization

Abstract

Preference alignment is central to improving large language models, but standard reward-based formulations can be restrictive when human preferences are cyclic, non-transitive. Nash Learning from Human Feedback (NLHF) addresses this limitation by modeling alignment as a preference game and targeting a Nash equilibrium. However, the learning-theoretic foundations of scalable NLHF remain limited. Existing regret guarantees requires training a general preference model and solving minimax oracles, while iterative NLHF methods are easier to implement but lack regret guarantees. In this paper, we study online iterative NLHF and identify exploration as the key obstacle. First, we show that standard iterative NLHF can suffer an exponential dependence on the KL-regularization parameter, revealing that implicit exploration through policy updates is insufficient for efficient exploration. Second, we propose Exploratory Nash Policy Optimization (ENPO) that combines SFT-based regularization with adversarial policy exploration. ENPO eliminates the exponential dependency without using any minimax oracles and explicit preference model estimations. Additionally, we present another theoretical algorithm: Bonus-explorer ENPO (BENPO) with an explicit exploration mechanism, which uses additionally oracles but can guarantee an regret bound. A practical algorithm: Direct ENPO (DENPO) was implemented for LLM fine-tuning. The empirical demonstrations show that DENPO consistently outperforms existing similar RLHF and NLHF baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
RLHF, Nash Learning, Policy Optimization, Efficient Exploration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZGQ2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies online iterative NLHF. It first shows that INPO-style iterative updates can need Ω(e1/β)\Omega(e^{1/\beta}) samples on a three-response instance. It then proposes ENPO, which adds an SFT-style likelihood term on the learner's own samples to the INPO squared loss and selects an explorer policy through a ratio-type maximization. Under realizability and bounded log-ratio assumptions, ENPO obtains a O~(T)\tilde{O}(\sqrt{T}) best-iterate bound without e1/βe^{1/\beta} dependence. BENPO, a variant built on minimax oracles, obtains O(log⁡T)O(\log T) cumulative gap. A practical surrogate, DENPO, is evaluated on Llama-3-8B-Instruct against INPO and XPO.

Critical strengths

The question of whether iterative NLHF can achieve sample efficiency without minimax oracles is relevant.

The lower bound construction for INPO is explicit and checkable, and the mirror descent target and residual formulation in Eqs. (3), (10), and (16) are internally consistent.

Critical weaknesses
  1. Theorem B.10, which supports the regret bound and the uniform mixture output, relies on Assumption B.4. For a finite Π\Pi containing πref\pi_{ref}, B.4 requires the mirror descent orbit to reach π⋆\pi^{\star} exactly in finitely many steps. The relaxed Assumption B.7 with εη≤c/T\varepsilon_{\eta} \le c/T ties the policy class to the horizon and yields only a best-iterate bound, T2gap(πbest)\frac{T}{2}\mathrm{gap}(\pi^{best}), whose selection requires evaluating the duality gap and hence knowledge of P⋆P^{\star}.

  2. The explorer step in Eq. (5) is a joint maximization over Π×Π\Pi \times \Pi, and the paper does not show it to be easier to implement than the minimax oracles it replaces. Re-estimating P^t\hat{P}_t on historical atoms costs O(t3)O(t^3) queries per round, so the rate in terms of preference queries is far slower than T\sqrt{T}. The algorithm draws prompt-independent opponents, whereas the proof assumes prompt-conditional ones.

  3. The theory requires η=2β/(1+4β2)<1\eta = 2\beta/(1+4\beta^2) < 1, while the experiments use η\eta between 10210^2 and 10310^3 with no stated correspondence to the theoretical parameters. DENPO replaces the explorer step with an unanalyzed surrogate, runs for only T=3T=3, and compares only with INPO and XPO.

Questions for authors
  1. Does the regret bound for the uniform mixture output hold under Assumption B.7 alone?
  2. How is πbest\pi^{best} selected without access to P⋆P^{\star}?
  3. In Lemma B.9, the INPO loss vanishes for every π\pi when only bb is sampled. Does the lower bound then rely on choosing πt=πref\pi_t = \pi_{ref} among the minimizers?
  4. Can the main text explain why a forward KL term toward πt−1\pi_{t-1} complements the reverse KL proximal term in Eq. (2)?
  5. How do the experimental η\eta and β\beta correspond to the theoretical stepsize η=2β/(1+4β2)\eta = 2\beta/(1+4\beta^2)?
5 comments
AuthorsOpenReview ↗

Author Rebuttals

We thank the reviewer for their thoughtful comments and constructive suggestions. We appreciate them. We respond to each point to address the questions and concerns raised.

Theorem B.10, which supports the regret bound and the uniform mixture output, relies on Assumption B.4. For a finite Π\Pi containing πref\pi_{\mathrm{ref}}, B.4 requires the mirror descent orbit to reach π⋆\pi^\star exactly in finitely many steps. The relaxed Assumption B.7 with εη≤c/T\varepsilon_\eta\le c/T ties the policy class to the horizon and yields only a best-iterate bound, T2gap⁡(πbest)\frac{T}{2}\operatorname{gap}(\pi^{\mathrm{best}}), whose selection requires evaluating the duality gap and hence knowledge of P⋆P^\star.

Does the regret bound for the uniform mixture output hold under Assumption B.7 alone?

How is πbest\pi^{\mathrm{best}} selected without access to P⋆P^\star?

Thank you for these helpful comments and questions.

We agree that to choose a best iterate we have to store all checkpoints and evaluate them with various benchmarks in practice, which is not ideal. To avoid that, we revised the paper to adapt to another approximate realizability assumption: for each mirror descent target, there exists a policy in the policy class that is close enough to it. This approximation condition allows a finite policy class and yields a uniform-mixture regret bound. We presented this new assumption and the revised regret bound in Section 3 of the revised paper.

We also note that, with an exact realizability condition and a potentially infinite policy class, we can still derive a similar regret bound by using a standard covering-number argument. By doing so, we are able to make the policy class and realizability condition independent of the time horizon. We included these guarantees in Section D.2.

The explorer step in Eq. (5) is a joint maximization over Π×Π\Pi\times\Pi, and the paper does not show it to be easier to implement than the minimax oracles it replaces.

Thank you for pointing out this point. We appreciate it.

We agree that the computational advantage of the explorer should be explained more clearly. The distinction lies in the optimization structure. The max-max problem can still be seen as a maximization problem over a joint policy class, in which both policy variables share a common scalar objective function. In contrast, the max-min problem is a saddle-point problem, where the two decision variables are optimizing the opposite objectives.

This distinction particularly matters in the gradient-based training. It is known that opposing gradient updates in games can exhibit rotational or cycling dynamics even in bilinear zero-sum examples [Balduzzi et al. (2018)]. In contrast, gradient descent does not incur such issues. More generally, computing a Nash equilibrium in a general finite game is PPAD-complete [Daskalakis et al. (2009)].

  • David Balduzzi, Sebastien Racaniere, James Martens, Jakob Foerster, Karl Tuyls, and Thore Graepel. The mechanics of n-player differentiable games, 2018.
  • Constantinos Daskalakis, Paul W. Goldberg, and Christos H. Papadimitriou. The complexity of computing a Nash equilibrium, 2009.

We have added a paragraph in the revised paper to explain this distinction.

Re-estimating P^t\widehat P_t on historical atoms costs O(t3)O(t^3) queries per round, so the rate in terms of preference queries is far slower than T\sqrt T. The algorithm draws prompt-independent opponents, whereas the proof assumes prompt-conditional ones.

Thank you for these helpful comments. We appreciate it.

We improved the per-iteration query cost by switching to an alternative empirical loss function in the first oracle, which is the loss function used in IPO and INPO. By doing so, we reduce the query cost from O(t3)O(t^3) to O(t)O(t). This change significantly saves on the per-iteration preference query and sampling cost: in round t, we only draw 2t+22t + 2 new responses and query 2t2t preference labels.

We also updated Algorithm 1 to make it be consistent with the proof. The algorithm now explicitly draws each new pair's prompt x∼ρ\mathbf{x}\sim\rho and both responses conditionally on that shared prompt to construct the dataset D2tD_2^t. The dataset D1tD_1^t uses fresh independent prompts from ρ\rho. To compute the first loss function, we sample ct,s∼πt−1(⋅∣xs)c_{t,s} \sim \pi_{t-1}(\cdot\mid\mathbf{x}_s) at each stored pair's prompt. Thus, the sampling rule agrees with the prompt conditioning used in the proof.

AuthorsOpenReview ↗

Author Rebuttals

In Lemma B.9, the INPO loss vanishes for every π\pi when only bb is sampled. Does the lower bound then rely on choosing πt=πref\pi_t=\pi_{\mathrm{ref}} among the minimizers?

Thank you for this question.

Yes, we agree that it would make the argument more robust if the lower bound did not depend on a special tie-breaking rule. We have replaced the earlier example with a new instance satisfying our updated assumptions, in which every empirical update has a unique minimizer on the event establishing the lower bound. Thus, the revised proposition is independent of how ties are resolved.

The same instance works for every batch size, including a batch size of one. INPO still requires exponentially many preference queries in 1/β1/\beta even to achieve a constant duality gap. We have updated Proposition 3.2 and provided the corresponding proof in the paper.

Can the team explain why a forward KL term toward πt−1\pi_{t-1} complements the reverse KL proximal term in Eq. (2)?

Thank you for the question. This is a very interesting question.

The SFT term acts as a forward KL penalty toward the current policy, and it discourages the learner from dropping probability on responses covered by the current data, while the reverse KL proximal term controls movement away from the previous policy. The two terms therefore provide complementary control of the policy update.

This additional control is useful when learning an equilibrium, where we must compare the learner against opponents that may be poorly covered by the data. We think this is a very interesting observation and might have broader implications in RL and learning in games. We added an explanation for this complementary control interpretation in Appendix C.1 of the revised paper.

The theory requires η=2β/(1+4β2)<1\eta=2\beta/(1+4\beta^2)<1, while the experiments use η\eta between 10210^2 and 10310^3 with no stated correspondence to the theoretical parameters.

How do the experimental η\eta and β\beta correspond to the theoretical stepsize η=2β/(1+4β2)\eta=2\beta/(1+4\beta^2)?

Thank you for raising this good point.

The theoretical stepsizes are conservative sufficient choices for controlling accumulated optimization and statistical errors over a long training horizon. In practice, the LLM post-training typically is implemented in a much smaller time horizon, e.g., three iterations. We therefore use more aggressive stepsizes to obtain substantial improvements within this short horizon.

We comment that, even in the classic tabular experiment settings, it is common that the gradient-based algorithms are implemented with a much more aggressive stepsize instead of the theoretical conservative stepsize. In our paper, the polynomial preference-query guarantee is itself a significant theoretical contribution: to the best of our knowledge, it is the first such guarantee for online iterative NLHF. We regard extending this guarantee to larger stepsizes as an interesting open problem.

DENPO replaces the explorer step with an unanalyzed surrogate, runs for only T=3T=3, and compares only with INPO and XPO.

Thank you for these helpful comments.

We added theoretical support for the explorer surrogate in Appendix C.1.1 of the revised paper. Under a coverage condition, we connect population DPO improvement to a lower bound on the theoretical exploration objective. This provides a mathematical justification for the explorer surrogate used in DENPO. We think this justification is an interesting and meaningful addition to the paper, since most practical algorithms only employ heuristic surrogates without any theoretical justification.

We agree that the three-iteration horizon is a small number compared to the usual gradient-based algorithm. Even so, it is a setup that is commonly used in the LLM post-training, partly due to its high cost, and we adapt that setup for the comparisons between our methods and previous baselines.

As per our response to Reviewer CF4g, we are implementing additional experiments to compare DENPO with more baselines. We will follow up with additional results once we collect a full set of results.

Thank you again for your detailed review. We sincerely appreciate your comments and suggestions.

We would like to follow up on our rebuttal. We have substantially revised the paper to address the valuable points you raised. We have invested significant effort in this revision, and we would greatly appreciate it if you could take a look at our responses and let us know whether they have adequately addressed your concerns.

Reviewer_H1y2OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies exploration in iterative Nash preference optimization under general preferences. It proposes ENPO, which combines an SFT regularizer with an adversarial explorer to avoid exponential dependence on the inverse KL coefficient. BENPO gives a logarithmic regret bound under stronger oracle assumptions. A practical variant, DENPO, is evaluated on Llama-3-8B-Instruct and shows gains over INPO and XPO, particularly at larger stepsizes.

Critical strengths
  1. The paper addresses exploration under general preferences. The adversarial sampler targets weaknesses in the current policy, giving the exploration design a clear motivation.

  2. BENPO’s logarithmic cumulative duality-gap bound is a substantive theoretical result.

Critical weaknesses
  1. ENPO uses O(T⁴) preference queries over T rounds, so its round-based bound hides a steep query cost. The comparison with prior methods should use total queries to clarify the price of removing exponential dependence on 1/β.

  2. DENPO trails INPO on all four metrics at the default stepsize, including 52.03 versus 53.69 on AE2-mini. Without an ablation removing the explorer while retaining the SFT regularizer, it is unclear whether the gains at larger stepsizes come from exploration or better regularization.

Questions for authors

Do you have a selection rule based on observable feedback that preserves Theorem 3.3’s guarantee? What additional query budget would it require?

5 comments
AuthorsOpenReview ↗

Author Rebuttals

We thank the reviewer for their thoughtful comments and constructive suggestions. We appreciate them. We respond to each point to address the questions and concerns raised.

ENPO uses O(T4)O(T^4) preference queries over TT rounds, so its round-based bound hides a steep query cost. The comparison with prior methods should use total queries to clarify the price of removing exponential dependence on 1/β1/\beta.

Thank you for this helpful comment.

We agree that there was a steep query cost in the presented algorithm. We found that the per-iteration query cost can be reduced from O(t3)O(t^3) to O(t)O(t) by simply using an alternative empirical loss function in the first oracle. In particular, by employing a similar loss function that was used in IPO and INPO, we significantly save on the per-iteration preference query and sampling cost: in round t, we only draw 2t+22t + 2 new responses and query 2t2t preference labels.

The new algorithm uses O(ST2β−8ϵ−4)O(S^2_T \beta^{-8} \epsilon^{-4}) preference queries to compute an ϵ\epsilon-duality gap (ST2S^2_T is a coefficient that describes the coverage, which is a small constant in the instance used to show the lower bound of INPO), while in Section 3.1 we show that INPO requires Ω(e1/(4β))\Omega(e^{1/(4\beta)}) preference queries to even compute a constant duality gap. This is a significant theoretical gap: the prior methods require a number of preference queries that is exponentially dependent on 1/β1/\beta. In contrast, our algorithm admits a polynomial dependence. This is also the main theoretical gap we close in this paper.

We have revised our algorithm and updated Theorem 3.3 and the corresponding discussion to discuss this point.

DENPO trails INPO on all four metrics at the default stepsize, including 52.03 versus 53.69 on AE2-mini. Without an ablation removing the explorer while retaining the SFT regularizer, it is unclear whether the gains at larger stepsizes come from exploration or better regularization.

Thank you for this helpful comment and suggestion. We appreciate it.

We agree that an ablation experiment on the explorer will be helpful, and the contributions of regularization and exploration should be examined separately. We are now implementing the ablation experiment: removing the explorer while retaining the SFT regularizer. In a preliminary Llama-3-8B-Instruct experiment, in the largest stepsize setting, iteration 3 of ENPO without the explorer scores 56.8056.80 with AE2-mini, which is worse than the full ENPO's 58.3358.33 but still better than other existing baselines. This suggests that the explorer is useful in improving the performance, while even without the explorer, ENPO still shows better performance than other baselines.

We are implementing more additional experiments, and will follow up with a full set of experiment results in several days.

Do you have a selection rule based on observable feedback that preserves Theorem 3.3's guarantee? What additional query budget would it require?

Thank you for the question and suggestion. This is a very good point.

Yes, we can use an alternative adaptive hyperparameter selection rule, which does not need any information that is unknown in practice (e.g., the coverage coefficient). In Section 3.4 of the revised paper, we present such a selection rule, in which we use the optimal values in the second oracle from the previous iterations 1,…,t−11, \ldots, t - 1 to set up the parameters γt\gamma_t and αt\alpha_t in the current iteration tt. The resulting algorithm therefore can be implemented without knowing any coverage coefficients in advance, and can potentially obtain better choices of γt\gamma_t and αt\alpha_t than the conservative uniform choices.

Moreover, the adaptive guarantee preserves the rate up to constants and uses the same preference query budget. We added a new subsection in the paper to develop this useful extension, which can be interesting both theoretically and practically. We think this strengthens the paper's contribution significantly.

Thank you for the detailed response and the substantial theoretical revisions. I appreciate the reduction in preference-query complexity from O(T4)O(T^4) to O(T2)O(T^2), as well as the new adaptive hyperparameter selection guarantee in Section 3.4. These are meaningful improvements that address my first weakness and an important aspect of my question.

My main remaining concern is the empirical contribution of the adversarial explorer.

The preliminary ablation result is helpful. At the largest stepsize, DENPO without the explorer achieves 56.80 on AE2-mini, compared with 58.33 for full DENPO. This suggests that the SFT regularizer alone may account for a substantial part of the observed stability improvement, while the incremental contribution of the explorer remains unclear. It would also be helpful to clarify why DENPO underperforms INPO on all four metrics at the default stepsize.

Regarding the adaptive selection rule, Theorem 3.4 is a useful addition. I would also appreciate a clarification of the statement in Section 3.2 that ENPO can output "the iterate with the smallest duality gap." Unlike the uniform mixture, selecting such an iterate appears to require additional feedback or evaluation queries. Is there an observable selection rule that preserves the theoretical guarantee, and what would its query cost be?

Thank you very much for your response, and for recognizing the theoretical improvements.

We follow up with additional explanation as follows.

At the largest stepsize, DENPO without the explorer achieves 56.80 on AE2-mini, compared with 58.33 for full DENPO. This suggests that the SFT regularizer alone may account for a substantial part of the observed stability improvement, while the incremental contribution of the explorer remains unclear.

We have now evaluated the four combinations of SFT regularization and exploration on Llama-3-8B-Instruct, which provides a more comprehensive view of each component's contribution. We also added a more structured discussion on the roles of the SFT regularization and the explorer to provide more explanation and insights.

The full ablation results are:

SFT regularizerExplorerIteration 1Iteration 2Iteration 3
OnOn42.88 ± 3.5456.22 ± 3.5658.33 ± 1.00
OnOff42.55 ± 0.6149.44 ± 0.2955.57 ± 3.11
Off (α=0\alpha = 0)On42.68 ± 0.2757.53 ± 2.7055.57 ± 4.81
Off (α=0\alpha = 0)Off42.91 ± 1.6449.35 ± 1.5552.75 ± 1.59
(updated to include more seeds)

We agree that SFT regularization alone shows benefits as measured by the performance metrics. Beyond that, the updated full table also shows that the explorer contributes to the performance improvements. In particular, we can see that, with the explorer, the algorithm has already achieved good performance at iteration 2 compared with the variants without the explorer.

Our understanding is that the main role of SFT regularization is stabilizing learning, while the main role of the explorer is generating out-of-distribution samples---both components are new and useful in our algorithm. We can observe both effects in our ablation experiments: with SFT regularization, performance improves steadily over iterations, while with the explorer, performance improves more rapidly in the early iterations.

It would also be helpful to clarify why DENPO underperforms INPO on all four metrics at the default stepsize.

Yes, we agree that we should clarify and explain these observations.

One plausible explanation is the difference in the training comparisons: INPO selects the best and worst of eight responses, whereas DENPO uses only one comparison pair per prompt, labeled against four current-policy samples. When both algorithms use small stepsizes, INPO's updates remain stable, and its more selective pairs may provide a stronger immediate training signal. In contrast, when both algorithms use larger stepsizes, INPO's updates become unstable, while DENPO achieves better empirical performance even with fewer samples per prompt.

Overall, DENPO achieves the best performance with corresponding best stepsizes. We presented the results for the small-stepsize cases to illustrate additional observations. The empirical advantage supported by the experiments is DENPO's tolerance of larger updates.

We have added a more detailed explanation of this in the newly updated revision.

I would also appreciate a clarification of the statement in Section 3.2 that ENPO can output "the iterate with the smallest duality gap." Unlike the uniform mixture, selecting such an iterate appears to require additional feedback or evaluation queries. Is there an observable selection rule that preserves the theoretical guarantee, and what would its query cost be?

Thank you for pointing this out. The phrase “the iterate with the smallest duality gap” was left over from an earlier version. We have corrected the sentence in Section 3.2 to read: “After TT rounds, ENPO outputs the uniformly mixed iterates.” This is consistent with the output already specified in Algorithm 1 and the guarantee of Theorem 3.4. In particular, by Proposition B.1, we have gap⁡Π(πˉT)≤2Reg⁡TT\operatorname{gap}_\Pi(\bar\pi_T)\leq\frac{2\operatorname{Reg}_T}{T}.

We have also gone through the paper to fix similar inconsistencies.

Reviewer_CF4gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies exploration in iterative Nash preference optimization under general preferences. It identifies an exponential sample-complexity barrier for on-policy INPO and proposes ENPO, combining SFT-style regularization with adversarial exploration, alongside the stronger-oracle variant BENPO. A practical surrogate, DENPO, is evaluated on Llama-3-8B-Instruct against INPO and XPO, showing improvements on several alignment benchmarks.

Critical strengths
  1. The paper addresses an important exploration problem in learning from general, potentially non-transitive preferences.

  2. The combination of stabilizing regularization and adversarial exploration provides a principled algorithmic design. The theoretical analysis contains meaningful results.

Critical weaknesses
  1. My main concern is the limited empirical evaluation. All experiments use a single backbone, Llama-3-8B-Instruct, leaving generalization to newer model families (e.g., Qwen 3) untorched. The experimental comparison is also limited to INPO and XPO; relevant baselines, including iterative DPO, SPPO, and ONPO, are absent. Although the appendix provides budget comparisons, evaluating a broader set of methods under matched computational and preference-feedback budgets would strengthen the empirical evidence.

  2. The presentation could also be improved. In particular, DENPO’s concrete optimization objectives are deferred to the appendix, making the main text insufficiently self-contained. Including these objectives and a concise description of the training procedure would improve clarity.

Questions for authors

Please see the weaknesses part.

At the current stage, I consider the paper borderline for acceptance at a top-tier ML conference, with a slight lean toward acceptance. I remain open to revising my assessment based on the authors’ response and subsequent discussion.

4 comments
AuthorsOpenReview ↗

Author Rebuttals

We thank the reviewer for their thoughtful comments and constructive suggestions. We appreciate them. We respond to each point to address the questions and concerns raised.

My main concern is the limited empirical evaluation. All experiments use a single backbone, Llama-3-8B-Instruct, leaving generalization to newer model families (e.g., Qwen 3) untouched. The experimental comparison is also limited to INPO and XPO; relevant baselines, including iterative DPO, SPPO, and ONPO, are absent. Although the appendix provides budget comparisons, evaluating a broader set of methods under matched computational and preference-feedback budgets would strengthen the empirical evidence.

Thank you for the suggestions. We appreciate it.

We are implementing additional experiments to expand the evaluation to include the baselines you mentioned: iterative DPO, SPPO, and ONPO, and to include a second model family, Qwen3-8B. We will also incorporate the additional experiment results into the revised manuscript, as soon as we collect all additional results.

Here, we present some preliminary results: on Llama-3-8B-Instruct, we compare ENPO with more baselines with matched costs for prompts, samples, and preference oracles. Iteration-3 checkpoint AE2-mini scores are 53.1253.12 for iterative DPO, 47.4647.46 for SPPO, and 55.7455.74 for ONPO, which are worse than our DENPO performance. We notice that SPPO's official recipe achieves 61.2961.29, using a different oracle (PairRM) and sampling budget. We instead implement SPPO within our settings for comparison. We will follow up with a complete set of experiment results including those on Qwen3 in the next several days.

The presentation could also be improved. In particular, DENPO's concrete optimization objectives are deferred to the appendix, making the main text insufficiently self-contained. Including these objectives and a concise description of the training procedure would improve clarity.

Thank you for your helpful comments and the suggestions on the presentation.

We have made Section 5.1 self-contained by moving the learner and explorer objectives into the main text. We also describe the alternating training sequence at the beginning of Section 5.1 to provide an overview of the training procedure.

We thank the reviewer again for suggesting a broader empirical evaluation.

Following our initial response, we have completed experiments with iterative DPO, SPPO, and ONPO, and extended the evaluation to Qwen3-8B. In these experiments, we use the same setups as those in the existing experiments. On Llama-3-8B-Instruct, we obtain the following iteration-3 results. Since GPT-4-Turbo was retired from the OpenAI API on October 23, 2026, we do not include the corresponding evaluation results in the following tables.

Method or outputStepsizeAE2-mini LC (%)MT-BenchArena-Hard (%)
Base model—34.836.6436.40
DENPO learner1/η=9.375×10−41/\eta = 9.375 \times 10^{-4}58.337.0455.10
INPO1/η=9.375×10−41/\eta = 9.375 \times 10^{-4}30.245.3817.67
ONPO1/η=9.375×10−41/\eta = 9.375 \times 10^{-4}57.807.1345.60
SPPO1/η=1×10−31/\eta = 1 \times 10^{-3}47.467.1348.60
Iterative DPO—53.126.7654.50
XPO—47.066.9145.33

On Qwen3-8B, we obtain the following iteration-3 results.

Method or outputStepsizeAE2-mini LC (%)MT-BenchArena-Hard (%)
Base model—71.217.8690.0
DENPO learner1/η=9.375×10−41/\eta = 9.375 \times 10^{-4}78.878.3891.5
DENPO explorer1/η=9.375×10−41/\eta = 9.375 \times 10^{-4}82.908.5293.5
INPO1/η=9.375×10−41/\eta = 9.375 \times 10^{-4}78.448.1392.7
ONPO1/η=9.375×10−41/\eta = 9.375 \times 10^{-4}77.768.0092.1
SPPO1/η=1×10−31/\eta = 1 \times 10^{-3}78.128.1992.5
Iterative DPO—76.818.0292.5
XPO—74.898.1893.2

We find that, in the Qwen3-8B experiments, the DENPO explorer policy exhibits stronger performance than the learner policy. We therefore include both the DENPO learner and explorer policies in the table above. Since we use the same matched computational budget and the explorer policy is obtained as part of the third training iteration, using the explorer policy does not incur any additional training cost.

Overall, the DENPO learner and explorer policies outperform the baselines on most evaluated metrics, providing stronger empirical support for the empirical effectiveness of DENPO.

Thanks for the response. My main concerns have been addressed, and I maintain my weak acceptance recommendation.

Thank you!

Comments on the paper

1 comment

We sincerely thank all PCs, SACs, ACs, and reviewers for their time and efforts. Our paper has benefited significantly from the detailed, extremely useful, and invaluable feedback provided by the reviewers. We are deeply grateful for their help and support. We also appreciate all the reviewers' responses to our rebuttals and their recognition of the efforts we have made to address their concerns.

For your convenience, we summarize our main contributions, along with the major new theoretical and empirical results added during the rebuttal period. We have also addressed the other concerns and questions raised in the reviews in our individual rebuttal responses.

Main contributions

  • ENPO: Combine SFT regularization with adversarial exploration to obtain query complexity polynomial in inverse KL regularization, the first such guarantee in iterative Nash preference optimization without minimax oracles or preference-model training.
  • BENPO: Prove O(log⁡T)O(\log T) regret without exponential dependence on inverse KL regularization, using additional minimax and best-response oracles.
  • DENPO: Implement the explorer with a DPO surrogate and demonstrate its empirical benefits in LLM post-training.

Major theoretical new results

  • Significantly improved per-iteration query and sample cost, with a duality-gap guarantee for the uniformly mixed policy.
  • A theoretical guarantee for the DPO-type explorer surrogate under a coverage condition.
  • An adaptive parameter selection rule that preserves the regret bound without using the coverage coefficient as an input.

Major experimental new results

  • Additional baselines: iterative DPO, SPPO, and ONPO,
  • Qwen3-8B additional experiments,
  • Ablation experiments: understand the roles of SFT regularization and adversarial exploration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.