acceptodds
Under review as a conference paper at ICLR 2027

Controlling Sharpening in LLM Post-Training

Abstract

LLM post-training via reinforcement learning has shown consistent improvements in accuracy at the cost of reducing behavioral coverage—a phenomenon known as sharpening that can stall progress on difficult tasks. Repeated-sampling objectives such as and are known to mitigate sharpening, but the mechanism behind it remains poorly understood. Moreover, existing approaches for optimization use dedicated algorithms, and no general framework exists for adapting established policy optimization methods from to . We derive the optimal policies for KL-regularized and objectives in closed form. Our explicit solutions reveal that optimization naturally redistributes learning pressure based on prompt difficulty, directing it toward harder problems. Crucially, we show that standard objectives achieve these exact same optimal policies through prompt-adaptive regularization or reward transformation. Ultimately, this yields an optimizer-independent explanation for why objectives reduce sharpening on easy tasks while prioritizing success on harder ones. Building on these insights, we introduce Adaptive Pressure for Policy Optimization (KAPPO). Following a one-time reference calibration, KAPPO seamlessly adapts existing methods to via prompt-adaptive regularization or reward transformation, without otherwise altering their training procedures. Experiments with several RL methods on mathematical reasoning and code generation demonstrate that KAPPO provides direct control over sharpening. Crucially, we demonstrate that pressure can be fully realized with single-completion training, matching the performance of multi-completion optimizers at a fraction of the cost.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, large language model, post training, pass at k, max at k, reasoning, exploration, sharpening, closed-form solution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oCgkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper derives optimal policies for KL-regularized pass@k and max@k objectives and introduces KAPPO, which adapts existing optimizers using reference-calibrated coefficients or reward transformations. For binary rewards, it computes

\qquad \beta_{\mathrm{eff}}(x)=\frac{\beta}{a_k(x)},$$ where $p^*(x)$ is the ideal regularized optimal success probability inferred from the reference model. Standard DPO uses a global coefficient in its preference loss, $$\mathcal L_{\mathrm{DPO}} =-\mathbb E[\log\sigma(\beta\Delta_\theta)], \quad \Delta_\theta= \log\frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} -\log\frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}.$$ KAPPO replaces $\beta$ with the prompt-dependent $\beta_{\mathrm{eff}}(x)$. Standard GRPO uses group-normalized advantages $$A_i=\frac{R_i-\bar R}{s_R}.$$ Its binary KAPPO adaptation instead uses $A_i^{(k)}=a_k(x)A_i$, applying the multiplier after normalization to prevent cancellation. For graded rewards, KAPPO applies a nonlinear reference win-rate transformation before group normalization. These modifications target multi-attempt performance, although exact optimum matching holds under the theoretical assumptions rather than automatically for practical DPO or GRPO training. Experiments on mathematical reasoning and code generation show that the single-completion A⋆PO implementation achieves performance close to multi-completion PKPO with fewer online rollouts.
Critical strengths
  1. The optimal-policy derivations provide a useful explanation of how pass@k optimization shifts pressure toward harder prompts.
  2. Single-completion A⋆PO achieves performance close to PKPO, with differences below 0.5 percentage points at matching validation budgets.
  3. Experiments cover multiple optimizers, two model families for math, and code generation.
Critical weaknesses
  1. The connection between the theory and the practical implementations needs clarification. Matching the ideal optimal policy does not guarantee the same behavior when training a neural network. In particular, the deterministic correct/incorrect pairs used for DPO do not directly satisfy the assumptions behind its reward-optimum correspondence [1].
  2. The method generates 30 reference-model answers per training question and uses them to fix its training coefficient. For difficult questions, observing zero versus one correct answer can substantially change that coefficient, making validation with larger or independently sampled banks important. This also introduce additional inference cost before training.
  3. For DPO and GRPO, which already sample multiple responses per question during training, the benefit of fixed reference calibration over using observed training success rates remains unclear. A comparison with online coefficient updates would strengthen the justification.
  4. The evaluation is somewhat narrow. The math experiments train and validate on MATH, then test only on AIME 2026; the code experiments test on the hard subset of LiveCodeBench v6. These results are encouraging, but broader testing would help establish how well the method generalizes. Error bars, clearer evaluation settings, and training costs that include reference calibration would also make the results easier to assess.
  5. The code experiments use test-case pass rates with ties, while the win-rate derivation assumes an atomless reward distribution. The authors acknowledge this mismatch but still use (Z=1). The effect on the A⋆PO results remains unclear.

References [1] Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. Rafael Rafailov et al., 2023.

Questions for authors
  1. The win-rate derivation assumes an atomless reward distribution, but test-case pass rates contain ties. How frequent are ties in your code experiments, and does correcting the partition term rather than assuming (Z=1) materially change the A⋆PO results?
  2. For DPO and GRPO, which already sample multiple responses per question during training, what is the advantage of fixed reference calibration over updating the coefficients using observed training success rates?
5 comments
AuthorsOpenReview ↗

Response by Authors [1/2]

We thank reviewer oCgk for the thoughtful and constructive feedback, for recognizing the theoretical insights of our optimal-policy derivations, and for highlighting the strong empirical performance of our single-completion approach and the broad experimental scope across optimizers, model families, and task domains.

We have carefully considered their comments and adjusted the paper to resolve all stated weaknesses and questions (resulting in 3 new ablations, 5 new sections in the appendix and additional clarifying paragraphs in the main text). All changes are highlighted in blue in the updated paper PDF.

In the following, we summarize how we addressed the reviewer's questions and weaknesses:


W1

We agree that matching an ideal optimum does not guarantee convergence to it under non-convex neural network training dynamics. Instead, we base our approach on the well-established empirical evidence that fitting the optimal policy often provides a good training signal [5,6,7], and we dedicate a lot of effort to verifying the proposed methods empirically in a broad experimental scope.

Regarding the connection between @k-DPO and the KL-regularized pass@k optimum, we have added a dedicated discussion in Appendix I. We highly encourage the reviewer to take a look at it.

Here is the TLDR:

  • If preference annotations follow the Bradley–Terry model, the derived @k-DPO loss (our KAPPO) is uniquely minimized by the pass@k closed-form optimal policy.
  • However, a widely used practical heuristic [1,2,3] is to assign correctness-based preferences deterministically instead of following Bradley-Terry: here, the chosen completion is given reward 1 and the rejected completion reward 0. The strong empirical performance of this well-established heuristic motivated us to evaluate KAPPO with this preference construction.
  • Under deterministic preferences, ordinary DPO, and consequently our @k-DPO adaptation, loses the guarantee of recovering a theoretically optimal policy [4]. This loss of guarantees therefore originates in the preference construction, independently of our KAPPO adaptation.
  • With deterministic labels, we can nevertheless prove that the adaptive KAPPO coefficient controls the policy-margin scale during finite training, providing a concrete connection to the pressure redistribution prescribed by our theory.

W2

Thank you for pushing back on this. We address the reviewer's concern related to the noise in the estimated coefficients for difficult questions (i.e. having low success probability) by considering the end-to-end algorithm performance when using less than 30 reference completions per prompt (we believe that it is easier to show the sensitivity this way than using a larger reference bank, as initially proposed by the reviewer). Appendix N demonstrates that in our setting, performance remains unchanged when reducing the reference bank from 30 to 16 or even 8 completions. With only 4 reference completions, performance decreases across the curve but remains substantially above the reference policy. We would also like to point the reviewer to Appendix M, where we theoretically analyze the sensitivity to finite reference calibration.

Additionally, we want to emphasize that reference calibration is computationally inexpensive (see Appenix P for more details). E.g., if we consider @k-A*PO runs on 4 B200 GPUs, we obtain the following estimate: reference calibration (with a reference bank of size 30) requires only approximately 8% of the total GPU-hours for a run, and this amount can be further amortized over all runs for the same model and dataset. More detailed results are provided in the table below

MethodCalibrationOnline generation + gradingTraining updates¹Overall²
@4\mathrm{@}4-A⋆A^{\star}PO3.00 h13.79 h5.97 h35.31 h
PKPO (K=4)0.00 h16.50 h15.86 h45.80 h

¹ For A⋆A^{\star}PO, training-update time includes reference-model log-probability computation and gradient updates.

² Overall time includes calibration, online generation and grading, training updates, validation, checkpointing, initialization, synchronization, and other overhead. Reducing the validation frequency would significantly cut down on this cost.

AuthorsOpenReview ↗

Response by Authors [2/2]

Q2 & W3

Thank you for suggesting this idea! We have added a section in Appendix O that directly investigate this alternative. For @2-DPO, we reuse the eight current-policy completions already generated for preference-pair construction to recompute the adaptive coefficient introduced by KAPPO (i.e., we estimate pref(x)p_{ref}(x) via p(x)p(x) before solving the fixed-point equation for p∗(x)p^*(x)). DPO’s reference-policy weights remain fixed, and no additional rollouts are required. The online variant closely tracks fixed-calibration @2-DPO and reproduces its qualitative reduction in sharpening. Thus, separate reference-bank generation can indeed be skipped in settings where several online completions are already required for the base @1 method (such as DPO in our setting and GRPO). This approach offers a valuable alternative for efficient KAPPO implementation and we thank the reviewer for bringing this up.

W4

We agree with the reviewer that the paper would benefit from even broader evaluation, but we would like to point out that our study already spans multiple optimizers, two model families, and both mathematical reasoning and code generation. At this scale, more than 100,000 GPU-hours had to be devoted to training, thorough hyper-parameter sweeps (for all baselines, see also Appendix J), and evaluation (Appendix K). We would also like to point the reviewer to the Sections 6.2 and 6.3, where we describe our efforts in ensuring a distribution shift in our final evals, thereby directly testing the generalization properties of KAPPO.

Beyond our initial submission, we believe that the calibration-size (Appendix N) and online-calibration (Appendix O) ablations provide additional robustness evidence for our method.

Concerning computational cost, Appendix P now reports calibration, online generation and grading, training updates, and the overall cost separately, see also our response to W2.

Concerning clearer evaluation settings, we took the reviewer's feedback on our presentation to heart and have added a dedicated and detailed section in Appendix J.

Q1 & W5

Assuming Z=1Z=1 does not significantly affect the performance of the algorithms used in our experiments. First, our GRPO adaptation does not use the partition term at all, and second, even though the partition term is present in the A*PO adaptation, the resulting gradients are still unbiased in the on-policy regime that we use (the gradients of A*PO have the form of a Policy Gradient with log⁡Z\log Z representing the baseline term). As a result, changing ZZ only affects the variance of the Monte Carlo gradient estimates. Since an atomless distribution would dictate Z=1Z=1, we find this a reasonable baseline to adopt, even in the presence of ties.

Here are some statistics on ties in our coding experiments: considering a random pair of completions sampled for the same prompt, the probability that their rewards are equal is 39%. However, this percentage is highly dominated by very easy or very hard tasks where the rewards are typically 1 or 0, respectively. Conditioned on the reward being between 0 and 1, the probability of ties decreases to just 15%. Since some tasks contain invalid tests (leading to a reward in (0,1)(0,1) even for the gold solution), further conditioning on the tests validity, the probability of ties drops to approximately 9%.


We thank the reviewer for helping us clarify the theoretical scope and practical trade-offs of KAPPO. We hope that the additional analysis, experiments, and other revisions to the submission address their concerns on presentation and would be grateful if they could reconsider their overall rating in light of our clarifications and the revised manuscript. We welcome any remaining questions.


References

[1] Jack Lanchantin et al. “Bridging Offline and Online Reinforcement Learning for LLMs.” arXiv:2506.21495, 2025.

[2] Team Olmo et al. “Olmo 3.” arXiv:2512.13961, 2025.

[3] Project Apertus. “Apertus 1.5: From Open Text Models to Open Multimodal Reasoners.” arXiv:2610.15542, 2026.

[4] Azar, Mohammad Gheshlaghi, et al. "A general theoretical paradigm to understand learning from human preferences." International conference on artificial intelligence and statistics. PMLR, 2024.

[5] Rafael Rafailov, et al. "Direct preference optimization: Your language model is secretly a reward model." In Advances in Neural Information Processing Systems, volume 36, pp. 53728–53741. Curran Associates, Inc., 2023.

[6] Kiante Brantley, et al. "Accelerating RL for LLM reasoning with optimal advantage regression." In Advances in Neural Information Processing Systems, volume 38, pp. 151492–151531. Curran Associates, Inc., 2025.

[7] Simon Matrenok, Skander Moalla, and Caglar Gulcehre. "Quantile reward policy optimization: Alignment with pointwise regression and exact partition functions." Advances in Neural Information Processing Systems 38 (2025)

AuthorsOpenReview ↗

Gentle Reminder: Rebuttal & Revised PDF Before Discussion Closes

Dear Reviewer oCgk,

As the discussion period wraps up, we wanted to briefly bring our rebuttal and revised manuscript to your attention. Following your helpful suggestions, we added an experiment comparing fixed reference calibration against online coefficient updates computed from current-policy rollouts (Appendix O), an end-to-end ablation with fewer samples in the reference bank (Appendix N), and a detailed GPU-hour breakdown separating calibration, generation, and training costs against PKPO (Appendix P). We also clarified the theoretical connection between the @k-DPO loss and the KL-regularized pass@k objective (Appendix I), quantified tie frequencies in our code experiments while explaining why setting Z=1Z=1 preserves unbiased on-policy gradients (Appendix F), and expanded our experimental setup details (Appendix J).

We would be very grateful if you could take a quick look at our response and revised PDF. We are happy to answer any remaining questions before the deadline!

Reviewer_QAt9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how pass@k and max@k objectives affect sharpening in LLM post-training. It characterizes their KL-regularized optimal policies and shows that, for binary rewards, the pass@k optimum can also be obtained through ordinary single-completion reward optimization with prompt-dependent reward scaling or KL regularization. Based on this connection, the authors propose KAPPO and instantiate it with DPO, A⋆PO, and GRPO. Experiments on mathematical reasoning and code generation show improved performance at larger evaluation sampling budgets. In particular, single-completion @k-A⋆PO achieves results close to the multi-completion PKPO baseline.

Critical strengths
  1. The paper provides a useful connection between repeated-sampling objectives and prompt-dependent optimization pressure. The binary-reward derivation is particularly clear: reference success probabilities determine a scalar equation, whose solution supplies the reward multiplier or regularization coefficient. The single-completion implementation is practically interesting. The comparison with PKPO supports the claim that online groups of k completions are not always necessary to obtain the benefits associated with pass@k training.

  2. The experiments cover two model families, two task domains, and several optimizers. The consistent shift toward better performance at larger sampling budgets supports the proposed interpretation.

Critical weaknesses
  1. The reliability of reference calibration at the actual training hyperparameters remains insufficiently established. Appendix H (p. 30) specifies M=30 and training β values mostly between 10⁻⁴ and 5×10⁻³, whereas the finite-bank analysis in Appendix K uses β=0.1. The resulting behavior can differ considerably. Robustness to the particular reference bank therefore matters, but no end-to-end bank-size or bank-resampling ablation is reported.

  2. The continuous-reward experiments provide a less direct validation of the theory. They optimize the reference-quantile surrogate introduced in Section 4.2 rather than raw-reward max@k. Moreover, Appendix F acknowledges that the explicit solution assumes an atomless reward distribution and that Z=1 need not hold for discrete rewards or empirical CDF estimates. Nevertheless, the implementation retains Z=1, as described in Appendix J.1. The effect of this approximation is not quantified. It is also unclear whether the k=1 baselines in Figure 4 use raw rewards or reference-quantile rewards, which are generally different objectives.

Questions for authors

See weaknesses

4 comments
AuthorsOpenReview ↗

Response by Authors

We thank reviewer QAt9 for the thoughtful and constructive feedback, for recognizing the theoretical insights and practical relevance of our approach, and for highlighting the broad experimental scope.

We have carefully considered their comments and adjusted the paper to resolve all stated weaknesses and questions (resulting in 3 new ablations, 5 new sections in the appendix and additional clarifying paragraphs in the main text). All changes are highlighted in blue in the updated paper PDF.

In the following, we summarize how we addressed the reviewer's questions and weaknesses:


W1

We thank the reviewer for pushing back on this. We have revised the Appendix K with sensitivity to finite reference calibration analysis (in the updated PDF it is Appendix M). Now, the numerical analysis is done for β=5×10−4\beta=5 \times 10^{-4}, a coefficient shared by the reported hyperparameter grids for binary and continuous-reward adaptations. As a result, our revised analysis now considers a much closer setup to the one we actually use in our empirical part.

Moreover, we have added an ablation on the required reference calibration size in Appendix N, directly addressing the reviewer's requests on testing the end-to-end algorithm sensitivity to the reference bank size. The ablation demonstrates that in our setting performance remains unchanged when reducing the reference bank from 30 to 16 or even 8 completions. With only 4 reference completions, performance decreases across the curve but remains substantially above the reference policy.

W2

Our experiments with win-rate rewards in the continuous case directly test the theory developed in Section 4.2 for the win-rate rewards, without claiming to test the theory developed for the raw rewards (as Appendix D details, the raw-rewards procedure is more prone to numerical approximation errors than the win-rate reward procedure, motivating our empirical focus). Therefore, considering this aspect, we respectfully disagree that our practical setting provides a less direct validation of the theory. In this light, we would also like to point the reviewer to the newly added Appendix G, which compares raw-reward max@k with win-rate max@k. Generally, neither win-rate nor raw-reward optimization is universally preferred, and neither is just a surrogate for the other. It all depends on the desired target behavior, see also the discussion in [1,2], which studies policy optimization with quantile rewards in the @1 setting and discusses their benefits and limitations.

We thank the reviewer for providing us the opportunity to clarify this point, and indeed, the wording in our initial submission was too defensive. Assuming Z=1Z=1 does not significantly affect the performance of the algorithms used in our experiments. First, our GRPO adaptation does not use the partition term at all, and second, even though the partition term is present in the A*PO adaptation, the resulting gradients are still unbiased in the on-policy regime that we use (the gradients of A*PO have the form of a Policy Gradient with log⁡Z\log Z representing the baseline term). As a result, changing ZZ only affects the variance of the Monte Carlo gradient estimates. Since an atomless distribution would dictate Z=1Z=1, we find this a reasonable baseline to adopt, even in the presence of ties. We have adjusted the submission accordingly.

Regarding the k=1 baselines in Figure 4, we concede that our presentation was ambiguous and thank the reviewer for bringing this to our attention. For A⋆A^{\star}PO, all configurations, including the k=1k=1 baseline, use reference-win-rate rewards. That helps isolating the effect of parameter kk. For GRPO, in contrast, the ordinary baseline uses raw test-case pass rates, which helps to establish a solid baseline for both panels. That distinction is reflected in the legend, where GRPO doesn't have a prefix "@k". However, we totally agree that it creates some confusion, therefore we have added a clarification to the main text.


We thank the reviewer again for identifying these points. The revised sensitivity analysis and additional ablations significantly strengthen the empirical evidence, while the expanded discussion makes the scope of the continuous-reward results more explicit. We hope these additions and revisions address the reviewer's concerns and would be grateful if they could reconsider their soundness assessment and overall rating in light of the revised manuscript. We welcome any remaining questions.

References

[1] Simon Matrenok, Skander Moalla, and Caglar Gulcehre. "Quantile reward policy optimization: Alignment with pointwise regression and exact partition functions." Advances in Neural Information Processing Systems 38 (2025)

[2] Ananth Balashankar, et al, “InfAlign: Inference-aware language model alignment.” In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research. PMLR, 2025

AuthorsOpenReview ↗

Gentle Reminder: Rebuttal & Revised PDF Before Discussion Closes

Dear Reviewer QAt9,

As the discussion period wraps up, we wanted to briefly bring our rebuttal and revised manuscript to your attention. Following your helpful suggestions, we added an end-to-end ablation on reference bank size (Appendix N) and updated our finite-bank sensitivity analysis to use a β\beta matching our actual training hyperparameter grid (Appendix M). We also added a characterization of the win-rate-based objective relative to raw-reward \max@k\max@k (Appendix G), clarified why setting Z=1Z=1 leaves on-policy gradients unbiased in the presence of discrete rewards and ties (Appendix F), and explicitly clarified the reward definitions for the k=1k=1 baselines in Figure 4 in the main text.

We would be very grateful if you could take a quick look at our response and revised PDF. We are happy to answer any remaining questions before the deadline!

Thank you for your response. I have no more concerns.

Dear reviewer QAt9

We are glad that our additional results and clarifications have addressed your concerns. Given that there are no other questions or concerns raised, could you consider revising your weak assessment in your overall score as well as your soundness score?

Once again, thank you for the time and effort spent on this work, which improved the paper a lot.

Reviewer_xixrOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies KL-regularized pass@k and max@k optimization. For binary rewards, it characterizes the optimal policy through a prompt-dependent effective KL coefficient. For continuous rewards, it derives an optimality condition and an explicit construction for win-rate-based rewards. Building on these results, the authors propose KAPPO, which uses a one-time reference-policy calibration to adapt A$^\star$PO, DPO, and GRPO. Experiments on mathematical reasoning and code generation show improved performance at larger evaluation budgets. In the mathematical reasoning experiments, @k-A$^\star$PO approaches PKPO’s performance while using one on-policy completion per prompt during training, after calibration.

Critical strengths
  1. A useful characterization of the KL-regularized pass@k optimum. Under the stated assumptions, the binary-reward analysis makes the relationship between prompt difficulty and optimization pressure precise. The resulting prompt-dependent coefficient offers a concrete way to adapt existing optimizers.

  2. A strong empirical comparison for the single-completion method. On MATH Level 5, @k-A$^\star$PO is within 0.5 percentage points of PKPO on the metric matching the training value of k, across the reported model and k settings. This is meaningful evidence provided the reference calibration is available.

Critical weaknesses
  1. Insufficient connection between theory and implementation. Substituting βeff\beta_{\mathrm{eff}} into the DPO loss does not directly guarantee that its optimum coincides with that of the KL-regularized pass@k objective. GRPO does not include KL regularization, while the code experiments optimize a win-rate surrogate with discrete rewards and possible ties.

  2. Unclear necessity of precise calibration and practical efficiency. The paper lacks a simple difficulty-weighted baseline, an ablation on the number of calibration samples, and measures of result variability. While the paper demonstrates a reduction in online generation counts, it does not separately report the end-to-end GPU time for @k-A$^\star$PO and PKPO. Additionally, the main text states that DPO uses "one pair" of completions per problem, whereas Appendix H specifies that eight completions are generated before selecting a pair. The number of completions used in the loss should be clearly distinguished from the number actually generated.

Questions for authors

I am happy to raise my score if you can address my question below:

  1. Under the actual preference-pair construction procedure, what provable relationship exists between the @k-DPO loss and the KL-regularized pass@k objective?

  2. How much improvement does precise calibration provide over simple reference-success-rate weighting, and how sensitive are the results to the number of calibration samples?

  3. Could the authors separately report the GPU time for calibration, generation, training, and the overall process, as well as the number of optimization passes over each batch of training data (i.e., data reuse frequency) for @k-APO and PKPO, and clarify the cost of generating eight completions per problem in DPO?

  4. How substantially does the win-rate surrogate objective used in the code experiments differ from the original-reward max@k objective?

7 comments
AuthorsOpenReview ↗

Response by Authors [1/3]

We thank reviewer xixr for the insightful feedback, for recognizing the utility of our closed-form pass@k expression, and for highlighting the strong empirical performance of the single-completion KAPPO adaptation of A*PO compared to PKPO.

We have carefully considered their comments and adjusted the paper to resolve all stated weaknesses and questions (resulting in 3 new ablations, 5 new sections in the appendix and additional clarifying paragraphs in the main text). All changes are highlighted in blue in the updated paper PDF.

In the following, we summarize how we addressed the reviewer's questions and weaknesses:


Q1 & W1

We have added a separate section in Appendix I that presents a detailed explanation of the connection between the @k-DPO loss and the KL-regularized pass@k objective. We highly encourage the reviewer to take a look at it.

Here is the TLDR:

  • If preference annotations follow the Bradley–Terry model, the derived @k-DPO loss (our KAPPO) is uniquely minimized by the pass@k closed-form optimal policy.
  • However, a widely used practical heuristic [1,2,3] is to assign correctness-based preferences deterministically instead of following Bradley-Terry: here, the chosen completion is given reward 1 and the rejected completion reward 0. The strong empirical performance of this well-established heuristic motivated us to evaluate KAPPO with this preference construction.
  • Under deterministic preferences, ordinary DPO, and consequently our @k-DPO adaptation, loses the guarantee of recovering a theoretically optimal policy [4]. This loss of guarantees therefore originates in the preference construction, independently of our KAPPO adaptation.
  • With deterministic labels, we can nevertheless prove that the adaptive KAPPO coefficient controls the policy-margin scale during finite training, providing a concrete connection to the pressure redistribution prescribed by our theory.

Below we also comment on the raised concern regarding KL term omission in GRPO:

GRPO was introduced as a KL-regularized policy optimization algorithm in [5], yet its version without KL regularization is also widely used [6]. Because we already demonstrate the effectiveness of KAPPO on two algorithms utilizing a KL-regularized objective (DPO and A*PO), a practitioner may want to know if our method breaks in case KL-regularization is removed. To address this concern, we deliberately picked the GRPO experiment to test this (and we clearly position that as an empirical extension of our theoretical result), demonstrating that KAPPO remains effective without explicit KL-regularization. This extension can nevertheless be motivated by the connection between KL regularization and finite-step optimization. Our theory first identifies a transformed reward, whose KL-regularized optimum corresponds to win-rate max@k. Limiting optimization of that same transformed reward can then play a similar regularizing role: under exact entropic mirror ascent from the reference policy, TT steps with learning rate η\eta recover the KL-regularized optimum with coefficient β=1/(ηT)\beta=1/(\eta T) [7]. Empirically, it has also been observed that KL penalty and early stopping reach the same gold-reward–KL frontier in RLHF [8]. Thus, a reward transformation derived with KL regularization remains theoretically motivated when the explicit penalty is replaced by a finite training horizon.

Q2 & W2

Great question! We have added the corresponding ablations in Appendix Q, N and O.

Here is the TLDR:

  • The simpler reward weighting by the failure probability (1−pref)(1-p_{ref}) instead of the derived k(1−p∗(x))k−1k(1-p^*(x))^{k-1} indeed produces a qualitative reduction in sharpening by overtaking the ordinary A*PO at larger evaluation budgets (Appendix Q). However, our theoretically derived weighting achieves higher performance than this simpler reward weighting heuristic across all evaluated budgets, demonstrating the utility of using theoretically sound methods.
  • Another ablation (Appendix N) demonstrates that in our setting performance remains unchanged when reducing the reference bank from 30 to 16 or even 8 completions. With only 4 reference completions, performance decreases across the curve but remains substantially above the reference policy.
  • Finally, in Appendix O we demonstrate that for optimizers where multiple online rollouts are sampled, such as DPO in our setting, we can use the online rollouts of the current policy as an approximation for the reference bank, eliminating the overhead of prior reference calibration. Indeed, using online rollouts to estimate p∗p^* via our fixed point equation results in equally strong performance for @k-DPO.
AuthorsOpenReview ↗

Response by Authors [2/3]

Q3 & W2

Thank you for pushing back on this. We have now added detailed measurements to Appendix P.

Here is the TLDR:

  • As we state in the main text, both DPO and our proposed @k-DPO require a single pair of completions per prompt. In our practical implementation we follow a common approach [1] and select a preference pair from a larger candidate group (8 completions per prompt). The only reason to use more than 2 completions per prompt is to increase the probability of obtaining an informative pair with different rewards. Since we use this approach for both the baseline DPO and the proposed @k-DPO, this implementation aspect does not affect our comparison.
  • Across all methods, each freshly generated rollout batch is used for exactly one optimizer update, with no additional optimization epochs or reuse in subsequent iterations, and all methods (within the same domain) use the same number of optimization steps. As a result, PKPO is given substantially more data to train on (4x more for K=4), yet @k-A*PO achieves equivalent performance.
  • On 4 B200 GPUs, compared to @4-A*PO, PKPO at K=4 incurs 1.6x the total cost (in GPU-hours) for online generation, grading, and training updates. Reference calibration requires approximately 8% of the total GPU-hours for a run, and this amount can be further amortized over all runs for the same model and dataset. If we further include validation, checkpointing, initialization, and synchronization, PKPO incurs 1.3x the cost for @4-A*PO. However, the total cost is highly diluted in our setup by those shared stages, e.g. by the expensive validation. More detailed results are provided in the table below
MethodCalibrationOnline generation + gradingTraining updates¹Overall²
@4\mathrm{@}4-A⋆A^{\star}PO3.00 h13.79 h5.97 h35.31 h
PKPO (K=4)0.00 h16.50 h15.86 h45.80 h

¹ For A⋆A^{\star}PO, training-update time includes reference-model log-probability computation and gradient updates.

² Overall time includes calibration, online generation and grading, training updates, validation, checkpointing, initialization, synchronization, and other overhead.

Finally, we would like to point out that we initially emphasized the gain over PKPO in terms of rollout multiplicity, since translation into GPU time largely depends on the specific setup deployed. E.g., @4-A*PO compared to PKPO (k=4) shows more than 2x gain in GPU-hours for the training step in our setup (as shown above). As a result, a setup that has more steps per generated batch or has larger data reuse (which is very typical in practice), and that doesn't dilute the gain by expensive evaluation, would result in much larger overall speedup compared to PKPO.

Q4 & W1

Great question! We have added this analysis in Appendix G, comparing the single-completion reward transformations derived for both raw-reward max@k and win-rate max@k. In our illustration, the difference is most pronounced on hard tasks: win rates emphasize improvements within the reference reward distribution but compress further gains above most reference outcomes, which raw rewards continue to distinguish. This changes the optimization emphasis, with benefits depending on the setting and required target behavior. We also refer the reviewer to [9, 10], which studies policy optimization with quantile rewards in the @1 setting and discusses their benefits and limitations.

AuthorsOpenReview ↗

Response by Authors [3/3]

We again thank the reviewer for their detailed feedback, which has pushed us to substantially improve our paper. We hope that, given our additional clarifications and experimental results (both in the rebuttal and in the revised manuscript), the reviewer agrees that their concerns on the presentation and soundness of our work have been addressed, and that we have convinced the reviewer to recommend acceptance. We welcome any remaining questions.


References

[1] Jack Lanchantin, et al. "Bridging offline and online reinforcement learning for LLMs." arXiv preprint arXiv:2506.21495, 2025

[2] Team Olmo, et al. " Olmo 3." arXiv preprint arXiv:2512.13961, 2025

[3] Project Apertus. "Apertus 1.5: From open text models to open multimodal reasoners." arXiv preprint arXiv:2610.15542, 2026

[4] Azar, Mohammad Gheshlaghi, et al. "A general theoretical paradigm to understand learning from human preferences." International conference on artificial intelligence and statistics. PMLR, 2024.

[5] Shao, Zhihong, et al. "Deepseekmath: Pushing the limits of mathematical reasoning in open language models." arXiv preprint arXiv:2402.03300 (2024).

[6] Yu, Qiying, et al. "DAPO: An open-source LLM reinforcement learning system at scale." Advances in Neural Information Processing Systems 38 (2025)

[7] Suggala, Arun, Adarsh Prasad, and Pradeep K. Ravikumar. "Connecting optimization and regularization paths." Advances in Neural Information Processing Systems 31 (2018).

[8] Gao, Leo, John Schulman, and Jacob Hilton. "Scaling laws for reward model overoptimization." International conference on machine learning. PMLR, 2023.

[9] Simon Matrenok, Skander Moalla, and Caglar Gulcehre. "Quantile reward policy optimization: Alignment with pointwise regression and exact partition functions." Advances in Neural Information Processing Systems 38 (2025)

[10] Ananth Balashankar, et al, “InfAlign: Inference-aware language model alignment.” In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 2646–2672. PMLR, 2025

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common response to the reviewers

We thank all reviewers for their insightful feedback. Their comments have helped us substantially improve our paper. To address all concerns, we have added considerable experimental evidence and clarified the connections between our theoretical results and practical implementations. All revisions are highlighted in blue in the updated manuscript.


To summarize, the main additions are:

  • Reference-bank-size ablation (Appendix N): tests sensitivity to calibration accuracy, showing no observed performance degradation when reducing the bank from 30 to 16 or 8 completions in the tested setting.
  • Online-calibration ablation (Appendix O): shows that @2-DPO can reuse existing training rollouts to estimate its adaptive coefficients, achieving comparable performance while eliminating the (slight) overhead of reference calibration.
  • Heuristic-weighting ablation (Appendix Q): tests whether a simpler difficulty-based weighting reproduces the benefits of KAPPO. It indeed demonstrates reduced sharpening, confirming our central intuition of adaptive optimization pressure w.r.t. task complexity. However, our derived weighting achieves higher performance across all evaluated budgets, highlighting the value of our developed theory.
  • DPO theory–implementation connection (Appendix I): provides a proposition and proof establishing exact optimal-policy recovery under a specified preference construction, and explains the consequences of the deterministic labels used in practice.
  • Raw-reward versus win-rate analysis (Appendix G): illustrates how the win-rate rewards change the value assigned to reward improvements compared to raw rewards, and discusses the resulting practical trade-offs.
  • Detailed computational accounting (Appendix P): separates calibration, generation and grading, training updates, and overall GPU costs, including data reuse and calibration amortization.
  • Clearer evaluation settings (Appendix J): specifies sampling settings and metric estimation to make the reported results easier to interpret and reproduce.

We hope that the additional experiments and clarifications address all raised concerns and strengthen the case for the paper’s contribution. We welcome any additional questions or discussions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.