- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper derives optimal policies for KL-regularized pass@k and max@k objectives and introduces KAPPO, which adapts existing optimizers using reference-calibrated coefficients or reward transformations. For binary rewards, it computes
\qquad \beta_{\mathrm{eff}}(x)=\frac{\beta}{a_k(x)},$$ where $p^*(x)$ is the ideal regularized optimal success probability inferred from the reference model. Standard DPO uses a global coefficient in its preference loss, $$\mathcal L_{\mathrm{DPO}} =-\mathbb E[\log\sigma(\beta\Delta_\theta)], \quad \Delta_\theta= \log\frac{\pi_\theta(y_w|x)}{\pi_{\mathrm{ref}}(y_w|x)} -\log\frac{\pi_\theta(y_l|x)}{\pi_{\mathrm{ref}}(y_l|x)}.$$ KAPPO replaces $\beta$ with the prompt-dependent $\beta_{\mathrm{eff}}(x)$. Standard GRPO uses group-normalized advantages $$A_i=\frac{R_i-\bar R}{s_R}.$$ Its binary KAPPO adaptation instead uses $A_i^{(k)}=a_k(x)A_i$, applying the multiplier after normalization to prevent cancellation. For graded rewards, KAPPO applies a nonlinear reference win-rate transformation before group normalization. These modifications target multi-attempt performance, although exact optimum matching holds under the theoretical assumptions rather than automatically for practical DPO or GRPO training. Experiments on mathematical reasoning and code generation show that the single-completion A⋆PO implementation achieves performance close to multi-completion PKPO with fewer online rollouts.Critical strengths
- The optimal-policy derivations provide a useful explanation of how pass@k optimization shifts pressure toward harder prompts.
- Single-completion A⋆PO achieves performance close to PKPO, with differences below 0.5 percentage points at matching validation budgets.
- Experiments cover multiple optimizers, two model families for math, and code generation.
Critical weaknesses
- The connection between the theory and the practical implementations needs clarification. Matching the ideal optimal policy does not guarantee the same behavior when training a neural network. In particular, the deterministic correct/incorrect pairs used for DPO do not directly satisfy the assumptions behind its reward-optimum correspondence [1].
- The method generates 30 reference-model answers per training question and uses them to fix its training coefficient. For difficult questions, observing zero versus one correct answer can substantially change that coefficient, making validation with larger or independently sampled banks important. This also introduce additional inference cost before training.
- For DPO and GRPO, which already sample multiple responses per question during training, the benefit of fixed reference calibration over using observed training success rates remains unclear. A comparison with online coefficient updates would strengthen the justification.
- The evaluation is somewhat narrow. The math experiments train and validate on MATH, then test only on AIME 2026; the code experiments test on the hard subset of LiveCodeBench v6. These results are encouraging, but broader testing would help establish how well the method generalizes. Error bars, clearer evaluation settings, and training costs that include reference calibration would also make the results easier to assess.
- The code experiments use test-case pass rates with ties, while the win-rate derivation assumes an atomless reward distribution. The authors acknowledge this mismatch but still use (Z=1). The effect on the A⋆PO results remains unclear.
References [1] Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. Rafael Rafailov et al., 2023.
Questions for authors
- The win-rate derivation assumes an atomless reward distribution, but test-case pass rates contain ties. How frequent are ties in your code experiments, and does correcting the partition term rather than assuming (Z=1) materially change the A⋆PO results?
- For DPO and GRPO, which already sample multiple responses per question during training, what is the advantage of fixed reference calibration over updating the coefficients using observed training success rates?
5 comments
Response by Authors [2/2]
Q2 & W3
Thank you for suggesting this idea! We have added a section in Appendix O that directly investigate this alternative. For @2-DPO, we reuse the eight current-policy completions already generated for preference-pair construction to recompute the adaptive coefficient introduced by KAPPO (i.e., we estimate via before solving the fixed-point equation for ). DPO’s reference-policy weights remain fixed, and no additional rollouts are required. The online variant closely tracks fixed-calibration @2-DPO and reproduces its qualitative reduction in sharpening. Thus, separate reference-bank generation can indeed be skipped in settings where several online completions are already required for the base @1 method (such as DPO in our setting and GRPO). This approach offers a valuable alternative for efficient KAPPO implementation and we thank the reviewer for bringing this up.
W4
We agree with the reviewer that the paper would benefit from even broader evaluation, but we would like to point out that our study already spans multiple optimizers, two model families, and both mathematical reasoning and code generation. At this scale, more than 100,000 GPU-hours had to be devoted to training, thorough hyper-parameter sweeps (for all baselines, see also Appendix J), and evaluation (Appendix K). We would also like to point the reviewer to the Sections 6.2 and 6.3, where we describe our efforts in ensuring a distribution shift in our final evals, thereby directly testing the generalization properties of KAPPO.
Beyond our initial submission, we believe that the calibration-size (Appendix N) and online-calibration (Appendix O) ablations provide additional robustness evidence for our method.
Concerning computational cost, Appendix P now reports calibration, online generation and grading, training updates, and the overall cost separately, see also our response to W2.
Concerning clearer evaluation settings, we took the reviewer's feedback on our presentation to heart and have added a dedicated and detailed section in Appendix J.
Q1 & W5
Assuming does not significantly affect the performance of the algorithms used in our experiments. First, our GRPO adaptation does not use the partition term at all, and second, even though the partition term is present in the A*PO adaptation, the resulting gradients are still unbiased in the on-policy regime that we use (the gradients of A*PO have the form of a Policy Gradient with representing the baseline term). As a result, changing only affects the variance of the Monte Carlo gradient estimates. Since an atomless distribution would dictate , we find this a reasonable baseline to adopt, even in the presence of ties.
Here are some statistics on ties in our coding experiments: considering a random pair of completions sampled for the same prompt, the probability that their rewards are equal is 39%. However, this percentage is highly dominated by very easy or very hard tasks where the rewards are typically 1 or 0, respectively. Conditioned on the reward being between 0 and 1, the probability of ties decreases to just 15%. Since some tasks contain invalid tests (leading to a reward in even for the gold solution), further conditioning on the tests validity, the probability of ties drops to approximately 9%.
We thank the reviewer for helping us clarify the theoretical scope and practical trade-offs of KAPPO. We hope that the additional analysis, experiments, and other revisions to the submission address their concerns on presentation and would be grateful if they could reconsider their overall rating in light of our clarifications and the revised manuscript. We welcome any remaining questions.
References
[1] Jack Lanchantin et al. “Bridging Offline and Online Reinforcement Learning for LLMs.” arXiv:2506.21495, 2025.
[2] Team Olmo et al. “Olmo 3.” arXiv:2512.13961, 2025.
[3] Project Apertus. “Apertus 1.5: From Open Text Models to Open Multimodal Reasoners.” arXiv:2610.15542, 2026.
[4] Azar, Mohammad Gheshlaghi, et al. "A general theoretical paradigm to understand learning from human preferences." International conference on artificial intelligence and statistics. PMLR, 2024.
[5] Rafael Rafailov, et al. "Direct preference optimization: Your language model is secretly a reward model." In Advances in Neural Information Processing Systems, volume 36, pp. 53728–53741. Curran Associates, Inc., 2023.
[6] Kiante Brantley, et al. "Accelerating RL for LLM reasoning with optimal advantage regression." In Advances in Neural Information Processing Systems, volume 38, pp. 151492–151531. Curran Associates, Inc., 2025.
[7] Simon Matrenok, Skander Moalla, and Caglar Gulcehre. "Quantile reward policy optimization: Alignment with pointwise regression and exact partition functions." Advances in Neural Information Processing Systems 38 (2025)
Gentle Reminder: Rebuttal & Revised PDF Before Discussion Closes
Dear Reviewer oCgk,
As the discussion period wraps up, we wanted to briefly bring our rebuttal and revised manuscript to your attention. Following your helpful suggestions, we added an experiment comparing fixed reference calibration against online coefficient updates computed from current-policy rollouts (Appendix O), an end-to-end ablation with fewer samples in the reference bank (Appendix N), and a detailed GPU-hour breakdown separating calibration, generation, and training costs against PKPO (Appendix P). We also clarified the theoretical connection between the @k-DPO loss and the KL-regularized pass@k objective (Appendix I), quantified tie frequencies in our code experiments while explaining why setting preserves unbiased on-policy gradients (Appendix F), and expanded our experimental setup details (Appendix J).
We would be very grateful if you could take a quick look at our response and revised PDF. We are happy to answer any remaining questions before the deadline!
Response by Authors [1/2]
We thank reviewer
oCgkfor the thoughtful and constructive feedback, for recognizing the theoretical insights of our optimal-policy derivations, and for highlighting the strong empirical performance of our single-completion approach and the broad experimental scope across optimizers, model families, and task domains.We have carefully considered their comments and adjusted the paper to resolve all stated weaknesses and questions (resulting in 3 new ablations, 5 new sections in the appendix and additional clarifying paragraphs in the main text). All changes are highlighted in blue in the updated paper PDF.
In the following, we summarize how we addressed the reviewer's questions and weaknesses:
W1
We agree that matching an ideal optimum does not guarantee convergence to it under non-convex neural network training dynamics. Instead, we base our approach on the well-established empirical evidence that fitting the optimal policy often provides a good training signal [5,6,7], and we dedicate a lot of effort to verifying the proposed methods empirically in a broad experimental scope.
Regarding the connection between @k-DPO and the KL-regularized pass@k optimum, we have added a dedicated discussion in Appendix I. We highly encourage the reviewer to take a look at it.
Here is the TLDR:
W2
Thank you for pushing back on this. We address the reviewer's concern related to the noise in the estimated coefficients for difficult questions (i.e. having low success probability) by considering the end-to-end algorithm performance when using less than 30 reference completions per prompt (we believe that it is easier to show the sensitivity this way than using a larger reference bank, as initially proposed by the reviewer). Appendix N demonstrates that in our setting, performance remains unchanged when reducing the reference bank from 30 to 16 or even 8 completions. With only 4 reference completions, performance decreases across the curve but remains substantially above the reference policy. We would also like to point the reviewer to Appendix M, where we theoretically analyze the sensitivity to finite reference calibration.
Additionally, we want to emphasize that reference calibration is computationally inexpensive (see Appenix P for more details). E.g., if we consider @k-A*PO runs on 4 B200 GPUs, we obtain the following estimate: reference calibration (with a reference bank of size 30) requires only approximately 8% of the total GPU-hours for a run, and this amount can be further amortized over all runs for the same model and dataset. More detailed results are provided in the table below
¹ For A⋆PO, training-update time includes reference-model log-probability computation and gradient updates.
² Overall time includes calibration, online generation and grading, training updates, validation, checkpointing, initialization, synchronization, and other overhead. Reducing the validation frequency would significantly cut down on this cost.