- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MaxPO introduces a leave-two-out baseline for max@K policy gradients, producing an unbiased estimator with exactly centered advantages. It derives connections to existing estimators and evaluates variance in controlled environments and performance on mathematical reasoning and code generation.
Critical strengths
- The analysis clarifies which existing max@K estimators share the same centered signal and how standard-deviation normalization changes their unbiasedness properties.
Critical weaknesses
-
Average gains are modest. Averaging the task-average scores across both models in Table 2, improvements over the strongest baseline at each model and sampling budget are approximately 2.1, 2.0, and 2.5 percentage points at pass@8, pass@32, and pass@256. The table does not report uncertainty across training runs.
-
The benefit over closely related prior methods is untested. Appendices F.5 and F.6 connect MaxPO to specific unnormalized or mean-centered forms of [1] and [2], but neither method is directly evaluated. It remains unclear whether MaxPO's estimator offers practical gains over their existing formulations.
[1] Chen et al. Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models. arXiv:2508.10751, 2025. [2] Bagirov et al. The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisation. arXiv:2510.23393, 2025.
Questions for authors
- Do the average accuracy gains persist across independent training seeds?
- Under matched training and tuning budgets, how does MaxPO compare with [1] and [2], and how much does the normalization choice affect performance?
3 comments
Thanks for the additional experiments and for clarifying the relationship to prior work. Both increased my confidence in the results, and I’ve raised my score to 3.
(And section 5.1 describes variance reduction as a theoretical property. Are there conditions under which the leave-two-out baseline provably reduces policy-gradient variance relative to PKPO, or should I read the reduction as an empirical finding? My understanding is that Proposition 3.1 establishes exact centering, which does not by itself imply lower gradient variance.
Thank you for the response
Thank you for the follow-up and for increasing the score. The other reviewers also asked about this point.
A precise sufficient condition is where and is the centered advantage of MaxPO.
The optimal variance reduction baseline (https://arxiv.org/abs/1301.2315) is given by , i.e., it is a weighted average of the different rewards, where the weighting is proportional to the square of the log-derivative. If we are agnostic to the magnitude of the log-derivative (e.g., assume that it is constant or independent of the reward), then the centered expected reward gives the minimum variance (i.e., our centered baseline gives the minimum variance). The main comparison method, PKPO, also has a baseline that is agnostic to the log-derivative terms. While in practice, it can happen that log-derivative terms align effectively so that PKPO has lower variance, in a typical setting, our approach will be better. Note also that this optimal baseline technique is rarely used in practice because it has not been shown to work better than the centered baseline in practice. Our experiments also showed that MaxPO has a lower gradient variance than PKPO both in toy experiments, and in the LLM experiments (Fig. 1,3,4,5,7)
A simple intuition for why centering helps is the following:
Consider advantages that are offset from the center by so that , where are centered advantages.
Let . It is known that and assume , and that and are uncorrelated.
By uncorrelation and centering, the cross term vanishes:
Therefore,
The offset thus adds an extra variance of . A centered baseline removes this offset.
More generally, because the advantages of PKPO are always bigger than the centered advantage, we have , and thus we do not need the cross term to vanish, but a sufficient condition is .
We added a paragraph about this at the end of Section 2.1. We agree that this improves the clarity and thank you for the suggestion.
Thank you for the review
Thank you for the review and for carefully reading our work.
The results on coding tasks in the appendix show greater gains (Table 6, Figure 11). Compared to GRPO: 2.6, 4.9, 5.3 at pass@(8,32,256) respectively. Also, to elaborate on your claim regarding Table 2, the improvements on the Qwen-7B model are up to around 4-5% points, and up to around 2% points for the Llama-3B model. So the performance may be model-dependent, and on some models the improvements appear significant.
We agree that checking persistence is important, and to address the concern we ran 2 extra seeds on the Qwen 7B model with math tasks (so a total of 3 seeds; the extra experiments cost around 2000USD).
Values are mean ± standard error. We added the table to App J, and will replace the results in the paper when accepted.
The results show that gains remain consistent for higher k values, but were perhaps overestimated in the previous experiments. An important observation to us was the comparison with PKPO. Namely, the improvements over PKPO remain consistent across all 3 seeds, and MaxPO always outperformed it. This provides supporting evidence to our main claim that the centered baseline outperforms the one from PKPO.
In practice, it is quite difficult to achieve fully statistically significant results in LLM training due to the cost. We believe the additional corroborative evidence (lowered gradient variance, toy experiments) provide a strong case that our main claim of improving over the non-centered PKPO is valid.
As a preliminary we note that Bagirov’s method was developed concurrently to ours, and that Chen’s work is limited to binary rewards (and their derivation does not extend to continuous rewards). Both are currently only available as arxiv papers and the ICLR reviewer policy notes that lack of comparisons to arxiv papers cannot be a basis for rejection, but authors are strongly encouraged to discuss such works (https://iclr.cc/Conferences/2027/ReviewerGuidelines). We believe the current discussion fits within these guidelines.
A bit more background:
Compared to Bagirov: While one of the baseline techniques that they use ends up proportional to ours, the magnitude of their estimator remains biased w.r.t. the correct magnitude. Moreover, they do not perform experiments with this variant. Their main experimental method divides the advantages with the std of the output advantages. However, this biases the estimator. For example, if the rewards are binary, then the normalization actually makes the normalized advantages identical to the advantages of GRPO (except that when there are sufficiently many correct responses in the batch, then the advantage will be 0 for Bagirov, but not for GRPO). Thus the estimator is not theoretically unbiased and it remains unclear how related to the max@k objective it is.
Chen’s work similarly has an std normalization, and the connection to max@k optimization in that case is not clear. Furthermore, Chen’s work is algorithmically limited to binary rewards, whereas ours works with continuous rewards. We also have coding experiments with non-binary pass rate rewards where Chen’s method is not applicable (Table 6, Figure 11). In practice, we believe the performance on binary reward tasks will be similar.
The main comparison algorithm is PKPO that is also unbiased, but their baseline technique is uncentered. Our experiments examined both the gradient variance and the performance, and provided convincing evidence that MaxPO outperforms PKPO and the gradient variance is reduced.
Again, thank you and do not hesitate to ask if there are any remaining concerns or questions.