- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper analyzes regression-based policy updates for RLVR, in which a policy is regressed onto reward minus a baseline relative to the rollout policy. The baseline is a soft value computed at a “baseline temperature” that may differ from the update temperature, as in A⋆PO’s two-temperature scheme or Kimi K2’s mean baseline. The paper shows that, at the population level, this regression has the same optimum as a mixed KL–χ² regularized objective. The induced χ² coefficient is zero when the two temperatures are equal and increases continuously and strictly as the baseline temperature grows, reaching its maximum at the mean baseline (Prop 1). The coefficient is bounded by reward variability and is always below 1/4 (Prop 2), and its dependence on the pass rate is characterized for binary rewards (Prop 3). Building on this, the paper optimizes the KL–χ² objective directly, with χ² strength as an explicit (fixed or adaptive) parameter. Experiments with Qwen2.5 on MATH (1.5B/3B/7B) and DAPO-Math-17k (7B) show that the choice of baseline and the explicit χ² strength both affect training stability. A paired experiment shows that matching the regression-induced coefficient does not reproduce the regression’s training behavior.
Critical strengths
- A clean characterization of an implicit regularizer behind common practice. The map from baseline temperature to the induced χ² coefficient is exact, continuous, and monotone, and its attainable range has a tight bound. This gives a principled way to understand why the baseline choices made by practical methods (A⋆PO, Kimi K2) differ in regularization strength. The binary-reward analysis is checked numerically to near machine precision.
- Candid treatment of the population-versus-training gap. The paper explicitly separates fully optimized solutions from finite-step training and reports a paired experiment in which matching the coefficient does not reproduce the regression’s behavior. The empirical observation itself is useful: mean-centered regression and positive explicit χ² coefficients stabilize training on MATH, where KL-only and equal-temperature variants collapse.
Critical weaknesses
- The theory does not explain the stability differences that the experiments focus on. (a) By the paper’s own numerical example (§3.2), at the β_u = 0.001 used in the experiments, every ideal policy across the coefficient spectrum has a success probability above 0.99995. The theory therefore predicts nearly identical outcomes, yet the experiments show that PMD-PART and T²-PART collapse while PMD-MEAN is stable. (b) The paired experiment confirms that the induced coefficient alone does not determine training behavior. (c) A more direct candidate cause is not examined: with β_u = 0.001 and no ratio clipping, the regression targets for failed responses reach log-ratios of about −477 to −878. As a result, the explicit KL–χ² objective offers no clear practical advantage. PMD-MEAN is the best method on MATH 3B/7B and clearly the best on DAPO (Avg 16.7 vs. at most 13.6 for the explicit surrogates). The stabilizing coefficient is also setting-dependent: ξ = 1 is stable on MATH but collapses on DAPO.
- The experimental protocol limits the strength of the empirical conclusions. (a) Checkpoints are selected on MATH500 and on AIME 2024, which are also reported as evaluation metrics. (b) Each configuration is run only once, even though failure points vary widely across runs. (c) The MATH GRPO and A⋆PO numbers are copied from Brantley et al. (2025), and the provenance, training budget, and checkpoint selection of the DAPO A⋆PO baseline are not stated. (d) Training budgets differ across runs: 15 vs. 9 epochs for the 3B surrogates; on DAPO, 2 vs. 4 GPUs, with endings that mix time limits and manual stops. Several DAPO runs shown in Figure 1 (no regularizer, KL-only, ξ = 1) are missing from the result tables. (e) Omitting ratio clipping is non-standard, so it is unclear whether the conclusions transfer to typical training setups.
Questions for authors
- Could you clarify precisely which results are new relative to Xu et al. (2026) and Sakhi et al. (2026)? My understanding is that the mean-baseline endpoint, the b=∞ case of Prop 2, the mean-endpoint limit in Prop 3, and the inclusive-mean shrinkage appear in those works. Is the main new contribution the extension to finite baseline temperatures and the bound on the attainable range?
- Does PMD-PART still collapse if you use a larger β_u, or add standard ratio clipping? This would help determine whether the collapse is driven by the χ² strength or by the scale of the regression targets.
- Can you report results with multiple seeds and with checkpoint selection on a held-out set that is disjoint from the reported benchmarks?
- Can you retrain GRPO and A⋆PO under your own setup and budget on both datasets, and include all DAPO runs (including the crashed ones) in the result tables?
- Given that matching the coefficient does not reproduce the behavior, what does the KL–χ² view predict about training stability that could be tested? Alternatively, is there practical guidance for choosing ξ beyond trial and error?
3 comments
Question 3. Multiple seeds and checkpoint selection on a disjoint set
Also addresses Critical Weaknesses 2(a) and 2(b).
Multiple seeds
We extended the eight-step regression-surrogate comparison to three seeds, keeping the model, data, baseline rule, loss scaling, and fitting schedule matched. The previously reported divergence in training behavior persists across all three pairs.
During training, all three regression runs had a minibatch in which nearly all sampled responses fell below one-thousandth of their rollout-policy probabilities. This event was absent from the corresponding surrogate runs.
At the last validation point reached by all six runs, mean MATH500 accuracy was for regression and for the surrogate, with variation reported as the sample standard deviation across three seeds. All three surrogate runs completed the full training horizon, compared with two of the three regression runs.
Checkpoint selection and disjoint benchmarks
For MATH training, we choose checkpoints by greedy MATH500 accuracy and also evaluate them on Minerva Math, OlympiadBench and AMC23. For DAPO training, we choose checkpoints by greedy AIME 2024 accuracy and also evaluate them on AIME 2025, HMMT 2024 and HMMT 2025. These additional benchmarks do not enter checkpoint selection. We will label MATH500 and AIME 2024 as validation scores and separate them from the other evaluations.
Question 4. GRPO and A*PO under our setup and budget, and the missing DAPO runs
Also addresses Critical Weaknesses 2(c) and 2(d).
Baseline provenance
Our MATH experiments build on the public A*PO implementation and use the same Qwen2.5 base models, MATH training dataset and benchmark suite as Brantley et al. (2025). We include their published GRPO and A*PO results as reference comparisons and mark their source explicitly in the tables.
The DAPO A*PO result comes from training in our setup, with a fixed base-model reference and an offline baseline at temperature . This run started from the base model and ended at the 48-hour limit on four H200 GPUs. We evaluated the checkpoint with the highest greedy AIME 2024 validation accuracy. We will include its completed training budget and selection rule alongside the PMD and surrogate settings. The controlled objective comparisons provide the evidence for the paper's claims about baseline and penalty choice.
Training budgets
The 3B fixed-coefficient runs were allowed 15 epochs, compared with nine for most other MATH runs. However, the checkpoints used for their reported scores were all saved within the first nine epochs, as Appendix D.2 reports. Those scores therefore come from models trained within the shorter budget. The paired comparison provides evidence under a common configuration, while Appendix D.3 lists the different DAPO budgets and stopping reasons.
Missing DAPO rows
Tables 2, 6 and 7 omit three DAPO configurations that appear in Figure 1 and the run-outcome discussion in Appendix D.3. We will add the unregularized and KL-only runs with the outcome of reward collapse, and the run with the outcome of reward and response-length collapse. All three were stopped manually.
Question 5 and Critical Weakness 1. Testable stability predictions and guidance for choosing
The coefficient correspondence describes a shared optimum. Comparing the losses' gradients gives a further prediction about fitting. At the small update temperature used here, regression can continue strongly suppressing an incorrect response after its probability has already fallen, while that response's contribution to the surrogate gradient diminishes. We tested whether this leads to stronger suppression under regression on identical frozen batches.
Critical Weakness 1(a). Similar ideal policies and finite fitting
The near-identical success probabilities in Section 3.2 of the paper describe fully optimized population policies. Our paired training comparison takes eight optimizer steps over disjoint minibatches from each rollout. These steps depend on the gradients at the current neural policy, which can differ even when the objectives share an optimum.
Critical Weakness 1(c) and Question 5. Negative regression targets
Our update temperature and baseline temperature follow A*PO's main configuration (Brantley et al., 2025, Section 4 and Appendix A.5). In groups containing both correct and incorrect answers, this places the failed-response targets far below the log ratios reached during short fits.
Both objectives use the same soft baseline , computed at temperature from each prompt's sampled rewards. For a frozen response, let be its probability ratio relative to the rollout policy, and . For the squared regression loss and the centered surrogate scaled by , the derivatives are
\qquad \frac{\partial\ell_S}{\partial l}=2\beta q[-A+\beta l+\xi(q-1)].$$ Their derivatives agree at $q=1$. As fitting changes the ratios, the two losses give responses different weights. For a failed response, regression targets $l=-v/\beta$ and its derivative is $2\beta(v+\beta l)$. While $|\beta l|$ is small compared with $v$, the downward coefficient remains close to $2\beta v$, even if the response has already become much less likely. The surrogate's coefficient tends to zero as $q$ tends to zero. Regression can therefore continue suppressing sampled failures after their contributions to the surrogate gradient have become small. Each coefficient multiplies the gradient of that response's log probability, so this difference affects fitting even with an exact baseline. **Critical Weakness 1(b) and Question 5. Short-fitting test** We used one saved rollout batch from the base Qwen2.5-1.5B model and another from an early regression checkpoint. At each checkpoint, both objectives start from identical parameters with fresh optimizers and use the same responses, baseline and response weights. The surrogate uses the coefficient induced by that baseline. Each fit takes eight optimizer steps in one pass through the batch. Before each step, we compare both losses' gradients at identical parameters on the same minibatch. They initially agree, with cosine similarity approximately $0.9999$ on both batches, where one means identical directions. Their directions separate as fitting changes the response ratios. The table reports the mean log probability ratio of incorrect responses in groups containing both correct and incorrect answers, measured after eight steps. Entries give the range over four fits per objective on each saved batch. More negative values mean stronger suppression relative to the starting policy. | Starting checkpoint | Regression | Corresponding surrogate | |---|---|---| | Base model | $-1.44$ to $-1.41$ | $-0.79$ to $-0.66$ | | Early checkpoint | $-4.91$ to $-4.84$ | $-3.21$ to $-2.42$ | Regression suppresses sampled failures more strongly in every repeated check on both batches, supporting the predicted difference in finite fitting. **Practical guidance for choosing $\xi$** The spectrum identifies which coefficients can be obtained by tuning the baseline temperature. PMD-MEAN gives the largest attainable coefficient at a fixed update temperature and reward distribution. Holding both temperatures fixed still lets the induced coefficient change as the pass rate changes. The explicit objective allows us to choose and hold $\xi$ fixed independently of those changes. PMD-MEAN remains a strong baseline in our experiments. The explicit settings $\xi=0.3$ and $\xi=1$ exceed the universal $1/4$ bound, but neither consistently outperforms PMD-MEAN, and $\xi=1$ deteriorates on DAPO. The attainable range therefore helps interpret coefficient choices, while selecting a coefficient for a particular training setting still requires empirical evaluation.
Thank you for the detailed review. We answer your five questions in order and identify the related critical weaknesses below.
Question 1. New results relative to Xu et al. and Sakhi et al.
We distinguish the prior mean-baseline and Lambert-target results from our additional coefficient map, bounds and finite-temperature expansion.
The mean-baseline mixed objective and leading binary endpoint limit come from Xu et al. Sakhi et al. study advantage centering and temperature decoupling through Lambert targets. Section 2.3 and the proofs credit these results, including the inclusive-mean effect.
Our additions are the exact, continuous, and strictly increasing coefficient map over all admissible baseline temperatures, its full attainable interval, and the finite-baseline binary expansion. Proposition 2 also gives a refined variance bound and the sharp universal ceiling ξ<1/4. At the mean endpoint, the baseline-gap bound recovers Xu et al.'s result. The variance bound and sharp ceiling are additional parts of the proposition.
Question 2. Larger update temperatures and ratio clipping
Also addresses Critical Weaknesses 1(c) and 2(e).
At the population level, the KL-only objective with coefficient 0.101 has the same optimum as PMD-PART with both temperatures set to 0.101. Our experiment optimizes this objective directly.
To address concerns about whether the stability improvements arise simply from stronger regularization, we conducted twelve runs comparing the mixed objective against a stronger KL-only baseline. We matched their local quadratic behavior near the reference policy. Across two learning rates and three paired seeds per rate, the mixed objective achieved higher final validation accuracy in all six paired comparisons.
To match the penalties, let h(q)=qlogq−q+1 be the KL integrand. Since h(q)=(q−1)2/2+O((q−1)3) near one,
=(\beta+\xi)h(q)+O((q-1)^3).$$ The corresponding KL-only coefficient is therefore $\beta_{\mathrm{KL}}=\beta+\xi$. Our comparison uses $(\beta,\xi)=(0.001,0.1)$ for mixed and $0.101$ for KL-only on Qwen2.5-1.5B and MATH. Both objectives use the baseline-free formulation, loss scale one, and the same training horizon, with eight optimizer steps in one pass through each rollout batch. The final MATH500 validation accuracies are shown below as means and sample standard deviations across three seeds, in percent. | Learning rate | Mixed objective | Locally matched KL only | |---|---|---| | $10^{-6}$ | $38.7\pm11.3$ | $17.7\pm21.0$ | | $5\times10^{-7}$ | $50.1\pm3.0$ | $35.4\pm21.2$ | At both learning rates, some KL-only runs lost much of their validation accuracy. To assess whether these findings extend beyond MATH500 validation, we evaluated the final checkpoints of all runs on three additional benchmarks, Minerva, OlympiadBench, and AMC23. At both learning rates, the mixed objective achieves higher mean accuracy across all three benchmarks. Results are reported as mean and sample standard deviation across three seeds, using greedy accuracy for Minerva and OlympiadBench and Avg@32 for AMC23, consistent with the paper's evaluation protocol. | Learning rate | Objective | Minerva | OlympiadBench | AMC23 | |---|---|---|---|---| | $10^{-6}$ | Mixed | $18.4\pm2.6$ | $18.1\pm2.2$ | $24.9\pm1.2$ | | $10^{-6}$ | Locally matched KL only | $9.9\pm8.0$ | $8.4\pm7.9$ | $16.1\pm8.9$ | | $5\times10^{-7}$ | Mixed | $19.1\pm1.7$ | $21.0\pm0.8$ | $26.8\pm1.2$ | | $5\times10^{-7}$ | Locally matched KL only | $13.7\pm5.4$ | $16.4\pm4.8$ | $20.9\pm7.1$ | Beyond their local match, the mixed penalty grows quadratically with large ratios, while KL grows as $q\log q$. The observed training difference persists across the two learning rates, supporting the relevance of penalty shape after local strength has been matched. Training the corresponding regression loss at this larger temperature and testing ratio-clipped regression are directions for future work.