acceptodds
Under review as a conference paper at ICLR 2027

A Unified KL– Geometry of Approximate Policy Mirror Descent

Abstract

Regression-based reinforcement learning for language models approximates policy mirror descent (PMD). The exact KL-regularized update requires a log-partition function that is difficult to estimate from a few rollouts. Practical methods modify this baseline, as in the two-temperature formulation of \APO and the mean baseline of Kimi K2. We study how these choices change the policy-improvement objective. For regression relative to the rollout policy, we characterize a mixed KL– objective with the same population optimum. The induced strength is zero at matched baseline and update temperatures, increases continuously with baseline temperature, and reaches its maximum at the mean baseline. We bound this strength using reward variation and characterize its dependence on the two temperatures and rollout pass rate for binary rewards. The bounds identify penalty strengths that cannot be reached through baseline temperature alone. We then optimize the mixed objective directly, making the strength an explicit parameter. Experiments with Qwen2.5 models on MATH and DAPO-Math-17k show that baseline choice and explicit penalty strength affect training stability. On MATH, positive coefficients sustain training in settings where KL-only updates fail across all three model sizes.

Primary area:
reinforcement learning
Keywords:
policy mirror descent, LLM post-training, reinforcement learning with verifiable rewards, KL-chi-squared regularization, chi-squared divergence

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dCqrOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper analyzes regression-based policy updates for RLVR, in which a policy is regressed onto reward minus a baseline relative to the rollout policy. The baseline is a soft value computed at a “baseline temperature” that may differ from the update temperature, as in A⋆PO’s two-temperature scheme or Kimi K2’s mean baseline. The paper shows that, at the population level, this regression has the same optimum as a mixed KL–χ² regularized objective. The induced χ² coefficient is zero when the two temperatures are equal and increases continuously and strictly as the baseline temperature grows, reaching its maximum at the mean baseline (Prop 1). The coefficient is bounded by reward variability and is always below 1/4 (Prop 2), and its dependence on the pass rate is characterized for binary rewards (Prop 3). Building on this, the paper optimizes the KL–χ² objective directly, with χ² strength as an explicit (fixed or adaptive) parameter. Experiments with Qwen2.5 on MATH (1.5B/3B/7B) and DAPO-Math-17k (7B) show that the choice of baseline and the explicit χ² strength both affect training stability. A paired experiment shows that matching the regression-induced coefficient does not reproduce the regression’s training behavior.

Critical strengths
  1. A clean characterization of an implicit regularizer behind common practice. The map from baseline temperature to the induced χ² coefficient is exact, continuous, and monotone, and its attainable range has a tight bound. This gives a principled way to understand why the baseline choices made by practical methods (A⋆PO, Kimi K2) differ in regularization strength. The binary-reward analysis is checked numerically to near machine precision.
  2. Candid treatment of the population-versus-training gap. The paper explicitly separates fully optimized solutions from finite-step training and reports a paired experiment in which matching the coefficient does not reproduce the regression’s behavior. The empirical observation itself is useful: mean-centered regression and positive explicit χ² coefficients stabilize training on MATH, where KL-only and equal-temperature variants collapse.
Critical weaknesses
  1. The theory does not explain the stability differences that the experiments focus on. (a) By the paper’s own numerical example (§3.2), at the β_u = 0.001 used in the experiments, every ideal policy across the coefficient spectrum has a success probability above 0.99995. The theory therefore predicts nearly identical outcomes, yet the experiments show that PMD-PART and T²-PART collapse while PMD-MEAN is stable. (b) The paired experiment confirms that the induced coefficient alone does not determine training behavior. (c) A more direct candidate cause is not examined: with β_u = 0.001 and no ratio clipping, the regression targets for failed responses reach log-ratios of about −477 to −878. As a result, the explicit KL–χ² objective offers no clear practical advantage. PMD-MEAN is the best method on MATH 3B/7B and clearly the best on DAPO (Avg 16.7 vs. at most 13.6 for the explicit surrogates). The stabilizing coefficient is also setting-dependent: ξ = 1 is stable on MATH but collapses on DAPO.
  2. The experimental protocol limits the strength of the empirical conclusions. (a) Checkpoints are selected on MATH500 and on AIME 2024, which are also reported as evaluation metrics. (b) Each configuration is run only once, even though failure points vary widely across runs. (c) The MATH GRPO and A⋆PO numbers are copied from Brantley et al. (2025), and the provenance, training budget, and checkpoint selection of the DAPO A⋆PO baseline are not stated. (d) Training budgets differ across runs: 15 vs. 9 epochs for the 3B surrogates; on DAPO, 2 vs. 4 GPUs, with endings that mix time limits and manual stops. Several DAPO runs shown in Figure 1 (no regularizer, KL-only, ξ = 1) are missing from the result tables. (e) Omitting ratio clipping is non-standard, so it is unclear whether the conclusions transfer to typical training setups.
Questions for authors
  1. Could you clarify precisely which results are new relative to Xu et al. (2026) and Sakhi et al. (2026)? My understanding is that the mean-baseline endpoint, the b=∞ case of Prop 2, the mean-endpoint limit in Prop 3, and the inclusive-mean shrinkage appear in those works. Is the main new contribution the extension to finite baseline temperatures and the bound on the attainable range?
  2. Does PMD-PART still collapse if you use a larger β_u, or add standard ratio clipping? This would help determine whether the collapse is driven by the χ² strength or by the scale of the regression targets.
  3. Can you report results with multiple seeds and with checkpoint selection on a held-out set that is disjoint from the reported benchmarks?
  4. Can you retrain GRPO and A⋆PO under your own setup and budget on both datasets, and include all DAPO runs (including the crashed ones) in the result tables?
  5. Given that matching the coefficient does not reproduce the behavior, what does the KL–χ² view predict about training stability that could be tested? Alternatively, is there practical guidance for choosing ξ beyond trial and error?
3 comments

Thank you for the detailed review. We answer your five questions in order and identify the related critical weaknesses below.

Question 1. New results relative to Xu et al. and Sakhi et al.

We distinguish the prior mean-baseline and Lambert-target results from our additional coefficient map, bounds and finite-temperature expansion.

The mean-baseline mixed objective and leading binary endpoint limit come from Xu et al. Sakhi et al. study advantage centering and temperature decoupling through Lambert targets. Section 2.3 and the proofs credit these results, including the inclusive-mean effect.

Our additions are the exact, continuous, and strictly increasing coefficient map over all admissible baseline temperatures, its full attainable interval, and the finite-baseline binary expansion. Proposition 2 also gives a refined variance bound and the sharp universal ceiling ξ<1/4\xi < 1/4. At the mean endpoint, the baseline-gap bound recovers Xu et al.'s result. The variance bound and sharp ceiling are additional parts of the proposition.

Question 2. Larger update temperatures and ratio clipping

Also addresses Critical Weaknesses 1(c) and 2(e).

At the population level, the KL-only objective with coefficient 0.1010.101 has the same optimum as PMD-PART with both temperatures set to 0.1010.101. Our experiment optimizes this objective directly.

To address concerns about whether the stability improvements arise simply from stronger regularization, we conducted twelve runs comparing the mixed objective against a stronger KL-only baseline. We matched their local quadratic behavior near the reference policy. Across two learning rates and three paired seeds per rate, the mixed objective achieved higher final validation accuracy in all six paired comparisons.

To match the penalties, let h(q)=qlog⁡q−q+1h(q)=q\log q-q+1 be the KL integrand. Since h(q)=(q−1)2/2+O((q−1)3)h(q)=(q-1)^2/2+O((q-1)^3) near one,

=(\beta+\xi)h(q)+O((q-1)^3).$$ The corresponding KL-only coefficient is therefore $\beta_{\mathrm{KL}}=\beta+\xi$. Our comparison uses $(\beta,\xi)=(0.001,0.1)$ for mixed and $0.101$ for KL-only on Qwen2.5-1.5B and MATH. Both objectives use the baseline-free formulation, loss scale one, and the same training horizon, with eight optimizer steps in one pass through each rollout batch. The final MATH500 validation accuracies are shown below as means and sample standard deviations across three seeds, in percent. | Learning rate | Mixed objective | Locally matched KL only | |---|---|---| | $10^{-6}$ | $38.7\pm11.3$ | $17.7\pm21.0$ | | $5\times10^{-7}$ | $50.1\pm3.0$ | $35.4\pm21.2$ | At both learning rates, some KL-only runs lost much of their validation accuracy. To assess whether these findings extend beyond MATH500 validation, we evaluated the final checkpoints of all runs on three additional benchmarks, Minerva, OlympiadBench, and AMC23. At both learning rates, the mixed objective achieves higher mean accuracy across all three benchmarks. Results are reported as mean and sample standard deviation across three seeds, using greedy accuracy for Minerva and OlympiadBench and Avg@32 for AMC23, consistent with the paper's evaluation protocol. | Learning rate | Objective | Minerva | OlympiadBench | AMC23 | |---|---|---|---|---| | $10^{-6}$ | Mixed | $18.4\pm2.6$ | $18.1\pm2.2$ | $24.9\pm1.2$ | | $10^{-6}$ | Locally matched KL only | $9.9\pm8.0$ | $8.4\pm7.9$ | $16.1\pm8.9$ | | $5\times10^{-7}$ | Mixed | $19.1\pm1.7$ | $21.0\pm0.8$ | $26.8\pm1.2$ | | $5\times10^{-7}$ | Locally matched KL only | $13.7\pm5.4$ | $16.4\pm4.8$ | $20.9\pm7.1$ | Beyond their local match, the mixed penalty grows quadratically with large ratios, while KL grows as $q\log q$. The observed training difference persists across the two learning rates, supporting the relevance of penalty shape after local strength has been matched. Training the corresponding regression loss at this larger temperature and testing ratio-clipped regression are directions for future work.

Question 3. Multiple seeds and checkpoint selection on a disjoint set

Also addresses Critical Weaknesses 2(a) and 2(b).

Multiple seeds

We extended the eight-step regression-surrogate comparison to three seeds, keeping the model, data, baseline rule, loss scaling, and fitting schedule matched. The previously reported divergence in training behavior persists across all three pairs.

During training, all three regression runs had a minibatch in which nearly all sampled responses fell below one-thousandth of their rollout-policy probabilities. This event was absent from the corresponding surrogate runs.

At the last validation point reached by all six runs, mean MATH500 accuracy was 33.3±26.3%33.3\pm26.3\% for regression and 52.3±0.8%52.3\pm0.8\% for the surrogate, with variation reported as the sample standard deviation across three seeds. All three surrogate runs completed the full training horizon, compared with two of the three regression runs.

Checkpoint selection and disjoint benchmarks

For MATH training, we choose checkpoints by greedy MATH500 accuracy and also evaluate them on Minerva Math, OlympiadBench and AMC23. For DAPO training, we choose checkpoints by greedy AIME 2024 accuracy and also evaluate them on AIME 2025, HMMT 2024 and HMMT 2025. These additional benchmarks do not enter checkpoint selection. We will label MATH500 and AIME 2024 as validation scores and separate them from the other evaluations.

Question 4. GRPO and A*PO under our setup and budget, and the missing DAPO runs

Also addresses Critical Weaknesses 2(c) and 2(d).

Baseline provenance

Our MATH experiments build on the public A*PO implementation and use the same Qwen2.5 base models, MATH training dataset and benchmark suite as Brantley et al. (2025). We include their published GRPO and A*PO results as reference comparisons and mark their source explicitly in the tables.

The DAPO A*PO result comes from training in our setup, with a fixed base-model reference and an offline baseline at temperature 0.50.5. This run started from the base model and ended at the 48-hour limit on four H200 GPUs. We evaluated the checkpoint with the highest greedy AIME 2024 validation accuracy. We will include its completed training budget and selection rule alongside the PMD and surrogate settings. The controlled objective comparisons provide the evidence for the paper's claims about baseline and penalty choice.

Training budgets

The 3B fixed-coefficient runs were allowed 15 epochs, compared with nine for most other MATH runs. However, the checkpoints used for their reported scores were all saved within the first nine epochs, as Appendix D.2 reports. Those scores therefore come from models trained within the shorter budget. The paired comparison provides evidence under a common configuration, while Appendix D.3 lists the different DAPO budgets and stopping reasons.

Missing DAPO rows

Tables 2, 6 and 7 omit three DAPO configurations that appear in Figure 1 and the run-outcome discussion in Appendix D.3. We will add the unregularized and KL-only runs with the outcome of reward collapse, and the ξ=1\xi=1 run with the outcome of reward and response-length collapse. All three were stopped manually.

Question 5 and Critical Weakness 1. Testable stability predictions and guidance for choosing ξ\xi

The coefficient correspondence describes a shared optimum. Comparing the losses' gradients gives a further prediction about fitting. At the small update temperature used here, regression can continue strongly suppressing an incorrect response after its probability has already fallen, while that response's contribution to the surrogate gradient diminishes. We tested whether this leads to stronger suppression under regression on identical frozen batches.

Critical Weakness 1(a). Similar ideal policies and finite fitting

The near-identical success probabilities in Section 3.2 of the paper describe fully optimized population policies. Our paired training comparison takes eight optimizer steps over disjoint minibatches from each rollout. These steps depend on the gradients at the current neural policy, which can differ even when the objectives share an optimum.

Critical Weakness 1(c) and Question 5. Negative regression targets

Our update temperature β=0.001\beta=0.001 and baseline temperature b=0.5b=0.5 follow A*PO's main configuration (Brantley et al., 2025, Section 4 and Appendix A.5). In groups containing both correct and incorrect answers, this places the failed-response targets far below the log ratios reached during short fits.

Both objectives use the same soft baseline vv, computed at temperature b=0.5b=0.5 from each prompt's sampled rewards. For a frozen response, let qq be its probability ratio relative to the rollout policy, l=log⁡ql=\log q and A=r−vA=r-v. For the squared regression loss and the centered surrogate scaled by 2β2\beta, the derivatives are

\qquad \frac{\partial\ell_S}{\partial l}=2\beta q[-A+\beta l+\xi(q-1)].$$ Their derivatives agree at $q=1$. As fitting changes the ratios, the two losses give responses different weights. For a failed response, regression targets $l=-v/\beta$ and its derivative is $2\beta(v+\beta l)$. While $|\beta l|$ is small compared with $v$, the downward coefficient remains close to $2\beta v$, even if the response has already become much less likely. The surrogate's coefficient tends to zero as $q$ tends to zero. Regression can therefore continue suppressing sampled failures after their contributions to the surrogate gradient have become small. Each coefficient multiplies the gradient of that response's log probability, so this difference affects fitting even with an exact baseline. **Critical Weakness 1(b) and Question 5. Short-fitting test** We used one saved rollout batch from the base Qwen2.5-1.5B model and another from an early regression checkpoint. At each checkpoint, both objectives start from identical parameters with fresh optimizers and use the same responses, baseline and response weights. The surrogate uses the coefficient induced by that baseline. Each fit takes eight optimizer steps in one pass through the batch. Before each step, we compare both losses' gradients at identical parameters on the same minibatch. They initially agree, with cosine similarity approximately $0.9999$ on both batches, where one means identical directions. Their directions separate as fitting changes the response ratios. The table reports the mean log probability ratio of incorrect responses in groups containing both correct and incorrect answers, measured after eight steps. Entries give the range over four fits per objective on each saved batch. More negative values mean stronger suppression relative to the starting policy. | Starting checkpoint | Regression | Corresponding surrogate | |---|---|---| | Base model | $-1.44$ to $-1.41$ | $-0.79$ to $-0.66$ | | Early checkpoint | $-4.91$ to $-4.84$ | $-3.21$ to $-2.42$ | Regression suppresses sampled failures more strongly in every repeated check on both batches, supporting the predicted difference in finite fitting. **Practical guidance for choosing $\xi$** The spectrum identifies which coefficients can be obtained by tuning the baseline temperature. PMD-MEAN gives the largest attainable coefficient at a fixed update temperature and reward distribution. Holding both temperatures fixed still lets the induced coefficient change as the pass rate changes. The explicit objective allows us to choose and hold $\xi$ fixed independently of those changes. PMD-MEAN remains a strong baseline in our experiments. The explicit settings $\xi=0.3$ and $\xi=1$ exceed the universal $1/4$ bound, but neither consistently outperforms PMD-MEAN, and $\xi=1$ deteriorates on DAPO. The attainable range therefore helps interpret coefficient choices, while selecting a coefficient for a particular training setting still requires empirical evaluation.
Reviewer_2eEaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how baseline temperature changes the implicit regularization of regression-based approximate policy mirror descent. The authors characterize a unique, continuous, and strictly monotone correspondence between a family of soft baselines and mixed KL–χ² objectives sharing the same optimal policy. They derive variance-dependent bounds on the attainable coefficients and analyze their dependence on temperature and success rate for binary rewards. Building on existing mean-baseline results, the paper also directly optimizes explicit mixed objectives. Experiments on MATH and DAPO-Math-17k examine training stability across baseline and coefficient choices. A paired experiment shows that matching the theoretically implied coefficient and initial gradients does not ensure similar subsequent training behavior.

Critical strengths
  1. A precise characterization of the regularization attainable through baseline temperature.
    The paper extends existing mean-baseline results to the full soft-baseline temperature family, establishing strict monotonicity and refined variance-dependent coefficient bounds. This quantifies which regularization strengths temperature tuning can achieve and provides a concrete theoretical contribution beyond the previously understood endpoint.
  2. An informative comparison of regression and its corresponding explicit surrogate.
    The paired experiment aligns the baseline rule, empirically implied coefficient, and gradients at unit probability ratios, yet observes substantially different training trajectories. This provides a useful empirical illustration of why population-optimum correspondence alone is insufficient to predict finite-step optimization behavior.
Critical weaknesses

The connection between the new coefficient geometry and practical training remains insufficiently validated. The main theoretical additions concern a continuous temperature spectrum and attainable-coefficient bounds, but the neural experiments test only one finite intermediate baseline temperature. They do not demonstrate how the new characterization predicts stable regimes or guides coefficient selection. Nor do they directly establish a benefit from exceeding the baseline-induced attainable range; PMD-MEAN remains competitive and achieves higher reported accuracy in several settings. Furthermore, every configuration uses only one training seed, leaving the reproducibility and scope of the paired observation uncertain. Consequently, the practical contribution beyond existing mean-baseline findings remains difficult to assess.

Questions for authors
  1. Validation of the regularization comparison: Does the mixed objective retain its stability advantage against a locally strength-matched KL-only baseline with comparable learning-rate tuning? This would help distinguish the effect of χ² geometry from increased overall regularization strength.
  2. Validation of robustness: Does the regression–surrogate divergence in Figure 3 recur across independent training seeds? Can the key stability and accuracy comparisons be confirmed under matched training budgets, with uncertainty estimates or failure frequencies?
3 comments

Thank you for the careful review. We answer your two questions in order, then address the connection between the theory and practical training.

Question 1. Validation of the regularization comparison

To address concerns about whether the stability improvements arise simply from stronger regularization, we conducted twelve runs comparing the mixed objective against a stronger KL-only baseline. We matched their local quadratic behavior near the reference policy. Across two learning rates and three paired seeds per rate, the mixed objective achieved higher final validation accuracy in all six paired comparisons.

To match the penalties, let h(q)=qlog⁡q−q+1h(q)=q\log q-q+1 be the KL integrand. Since h(q)=(q−1)2/2+O((q−1)3)h(q)=(q-1)^2/2+O((q-1)^3) near one,

=(\beta+\xi)h(q)+O((q-1)^3).$$ The corresponding KL-only coefficient is therefore $\beta_{\mathrm{KL}}=\beta+\xi$. Our comparison uses $(\beta,\xi)=(0.001,0.1)$ for mixed and $0.101$ for KL-only on Qwen2.5-1.5B and MATH. Both objectives use the baseline-free formulation, loss scale one, and the same training horizon, with eight optimizer steps in one pass through each rollout batch. The final MATH500 validation accuracies are shown below as means and sample standard deviations across three seeds, in percent. | Learning rate | Mixed objective | Locally matched KL only | |---|---|---| | $10^{-6}$ | $38.7\pm11.3$ | $17.7\pm21.0$ | | $5\times10^{-7}$ | $50.1\pm3.0$ | $35.4\pm21.2$ | At the original learning rate, two of the KL-only runs exhibit substantial response-length growth and deterioration in sampled rewards, a combination absent from the mixed-objective runs. Even at the lower learning rate, one of the KL-only runs experiences a substantial decline in validation accuracy. To assess whether these findings extend beyond MATH500 validation, we evaluated the final checkpoints of all runs on three additional benchmarks, Minerva, OlympiadBench, and AMC23. At both learning rates, the mixed objective achieves higher mean accuracy across all three benchmarks. Results are reported as mean and sample standard deviation across three seeds, using greedy accuracy for Minerva and OlympiadBench and Avg@32 for AMC23, consistent with the paper's evaluation protocol. | Learning rate | Objective | Minerva | OlympiadBench | AMC23 | |---|---|---|---|---| | $10^{-6}$ | Mixed | $18.4\pm2.6$ | $18.1\pm2.2$ | $24.9\pm1.2$ | | $10^{-6}$ | Locally matched KL only | $9.9\pm8.0$ | $8.4\pm7.9$ | $16.1\pm8.9$ | | $5\times10^{-7}$ | Mixed | $19.1\pm1.7$ | $21.0\pm0.8$ | $26.8\pm1.2$ | | $5\times10^{-7}$ | Locally matched KL only | $13.7\pm5.4$ | $16.4\pm4.8$ | $20.9\pm7.1$ | Beyond their local match, the mixed penalty grows quadratically with large ratios, while KL grows as $q\log q$. The observed training difference persists across the two learning rates, supporting the relevance of penalty shape after local strength has been matched. **Question 2. Validation of robustness** *Also addresses the single-seed concern in the critical weaknesses.* We extended the paired regression-surrogate comparison to three seeds, keeping the model, data, baseline rule, loss scaling, and fitting schedule matched. The previously reported divergence in training behavior persists across all three pairs. Specifically, all three regression runs exhibit severe suppression of sampled-response probability ratios, while none of the corresponding surrogate runs exhibit the same behavior. We measure the sequence-probability ratio between the current policy and the rollout policy before the eighth optimizer step, after seven updates on the frozen rollout batch. In all three regression runs, the 99th percentile of this ratio falls below $10^{-3}$, indicating that nearly all responses in that minibatch have measured probabilities below one-thousandth of their rollout-policy values. The table summarizes this event and the validation outcomes. Accuracy is measured at the last validation point reached by all six runs and reported as mean and sample standard deviation across three seeds, in percent. Event counts and completion describe each run's full observed training. | Objective | MATH500 validation at shared training point | Runs with ratio 99th percentile below $10^{-3}$ | Runs completing the full horizon | |---|---|---|---| | Regression | $33.3\pm26.3$ | 3 of 3 | 2 of 3 | | Corresponding surrogate | $52.3\pm0.8$ | 0 of 3 | 3 of 3 | These results strengthen the evidence that regression and its corresponding surrogate can exhibit substantially different training behavior despite their population-level correspondence. The regression runs show considerable variability in validation accuracy and deterioration timing, whereas all three surrogate runs complete training and have higher mean accuracy with less variation across runs at the shared evaluation point. We will include the individual trajectories and stopping outcomes with these summaries.

Critical weaknesses. Connection between the coefficient geometry and practical training

Our contribution is not a prescription for an optimal regularization coefficient, but a characterization of the regularization attainable through regression baselines and an empirical investigation of how baseline choice, coefficient strength, and fitting objective affect training stability.

First, the theory characterizes what can be controlled through baseline temperature. Proposition 1 establishes that varying the baseline temperature produces a continuous interval of induced χ2\chi^2 coefficients, with PMD-MEAN attaining its maximum. Proposition 2 bounds this interval, while Proposition 3 characterizes its dependence on the rollout pass rate. These results show that baseline-temperature tuning cannot produce arbitrary regularization strengths and that fixed temperatures do not imply a fixed coefficient throughout training. Explicitly choosing ξ\xi removes these restrictions, although the theory does not establish which coefficient will yield the best empirical performance.

Second, the experiments investigate the practical consequences of these distinctions. The MATH comparisons show that baseline choice and explicit coefficient strength affect training stability, although accuracy does not vary monotonically with regularization. We also compare regression with its corresponding surrogate using the same induced coefficient. To address the reproducibility concern, we extended this paired comparison to three seeds. All three regression runs exhibit severe sequence-ratio suppression, while none of the surrogate runs exhibit the same behavior. All three surrogate runs complete training, compared with two of three regression runs, and achieve more consistent validation accuracy.

Third, we acknowledge that exceeding the attainable coefficient range does not necessarily improve performance. The explicit settings ξ=0.3\xi=0.3 and ξ=1\xi=1 exceed the universal 1/41/4 bound, but neither consistently outperforms PMD-MEAN, and ξ=1\xi=1 deteriorates on DAPO. PMD-MEAN remains a strong practical baseline. These findings indicate that greater control over regularization does not automatically translate into higher accuracy.

Thank you again for your thoughtful review. Your suggestions helped us strengthen the experimental comparisons. We hope the answers and additional results above address your main concerns, and we would be glad to clarify anything further during the discussion. If they do, we would be grateful if you would consider updating your rating.

Thanks for the additional experiments and clarifications. The KL comparisons with matched local regularization strength and the results across three seeds address the concerns about the original controls and reproducibility. The analysis on identical frozen batches also provides a more concrete explanation of the different training dynamics. I will raise the rating to 3.

Reviewer_ybMTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how baseline temperature affects the implicit regularization of regression-based policy mirror descent (PMD) for large language model post-training. The authors show that the regression objective shares the same population optimum as a mixed KL–χ² objective and characterize how the induced χ² coefficient increases continuously from zero at matched temperatures to a finite maximum at the mean baseline. They also derive bounds on the attainable regularization strength based on reward variation.

The authors further investigate directly optimizing the mixed KL–χ² objective with an explicitly chosen regularization coefficient, including an adaptive heuristic based on response pass rate. Experiments with Qwen2.5 models across different sizes and mathematical reasoning datasets show that baseline choice and χ² regularization strength affect training stability. Positive χ² regularization sustains training in settings where KL-only updates fail. The experiments also demonstrate that regression and its corresponding explicit surrogate can exhibit different training dynamics despite sharing the same population optimum.

Critical strengths
  1. The paper provides mathematically sound derivations characterizing the relationship between baseline temperature and the induced χ² regularization coefficient. The continuous coefficient spectrum and bounds on the attainable regularization strength help explain the limitations of controlling regularization through baseline temperature alone.

  2. The paper evaluates the explicit KL–χ² objective across three Qwen model sizes and two mathematical reasoning datasets. The results demonstrate that positive χ² regularization can sustain training in settings where KL-only updates fail, while also revealing sensitivity to regularization strength.

  3. The paired experiment demonstrates that regression and its corresponding explicit surrogate can exhibit different finite-step training dynamics despite sharing the same population optimum.

Critical weaknesses
  1. Although explicitly choosing the χ² coefficient provides greater flexibility, the proposed approach does not consistently outperform PMD-MEAN. PMD-MEAN achieves higher MATH500 accuracy at 3B and 7B and the highest average accuracy in the DAPO comparison. The practical benefit of explicitly controlling the coefficient over existing approaches remains unclear.

  2. The KL-only baseline uses a fixed coefficient of 0.001. Since KL and χ² penalties have similar local quadratic behavior near the reference policy, it remains unclear whether the observed stability improvements result specifically from χ² regularization or simply from stronger overall regularization.

  3. The effectiveness of explicit χ² regularization depends on the selected coefficient, which varies across benchmarks and potentially model sizes. This introduces additional tuning requirements and limits the practical convenience of the proposed approach.

  4. The experiments are restricted to the Qwen model family and mathematical reasoning tasks. Evaluations on other model families and reasoning tasks, such as coding, would strengthen the evidence for broader applicability.

Questions for authors
  1. What motivates the adaptive choice ? At small update temperatures, the mean-baseline coefficient approaches (p(1-p)), whereas the proposed rule increases with the pass rate. Is this stronger regularization at high pass rates intentional, and what benefit is it intended to provide?

  2. Could the authors investigate whether increasing the KL-regularization coefficient would achieve similar training stability improvements to adding the χ² penalty, particularly given their similar behavior for small policy updates?

  3. Appendix D.4 states that the corresponding χ² coefficient is computed for a uniformly weighted reward group, while dynamic microbatching introduces unequal sequence weights during fitting. How substantial is this imbalance in the paired runs, and how does it affect the interpretation of the regression–surrogate correspondence and their observed divergence?

3 comments

Thank you for the careful review. We answer your three questions in order, then address the remaining critical weaknesses.

Question 1. Motivation for the adaptive choice and stronger regularization at high pass rates

Our choice ξt(x)=p^t(x)\xi_t(x)=\widehat p_t(x) is motivated by preserving the relative balance between reward and regularization for correct responses when moving from the mean-baseline to the baseline-free objective.

Unlike the regression-induced coefficient, this heuristic (as stated in Section 3.1) intentionally maintains regularization as the pass rate approaches one. Its motivation concerns the reward contribution of a correct response. With the mean baseline, that contribution is proportional to 1−p1-p, and the induced coefficient approaches p(1−p)p(1-p) at small update temperatures. Ignoring the KL term, this contribution is

=(1-p)\left[q-\frac{p}{2}(q-1)^2\right].$$ Choosing $\xi=p$ in the baseline-free objective preserves this relative balance for correct responses after their reward signal changes from $1-p$ to $1$. It uses a quantity already available from the rollout group and retains a penalty as sampled success approaches one. **Question 2. Increasing the KL-regularization coefficient** *Also addresses Critical Weakness 2 about stronger overall regularization.* To address concerns about whether the stability improvements arise simply from stronger regularization, we conducted twelve runs comparing the mixed objective against a stronger KL-only baseline. We matched their local quadratic behavior near the reference policy. Across two learning rates and three paired seeds per rate, the mixed objective achieved higher final validation accuracy in all six paired comparisons. To match the penalties, let $h(q)=q\log q-q+1$ be the KL integrand. Since $h(q)=(q-1)^2/2+O((q-1)^3)$ near one, $$\beta h(q)+\frac{\xi}{2}(q-1)^2 =(\beta+\xi)h(q)+O((q-1)^3).$$ The corresponding KL-only coefficient is therefore $\beta_{\mathrm{KL}}=\beta+\xi$. Our comparison uses $(\beta,\xi)=(0.001,0.1)$ for mixed and $0.101$ for KL-only on Qwen2.5-1.5B and MATH. Both objectives use the baseline-free formulation, loss scale one, and the same training horizon, with eight optimizer steps in one pass through each rollout batch. The final MATH500 validation accuracies are shown below as means and sample standard deviations across three seeds, in percent. | Learning rate | Mixed objective | Locally matched KL only | |---|---|---| | $10^{-6}$ | $38.7\pm11.3$ | $17.7\pm21.0$ | | $5\times10^{-7}$ | $50.1\pm3.0$ | $35.4\pm21.2$ | At the original learning rate, two of the KL-only runs exhibit substantial response-length growth and deterioration in sampled rewards, a combination absent from the mixed-objective runs. Even at the lower learning rate, one of the KL-only runs experiences a substantial decline in validation accuracy. To assess whether these findings extend beyond MATH500 validation, we evaluated the final checkpoints of all runs on three additional benchmarks, Minerva, OlympiadBench, and AMC23. At both learning rates, the mixed objective achieves higher mean accuracy across all three benchmarks. Results are reported as mean and sample standard deviation across three seeds, using greedy accuracy for Minerva and OlympiadBench and Avg@32 for AMC23, consistent with the paper's evaluation protocol. | Learning rate | Objective | Minerva | OlympiadBench | AMC23 | |---|---|---|---|---| | $10^{-6}$ | Mixed | $18.4\pm2.6$ | $18.1\pm2.2$ | $24.9\pm1.2$ | | $10^{-6}$ | Locally matched KL only | $9.9\pm8.0$ | $8.4\pm7.9$ | $16.1\pm8.9$ | | $5\times10^{-7}$ | Mixed | $19.1\pm1.7$ | $21.0\pm0.8$ | $26.8\pm1.2$ | | $5\times10^{-7}$ | Locally matched KL only | $13.7\pm5.4$ | $16.4\pm4.8$ | $20.9\pm7.1$ | Beyond their local match, the mixed penalty grows quadratically with large ratios, while KL grows as $q\log q$. The observed training difference persists across the two learning rates, supporting the relevance of penalty shape after local strength has been matched.

Question 3. Dynamic microbatching and unequal sequence weights

We measured the actual response weights in paired fits on two frozen batches and found them nearly uniform, with identical weights for regression and its surrogate.

First, we verified that both objectives assign the same weights to individual responses. Using the base model and an early regression checkpoint, we fitted regression and its corresponding surrogate on identical frozen responses. Both objectives produced identical microbatch partitions and assigned each response the same weight.

Second, we quantified the degree of weight imbalance. The normalized effective sample size (ESS) ranged from 0.9960.996 to 1.0001.000, where 11 indicates perfectly uniform weighting. Response weights showed little association with response length or correctness, and mixed-reward groups received nearly the same proportion of total weight as their proportion of responses.

Finally, unequal weights do not inherently break the initial-gradient correspondence. At q=1q=1, regression and the appropriately scaled centered surrogate have matching per-response gradients. Applying the same response weights to both objectives preserves this equality, even when the weights are nonuniform. However, their gradients can diverge as fitting progresses.

Critical Weaknesses 1 and 3. Practical benefit of explicit control and coefficient tuning

We agree that explicit coefficient control does not consistently outperform PMD-MEAN, and we do not claim that it does. Rather, its benefit is to make the regularization induced by regression methods explicit, independently controllable, and amenable to systematic study.

Our analysis identifies the regularization induced by PMD-MEAN. PMD-MEAN belongs to the regression family we analyze. It corresponds to the mean-baseline endpoint, obtained as the baseline temperature approaches infinity. For a fixed update temperature and rollout reward distribution, this endpoint induces the largest coefficient ξ∞\xi_\infty. Its strong empirical performance therefore identifies a competitive existing choice within the regularization spectrum.

Second, explicit control allows us to investigate regularization regimes that regression methods cannot attain. Proposition 2 bounds every induced coefficient below 1/41/4 for rewards in [0,1][0,1]. We therefore evaluate ξ=0.3\xi=0.3 and ξ=1\xi=1, both above that ceiling. In Table 1, neither improves MATH500 accuracy over PMD-MEAN at any of the three model sizes. The full benchmark tables show some gains on other benchmarks, but no consistent benefit, and ξ=1\xi=1 deteriorates on DAPO. These experiments provide evidence that increasing regularization beyond the regression family's attainable range is not a reliable way to improve performance in the settings studied.

Finally, direct control changes how regularization can be specified and tuned. The explicit objective allows us to hold ξ\xi fixed as pass rates evolve, without estimating a reward baseline. Directly selecting ξ\xi in the explicit objective also provides a more interpretable alternative to tuning the baseline temperature while holding the update temperature fixed.

Critical Weakness 4. Restricted model family and mathematical reasoning tasks

We agree that evaluations on other model families and reasoning tasks, such as coding, would strengthen the empirical evidence for broader applicability. Our experiments span three model sizes, two mathematical reasoning training sets, and different numbers of responses per prompt. Importantly, Propositions 1 and 2 characterize the induced regularization under the stated population assumptions and bounded rewards, independently of the model architecture or task domain. Thus, while our empirical results focus on mathematical reasoning with Qwen models, the theoretical characterization is not specific to these settings. Evaluating whether the same empirical trends hold for coding tasks and other model families is an important direction for future work.

Thank you for the detailed rebuttal and additional experiments. I particularly appreciate the new results on dynamic microbatch weighting, which provide strong supporting evidence for the paper's contributions and effectively address several of my original concerns (most notably W2).

Overall, I am encouraged by these additions and maintain my positive assessment of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.