acceptodds
Under review as a conference paper at ICLR 2027

Learn What’s Left, Not What’s Mastered: Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization

Abstract

Reinforcement learning (RL) with group-relative advantages has become the de facto standard for post-training language model reasoners. However, when optimizing multiple reward objectives, existing methods typically scalarize the reward vector with a fixed weighted sum before group-wise standardization. We show that this design leads to two fundamental problems: rollouts with distinct reward profiles can receive identical advantages, and all objectives are optimized with fixed relative weights regardless of their current level of saturation. Consequently, an objective whose rewards are already near their upper bound can retain substantial influence when its rewards still vary within rollout groups. We introduce **Saturation Aware Advantage Reweighting for Multi-Reward Policy Optimization** (SA-MRPO), which standardizes each reward objective independently and adaptively discounts its contribution according to a batch-level estimate of objective saturation. This changes the relative contribution of each objective according to its observed reward headroom. We further derive an exact condition under which saturation aware reweighting reverses the sign of a rollout's aggregate advantage. Across mathematical reasoning with two- and three-objective reward combinations, SA-MRPO improves the harder correctness objective over GDPO in 12 of 15 benchmark comparisons, with gains of up to 5% on AIME24. On adaptive reasoning it improves accuracy on all five benchmarks, by 3.8% on average and up to 9.2% on AMC23, and on coding benchmarks it improves pass rate by up to 2.3%, while in all settings maintaining the easier objectives near their already satisfied levels. Additional experiments characterize the accompanying reward tradeoffs and sensitivity to corrupted rewards.

Primary area:
optimization
Keywords:
Reinforcement Learning, Large Language Model Post-Training, Multi-Reward Optimization, Group Relative Policy Optimization, Adaptive Reward Reweighting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ipbmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies GRPO-style training with multiple reward objectives, such as correctness, length, and format. It argues that summing rewards with fixed weights loses information and keeps optimizing objectives that are already nearly saturated. The proposed SA-MRPO standardizes each objective within the group, as GDPO does, and then scales each objective's weight by (1 − s)^γ, where s is the batch-level saturation of that objective relative to its reward bounds. With γ = 0 it reduces to GDPO, and the paper derives the value of γ at which a rollout's aggregate advantage changes sign. Experiments cover mathematical reasoning with two or three objectives, adaptive reasoning with a length reward, and code generation, mainly against GDPO and, for adaptive reasoning, several recent multi-reward methods.

Critical strengths
  1. A simple and well-motivated change. The problem is clearly illustrated in Figure 1: under fixed weights, a nearly saturated objective such as format can still dominate the ranking of rollouts. The fix is a single multiplicative factor on top of GDPO, is easy to implement, and leaves the policy update unchanged.

  2. Broad evaluation with relevant baselines. The method is tested on math, adaptive reasoning, and code. In the adaptive reasoning setting, it is compared with five recent multi-reward methods under several seeds and achieves the best average accuracy (Table 4). The paper also reports the accompanying costs, such as longer responses and higher exceed rates, instead of only accuracy.

Critical weaknesses
  1. The gains are not separated from simply down-weighting the length objective. In most experiments, the objective that saturates first is length or format, so SA-MRPO mainly shifts weight from length toward correctness. The results follow this pattern. In Table 2, accuracy rises by 3.8 points while the mean response length grows from 333 to 459 tokens. In Table 1, the exceed rate is higher than GDPO's in 10 of 15 comparisons. Section 5.5 describes γ as trading length reward for correctness. Table 4 compares SA-MRPO with SAW and GRPO at similar lengths, but neither of them is GDPO with a smaller fixed weight on length. That comparison is the direct test of whether the saturation-dependent schedule matters beyond a different static allocation.

  2. The allocation is "adaptive" in name, but in practice it is set by hand. How much weight moves between objectives depends on two manual choices: the exponent γ and the reward bounds used to compute saturation. Both change across experiments. γ is 0.25 in the 3B math setting and 0.75 in adaptive reasoning, and the best value differs again in the separate 3B study (Section 5.5). The length bounds are (0, 4000) in Table 1 and (1024, 2048) in Table 4. Section 5.5 also shows that γ simply moves the model along the correctness–length tradeoff. The paper gives no rule for setting either choice on a new task, which makes the method's behavior hard to predict outside the reported configurations.

Questions for authors
  1. How does GDPO perform with a smaller fixed weight on the length objective, chosen to reach a response length similar to SA-MRPO's? A sweep over a few fixed weights would show whether the saturation-aware schedule beats every static allocation.

  2. How should γ be chosen for a new task? The current experiments use different values, and the best one varies across settings.

  3. Could the paper show how the effective weights w̃_k change during training? This would show whether the reweighting stays dynamic or settles early into a fixed ratio.

5 comments
AuthorsOpenReview ↗

Part 1: Static-weight trade-off

Thank you for recognizing the simplicity of the method and the evidence across tasks. We address W1 and W2 together with their corresponding questions.

W1 / Q1

How does GDPO perform with a smaller fixed weight on the length objective, chosen to reach a response length similar to SA-MRPO’s?

The submitted paper already provides evidence that the accuracy gain is not solely explained by longer responses. In Table 4, SA-MRPO achieves 49.9 ± 2.1% versus SAW's 47.4 ± 1.7% at 856 versus 855 tokens, using two and three training seeds, respectively. SA-MRPO's mean accuracy is higher on each of the four external mathematical benchmarks. This addresses the generation-cost concern in W1; the smaller-fixed-weight GDPO alternative is examined below.

We evaluated GDPO with a fixed length weight of 0.3 while keeping the correctness weight at 1, using three training and external-evaluation seeds. The table also includes the two-stage diagnostic discussed under W2 / Q3:

MethodDeepScaleR correctnessLength rewardaccuracy %Mean tokens
GDPO, w_len=10.4480.99742.6 ± 1.0487
GDPO, w_len=0.30.5180.95449.26 ± 0.67742
Two-stage schedule (W2 / Q3)0.5200.95549.17 (single run)715
SAW0.5100.92747.4 ± 1.7855
SA-MRPO, γ\gamma=0.750.5330.92349.9 ± 2.1856

Reducing the fixed length weight recovers a substantial part of the gap to equal-weight GDPO. The 0.3 length weight yields shorter responses and lower mean accuracy than SA-MRPO, with a higher length reward, so the two occupy different tradeoff points. Together, these results support competitive observed tradeoffs; the 742-token is not an exact length match or a complete static frontier. The near-equal-length SAW comparison and the smaller-weight GDPO performance answer complementary questions.

AuthorsOpenReview ↗

Part 2: Exponent choice and effective-weight trajectories

W2 / Q2

How should γ\gamma be chosen for a new task?

Our Table 6 provides a three-seed sensitivity study: correctness means are 0.476, 0.485, 0.533, and 0.540 for γ\gamma=0.25, 0.5, 0.75, and 1.0, respectively, versus 0.448 for GDPO. Every tested positive exponent improves mean correctness over GDPO in this setting. As discussed in section 5.5, moving from γ\gamma=0.75 to 1.0 increases correctness from 0.533 to 0.540 while reducing length reward from 0.923 to 0.899. These results support choosing the operating point according to the desired tradeoff.

A practical starting point is γ\gamma=0.75 when correctness is prioritized and length or format is easy to learn. In that same regime, γ\gamma=0.25 or 0.5 retains more emphasis on auxiliary performance. Keep the base preference weights fixed and compare held-out correctness together with actual auxiliary requirements, such as exceedance or format-compliance rates. The preferred operating point can differ across tasks because their reward states and requirements differ; task difficulty alone is not the criterion.

γ\gamma controls response strength; the current reward state determines the effective allocation. We respectfully disagree with W2's characterization that the allocation is adaptive only in name because γ\gamma is chosen by hand. Selecting the sensitivity of a state-dependent rule does not prescribe its resulting objective coefficients. At a fixed positive γ\gamma, the effective coefficient ratios change as the relative observed headrooms change; γ\gamma=0 recovers GDPO. Increasing γ\gamma strengthens the relative emphasis on objectives with greater remaining headroom. It discounts objective-advantage contributions according to saturation, without changing the reward functions or prescribing a transition step.

The reward bounds and token thresholds cited in W2 are also different quantities. Appendix B.2 specifies reward-value bounds [0,1] for the saturation calculation. The ranges (0,4000) and (1024,2048) are token thresholds defining the length reward function, as listed in Table 5. Within each comparison, the competing methods share that reward function. These thresholds affect the accuracy–length tradeoff, but they are not different saturation upper bounds selected specifically for SA-MRPO.

W2 / Q3

Could the paper show how the effective weights w~k\widetilde{w}_k change during training?

Our Figure 5 reports these trajectories over three training seeds, together with reward means and mean response length. Appendix C.1 explains the changing coefficient ratio: the length coefficient declines as the length reward approaches its upper bound, while the correctness coefficient remains larger later in training. The figure shows an early reallocation phase followed by a more stable regime.

This directly addresses whether the coefficients evolve or settle: both behaviors occur at different training stages. Stabilization under a more stable observed reward state is consistent with an adaptive rule; it does not imply that the coefficients were set by hand. The same fixed γ\gamma continues to map current saturation statistics to effective weights throughout training. State dependence does not require the weights to keep changing substantially after the observed state stabilizes.

The two-stage diagnostic in the table under W1 / Q1 uses a schedule informed by the observed SA-MRPO trajectory: the length-to-correctness weight ratio changes from 0.55 to 0.25 at step 30. In one run, it reaches DeepScaleR correctness 0.520 and length reward 0.955, with external accuracy 49.17% at 715 tokens. This differs in auxiliary performance and generation length from SA-MRPO's 0.533 / 0.923 and 49.9% / 856 tokens, so it does not establish equivalence or a matched-cost advantage for dynamic weighting. The schedule is prescribed from an observed trajectory, whereas SA-MRPO obtains its coefficients from current reward statistics.

The submitted paper therefore already contains a near-equal-length comparison, a multi-seed exponent study, and measured weight trajectories. Together with the additional static experiment, these results clarify both the practical contribution and the state-dependent mechanism. We would appreciate reconsideration of the assessment in light of this evidence.

Reviewer_ipbmOpenReview ↗

Response to authors

The response resolves some of my concerns. I mixed up the saturation bounds with the token thresholds of the length reward, and I withdraw that point. Figure 5 already answers Q3, and I agree that the rule is state-dependent.
My main concern remains. With a fixed length weight of 0.3, GDPO reaches 49.26%, close to SA-MRPO's 49.9%, and it uses fewer tokens. This gap is within seed variation, and the two-stage schedule performs about the same. Like the fixed weight, γ needs to be tuned for each task. So it is still unclear how much the saturation-aware weighting adds beyond a well-chosen fixed weight. This weighting is the only change SA-MRPO makes to GDPO, so I keep my score at 2.

Reviewer_eTTmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SA-MRPO independently normalizes multiple reward objectives and reweights their advantages according to remaining reward headroom. The method reduces the influence of nearly saturated objectives and is evaluated on mathematical reasoning, adaptive reasoning, and code generation.

Critical strengths

Simple and interpretable design. The method introduces a single saturation exponent while retaining the standard policy update. Its relationship to GDPO and its effect on relative objective weights are transparent.

Evidence across several task settings. SA-MRPO improves mathematical accuracy over GDPO in 12 of 15 comparisons and improves coding test-case pass rates on three of four benchmarks.

A useful comparison at similar generation cost. In Table 4, SA-MRPO reaches 49.9% average accuracy versus 47.4% for SAW at almost identical mean response lengths—856 versus 855 tokens. This suggests that the improvement is not entirely explained by generating longer answers.

Limitations are examined rather than ignored. The paper reports auxiliary-reward degradation, corrupted-reward experiments, and multiple-seed results for the controlled adaptive setting. It explicitly avoids claiming that previously optimized objectives are guaranteed to remain preserved.

Critical weaknesses

The benefit over well-tuned static weighting remains unclear. The main comparison uses GDPO with prescribed weights, but does not establish a static-weight trade-off frontier. A lower fixed length-reward weight might recover some of the observed gains, so the value of adapting weights during training needs stronger isolation.

Primary-objective gains sometimes weaken auxiliary objectives. Length exceedance increases in 10 of 15 mathematical comparisons, and coding bug rates increase on several benchmarks. On TACO, both pass rate and bug rate worsen. Format compliance is not separately reported, limiting the claim that easier objectives remain satisfactorily maintained.

Statistical coverage and experimental controls are uneven. The main sampled mathematical results use one training run, while the external adaptive comparison uses only two SA-MRPO runs. Appendix B also notes that the GRPO baseline includes additional format and KL terms, preventing a clean attribution of that comparison to advantage construction.

Reward headroom can differ from learnable headroom. The method emphasizes objectives whose means are far below prescribed upper bounds, even when this gap reflects irreducible difficulty or poorly calibrated bounds. The corruption experiment is useful, but does not fully establish robustness to unattainable reward ceilings or heterogeneous task difficulty.

Questions for authors

Does SA-MRPO outperform a tuned static-weight GDPO frontier when response length and auxiliary-reward satisfaction are matched? How were objective weights and the saturation exponent selected?

Can the principal mathematical and coding gains be confirmed across additional seeds, with format compliance and all auxiliary rewards reported? Can GRPO be rerun with identical objectives and KL settings?

How sensitive is performance to conservative or unattainable reward bounds? Would the conclusions change when some objectives have persistent headroom but little achievable improvement?

3 comments
AuthorsOpenReview ↗

Part 1: Static-weight trade-offs and parameter choices

Thank you for recognizing the interpretable design, the cross-task evidence, and the comparison at similar generation cost. We address the weaknesses together with their corresponding questions.

W1 / Q1.1

Does SA-MRPO outperform a tuned static-weight GDPO frontier when response length and auxiliary-reward satisfaction are matched?

We added an additional static-weight experiment that keeps GDPO's correctness weight at 1 and reduces its length weight to 0.3, using three training and external-evaluation seeds:

MethodDeepScaleR correctnessLength rewardaccuracy %Mean tokens
GDPO, w_len=10.4480.99742.6 ± 1.0487
GDPO, w_len=0.30.5180.95449.26 ± 0.67742
SAW0.5100.92747.4 ± 1.7855
SA-MRPO, γ\gamma=0.750.5330.92349.9 ± 2.1856

Reducing the fixed length weight recovers a substantial part of the gap to equal-weight GDPO. The new entry's response length is shorter but has lower mean accuracy than SA-MRPO. This supports competitive observed tradeoffs, without establishing dominance over a complete tuned static frontier.

The existing near-equal-length comparison provides complementary evidence: SA-MRPO reaches 49.9% versus SAW's 47.4% at 856 versus 855 tokens, with similar internal length rewards of 0.923 and 0.927. Beyond this setting, CodeContests test-case pass rate increases from 19.2% to 20.6% with the reported bug rate unchanged at 15.5%. In the 3B three-objective mathematical setting, Minerva accuracy increases from 16.9% to 18.1% with the same reported exceedance rate of 0.1%. Thus, the evidence includes gains at similar generation length or unchanged reported auxiliary metrics.

Q1.2

How were objective weights and the saturation exponent selected?

The role of γ\gamma is to control how strongly the coefficients respond to observed saturation and the effective allocation is determined by the current reward state. Fixing a positive γ\gamma does not fix the coefficient ratios across training batches: γ\gamma strengthens the response to relative headroom, rather than assigning a fixed correctness weight.

A practical starting point is γ\gamma=0.75 when correctness is prioritized and length or format is easy to learn. In that same regime, γ\gamma=0.25 or 0.5 retains more emphasis on auxiliary performance. The choice should follow held-out correctness together with actual auxiliary requirements, such as exceedance or format-compliance rates. Task difficulty alone is not the criterion: γ\gamma strengthens the response to relative headroom rather than assigning a fixed correctness weight.

Table 6 provides three-seed sensitivity evidence: correctness means are 0.476, 0.485, 0.533, and 0.540 for γ\gamma=0.25, 0.5, 0.75, and 1.0, respectively, versus 0.448 for GDPO. Every tested positive exponent improves mean correctness over GDPO in this setting. Moving from γ\gamma=0.75 to 1.0 increases correctness from 0.533 to 0.540 while reducing length reward from 0.923 to 0.899, illustrating why the preferred operating point depends on the desired tradeoff.

AuthorsOpenReview ↗

Part 2: Statistical coverage, and reward bounds

W2 / W3 / Q2.1

Can the principal mathematical and coding gains be confirmed across additional seeds with format compliance and all auxiliary rewards reported?

Due to the lmited compute, we probably not able to provide additional experiments over different seeds. The central correctness–length result has independent-training support in our paper. Table 6 reports three independent runs per configuration (Appendix B.3): SA-MRPO (γ\gamma=0.75) reaches 0.533 ± 0.008 correctness, versus 0.448 ± 0.005 for GDPO and 0.510 ± 0.006 for SAW, with similar SA-MRPO/SAW length rewards (0.923/0.927). Table 4 reports higher SA-MRPO means than SAW on all four external benchmarks, averaging 49.9 ± 2.1% versus 47.4 ± 1.7% at 856 versus 855 tokens (two versus three seeds). The other mathematical and coding tables assess cross-setting coverage, not additional training-seed replication.

Joint auxiliary reporting includes EXCEED in Table 1, length reward in Table 6, and PASS/BUG for every coding benchmark in Table 3. CodeContests and Minerva improve without worsening the reported auxiliary metric (W1 / Q1.1). Table 1's exceedance rates are at most 1.7%, although they rise in 10 of 15 comparisons; Table 3 also includes TACO's negative result. Table 1 lacks a separate format metric; our auxiliary evidence concerns measured length and executability.

W3 / Q2.2

Can GRPO be rerun with identical objectives and KL settings?

GDPO already provides a direct ablation of the proposed factor: it is the γ\gamma=0 case under the same base weights and final normalization (section 4.2; Appendix A.2). The adaptive GDPO/SAW/SA-MRPO comparisons share correctness and length objectives and use no KL (Appendix B). In coding, section 5.4 specifies identical data, rewards, and optimization hyperparameters, changing only aggregate-advantage construction. Thus, the core comparisons do not depend on the GRPO row. Its extra format reward and reward-level KL are disclosed in Appendix B; we use it as a reference, without claiming a matched-objective GRPO rerun.

W4 / Q3.1

How sensitive is performance to conservative or unattainable reward bounds?

We added a diagnostic that changes the upper reward bound used to compute the length objective's saturation from 1 to 1.25 while leaving its reward function unchanged. Since rewards still cannot exceed 1, at least 20% of the nominal normalized range is now unattainable:

SettingDeepScaleR correctnessLength rewardaccuracyMean tokens
GDPO reference0.4480.99742.6 ± 1.0487
SA-MRPO, nominal bound0.5330.92349.9 ± 2.1856
SA-MRPO, upper bound 1.250.4690.99442.11 ± 1.26578
Bound-calibration variant using the batch maximum0.5170.92648.46 ± 0.26821

The batch-max calibration variant computes each objective’s saturation using the smaller of its nominal reward upper bound and the maximum reward observed for that objective in the current training batch, while leaving the reward function unchanged.

The two new configurations each use two training and external-evaluation seeds; reference counts are as stated above. The enlarged bound preserves excess weight on the length objective and substantially reduces the accuracy benefit. In this diagnostic, the calibration variant recovers much of the nominal-bound performance. Generation costs also change, so the table reports both accuracy and tokens.

W4 / Q3.2

Would the conclusions change when some objectives have persistent headroom but little achievable improvement?

Persistent, unlearnable headroom can weaken the benefit, as the bound perturbation above demonstrates. SA-MRPO uses headroom as an observable allocation signal, not an estimate of how much improvement further optimization can achieve; section 4 already states this distinction.

The submitted corruption experiment provides a separate stress test: with 30% length-reward corruption, correctness falls by 2.1 percentage points for SA-MRPO with γ\gamma=0.75, compared with 5.9 for GDPO, 10.2 for DVAO, and 2.3 for SAW. Each corrupted point is a single run compared with its same-seed clean run, so it describes behavior under that perturbation rather than general noise robustness.

Reward-range calibration can be tested and partly corrected, but identifying whether every remaining error is learnable requires information beyond batch reward extrema. For example, the same batch may contain unsolvable prompts and other prompts attaining reward 1; the calibration variant cannot resolve that case. We will present it as a diagnostic extension. The conclusions remain supported for the tested settings, including the matched-length and auxiliary-matched examples above, without assuming that every unrealized reward increment is learnable.

We hope the added static experiment, and bound-calibration diagnostic strengthen your assessment of the practical contribution.

Dear Reviewer eTTm,

Thank you for your feedback and suggestions. We fully understand you are busy with other commitments and we appreciate the time and effort you have devoted to evaluating our work. Your feedback has helped us strengthen the manuscript through additional experiments, theoretical analysis, and clarifications.

With the discussion period approaching its conclusion, we would welcome any remaining questions or concerns regarding our responses and revisions. We would be happy to address them.

Thank you again for your consideration. Best, Authors

Reviewer_67HKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SA-MRPO, which shifts reinforcement learning toward reward objectives with more room to improve by downweighting those already near their maximum. Its main contributions are a simple adaptive weighting rule and a theoretical condition showing when it changes which responses training favors. Experiments show improved math accuracy in 12/15 comparisons, adaptive reasoning gains averaging 3.8 percentage points, and coding gains on 3/4 benchmarks. These gains sometimes cost longer responses or more execution errors, and most headline results use a single training run.

Critical strengths

A simple, well-motivated extension to multi-reward optimization. SA-MRPO explicitly accounts for reward saturation when combining independently normalized advantages. It integrates directly into existing GRPO training, and the sign-reversal analysis shows that it can meaningfully change training preferences. This makes the contribution both practical and analytically interpretable.

Empirical support beyond a single task, including a useful matched-cost comparison. Improvements span mathematical reasoning, adaptive reasoning, and coding. Particularly consequential is the multi-seed adaptive comparison: SA-MRPO achieves 49.9% average accuracy versus SAW’s 47.4% at nearly identical response length. This provides evidence that its benefits extend beyond simply allowing more generation.

Critical weaknesses

The evidence for reliable superiority remains limited. Most headline math and coding comparisons use a single training run, and gains often accompany longer responses or worse auxiliary metrics. The multi-seed comparison against SAW at similar response length is encouraging, but does not establish consistent superiority across settings. This limits confidence in the generality of the empirical claims.

Reward headroom is not necessarily achievable learning progress. The method treats distance from a prescribed reward ceiling as a reason to increase an objective’s relative weight, even when that gap reflects noise or unattainable performance. The corruption experiments examine this concern only narrowly, while the theory establishes an advantage-sign threshold rather than an optimization benefit. This leaves the central weighting assumption insufficiently validated.

Questions for authors
  1. Clarification on reliability: How were the saturation exponent and reported configurations selected? Do the improvements over GDPO and SAW persist across training seeds and nearby exponent values? This would help assess whether the gains are robust or sensitive to configuration selection.

  2. Clarification on reward tradeoffs: Beyond the matched-length comparison with SAW, what evidence shows that SA-MRPO improves correctness at comparable generation cost or auxiliary-reward compliance? This would clarify how much of the reported benefit comes from accepting longer responses or more execution errors.

  3. Clarification on the central assumption: What evidence distinguishes reward headroom that is learnable from headroom caused by noise or an unattainable reward ceiling? In particular, how sensitive are the results to the prescribed bounds? This bears directly on when saturation is a reliable basis for allocating optimization effort.

3 comments
AuthorsOpenReview ↗

Part 1: Cost, reliability, and static-weight trade-offs

Thank you for recognizing both the interpretability of the method and the comparison at similar generation cost. We address the weaknesses together with their corresponding questions.

W1 / Q1

How were the saturation exponent and reported configurations selected?

The role of γ\gamma is to control how strongly the coefficients respond to observed saturation and the effective allocation is determined by the current reward state. Fixing a positive γ\gamma does not fix the coefficient ratios across training batches: γ\gamma strengthens the response to relative headroom, rather than assigning a fixed correctness weight.

A practical starting point is γ\gamma=0.75 when correctness is prioritized and length or format is easily to learn. In that same regime, γ\gamma=0.25 or 0.5 retains more emphasis on auxiliary performance. The choice should follow held-out correctness together with actual auxiliary requirements, such as exceedance or format-compliance rates. Task difficulty alone is not the criterion.

Do the improvements over GDPO and SAW persist across training seeds and nearby exponent values?

Table 6 in our paper reports three training seeds for every configuration in the adaptive setting:

Method / γ\gammaDeepScaleR correctnessMean length reward
GDPO / 00.448 ± 0.0050.997
SA-MRPO / 0.250.476 ± 0.0040.986
SA-MRPO / 0.50.485 ± 0.0150.966
SA-MRPO / 0.750.533 ± 0.0080.923
SA-MRPO / 1.00.540 ± 0.0080.899
SAW0.510 ± 0.0060.927

Thus, the higher mean correctness relative to GDPO occurs at every tested positive exponent, rather than at an isolated γ\gamma. Both γ\gamma=0.75 and γ\gamma=1.0 also exceed SAW's mean correctness, with different auxiliary tradeoffs. Additional three-seed external evaluations give 43.63 ± 0.99% at 694 tokens for γ\gamma=0.5 and 49.07 ± 2.00% at 928 tokens for γ\gamma=1.0. The latter remains above SAW's mean accuracy. This supports a useful neighboring range in this setting, without implying uniform superiority across all tasks or γ\gamma values.

W1 / Q2

Beyond the matched-length comparison with SAW, what evidence shows that SA-MRPO improves correctness at comparable generation cost or auxiliary-reward compliance?

In our Table 4, SA-MRPO achieves 49.9 ± 2.1% average accuracy versus 47.4 ± 1.7% for SAW at nearly identical mean response lengths: 856 versus 855 tokens. These are the submitted external results, using two SA-MRPO seeds and three SAW seeds. There is also evidence in a different domain: on CodeContests, test-case pass rate increases from 19.2% to 20.6%, while the reported bug rate remains 15.5% for both methods. In the 3B three-objective mathematical setting, Minerva accuracy increases from 16.9% to 18.1% with the same reported exceedance rate of 0.1%. These comparisons support gains at similar generation length or unchanged reported auxiliary metrics; they do not require claiming that every reported gain is cost-free.

We also evaluated smaller fixed GDPO length weights, keeping the correctness weight at 1. External means and sample standard deviations were recomputed from the per-run evaluation records:

Methodaccuracy %Mean tokensExternal seeds
GDPO, w_len=142.6 ± 1.04873
GDPO, w_len=0.349.26 ± 0.677423
SA-MRPO, γ\gamma=0.7549.9 ± 2.18562

Reducing the fixed length weight recovers a substantial part of the gap to equal-weight GDPO. At the reported mean accuracy–token pairs, GDPO with w_len=0.3 is shorter but less accurate than SA-MRPO. This supports competitive observed tradeoffs.

AuthorsOpenReview ↗

Part 2/2: Headroom, bound sensitivity

W2 / Q3

SA-MRPO uses reward headroom as an observable allocation signal. It does not estimate the amount of improvement that further optimization can achieve. This distinction is stated in our section 4, and we will make it more prominent. The positive evidence is that this signal produces useful allocations in the tested settings, including the matched-generation-length comparison above; it is not a claim that every unrealized reward increment is learnable.

The submitted corruption experiment provides one direct stress test. With 30% length-reward corruption, correctness falls by 2.1 percentage points for SA-MRPO with γ\gamma=0.75, compared with 5.9 for GDPO and 10.2 for DVAO; SAW loses 2.3 points. Each corrupted point is a single run compared with its same-seed clean run, so this establishes behavior under that perturbation rather than general noise robustness.

We additionally tested sensitivity to a miscalibrated upper bound. We changed the upper reward bound used to compute the length objective's saturation from 1 to 1.25 while leaving its reward function unchanged. Since rewards still cannot exceed 1, at least 20% of the nominal normalized range is now unattainable. This isolates a persistent headroom component introduced by bound misspecification:

SettingDeepScaleR correctnessLength rewardaccuracy %Mean tokens
GDPO reference0.4480.99742.6 ± 1.0487
SA-MRPO, nominal bound0.5330.92349.9 ± 2.1856
SA-MRPO, upper bound 1.250.4690.99442.11 ± 1.26578
Bound-calibration variant using the batch maximum0.5170.92648.46 ± 0.26821

The batch-max calibration variant computes each objective’s saturation using the smaller of its nominal reward upper bound and the maximum reward observed for that objective in the current training batch, while leaving the reward function unchanged.The two new configurations each use two training and external-evaluation seeds with reference counts are as stated above. The enlarged bound preserves excess weight on the length objective and substantially reduces the accuracy benefit. In this diagnostic, a batch-max calibration variant recovers much of the nominal-bound performance. For a calibrated objective, the variant computes saturation using the smaller of the nominal upper bound and the observed batch maximum. Generation costs also change, so the table reports both accuracy and tokens.

This result clarifies an actionable distinction: calibration of a reward range can be tested and partly corrected, while determining whether every remaining error is learnable requires information beyond batch reward extrema. For example, a batch can contain both unsolvable prompts and other prompts attaining reward 1. The calibration variant does not resolve that case. We will make this limitation explicit, consistent with section 4's existing distinction between nominal and optimizable headroom, and present the calibration result as a diagnostic extension.

These additions retain the positive matched-length and auxiliary-matched evidence, add the requested static experiment, and make the limits of the statistical and mechanistic claims explicit. We hope they strengthen your assessment of the practical contribution.

Dear Reviewer 67HK,

Thank you for your feedback and suggestions. We fully understand you are busy with other commitments and we appreciate the time and effort you have devoted to evaluating our work. Your feedback has helped us strengthen the manuscript through additional experiments, theoretical analysis, and clarifications.

With the discussion period approaching its conclusion, we would welcome any remaining questions or concerns regarding our responses and revisions. We would be happy to address them.

Thank you again for your consideration. Best, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal

We thank Reviewers 67HK, eTTm and ipbm for their careful reading and constructive questions. We thank the Area Chair Ey5W for handling our submission and the careful reminder. The reviewers’ concerns primarily involve the following points.

(i) Can GDPO with a smaller fixed length weight reach the same trade-off? (ipbm, eTTm, 67HK)

We ran the requested experiment in the controlled adaptive setting: GDPO with the length weight reduced to 0.3 (three seeds), and a two-stage weight schedule (one run). The fixed weight reaches 49.26 ± 0.67% external accuracy at 742 tokens, against 49.9 ± 2.1% at 856 tokens for SA-MRPO and 42.6 ± 1.0% for equal-weight GDPO. We report this openly: a well-chosen fixed weight is competitive in this two-objective setting. Our main contribution sits on the automatically and adaptively weight allocation instead of hand-tuned weight allocation. The weight 0.3 and the schedule setting were both chosen after observing the SA-MRPO result and trajectory. SA-MRPO reaches this operating point from equal base weights, with the allocation computed from the reward state during training.

(ii) How is γ\gamma chosen, and are the gains stable across seeds and nearby values? (ipbm, 67HK, eTTm)

Table 6 already reports a three-seed sweep: every tested γ\gamma > 0 improves correctness over GDPO (0.476 / 0.485 / 0.533 / 0.540 for γ\gamma = 0.25 / 0.5 / 0.75 / 1.0, against 0.448). We added additional three-seed external evaluations for γ\gamma = 0.5 and 1.0 (43.63 ± 0.99% and 49.07 ± 2.00%) and a practical guide for choosing γ\gamma.

(iii) Is reward headroom a reliable signal when it is noisy or unattainable? (67HK, eTTm)

We thank the reviewer again for this valuable question. The paper already tests noisy rewards: with 30% of the length rewards corrupted, SA-MRPO loses 2.1 points of correctness, against 5.9 for GDPO and 10.2 for DVAO. For unattainable headroom we added the additional experiment that the reviewers asked for, setting the upper bound of the length objective’s to 1.25 so that 20% of the nominal range can never be reached (two seeds). A simple calibration that uses the smaller of the nominal bound and the batch maximum keeps SA-MRPO at 48.46 ± 0.26% external accuracy under this misspecification, close to the 49.9% obtained with the correct bound and well above the 42.11 ± 1.26% without calibration. The calibration is identical to SA-MRPO whenever some response in the batch attains the nominal maximum, which is the case for the rule-based rewards in the paper. We will add this experiment and state the scope of the headroom signal explicitly.

(iv) Statistical coverage, auxiliary metrics and the GRPO baseline (67HK, eTTm)

The central claim is already supported by multi-seed evidence in our origin paper. In the controlled adaptive setting, every configuration is trained with three seeds (Table 6): SA-MRPO reaches 0.533 ± 0.008 correctness, against 0.448 ± 0.005 for GDPO and 0.510 ± 0.006 for SAW, and the improvement over GDPO holds at every tested γ\gamma > 0. On four external benchmarks (Table 4), SA-MRPO has the highest average accuracy among all methods compared and is higher than SAW on every benchmark at the same response length (856 vs. 855 tokens). The gains also appear across settings: 12 of 15 comparisons in mathematical reasoning and three of four coding benchmarks. The comparison with GDPO isolates the proposed factor exactly, since GDPO is the γ\gamma = 0 case of our method with the same weights and normalization, so the conclusions do not rely on the GRPO row. The remaining mathematical and coding tables use one training run per configuration. We could not add seeds for them or rerun GRPO with matched settings within the compute available during the discussion period. The revision will add these seeds, report format compliance, and present GRPO as a reference row.

We thank the Area Chair and the reviewers again for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.