- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies GRPO-style training with multiple reward objectives, such as correctness, length, and format. It argues that summing rewards with fixed weights loses information and keeps optimizing objectives that are already nearly saturated. The proposed SA-MRPO standardizes each objective within the group, as GDPO does, and then scales each objective's weight by (1 − s)^γ, where s is the batch-level saturation of that objective relative to its reward bounds. With γ = 0 it reduces to GDPO, and the paper derives the value of γ at which a rollout's aggregate advantage changes sign. Experiments cover mathematical reasoning with two or three objectives, adaptive reasoning with a length reward, and code generation, mainly against GDPO and, for adaptive reasoning, several recent multi-reward methods.
Critical strengths
-
A simple and well-motivated change. The problem is clearly illustrated in Figure 1: under fixed weights, a nearly saturated objective such as format can still dominate the ranking of rollouts. The fix is a single multiplicative factor on top of GDPO, is easy to implement, and leaves the policy update unchanged.
-
Broad evaluation with relevant baselines. The method is tested on math, adaptive reasoning, and code. In the adaptive reasoning setting, it is compared with five recent multi-reward methods under several seeds and achieves the best average accuracy (Table 4). The paper also reports the accompanying costs, such as longer responses and higher exceed rates, instead of only accuracy.
Critical weaknesses
-
The gains are not separated from simply down-weighting the length objective. In most experiments, the objective that saturates first is length or format, so SA-MRPO mainly shifts weight from length toward correctness. The results follow this pattern. In Table 2, accuracy rises by 3.8 points while the mean response length grows from 333 to 459 tokens. In Table 1, the exceed rate is higher than GDPO's in 10 of 15 comparisons. Section 5.5 describes γ as trading length reward for correctness. Table 4 compares SA-MRPO with SAW and GRPO at similar lengths, but neither of them is GDPO with a smaller fixed weight on length. That comparison is the direct test of whether the saturation-dependent schedule matters beyond a different static allocation.
-
The allocation is "adaptive" in name, but in practice it is set by hand. How much weight moves between objectives depends on two manual choices: the exponent γ and the reward bounds used to compute saturation. Both change across experiments. γ is 0.25 in the 3B math setting and 0.75 in adaptive reasoning, and the best value differs again in the separate 3B study (Section 5.5). The length bounds are (0, 4000) in Table 1 and (1024, 2048) in Table 4. Section 5.5 also shows that γ simply moves the model along the correctness–length tradeoff. The paper gives no rule for setting either choice on a new task, which makes the method's behavior hard to predict outside the reported configurations.
Questions for authors
-
How does GDPO perform with a smaller fixed weight on the length objective, chosen to reach a response length similar to SA-MRPO's? A sweep over a few fixed weights would show whether the saturation-aware schedule beats every static allocation.
-
How should γ be chosen for a new task? The current experiments use different values, and the best one varies across settings.
-
Could the paper show how the effective weights w̃_k change during training? This would show whether the reweighting stays dynamic or settles early into a fixed ratio.
5 comments
Part 2: Exponent choice and effective-weight trajectories
W2 / Q2
How should be chosen for a new task?
Our Table 6 provides a three-seed sensitivity study: correctness means are 0.476, 0.485, 0.533, and 0.540 for =0.25, 0.5, 0.75, and 1.0, respectively, versus 0.448 for GDPO. Every tested positive exponent improves mean correctness over GDPO in this setting. As discussed in section 5.5, moving from =0.75 to 1.0 increases correctness from 0.533 to 0.540 while reducing length reward from 0.923 to 0.899. These results support choosing the operating point according to the desired tradeoff.
A practical starting point is =0.75 when correctness is prioritized and length or format is easy to learn. In that same regime, =0.25 or 0.5 retains more emphasis on auxiliary performance. Keep the base preference weights fixed and compare held-out correctness together with actual auxiliary requirements, such as exceedance or format-compliance rates. The preferred operating point can differ across tasks because their reward states and requirements differ; task difficulty alone is not the criterion.
controls response strength; the current reward state determines the effective allocation. We respectfully disagree with W2's characterization that the allocation is adaptive only in name because is chosen by hand. Selecting the sensitivity of a state-dependent rule does not prescribe its resulting objective coefficients. At a fixed positive , the effective coefficient ratios change as the relative observed headrooms change; =0 recovers GDPO. Increasing strengthens the relative emphasis on objectives with greater remaining headroom. It discounts objective-advantage contributions according to saturation, without changing the reward functions or prescribing a transition step.
The reward bounds and token thresholds cited in W2 are also different quantities. Appendix B.2 specifies reward-value bounds [0,1] for the saturation calculation. The ranges (0,4000) and (1024,2048) are token thresholds defining the length reward function, as listed in Table 5. Within each comparison, the competing methods share that reward function. These thresholds affect the accuracy–length tradeoff, but they are not different saturation upper bounds selected specifically for SA-MRPO.
W2 / Q3
Could the paper show how the effective weights change during training?
Our Figure 5 reports these trajectories over three training seeds, together with reward means and mean response length. Appendix C.1 explains the changing coefficient ratio: the length coefficient declines as the length reward approaches its upper bound, while the correctness coefficient remains larger later in training. The figure shows an early reallocation phase followed by a more stable regime.
This directly addresses whether the coefficients evolve or settle: both behaviors occur at different training stages. Stabilization under a more stable observed reward state is consistent with an adaptive rule; it does not imply that the coefficients were set by hand. The same fixed continues to map current saturation statistics to effective weights throughout training. State dependence does not require the weights to keep changing substantially after the observed state stabilizes.
The two-stage diagnostic in the table under W1 / Q1 uses a schedule informed by the observed SA-MRPO trajectory: the length-to-correctness weight ratio changes from 0.55 to 0.25 at step 30. In one run, it reaches DeepScaleR correctness 0.520 and length reward 0.955, with external accuracy 49.17% at 715 tokens. This differs in auxiliary performance and generation length from SA-MRPO's 0.533 / 0.923 and 49.9% / 856 tokens, so it does not establish equivalence or a matched-cost advantage for dynamic weighting. The schedule is prescribed from an observed trajectory, whereas SA-MRPO obtains its coefficients from current reward statistics.
The submitted paper therefore already contains a near-equal-length comparison, a multi-seed exponent study, and measured weight trajectories. Together with the additional static experiment, these results clarify both the practical contribution and the state-dependent mechanism. We would appreciate reconsideration of the assessment in light of this evidence.
Response to authors
The response resolves some of my concerns. I mixed up the saturation bounds with the token thresholds of the length reward, and I withdraw that point. Figure 5 already answers Q3, and I agree that the rule is state-dependent.
My main concern remains. With a fixed length weight of 0.3, GDPO reaches 49.26%, close to SA-MRPO's 49.9%, and it uses fewer tokens. This gap is within seed variation, and the two-stage schedule performs about the same. Like the fixed weight, γ needs to be tuned for each task. So it is still unclear how much the saturation-aware weighting adds beyond a well-chosen fixed weight. This weighting is the only change SA-MRPO makes to GDPO, so I keep my score at 2.
Part 1: Static-weight trade-off
Thank you for recognizing the simplicity of the method and the evidence across tasks. We address W1 and W2 together with their corresponding questions.
W1 / Q1
How does GDPO perform with a smaller fixed weight on the length objective, chosen to reach a response length similar to SA-MRPO’s?
The submitted paper already provides evidence that the accuracy gain is not solely explained by longer responses. In Table 4, SA-MRPO achieves 49.9 ± 2.1% versus SAW's 47.4 ± 1.7% at 856 versus 855 tokens, using two and three training seeds, respectively. SA-MRPO's mean accuracy is higher on each of the four external mathematical benchmarks. This addresses the generation-cost concern in W1; the smaller-fixed-weight GDPO alternative is examined below.
We evaluated GDPO with a fixed length weight of 0.3 while keeping the correctness weight at 1, using three training and external-evaluation seeds. The table also includes the two-stage diagnostic discussed under W2 / Q3:
Reducing the fixed length weight recovers a substantial part of the gap to equal-weight GDPO. The 0.3 length weight yields shorter responses and lower mean accuracy than SA-MRPO, with a higher length reward, so the two occupy different tradeoff points. Together, these results support competitive observed tradeoffs; the 742-token is not an exact length match or a complete static frontier. The near-equal-length SAW comparison and the smaller-weight GDPO performance answer complementary questions.