- Rating
- 1
- Confidence
- 2(−1)
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
Summary: This paper identifies the reward collapse issue in group-relative methods for LLM text-to-text regression, proposing MM-GRPO to normalize advantage with reward range and R-GRPO to design advantage with relative ranking.
Contributions: The reported contributions of this paper are:
- it identifies the reward collapse issue;
- it proposes MM-GRPO;
- it proposes R-GRPO.
Validations: The paper examines the within group reward average and deviation before and after training; it evaluates the proposed algorithms and baselines on two real world regression tasks.
Critical strengths
Transparent reporting of experimental details. The paper transparently documents its training configurations, dataset construction prompts, and hyperparameter choices.
Critical weaknesses
- Unclear motivation for MM-GRPO and limited contribution of R-GRPO.
- Both standard GRPO, Dr. GRPO and MM-GRPO multiplies the centered reward by a group-dependent weight . The paper criticizes Dr. GRPO () for allowing low variance group to have diminished weight, while for a non-diminishing group weight of GRPO (), criticizes with a confusing reason ("normalize away the extent of reward concentration"). The criteria for a decent group weight, e.g. why normalize with reward range rather than standard deviation is better (in specific why group B deserves higher advantage magnitude than group A), remains unclear by the motivation of tackling reward collapse.
- R-GRPO is not original as the exact same algorithm was already in GOPO. Although the paper cites GOPO, applying it to a slightly different task is not a justified contribution.
- Uncontrolled baseline comparisons. The comparison with baselines is not controlled, leaving the empirical verification questionable:
- For Qwen3-1.7B experiments, MM-GRPO and R-GRPO advantages are multiplied by a constant scaling factor carefully tuned on the validation set (Table 3), while the baselines are not.
- A KL penalty is enabled for Standard GRPO and Batch GRPO, but disabled for MM-GRPO, R-GRPO, and Dr.GRPO (Section 5.3), directly confounding the comparison between choices of group weight.
Questions for authors
- Can the authors clarify that in principle, what properties should a good group weight satisfy in the face of reward collapse, and why normalizing by reward range instead of reward standard deviation is theoretically or intuitively preferrable?
- Can the authors provide a strictly controlled comparison with at least GRPO, where either both use or both not use a tuned global weight () and KL penalty?
4 comments
KL Penalty and Scaling Factor Ablations
Q2/ W2: KL Penalty and Scaling Factor Ablations
We have added two new studies that directly address these concerns and will be included in the revised manuscript.
Advantage Estimation Ablation (KL penalty)
We have conducted an ablation study to isolate the effects of changing the advantage estimator while keeping everything else constant, including the omission of the KL regularization penalty. Specifically, we evaluate the GRPO and Batch GRPO advantage estimators using the same Dr.GRPO policy-optimization recipe as our methods. As shown in the tables below, MM-GRPO achieves the overall best performance, with the highest average for both model sizes. The results further demonstrate the effectiveness of the MM-GRPO advantage formulation. We will be including this analysis in the Results section of the revised manuscript:
Qwen3-4B
| Method | Nutrition ↑ | Nutrition MAE ↓ | Nutrition RMSE ↓ | Price ↑ | Price MAE ↓ | Price RMSE ↓ | Avg. ↑ |
|---|---|---|---|---|---|---|---|
| Dr.GRPO | 0.694 | 9.951 | 19.185 | 0.586 | 11.342 | 18.348 | 0.640 |
| Dr.GRPO + | 0.704 | 10.122 | 18.895 | 0.455 | 11.301 | 21.048 | 0.579 |
| Dr.GRPO + | 0.739 | 9.771 | 17.718 | 0.621 | 10.942 | 17.542 | 0.680 |
| R-GRPO | 0.732 | 9.750 | 17.953 | 0.610 | 11.212 | 17.810 | 0.671 |
| MM-GRPO | 0.756 | 8.978 | 17.140 | 0.621 | 10.902 | 17.542 | 0.689 |
Qwen3-1.7B
| Method | Nutrition ↑ | Nutrition MAE ↓ | Nutrition RMSE ↓ | Price ↑ | Price MAE ↓ | Price RMSE ↓ | Avg. ↑ |
|---|---|---|---|---|---|---|---|
| Dr.GRPO | 0.375 | 17.634 | 27.434 | 0.498 | 12.928 | 20.207 | 0.437 |
| Dr.GRPO + | 0.470 | 15.344 | 25.260 | 0.469 | 13.166 | 20.781 | 0.469 |
| Dr.GRPO + | 0.491 | 15.020 | 24.761 | 0.491 | 13.044 | 20.345 | 0.491 |
| R-GRPO | 0.490 | 14.367 | 24.794 | 0.536 | 12.515 | 19.411 | 0.513 |
| MM-GRPO | 0.608 | 12.690 | 21.719 | 0.591 | 11.716 | 18.227 | 0.600 |
Tuning Scaling Factor for GRPO
We have conducted the same scaling factor validation-based tuning procedure for GRPO for Qwen3-1.7B and found that the original GRPO scale of 1 achieved the best validation performance. Therefore, the reported GRPO result already corresponds to its best scale among the evaluated values. For Qwen3-4B, no global advantage scaling was applied to any method. We will expand Table 3 in the Appendix to include the following GRPO tuned results:
| Scale | Nutrition Estimation MAE ↓ | Product Price Prediction MAE ↓ |
|---|---|---|
| 1 | 12.904 | 13.257 |
| 4 | 13.148 | 14.167 |
| 8 | 13.679 | 13.483 |
| 16 | 14.852 | 14.258 |
Follow up on Author's Response to Q1
I thank the author for the response.
I am still confused on what a good advantage estimator should be according to the authors. Could the authors further clarify the above 4 criteria with a few examples (e.g. reward distribution within groups)?
In specific, why this paper is confusing to me is that the authors are not clearly showing which of the following issues they are really tackling, and why the method they propose can tackle it, in specific better than GRPO:
- different problems have different reward range and relative difference
- as policy evolves during RL, the relative difference can diminish globally
Also, can the authors explain why extending RL when the reward signal fails to distinguish can be worthy of the cost? Won't this risk reward hacking as reward proxy (either model based or not) might not be accurate at distinguishing tiny difference?
I keep my score but reduce confidence until the above key confusion is clarified.
We thank the reviewer for the follow-up.
Why normalize with range (MM-GRPO) instead of standard deviation (GRPO):
Of the two issues you listed, our work addresses the second: reward differences within rollout groups diminish as the policy evolves. MM-GRPO does not attempt to prevent the diminishing of reward differences. Instead range normalization produces advantages that remain informative as the reward range contracts. This allows the distribution of rewards within that range to determine the group’s advantage magnitude (how much that group will affect the policy update). In contrast, GRPO does not have this property because standard-deviation normalization equalizes advantage magnitude across all groups as described below.
As explained in the “Adaptive Group Scaling” section (Section 4.1) and illustrated in Figure 1(c), the advantage magnitude of a group is
For GRPO, standard-deviation normalization gives
\sqrt{\frac{G-1}{G}} \frac{\sigma_g}{\sigma_g+\epsilon} \approx \sqrt{\frac{G-1}{G}}.$$ With $G=64$, this is approximately $0.992$ for every group with $\sigma_g \gg \epsilon$, regardless of whether its reward range is established by one outlier value or supported by differences across many rollouts (well-dispersed). For MM-GRPO, range normalization instead gives $$M_g^{\mathrm{MM}}= \alpha \sqrt{\frac{G-1}{G}} \frac{\sigma_g} {r_{\max,g}-r_{\min,g}+\epsilon} \approx 0.992\ \alpha c_g, \qquad c_g = \frac{\sigma_g} {r_{\max,g}-r_{\min,g}}.$$ In MM-GRPO each group’s relative reward dispersion directly determines its advantage scale through the factor $c_g$. Groups whose rewards are broadly distributed across their observed range have larger $c_g$ and receive greater advantage magnitude, whereas groups in which most rewards are concentrated and only a few outlier values establish the range have smaller $c_g$. This adaptive group weighting is the inductive bias introduced by MM-GRPO and evaluated in our experiments. For a **concrete example**, consider two groups of $G=64$ rollouts whose rewards have the same range: - **Group A:** 63 rollouts receive reward $0$ and one rollout receives reward $1$. - **Group B:** 32 rollouts receive reward $0$ and 32 rollouts receive reward $1$. In Group A, the range is established by a single isolated reward, giving $$c_A=\frac{\sigma_A}{r_{\max,A}-r_{\min,A}}=0.125.$$ In Group B, the reward difference is supported by half of the rollouts on each side, giving $$c_B=\frac{\sigma_B}{r_{\max,B}-r_{\min,B}}\approx0.504.$$ GRPO assigns both groups approximately the same RMS advantage magnitude: $$M_A^{\mathrm{GRPO}} \approx M_B^{\mathrm{GRPO}} \approx \sqrt{\frac{63}{64}} = 0.992.$$ With $\alpha=1$, MM-GRPO instead assigns $$M_A^{\mathrm{MM}} \approx 0.992(0.125) = 0.124, \qquad M_B^{\mathrm{MM}} \approx 0.992(0.504) = 0.500.$$ Thus, GRPO gives the same advantage scale to a group whose range is produced by one isolated reward (group A) and a group whose reward separation is supported by many rollouts (group B). MM-GRPO gives greater weight to the latter. This example demonstrates the inductive bias introduced by range normalization; Figure 1(c) provides two further visual examples of the same effect. Empirically we observe that this inductive bias leads to lower mean absolute error (MAE) despite having a greater proportion of collapsed groups in comparison to other methods (see table in the first response block). This suggests that weighting well-dispersed groups higher improves the overall policy and thereby achieves better performance, including on prompts whose rewards have collapsed. The calculation of advantage magnitude of MM-GRPO is included in the manuscript, and we will add the above calculation for GRPO to make the distinction between range and standard deviation normalization more clear. **Will this cause reward hacking?** Do you mean that continuing RL after some groups collapse could force the model to exploit unreliable reward differences rather than learn meaningful behavior? We observe that collapsed groups remain inaccurate (23.7–62.0% error), so this collapse does not imply that further learning is unnecessary. Moreover, our reward is deterministic negative absolute error computed against a known ground-truth target. We evaluate on held-out test data and observe improvements not only in MAE, which defines the reward, but also in $R^2$. This shows that MM-GRPO improves numerical prediction without reward hacking.
Motivations and Reward Collapse
Q1 / W1: Motivations and Reward Collapse
We thank the reviewer for raising these questions. We will clarify the discussion in the manuscript on the following motivations behind the MM-GRPO advantage estimator and further clarify that not every high-dispersion group is intrinsically more valuable. MM-GRPO introduces a specific inductive bias for continuous regression rewards. We seek an estimator that:
Dr.GRPO preserves raw gaps but makes advantage magnitude proportional to the raw reward scale. Standard GRPO removes raw scale but forces every nonconstant group to have approximately unit advantage variance. MM-GRPO instead fixes the advantage range while allowing its standard deviation to equal approximately
αrmax−rminσ
Consequently, a group whose rewards are broadly distributed across its range receives a larger advantage scale than one in which most rewards are concentrated and only an isolated value establishes the range. We do not claim that this weighting is universally optimal; it is a transparent, testable inductive bias, and the controlled experiments evaluate whether it is useful for regression RL.
We will clarify these points in the revised manuscript in addition to adding the following Results subsection presenting the empirical performance under reward collapse:
" Reward collapse is a natural and potentially desirable consequence of learning when predictions converge toward the ground-truth target. However, reward concentration does not necessarily indicate that a prompt has been solved. As shown in Table (below), collapsed groups retain substantial prediction error, ranging from 23.7% to 36.7% of the ground-truth value for Qwen3-4B and from 32.6% to 62.0% for Qwen3-1.7B. A group is considered collapsed when all of its rollout errors are within 10% of one another for Qwen3-4B or 15% for Qwen3-1.7B. We use the wider threshold for Qwen3-1.7B because its groups contain 64 rather than 8 rollouts, making complete agreement across the group less likely. This analysis shows that collapsed groups are substantially closer to one another than to the ground-truth, demonstrating that reward collapse does not necessarily imply convergence to the correct prediction. Despite having the largest proportion of collapsed groups, MM-GRPO achieves the lowest overall collapsed-group error, with R-GRPO achieving the second lowest average error."
Regarding R-GRPO, we agree that rank-based advantage estimation itself is not algorithmically novel, and we will revise the manuscript to make that more clear. Our primary methodological contribution is MM-GRPO and we use the rank-based estimator as a diagnostic control that removes reward magnitudes entirely, allowing us to test whether maintaining a scale-stable ordinal signal is sufficient under reward concentration for regression RL. Its contribution to this paper is therefore the empirical finding it enables in text-to-text regression. We will revise the manuscript to make this clear and distinguish prior rank-based optimization from our diagnostic use of ranked rewards in text-to-text regression settings.