acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Group-Relative Advantage Estimation in Policy Optimization for Text-to-Text Regression

Abstract

Fine-grained scalar rewards provide rich supervision for reinforcement learning with large language models and naturally support text-to-text regression, where numerical prediction error can directly define the reward. However, as model predictions improve, rewards within rollout groups become increasingly concentrated, a phenomenon we refer to as reward collapse. Existing group-relative advantage estimators either allow the learning signal to shrink with these reward differences or normalize away the extent of reward concentration. We introduce Min-Max GRPO (**MM-GRPO**), which normalizes centered rewards by their within-group range, maintaining a well-scaled learning signal while allowing relative reward diversity to determine each group's contribution to optimization. We additionally introduce Rank-based GRPO (**R-GRPO**), which discards reward magnitudes and constructs advantages from within-group ranks to verify the importance of maintaining a well-spread learning signal throughout training. Across two real-world text-to-text regression benchmarks and two model sizes, MM-GRPO achieves the strongest overall regression performance, while R-GRPO remains competitive despite using only reward ordering. Additional analyses show that MM-GRPO and R-GRPO's improvements extend to long-tail target values and multilingual out-of-distribution inputs.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, large language models, group relative policy optimization, regression

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_RJEFOpenReview ↗
Rating
1
Confidence
2(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Summary: This paper identifies the reward collapse issue in group-relative methods for LLM text-to-text regression, proposing MM-GRPO to normalize advantage with reward range and R-GRPO to design advantage with relative ranking.

Contributions: The reported contributions of this paper are:

  • it identifies the reward collapse issue;
  • it proposes MM-GRPO;
  • it proposes R-GRPO.

Validations: The paper examines the within group reward average and deviation before and after training; it evaluates the proposed algorithms and baselines on two real world regression tasks.

Critical strengths

Transparent reporting of experimental details. The paper transparently documents its training configurations, dataset construction prompts, and hyperparameter choices.

Critical weaknesses
  1. Unclear motivation for MM-GRPO and limited contribution of R-GRPO.
  • Both standard GRPO, Dr. GRPO and MM-GRPO multiplies the centered reward by a group-dependent weight wgw_g. The paper criticizes Dr. GRPO (wg=1w_g = 1) for allowing low variance group to have diminished weight, while for a non-diminishing group weight of GRPO (wg=1/σgw_g = 1 / \sigma_g), criticizes with a confusing reason ("normalize away the extent of reward concentration"). The criteria for a decent group weight, e.g. why normalize with reward range rather than standard deviation is better (in specific why group B deserves higher advantage magnitude than group A), remains unclear by the motivation of tackling reward collapse.
  • R-GRPO is not original as the exact same algorithm was already in GOPO. Although the paper cites GOPO, applying it to a slightly different task is not a justified contribution.
  1. Uncontrolled baseline comparisons. The comparison with baselines is not controlled, leaving the empirical verification questionable:
  • For Qwen3-1.7B experiments, MM-GRPO and R-GRPO advantages are multiplied by a constant scaling factor α\alpha carefully tuned on the validation set (Table 3), while the baselines are not.
  • A KL penalty is enabled for Standard GRPO and Batch GRPO, but disabled for MM-GRPO, R-GRPO, and Dr.GRPO (Section 5.3), directly confounding the comparison between choices of group weight.
Questions for authors
  1. Can the authors clarify that in principle, what properties should a good group weight satisfy in the face of reward collapse, and why normalizing by reward range instead of reward standard deviation is theoretically or intuitively preferrable?
  2. Can the authors provide a strictly controlled comparison with at least GRPO, where either both use or both not use a tuned global weight (α\alpha) and KL penalty?
4 comments
AuthorsOpenReview ↗

Motivations and Reward Collapse

Q1 / W1: Motivations and Reward Collapse

We thank the reviewer for raising these questions. We will clarify the discussion in the manuscript on the following motivations behind the MM-GRPO advantage estimator and further clarify that not every high-dispersion group is intrinsically more valuable. MM-GRPO introduces a specific inductive bias for continuous regression rewards. We seek an estimator that:

  1. is invariant to reward translation and positive rescaling
  2. remains bounded as reward differences contract
  3. preserves the ordering and relative numerical gaps between responses
  4. allows the normalized structure of a group’s rewards to affect its overall advantage scale.

Dr.GRPO preserves raw gaps but makes advantage magnitude proportional to the raw reward scale. Standard GRPO removes raw scale but forces every nonconstant group to have approximately unit advantage variance. MM-GRPO instead fixes the advantage range while allowing its standard deviation to equal approximately

ασrmax⁡−rmin⁡\alpha \frac{\sigma}{r_{\max}-r_{\min}}

Consequently, a group whose rewards are broadly distributed across its range receives a larger advantage scale than one in which most rewards are concentrated and only an isolated value establishes the range. We do not claim that this weighting is universally optimal; it is a transparent, testable inductive bias, and the controlled experiments evaluate whether it is useful for regression RL.

We will clarify these points in the revised manuscript in addition to adding the following Results subsection presenting the empirical performance under reward collapse:

" Reward collapse is a natural and potentially desirable consequence of learning when predictions converge toward the ground-truth target. However, reward concentration does not necessarily indicate that a prompt has been solved. As shown in Table (below), collapsed groups retain substantial prediction error, ranging from 23.7% to 36.7% of the ground-truth value for Qwen3-4B and from 32.6% to 62.0% for Qwen3-1.7B. A group is considered collapsed when all of its rollout errors are within 10% of one another for Qwen3-4B or 15% for Qwen3-1.7B. We use the wider threshold for Qwen3-1.7B because its groups contain 64 rather than 8 rollouts, making complete agreement across the group less likely. This analysis shows that collapsed groups are substantially closer to one another than to the ground-truth, demonstrating that reward collapse does not necessarily imply convergence to the correct prediction. Despite having the largest proportion of collapsed groups, MM-GRPO achieves the lowest overall collapsed-group error, with R-GRPO achieving the second lowest average error."

Method4B Nutrition % Error ↓4B Price % Error ↓1.7B Nutrition % Error ↓1.7B Price % Error ↓Average % Error ↓% of Groups Collapsed
Dr.GRPO36.733.252.240.940.841.9
GRPO27.830.841.532.933.344.8
Batch GRPO27.033.643.862.041.635.5
R-GRPO26.030.132.637.131.543.7
MM-GRPO23.728.236.732.630.351.5

Regarding R-GRPO, we agree that rank-based advantage estimation itself is not algorithmically novel, and we will revise the manuscript to make that more clear. Our primary methodological contribution is MM-GRPO and we use the rank-based estimator as a diagnostic control that removes reward magnitudes entirely, allowing us to test whether maintaining a scale-stable ordinal signal is sufficient under reward concentration for regression RL. Its contribution to this paper is therefore the empirical finding it enables in text-to-text regression. We will revise the manuscript to make this clear and distinguish prior rank-based optimization from our diagnostic use of ranked rewards in text-to-text regression settings.

AuthorsOpenReview ↗

KL Penalty and Scaling Factor Ablations

Q2/ W2: KL Penalty and Scaling Factor Ablations

We have added two new studies that directly address these concerns and will be included in the revised manuscript.

Advantage Estimation Ablation (KL penalty)

We have conducted an ablation study to isolate the effects of changing the advantage estimator while keeping everything else constant, including the omission of the KL regularization penalty. Specifically, we evaluate the GRPO and Batch GRPO advantage estimators using the same Dr.GRPO policy-optimization recipe as our methods. As shown in the tables below, MM-GRPO achieves the overall best performance, with the highest average R2R^2 for both model sizes. The results further demonstrate the effectiveness of the MM-GRPO advantage formulation. We will be including this analysis in the Results section of the revised manuscript:

Qwen3-4B

MethodNutrition R2R^2 ↑Nutrition MAE ↓Nutrition RMSE ↓Price R2R^2 ↑Price MAE ↓Price RMSE ↓Avg. R2R^2 ↑
Dr.GRPO0.6949.95119.1850.58611.34218.3480.640
Dr.GRPO + AGRPOA^{\mathrm{GRPO}}0.70410.12218.8950.45511.30121.0480.579
Dr.GRPO + ABatchA^{\mathrm{Batch}}0.7399.77117.7180.62110.94217.5420.680
R-GRPO0.7329.75017.9530.61011.21217.8100.671
MM-GRPO0.7568.97817.1400.62110.90217.5420.689

Qwen3-1.7B

MethodNutrition R2R^2 ↑Nutrition MAE ↓Nutrition RMSE ↓Price R2R^2 ↑Price MAE ↓Price RMSE ↓Avg. R2R^2 ↑
Dr.GRPO0.37517.63427.4340.49812.92820.2070.437
Dr.GRPO + AGRPOA^{\mathrm{GRPO}}0.47015.34425.2600.46913.16620.7810.469
Dr.GRPO + ABatchA^{\mathrm{Batch}}0.49115.02024.7610.49113.04420.3450.491
R-GRPO0.49014.36724.7940.53612.51519.4110.513
MM-GRPO0.60812.69021.7190.59111.71618.2270.600

Tuning Scaling Factor for GRPO

We have conducted the same scaling factor validation-based tuning procedure for GRPO for Qwen3-1.7B and found that the original GRPO scale of 1 achieved the best validation performance. Therefore, the reported GRPO result already corresponds to its best scale among the evaluated values. For Qwen3-4B, no global advantage scaling was applied to any method. We will expand Table 3 in the Appendix to include the following GRPO tuned results:

ScaleNutrition Estimation MAE ↓Product Price Prediction MAE ↓
112.90413.257
413.14814.167
813.67913.483
1614.85214.258
Reviewer_RJEFOpenReview ↗

Follow up on Author's Response to Q1

I thank the author for the response.

I am still confused on what a good advantage estimator should be according to the authors. Could the authors further clarify the above 4 criteria with a few examples (e.g. reward distribution within groups)?

In specific, why this paper is confusing to me is that the authors are not clearly showing which of the following issues they are really tackling, and why the method they propose can tackle it, in specific better than GRPO:

  • different problems have different reward range and relative difference
  • as policy evolves during RL, the relative difference can diminish globally

Also, can the authors explain why extending RL when the reward signal fails to distinguish can be worthy of the cost? Won't this risk reward hacking as reward proxy (either model based or not) might not be accurate at distinguishing tiny difference?

I keep my score but reduce confidence until the above key confusion is clarified.

We thank the reviewer for the follow-up.

Why normalize with range (MM-GRPO) instead of standard deviation (GRPO):

Of the two issues you listed, our work addresses the second: reward differences within rollout groups diminish as the policy evolves. MM-GRPO does not attempt to prevent the diminishing of reward differences. Instead range normalization produces advantages that remain informative as the reward range contracts. This allows the distribution of rewards within that range to determine the group’s advantage magnitude (how much that group will affect the policy update). In contrast, GRPO does not have this property because standard-deviation normalization equalizes advantage magnitude across all groups as described below.

As explained in the “Adaptive Group Scaling” section (Section 4.1) and illustrated in Figure 1(c), the advantage magnitude of a group is

Mg=1G∑i=1GAg,i2.M_g = \sqrt{\frac{1}{G}\sum_{i=1}^{G} A_{g,i}^{2}}.

For GRPO, standard-deviation normalization gives

\sqrt{\frac{G-1}{G}} \frac{\sigma_g}{\sigma_g+\epsilon} \approx \sqrt{\frac{G-1}{G}}.$$ With $G=64$, this is approximately $0.992$ for every group with $\sigma_g \gg \epsilon$, regardless of whether its reward range is established by one outlier value or supported by differences across many rollouts (well-dispersed). For MM-GRPO, range normalization instead gives $$M_g^{\mathrm{MM}}= \alpha \sqrt{\frac{G-1}{G}} \frac{\sigma_g} {r_{\max,g}-r_{\min,g}+\epsilon} \approx 0.992\ \alpha c_g, \qquad c_g = \frac{\sigma_g} {r_{\max,g}-r_{\min,g}}.$$ In MM-GRPO each group’s relative reward dispersion directly determines its advantage scale through the factor $c_g$. Groups whose rewards are broadly distributed across their observed range have larger $c_g$ and receive greater advantage magnitude, whereas groups in which most rewards are concentrated and only a few outlier values establish the range have smaller $c_g$. This adaptive group weighting is the inductive bias introduced by MM-GRPO and evaluated in our experiments. For a **concrete example**, consider two groups of $G=64$ rollouts whose rewards have the same range: - **Group A:** 63 rollouts receive reward $0$ and one rollout receives reward $1$. - **Group B:** 32 rollouts receive reward $0$ and 32 rollouts receive reward $1$. In Group A, the range is established by a single isolated reward, giving $$c_A=\frac{\sigma_A}{r_{\max,A}-r_{\min,A}}=0.125.$$ In Group B, the reward difference is supported by half of the rollouts on each side, giving $$c_B=\frac{\sigma_B}{r_{\max,B}-r_{\min,B}}\approx0.504.$$ GRPO assigns both groups approximately the same RMS advantage magnitude: $$M_A^{\mathrm{GRPO}} \approx M_B^{\mathrm{GRPO}} \approx \sqrt{\frac{63}{64}} = 0.992.$$ With $\alpha=1$, MM-GRPO instead assigns $$M_A^{\mathrm{MM}} \approx 0.992(0.125) = 0.124, \qquad M_B^{\mathrm{MM}} \approx 0.992(0.504) = 0.500.$$ Thus, GRPO gives the same advantage scale to a group whose range is produced by one isolated reward (group A) and a group whose reward separation is supported by many rollouts (group B). MM-GRPO gives greater weight to the latter. This example demonstrates the inductive bias introduced by range normalization; Figure 1(c) provides two further visual examples of the same effect. Empirically we observe that this inductive bias leads to lower mean absolute error (MAE) despite having a greater proportion of collapsed groups in comparison to other methods (see table in the first response block). This suggests that weighting well-dispersed groups higher improves the overall policy and thereby achieves better performance, including on prompts whose rewards have collapsed. The calculation of advantage magnitude of MM-GRPO is included in the manuscript, and we will add the above calculation for GRPO to make the distinction between range and standard deviation normalization more clear. **Will this cause reward hacking?** Do you mean that continuing RL after some groups collapse could force the model to exploit unreliable reward differences rather than learn meaningful behavior? We observe that collapsed groups remain inaccurate (23.7–62.0% error), so this collapse does not imply that further learning is unnecessary. Moreover, our reward is deterministic negative absolute error computed against a known ground-truth target. We evaluate on held-out test data and observe improvements not only in MAE, which defines the reward, but also in $R^2$. This shows that MM-GRPO improves numerical prediction without reward hacking.
Reviewer_AXvaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits group-relative advantage estimation for text-to-text regression. The authors argue that standard within-group normalization can be problematic when reward ranges, within-group dispersion, and tail behavior vary substantially across inputs. In such settings, normalizing each group independently may distort the relative importance of examples and produce unstable or poorly calibrated policy-gradient signals. The paper proposes two alternatives. MM-GRPO adaptively adjusts the group scale according to the relationship between within-group dispersion and the reward range, while R-GRPO further reduces or removes dependence on absolute reward magnitudes and relies primarily on within-group ranking information. The methods are evaluated in several text regression settings, with a focus on regression accuracy, long-tail performance, and training stability. The results suggest that MM-GRPO provides more stable scaling under varying reward ranges, while R-GRPO remains competitive despite discarding reward magnitude and in some settings outperforms estimators that preserve it.

Critical strengths
  1. The paper raises an important and underexplored issue: group-wise normalization is not necessarily appropriate for text regression tasks with heterogeneous reward scales. Connecting reward range, within-group dispersion, and advantage estimation provides a useful perspective for rethinking GRPO-style optimization.

  2. The paper studies both MM-GRPO, which retains reward-scale information, and R-GRPO, which relies mainly on relative ordering. This design helps separate the contributions of reward magnitude and ranking information. The strong performance of R-GRPO in some settings suggests that relative ordering may contain a substantial portion of the useful policy-optimization signal.

Critical weaknesses
  1. The paper does not yet sufficiently distinguish improvements in regression accuracy from improvements in ranking behavior. Since R-GRPO discards reward magnitude while remaining competitive, the result may indicate that the task or evaluation metric is primarily sensitive to relative ordering rather than showing that absolute reward magnitude is generally unnecessary. More detailed regression, ranking, calibration, and interval-specific error metrics are needed to determine what capability is actually improved.

  2. The scale adjustment in MM-GRPO and the ranking-based design of R-GRPO remain somewhat heuristic. The paper should provide systematic ablations for reward range, within-group standard deviation, group size, tail proportion, and normalization constants, together with comparisons against simpler alternatives such as reward clipping, global normalization, quantile normalization, rank transformation, and other robust scaling methods.

Questions for authors
  1. Can the MM-GRPO scale be derived from an optimization principle, such as minimizing gradient variance, or is it primarily based on empirical observations?

  2. How do MM-GRPO and R-GRPO handle groups with nearly identical rewards, near-zero standard deviation, or extreme outliers? Can these cases lead to exploding gradients or vanishing training signals?

  3. What is the relationship between the R-GRPO objective and the original continuous regression objective? Can the authors report MAE, RMSE, Spearman correlation, long-tail error, and calibration metrics for all methods?

  4. Do the advantages of MM-GRPO and R-GRPO remain stable when group size, reward noise, reward range, and the proportion of long-tail examples are varied?

  5. Have the authors compared against reward clipping, global normalization, quantile normalization, rank transformation, and other robust advantage-normalization methods?

4 comments

W1: Evaluation of Ranked Rewards

We thank the reviewer for raising these questions. We would like to clarify that R-GRPO uses reward ranks only to construct advantages within the sampled rollout group for each input, rather than optimizing the ordering of predicted targets across different inputs.

Our evaluation directly measures numerical regression accuracy using MAE, RMSE, and R² (Table 1), rather than ranking-based metrics. These metrics depend on the absolute numerical deviations between predictions and targets, meaning that improvements cannot be attributed solely to better ordering of predictions. For example, on Qwen3-4B nutrition estimation, R-GRPO improves RMSE from 19.185 (Dr.GRPO) to 17.953 while increasing R² from 0.694 to 0.732.

Furthermore, we have provided complementary analyses beyond aggregate metrics: Appendix A includes prediction-versus-target calibration plots, while Table 2 evaluates MAE on the highest-value 10% of targets. Both analyses show that the improvements extend to numerical prediction accuracy in challenging target regions.

We agree that these results do not establish that reward magnitudes are universally unnecessary. Rather, our findings demonstrate that within-group reward ordering can provide an effective learning signal for improving numerical regression accuracy, even without preserving the precise reward differences. We will further clarify this distinction in the revised manuscript.

W2: Additional Ablation Studies

We follow the standard GRPO training recipe, including PPO-style policy-ratio clipping; this is distinct from scalar reward clipping, which we do not apply. Our baselines cover mean-centered rewards, within-group normalization, batch-level normalization, and rank-based normalization. To further address the reviewer’s concern, we focus our additional experiments on the most directly relevant factors for isolating the contribution of our method: the advantage estimator under an identical optimization recipe and the global advantage-scaling factor. We have added two new studies that will be included in the revised manuscript.

Advantage Estimation Ablation (KL penalty)

We have conducted an ablation study to isolate the effects of changing the advantage estimator while keeping everything else constant, including the omission of the KL regularization penalty. Specifically, we evaluate the GRPO and Batch GRPO advantage estimators using the same Dr.GRPO policy-optimization recipe as our methods. As shown in the tables below, MM-GRPO achieves the overall best performance, with the highest average R2R^2 for both model sizes. The results further demonstrate the effectiveness of the MM-GRPO advantage formulation. We will be including these results in the Results section of the revised manuscript:

Qwen3-4B

MethodNutrition R2R^2 ↑Nutrition MAE ↓Nutrition RMSE ↓Price R2R^2 ↑Price MAE ↓Price RMSE ↓Avg. R2R^2 ↑
Dr.GRPO0.6949.95119.1850.58611.34218.3480.640
Dr.GRPO + AGRPOA^{\mathrm{GRPO}}0.70410.12218.8950.45511.30121.0480.579
Dr.GRPO + ABatchA^{\mathrm{Batch}}0.7399.77117.7180.62110.94217.5420.680
R-GRPO0.7329.75017.9530.61011.21217.8100.671
MM-GRPO0.7568.97817.1400.62110.90217.5420.689

Qwen3-1.7B

MethodNutrition R2R^2 ↑Nutrition MAE ↓Nutrition RMSE ↓Price R2R^2 ↑Price MAE ↓Price RMSE ↓Avg. R2R^2 ↑
Dr.GRPO0.37517.63427.4340.49812.92820.2070.437
Dr.GRPO + AGRPOA^{\mathrm{GRPO}}0.47015.34425.2600.46913.16620.7810.469
Dr.GRPO + ABatchA^{\mathrm{Batch}}0.49115.02024.7610.49113.04420.3450.491
R-GRPO0.49014.36724.7940.53612.51519.4110.513
MM-GRPO0.60812.69021.7190.59111.71618.2270.600

Tuning Scaling Factor for GRPO

We have conducted the same scaling factor validation-based tuning procedure for GRPO for Qwen3-1.7B and found that the original GRPO scale of 1 achieved the best validation performance. Therefore, the reported GRPO result already corresponds to its best scale among the evaluated values. For Qwen3-4B, no global advantage scaling was applied to any method. We will expand the advantage-scaling table in the Appendix to include the following GRPO results:

ScaleNutrition Estimation MAE ↓Product Price Prediction MAE ↓
112.90413.257
413.14814.167
813.67913.483
1614.85214.258

We thank the reviewer for raising these important questions. We address each question individually below and will incorporate the corresponding clarifications into the revised manuscript.

Q1: Can the MM-GRPO scale be derived from an optimization principle, such as minimizing gradient variance, or is it primarily based on empirical observations?

MM-GRPO is not derived by minimizing policy-gradient variance, but rather is motivated by a set of desirable properties for continuous regression rewards. We seek an estimator that:

  1. is invariant to reward translation and positive rescaling
  2. remains bounded as reward differences contract
  3. preserves the ordering and relative numerical gaps between responses
  4. allows the normalized structure of a group’s rewards to affect its overall advantage scale.

Dr.GRPO preserves raw gaps but makes advantage magnitude proportional to the raw reward scale. Standard GRPO removes raw scale but forces every nonconstant group to have approximately unit advantage variance. MM-GRPO instead fixes the advantage range while allowing its standard deviation to equal approximately

ασrmax⁡−rmin⁡\alpha \frac{\sigma}{r_{\max}-r_{\min}}

Consequently, a group whose rewards are broadly distributed across its range receives a larger advantage scale than one in which most rewards are concentrated and only an isolated value establishes the range. We do not claim that this weighting is universally optimal; it is a transparent, testable inductive bias, and the controlled experiments evaluate whether it is useful for regression RL.

Q2: How do MM-GRPO and R-GRPO handle groups with nearly identical rewards, near-zero standard deviation, or extreme outliers? Can these cases lead to exploding gradients or vanishing training signals?

1. Nearly identical rewards and near-zero standard deviation: Neither estimator can produce exploding advantages solely because the reward dispersion approaches zero. MM-GRPO computes

AiMM=αri−μrmax⁡−rmin⁡+ϵ.A_i^{\mathrm{MM}}=\alpha\frac{r_i-\mu}{r_{\max}-r_{\min}+\epsilon}.

Because ∣ri−μ∣≤rmax⁡−rmin⁡|r_i-\mu|\leq r_{\max}-r_{\min}, every advantage satisfies ∣AiMM∣≤α|A_i^{\mathrm{MM}}|\leq\alpha. When the rewards are nearly identical, the centered rewards and their range contract together; if the range becomes comparable to ϵ\epsilon, the advantages shrink rather than explode. An exactly constant group produces zero MM-GRPO advantages because it contains no reward-supported comparison.

R-GRPO does not divide by a dispersion statistic and uses only bounded normalized ranks. Its advantage magnitude therefore does not shrink for distinct but numerically close rewards, although the resulting ordering may become more sensitive to reward noise.

2. Extreme outliers: For MM-GRPO, the within-group advantage standard deviation is approximately

ασrmax⁡−rmin⁡=αcg.\alpha\frac{\sigma}{r_{\max}-r_{\min}}=\alpha c_g.

When an isolated extreme reward establishes a large range while most rewards remain concentrated, cgc_g is smaller than for a group whose rewards are broadly distributed across that range. MM-GRPO therefore assigns a smaller overall advantage scale to outlier-dominated groups and a larger scale to groups whose rewards are distributed more broadly within their range. R-GRPO also limits the influence of an outlier because its advantage depends on rank rather than the outlier’s numerical magnitude.

These properties prevent near-zero dispersion or extreme reward magnitudes from directly producing unbounded advantages. We will clarify these edge-case behaviors in the revised manuscript.

Q3: What is the relationship between the R-GRPO objective and the original continuous regression objective? Can the authors report MAE, RMSE, Spearman correlation, long-tail error, and calibration metrics for all methods?

R-GRPO applies a monotonic transformation to rewards within each group: because the reward is negative absolute error, ranking rewards is equivalent to ranking responses by absolute prediction error. Therefore the objective of R-GRPO is similar to that of the original regression objective in that it encourages responses with lower numerical error relative to other responses sampled for that prompt. However, R-GRPO discards information about how much lower that error is. Our main results already report MAE, RMSE, and R2R^2, together with long-tail MAE in the main Results section and calibration plots in the Appendix.

Q4: Do the advantages of MM-GRPO and R-GRPO remain stable when group size, reward noise, reward range, and the proportion of long-tail examples are varied?

MM-GRPO is invariant to reward translation and approximately invariant to positive reward rescaling, while its advantages remain bounded by α\alpha. R-GRPO is independent of reward magnitude and range, and its normalized rank advantages are also bounded. Thus, neither estimator produces exploding advantages when the reward range changes. However, reward noise can alter the extrema used by MM-GRPO and the ordering used by R-GRPO, so neither formulation guarantees invariance to noisy rewards. Our experiments use group sizes of 8 and 64 for the two model sizes (4B and 1.7B), and both methods perform well in these settings. Our long-tail evaluation demonstrates performance on rare high-value targets.

As related evidence, our new analysis on the performance under reward collapse shows that MM-GRPO achieves the lowest average error among collapsed groups despite having the largest proportion of such groups, showing that it remains effective under reward concentration. As shown in the Table (below), collapsed groups retain substantial prediction error, ranging from 23.7% to 36.7% of the ground-truth value for Qwen3-4B and from 32.6% to 62.0% for Qwen3-1.7B. A group is considered collapsed when all of its rollout errors are within 10% of one another for Qwen3-4B or 15% for Qwen3-1.7B. We use the wider threshold for Qwen3-1.7B because its groups contain 64 rather than 8 rollouts, making complete agreement across the group less likely. This analysis shows that collapsed groups are substantially closer to one another than to the ground-truth, demonstrating that reward collapse does not necessarily imply convergence to the correct prediction. Despite having the largest proportion of collapsed groups, MM-GRPO achieves the lowest overall collapsed-group error, with R-GRPO achieving the second lowest average error.

Method4B Nutrition % Error ↓4B Price % Error ↓1.7B Nutrition % Error ↓1.7B Price % Error ↓Average % Error ↓% of Groups Collapsed
Dr.GRPO36.733.252.240.940.841.9
GRPO27.830.841.532.933.344.8
Batch GRPO27.033.643.862.041.635.5
R-GRPO26.030.132.637.131.543.7
MM-GRPO23.728.236.732.630.351.5

Q5: Have the authors compared against reward clipping, global normalization, quantile normalization, rank transformation, and other robust advantage-normalization methods?

We thank the reviewer for raising these questions. Several of the requested alternatives are already represented in our evaluation. GRPO uses within-group standard-deviation normalization, while Batch GRPO uses a batch-level standard deviation and therefore provides a global scale-normalization baseline while retaining group-level centering. Dr.GRPO evaluates mean-centered rewards without scale normalization. R-GRPO directly evaluates rank transformation; because its normalized ranks correspond to empirical cumulative probabilities, it is also equivalent to empirical quantile normalization up to affine scaling and tie handling.

We did not evaluate reward clipping or every possible robust scale estimator. Reward clipping is not itself a normalization strategy and requires a task-dependent clipping threshold; it also discards reward differences beyond that threshold. Similarly, median- or quantile-based dispersion estimators introduce additional design choices that are outside the scope of the current study. We will clarify that our comparison covers the principal mean-centered, group-normalized, batch-normalized, range-normalized, and rank/quantile-based formulations rather than claiming an exhaustive evaluation of every robust transformation.

We further added controlled experiments that apply the GRPO and Batch GRPO advantage estimators within the exact Dr.GRPO optimization recipe, removing the KL and policy-objective confounds (results of this analysis in Official Comment 1 above). MM-GRPO retains the strongest overall performance under this comparison. We also apply the same validation-based advantage-scaling procedure to GRPO and find that its original scale of 1 performs best, showing that global rescaling alone does not reproduce MM-GRPO’s gains. Together, these results isolate the benefit of range-normalized, group-dependent advantage scaling from batch normalization, rank transformation, optimization-recipe differences, and global advantage scaling.

Thank you again for your feedback. As the discussion period ends today, we wanted to follow up to see whether our response has addressed your concerns or if any additional clarification would be helpful.

Reviewer_2kfdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies group-relative advantage estimation for text-to-text regression, where the reward is the negative absolute error of a numerical prediction. It shows that as the policy improves, rewards within a rollout group become concentrated. Based on this phenomenon, the paper proposes MM-GRPO, which divides centered rewards by the within-group range so that groups with greater relative reward dispersion contribute more strongly to the learning process. It further introduces R-GRPO, which replaces reward magnitudes with within-group ranks to test whether ordering alone is enough. Finally, the paper presents an empirical analysis comparing the proposed methods against existing methods using the Qwen3-4B and Qwen3-1.7B models.

Critical strengths
  • The main strength is that the paper identifies a real problem in text-to-text regression, and the proposed method is a simple solution for that. Both MM-GRPO and R-GRPO show consistent improvement against baselines.
Critical weaknesses
  • My primary concern is that the results may be restricted to small models and specific seeds. All experiments use 1.7B and 4B models, and the gains decrease as the model size decreases from 1.7B to 4B. It is unclear whether they hold at 7B or above. Since we are asking the model to provide a numerical value, the results will depend heavily on the decoding temperature. A single seed and a specific decoding temperature make the result less trustworthy.
Questions for authors
  1. What are the results when tested on larger models, say 7B or above? We can notice that gains are much larger for 1.7B than for 4B, so it is not clear to me whether the results will be similar when extending to larger models.
  2. How stable are the results across random seeds and sampling temperatures? Since we ask the model to produces a number, the predictions and the within-group reward spread depend directly on the decoding temperature. Can we notice the same improvement given different temperatures and seeds?
1 comment

We thank the reviewer for raising these concerns. We have conducted an additional full comparison using a second random seed, in addition to a new ablation study for advantage estimators, and we clarify the current scope regarding model size and sampling temperature below.

Q1: What are the results on models with 7B parameters or more?

Our current experiments cover Qwen3-1.7B and Qwen3-4B, and we were unable to complete an additional 7B-scale training suite within our available compute budget. We therefore do not claim that the empirical gains necessarily extend unchanged beyond 4B.

At the same time, evaluating smaller models is practically relevant for text-to-text regression. Fine-tuning compact models for specialized numerical prediction tasks substantially reduces training and inference costs and supports deployment in resource-constrained settings. Our results establish the effectiveness of MM-GRPO across two model sizes and two real-world regression tasks, but evaluating whether the gains persist at 7B and larger scales remains an important direction for future work.

Q2: How stable are the results across random seeds and sampling temperatures?

To evaluate seed sensitivity, we conducted a complete second-seed comparison of all advantage-estimation methods using Qwen3-4B under a fixed compute budget. Every method was trained with 256 prompts and 8 rollouts per prompt. We used a mini-batch size of 64 prompts, while keeping all other hyperparameters consistent with the main experiments. We evaluated each method on the same fixed random subset of 1,000 validation examples using greedy decoding. We selected each method's best validation checkpoint and evaluate that checkpoint on the complete held-out test sets.

Second-seed Qwen3-4B results.

MethodNutrition R2R^2 ↑Nutrition MAE ↓Nutrition RMSE ↓Price R2R^2 ↑Price MAE ↓Price RMSE ↓Avg. R2R^2 ↑
Dr.GRPO0.56011.64723.0280.33912.11923.1840.450
GRPO0.68510.76919.4840.22911.36325.0320.457
Batch GRPO0.67411.54919.8060.61911.58717.6040.647
R-GRPO0.63311.04521.0250.54411.18019.2540.589
MM-GRPO0.71110.28118.6440.63010.93417.3480.671

MM-GRPO again achieves the best R2R^2, MAE, and RMSE on both tasks, reproducing the central result under a second random seed.

The training rollouts in this experiment use the same sampling temperature as the main experiments, while validation and test predictions are generated with greedy decoding, for which no sampling temperature is applied. Consequently, the reported test metrics are not affected by the decoding temperature, although the training temperature can still influence the diversity of sampled rollout groups.

Advantage Estimation Ablation

We also performed an additional controlled ablation to separate the effect of advantage estimation from differences in the policy-optimization recipe, such as KL regularization. Specifically, we evaluated the GRPO and Batch GRPO advantage estimators using the same Dr.GRPO optimization recipe used by our methods, including the omission of the KL penalty. The table below summarizes these results and will be included in the revised manuscript.

Qwen3-4B

MethodNutrition R2R^2 ↑Nutrition MAE ↓Nutrition RMSE ↓Price R2R^2 ↑Price MAE ↓Price RMSE ↓Avg. R2R^2 ↑
Dr.GRPO0.6949.95119.1850.58611.34218.3480.640
Dr.GRPO + AGRPOA^{\mathrm{GRPO}}0.70410.12218.8950.45511.30121.0480.579
Dr.GRPO + ABatchA^{\mathrm{Batch}}0.7399.77117.7180.62110.94217.5420.680
R-GRPO0.7329.75017.9530.61011.21217.8100.671
MM-GRPO0.7568.97817.1400.62110.90217.5420.689

Qwen3-1.7B

MethodNutrition R2R^2 ↑Nutrition MAE ↓Nutrition RMSE ↓Price R2R^2 ↑Price MAE ↓Price RMSE ↓Avg. R2R^2 ↑
Dr.GRPO0.37517.63427.4340.49812.92820.2070.437
Dr.GRPO + AGRPOA^{\mathrm{GRPO}}0.47015.34425.2600.46913.16620.7810.469
Dr.GRPO + ABatchA^{\mathrm{Batch}}0.49115.02024.7610.49113.04420.3450.491
R-GRPO0.49014.36724.7940.53612.51519.4110.513
MM-GRPO0.60812.69021.7190.59111.71618.2270.600

MM-GRPO retains the overall best performance under this controlled optimization recipe, with the highest average R2R^2.

Together with the new second-seed experiment, these results further demonstrate the effectiveness of the MM-GRPO advantage formulation.

Reviewer_qgT1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes two group-relative advantage estimators for GRPO-style RL fine-tuning of LLMs on text-to-text regression, motivated by “reward collapse” (within-group rewards concentrating as training progresses). MM-GRPO scales centered rewards by the within-group reward range, which amounts to rescaling GRPO advantages by a group-dependent factor. R-GRPO uses within-group reward ranks instead of magnitudes. On carbohydrate estimation and Amazon price prediction with Qwen3-4B and 1.7B, MM-GRPO gives the best overall results against baselines, with R-GRPO competitive and both showing gains on long-tail and multilingual inputs.

Critical strengths
  1. The method is clearly presented and the paper is easy to follow.
  2. The paper provides reasonably broad evaluation. The paper goes beyond average accuracy to long-tail targets, calibration plots, and multilingual out-of-distribution inputs, and covers two tasks and two model sizes. The direction of the results is consistent, even though the evidence is questionable due to experiment design (see Critical Weaknesses).
Critical weaknesses
  1. “Reward collapse” is presented as a defect without evidence that it is one. The paper itself calls this “a natural consequence of learning,” and it is arguably the desired outcome for a regression task, where the goal is for sampled predictions to agree with each other and with the target. Therefore, the motivation of the proposal seems questionable.
  2. Since the method is just a change of the normalization strategy of the (Dr.)GRPO for the special task of regression, the novelty is compromised. While the moderate novelty is not disqualifying on its own, the experiments are not rigorously conducted. For example, since the KL penalty is removed from the method while vanilla GRPO and Batch GRPO is tested with the KL penalty, it is unfair to compare them, which hardly justify if the performance gains come from the normalization strategy alone.
  3. Another problem with the experimental methodology is that a key ablation experiment is missing. Since MM ≈ α·c_g·GRPO as acknowledged in the paper, the gains may come from tuning α rather than from range normalization or adaptivity. There needs to be a “GRPO × α”-like baseline.
Questions for authors
  1. To resolve my concern about the motivation of the paper (ie., addressing reward collapse), can the author provide results of training dynamics, such as reward curves, entropy, or across-rollout prediction variance?
  2. Can the authors provide additional ablation experiments as mentioned previously in weakness 2 and 3?
5 comments
AuthorsOpenReview ↗

Addressing Reward Collapse

Q1/ W1: Addressing Reward Collapse

We thank the reviewer for the insightful comment, which motivated the additional analysis presented below. We agree that reward concentration can be a natural and desirable consequence of learning when predictions converge toward the ground-truth target. However, reward collapse can become an issue when the collapsed groups are still far from the ground truth. We find that collapsed groups still have substantial prediction errors, ranging from 23.7% to 62.0% of the ground-truth value.

To investigate this distinction, we conducted an additional analysis of collapsed rollout groups. A group is considered collapsed when its rollout errors are within 10% of one another for Qwen3-4B or 15% for Qwen3-1.7B. The wider threshold for Qwen3-1.7B accounts for its larger rollout group size (64 versus 8), which makes close agreement across all samples less likely. As shown in the table below, even among collapsed groups, the average prediction error remains substantial, ranging from 23.7% to 36.7% for Qwen3-4B and from 32.6% to 62.0% for Qwen3-1.7B. This demonstrates that rollouts can exhibit highly concentrated rewards while remaining inaccurate, and that reward concentration alone is not a reliable indicator that a prompt has been solved.

We observe that MM-GRPO achieves the lowest average prediction error among collapsed groups, followed by R-GRPO, despite MM-GRPO exhibiting a high proportion of collapsed groups. These findings reinforce our motivation: the goal is not to prevent predictions from converging, but to develop advantage estimators that remain effective in the presence of concentrated rewards.

We will incorporate this analysis into the main Results section of the revised manuscript, together with the table below:

Method4B Nutrition % Error ↓4B Price % Error ↓1.7B Nutrition % Error ↓1.7B Price % Error ↓Average % Error ↓% of Groups Collapsed
Dr.GRPO36.733.252.240.940.841.9
GRPO27.830.841.532.933.344.8
Batch GRPO27.033.643.862.041.635.5
R-GRPO26.030.132.637.131.543.7
MM-GRPO23.728.236.732.630.351.5
AuthorsOpenReview ↗

KL Penalty and Scaling Factor Ablations

Q2/ W2 & W3 : KL and Scaling Factor Ablations

We have added two new studies that directly address these concerns and will be included in the revised manuscript.

Advantage Estimation Ablation (KL penalty)

We have conducted an ablation study to isolate the effects of changing the advantage estimator while keeping everything else constant, including the omission of the KL regularization penalty. Specifically, we evaluate the GRPO and Batch GRPO advantage estimators using the same Dr.GRPO policy-optimization recipe as our methods. As shown in the tables below, MM-GRPO achieves the overall best performance, with the highest average R2R^2 for both model sizes. The results further demonstrate the effectiveness of the MM-GRPO advantage formulation. We will be including this analysis in the Results section of the revised manuscript:

Qwen3-4B

MethodNutrition R2R^2 ↑Nutrition MAE ↓Nutrition RMSE ↓Price R2R^2 ↑Price MAE ↓Price RMSE ↓Avg. R2R^2 ↑
Dr.GRPO0.6949.95119.1850.58611.34218.3480.640
Dr.GRPO + AGRPOA^{\mathrm{GRPO}}0.70410.12218.8950.45511.30121.0480.579
Dr.GRPO + ABatchA^{\mathrm{Batch}}0.7399.77117.7180.62110.94217.5420.680
R-GRPO0.7329.75017.9530.61011.21217.8100.671
MM-GRPO0.7568.97817.1400.62110.90217.5420.689

Qwen3-1.7B

MethodNutrition R2R^2 ↑Nutrition MAE ↓Nutrition RMSE ↓Price R2R^2 ↑Price MAE ↓Price RMSE ↓Avg. R2R^2 ↑
Dr.GRPO0.37517.63427.4340.49812.92820.2070.437
Dr.GRPO + AGRPOA^{\mathrm{GRPO}}0.47015.34425.2600.46913.16620.7810.469
Dr.GRPO + ABatchA^{\mathrm{Batch}}0.49115.02024.7610.49113.04420.3450.491
R-GRPO0.49014.36724.7940.53612.51519.4110.513
MM-GRPO0.60812.69021.7190.59111.71618.2270.600

Tuning Scaling Factor for GRPO

We have conducted the same scaling factor validation-based tuning procedure for GRPO for Qwen3-1.7B and found that the original GRPO scale of 1 achieved the best validation performance. Therefore, the reported GRPO result already corresponds to its best scale among the evaluated values. For Qwen3-4B, no global advantage scaling was applied to any method. We will expand Table 3 in the Appendix to include the following GRPO tuned results:

ScaleNutrition Estimation MAE ↓Product Price Prediction MAE ↓
112.90413.257
413.14814.167
813.67913.483
1614.85214.258

Thank you again for your feedback. As the discussion period ends today, we wanted to follow up to see whether our response has addressed your concerns or if any additional clarification would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.