acceptodds
Under review as a conference paper at ICLR 2027

Save Your Saturated Data: Learning Beyond Reward Saturation in Group-Based RL

Abstract

Group-relative reinforcement learning (RL) relies on reward variation among sampled responses to estimate informative relative advantages. As language models become increasingly capable, existing training data can become reward-saturated: all sampled responses to the same problem might receive equally high rewards, where the group-relative learning signals vanish and leave previously useful data obsolete. In this work, we investigate whether useful learning signals can be recovered from such saturated data. We study interventions at four levels of group-policy RL pipelines—data, rollout, reward, and advantage—and conduct extensive RL training on saturated reasoning data only. While standard GRPO on saturated data would almost always yield near-0 advantages and near-noise signals, diverse interventions successfully recycle and repurpose such data: among the proposed strategies, interventions at rollout generation are consistently most effective: nudging the policy to generate “high-quality”, incorrect solutions introduces rollouts with poor rewards into saturated groups as negative samples, which turns out to improve GRPO by 6.4% to 9.0% across Qwen3-1.7B and 4B. Other interventions such as increasing rollout temperature or adding auxiliary rewards can also restore non-zero advantages, but yield less consistent gains. Further analyses show that effective negative rollouts require informative negative trajectories, that the method remains effective alongside unsaturated data, and that it supports iterative recycling of newly saturated examples. While increasingly stronger LLMs would render more data as saturated, our results demonstrate that don't waste your saturated data: with the right strategies they can be recycled into useful RL training signals in an increasingly data-scarce world. Our code is available.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Post-Training, Reinforcement Learning, GRPO, Reward Saturation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eMbTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies GRPO on reward-saturated prompts, where every sampled response is correct and the group-relative advantage vanishes. It compares interventions at four stages of training: perturbing the prompt, changing rollout generation, adding auxiliary rewards, and padding the advantage computation. The main proposal asks the current policy to write a plausible but wrong solution and adds it to the group as a negative sample. Training Qwen3 models only on saturated MATH problems, the paper finds this works best and adds analyses on how negatives are built, mixed data, iterative recycling and another model family.

Critical strengths

The negative rollout trick is simple, since it only needs an extra instruction in the prompt, and it beats GRPO on both Qwen3 sizes, with the gain holding across three seeds on the 1.7B model. For anyone who currently throws away prompts the model always solves, this is a cheap way to get some use out of them, and it is the main reason I see value in the paper.

I also found the comparison between different ways of restoring variance instructive. Zero padding makes the advantages non-zero without adding any wrong trajectory and does not help much, and swapping only the final answer of a correct solution actually hurts. So what seems to matter is having a coherent wrong solution to push away from, not just a non-zero advantage. That is useful to know for anyone working on zero-variance groups.

Critical weaknesses

The paper starts from the claim that GRPO gets almost no signal from saturated data, but its own GRPO baseline improves quite a bit on this data, especially on the harder math benchmarks. That suggests many groups stop being all-correct once training starts, which the paper never checks. So I cannot tell whether negative rollouts restore a missing signal or add a negative gradient on top of an existing one. I also could not find whether the loss on the negatives uses the original prompt or the prompt asking for a mistake, and the two would teach different things.

The paper is pitched as a comparison of which part of the pipeline best recovers signal, but each part gets one or two configurations, untuned and run once. The authors' own seed runs and mixed-data experiment show results moving between runs about as much as the methods differ, so I do not think the ranking across levels holds up. What I am convinced of is the narrower point that negative rollouts beat plain GRPO. The recycling and LLaMA experiments have a similar problem: the first has no matched baseline, and the second shows a small, mixed gain.

Questions for authors
  1. How often are groups actually all-correct during training, under GRPO and under each intervention, and how does that fit with the gains plain GRPO gets on this data?
  2. Is the loss on the constructed negatives computed with or without the wrong-answer instruction in the context?
  3. Do you check that the constructed negatives are actually wrong, and what happens when the model still answers correctly?
3 comments

Question 2: Implementation Details

We thank the reviewer for highlighting the need for clarification. The negative prefix is used only during response generation, while all policy log-likelihoods and KL regularization are computed under the original prompt, without the additional prefix. This design is intentional. In reward-saturated groups, the on-policy GRPO gradient is zero, so an unbiased estimate of this gradient provides no learning signal. Our objective is to restore meaningful learning signals through informative negative trajectories, rather than estimate the original on-policy gradient.

For negative response filtering, all generated candidates are evaluated using the original reward function, and only verified incorrect responses are retained. If no incorrect candidate is found, no replacement occurs. During optimization, retained negatives keep their verified rewards and participate in group-wise advantage normalization and policy optimization under the original prompt. Only response tokens contribute to the GRPO loss, while the additional misconception instruction is excluded from optimization. We add these details to the Section 4.1 Experiments Settings in the revised paper.

Question 1 and 3: More Learning Dynamics

We further track the original group saturation ratio throughout training, which averages 78.30% for Qwen3-1.7B and 91.05% for Qwen3-4B. This indicates that reward saturation remains prevalent during training, while the remaining non-saturated groups still provide learning signals that explain the improvements of vanilla GRPO. To better understand these dynamics, we additionally track the saturation ratio, and negative generation success rate throughout training. We add these learning dynamics to the Section 6.1 Learning Dynamics of Reward Saturation in the revised paper.

More ablations

To isolate the contribution of directly optimizing negative trajectories, we conduct two additional ablations: Negative Rollout w/o Gradient: The same six ordinary rollouts and two constructed negative rollouts, but masking the gradients of the negative trajectories while keeping the group advantage computation unchanged. 6+2 Zero Padding: Six ordinary rollouts with two zero-paddings. The results are summarized below:

MethodMATHGPQAMinervaAIME24AIME25IFBench
1.7B
Negative Rollout64.8029.8018.3820.3114.3824.00
w/o Gradient63.2023.7415.0719.2710.2125.20
6+2 Zero Padding64.6026.7718.0116.6713.5422.00
4B
Negative Rollout70.6040.4026.1033.0225.5233.20
w/o Gradient69.4034.8525.3735.5215.4228.80
6+2 Zero Padding58.0026.7722.434.271.0428.80

These results show that directly optimizing negative trajectories generally outperforms both alternatives across model scales. Compared with the gradient-masked variant, Negative Rollout improves average performance by 9.5% and 9.4% on the 1.7B and 4B models, respectively. It also consistently achieves higher average performance than the 6+2 zero-padding control. These findings demonstrate that the improvements extend beyond changes in group-wise advantage estimation, highlighting the additional benefits of directly optimizing informative negative trajectories. We have included these experiments in Appendix D.1 (​​Ablation on Negative Trajectory Optimization) of the revised paper.

Reviewer_eMbTOpenReview ↗

Response to authors

Thank you for clarifying the implementation and for adding the training dynamics and ablations. These address several of my concerns and support the benefit of the negative rollouts.
However, my concern about the comparison across intervention levels, which is central to the paper, remains. In Table 1, Negative Rollout leads the strongest alternatives from other levels by 0.83 points on Qwen3-1.7B (zero padding, k = 3) and 0.98 points on Qwen3-4B (reward model). These alternatives were run only once, so it is unclear whether these margins hold across seeds. The recycling experiment still lacks a matched baseline, and the gain on LLaMA remains small (0.66 points over GRPO). I recognize the improvement over GRPO, but the evidence for the broader conclusions remains insufficient to change my assessment. I therefore keep my score at 2.

Thank you for your constructive feedback. We have conducted additional experiments and would like to clarify the following points.

Multi-seed evaluation of competing interventions. To address the concern about single-seed comparisons, we evaluated the strongest alternative intervention for each model using three random seeds.

ModelInterventionSeed 1Seed 2Seed 3Mean ± Std
Qwen3-1.7BZero Padding (k=3)30.7030.5429.2630.17 ± 0.79
Qwen3-4BReward Model40.2439.1938.8139.41 ± 0.74

These results provide a more reliable estimate of the performance of the strongest competing interventions and strengthen the comparison across intervention levels.

Recycling experiment. Our primary goal is to demonstrate a potential setting for reusing saturated data, rather than to establish that recycling outperforms all alternative data augmentation strategies. We acknowledge that additional training samples can naturally improve performance and will clarify the scope of this experiment in the revised paper.

Generalization to LLaMA. The LLaMA experiment primarily aims to demonstrate that Negative Rollout can generalize beyond the Qwen model family. While the improvement is relatively modest, it provides initial evidence of cross-family applicability. We will consider evaluating additional model families and scales in future work.

Reviewer_cSh7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies learning from empirically all-correct groups in group-relative RL. It compares data-, rollout-, reward-, and advantage-level interventions and finds that constructing coherent negative rollouts gives the strongest aggregate gains. It also examines mixed-saturation data, iterative mining, and training-seed variation.

Critical strengths

The paper systematically studies an underexplored failure mode rather than equating nonzero advantage with useful learning. The negative-construction comparisons and three-seed Qwen3-1.7B results provide useful evidence that trajectory content matters.

Critical weaknesses
  1. The sampling and update contexts for constructed negatives are unclear. Appendix B.2 generates negatives using an explicit wrong-answer suffix. If that suffix is removed when computing policy log probabilities, there is a behavior-context mismatch; if retained, the update may partly penalize compliance with the directive. This is an unresolved methodological question, not a confirmed implementation error (pp. 3–5, 15).
  2. Eight correct samples identify an observed all-correct group, not an intrinsically solved question. For example, a policy with true success probability 0.9 passes that screen with probability about 0.43. Without larger independent resampling, the extent of saturation and the interpretation of recovered learning remain uncertain (pp. 3–5).
Questions for authors
  1. Are the suffix and original prompt identical for generation, old-policy scoring, and current-policy scoring? If not, how is the sampling mismatch handled? What control isolates informative reasoning errors from the wrong-answer directive?
  2. What are the success-rate distributions under larger independent resampling for the selected pool, and how often do those groups remain all-correct during training?
2 comments

Weakness 1 and Question 1: Implementation Details

We thank the reviewer for highlighting the need for clarification. The negative prefix is used only during response generation, while all policy log-likelihoods and KL regularization are computed under the original prompt, without the additional prefix. This design is intentional. In reward-saturated groups, the on-policy GRPO gradient is zero, so an unbiased estimate of this gradient provides no learning signal. Our objective is to restore meaningful learning signals through informative negative trajectories, rather than estimate the original on-policy gradient.

For negative response filtering, all generated candidates are evaluated using the original reward function, and only verified incorrect responses are retained. If no incorrect candidate is found, no replacement occurs. During optimization, retained negatives keep their verified rewards and participate in group-wise advantage normalization and policy optimization under the original prompt. Only response tokens contribute to the GRPO loss, while the additional misconception instruction is excluded from optimization. We add these details to the Section 4.1 Experiments Settings in the revised paper.

Weakness 2: More Resampling

To assess whether saturation remains prevalent under substantially more independent resampling, we independently resample 32 responses per prompt at temperature 1.0, using 200 prompts across three seeds (19,200 responses total). Notably, 82.50% prompts initially identified as saturated under 8 rollouts still achieve at least 90% accuracy under 32 independent resamplings, with 57.17% ± 2.36% remaining fully saturated (32/32 correct). The overall response accuracy is 93.96%. The distribution across seeds is summarized below (mean ± std):

Metric100%[90%, 100%)[80%, 90%)[60%, 80%)[40%, 60%)[20%, 40%)(0%, 20%)0%
Prompts114.33 ± 4.7350.67 ± 6.6615.67 ± 1.5312.67 ± 4.165.00 ± 2.650.67 ± 0.581.00 ± 0.000.00 ± 0.00
Percentage (%)57.17 ± 2.3625.33 ± 3.337.83 ± 0.766.33 ± 2.082.50 ± 1.320.33 ± 0.290.50 ± 0.000.00 ± 0.00

These results show that most selected prompts have high success rates, though they are not necessarily intrinsically solved. Our method targets observed group-level saturation, where identical rewards yield zero advantages. We have included these experiments in Appendix D.2 (​​Independent Resampling Analysis) of the revised paper.

Question 2: Learning Dynamics

We further track the original group saturation ratio throughout training, which averages 78.30% for Qwen3-1.7B and 91.05% for Qwen3-4B. This indicates that reward saturation remains prevalent during training, while the remaining non-saturated groups still provide learning signals that explain the improvements of vanilla GRPO. To better understand these dynamics, we additionally track the saturation ratio, and negative generation success rate throughout training. We add these learning dynamics to the Section 6.1 Learning Dynamics of Reward Saturation in the revised paper.

More ablations

To isolate the contribution of directly optimizing negative trajectories, we conduct two additional ablations: Negative Rollout w/o Gradient: The same six ordinary rollouts and two constructed negative rollouts, but masking the gradients of the negative trajectories while keeping the group advantage computation unchanged. 6+2 Zero Padding: Six ordinary rollouts with two zero-paddings. The results are summarized below:

MethodMATHGPQAMinervaAIME24AIME25IFBench
1.7B
Negative Rollout64.8029.8018.3820.3114.3824.00
w/o Gradient63.2023.7415.0719.2710.2125.20
6+2 Zero Padding64.6026.7718.0116.6713.5422.00
4B
Negative Rollout70.6040.4026.1033.0225.5233.20
w/o Gradient69.4034.8525.3735.5215.4228.80
6+2 Zero Padding58.0026.7722.434.271.0428.80

These results show that directly optimizing negative trajectories generally outperforms both alternatives across model scales. Compared with the gradient-masked variant, Negative Rollout improves average performance by 9.5% and 9.4% on the 1.7B and 4B models, respectively. It also consistently achieves higher average performance than the 6+2 zero-padding control. These findings demonstrate that the improvements extend beyond changes in group-wise advantage estimation, highlighting the additional benefits of directly optimizing informative negative trajectories. We have included these experiments in Appendix D.1 (​​Ablation on Negative Trajectory Optimization) of the revised paper.

Dear Reviewer,

Thank you very much for your valuable feedback on our work! As the discussion period draws to a close, we would welcome any further questions or feedback on our responses and additional results. If any concerns remain, we would be happy to address them and provide further clarification.

Best regards,

The Authors

Reviewer_HBBpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how to recover learning signals from prompts whose sampled responses all receive maximum reward. It compares interventions at the data, rollout, reward, and advantage levels. Its main method asks the current policy to generate plausible incorrect solutions and includes two constructed negatives in a group of eight. Average gains over GRPO are 2.60 and 2.48 percentage points on Qwen3-1.7B and Qwen3-4B, respectively (9.0% and 6.4% relative). Further experiments examine negative construction, mixed saturation ratios, iterative mining, and transfer to LLaMA-3.1-8B-Instruct.

Critical strengths
  1. A useful comparison of ways to recover learning signals. The paper addresses a relevant bottleneck and compares multiple intervention levels, including the existing Mixed-CUTS method. The negative-construction comparison in Table 2 suggests that coherent incorrect trajectories can be more useful than replacing only the answer or introducing arbitrary reward variation. This is a practically interesting observation beyond the basic fact that unequal rewards restore nonzero advantages.

  2. Evidence that the gains are not confined to one run or a fully saturated mixture. Appendix C reports three-seed averages of 29.38 ± 0.41 for GRPO and 32.04 ± 0.60 for Negative Rollout on Qwen3-1.7B. The mixed-data experiments also show benefits when saturated prompts are only part of the training set. These results make the empirical direction promising, although the evidence is stronger for some models and tasks than others.

Critical weaknesses
  1. The central negative-rollout update is underspecified. Section 3.2 and Appendix B.2 generate negatives with an added wrong-answer instruction, while Equation (2) gives an ordinary on-policy GRPO objective. The paper does not clearly state whether that instruction is retained during optimization, which context supplies old-policy likelihoods and KL, or how negative tokens enter the loss. Removing the instruction creates a conditional behavior-distribution mismatch; retaining it mixes trajectories conditioned on different prompts. Neither design is necessarily invalid, but the actual objective and its treatment of this mismatch are essential to assess and reproduce the method. The handling of an instructed negative that is nevertheless correct is also unclear.

  2. The main ablation does not cleanly attribute the gain to learning from negative trajectories. The proposed method uses six ordinary responses and two constructed negatives, whereas zero padding augments an original group of eight successes. This changes positive-sample counts and advantage scaling as well as negative-token learning. A matched control is needed. The scope of the gain also warrants care: 4B MATH-500 is unchanged, AIME25 decreases from 27.40 to 25.52, and the LLaMA average gain is 0.66 points with several regressions. The three-seed 1.7B result is helpful, but eight-success screening alone does not show that groups remain saturated under fresh training samples. Together, these limitations leave uncertainty about what produces the benefit and when it should be expected.

Questions for authors
  1. Please specify generation versus optimization prompts, old-policy likelihoods, KL context, loss masks, and reward verification for constructed negatives. How are accidentally correct negatives handled, and is any conditional distribution mismatch intentional?

  2. Does the gain persist with the same six positive trajectories and two negative reward entries, but gradients through negative trajectories disabled and the loss normalization held fixed? Please report total sampling/verifier cost, including screening or replacement, and the fraction of screened prompts that remains saturated under independent resampling.

2 comments

Weakness 1 and Question 1: Implementation Details

We thank the reviewer for highlighting the need for clarification. The negative prefix is used only during response generation, while all policy log-likelihoods and KL regularization are computed under the original prompt, without the additional prefix. This design is intentional. In reward-saturated groups, the on-policy GRPO gradient is zero, so an unbiased estimate of this gradient provides no learning signal. Our objective is to restore meaningful learning signals through informative negative trajectories, rather than estimate the original on-policy gradient.

For negative response filtering, all generated candidates are evaluated using the original reward function, and only verified incorrect responses are retained. If no incorrect candidate is found, no replacement occurs. During optimization, retained negatives keep their verified rewards and participate in group-wise advantage normalization and policy optimization under the original prompt. Only response tokens contribute to the GRPO loss, while the additional misconception instruction is excluded from optimization. We add these details to the Section 4.1 Experiments Settings in the revised paper.

Weakness 2 and Question 2: More Ablations

To isolate the contribution of directly optimizing negative trajectories, we conduct two additional ablations: Negative Rollout w/o Gradient: The same six ordinary rollouts and two constructed negative rollouts, but masking the gradients of the negative trajectories while keeping the group advantage computation unchanged. 6+2 Zero Padding: Six ordinary rollouts with two zero-paddings. The results are summarized below:

MethodMATHGPQAMinervaAIME24AIME25IFBench
1.7B
Negative Rollout64.8029.8018.3820.3114.3824.00
w/o Gradient63.2023.7415.0719.2710.2125.20
6+2 Zero Padding64.6026.7718.0116.6713.5422.00
4B
Negative Rollout70.6040.4026.1033.0225.5233.20
w/o Gradient69.4034.8525.3735.5215.4228.80
6+2 Zero Padding58.0026.7722.434.271.0428.80

These results show that directly optimizing negative trajectories generally outperforms both alternatives across model scales. Compared with the gradient-masked variant, Negative Rollout improves average performance by 9.5% and 9.4% on the 1.7B and 4B models, respectively. It also consistently achieves higher average performance than the 6+2 zero-padding control. These findings demonstrate that the improvements extend beyond changes in group-wise advantage estimation, highlighting the additional benefits of directly optimizing informative negative trajectories. We have included these experiments in Appendix D.1 (​​Ablation on Negative Trajectory Optimization) of the revised paper.

Question 2: Extra Costs and Learning Dynamics

We thank the reviewer for this suggestion. We further quantify the sampling and verification costs, as well as saturation persistence under independent resampling throughout training. The 1.7B and 4B models incur additional sampling overheads of 20.21% and 23.28%, respectively, with each additional rollout evaluated using a lightweight rule-based math verifier without LLM-based verification costs. For saturation persistence, the re-saturation rate varies throughout training. For 4B, with negative rollouts, approximately 20% of prompts fail to remain saturated at certain early and late training steps, while this fraction drops to around 1–5% in the middle. For 1.7B, the fraction starts at approximately 15% and generally decreases as training progresses. These results suggest that reward saturation largely persists beyond initial screening, although its prevalence changes during training.

To better understand these dynamics, we additionally track the saturation ratio, and negative generation success rate throughout training. We add these learning dynamics to the Section 6.1 Learning Dynamics of Reward Saturation in the revised paper.

Dear Reviewer,

Thank you very much for your valuable feedback on our work! As the discussion period draws to a close, we would welcome any further questions or feedback on our responses and additional results. If any concerns remain, we would be happy to address them and provide further clarification.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.