- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies GRPO on reward-saturated prompts, where every sampled response is correct and the group-relative advantage vanishes. It compares interventions at four stages of training: perturbing the prompt, changing rollout generation, adding auxiliary rewards, and padding the advantage computation. The main proposal asks the current policy to write a plausible but wrong solution and adds it to the group as a negative sample. Training Qwen3 models only on saturated MATH problems, the paper finds this works best and adds analyses on how negatives are built, mixed data, iterative recycling and another model family.
Critical strengths
The negative rollout trick is simple, since it only needs an extra instruction in the prompt, and it beats GRPO on both Qwen3 sizes, with the gain holding across three seeds on the 1.7B model. For anyone who currently throws away prompts the model always solves, this is a cheap way to get some use out of them, and it is the main reason I see value in the paper.
I also found the comparison between different ways of restoring variance instructive. Zero padding makes the advantages non-zero without adding any wrong trajectory and does not help much, and swapping only the final answer of a correct solution actually hurts. So what seems to matter is having a coherent wrong solution to push away from, not just a non-zero advantage. That is useful to know for anyone working on zero-variance groups.
Critical weaknesses
The paper starts from the claim that GRPO gets almost no signal from saturated data, but its own GRPO baseline improves quite a bit on this data, especially on the harder math benchmarks. That suggests many groups stop being all-correct once training starts, which the paper never checks. So I cannot tell whether negative rollouts restore a missing signal or add a negative gradient on top of an existing one. I also could not find whether the loss on the negatives uses the original prompt or the prompt asking for a mistake, and the two would teach different things.
The paper is pitched as a comparison of which part of the pipeline best recovers signal, but each part gets one or two configurations, untuned and run once. The authors' own seed runs and mixed-data experiment show results moving between runs about as much as the methods differ, so I do not think the ranking across levels holds up. What I am convinced of is the narrower point that negative rollouts beat plain GRPO. The recycling and LLaMA experiments have a similar problem: the first has no matched baseline, and the second shows a small, mixed gain.
Questions for authors
- How often are groups actually all-correct during training, under GRPO and under each intervention, and how does that fit with the gains plain GRPO gets on this data?
- Is the loss on the constructed negatives computed with or without the wrong-answer instruction in the context?
- Do you check that the constructed negatives are actually wrong, and what happens when the model still answers correctly?
3 comments
Response to authors
Thank you for clarifying the implementation and for adding the training dynamics and ablations. These address several of my concerns and support the benefit of the negative rollouts.
However, my concern about the comparison across intervention levels, which is central to the paper, remains. In Table 1, Negative Rollout leads the strongest alternatives from other levels by 0.83 points on Qwen3-1.7B (zero padding, k = 3) and 0.98 points on Qwen3-4B (reward model). These alternatives were run only once, so it is unclear whether these margins hold across seeds. The recycling experiment still lacks a matched baseline, and the gain on LLaMA remains small (0.66 points over GRPO). I recognize the improvement over GRPO, but the evidence for the broader conclusions remains insufficient to change my assessment. I therefore keep my score at 2.
Thank you for your constructive feedback. We have conducted additional experiments and would like to clarify the following points.
Multi-seed evaluation of competing interventions. To address the concern about single-seed comparisons, we evaluated the strongest alternative intervention for each model using three random seeds.
| Model | Intervention | Seed 1 | Seed 2 | Seed 3 | Mean ± Std |
|---|---|---|---|---|---|
| Qwen3-1.7B | Zero Padding (k=3) | 30.70 | 30.54 | 29.26 | 30.17 ± 0.79 |
| Qwen3-4B | Reward Model | 40.24 | 39.19 | 38.81 | 39.41 ± 0.74 |
These results provide a more reliable estimate of the performance of the strongest competing interventions and strengthen the comparison across intervention levels.
Recycling experiment. Our primary goal is to demonstrate a potential setting for reusing saturated data, rather than to establish that recycling outperforms all alternative data augmentation strategies. We acknowledge that additional training samples can naturally improve performance and will clarify the scope of this experiment in the revised paper.
Generalization to LLaMA. The LLaMA experiment primarily aims to demonstrate that Negative Rollout can generalize beyond the Qwen model family. While the improvement is relatively modest, it provides initial evidence of cross-family applicability. We will consider evaluating additional model families and scales in future work.
Question 2: Implementation Details
We thank the reviewer for highlighting the need for clarification. The negative prefix is used only during response generation, while all policy log-likelihoods and KL regularization are computed under the original prompt, without the additional prefix. This design is intentional. In reward-saturated groups, the on-policy GRPO gradient is zero, so an unbiased estimate of this gradient provides no learning signal. Our objective is to restore meaningful learning signals through informative negative trajectories, rather than estimate the original on-policy gradient.
For negative response filtering, all generated candidates are evaluated using the original reward function, and only verified incorrect responses are retained. If no incorrect candidate is found, no replacement occurs. During optimization, retained negatives keep their verified rewards and participate in group-wise advantage normalization and policy optimization under the original prompt. Only response tokens contribute to the GRPO loss, while the additional misconception instruction is excluded from optimization. We add these details to the Section 4.1 Experiments Settings in the revised paper.
Question 1 and 3: More Learning Dynamics
We further track the original group saturation ratio throughout training, which averages 78.30% for Qwen3-1.7B and 91.05% for Qwen3-4B. This indicates that reward saturation remains prevalent during training, while the remaining non-saturated groups still provide learning signals that explain the improvements of vanilla GRPO. To better understand these dynamics, we additionally track the saturation ratio, and negative generation success rate throughout training. We add these learning dynamics to the Section 6.1 Learning Dynamics of Reward Saturation in the revised paper.
More ablations
To isolate the contribution of directly optimizing negative trajectories, we conduct two additional ablations: Negative Rollout w/o Gradient: The same six ordinary rollouts and two constructed negative rollouts, but masking the gradients of the negative trajectories while keeping the group advantage computation unchanged. 6+2 Zero Padding: Six ordinary rollouts with two zero-paddings. The results are summarized below:
These results show that directly optimizing negative trajectories generally outperforms both alternatives across model scales. Compared with the gradient-masked variant, Negative Rollout improves average performance by 9.5% and 9.4% on the 1.7B and 4B models, respectively. It also consistently achieves higher average performance than the 6+2 zero-padding control. These findings demonstrate that the improvements extend beyond changes in group-wise advantage estimation, highlighting the additional benefits of directly optimizing informative negative trajectories. We have included these experiments in Appendix D.1 (Ablation on Negative Trajectory Optimization) of the revised paper.