- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The main idea of the paper is to introduce a curriculum step before applying GRPO to a language model being trained with verifiable rewards. The new step modifies the rollout sampling to increase the rate of positive examples (aiming for 50% of the pool). It does this by using an originally sampled positive example's (called a "sibling") prefix as a hint and resampling additional complete candidates, then keeping the positive examples produced by that sampling. The paper calls these "conditional positives." The overall method is called "success conditioned reconstruction and augmentation" (SCRA).
The authors train a Qwen model using DAPO-Math-17k and test on six benchmarks. The method matches or slightly outperforms the alternative approaches.
The technique is reasonable and for the narrow case of verifiable reward RL on language models seems worth considering as a learning stage prior to GRPO. The paper might be reasonable as draft documentation for a pull request to a conventional GRPO codebase, adding the new idea as a feature. It does not offer any illumination of the tasks being considered, the datasets or models used in experiments, or RL in general.
Critical strengths
The basic idea of rebalancing the pools during an initial phase of learning is reasonably simple, even if the exposition makes it sound complicated.
The method achieves gains of 0.5-2 points absolute on five of six tests, compared to the strongest baseline (HIPO) and nearly matches it on the sixth.
Critical weaknesses
The paper dives in to discussing the details of an RL algorithm without giving much context; it doesn't explicitly state, for example, what RL is being applied to in this paper (language models as policies) or what the primary problem that the authors are trying to solve is. The assumption -- for which no evidence is given from prior literature, and no intuition is given, only preliminary experimental support -- is that RL works better with a balanced set of positive and negative examples for each prompt. Section 3, where the method is introduced, has the same problem: it dives into mathematical expressions without situating the reader at all. The result of this jarring writing choice is that the paper fails to make the novelty of the core idea clear relative to the many existing methods.
Meanwhile, the related work section reads like a laundry list of variations on the GRPO theme, with no attempt to synthesize or abstract into deeper lessons learned. The proposed method therefore comes across as just one more RL hack with no intuitive motivation, and the reader is left thinking that the novelty is likely overplayed. It's also entirely unclear whether these competing methods are actually orthogonal tricks that could be combined with the new idea.
The paper needs more explanation of the choice of baselines and their hyperparameters. Why were these chosen? Is the computational effort that went into tuning each of these matched fairly? The N=16 GRPO run is appreciated; more detail is needed to reassure the reader that the rest of the comparisons are reasonable. One has the feeling that the experiments after table 1 are basically reporting the exploratory work as if it were a sensitivity analysis. No generalizable lessons are derived from these experiments.
In many places the text is very terse, assuming the reader is deeply familiar with the ideas that the paper ought to clearly explain in a self-contained way. Examples:
- "With binary rewards, uniform-outcome groups yield zero relative advantages" -- relative to what? (line 078)
- "using a reflection template T" -- this is left undefined and undescribed (line 116); in figure 1 it becomes T(x, s) but s is also never defined. Around line 198 we get a little more discussion of the templates, but the paper doesn't explain what they are or their purpose. One must go to an appendix to see examples.
- "The selected trajectories are trained against the original no-hint prompt, with tokenwise behavior correction for the conditional samples." -- trained using what method/objective function? Is this basically GRPO with a different strategy for choosing the rollouts? It's never stated directly. (line 120)
- Section 3.3's style is especially hard to follow. I think it's an attempt to justify the 1:1 ratio, but it's full of unexplained concepts (e.g., "terms with no selected trajectories," "reconstruction ... compares", "hinted group control," "complete reconstruction rules") and there's no conclusion to the section that helps the reader understand what the point of the math was.
- Ideas that are fairly simple are given way more notation than they need; equations 3 and 8 are examples.
- "rebased" is used repeatedly but never defined
The authors ran multiple seeds for some experimental conditions but leave any measure of variation or confidence out of the table (some values are given in the text). The very high precision in the reported scores is suspicious.
The paper makes no attempt to quantify how often the rollouts are unbalanced (rare positive cases) during conventional algorithms' training runs on the datasets under consideration. There is no analysis of what conditions lead to this issue, e.g., what kinds of problems the method will be most useful for.
Questions for authors
- Why 81 steps of SCRA? How was this value chosen? Because there's no discussion of this hyperparameter choice, one suspects that it might have been selected because it gave the best results on the final evaluation -- which would of course be cheating. Same question about the prefix length of 192 tokens.