- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to sovle a main challenge of RL that rollout quality collapse when given hard prompts. The paper claims that exsiting methods solve this by resampling or skip such prompts, which skew its distribution or introduce context mismatch. To close this gap, this paper propose SAGE, in which the policy model generate tiwce: (1) first generate a hint given a hard prompt and its reference solution, (2) then the policy model rollouts given the generated hint and the hard prompt. The following process of rewarding and policy updating are based on the second generation. During the test time, the hints are removed from the policy models' context but just given prompts. The experiments show that SAGE achieves the best average among all baselines (GRPO, LUFFY, Scaf-GRPO, SFT) with three LLMs (Llama-3.2-3B, Qwen2.5-7B, Qwen3-4B) on six mathematical benchmark.
Critical strengths
-
Simple and practical method: The rollout quality collapse is an important challenge in RL. This paper aims solve this with only adding self-generated hints, keeping reward strategies unchanged. So it is easy to plug into existing training pipelines.
-
Relatively extensive experiments: The paper conducts extensive analytical experiments, including main comparisons with existing methods on both easy and hard training sets across three LLMs, ablations of the key design choices, and an analysis of training dynamics. SAGE shows relatively consistent advantages over the baselines cross these experiments.
Critical weaknesses
- The motivation is somewhat incremental, and the comparison with existing methods is not well supported. The paper claims that existing methods "skew the training distribution or introduce context mismatch", but SAGE does not appear to solve the context mismatch it criticizes. For example, compared with Scaf-GRPO, which already triggers hints only on collapsed groups and samples hinted rollouts on-policy, SAGE mainly introduces online self-generated hints. The context mismatch seemly remains: training is conditioned on the prompt plus the hint, while at test time the model sees only the prompt. In fact, since this paper discards more no-hint rollouts, the mismatch might be larger. So possibly the main gain lies self-generated hints provide a better-calibrated curriculum during training.
- The context mismatch may need direct analysis and evidence. Figure 7 shows that the number of prompts failing without hints decreases during training, but it does not tell where the improvement comes from. Since batches are randomly sampled, the improvement might come from the model's general improvement through RL on non-hinted prompts, rather than from the hinted updates. A GRPO counterpart of Figure 7, or tracking the prompts previously trained with hints, may help.
Questions for authors
- Could the authors clarify more about the motivation and the gain over existing methods, like Scaf-GRPO?
- Could the authors add a direct analysis of the context mismatch?