- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to mitigate entropy collapse in RLVR training and the resulting drop in pass@k at large k. The authors propose introducing partial solutions generated by weaker small models (Gemma-2-2B and LLaMA-3.2-1B) as prefixes during GRPO training. Specifically, the weak model first generates a complete response. The response is then truncated at the position with the largest entropy drop, measured by the magnitude of entropy change. The target policy model continues generation from this point, and its gradient updates are applied only to the completion it generates itself. Notably, no prefix is used at evaluation time; instead, the policy model directly generates the complete response. Experiments on mathematical tasks are conducted with two Qwen2.5 models.
Critical strengths
- The method is simple and fairly interesting. It introduces generations from another model to avoid diversity collapse during RL training, giving the policy an opportunity to learn alternative reasoning paths.
- The evaluation takes practical value into account by returning to the standard paradigm of direct generation by the policy model.
- Generating prefixes with a small model is relatively cheap and does not introduce much additional overhead.
Critical weaknesses
- Limited novelty. Using partial solutions as prefixes or hints in RLVR training and letting the model continue the remaining reasoning has already been explored in multiple works accepted at top conference [1–3]. The main difference in this paper is only that the prefix comes from a weak model and is interpreted as a “perturbation.” However, the paper cites only QuestA and does not compare against any prefix- or hint-based method in its experiments.
- No correctness check on weak-model prefixes. The paper does not perform any correctness check on the prefixes generated by the weak model. Two situations can be expected: (1) when the prefix is wrong, all 8 continuations often fail; (2) when the prefix is wrong, the subsequent reasoning deviates from the prefix’s logic and reaches the correct answer by chance, and this kind of error is also reinforced.
- Relation to data augmentation and off-policy training. This paper trains the model on states produced by an external model, which is closer to data augmentation or an off-policy state distribution. The paper should discuss this more.
References
[1] QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation. ICLR 2026.
[2] BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning. NeurIPS 2025.
[3] StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason. ACL 2026.
Questions for authors
See above.
We thank the reviewer for the detailed comments on related work and the training mechanism.
Regarding the distinction from prior work, we agree that the original manuscript did not sufficiently discuss QuestA, BREAD, and StepHint. We will expand the related-work section and clarify that our novelty does not lie in the general idea of using partial trajectories as prefixes. Rather, we investigate a different role for auxiliary trajectories: instead of treating them primarily as correct hints or demonstrations, we use trajectories from models with different generation distributions as training-time conditioning contexts, even when their semantic guidance is imperfect. The target policy generates and learns from the continuation, while the auxiliary prefix itself is excluded from the policy loss. No auxiliary prefix is used during inference or evaluation.
Regarding incorrect or misleading prefixes, we agree that they can negatively affect the conditioning context and potentially lead to unproductive rollouts. In our implementation, all eight rollouts for a given question share the same prefix. If all eight completions receive zero reward, their group-normalized advantages are zero, and this group provides no effective policy-gradient signal under the standard GRPO formulation. Such cases therefore waste rollout computation and reduce the effective training signal. However, if the target policy successfully continues from a prefix and obtains a non-zero reward relative to other completions, the generated suffix can still receive a useful learning signal. Importantly, the prefix itself is never directly optimized. We also retain 80% question-only GRPO rollouts, which allows the target policy to continue learning from the original question distribution rather than relying entirely on auxiliary contexts.
Regarding the on-policy nature of the method, we agree that describing the entire training procedure as standard question-only on-policy GRPO would be imprecise. The suffix is sampled from the current target policy conditioned on an externally generated prefix, and its policy loss is computed using the target policy. However, the context distribution includes externally generated prefixes and therefore differs from the original question-only rollout distribution. We will describe the method more accurately as GRPO with a mixed context distribution, or mixed-context trajectory augmentation. Unlike conventional distillation, our method does not optimize the target policy to imitate the auxiliary prefix, nor does it require auxiliary-model token probabilities for the prefix. The learning objective is applied only to the target-generated continuation.
We will revise the method description to make this distinction explicit and discuss the potential negative effects of unproductive prefixes, including reduced effective rollout efficiency.