- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper incorporates loop-based refinement into reinforcement learning. Specifically, the authors provide the model being trained with incorrect responses and corresponding failure analyses generated by a high-performing model. They also use failure diversity as a criterion for selecting training data. In my view, the core idea is interesting. However, the practical relevance of the proposed method remains unclear, and the overall framework appears overly complex and computationally expensive.
Critical strengths
- The paper is well written and easy to follow.
- The paper presents an interesting perspective by combining loop-based refinement with RL training.
Critical weaknesses
- The proposed method incurs substantial computational costs and relies on an external high-performing model to generate feedback. It remains unclear whether this approach offers advantages over directly distilling knowledge from that model.
- The comparison between LoopRL and the other models in Figure 1 appears unfair, as the other models do not have access to previous trajectories and feedback.
Questions for authors
- How can prompts for which all sampled solutions fail benefit from GRPO training? If all solutions receive the same reward, their advantages should be zero, resulting in no policy-gradient learning signal.
- Feedback from a stronger model may reveal the correct solution or the reasoning path leading to it. How do the authors rule out solution leakage as the source of the observed gains? More broadly, which contributes more to LoopRL’s performance: the previous trajectory or the feedback?
- What does the reported value of 82.5% represent? The main text describes it as pass@8 at loop 11, whereas line 697 attributes it to loop 0. Could the authors clarify this discrepancy?
- Why does feedback from later loops lead to better performance? Do the authors have empirical observations or analyses that explain this finding?
- What real-world application scenarios do the authors envision for the proposed method?
Response to Reviewer gMn3
Thank you for the comments. The stronger teacher, Qwen3-235B-A22B, only writes feedback during offline data collection. During RL, the policy generates a fresh solution and earns a reward for its final answer. At test time, the evaluated model writes its own feedback. Moreover, we trained Qwen3-4B on the same LoopRL’s selected contexts after removing their feedback. At inference time, this variant directly revises or aggregates its previous solutions, whereas standard LoopRL also has the evaluated model generate feedback. The two models reach 79.33% and 79.32% mean accuracy. These results indicate that context selection is the main source of LoopRL’s gains. Appendix B.8 reports all scores alongside the solution-only training curves.
W1 / Q2: The stronger teacher is used only for offline feedback collection. At inference, the evaluated 4B model produces both solutions and feedback itself, where feedback receives only the question and the model's solution. If the model derives the correct answer in its own feedback, that is part of its inference, not information supplied by a teacher. LoopRL studies RL from interaction states: the previous attempt and feedback condition a fresh solution that must earn a verified reward. We don’t study SFT, and thus neither stored solution nor feedback is a supervised response target.
Moreover, we add an experiment for comparing solution-only and solution-plus-feedback training (Appendix B.8). Both variants train on the same selected questions and stored solutions. Standard LoopRL (solution-plus-feedback) also conditions on stored feedback, whose tokens are excluded from the policy loss. At inference it generates its own feedback, while the solution-only model revises or aggregates its previous solution directly. Standard LoopRL scores higher on AIME 2026 and IMO-Answerbench, and the solution-only variant scores higher on both HMMT benchmarks. With selected contexts, solution-only reaches 79.33% versus 79.32% with feedback (Table 5). Figure 4 additionally demonstrates the selection benefit on the same questions. These results indicate that context selection is the main source of LoopRL’s gains; explicit feedback provides no gain in mean accuracy.
W2: We report question-only loop-0 accuracy: 63.8% for LoopRL-Qwen3-4B and 69.0% for LoopRL-Qwen3.5-4B. During iterative inference, each evaluated model generates its own solutions and feedback. Figure 5 compares base and trained models under the same feedback scaffold. We additionally give QED-Nano access to previous trajectories and self-generated feedback through this same scaffold (Appendix B.9, Table 7). It reaches 76.19% at 57.85 effective PFLOPs, compared with 76.44% at 33.21 for LoopRL-Qwen3-4B and 82.50% at 42.81 for LoopRL-Qwen3.5-4B. Applying extended scaling to all larger models exceeds our inference resources.
Q1: The reviewer is correct that a fresh GRPO group with identical rewards has zero group-relative advantages. “All incorrect” describes eight offline continuations used to characterize a context. Training samples a fresh group from the current policy, which can contain both correct and incorrect solutions. Zero offline successes therefore does not imply zero training signal. Diversity distinguishes these failure states: diversity=2 contexts reach 79.8%, compared with 75.0% for diversity=1 contexts.
Q3: The headline 82.5% is LoopRL-Qwen3.5-4B's mean accuracy at inference loop 11. Appendix B.2 reports a separate result: LoopRL-Qwen3-4B's loop-0 pass@8 is also 82.5%.
Q4: Appendix B.7 already gives a possible way to explain the result that different loops lead to different RL outcomes. We compute NLL of prompts from loops 1–4 and compare it with mean prompt length. Mean prompt NLL decreases from 1.1473 at Loop-1 to 0.9855 at Loop-4, while prompt length is not monotonic. (Table 6)
Q5: Agent harnesses repeatedly propose, observe, and revise. Code generation with test failures, proof search with verifier feedback, and tool-using agents all produce reusable intermediate states. LoopRL provides a training recipe for these loops: select useful states and train short on-policy continuations, even when deployment uses repeated revision or aggregation.
In conclusion, we systematically studied which data from multi-loop reasoning trajectories are most useful for RL training, and proposed the data selection method from the experiments. Training on the selected 5.2K contexts brought a 4B model to the IMO gold medal level. The 4B model reaches 82.5% on IMO-AnswerBench, above the reported 80.0% score of Gemini Deep Think (IMO Gold) on that benchmark. We release all training code and data to enable reproduction and further study of RL on feedback loops.