- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RLVR will fail when task is difficult that the agent has a low or even no chance of success. This paper enables effective RLVR by continuously generating feedback during iterative self-improvement and incorporating it into subsequent training to enhance the agent's trial success rate. Experimental results illustrate that training on only 4k of these tuples recovers almost the full performance of RLTL;DR and classical SFT on full rollouts.
Critical strengths
- This paper enables effective RLVR by continuously generating feedback during iterative self-improvement and incorporating it into subsequent training to enhance the agent's trial success rate.
Critical weaknesses
- The paper only used Qwen 3.5 9B for experiments, and the reliability of the experimental conclusions is insufficient
- The paper only used three datasets for the experiment and the Synthetic API is a pure synthetic environment. Moreover, the method proposed in this article has significantly poor performance on Leetcode as shown in Table 1. The effectiveness of our method is questionable.
- The paper does not have open source code. This makes me doubt this article.
Questions for authors
- Can you provide more model experimental results besides Qwen3.5-9b?
- Can you increase the experimental dataset to observe whether this method is truly effective in real-world scenarios?
2 comments
Confirm
I have reviewed the rebuttal and maintained my score
We would like to thank the reviewer for their time. We’re happy to provide clarifications below.
Thanks for the suggestion! We have now replicated our experiments on the Granite 4.2 8B model. We observe the same effect: GRPO fails to learn, whereas our insight internalization breaks through this learning barrier. This worked out of the box, without any editing to the insight generation or internalization logic or hyperparameters. This is unsurprising because our method is simple and generic, agnostic of the model. The numbers below are Pass@1 on rollouts without insights in context on the SAPI frontier-difficult split, just like in the main paper.
Synthetic API is not "a pure synthetic environment" in the sense of some toy task. It is the largest collection of realistic on-device tasks to our knowledge, and training on it gives SOTA performance on other benchmarks, (e.g., the human-written Appworld benchmark with real-world on-device tasks) [1].
We do not see which results would permit to conclude that the method is ineffective. It learns on all three datasets on tasks where the agent has Pass@128=0 and GRPO flatlines.
The only result that is negative is the generalization in coding tasks, in which all baselines we tested do not generalize to the test set either. We believe it's important to sketch limitations, and we communicated this openly in the main text, conclusion, and discussion in the originally submitted manuscript. We can highlight it a bit more in the revised version if you deem it important.
We unfortunately cannot provide full source code for legal reasons. But we do provide a full step-by-step reproduction guide in Appendix B, disclose all training hyperparameters and prompts in Appendix A, and provide a guide on how to evaluate correctly by deconfounding evaluation in Appendix D (which we hope is going to be helpful even outside our own experiments). We invite to also have a look at the experiments requested by the other reviewers. All of them confirmed the effects we have reported in the paper.
After the ICLR deadline, two other papers have independently confirmed our paper's claims. [2, 3] So if your argument is that you doubt the article because there is no code (which in the era of AI slop papers might be a good prior), we think it's apparent our article shouldn't fall into this category.
See our response to W1.
We provide results on three datasets which all have the same conclusion. All three are "real world scenarios". We'd be happy to cater for you if you provided more details on what experiment you want to see.
Let us know what concrete evidence you would like to see to move from a clear rejection score.
[1] Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis, Szot et al., 2026, https://arxiv.org/abs/2610.37648
[2] Shockingly Simple Self-retrospection Improves Agentic Models Without RL, Jonathan Light et al., 2026, https://openreview.net/forum?id=fcQaMnBdzB
[3] Teaching NVIDIA's Nemotron 3.5 Super-VL ML Engineering, Bespoke Labs, 2026, https://bespokelabs.ai/blog/teaching-nvidias-nemotron-ml-engineering