- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates recursive self-improvement (RSI) and identifies two limitations of existing population-based approaches: fragmented knowledge across different agent lineages and the coupling of task performance with self-improvement capability. To address these issues, the authors propose Looped-RSI, a framework that periodically consolidates knowledge from multiple agents, including unsuccessful ones, to construct stronger task agents. Experiments on BabyAI and Crafter demonstrate that Looped-RSI outperforms HyperAgents, achieving better task performance with fewer evolutionary expansions.
Critical strengths
-
This paper effectively integrates complementary knowledge and valuable experiences from unsuccessful agents.
-
The method separates task-solving capability from self-improvement capability, and comprehensive experiments demonstrate improved performance, more effective recursive improvement, and strong cross-environment generalization.
Critical weaknesses
-
How do the authors determine which pieces of knowledge acquired by each agent are useful? This seems to be a prerequisite for the proposed consolidation process. From my understanding, the useful knowledge contributed by each agent is identified based on real-time feedback from the environment (e.g., Table 3 in Appendix A.3). However, how would the proposed method work in environments where such feedback is unavailable, such as the Paper Review and IMO-GradingBench benchmarks used in the baseline paper?
-
The paper evaluates the proposed method on only two benchmarks, BabyAI and Crafter, which limits the comprehensiveness of the experimental evaluation. Moreover, since the proposed method builds upon HyperAgents, the authors should at least conduct comparisons on some of the benchmarks used in the original HyperAgents paper. In particular, I suggest evaluating the proposed method on Paper Review and IMO-GradingBench to investigate how much improvement it can achieve over the baseline in these more challenging settings.
Questions for authors
See the weakness
2 comments
Dear Reviewer 2j18,
Just a gentle reminder as the discussion deadline is approaching. We hope our response and the additional clarifications&expirements addressed your concerns, and we would be grateful if you could reconsider your assessment in light of our revised response. We would also greatly appreciate any further feedback you may have before the deadline. Thank you again for your time and consideration.
Official Comment by Authors
Thank you for the constructive feedback. Your questions point to an important distinction that was not sufficiently clear in the original presentation: Looped-RSI does not require interactive environment feedback such as valid/invalid actions. What it requires is evaluation evidence about the agents generated during self-improvement. Following your suggestion, we also added experiments on the non-interactive Paper Review and IMO-GradingBench settings used by HyperAgent.
Q1. How is useful knowledge identified when real-time environment feedback is unavailable, as in Paper Review or IMO-GradingBench?
The evidence used by consolidation does not need to come from real-time environment interaction. Looped-RSI uses the same general type of signal required by the underlying self-improvement process: what changed in an agent and what happened when that agent was evaluated. The precise evidence naturally depends on the task. In BabyAI, evaluation can provide task success together with behavioral evidence such as rejected actions. In Paper Review, there is no action interface or valid/invalid feedback; self-improvement instead operates on the agent's reasoning and decision procedure. The consolidator reasons over the population's modifications, measured task outcomes, and self-improvement traces, and synthesizes changes supported by this evidence. It does not require a predefined notion of what the useful capability should be.
Paper Review provides a particularly clean demonstration. Looped-RSI and HyperAgent share exactly the same first 15 generations, and all 15 obtain zero validation accuracy. At the first consolidation boundary, task-agent consolidation is rejected, but improver consolidation is accepted. The consolidated improver then changes the subsequent trajectory: Looped-RSI reaches 68% validation accuracy by generation 20 and 90% by generation 29, while HyperAgent remains at zero through generation 30 and eventually reaches 84%. Thus, useful self-improvement can be recovered without action-level feedback, even when the current population contains no positive task-level reward.
The key requirement is therefore evaluation feedback, not interactive feedback. If there were no signal at all for evaluating generated agents, neither Looped-RSI nor a selection-based method such as HyperAgent would have a basis for identifying useful changes. This is a requirement of the underlying RSI setting rather than an additional assumption introduced by consolidation.
Q2. Does Looped-RSI also improve over HyperAgent on the non-interactive benchmarks used in the original HyperAgent paper?
Yes. Following your suggestion, we added both datasets. For Paper Review, we use the same task formulation, prompt, and evaluation protocol as HyperAgent. The first 15 generations are shared.
Looped-RSI therefore obtains higher validation and held-out test performance while using substantially less total inference. Importantly for Q1, this improvement occurs in a setting with no interactive environment or action-level feedback. For IMO-GradingBench, we also follow HyperAgent's transfer protocol rather than performing a new Looped-RSI consolidation run on the target task. The source-domain improver is frozen and used to improve the target-domain agent:
The improvement on IMO is modest, and we do not overinterpret it. It provides complementary evidence that the improvement procedure learned by Looped-RSI transfers to a substantially different non-interactive structured-reasoning task.
We originally emphasized BabyAI and Crafter because our central claim concerns fragmented progress across branches, which is particularly exposed in open-ended settings where useful interfaces, behaviors, strategies, and failure information may be discovered separately during search. However, the new Paper Review/IMO-GradingBench results shows that the benefit is not restricted to interactive control. Beyond the two benchmarks, we also broadened the evaluation across model families and environments. With Gemini, Looped-RSI reaches 49.1% versus 9.8% for HyperAgent on Crafter (5.0× higher) and 41.7% versus 4.2% on MiniHack (9.9× higher), while using lower total API cost in both cases.
Together, the revised evaluation now spans interactive and non-interactive tasks, open-ended and structured settings, direct self-improvement and transfer, and both open- and closed-source model families. We have revised the manuscript accordingly to make clear that Looped-RSI requires empirical evaluation evidence, not real-time or action-level environment feedback.