- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper addresses the lack of environments and tasks for improving RSI agents, and proposes a scalable pipeline for generating them. Specifically, it uses published papers as sources, then runs a multi-agent workflow to generate task environments and corresponding baselines, and uses LM judges to score them. The authors used this approach to build numerous tasks and environments, running experiments with different models across 120 of them to demonstrate the pipeline's effectiveness.
Critical strengths
-
the problem this paper tries to solve is crucial and very practical. If our AI agents need to continuously improve their research capabilities, we genuinely need a mature pipeline to automatically generate a large volume of high-quality research tasks with automated evaluation.
-
the scale and coverage of the tasks built in the paper are impressive, featuring 210 environments, 18 research domains, and multiple models. Furthermore, all the generated tasks have been evaluated by human experts.
Critical weaknesses
- regarding the value of the tasks generated by this pipeline: I'm skeptical about whether they're actually useful for an agent to learn new capabilities, and that's my main concern. For instance, the RSI experiment in the paper only consists of three sessions, with the first two lasting just one hour, and most of the gains come from the very first session. This suggests the tasks might not be particularly difficult, meaning they may not really help agents learn new skills. Furthermore, since these environments are derived from papers, we don't know whether the generated tasks genuinely capture the research problems explored in the original papers or if they're just oversimplified versions. If they are oversimplified, they might be too basic to support a true RSI experiment.
- the pipeline uses a multi-agent workflow where an author agent generates the content, a reviewer reviews it, an independent reproducer reproduces the baseline, and then another reviewer reviews it again. However, the paper provides no ablation studies to justify whether all these design choices are actually necessary. For example, I could potentially have a single, powerful agent generate the tasks, evaluation, and baseline algorithms all at once based on the paper. While that might consume more upfront compute cost, if the multi-step design doesn't bring a significant improvement in task quality, wouldn't a one-shot generation approach be better?Also, the authors claim the pipeline is scalable. When it comes to scalability, there are two aspects: whether it can automatically generate environments in bulk, and what it takes to produce a high-quality environment. Regarding the former, it remains unclear how much capital, compute, and time are required to generate a single high-quality environment, and we don't yet know whether this approach scales effectively or not.
Questions for authors
-
are these generated environments sufficient to support long-term self-improvement? Specifically, if we run real RSI on them, won't agent performance plateau after the first couple of sessions?
-
regarding the multi-agent pipeline built in this paper for generating task environments: how important is the independent reproducer agent within it? I think that adding more agents increases the pipeline overhead, so I'd like to know what the advantages are of using this multi-agent pipeline compared to just using a single, capable agent to generate all the environments and tasks directly.
6 comments
Dear Reviewer uP8v,
Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.
Best regards,
The Authors
Thanks for the authors' response, but my core confusion remains unresolved. First, regarding the ratings by domain experts, the paper shows that on the resistance-to-saturation dimension, the human domain experts scored it very low. This demonstrates that they consider these tasks prone to saturation. Second, regarding the authors' mention in the rebuttal that it improves its best-graded solution by more than 0.01, I find this 0.01 magnitude hard to interpret. To me, it simply means that this scale is already saturated. Additionally, the authors noted that these sessions fail to establish long-term capability gains. Therefore, while the paper describes a scalable task generation pipeline, I don't believe the current results are sufficient to show that this is a meaningful approach for recursive self-improvement tasks. Correct me if you think my interpretation is wrong.
Thank you for the thoughtful comments. We address W1/Q1 together and W2/Q2 together below.
W1/Q1: Value of the generated tasks.
We'd like to highlight a few findings that showcase the value of the generated tasks, focusing on difficulty, fidelity, and room for continued improvement.
To assess difficulty beyond the starting solution, we compare agents against an independently reproduced paper method under the same task interface and execution budget. Models surpassed that reference in 67/120 environments after excluding all three sessions of model-environment chains flagged for access to restricted materials. Our transcript coding identifies work beyond parameter tuning in 94% of the attempts that surpassed the reference. These comparisons assess progress against a reproduced research method within the adapted task (Sections 5.2-5.3).
On fidelity, domain experts reviewed 90 of the 180 main-run environments with access to the source papers, giving mean ratings of 4.39/5 for research content and 3.72/5 for faithfulness. Their assessments support the presence of research content while identifying losses in fidelity from adapting problems to bounded compute (Section 4.4). The improvement result also holds in the higher-fidelity subset: 24/34 environments rated 4-5 show later gains (Appendix J.2).
Although most gains occur early, improvement continues beyond the first session. After the same whole-chain exclusion, at least one model improves its best graded solution from session 1 to session 3 by more than 0.01 in normalized held-out score in 100/120 environments (83%). This includes 15/20 environments rated 4-5 for resistance to saturation. Some experts saw solver summaries, so the subgroup comparisons are descriptive. All three sessions have the same 60-minute cap, including the third, with fixed model weights. The evidence supports the tasks' use for iterative solution improvement; three sessions cannot establish long-term capability gains or rule out later plateaus, and do not isolate inherited records from additional attempts. We have updated the results in Sections 5.2-5.3, added the expert-rated subset analysis in Appendix J.2, and clarified the scope in Section 6.
W2/Q2: Value and cost of separate reproduction and review.
To test whether the separate agents are worth the overhead, we took 15 papers for which our pipeline had completed a build and asked one agent to build, reproduce, and review each environment in a single session. We kept the model, prompts, and checks the same. With separate agents, the reproducer works from the paper and solver materials without seeing the author's implementation.
The single-agent version completed 11 of the 15 builds. In the other four, it stopped because it could not independently review work it had created itself. Even among the 11 completed builds, nine were marked as passing while the review report said an independent assessment was not possible. None of the 15 builds with separate agents had this contradiction (Appendix G.2).
Taking one of the tasks as an example, the solver was supposed to use observations as they arrived, but the environment built by a single agent also exposed future observations. When we changed only those future observations, the solver's first output changed in all 12 tested runs. The environment built with separate agents supplied observations one step at a time, and the same test changed the first output in none of the 12 runs (Appendix G.3). The broader robustness tests were tied at 9 of 11 tasks.
For these 15 builds, recorded model costs were 6,083 (USD) with separate agents and 5,811 with one agent, a 4.7% difference. The single-agent cost includes the four incomplete builds. In the main production run, we built 180 environments over 14 days. The recorded median cost was 377 per environment across the 163 builds with usage logs. Appendix G.1 gives the cost accounting and logging coverage.
We hope these clarifications and additional results address your concerns. Please let us know if there is anything else we can clarify.