- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes AReaL-RSI, a multi-agent framework for agentic research that consists of a leader agent that assigns "harnesses" to a fixed team of researcher agents. AReaL-RSI routes findings between them under a conditional sharing protocol, where unresolved problems go to at most one additional researcher, while supported findings are shared more widely. After each campaign, the leader revises the harnesses for future campaigns. The paper shows that this framework improves against Codex's goal functionality on three task families.
Critical strengths
-
The paper's evaluation spans three quite different task families and two backbones. The result points the same way in all 12 comparisons against Codex's goal, which is consistent in direction (though see the compute concern below).
-
I appreciate that the conditional sharing protocol is a concrete and competent response to a real problem: shared failures homogenising exploration. The configuration study gives some support for combining sharing with specialised roles.
Critical weaknesses
-
I'm not quite convinced that the experiments test the paper's main claims. The recursive part of the method, which is the post-campaign harness revision (Section 4), has not yet been evaluated. The paper is framed around adaptive scaling but it fixes the team size (see my question below). As it stands, I think the paper should lean more into framing itself as a competent multi-agent research system that can improve upon Codex's goal, rather than adopting a framing around RSI.
-
My other main concern is the strength of the main comparison. The paper has Codex's goal as the only external baseline, however, Codex's goal is a single agent loop. For KernelBench, Codex was also limited to one effective hour in total, whereas each AReaL-RSI researcher receives one hour, not mentioning the leader and review costs. The paper itself notes that RQ1 does not establish equal-compute efficiency -- if so, the gains cannot be separated from the extra compute.
-
Following up on the above point, the paper discusses several research optimiser systems (AIDE, AlphaEvolve, DGM, ADAS) but does not include any of them as baselines. It's unclear to me how AReaL-RSI compares with methods designed for these tasks rather than a general-purpose agent loop.
-
The paper defines the objective in terms of (tokens, model charges, experiment compute) in Eq. 1 and Eq. 13, yet no cost is reported for any run.
Questions for authors
-
Could the authors please provide the costs for each run?
-
What was the team size in RQ1? Had the RQ1 harnesses been revised by earlier campaigns?
-
Could the author include a scaling curve? What happens if you vary the size of the agent team?
-
Could the authors include one dedicated optimiser system baseline (AIDE or OpenEvolve)? I'd be happy to revise my score if the result holds against these baselines. Additionally, could you please include a few repeated runs to account for run-to-run variance -- though I'm conscious of the potential run costs.
As a suggestion, especially for the Terminal-Bench experiment, a dedicated harness optimiser such as Meta-Harness that reports Terminal-Bench 2.0 results with released code would be a more informative comparison than Codex's goal.
Thank you for recognizing the breadth of the evaluation and the conditional sharing mechanism, and for clearly identifying the gap between the paper’s framing and its evidence. Your compute concern is well founded: allocating time separately to each researcher is not equivalent to a single agent’s total budget. The Codex-Goal comparison therefore should not be interpreted as demonstrating equal-compute efficiency. Team sizes, harness versions, and prior revision histories also need clearer reporting.
We agree that comparable-compute baselines, varying team sizes, repeated runs, and direct evaluation of cross-campaign improvement would strengthen the study. Your suggestions of AIDE, OpenEvolve, and Meta-Harness provide concrete directions for more informative comparisons. This feedback helps us define the current contribution more precisely and identify the evidence needed to support broader RSI claims.