- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies an iterative way of using AI-scientist systems, called Runtime Agent Coordination (RAC). It allows existing agents to choose the next agent based on the current research state, with task contracts and output verification. The authors evaluate RAC on three AI-scientist systems.
Critical strengths
Studying cooperation among agents in AI-scientist systems is a meaningful direction. Research tasks often require several rounds of work, and understanding how agents should coordinate could help improve these systems.
Critical weaknesses
-
The contribution beyond existing coordination methods is not clear enough. The experiments compare RAC with each system’s native workflow and RAC’s own variants. They do not directly compare it with other established multi-agent coordination methods applied to the same AI-scientist systems.
-
The RAC framework does not consistently improve performance across the three systems. For some systems, estimated costs increase substantially while performance improves only slightly or remains almost unchanged.
Questions for authors
-
What does RAC add beyond existing dynamic delegation methods?
-
Is there a direct comparison with another coordination method under the same experimental setup?
2 comments
Thank you for the response. However, my main concerns remain unresolved.
The proposed coordination mechanism should be compared with existing coordination methods in end-to-end AI-scientist workflows under comparable settings. I consider this comparison central to the paper’s motivation. It would help discuss which coordination mechanisms work well for AI scientists and provide a basis for further work. The ablations of RAC’s own components are useful, but their value is limited.
So I lean toward keeping my original score.
Thank you for raising novelty, external baselines and cost-effectiveness.
Contribution relative to dynamic delegation. RAC provides a shared coordination intervention across existing AI-scientist hosts and an experimental separation of communication from successor selection. R1 adds communication with the native successor; R2 changes selection under the same communication; R3 adds contracts and advisory verification. Keeping native capabilities, tools and permissions fixed within each host makes the R1–R2 contrast informative about changing who acts next.
AutoGen supports flexible multi-agent conversations, Magentic-One orchestrates specialists with progress tracking and replanning, and MetaGPT organizes collaboration around operating procedures. RAC builds on these broad coordination ideas. Its contribution is the cross-host intervention and comparison, rather than any one delegation or feedback component in isolation.
External baseline. We have not completed a comparison with an external coordinator under the same host, tasks, model and budget. This remains a gap. The current experiments answer whether changing coordination improves each host relative to its native workflow and communication-only condition; they do not establish superiority over other coordination methods.
Additional executions. With GPT-6 Luna (medium), R2 again exceeds R1 on average in all three hosts. The additional study covers 10 tasks each for Agent Laboratory (AL) and EvoScientist (Evo), and five for ARK. Scores are on a 0–100 scale.
R2 beats R1 on 5/10, 6/10 and 4/5 tasks, respectively. For Δt=sR2,t−sR1,t, 100,000 within-host task-pair bootstrap resamples (seed 20261117) give 95% percentile intervals of [−2.015, 2.450], [−2.410, 4.860] and [−0.790, 5.840]. All include zero. The positive mean direction recurs across the original DeepSeek V4 Pro and additional Luna cohorts, but each additional cell has one selected execution; these intervals describe task-mix sensitivity, not fixed-model execution repeatability. Each score is the median of three judge evaluations.
Component ablations. We added contracts-only (C) and verification-only (V) conditions. Means are:
Evo Math_003 R2+C exhausted its budget without a report; it is missing, not zero. The complete ARK ablations reinforce that neither added component improves R2's aggregate mean. Verifier-enabled error/completion handling also differs, so the ablations compare implemented component variants.
Resource use and realized routing. For the additional ten-task AL cohort, R1/R2/R3 mean input usage is 0.810/0.740/1.011M tokens, output usage is 74.63/76.60/69.04K, and wall time is 39.69/34.81/27.08 minutes. Evo's corresponding values are 2.232/2.331/2.263M, 35.34/34.38/36.42K and 15.14/15.90/15.42 minutes. The small R2−R1 gain accompanies lower input usage/runtime in AL but slightly higher usage/runtime in Evo. The resource benefit is therefore host-dependent. Subscription billing is unavailable; original dollar estimates use a standardized token tariff.
Routing needs both an actionable issue and an opportunity to act. AL R2 follows the native phase sequence, and none of the 30 AL R1–R3 traces selects a capability after report refinement. In ARK Physics_002, all 11 R1 post-review handoffs go to the planner, while R2 includes three direct handoffs to the coder. After one, the hop-18 coder reported parsing 4,610 JSON inputs, producing results for 550 eligible XEB instances and regenerating figures, while retaining unresolved scientific gaps. This shows how runtime selection can send a concrete review issue directly to a role that performs the corrective work.
The R3 cases further distinguish local checks from useful correction. AL Physics_002 V-only receives seven supported verdicts despite reporting no empirical estimates and using synthetic plotting demonstrations. ARK Neuroscience_002 R3 repeatedly records a writer read-access blocker rather than revising the manuscript. In these cases, local verification passed without target measurements, or the contract prevented a needed revision. Adding checks therefore did not ensure completion of the underlying research task.