- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AutoResearchClaw, a multi-agent framework for autonomous scientific research. The framework consists of 23 stages, covering the entire research process from hypothesis generation, experiment design, and code execution to result analysis and manuscript writing. The authors mainly combine several mechanisms, including multi agent debate, self-healing experiment execution, result verification, human intervention, and cross-run experience accumulation. The system can automatically repair code after experimental failures, reconsider research hypotheses when necessary, and leverage experience from previous research runs to improve future performance. The authors also introduce ARC-BENCH, a benchmark containing 55 research topics across five scientific domains. Experimental results show that ARC outperforms AIDE and AI Scientist v2 on machine learning tasks, while cross-run experience accumulation further improves performance. The authors also conduct ablation studies to analyze the contributions of different components and find that targeted human intervention at critical stages is more effective than full autonomy or step-by-step supervision.
Critical strengths
-
The paper presents a valuable system design for improving the traceability and reliability of experimental results in autonomous scientific research. The paper goes beyond evaluating whether autonomous research systems can successfully execute experiments and places particular emphasis on the authenticity and verifiability of experimental results. The authors introduce a Measurement Registry-based verification mechanism that programmatically checks numerical claims in generated manuscripts against actual execution records, reducing the risk of reporting unsupported experimental results. The ablation study further demonstrates the effectiveness of this mechanism: without the registry, 3 out of 5 manuscripts that passed the quality gate contained numerical claims unsupported by experimental records, whereas none of the 3 passing manuscripts produced by the full system exhibited such issues. Despite the limited evaluation scale, this mechanism provides a practically meaningful approach to improving data provenance and research integrity in autonomous scientific research systems.
-
The cross-run self-reinforcement mechanism is a valuable research direction and demonstrates promising empirical results. The proposed Cross-Run Evolution mechanism enables autonomous research systems to extract reusable lessons from previous experimental failures, research decisions, and execution anomalies, and incorporate these lessons into subsequent research cycles. This design extends feedback-driven improvement beyond individual research runs, providing a practical approach to accumulating and reusing experience across research cycles. The experiments show that three independent memory-enabled first-iteration runs achieve an higher average score compared with the no-memory baseline. The overall score further increases in a persistent sequence with accumulated experience. Although the improvements over successive iterations would benefit from more extensive independent replication, the reported results provide encouraging empirical evidence for the potential value of cross-run experience accumulation.
Critical weaknesses
-
The experimental validation of cross run self-reinforcement remains insufficient to fully support the claimed ability of continuous improvement. Cross-Run Self-Reinforcement is one of the paper's central contributions, but the current experiments do not sufficiently establish its effectiveness and generalizability. Although the authors report a substantial improvement from 0.564 to 0.827, the three-iteration results are based on only a single persistent trajectory, without multiple independent replications to assess the stability of the improvements. The experiments also repeatedly use the same ML01 - ML25 tasks, meaning that the accumulated experience may contain task-specific repair strategies and experimental information. As a result, it remains unclear whether the performance gains come from generalizable research experience or adaptation to previously encountered tasks. Further evaluation on unseen research tasks and more rigorous controlled experiments would strengthen the central claim of self-reinforcement.
-
The Human-in-the-loop experimental setup differs from realistic human AI collaboration, limiting the generalizability of the conclusions. The paper presents Human AI Collaboration as one of its main contributions and demonstrates the advantages of the CoPilot mode through comparisons across seven intervention regimes. However, the human guidance in these experiments is not provided dynamically by researchers based on real-time experimental outcomes. Instead, the authors use predefined Guidance Payloads, initially generated by an LLM and subsequently edited by the authors, which are injected at designated stages through a scripted adapter. This setup primarily evaluates the effects of predefined guidance at different stages rather than realistic, interactive human AI collaboration. The evaluation also covers only 10 ML tasks, with a single run per task and configuration. The intervention regimes differ in both the amount and timing of guidance, making it difficult to isolate the effects of intervention strategy. These limitations weaken the evidence supporting the claim that CoPilot consistently outperforms other human intervention strategies in real-world scientific research settings.
Questions for authors
-
Can the authors evaluate whether Cross-Run Self-Reinforcement improves performance on unseen research tasks, rather than repeatedly using the same tasks?
-
Can the authors provide evidence that CoPilot's advantages generalize to realistic human AI collaboration with dynamic human feedback, rather than predefined guidance? I understand that organizing experiments involving real human participants may be challenging in the short term. The authors could instead provide a reasonable justification and explain how the current experimental setup addresses some of these concerns.
Rebuttal, Part III&IV/IV
Part 3/4 — Human-in-the-loop evaluation and component ablation (W4)
Human-in-the-loop (HITL). In Table 4 (10 topics, one run per regime), CoPilot outperformed Full-Auto by 3.21 points on matched topics. Human reviewers also passed an average of 6.3 CoPilot manuscripts, compared with 2.3 for Full-Auto.
To assess whether this advantage persists beyond a single run, we conducted two additional independent runs per regime on the 10 ML topics, yielding three runs per regime.
Across the three runs, CoPilot's mean stage-20 score was 2.86 points higher than Full-Auto's (7.30 vs. 4.44). CoPilot also passed the gate on 20/30 topic–run pairs, compared with 8/30 for Full-Auto. Both findings are consistent in direction with Table 4.
Part 4/4 — External, harder benchmarks (W1, Q4)
We agree that the 25 ML topics are relatively short, CPU-based comparisons. They were selected so that each experimental cell could be rerun and regraded. We will therefore characterize ARC-Bench-ML as a diagnostic of the experimental stage, not as a benchmark of open-ended research.
For external tasks with hidden-test graders, we evaluate ARC-Optimize:
We will report these results in a separate appendix, keeping them distinct from the main results for ARC.
MLS-Bench. We evaluate four tasks (subgroup calibration, clustering, imputation, and multi-objective optimization) with two seeds each, a budget of $1.20 per run, and the official normalized hidden-test score. All arms follow the same evaluation and anomaly-verification rules.
MLE-bench Lite. We evaluate four CPU competitions with two seeds each, a budget of $1.20 per run, and the official grader. All eight runs exceed the leaderboard median. The Kaggle-percentile-equivalent scores for each seed are:
We thank the reviewer again. The revised paper will report these replicated and held-out results alongside the original tables.