acceptodds
Under review as a conference paper at ICLR 2027

AutoResearchClaw: Self-Reinforcing Autonomous Research with Human-AI Collaboration

Abstract

Automating scientific discovery requires iterative research pipelines that generate testable hypotheses, recover from experimental failures, retain useful experience across research cycles, and maintain strict data provenance. Existing systems often struggle with these demands: hypothesis generation and evaluation are frequently conflated, failed experiments are repaired without revisiting the underlying research design, and reported numerical claims are rarely verified against execution records. We present AutoResearchClaw (ARC), an open-source 23-stage multi-agent research pipeline that turns a research question into executed experiments, per-hypothesis verdicts, and a manuscript. ARC combines structured multi-agent debate, self-healing execution with Pivot/Refine decisions, registry-based result verification, configurable human intervention, and cross-run evolution that converts failures and research decisions into reusable lessons. We evaluate ARC on ARC-Bench, a 55-topic benchmark spanning machine learning, high-energy physics, metabolic modeling, statistics, and quantum computing. On 25 machine-learning topics, ARC scores 0.596, compared with 0.511 for AIDE and 0.419 for AI Scientist v2. Cross-run self-reinforcement raises Overall performance from a mean fresh-run baseline of 0.564 to 0.827 after three persistent iterations. End-to-end studies further show that targeted intervention at high-leverage stages outperforms both full autonomy and step-by-step oversight, while component ablations show that debate improves quality, self-healing improves completion, and registry-grounded verification prevents unsupported numerical claims in audited gate-passing manuscripts.

Primary area:
datasets and benchmarks
Keywords:
Autoresearch, End to end, AI4ML, Multiagent

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9MqtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AutoResearchClaw, a multi-agent framework for autonomous scientific research. The framework consists of 23 stages, covering the entire research process from hypothesis generation, experiment design, and code execution to result analysis and manuscript writing. The authors mainly combine several mechanisms, including multi agent debate, self-healing experiment execution, result verification, human intervention, and cross-run experience accumulation. The system can automatically repair code after experimental failures, reconsider research hypotheses when necessary, and leverage experience from previous research runs to improve future performance. The authors also introduce ARC-BENCH, a benchmark containing 55 research topics across five scientific domains. Experimental results show that ARC outperforms AIDE and AI Scientist v2 on machine learning tasks, while cross-run experience accumulation further improves performance. The authors also conduct ablation studies to analyze the contributions of different components and find that targeted human intervention at critical stages is more effective than full autonomy or step-by-step supervision.

Critical strengths
  1. The paper presents a valuable system design for improving the traceability and reliability of experimental results in autonomous scientific research. The paper goes beyond evaluating whether autonomous research systems can successfully execute experiments and places particular emphasis on the authenticity and verifiability of experimental results. The authors introduce a Measurement Registry-based verification mechanism that programmatically checks numerical claims in generated manuscripts against actual execution records, reducing the risk of reporting unsupported experimental results. The ablation study further demonstrates the effectiveness of this mechanism: without the registry, 3 out of 5 manuscripts that passed the quality gate contained numerical claims unsupported by experimental records, whereas none of the 3 passing manuscripts produced by the full system exhibited such issues. Despite the limited evaluation scale, this mechanism provides a practically meaningful approach to improving data provenance and research integrity in autonomous scientific research systems.

  2. The cross-run self-reinforcement mechanism is a valuable research direction and demonstrates promising empirical results. The proposed Cross-Run Evolution mechanism enables autonomous research systems to extract reusable lessons from previous experimental failures, research decisions, and execution anomalies, and incorporate these lessons into subsequent research cycles. This design extends feedback-driven improvement beyond individual research runs, providing a practical approach to accumulating and reusing experience across research cycles. The experiments show that three independent memory-enabled first-iteration runs achieve an higher average score compared with the no-memory baseline. The overall score further increases in a persistent sequence with accumulated experience. Although the improvements over successive iterations would benefit from more extensive independent replication, the reported results provide encouraging empirical evidence for the potential value of cross-run experience accumulation.

Critical weaknesses
  1. The experimental validation of cross run self-reinforcement remains insufficient to fully support the claimed ability of continuous improvement. Cross-Run Self-Reinforcement is one of the paper's central contributions, but the current experiments do not sufficiently establish its effectiveness and generalizability. Although the authors report a substantial improvement from 0.564 to 0.827, the three-iteration results are based on only a single persistent trajectory, without multiple independent replications to assess the stability of the improvements. The experiments also repeatedly use the same ML01 - ML25 tasks, meaning that the accumulated experience may contain task-specific repair strategies and experimental information. As a result, it remains unclear whether the performance gains come from generalizable research experience or adaptation to previously encountered tasks. Further evaluation on unseen research tasks and more rigorous controlled experiments would strengthen the central claim of self-reinforcement.

  2. The Human-in-the-loop experimental setup differs from realistic human AI collaboration, limiting the generalizability of the conclusions. The paper presents Human AI Collaboration as one of its main contributions and demonstrates the advantages of the CoPilot mode through comparisons across seven intervention regimes. However, the human guidance in these experiments is not provided dynamically by researchers based on real-time experimental outcomes. Instead, the authors use predefined Guidance Payloads, initially generated by an LLM and subsequently edited by the authors, which are injected at designated stages through a scripted adapter. This setup primarily evaluates the effects of predefined guidance at different stages rather than realistic, interactive human AI collaboration. The evaluation also covers only 10 ML tasks, with a single run per task and configuration. The intervention regimes differ in both the amount and timing of guidance, making it difficult to isolate the effects of intervention strategy. These limitations weaken the evidence supporting the claim that CoPilot consistently outperforms other human intervention strategies in real-world scientific research settings.

Questions for authors
  1. Can the authors evaluate whether Cross-Run Self-Reinforcement improves performance on unseen research tasks, rather than repeatedly using the same tasks?

  2. Can the authors provide evidence that CoPilot's advantages generalize to realistic human AI collaboration with dynamic human feedback, rather than predefined guidance? I understand that organizing experiments involving real human participants may be challenging in the short term. The authors could instead provide a reasonable justification and explain how the current experimental setup addresses some of these concerns.

Reviewer_gzmPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents AutoResearchClaw (ARC), a 23-stage multi-agent pipeline that goes from a research question to experiments and a manuscript. Its main pieces are multi-agent debate at hypothesis generation and result analysis, self-healing execution with refine/pivot decisions, a measurement registry that blocks unverified numbers, human-in-the-loop modes, and a lesson store carried across runs. It is evaluated on ARC-Bench, a newly curated 55-topic benchmark which mainly has 25 small ML topics. It beats AIDE and AI Scientist v2. The paper also reports cross-run improvement from the lesson store, a human-in-the-loop study and component ablations on 10 topics.

Critical strengths
  1. The system is open source and covers the whole loop, with some useful engineering ideas. The measurement registry, which replaces any number not found in the execution records with a placeholder, is a simple and concrete fix for fabricated results, and the ablation shows the LLM quality gate alone misses them.

  2. The paper reports per-topic scores, failure cases and human pass rates rather than only LLM-judge averages, which makes the results easy to check.

  3. it is a pretty complete work. The presentation and figures are easy to follow and do a good job.

Critical weaknesses
  1. The benchmark is too easy to say much about research ability.

ARC-Bench is built by the authors, and the 25 ML topics are textbook comparisons that must run on one CPU core in under ten minutes with sklearn-level code, e.g. feature scaling for k-NN or cross-validation strategies. These are closer to homework exercises than research questions, so a high score says little about whether the system can do research. Technically for a 23 stage AI scientist pipeline, you would want a much deeper task to accurately reflect the advantage of the system. Otherwise a lot of benefits measured can be drawn in noise and superficial pattern changes. At the very least there should be hour long problems, meaning problems that would take an hour for the agent to plateau. The scoring is also done entirely by LLM judges.

  1. The main comparison favors ARC by design.

Half of the score is Result Analysis, and its rubric rewards a supported / refuted / inconclusive verdict per hypothesis, which is exactly what ARC's stage-14 synthesizer is built to produce, while AIDE is a code-search agent that isn't built to write them. On code development ARC is no better than AIDE. Also, six of the 25 AIDE runs hit the 1,800 s budget and get zero execution score (Table 13), so part of the gap is timeouts rather than research quality.

  1. The self-reinforcement gain is measured on the same topics the memory was built from.

Lessons are extracted from runs on ML01–ML25 and then injected when re-running the same 25 topics, so the gain can come from remembering topic-specific fixes rather than doing better research. The headline 0.564 → 0.827 is also a single trajectory. Only Iteration 1 is replicated, and there the gain is much smaller. A fair test would build lessons on one set of topics and evaluate on held-out ones.

  1. The human-in-the-loop and ablation studies are too small to support their conclusions.

Each regime is run once per topic on 10 topics, with fixed guidance written in advance. Step-by-Step receives every payload CoPilot gets plus one more, yet scores lower, which looks more like run-to-run noise than an effect of "targeted" intervention.

Questions for authors
  1. How much of the self-reinforcement gain remains if lessons are built on one set of topics and evaluated on held-out topics? Can you replicate the three-iteration trajectory with several independent sequences?
  2. How do the systems compare if AIDE is also asked to write per-hypothesis verdicts, or on the topics where no system timed out?
  3. How well do the LLM judges agree with human experts on a sample of runs?
  4. How does ARC do on an external, harder benchmark such as MLE-bench or MLR-Bench or MLS-Bench?
3 comments
AuthorsOpenReview ↗

Rebuttal, Part III&IV/IV

Part 3/4 — Human-in-the-loop evaluation and component ablation (W4)

Human-in-the-loop (HITL). In Table 4 (10 topics, one run per regime), CoPilot outperformed Full-Auto by 3.21 points on matched topics. Human reviewers also passed an average of 6.3 CoPilot manuscripts, compared with 2.3 for Full-Auto.

To assess whether this advantage persists beyond a single run, we conducted two additional independent runs per regime on the 10 ML topics, yielding three runs per regime.

MetricRegimeRun 1Run 2Run 3Aggregate
Mean stage-20 scoreFull-Auto4.034.924.374.44
Mean stage-20 scoreCoPilot7.277.756.897.30
Gate passes (score ≥ 5)Full-Auto2/103/103/108/30
Gate passes (score ≥ 5)CoPilot7/107/106/1020/30

Across the three runs, CoPilot's mean stage-20 score was 2.86 points higher than Full-Auto's (7.30 vs. 4.44). CoPilot also passed the gate on 20/30 topic–run pairs, compared with 8/30 for Full-Auto. Both findings are consistent in direction with Table 4.


Part 4/4 — External, harder benchmarks (W1, Q4)

We agree that the 25 ML topics are relatively short, CPU-based comparisons. They were selected so that each experimental cell could be rerun and regraded. We will therefore characterize ARC-Bench-ML as a diagnostic of the experimental stage, not as a benchmark of open-ended research.

For external tasks with hidden-test graders, we evaluate ARC-Optimize:

  • What it is: A standalone, post-submission extension of ARC's experimental phase (stages 9–15) designed for longer ML experiments.
  • What it retains and adds: It retains ARC's iterate–evaluate–refine loop while adding development, holdout, and single-use audit evaluation tiers, along with budget-aware, multi-branch search.

We will report these results in a separate appendix, keeping them distinct from the main results for ARC.

MLS-Bench. We evaluate four tasks (subgroup calibration, clustering, imputation, and multi-objective optimization) with two seeds each, a budget of $1.20 per run, and the official normalized hidden-test score. All arms follow the same evaluation and anomaly-verification rules.

ArmHidden score
Starter code, no search0.218
One LLM revision0.457
ARC's refine loop (4 iterations)0.473
Budget-aware multi-branch search0.517
  • Refine loop vs. starter: The refine loop improves the hidden-test score by 0.255 [0.156, 0.378] across all eight task–seed pairs. Most of this improvement occurs with the first revision.
  • Larger budget: Increasing the budget to $5 (34–76 minutes per run) yields a further gain of 0.031 [−0.003, 0.066]. We therefore do not claim that these tasks require an hour to reach a plateau.

MLE-bench Lite. We evaluate four CPU competitions with two seeds each, a budget of $1.20 per run, and the official grader. All eight runs exceed the leaderboard median. The Kaggle-percentile-equivalent scores for each seed are:

CompetitionSeed 1Seed 2
leaf-classification0.540.59
random-acts-of-pizza0.780.92
nomad20180.960.98
detecting-insults1.001.00
  • Interpreting detecting-insults: MLE-bench evaluates this competition on the original public test set, so we do not interpret these scores as evidence of a top leaderboard rank.
  • Not leaderboard submissions: These models are trained on subsets of the public training data using a lightweight proposer; the results should not be interpreted as direct Kaggle leaderboard submissions.

We thank the reviewer again. The revised paper will report these replicated and held-out results alongside the original tables.

Thanks for the detailed rebuttal and all the new experiments. The rebuttal is very detailed and a lot of effort has been put into it. The timeout analysis and the held-out lesson test partly address my concerns. I will raise my score for the comprehensiveness of the response. However, the held-out test makes it look like most of the self-reinforcement gain comes from reusing lessons on the same topic. The new external benchmarks also run ARC-Optimize, which is a system you built after submission. Those runs have no baselines and cover only 4 tasks, and even there the refine loop barely beats a single LLM revision (0.473 vs 0.457). So I’m still not convinced that the full pipeline helps on harder research tasks. More broadly, I’m not sure how practical this design is. A 23-stage pipeline is hard to maintain and hard to debug. And without a per-component breakdown it’s unclear how many of the stages actually contribute versus just adding complexity. Recent autoresearch systems are moving toward simpler loops built around a strong agent, so I’d like to see the authors argue why this level of hand-designed structure is still needed.

AuthorsOpenReview ↗

Thank You for Your Thoughtful Feedback

Thank you for carefully reviewing our rebuttal and additional experiments, and for reconsidering your assessment. We appreciate your recognition of the effort behind our response, as well as the time you have devoted to this discussion. Your thoughtful questions and constructive feedback have helped us improve the paper. Thank you again for your careful review and engagement.

Reviewer_YEQ7OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents AutoResearchClaw (ARC) and ARC-Bench. ARC is an open-source autoresearch system that combines hypothesis-level feedback, programmatic result verification, citation verification, configurable human intervention, and access to non-ML scientific tools within a single (23-stage) pipeline.

ARC-Bench is benchmark that covers 55 topics that span machine learning, high-energy physics, metabolic modeling, statistics, and quantum computing. It uses hierarchical rubrics to evaluate systems.

ARC is evaluated on ARC-Bench and compared to two good autoresearch systems, AIDE and AI cientist v2. ARC outperforms both.

Critical strengths

I very much like the paper's driving assumption that "Real research...does not proceed in a straight line from idea to paper: a researcher proposes a hypothesis, designs an experiment, observes what fails, revises the plan, and tries again." (040-042) And the (very complex) ARC system is described clearly at a high level. The system is clearly powerful and the fact that it is open-source is great news for the community.

Critical weaknesses

There is a LOT in the paper --- it covers the 23-stage ARC autoresearch system, the ARCBench benchmark, and an evaluation of ARC vs. two very good baselines.

But the paper's contributions are statements about what the proposed system (ARC) and benchmark (ARC-Bench) are in terms of their components; and how ARC performs on ARC-Bench vs. two baselines using a benchmark-specifc evaluation rubric. For a research paper I would want the contributions to point out what is novel about ARC and/or ARC-Bench (and the intro to motivate why the novel bits are necessary / important / the right novel bits) and the evaluation to focus on ablations with and without those novel components (or with simple vs. fuller implementations of each). Otherwise, there are no general pieces of wisdom to provide for the research community.

It is possible that ICLR is maybe not the right venue for this type of paper.

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.