- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces AutoResearcher-HumanResearch Similarity (AutoHRS), which is a framework for evaluating how well AutoResearchers can reconstruct human researchers across different steps in AI/ML-research workflows. Stated insights include that AutoResearchers tend to rely on recombining elements from relevant references provided to them when being tasked to generate research ideas, and that they favor established models/metrics rather than more problem-specific ones when being tasked to generate experiment plans.
Critical strengths
- Despite the lack of novelty, the qualitative analysis of the AutoResearcher outputs and its related findings and its related findings are sound and agree with previous works.
Critical weaknesses
-
Lack of generalizability and novelty of claimed statements. The experiment is only performed with ClaudeCode / Sonnet 4.5. First of all, recent scientific research benchmarks show that this model does not serve as a strong or reliable scientific auto-researchers (some relevant evaluation works are also not mentioned in the relevant work section). There were many other, much stronger models available well before the submission deadline and the ICLR 2026 knowledge cutoff, and analysis based on those would have made many of the claimed findings stronger, more relevant, and less obsolete. Many of these statements are also not new findings: the fact that these weaker models e.g. rely on established models/metrics rather than more problem-specific ones is not a new insight. While acknowledged in the limitations, the analysis only includes ICLR papers without considering any other scientific domains, giving the findings even less credibility. Moreover, the paper suggests that a subset of the experiments were replicated with Gemini CLI and Codex, pointing to Appendix B.1 and B.2. No appendix is available in the submitted PDF, so none of the appendix pointers help understand the work's full methodology, results, and conclusions.
-
Evaluation Design: The checklist evaluation pipeline is limited to measuring reconstruction only via simple LLM judge labels "yes", "no", "partially" compared to one human-reference. This raises the following concerns: (i) the evaluation measures whether reconstruction of one example reference by human researchers. It does not take into account how other human researchers would e.g. design an experiment plan given a research idea, i.e. here the verification is over-constrained to one specific way of how a single human researcher or research team would write the experiment plan - this is a classic failure mode of AI evaluation works measuring scientific research capabilities. Giving the same task to another human researcher/research team would likely also result in a different approach, so it's not obvious that this evaluation setup is measuring something that applied to AutoResearchers only; (ii) even if the judge labels the AutoResearchers' approach as "no", there is no measure of the direction of the "failed" reconstruction: did the AutoResearcher suggest an approach that is worse or better than the human reference? What if the AutoResearcher comes up with a better research idea or experiment plan than the human?
Questions for authors
see questions posed in the "critical weaknesses" section.
4 comments
Author Response (2/2)
W2.1: The evaluation measures whether reconstruction of one example reference by human researchers. It does not take into account how other human researchers would e.g. design an experiment plan given a research idea, i.e. here the verification is over-constrained to one specific way of how a single human researcher or research team would write the experiment plan - this is a classic failure mode of AI evaluation works measuring scientific research capabilities
We thank the reviewer for raising this concern. The reviewer is concerned that using a single human reference for each research idea may not capture the diversity of experimental plans that different human researchers could develop.
However, while this concern is conceptually valid, addressing it through multiple human references is challenging in practice. Unlike tasks with standardized inputs and multiple acceptable answers, scientific research is inherently open-ended, and collecting multiple independently developed human research papers addressing exactly the same idea under comparable research contexts is generally infeasible. Consequently, prior benchmarks evaluating AI research capabilities have also adopted published human research papers as references. For example, PaperBench (ICML 2025) evaluates AI agents' ability to replicate published AI research, while FIRE-Bench (ICML 2026) evaluates whether AI agents can rediscover scientific findings from human research papers. These established benchmarks demonstrate the practicality of using published research as a reference for evaluating AI research agents.
In our work, we further mitigate this concern through two design choices.
First, we select high-quality and diverse human research references. Our evaluation covers 152 ICLR oral papers from different research teams, capturing a broad range of research decisions and experimental designs across the AI field. We specifically select ICLR oral papers, rather than poster papers, papers from less selective venues, or rejected papers, because oral papers undergo rigorous peer review and represent highly regarded research contributions. We therefore consider their experimental designs to be strong reference examples reflecting the decisions of experienced human researchers. Importantly, we use these papers as references for reconstruction, rather than assuming that their experimental designs are the only scientifically valid solutions.
Second, our weighting scheme emphasizes relatively well-constrained aspects of experimental design. Specifically, Baselines and Comparisons (0.30) and Evaluation Protocol (0.25) together account for 55% of the Experiment Design score. These dimensions are more strongly constrained by the research problem and established experimental practices than other aspects of experimental design. For example, a new method should generally be compared against relevant existing approaches and evaluated using appropriate protocols. Although alternative valid choices may exist, we expect less variation in these dimensions across well-designed experimental plans addressing the same research idea.
W2.2: even if the judge labels the AutoResearchers' approach as "no", there is no measure of the direction of the "failed" reconstruction: did the AutoResearcher suggest an approach that is worse or better than the human reference? What if the AutoResearcher comes up with a better research idea or experiment plan than the human?
We thank the reviewer for raising this concern. However, we respectfully disagree with this criticism, as our work does not aim to evaluate whether AutoResearchers produce better or worse research than human researchers. Instead, our goal is to investigate whether AutoResearchers can reconstruct research decisions similar to those made by human researchers under comparable research contexts. Therefore, a "No" label indicates a divergence from the human reference, rather than suggesting that the AutoResearcher's approach is necessarily inferior. We acknowledge that an alternative approach may be equally valid or even better than the human reference, but evaluating such differences in research quality is beyond the scope of AutoHRS.
Thank you again for your time and attention in helping us to improve our paper. We welcome any further feedback or discussion that you are able to provide.
Kind Request to Review Our Rebuttal
Dear Reviewer,
As the author–reviewer discussion comes to a close, we would be grateful if you could share any remaining concerns or points you'd like to discuss. We understand this is a busy time, and we truly value the time and effort you've spent to reviewing our work and engaging with our responses.
Thank you!
Kind Request to Review Our Rebuttal. Thank you!
Dear Reviewer,
As the author–reviewer discussion will ends tomorrow, we would be grateful if you could share any remaining concerns or points you'd like to discuss. We truly value the time and effort you've spent to reviewing our work and engaging with our responses.
Thank you!
Author Response (1/2)
We thank you for the evaluation of our paper and the feedback. We would like to clarify that, in the original submission, the appendix was included in the supplementary material PDF. We have now incorporated the appendix into the main PDF to make it easier to access through the hyperlinks in the main text. And we would like to address your other concerns and questions as follows:
We thank the reviewer for raising this concern. We would like to clarify that our evaluation is not limited to Claude Sonnet 4.5. In addition to our main experiments, we also evaluated Gemini CLI (Gemini 3.5 Flash) and Codex (GPT-5.2), with the corresponding results already reported in Appendix B.2 in the supplementary material pdf file. Although ICLR allows authors to include the appendix in the supplementary materials, we acknowledge that its location was not clearly indicated in the main paper, which may have caused confusion.
Furthermore, we took your concern about the availability of more powerful models with knowledge cutoffs before the ICLR 2026 submission deadline seriously. We additionally evaluated Claude Opus 4.6 and Codex GPT-5.4 on the same test set to further examine the generalizability of our results. Here are the full results across different harness-model configs on the test set:
From these results, although we observe some improvements with Opus 4.6 and GPT-5.4 as the backbone models, the overall performance patterns remain consistent across different models and harnesses. In particular, idea generation consistently achieves the lowest scores, while paper synthesis achieves the highest scores. These findings further support the generalizability of our main conclusions across the evaluated models. To control the computational costs of running the AutoResearcher pipelines, we selected Claude Sonnet 4.5 for the main experiments. We believe these results help address the generalization concern.
Thank you for pointing this out. We respectfully disagree that limiting our evaluation to the AI field undermines the credibility of our findings, as our work explicitly focuses on AI research, as already stated in both the title and the main text. Extending our evaluation to other scientific domains is beyond the scope and resources of this study, particularly because our human annotators have expertise primarily in AI research. We have clarified the scope in Section Abstract (line 011) and Introduction (line 056-057) of the revised manuscript.
Thank you for pointing this out. As mentioned earlier, in the original submission, the appendix was included in the supplementary material PDF. Appendices B.1 and B.2 provide additional experimental details and cross-harness evaluation results for Gemini CLI and Codex. Although ICLR allows appendices to be submitted separately as supplementary material, we have now incorporated the appendix into the main PDF of the revised paper for easier access and a better reading experience.