- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces EurekaBench, a benchmark for evaluating whether AI agents can discover scientific mechanisms through iterative experiments and produce meaningful insights. It contains 26 tasks across six domains, with 306 expert-verified insight questions. Agents interact with computational environments and submit a mechanism description, executable implementation, and experiment records. Evaluation covers scientific constraints, predictive accuracy, and scientific insights. Across seven agent systems, the strongest agents approach the predictive accuracy of human reference solutions but receive substantially lower insight scores. Additional analyses examine guidance prompts, insights recorded during experimentation, and expert assessments of selected submissions.
Critical strengths
The benchmark tests whether a model explains a scientific phenomenon and supports useful insights, alongside how well it predicts the data. This is a valuable focus: the results show that agents can approach the predictive accuracy of human reference solutions while their mechanisms support far fewer insights. The involvement of domain experts in selecting problems and checking insight questions also strengthens the benchmark. With tasks across six domains, it provides a useful setting for studying where scientific agents succeed and where their explanations fall short.
Critical weaknesses
My main concern is the reliability of the insight scores. Judge agents interpret the submitted mechanisms and run further experiments to decide which insights they support, but the expert reviews provide broad assessments of selected submissions rather than checking each insight verdict. This leaves limited evidence that the scores reliably measure scientific understanding, which is central to the paper. The evaluation data and full task-specific rubrics also remain private. This makes it difficult for other researchers to reproduce the results, inspect disputed judgments, or check whether the benchmark gives fair credit to alternative scientific explanations.
Questions for authors
Here are my questions:
1. What evidence shows that the automated insight judgments agree with domain experts at the level of individual insight questions, beyond the broad reviews of selected submissions?
2. How can independent researchers access the evaluation data and full task-specific rubrics to reproduce the results and check disputed judgments?
5 comments
Thanks for the response!
I appreciate authors taking the time to address my concerns. I will keep my scores.
Thank you for your prompt response and for taking the time to review our rebuttal! Please let us know if you have any remaining concerns that we have not fully addressed. We would be happy to provide additional information and discuss them further.
Thank you for your comments and acknowledging that EurekaBench addresses an important problem in scientific discovery.
W1: Reliability of the scientific insights (SI) score
We agree that it is important for experts to evaluate scientific insights using the same rubric as the agent judges and compare the results. We randomly selected 9 problems across 6 domains, covering 56 scientific constraints and 100 scientific insights, and calculated human-agent agreement scores for GPT-6 Astra and Claude Fable 5.1.
Scientific constraints
Scientific insights
In addition to the agreement study, we had already made efforts in our original manuscript to ensure that automated insight judgments agree with those of domain experts at the level of individual insight questions. The agentic judge instructions were based on multiple rounds of expert feedback. We refined these instructions iteratively to align the agentic judge's evaluations as closely as possible with the standards of human scientists.
W2 & Q2: Accessibility of our code and evaluation data
We agree that making the evaluation data publicly available is important for transparency and reproducibility. It has always been our intention to release all evaluation data, including the full task-specific rubrics, to allow independent researchers to reproduce our results, verify individual judgments, and conduct further analysis. We will add a link to these resources in the deanonymized version.