acceptodds
Under review as a conference paper at ICLR 2027

EurekaBench: Measuring Agentic Ability to Discover New Scientific Insights

Abstract

Isaac Newton discovered the law of gravitation through an iterative process of analyzing observed data such as planetary patterns, finding the mechanisms by describing them in mathematical equations, and refining his theory against the Moon's orbit, revealing the startling insight that the same force governs both falling apples and orbiting planets, and successfully predicted the existence of planet Neptune. As AI agents rapidly advance, whether they can automate such a process has become a major point of contention. To measure this gap, we introduce EurekaBench, a cross-domain benchmark that tests AI agents' ability to conduct long-horizon experiments, find the mechanism to explain the observations, and eventually be judged by its derivable scientific insights. To measure this gap, we introduce EurekaBench, a cross-domain benchmark that tests AI agents' ability to conduct long-horizon experiments, find the mechanism to explain the observations, and eventually be judged by its derivable scientific insights. Our results show that current AI agents often overly fixate on predictive accuracy optimization, surpassing human scientists, while falling substantially short in deriving scientific insights.

Primary area:
datasets and benchmarks
Keywords:
scientific discovery, ai scientist agent, ai for science, benchmarks

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_uNgDOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EurekaBench, a benchmark for evaluating whether AI agents can discover scientific mechanisms through iterative experiments and produce meaningful insights. It contains 26 tasks across six domains, with 306 expert-verified insight questions. Agents interact with computational environments and submit a mechanism description, executable implementation, and experiment records. Evaluation covers scientific constraints, predictive accuracy, and scientific insights. Across seven agent systems, the strongest agents approach the predictive accuracy of human reference solutions but receive substantially lower insight scores. Additional analyses examine guidance prompts, insights recorded during experimentation, and expert assessments of selected submissions.

Critical strengths

The benchmark tests whether a model explains a scientific phenomenon and supports useful insights, alongside how well it predicts the data. This is a valuable focus: the results show that agents can approach the predictive accuracy of human reference solutions while their mechanisms support far fewer insights. The involvement of domain experts in selecting problems and checking insight questions also strengthens the benchmark. With tasks across six domains, it provides a useful setting for studying where scientific agents succeed and where their explanations fall short.

Critical weaknesses

My main concern is the reliability of the insight scores. Judge agents interpret the submitted mechanisms and run further experiments to decide which insights they support, but the expert reviews provide broad assessments of selected submissions rather than checking each insight verdict. This leaves limited evidence that the scores reliably measure scientific understanding, which is central to the paper. The evaluation data and full task-specific rubrics also remain private. This makes it difficult for other researchers to reproduce the results, inspect disputed judgments, or check whether the benchmark gives fair credit to alternative scientific explanations.

Questions for authors

Here are my questions:

1. What evidence shows that the automated insight judgments agree with domain experts at the level of individual insight questions, beyond the broad reviews of selected submissions?

2. How can independent researchers access the evaluation data and full task-specific rubrics to reproduce the results and check disputed judgments?

5 comments

Thank you for your comments and acknowledging that EurekaBench addresses an important problem in scientific discovery.

W1: Reliability of the scientific insights (SI) score

We agree that it is important for experts to evaluate scientific insights using the same rubric as the agent judges and compare the results. We randomly selected 9 problems across 6 domains, covering 56 scientific constraints and 100 scientific insights, and calculated human-agent agreement scores for GPT-6 Astra and Claude Fable 5.1.

Scientific constraints

ModelHuman vs GPT-5.6 SolHuman vs Claude Opus 4.8Human vs average of the two judges
GPT-6 Astra100.0%96.4%98.2%
Claude Fable 5.1100.0%98.2%99.1%

Scientific insights

ModelHuman vs GPT-5.6 SolHuman vs Claude Opus 4.8Human vs average of the two judges
GPT-6 Astra94.0%90.0%92.0%
Claude Fable 5.190.0%75.0%82.5%

Q1: What evidence shows that the automated insight judgments agree with domain experts at the level of individual insight questions, beyond the broad reviews of selected submissions?

In addition to the agreement study, we had already made efforts in our original manuscript to ensure that automated insight judgments agree with those of domain experts at the level of individual insight questions. The agentic judge instructions were based on multiple rounds of expert feedback. We refined these instructions iteratively to align the agentic judge's evaluations as closely as possible with the standards of human scientists.

W2 & Q2: Accessibility of our code and evaluation data

We agree that making the evaluation data publicly available is important for transparency and reproducibility. It has always been our intention to release all evaluation data, including the full task-specific rubrics, to allow independent researchers to reproduce our results, verify individual judgments, and conduct further analysis. We will add a link to these resources in the deanonymized version.

Reviewer_uNgDOpenReview ↗

Thanks for the response!

I appreciate authors taking the time to address my concerns. I will keep my scores.

Thank you for your prompt response and for taking the time to review our rebuttal! Please let us know if you have any remaining concerns that we have not fully addressed. We would be happy to provide additional information and discuss them further.

Reviewer_5dWEOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents EurekaBench, a benchmark for evaluating the performance of LLM agents on discovering new scientific insights. The benchmark consists of 26 long-horizon problems crafted by experts over 6 scientific domains (e.g., physics, neuroscience, computer science). For each task, the agent receives relevant scientific background information about the problem, an initial observation that challenges existing scientific understanding and a specialized environment to run simulations to test hypotheses. Then, the agent is instructed to discover the mechanism underlying the phenomenon described in the problem statement (text, code, and experimental logs).

The evaluation is split in two main parts:

  • predictive accuracy (PA): does its executable implementation predict held-out observations accurately?
  • scientific insight (SI): does the discovered mechanism accurately explain a set of insights provided by the experts?

The authors empirically show that while top LLMs are capable of achieving high predictive accuracy (almost matching that of human experts), they struggle at scientific insight.

Critical strengths
  • The benchmark addresses a timely problem: evaluating LLMs on scientific discovery. This is a hard problem to evaluate because scientific discovery involves complex reasoning and the generation of novel insights. The proposed evaluation method, which separates predictive accuracy from scientific insight, provides a structured way to assess the capabilities of LLM agents in this context. That requires careful task design and several rounds of discussions with experts in each domain. To me, this is very valuable for the research community.
  • The qualitative analysis of the failures provides valuable insights into the limitations of current LLM agents in scientific discovery. By examining where and why the agents struggle, the authors highlight specific challenges that need to be addressed in future research.
Critical weaknesses
  • This benchmark has a short shelf life, as those scientific problems may quickly become part of the pretraining data for LLMs, reducing the challenge for future models.
  • I found the paper makes many strong claims that are not fully substantiated by the evidence provided. For instance, the claim about agents intentionally prioritizing predictive accuracy over scientific insight might be overstated; it might also be that predictive accuracy is simply easier to optimize for, and/or the judges could systematically underestimate or overestimate certain kinds of insight.
Questions for authors
  • Was the 4 hours time limit and 1000 turns sufficient for the tasks?
  • How many runs were performed for each task to obtain the reported results? What's the variance across runs?
4 comments

Thank you for your positive scores and your acknowledgment that EurekaBench addresses a very timely, hard, and important problem in scientific discovery: evaluating whether AI agents can move beyond simply finding solutions to discovering new scientific insights through the long-horizon experimentation loop.

W1: Concerns about the short shelf life of EurekaBench

We think the fact that EurekaBench challenges existing best models already shows its value. If future generations of LLMs can solve it well, it means our effort has effectively advanced the development of frontier LLM training.

At the same time, discovering novel scientific insights is inherently difficult and time-intensive, even for experienced human scientists. This difficulty itself creates a time barrier.

Moreover, EurekaBench is a live benchmark, and we will continue working with more domain experts to expand it in two directions: adding new scientific problems and broadening the insights covered by existing problems. We believe that even as AI agents’ scientific discovery capabilities improve rapidly, EurekaBench will remain challenging.

W2.1: Claim about “intentionally prioritizing predictive accuracy over scientific insight”

Thank you for pointing out this important overstated wording issue. We have changed “AI agents intentionally\color{red}{\textit{intentionally}} treat scientific mechanism discovery as an optimization task” to AI agents tend to\color{green}{\textit{tend to}} treat scientific mechanism discovery as an optimization task” to describe the observed behavior without implying conscious intent for AI agent. Following your feedback, we carefully reviewed the rest of the manuscript and also replaced “overly\color{red}{\textit{overly}} fixate on predictive accuracy optimization, surpassing human scientists” with “overemphasize\color{green}{\textit{overemphasize}} predictive accuracy optimization, surpassing human scientists.” We include the changes in the revised manuscript.

W2.2: Concerns about the judges could systematically underestimate or overestimate”

Predictive accuracy can be easier to optimize. This difference may reflect greater emphasis in language model training on solving tasks with explicit, verifiable objectives than on open-ended experimentation and mechanism discovery. EurekaBench captures this capability gap in current AI scientist agents and asks whether they can move beyond finding solutions to discovering useful, novel scientific insights.

We also agree that agent judges may introduce noise or systematic bias. We therefore added a human–agent agreement study on 9 randomly selected problems across 6 scientific domains, covering 56 scientific constraints (SC) and 100 scientific insight (SI) questions. Our domain experts evaluated submissions from GPT-6 Astra and Claude Fable 5.1 and we provide the agreement score between experts and agents in the below tables:

Scientific constraints

ModelHuman vs GPT-5.6 SolHuman vs Claude Opus 4.8Human vs average of the two judges
GPT-6 Astra100.0%96.4%98.2%
Claude Fable 5.1100.0%98.2%99.1%

Scientific insights

ModelHuman vs GPT-5.6 SolHuman vs Claude Opus 4.8Human vs average of the two judges
GPT-6 Astra94.0%90.0%92.0%
Claude Fable 5.190.0%75.0%82.5%

Our agreement scores show that our agentic judge evaluation for the scientific constraints and scientific insights are reliable.

Questions

Q1: Was the 4 hours time limit and 1000 turns sufficient for the tasks?

Yes, 4 hours and 1000 iterations were sufficient for current agents to complete the tasks in EurekaBench. We provide the runtime table of our 7 agents. Most runs stopped within 1 to 2 hours, and the longest run took 3.97 hours on a Computer Science problem.

Runtime (hours): avg / max

AgentNeuroscienceGeophysicsPlasma PhysicsAstrophysicsComputer ScienceChemistry
Claude Fable 5.11.82 / 2.362.41 / 2.891.30 / 2.271.26 / 1.932.04 / 2.780.88 / 0.91
Claude Opus 51.56 / 3.081.52 / 1.531.65 / 1.871.27 / 2.191.96 / 2.690.79 / 0.87
Claude Opus 4.80.94 / 2.160.97 / 1.351.30 / 1.950.94 / 2.331.10 / 2.240.76 / 1.30
GPT 6 Astra1.35 / 3.681.46 / 2.781.26 / 2.691.22 / 2.931.20 / 3.830.31 / 0.70
GPT 5.6 Sol0.38 / 0.700.18 / 0.310.32 / 0.400.57 / 1.030.45 / 0.950.33 / 0.74
DeepSeek V4 Flash1.44 / 2.622.40 / 3.471.07 / 1.371.53 / 2.991.86 / 3.971.07 / 1.93
Kimi K32.03 / 3.801.03 / 1.031.43 / 1.871.79 / 3.131.82 / 3.520.87 / 1.07

Q2: How many runs were performed for each task to obtain the reported results? What's the variance across runs?

We report a single run per task in the paper because evaluating many of the problems typically takes more than 24 H100 GPU hours, and running scientist agents is also expensive. We agree that studying the variance across runs is important, so we randomly selected 6 problems across 4 scientific domains and ran each a second time with Claude Fable 5.1, Claude Opus 5, Claude Opus 4.8, GPT-6 Astra, and GPT 5.6 Sol. The variance and standard deviation across the two runs are reported below:

Agentjudge = GPT 5.6 Soljudge = Claude Opus 4.8combined (GPT 5.6 Sol + Claude Opus 4.8)
varianceSDvarianceSDvarianceSDagreement
Claude Fable 5.10.00820.09070.01730.13160.01280.113084.7%
Claude Opus 50.05030.22420.03780.19440.04400.209870.8%
Claude Opus 4.80.02320.15240.01030.10140.01680.129582.0%
GPT 6 Astra0.00590.07710.00150.03890.00370.061093.6%
GPT 5.6 Sol0.00220.04670.00190.04340.00200.045186.5%

This is great. Thank you for the additional information and answering my questions.

I'd encourage other reviewers to avoid selecting a borderline rating (i.e. score of 2 or 3), especially if their concerns have been addressed.

Thank you for your kind words! We are glad to hear that our responses have addressed your questions and concerns.

Reviewer_i8Y3OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a new benchmark on evaluating whether AI agents can go beyond fitting data, and to discover mechanisms that can forther produce science insights. The benchmark dataset includes tasks containing expert built, long-horizon tasks covering neuro-science, astrophysics, plasma physics, computer science, grophysics, and chemistry. Each tasks provides initial observations, a simulator or tool interface. Agents needs to submit mechanims description, executable mechanism, and experiment logs. In the evaluation part, there are three parts to be evaluated: 1. agent judged scientific-constraints; 2. prodictive analysis on validation data; 3. Insight score computed from expert written questions that the judge can derive from the submitted mechanisms, using the simulator.

Critical strengths
  1. The question is very important and well-formulated. Agent are not yet evaluated by the current benchmarks on the perspective that whether they can turn optimization, which is mostly auto-reseasrch agents do today, into the understanding and insights of the research subjects.

  2. Real experts are involved in the experiments and the part of the project. This is very good and can therefore ensure the quality of the datasets.

  3. Separating predictive accuracy from insight is valuable, and the decoupling between the two is supported quantitatively. The expert analysis of failure modes is also insightful: preselected hypotheses, fitting presented as mechanism, and stopping early.

Critical weaknesses
  1. For the insight score part, if I understand the content correctly, the scoring relies on LLM judges that have not been validated against humans. This parts involves how the agent can extend the mechanisms to insights, and if we use the same family of llm as judge, would it be possible that the results would be affected by self-preference?

  2. I think the same concerns may apply the human baselines. The human baseline could be circular because the insight questions are drawn from the same human research that serves as the human solution, maybe this is the reason human SI is high? In this case why the human mechanisms does not score close to perfect?

  3. The benchmark itself is still limited in terms of scale. Also it lacks variance analysis. In terms of agent research, or even science research, sometimes the variance could be very large even if mechanisms or insights are clear. I do think its necessary to report mutliple run results of the most important experiments.

Questions for authors
  1. What is the agreement between expert scientists and the agentic judges on a random sample of insight and SC verdicts, and what is the agreement between the two judges?
  2. How was the human-scientist submission constructed and evaluated, and why does it fall well short of a perfect SI when the questions derive from that work?
6 comments

Thank you for acknowledging that EurekaBench has well-formulated a very important problem for scientific discovery, which is evaluating whether AI scientist agents can move beyond simply optimizing the solution and discover new scientific insights.

W1: Reliability of the scientific insights (SI) score

This is a good point! First, we note that to maximize the reliability of scientific insight scores assigned by judge agents, we made the following efforts in our original manuscript:

  • We iteratively refined the judge agent's instructions based on expert feedback to make its evaluations align as closely as possible with the standards of human scientists. We briefly mention in line 248-254 and we clarified this in our revised manuscript.
  • To avoid self-preference when the judge and scientist agents come from the same model family, we used both GPT-5.6 Sol and Claude Opus 4.8 to score the scientific insights from each run (see lines 219–228). All results reported in the paper are averaged across the two judges.

Further, based on your comment, to have a deeper understanding of the self-preference problem, we further compared the two agentic judges' scores across domains to check for self-preference. Neither judge consistently gave higher scores to scientist agents from its own model family. We also found that Claude Opus 4.8 was generally more lenient toward agents from both model families, as it conducted more exploration during evaluation. We will add these results to the revised version.

DomainClaude Fable 5.1 – GPT-5.6 Sol judgeClaude Fable 5.1 – Claude Opus 4.8 judgeGPT-6 Astra – GPT-5.6 Sol judgeGPT-6 Astra – Claude Opus 4.8 judge
Astrophysics45.6%48.9%28.9%28.9%
Fusion plasma29.0%29.0%22.6%22.6%
Geophysics19.9%16.3%29.0%23.4%
CS17.8%40.0%20.0%24.4%
Neuroscience23.8%23.8%9.0%10.7%
Chemistry30%30%30%30%

Q1 What is the agreement between expert scientists and the agentic judges on a random sample of insight and SC verdicts, and what is the agreement between the two judges?

We agree that agent-expert agreement and agent-agent agreement analyses are important for demonstrating the reliability of the scientific insight scores. We first provide the additional agreement study between two judge agents.

AgentSCSI
Claude Fable 5.191.0%86.0%
Claude Opus 588.4%91.0%
Claude Opus 4.887.8%91.6%
GPT-6 Astra93.1%92.2%
GPT-5.6 Sol90.7%90.9%
DeepSeek V4 Flash89.9%96.0%
Kimi K393.0%93.5%

We also randomly selected 9 problems across 6 domains, covering 56 scientific constraints and 100 scientific insights, to calculate the human-agent agreement scores for GPT-6 Astra and Claude Fable 5.1. Our expert used the same SC and SI rubrics as the judge agents to evaluate the mechanisms discovered by the scientist agents and their experiment logs.

Scientific constraints

ModelHuman vs GPT-5.6 SolHuman vs Claude Opus 4.8Human vs average of the two judges
GPT-6 Astra100.0%96.4%98.2%
Claude Fable 5.1100.0%98.2%99.1%

Scientific insights

ModelHuman vs GPT-5.6 SolHuman vs Claude Opus 4.8Human vs average of the two judges
GPT-6 Astra94.0%90.0%92.0%
Claude Fable 5.190.0%75.0%82.5%

Our agreement analysis shows that both scores fall within the substantial agreement range. We updated the agreement analysis in the revision.

W2: Human insights score is not close to prefect

Q2: How was the human-scientist submission constructed and evaluated, and why does it fall well short of a perfect SI when the questions derive from that work?

We want to clarify that EurekaBench contains partially\color{red}{\text{partially}} solved scientific problems. We construct the scientific insight questions from 1) recent discoveries made by scientists, 2) unresolved insight questions in the source studies, and 3) potential insights proposed by our domain experts.

We agree that drawing some scientific insight questions from the same research used for the human reference may introduce a bias in human scientists' SI scores. We chose partially solved scientific problems because of a fundamental challenge in evaluating scientific discovery: unlike mathematical results, many scientific discoveries cannot be verified through definitive proofs. Even human experts find it difficult to evaluate newly discovered mechanisms for completely open-ended scientific problems.

To avoid overlooking any insights discovered by AI scientists beyond our SI rubric, our domain experts reviewed the mechanisms discovered by GPT-6 Astra and Claude Fable 5.1, along with their experiment logs. Although the agents showed some positive behaviors during their scientific experimentation loops, the experts agreed that they did not discover any additional interesting insights beyond those already evaluated in our SI score. We provide examples of these common positive behaviors in Fig. 8 (bottom) and Appendix D.4 in the revision manuscript.

W3.1: Scale of this benchmark

EurekaBench is a high-quality, expert-verified benchmark that requires substantial effort to construct. We worked closely with domain experts to develop 26 open scientific problems across six scientific domains (Neuroscience, Geophysics, Plasma Physics, Astrophysics, Computer Science, and Chemistry), with over 300 scientific insight questions. In Appendix D.1, we provide statistical analysis showing that the current scale of our benchmark is already sufficient to support our main conclusion that predictive accuracy alone does not reliably indicate scientific insight.

We also want to mention that EurekaBench is a live benchmark and we plan to continue to expand in two directions: 1) Work with more domain experts to add new scientific problems and insights, increasing the scale of the benchmark. 2) As discussed in our Limitations (lines 460–465 in original manuscript), scientists may approach the same problem differently and uncover different scientific insights. We will also expand the insight questions for existing problems to reflect this broader range of discoveries.

W3.2: Variance analysis

We agree that variance analysis across runs is important. We randomly selected 6 problems across 4 science domains and performed a second run for each problem to calculate the variance and standard deviation for Claude Fable 5.1, Claude Opus 5, Claude Opus 4.8, GPT-6 Astra, and GPT 5.6 sol below:

modeljudge = GPT 5.6 Soljudge = Claude Opus 4.8combined (GPT 5.6 Sol + Claude Opus 4.8)
varianceSDvarianceSDvarianceSDagreement
Claude Fable 5.10.00820.09070.01730.13160.01280.113084.7%
Claude Opus 50.05030.22420.03780.19440.04400.209870.8%
Claude Opus 4.80.02320.15240.01030.10140.01680.129582.0%
GPT 6 Astra0.00590.07710.00150.03890.00370.061093.6%
GPT 5.6 Sol0.00220.04670.00190.04340.00200.045186.5%

One example from Plasma Physics – Claude Fable 5.1 – different discovered mechanism with same SI score

Run 1 (discovered mechanism): …Negative triangularity lets the boundary elongation penetrate to the core (κ at ρ = 0.2: 1.96 vs 1.47) and lowers q outside mid-radius. Because the turbulent heat flux falls with elongation (Q_i ∝ κ^-1.15) and rises with q (Q ∝ q^1.2), the same power flows with a smaller gradient, lifting the profile: +5.4 % stored energy…

Run 2 (discovered mechanism): …The shape acts through the safety factor: turbulent stiffness grows as q^1.08, and at equal current the NT shape has a lower q95 (5.2 vs 6.9). Matching q95 removes the gain (PT at q95 = 5.16 reaches τ_E = 37.9 ms vs 37.8 ms for NT), so the phenomenon is "strong positive triangularity confines worse", not "negative confines better"… Both runs use the same task specification: discover a mechanism that explains why the reversed shape retains heat better and has weaker turbulence in the outer half of the plasma radius.

In the first run, Claude Fable 5.1 measures how each shape changes the plasma’s internal structure and discovers that taller inner layers and tighter magnetic winding reduce heat loss.

In the second run, Claude Fable 5.1 gives both shapes the same magnetic winding and discovers that magnetic winding drives the gain.

Although Claude Fable 5.1 takes different approaches during experimentation and discovers different mechanisms across the two runs, both runs focus on fitting a formula to the simulator rather than understanding why turbulence weakens. As a result, they fail on the same insight questions. The agent mainly focuses on optimizing solutions and lacks the ability to discover useful scientific insights. This is the capability gap we want to capture in EurekaBench.

Comments on the paper

1 comment

Dear Reviewers,

As the discussion phase draws to a close, please acknowledge the author's response and let them know whether it has addressed some or all of your concerns.

Thank you for the time and effort you and the other reviewers have put into this discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.