acceptodds
Under review as a conference paper at ICLR 2027

MechBench: Can AI Scientific Agents Discover Mechanisms Beyond Phenomenal Laws?

Abstract

Scientific discovery requires not only recovering mathematical laws that describe observable behavior, but also identifying the mechanisms that generate them. Existing benchmarks for symbolic regression and scientific agents primarily evaluate phenomenal-law recovery, leaving mechanism discovery largely untested. We introduce MechBench, a benchmark that explicitly separates these two capabilities. Each task is defined by a mechanistic model, a structured set of scientifically meaningful relations whose joint consequences entail an observable phenomenal law, while agents receive only observational data and scientific context. We evaluate mechanism recovery through mechanism probes, which query internal scientific consequences that cannot be inferred from the phenomenal law alone. To reduce reliance on memorized textbook mechanisms, we construct unfamiliar variants through controlled, scientifically interpretable mutations of canonical mechanisms, and screen for mechanistic indistinguishability to exclude ambiguous instances admitting comparable competing mechanisms. Experiments across representative scientific agents reveal a substantial phenomenal–mechanism recovery gap: for Codex with GPT-5.6-sol, phenomenal-law accuracy reaches 35.00% on the Core-set while mechanism accuracy is only 13.75%, with mechanism recovery failing in 64.29% of cases where the phenomenal law is correctly recovered. The gap widens as mechanisms become increasingly mutated, and even providing the correct phenomenal law leaves mechanism recovery below 50%. These results reveal a substantial generalization gap in mechanistic reasoning and establish mechanism discovery as a distinct challenge beyond recovering observable scientific laws.

Primary area:
datasets and benchmarks
Keywords:
Mechanism Discovery, Scientific Discovery, LLM Agents, Symbolic Regression, Equation Discovery, AI for Science, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Q7DfOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MechBench, a benchmark designed to evaluate whether AI scientific agents can recover the internal scientific mechanisms governing observed phenomena. The authors distinguish between "phenomenal-law recovery" and "mechanism recovery": the former requires only the recovery of observable input-output relationships, whereas the latter demands the recovery of internal variables and their scientific interrelationships. The dataset comprises 16 task families spanning physics, chemistry, biology, and materials science. Models are required to propose mechanisms based on scientific context and observational data, after which they are evaluated using two categories of metrics. The authors also employ a "mechanistic indistinguishability screening" process to exclude tasks where plausible competing mechanisms exist. Experimental results indicate that while models achieve high accuracy in recovering phenomenal laws, their accuracy in recovering mechanisms is significantly lower; furthermore, performance in mechanism recovery degrades rapidly as the number of mechanistic variations increases.

Critical strengths

Strengths:

  1. The problem is significant and practically relevant. Most existing benchmarks for scientific discovery focus solely on fitting observed patterns; by isolating the recovery of mechanistic structures as a distinct challenge, this paper addresses a valuable research problem.
  2. Experimental controls are comprehensive. The paper reports performance across phenomenal-law recovery, mechanism recovery, and both in-distribution (ID) and out-of-distribution (OOD) settings. It also incorporates a "Gold-P control" to help distinguish between failures in pattern discovery and failures in mechanistic reasoning.
  3. Broad cross-disciplinary coverage. The tasks span physics, chemistry, biology, and materials science, and the paper provides detailed explanations regarding task construction and evaluation procedures.
Critical weaknesses

Weaknesses:

  1. Mechanistic identifiability is not fully guaranteed. This is the paper's most critical issue. Multiple distinct internal mechanisms (M1,M2,…M_1, M_2, \ldots) can often produce the same phenomenal law. Even with noise-free data and an infinite sample size, the observed relationships alone are typically insufficient to uniquely identify the underlying mechanism. Although the authors propose a "mechanistic indistinguishability screening" and search for competing mechanisms (Section 4.2), the absence of competing mechanisms does not prove that the current mechanism is the only plausible one. Consequently, the problem might effectively reduce to whether the inferred mechanism merely satisfies the observed phenomena, a conclusion that runs counter to the paper's original intent.
  2. Lack of baselines involving human experts or other measures of task solvability. Human expert performance is crucial for understanding the inherent solvability of these tasks. The current model mechanism recovery rate is low; however, the paper fails to present baselines involving domain experts, graduate students, or other humans. Nor does it clarify whether humans could recover the reference mechanism for these tasks based solely on the same observations and scientific background.
Questions for authors
  1. Please see the discussion of mechanistic identifiability above. How do the authors establish that the retained benchmark instances are mechanistically identifiable within the intended scientific hypothesis class?
  2. The judgments regarding "comparable complexity" and "deviation from scientific background" lack formal definitions. The paper states: "If such an alternative requires comparable or lower structural complexity and no greater departure from the relevant scientific background knowledge than the intended modification..." However, the criteria for these judgments remain unclear. Do these assessments rely primarily on the subjective decisions?
  3. Some configurations in the paper utilize the average of three runs, while others are run only once. Appendix B.1 notes that for the Core-set, Codex+GLM-5.3-flash and Codex+DeepSeek-v4-flash-0731 involved three runs, whereas other configurations typically involved only one. Why were only certain configurations replicated? Did the authors consider standardizing the experimental workflow?
  4. PySR’s recovery rate for "phenomenal laws" is virtually zero. What is the reason for this? Is PySR a reasonable baseline for this task?

My rating might improve if the first issue is well resolved. If the other issues are also addressed, I would be happy to consider raising the score further.

9 comments
AuthorsOpenReview ↗

Reply to Reviewer Q7Df (1/5)

W1. Mechanistic identifiability is not fully guaranteed. This is the paper's most critical issue. Multiple distinct internal mechanisms (M1,M2,…\mathcal{M}_1, \mathcal{M}_2, \ldots) can often produce the same phenomenal law. Even with noise-free data and an infinite sample size, the observed relationships alone are typically insufficient to uniquely identify the underlying mechanism. Although the authors propose a "mechanistic indistinguishability screening" and search for competing mechanisms (Section 4.2), the absence of competing mechanisms does not prove that the current mechanism is the only plausible one. Consequently, the problem might effectively reduce to whether the inferred mechanism merely satisfies the observed phenomena, a conclusion that runs counter to the paper's original intent.

Q1. Please see the discussion of mechanistic identifiability above. How do the authors establish that the retained benchmark instances are mechanistically identifiable within the intended scientific hypothesis class?

Thanks for highlighting this issue. Since mechanistic identifiability is central to MechBench’s design and validity, we agree that it deserves a clearer explanation.

Following §3.1, a mechanism M={m1,…,mJ}\mathcal{M}=\{m_1,\ldots,m_J\} comprises scientifically meaningful relations mjm_j involving observable variables (x,y)(\mathbf{x},y) and unobservable internal quantities h\mathbf{h}. Eliminating h\mathbf{h} yields a phenomenal model P:FP(x,y)=0\mathcal{P}:F_{\mathcal{P}}(\mathbf{x},y)=0, from which observations D\mathcal D can be generated. From a purely mathematical perspective, recovering a unique M\mathcal{M} from P\mathcal{P} appears underdetermined, since infinitely many mathematically possible mechanisms may yield the same P\mathcal{P}. However, in scientific discovery from data, we typically seek a mechanism M\mathcal{M} that not only yields an appropriate P\mathcal{P} to describe the observations DD, but also stays as close as possible to established scientific knowledge—that is, requires minimal modifications to existing scientific assumptions. This motivates the following constrained optimization: M∗=argminM:M⇒PdC(M),\mathcal{M}^{*} = \text{argmin}_{\mathcal{M}:\mathcal{M}\Rightarrow\mathcal{P}} d_C(\mathcal{M}), where dC(M)d_C(\mathcal{M}) measures the minimum number of modifications needed to obtain M\mathcal{M} from established mechanisms under CC.

For example, a textbook mechanism M(0)\mathcal{M}^{(0)} has dC(M(0))=0d_C(\mathcal{M}^{(0)})=0, since it is already part of established scientific knowledge. Through LLM-assisted investigation, we check that the established mechanisms under CC contain no scientifically inequivalent alternative M(0)′\mathcal{M}^{(0)\prime} yielding the same P(0)\mathcal{P}^{(0)}, thereby ruling out alternatives satisfying both dC(M(0)′)=0d_C(\mathcal{M}^{(0)\prime})=0 and M(0)′⇒P(0)\mathcal{M}^{(0)\prime}\Rightarrow\mathcal{P}^{(0)}. This ensures that M(0)\mathcal{M}^{(0)} (or a mechanistic model that is scientifically equivalent to it, differing only in its literal formulation) can be uniquely determined by P(0)\mathcal{P}^{(0)} in the sense of the constrained optimization above. Furthermore, a variant M\mathcal{M} obtained by a simple modification to a relation mj↦δ(mj)m_j \mapsto \delta(m_j) in M(0)\mathcal{M}^{(0)} has dC(M)=1d_C(\mathcal{M})=1 (since dC(M)>0d_C(\mathcal{M})>0 and dC(M)≤1d_C(\mathcal{M})\leq1). Therefore, it can be uniquely identified in the sense of the constrained optimization above if no equally simple modification to another single relation mi↦δ~(mi)m_i \mapsto \tilde\delta(m_i) yields the same P\mathcal{P} induced by M\mathcal{M}. We check this by fixing P\mathcal{P} and the remaining relations mk≠im_{k\neq i}, then solving inversely for mim_i to determine whether such an alternative exists and, if so, how the relation must be modified. If, for every i≠ji\neq j, the inverse solution either does not exist or requires a substantially more complex modification (len[δ~(mi)]−len[mi]>len[δ(mj)]−len[mj]+20\text{len}[\tilde{\delta}(m_i)]-\text{len}[m_i] > \text{len}[\delta(m_j)]-\text{len}[m_j]+20 in our implementation), we accept M\mathcal{M} as an identifiable task: other possible mechanisms being able to yield the same P\mathcal{P} require either more assumptions or substantially more complex assumptions, making them sufficiently distinct from M\mathcal{M} and thus supporting M\mathcal{M} to be uniquely identified from P\mathcal{P} (and the observations DD generated from P\mathcal{P}). For variants combining q>1q>1 modifications, this design similarly ensures that M(0)\mathcal{M}^{(0)} cannot yield the same P\mathcal{P} through fewer than qq modifications (or qq alternative modifications that are not substantially more complex than those in M\mathcal{M}), thereby ensuring that M\mathcal{M} is uniquely identifiable from P\mathcal{P}.

Although these considerations are already reflected in §3.3–4.2, we believe they warrant a clearer discussion. We have therefore incorporated the above explanation into the revised manuscript.

AuthorsOpenReview ↗

Reply to Reviewer Q7Df (2/5)

W2. Lack of baselines involving human experts or other measures of task solvability. Human expert performance is crucial for understanding the inherent solvability of these tasks. The current model mechanism recovery rate is low; however, the paper fails to present baselines involving domain experts, graduate students, or other humans. Nor does it clarify whether humans could recover the reference mechanism for these tasks based solely on the same observations and scientific background.

Thanks the reviewer for this suggestion. MechBench can be viewed as a more demanding extension of symbolic regression, where agents must identify not only the observable law P\mathcal P but also the underlying mechanism M\mathcal M. Therefore, following widely used symbolic regression benchmarks (e.g., the Feynman dataset, SRBench, and LLM-SRBench), which focus on evaluating automated discovery methods rather than human performance, we did not include a human performance baseline.

Nevertheless, we agree that task solvability and difficulty warrant a more explicit analysis and discussion. As clarified in our response to W1 and Q1, our construction and screening support the solvability of recovering the reference mechanism M\mathcal M from observations DD and scientific context CC under the stated scientific assumptions, since the screened scientifically inequivalent alternatives yielding the same P\mathcal P require either more modifications to established scientific knowledge or substantially more complex modifications. To analyze task difficulty, we draw an analogy with symbolic regression, where the operator set and expression length help characterize the search space. Similarly, mechanism discovery can be examined along two dimensions: the number of candidate mechanistic relations and variants admissible under CC, and the number of relations mjm_j composing a mechanism model M\mathcal M.

For the number of relations composing a mechanism model, we find it generally small: across all 512 instances, the median is 10, and 78.5% contain at most 15. On the other hand, these relations already encode scientific knowledge, allowing agents to combine meaningful scientific relations instead of constructing an entire expression operator by operator as in symbolic regression. For the candidate mechanisms, although their set is not prescribed like the operator set in symbolic regression, they are constrained by the scientific context CC and recur across problems. Specifically, a semantic audit of the 16 textbook reference models identifies 50 distinct scientific relation types represented by 89 relation entries. These types fall into 17 broader mechanism classes, with 5–10 classes represented in each domain. Counting each class only once per problem gives 46 class occurrences, which reduce to 17 distinct classes across problems (37.0%). This indicates that the benchmark contexts repeatedly draw on relatively small collections of scientific mechanisms, such as population balances, reaction kinetics, and linear transport. Therefore, a well-designed AI system equipped with established domain knowledge could feasibly use the scientific context CC to identify a relatively small set of plausible candidate mechanisms for further evaluation against the observations. Together, these findings suggest that discovery can proceed by selecting, adapting, and combining a small number of familiar scientific relations, guided by the observations and scientific context, which provides concrete support for task feasibility.

Figure (anonymous link): https://hackmd.io/_uploads/r1oceCV0fg.png

AuthorsOpenReview ↗

Reply to Reviewer Q7Df (3/5)

Q2. The judgments regarding "comparable complexity" and "deviation from scientific background" lack formal definitions. The paper states: "If such an alternative requires comparable or lower structural complexity and no greater departure from the relevant scientific background knowledge than the intended modification..." However, the criteria for these judgments remain unclear. Do these assessments rely primarily on the subjective decisions?

The quantitative evaluation is performed objectively by Python code, without subjective judgment from LLMs or human evaluators. Since the reference mechanism M\mathcal{M} is derived from the textbook mechanism M(0)\mathcal{M}^{(0)} and no scientifically inequivalent textbook alternatives yielding the same phenomenal model have been found, the degree of “departure from relevant scientific background knowledge” can be measured by the departure from M(0)\mathcal{M}^{(0)}. For a reference mechanism M\mathcal{M} obtained by applying qq modifications to M(0)\mathcal{M}^{(0)}, we check whether a competing mechanism M′\mathcal{M}' yielding the same P\mathcal{P} can be obtained through fewer than qq modifications (a smaller departure from scientific background knowledge), or through qq modifications that are not substantially more complex than those in M\mathcal{M} (with an increase in expression length exceeding that of the reference modifications by no more than 20 mathematical symbols).

Reviewer_q7CSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MechBench tests whether scientific agents can uncover underlying mechanisms, rather than just fit equations to observed data. It includes 512 synthetic instances across 16 scientific families, built by combining interpretable changes to the mechanisms. Agents receive observations and scientific context, then submit a mechanism before answering hidden probes. Across different models and evaluation setups, recovering mechanisms through these probes is harder than recovering observable laws. Performance also drops on mutated instances, even when the correct observable law is already given to agents.

Critical strengths
  • New distinction introduced. Probing scientifically defined internal quantities offers a concrete way to assess explanatory structure beyond input–output fit. Using quantities with scientific meanings also allows equivalent mechanisms to use different intermediate symbols or formulations.
  • The experiments expose a substantial gap worth understanding. GPT-5.6-sol with Codex scores 35% on law recovery and 14% on mechanism recovery on the core-set.
Critical weaknesses
  • Limited Novelty. Scientific equation discovery is extensively studied in literature, both SR-like tasks and agentic tasks. Mechbench mainly introduced the mechanism probe for recovering latent scientific variables, which is also not entirely novel. The experimental findings are meaningful though.
Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Reply to Reviewer q7CS (1/1)

We thank the reviewer for the careful review and for recognizing the significance of investigating scientific mechanism discovery beyond phenomenal-law recovery. Below is our discussion regarding the weakness you mentioned:

Limited Novelty. Scientific equation discovery is extensively studied in literature, both SR-like tasks and agentic tasks. Mechbench mainly introduced the mechanism probe for recovering latent scientific variables, which is also not entirely novel. The experimental findings are meaningful though.

Although scientific equation discovery has been extensively studied through both traditional symbolic regression (SR) methods and more recent LLM-agent-based approaches, MechBench does not seek to introduce yet another equation-search algorithm. Rather, it revisits the formulation of existing scientific equation discovery tasks and the scientific discovery capabilities they are designed to provide. Specifically, existing SR tasks and related benchmarks typically operationalize scientific discovery as recovering a mathematical expression that describes the input-output relationship in observational data (i.e., phenomenal-law recovery). In contrast, a mechanistic model characterizes the system through a set of scientifically meaningful internal relations and the physical quantities involved. These relations jointly entail the phenomenal law, which captures the observable input-output relationship resulting from the underlying mechanism. Consequently, recovering a correct phenomenal law does not necessarily establish that the underlying scientific mechanism has also been recovered. This gap motivates MechBench, which extends the scope of existing scientific equation discovery tasks beyond observable equation recovery to mechanism recovery, and operationalizes this distinction in a benchmark that enables quantitative evaluation of both capabilities. This systematic operationalization of the distinction is precisely what makes MechBench novel.

Moreover, we surveyed existing studies on discovering latent scientific variables [1–3] and found that they generally aim to identify latent or intrinsic variables/coordinates z=h(x)z=h(x) from observational data xx, so as to obtain a compact and non-redundant representation of the system state and/or a tractable description of its dynamics (e.g., z˙=g(z)\dot{z}=g(z)). These studies therefore primarily seek a mathematically convenient representation of the observed system and its dynamics. Even when the identified latent variables can be associated with interpretable physical quantities, identifying these quantities alone does not establish the scientific relations governing them. For example, recovering Kepler's third law T2=4π2r3/(GM)T^2=4\pi^2r^3/(GM) from orbital observations, or finding a coordinate representation that simplifies the description of orbital dynamics, does not by itself recover the underlying Newtonian mechanism. The latter requires identifying physically meaningful intermediate quantities, such as gravitational force and orbital acceleration, and the fundamental relations connecting them, from which Kepler's law can be derived. This is precisely where MechBench differs: it requires agents to recover a set of scientifically meaningful internal quantities and, more importantly, the fundamental scientific relations that connect these quantities to one another and to observable variables. These relations must be grounded in relevant physical principles and domain knowledge. An arbitrary mathematical transformation hh, even if it leads to a simpler representation of the input-output relationship or system dynamics, does not by itself constitute a mechanistic model if the resulting variables lack appropriate physical interpretations or the relations are inconsistent with established scientific knowledge. We have incorporated this discussion into the revised manuscript.

[1] Champion et al. Data-driven discovery of coordinates and governing equations. PNAS (2019).

[2] Chen et al. Automated discovery of fundamental variables hidden in experimental data. Nature Computational Science (2022).

[3] Floryan & Graham. Data-driven discovery of intrinsic dynamics. Nature Machine Intelligence (2022).

AuthorsOpenReview ↗

Follow-up on Our Responses

Thank you again for your thoughtful review. We have provided detailed responses to your comments, including additional analyses and discussions. We would greatly appreciate it if you could take a look at our responses when you have a chance. Please let us know if any concerns remain, and we would be happy to provide further clarification. Thank you for your time and consideration.

Reviewer_CYNoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MechBench, a benchmark designed to evaluate the discovery of underlying scientific mechanisms rather than simply recovering mathematical laws that describe observed phenomena. The benchmark addresses limitations in existing symbolic regression and scientific discovery benchmarks by introducing mechanism probes to assess internal scientific relationships, constructing unfamiliar mechanisms through controlled mutations of canonical models, and screening for competing mechanisms to reduce ambiguity. Experiments across multiple LLM-based scientific agents reveal a substantial gap between phenomenal-law recovery and mechanism recovery, particularly as the underlying mechanisms become increasingly unfamiliar.

Critical strengths

S1. The distinction between phenomenal-law discovery and mechanism discovery is compelling and addresses an important limitation in current benchmarks for symbolic regression and AI-driven scientific discovery. The authors provide a clear motivation for why recovering mathematical relationships based only on observable variables is insufficient to demonstrate scientific understanding.

S2. The benchmark construction is good, particularly the use of mechanism probes to evaluate internal scientific relationships and the screening for mechanistic indistinguishability. These are important considerations for evaluating mechanism discovery, as different formulations may represent equivalent mechanisms, while different mechanisms may produce the same observable behavior.

Critical weaknesses

W1. While the paper positions MechBench as addressing limitations in existing symbolic regression and scientific discovery benchmarks, the experimental evaluation focuses almost exclusively on LLM-based scientific agents, with PySR as the only non-LLM baseline. Symbolic regression is a broader research area, with several established deep learning-based approaches, including transformer-based and physics-informed methods. The lack of these baselines limits the comprehensiveness of the evaluation and makes it difficult to assess how existing approaches to scientific discovery perform on the proposed benchmark. These methods could be evaluated using the same two-stage pipeline introduced for PySR, where a recovered phenomenal law is used for subsequent mechanism reasoning.

W2. PySR achieves nearly zero symbolic recovery accuracy, without a clear explanation for such poor performance. While the numerical results suggest that PySR can approximate some of the underlying relationships, it remains unclear whether its poor symbolic recovery is due to the intrinsic difficulty of the benchmark, limitations in the search space or optimization, or the strictness of the evaluation criteria. A more detailed failure analysis would help establish whether the benchmark is appropriately evaluating existing symbolic regression approaches.

W3. The evaluation of mechanism recovery requires all mechanism probes to be correct, whereas phenomenal-law recovery evaluates a single mathematical relationship. While recovering a mechanism naturally involves multiple relationships, this difference in evaluation criteria could partially explain the substantial gap observed between phenomenal-law and mechanism recovery. It remains unclear to what extent this gap reflects limitations in mechanistic reasoning rather than differences in task complexity and evaluation strictness. Reporting individual-probe accuracy, in addition to the current task-level metric, and analyzing performance as a function of the number of probes would help disentangle these factors and strengthen the paper's main empirical claim.

Questions for authors

Q1. Symbolic regression is a broad research area that extends well beyond LLM-based scientific agents, with established approaches (noted above). For this paper to serve as a benchmark for AI-driven scientific discovery, it is important to evaluate a representative range of these approaches rather than focusing almost exclusively on LLM-based agents. Could the authors expand the evaluation to include representative symbolic regression approaches?

Q2. What happens when an agent proposes a scientifically plausible mechanism that was not considered during benchmark construction?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer CYNo (1/3)

W1. While the paper positions MechBench as addressing limitations in existing symbolic regression and scientific discovery benchmarks, the experimental evaluation focuses almost exclusively on LLM-based scientific agents, with PySR as the only non-LLM baseline. Symbolic regression is a broader research area, with several established deep learning-based approaches, including transformer-based and physics-informed methods. The lack of these baselines limits the comprehensiveness of the evaluation and makes it difficult to assess how existing approaches to scientific discovery perform on the proposed benchmark. These methods could be evaluated using the same two-stage pipeline introduced for PySR, where a recovered phenomenal law is used for subsequent mechanism reasoning.

Q1. Symbolic regression is a broad research area that extends well beyond LLM-based scientific agents, with established approaches (noted above). For this paper to serve as a benchmark for AI-driven scientific discovery, it is important to evaluate a representative range of these approaches rather than focusing almost exclusively on LLM-based agents. Could the authors expand the evaluation to include representative symbolic regression approaches?

We thank the reviewer for highlighting the importance of evaluating a broader range of symbolic regression approaches. We recognize that symbolic regression is a broad research field with various well-established methods. Beyond traditional genetic programming-based approaches such as PySR, representative methods include physics-inspired approaches such as AI-Feynman, Transformer-based deep learning approaches such as E2ESR, and methods that incorporate LLMs into predefined evolutionary search workflows, such as LLM-SR. These approaches have all been extensively studied and developed, representing mature lines of research in symbolic regression.

Our original evaluation primarily focused on LLM-based scientific agents because MechBench aims to evaluate scientific mechanism discovery (M\mathcal{M}), rather than merely symbolic recovery of the phenomenal law (P\mathcal{P}). Although an SR method can be coupled with an LLM to form a two-stage pipeline, where the recovered phenomenal law is subsequently used for mechanism reasoning (as in our PySR + Direct-Ask evaluations), our "Gold-P Direct-Ask" experiment already provides an oracle upper-bound control for the second-stage mechanism reasoning, in which the ground-truth phenomenal law P\mathcal{P} is directly provided to the LLM, bypassing the equation-discovery stage entirely. Nevertheless, a substantial gap in mechanism recovery remains even with access to the correct phenomenal law. For example, GPT-5.6-sol achieves only 45.00% and 49.02% mechanism recovery accuracy on the Core-set and Full-set, respectively, under this oracle setting. With DeepSeek-v4-flash-0731 (the model used in our SR + Direct-Ask pipeline), the accuracy is even lower, at just 10.00% and 4.10%. This indicates that recovering the phenomenal law alone does not guarantee successful mechanism discovery, and that the two-stage pipeline remains fundamentally limited by the subsequent mechanism-reasoning capability.

Nevertheless, to directly address the reviewer's suggestion, we have expanded our evaluation to include three additional representative SR methods: AI-Feynman, E2ESR, and LLM-SR. We used their official implementations, together with the officially released pretrained weights for E2ESR and the same DeepSeek-v4-flash-0731 model for LLM-SR. As shown in the table below, all three additional methods achieve 0.00% symbolic phenomenal-law recovery accuracy on MechBench, consistent with the result of PySR. The reasons behind this consistently low recovery performance are further discussed in our response to W2 below.

SR MethodApproachSA(P)↑SA(\mathcal P)\uparrow
PySRGenetic programming0.00%
AI-FeynmanPhysics-inspired SR0.00%
E2ESRTransformer-based SR0.00%
LLM-SRLLM-guided evolutionary SR0.00%
AuthorsOpenReview ↗

Reply to Reviewer CYNo (2/3)

W2. PySR achieves nearly zero symbolic recovery accuracy, without a clear explanation for such poor performance. While the numerical results suggest that PySR can approximate some of the underlying relationships, it remains unclear whether its poor symbolic recovery is due to the intrinsic difficulty of the benchmark, limitations in the search space or optimization, or the strictness of the evaluation criteria. A more detailed failure analysis would help establish whether the benchmark is appropriately evaluating existing symbolic regression approaches.

The near-zero symbolic recovery accuracy of PySR (and the three additional SR methods) is largely attributable to the substantially higher complexity of MechBench's target phenomenal laws, which goes beyond the practical capabilities of direct symbolic search. As shown in the figure below, the median expression-tree length of phenomenal laws P\mathcal{P} in MechBench is 80, far exceeding the median of 13 in SRBench, a conventional benchmark for symbolic regression. To better illustrate this, the table below provides representative examples from the two benchmarks. In particular, the Allosteric Regulation example combines contributions from multiple ligand-binding states and conformation-dependent catalytic activities, resulting in a highly nested rational expression with 65 expression-tree nodes, substantially exceeding the sizes of the representative SRBench equations. This expression is too complex to be discovered by traditional symbolic regression methods such as PySR, which can hardly discover expressions longer than 20 expression-tree nodes.

Nevertheless, the high complexity of these phenomenal laws does not make MechBench an unreasonable benchmark. Despite their substantially greater complexity compared with conventional SR benchmarks, LLM-based scientific agents can still successfully recover a considerable fraction of them. For example, although the Allosteric Regulation task shown in the table is beyond the capabilities of traditional symbolic regression methods, the Codex agent powered by GPT-5.6-sol still successfully recovered an expression algebraically equivalent to the reference equation. This is partly attributable to LLM agents' stronger compositional reasoning in constructing complex expressions from simpler relations, which contributes to their stronger symbolic-regression capabilities. More importantly, MechBench does not require agents to directly recover the fully expanded P\mathcal{P}. Instead, agents are asked to construct a mechanistic model M={m1,…,mJ}\mathcal{M}=\{m_1,\ldots,m_J\}, where each mjm_j is a comparatively simple and scientifically meaningful relation constrained by domain knowledge. This structured formulation allows LLM agents to exploit their compositional reasoning capabilities to identify and assemble these simpler relations into a coherent mechanism. In contrast, traditional SR methods such as PySR directly search for the entire expression P\mathcal{P} without exploiting such mechanistic decomposition, making exact symbolic recovery considerably more challenging.

Figure (anonymous link): https://hackmd.io/_uploads/HJlXS0fAGl.png

TaskFormulaLength
(SRBench) Feynman II.3.24y=Pwr4πr2\displaystyle y=\frac{P_{\mathrm{wr}}}{4\pi r^2}9
(SRBench) Feynman I.13.12y=Gm1m2(1r2−1r1)\displaystyle y=Gm_1m_2\left(\frac1{r_2}-\frac1{r_1}\right)13
(SRBench) Feynman II.24.17y=ω2c2−π2d2\displaystyle y=\sqrt{\frac{\omega^2}{c^2}-\frac{\pi^2}{d^2}}18
(MechBench) Allosteric Regulation (Variant)P=s[1200(1+14s)+1403(1+r0.003(1+10s))2]1+100s(1+14s)+2(1+100s3)(1+r0.003(1+10s))2+0.16(r0.003(1+10s))3P=\frac{s\left[1200(1+14s)+\frac{140}{3}\left(1+\frac{r}{0.003(1+10s)}\right)^2\right]}{1+100s(1+14s)+2\left(1+\frac{100s}{3}\right)\left(1+\frac{r}{0.003(1+10s)}\right)^2+0.16\left(\frac{r}{0.003(1+10s)}\right)^3}65
AuthorsOpenReview ↗

Reply to Reviewer CYNo (3/3)

W3. The evaluation of mechanism recovery requires all mechanism probes to be correct, whereas phenomenal-law recovery evaluates a single mathematical relationship. While recovering a mechanism naturally involves multiple relationships, this difference in evaluation criteria could partially explain the substantial gap observed between phenomenal-law and mechanism recovery. It remains unclear to what extent this gap reflects limitations in mechanistic reasoning rather than differences in task complexity and evaluation strictness. Reporting individual-probe accuracy, in addition to the current task-level metric, and analyzing performance as a function of the number of probes would help disentangle these factors and strengthen the paper's main empirical claim.

Thanks for this helpful suggestion. To examine whether the observed performance gap is affected by the aggregation of multiple mechanism probes into a single task-level metric, we introduce two alternative evaluation metrics. Specifically, alongside the original all-probes-correct accuracy, we introduce an expected single-probe accuracy that assigns equal weight to each probe within a task and averages the per-task probe accuracy across all tasks, i.e., 1N∑i=1NciKi\frac{1}{N}\sum_{i=1}^{N}\frac{c_i}{K_i} (KiK_i is the number of probes for task ii and cic_i is the number of correctly answered probes), as well as an any-probe-correct accuracy, which considers a task successful if at least one of its probes is answered correctly.

As shown in the figure below, compared with the original all-probes-correct accuracy of 13.75% on the Core-set, the expected single-probe accuracy increases by only 2.71 percentage points. Even under the more permissive any-probe-correct criterion, the accuracy increases by only 5.00 percentage points. These increases are substantially smaller than the performance gap between phenomenal-law recovery and mechanism recovery, which is 21.25 percentage points. Therefore, although the all-probes-correct criterion is indeed more stringent, it cannot account for most of the observed performance gap. We have added these results and the corresponding discussion to the revised manuscript.

Figure (anonymous link): https://hackmd.io/_uploads/HJldDKV0Mg.png


Q2. What happens when an agent proposes a scientifically plausible mechanism that was not considered during benchmark construction?

During task construction, we systematically screen for and rule out competing mechanisms to ensure that M\mathcal{M} is the mechanism closest to established scientific knowledge among those yielding the same P\mathcal{P}. Consequently, any scientifically inequivalent mechanism M′\mathcal{M}' proposed by an agent that yields the same P\mathcal{P} requires either more assumptions or substantially more complex assumptions (e.g., assumptions requiring more than 20 additional mathematical symbols to express) than the reference mechanism M\mathcal{M}. We therefore regard such mechanisms as less scientifically plausible than the reference mechanism M\mathcal{M} and classify them as failures of mechanism discovery.

To explain how this is achieved, we provide a brief account here, which, although already discussed in §3.3–4.2, is presented more formally here for clarity: In MechBench, we seek not just any mechanism M\mathcal{M} that mathematically yields P\mathcal{P} and the corresponding observations DD, but the mechanism M⋆\mathcal{M}^\star requiring the fewest modifications to established knowledge. Specifically, we seek M⋆=argmin⁡M:M⇒PdC(M)\mathcal{M}^\star=\operatorname{argmin}_{\mathcal{M}:\mathcal{M}\Rightarrow\mathcal{P}}d_C(\mathcal{M}), where dC(M)d_C(\mathcal{M}) is the minimum number of modifications needed to obtain M\mathcal{M} from established mechanisms under CC, with scientifically equivalent formulations treated as the same mechanism. This formulation reflects scientists’ pursuit of parsimonious explanations grounded in existing knowledge and aligns with Kuhn’s account of “normal science” as puzzle-solving within an established paradigm. Thus, a textbook mechanism M(0)\mathcal{M}^{(0)} has dC(M(0))=0d_C(\mathcal{M}^{(0)})=0 (since it is already part of established scientific knowledge) and can be uniquely identified from P(0)\mathcal{P}^{(0)} under the above criterion (since we exclude tasks admitting scientifically inequivalent established mechanisms that yield the same P(0)\mathcal{P}^{(0)} during construction). On the other hand, for a mechanism M\mathcal{M} obtained from M(0)\mathcal{M}^{(0)} by modifying mj↦δ(mj)m_j\mapsto\delta(m_j), we check for competing mechanisms by fixing P\mathcal{P} and the remaining relations {mk:k≠i}\{m_k:k\neq i\}, then solving inversely for alternative modifications mi↦δ~(mi)m_i\mapsto\widetilde{\delta}(m_i) that yield the same P\mathcal{P}. If, for every i≠ji\neq j, no such solution exists or the required modification is substantially more complex than δ(mj)\delta(m_j), we accept M\mathcal{M} as an identifiable task under the above criterion.

AuthorsOpenReview ↗

Follow-up on Our Responses

Thank you again for your thoughtful review. We have provided detailed responses to your comments, including additional analyses and experimental results. We would greatly appreciate it if you could take a look at our responses when you have a chance. Please let us know if any concerns remain, and we would be happy to provide further clarification. Thank you for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Key Points from the Discussion

Dear Area Chair,

Thank you for overseeing the review discussion. As the rebuttal period comes to a close, we would like to briefly summarize MechBench’s core contribution along with the new results and clarifications added during the discussion.

Benchmark contribution. MechBench separately evaluates whether an agent can recover an observable law and whether it can recover the scientific relations that jointly generate that law. After the agent submits a mechanistic model, hidden probes test its predictions about scientifically meaningful internal quantities. This design lets us measure how often agents recover the correct law while missing the mechanism that explains it. In this process, our use of unobservable variables as mechanism probes plays a crucial role and serves a distinct purpose from the latent-variable discovery discussed by q7CS, which typically seeks compact representations of system states or dynamics.

Experiments and results. In response to CYNo’s questions about our experimental setups and results, we evaluated three additional symbolic regression baselines and examined how probe scoring affects the recovery gap. The results show that AI-Feynman, E2ESR and LLM-SR achieved 0% exact symbolic recovery of observable laws, like PySR. To help explain this result, we compared target expression sizes and found a median of 80 expression-tree nodes in MechBench, compared with 13 in SRBench. This difference helps explain why direct symbolic search struggles on MechBench. To assess how probe scoring affects the recovery gap, we counted a task as successful whenever at least one mechanism probe was answered correctly and found that the gap remained substantial at 16.25 percentage points. Taken together, these results show that symbolic regression baselines struggle with law recovery and that strict probe scoring explains little of the recovery gap, thus further strengthening our empirical evaluation.

Theoretical identifiability and solvability. To address Q7Df’s remaining concerns, we explained how MechBench establishes theoretical identifiability through a minimal-change criterion. For each candidate set of modifications, we hold the remaining textbook relations fixed and solve for changes that could reproduce the same observable law. We retain a task only if this screening rules out scientifically distinct alternatives that require fewer modifications or the same number of comparably simple modifications. This makes the reference mechanism uniquely identifiable under the criterion, and further gives a constructive route to solving the tasks: In principle, an agent can identify the textbook mechanism from the scientific context and systematically test candidate modifications, proceeding from changes to individual relations to combinations of increasing size, until it finds the minimal set of changes that reproduces the observable law recovered from the data.

Thank you for your time and consideration in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.