- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MechBench, a benchmark designed to evaluate whether AI scientific agents can recover the internal scientific mechanisms governing observed phenomena. The authors distinguish between "phenomenal-law recovery" and "mechanism recovery": the former requires only the recovery of observable input-output relationships, whereas the latter demands the recovery of internal variables and their scientific interrelationships. The dataset comprises 16 task families spanning physics, chemistry, biology, and materials science. Models are required to propose mechanisms based on scientific context and observational data, after which they are evaluated using two categories of metrics. The authors also employ a "mechanistic indistinguishability screening" process to exclude tasks where plausible competing mechanisms exist. Experimental results indicate that while models achieve high accuracy in recovering phenomenal laws, their accuracy in recovering mechanisms is significantly lower; furthermore, performance in mechanism recovery degrades rapidly as the number of mechanistic variations increases.
Critical strengths
Strengths:
- The problem is significant and practically relevant. Most existing benchmarks for scientific discovery focus solely on fitting observed patterns; by isolating the recovery of mechanistic structures as a distinct challenge, this paper addresses a valuable research problem.
- Experimental controls are comprehensive. The paper reports performance across phenomenal-law recovery, mechanism recovery, and both in-distribution (ID) and out-of-distribution (OOD) settings. It also incorporates a "Gold-P control" to help distinguish between failures in pattern discovery and failures in mechanistic reasoning.
- Broad cross-disciplinary coverage. The tasks span physics, chemistry, biology, and materials science, and the paper provides detailed explanations regarding task construction and evaluation procedures.
Critical weaknesses
Weaknesses:
- Mechanistic identifiability is not fully guaranteed. This is the paper's most critical issue. Multiple distinct internal mechanisms () can often produce the same phenomenal law. Even with noise-free data and an infinite sample size, the observed relationships alone are typically insufficient to uniquely identify the underlying mechanism. Although the authors propose a "mechanistic indistinguishability screening" and search for competing mechanisms (Section 4.2), the absence of competing mechanisms does not prove that the current mechanism is the only plausible one. Consequently, the problem might effectively reduce to whether the inferred mechanism merely satisfies the observed phenomena, a conclusion that runs counter to the paper's original intent.
- Lack of baselines involving human experts or other measures of task solvability. Human expert performance is crucial for understanding the inherent solvability of these tasks. The current model mechanism recovery rate is low; however, the paper fails to present baselines involving domain experts, graduate students, or other humans. Nor does it clarify whether humans could recover the reference mechanism for these tasks based solely on the same observations and scientific background.
Questions for authors
- Please see the discussion of mechanistic identifiability above. How do the authors establish that the retained benchmark instances are mechanistically identifiable within the intended scientific hypothesis class?
- The judgments regarding "comparable complexity" and "deviation from scientific background" lack formal definitions. The paper states: "If such an alternative requires comparable or lower structural complexity and no greater departure from the relevant scientific background knowledge than the intended modification..." However, the criteria for these judgments remain unclear. Do these assessments rely primarily on the subjective decisions?
- Some configurations in the paper utilize the average of three runs, while others are run only once. Appendix B.1 notes that for the Core-set, Codex+GLM-5.3-flash and Codex+DeepSeek-v4-flash-0731 involved three runs, whereas other configurations typically involved only one. Why were only certain configurations replicated? Did the authors consider standardizing the experimental workflow?
- PySR’s recovery rate for "phenomenal laws" is virtually zero. What is the reason for this? Is PySR a reasonable baseline for this task?
My rating might improve if the first issue is well resolved. If the other issues are also addressed, I would be happy to consider raising the score further.
9 comments
Reply to Reviewer Q7Df (2/5)
W2. Lack of baselines involving human experts or other measures of task solvability. Human expert performance is crucial for understanding the inherent solvability of these tasks. The current model mechanism recovery rate is low; however, the paper fails to present baselines involving domain experts, graduate students, or other humans. Nor does it clarify whether humans could recover the reference mechanism for these tasks based solely on the same observations and scientific background.
Thanks the reviewer for this suggestion. MechBench can be viewed as a more demanding extension of symbolic regression, where agents must identify not only the observable law but also the underlying mechanism . Therefore, following widely used symbolic regression benchmarks (e.g., the Feynman dataset, SRBench, and LLM-SRBench), which focus on evaluating automated discovery methods rather than human performance, we did not include a human performance baseline.
Nevertheless, we agree that task solvability and difficulty warrant a more explicit analysis and discussion. As clarified in our response to W1 and Q1, our construction and screening support the solvability of recovering the reference mechanism from observations and scientific context under the stated scientific assumptions, since the screened scientifically inequivalent alternatives yielding the same require either more modifications to established scientific knowledge or substantially more complex modifications. To analyze task difficulty, we draw an analogy with symbolic regression, where the operator set and expression length help characterize the search space. Similarly, mechanism discovery can be examined along two dimensions: the number of candidate mechanistic relations and variants admissible under , and the number of relations composing a mechanism model .
For the number of relations composing a mechanism model, we find it generally small: across all 512 instances, the median is 10, and 78.5% contain at most 15. On the other hand, these relations already encode scientific knowledge, allowing agents to combine meaningful scientific relations instead of constructing an entire expression operator by operator as in symbolic regression. For the candidate mechanisms, although their set is not prescribed like the operator set in symbolic regression, they are constrained by the scientific context and recur across problems. Specifically, a semantic audit of the 16 textbook reference models identifies 50 distinct scientific relation types represented by 89 relation entries. These types fall into 17 broader mechanism classes, with 5–10 classes represented in each domain. Counting each class only once per problem gives 46 class occurrences, which reduce to 17 distinct classes across problems (37.0%). This indicates that the benchmark contexts repeatedly draw on relatively small collections of scientific mechanisms, such as population balances, reaction kinetics, and linear transport. Therefore, a well-designed AI system equipped with established domain knowledge could feasibly use the scientific context to identify a relatively small set of plausible candidate mechanisms for further evaluation against the observations. Together, these findings suggest that discovery can proceed by selecting, adapting, and combining a small number of familiar scientific relations, guided by the observations and scientific context, which provides concrete support for task feasibility.
Figure (anonymous link): https://hackmd.io/_uploads/r1oceCV0fg.png
Reply to Reviewer Q7Df (3/5)
Q2. The judgments regarding "comparable complexity" and "deviation from scientific background" lack formal definitions. The paper states: "If such an alternative requires comparable or lower structural complexity and no greater departure from the relevant scientific background knowledge than the intended modification..." However, the criteria for these judgments remain unclear. Do these assessments rely primarily on the subjective decisions?
The quantitative evaluation is performed objectively by Python code, without subjective judgment from LLMs or human evaluators. Since the reference mechanism is derived from the textbook mechanism and no scientifically inequivalent textbook alternatives yielding the same phenomenal model have been found, the degree of “departure from relevant scientific background knowledge” can be measured by the departure from . For a reference mechanism obtained by applying modifications to , we check whether a competing mechanism yielding the same can be obtained through fewer than modifications (a smaller departure from scientific background knowledge), or through modifications that are not substantially more complex than those in (with an increase in expression length exceeding that of the reference modifications by no more than 20 mathematical symbols).
Reply to Reviewer Q7Df (1/5)
Thanks for highlighting this issue. Since mechanistic identifiability is central to MechBench’s design and validity, we agree that it deserves a clearer explanation.
Following §3.1, a mechanism M={m1,…,mJ} comprises scientifically meaningful relations mj involving observable variables (x,y) and unobservable internal quantities h. Eliminating h yields a phenomenal model P:FP(x,y)=0, from which observations D can be generated. From a purely mathematical perspective, recovering a unique M from P appears underdetermined, since infinitely many mathematically possible mechanisms may yield the same P. However, in scientific discovery from data, we typically seek a mechanism M that not only yields an appropriate P to describe the observations D, but also stays as close as possible to established scientific knowledge—that is, requires minimal modifications to existing scientific assumptions. This motivates the following constrained optimization: M∗=argminM:M⇒PdC(M), where dC(M) measures the minimum number of modifications needed to obtain M from established mechanisms under C.
For example, a textbook mechanism M(0) has dC(M(0))=0, since it is already part of established scientific knowledge. Through LLM-assisted investigation, we check that the established mechanisms under C contain no scientifically inequivalent alternative M(0)′ yielding the same P(0), thereby ruling out alternatives satisfying both dC(M(0)′)=0 and M(0)′⇒P(0). This ensures that M(0) (or a mechanistic model that is scientifically equivalent to it, differing only in its literal formulation) can be uniquely determined by P(0) in the sense of the constrained optimization above. Furthermore, a variant M obtained by a simple modification to a relation mj↦δ(mj) in M(0) has dC(M)=1 (since dC(M)>0 and dC(M)≤1). Therefore, it can be uniquely identified in the sense of the constrained optimization above if no equally simple modification to another single relation mi↦δ~(mi) yields the same P induced by M. We check this by fixing P and the remaining relations mk=i, then solving inversely for mi to determine whether such an alternative exists and, if so, how the relation must be modified. If, for every i=j, the inverse solution either does not exist or requires a substantially more complex modification (len[δ~(mi)]−len[mi]>len[δ(mj)]−len[mj]+20 in our implementation), we accept M as an identifiable task: other possible mechanisms being able to yield the same P require either more assumptions or substantially more complex assumptions, making them sufficiently distinct from M and thus supporting M to be uniquely identified from P (and the observations D generated from P). For variants combining q>1 modifications, this design similarly ensures that M(0) cannot yield the same P through fewer than q modifications (or q alternative modifications that are not substantially more complex than those in M), thereby ensuring that M is uniquely identifiable from P.
Although these considerations are already reflected in §3.3–4.2, we believe they warrant a clearer discussion. We have therefore incorporated the above explanation into the revised manuscript.