acceptodds
Under review as a conference paper at ICLR 2027

MechHypoBench: Evaluating Data-Driven Mechanistic Hypothesis Generation in AI Agents

Abstract

Data-driven mechanistic hypotheses are essential to scientific discovery because they explain how underlying processes produce observed phenomena. AI agents and AI scientists increasingly support scientific data analysis. However, their ability to turn empirical findings into mechanistic hypotheses remains insufficiently examined. To address this gap, we introduce MechHypoBench, the first benchmark for evaluating whether AI agents and AI scientists can generate such hypotheses from empirical data. It combines paper-derived mechanisms from 14 scientific fields with real-world datasets containing 17.98 million records. The construction retains the observational complexity of empirical data while providing a specified underlying mechanism. Agents analyze the observations and propose open-form hypotheses. We develop an evaluation framework that assesses open-form mechanistic hypotheses through their consequence under withheld conditions. Experiments with general agents and AI scientists reveal a substantial gap between generated hypotheses and the underlying mechanisms. Our datasets and codes are available at https://anonymous.4open.science/r/ICLR-2027-benchmark-E1FB.

Primary area:
datasets and benchmarks
Keywords:
Hypothesis Generation, Scientific Hypothesis, LLM Agents, Benchmarking, AI Scientists, Scientific Discovery

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_MDcqOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MechHypoBench, a 300-task benchmark (120 easy, 180 hard) designed to evaluate whether AI agents (Codex, Claude Code) and AI-scientist workflows (AI-Scientist-v2, AutoDiscovery) can generate open-form mechanistic hypotheses from tabular observational data. To construct tasks with known ground truth while preserving empirical data structures (grouping, longitudinal/time ordering, missingness, auxiliary covariates), the authors extract qualitative mechanistic claims from 244 papers across 14 fields, compile each into a fixed numerical DAG ($M^\ast $) built from multilinear interactions, logistic saturation, and exponential decay, and evaluate $M^\ast $ on percentile-ranked columns (Driver, Context, History, Phase) of 150 unrelated real-world datasets to synthesize five response channels (A–E). Agents observe a public subset of rows and propose open-form hypotheses. A frozen LLM interpreter (GPT-5.6 Terra) reads only the submitted hypothesis and predicts directional response changes (decrease, no_change, increase) across 12 withheld one-factor conditional comparisons, scored by Mechanism Consequence Agreement (MCA) and negative log-likelihood (MCNLL). Experiments show that the best agent (Codex + GPT-5.6 Sol) reaches 64.9% easy / 50.8% hard MCA (well below the qualitative and parameterized references at 87.6% / 82.9% and 96.8% / 97.5%), that routed GitHub scientific skills and AI-scientist workflows do not reliably improve MCA, and that supplying precomputed conditional contrasts from the public data (Evidence Summaries) improves MCA by +10.0 to +14.1 percentage points.

Critical strengths
  • Consequence-based evaluation with falsification controls and a clean diagnostic ablation: Scoring open-form hypotheses via their implied directional predictions on withheld one-factor conditional comparisons—rather than surface text overlap—is a well-motivated evaluation design, and the Empty, Shuffled, QMR, and PMR controls confirm that the interpreter distinguishes genuine conditional structure from vacuous or mismatched text. More importantly, the diagnostic interventions in Section 4.5 and Appendices E.5–E.6 (Evidence Summaries vs. Structured Records, plus retrospective hypothesis selection in Table 18) yield a clear, useful insight: agents primarily fail to surface conditional contrasts and interaction sign-reversals during exploratory data analysis (+10 to +14 pp when given precomputed conditional summaries), rather than merely failing to retain notes (+2 pp, n.s.).
Critical weaknesses
  • Substantial mismatch between the "14 scientific fields / domain mechanistic discovery" framing and the anonymized cross-domain task: Inspection of Appendix D, Appendix F, and the supplementary tasks (MechHypoBench.zip) shows that tasks graft a mechanism from one field onto an unrelated dataset from another field (e.g., a Toxoplasma gondii signaling mechanism grafted onto a rural child mental-health survey in task_fcc33a74fab2, or a molecular biology mechanism grafted onto an oceanography CTD dataset in task_02ddb9c16878) and strip all domain semantics (Factor Alpha, Factor Beta, Factor Gamma history, Observation Phase -> Response Channels A–E, where BACKGROUND.md explicitly states "All names are abstract study roles; external domain interpretations are neither defined nor required"). Consequently, the benchmark actually evaluates abstract nonlinear response-surface / conditional-sign recovery on grouped tabular data, not domain-grounded scientific mechanism discovery. This also makes the "scientific skills" experiment (Section 4.3) largely vacuous by construction, since domain-specific scientific skills from GitHub have no domain semantics to match against anonymized variables.
  • Confounding of non-termination with hypothesis quality in headline metrics, QMR sign-indeterminacy, and incomplete human-evaluation reporting:
    1. Across the 12 general-agent configurations, valid output completion rates range from 71.3% to 96.7% (Table 12), and missing/invalid runs are imputed with the Empty control (16.7% MCA). This conflates harness completion rate with scientific hypothesis quality in the main tables (for example, DeepSeek-V4-Flash jumps from 73.3%/76.7% valid in Codex to 92.5%/90.0% valid in Claude Code, accounting for much of the reported harness gap).
    2. In the supplement (ORACLE_HANDOFF.md), the Qualitative Mechanism Reference (QMR) is a symbolic structural-equation dump (component_b := b + w1*(a*decay) - w2*(a*b)), yet it achieves only 87.6% (easy) and 82.9% (hard) MCA because competing positive and negative interaction terms have algebraically indeterminate signs without exact numerical coefficients (PMR).
    3. The human evaluation in Section 3.5 (Table 2) reports post-revision means of 4.54–4.97 across "ten reviewers with scientific backgrounds," but omits how many tasks were evaluated, how many raters scored each task, whether raters were independent of the authors, and inter-rater agreement.
Questions for authors
  1. Conditional-on-valid performance (Clarification): What are the MCA and MCNLL of all 12 general-agent configurations in Table 3 / Table 12 when computed (a) conditional on valid outputs for each configuration and (b) on the common intersection of tasks where compared harnesses/skill settings both produced valid outputs?
  2. Relevance of scientific skills under variable anonymization (Clarification): Given that TASK.md and BACKGROUND.md replace all domain variables with Factor Alpha/Beta/Gamma and Response Channels A–E, what kinds of skills from the 4,043 GitHub skill pool were selected by SkillRouter-Reranker-0.6B, and how often did agents actually load or invoke them?
  3. Human evaluation protocol (Clarification): In Section 3.5, did all 10 reviewers evaluate all 300 tasks, were the reviewers external to the author team, what were the pre-revision scores on the 17 reconstructed mechanisms, and what was the inter-annotator agreement?
9 comments

We sincerely thank you for your careful reading and constructive feedback. We especially appreciate your recognition of our consequence-based evaluation as “a well-motivated evaluation design” and of our diagnostic interventions as yielding “a clear, useful insight” into the roles of evidence extraction and retention in mechanistic hypothesis generation. We address your questions and concerns point by point below.

AuthorsOpenReview ↗

Q1: Performance on valid outputs

All entries separated by / are Easy/Hard. n is the number of included tasks out of 120/180, respectively. MCA is in %, higher is better; lower MCNLL is better. A. Each configuration's valid outputs.

ConfigurationWithout skills: nMCAMCNLLWith skills: nMCAMCNLL
Codex / GPT-5.6 Sol116/17366.2/50.60.899/1.360113/17467.9/52.00.859/1.298
Codex / DeepSeek-V4-Flash88/13850.4/35.61.624/2.12691/14347.6/33.41.765/2.247
Codex / GLM-5.289/12549.3/33.91.573/2.12494/12749.6/32.71.497/2.020
Claude Code / Claude Sonnet 597/15350.1/31.01.422/1.898101/15146.9/31.11.523/1.981
Claude Code / DeepSeek-V4-Flash111/16251.8/36.01.587/2.047114/16546.8/33.21.712/2.204
Claude Code / GLM-5.293/12954.1/33.51.380/2.07595/14548.9/32.01.600/2.145

B. Common-valid tasks: harness comparisons.

BackendSkillsnCodex MCAClaude Code MCACodex MCNLLClaude Code MCNLL
DeepSeek-V4-Flashoff83/12551.1/35.649.9/35.11.596/2.1331.612/2.085
DeepSeek-V4-Flashon89/13048.0/33.647.7/33.71.774/2.2291.708/2.211
GLM-5.2off69/9448.4/34.854.8/32.91.546/2.0611.353/2.109
GLM-5.2on77/10849.8/32.447.9/31.61.493/2.0001.627/2.172

C. Common-valid tasks: skills off/on.

ConfigurationnMCA offMCA onMCNLL offMCNLL on
Codex / GPT-5.6 Sol109/16766.7/51.467.5/52.40.890/1.3430.867/1.290
Codex / DeepSeek-V4-Flash69/11454.0/34.750.5/33.81.516/2.1541.654/2.242
Codex / GLM-5.272/9248.7/33.649.8/33.81.548/2.1061.487/1.952
Claude Code / Claude Sonnet 588/13948.6/30.746.0/30.51.459/1.9051.541/2.006
Claude Code / DeepSeek-V4-Flash105/14952.6/35.946.8/34.11.568/2.0471.711/2.149
Claude Code / GLM-5.275/10754.9/33.649.6/33.61.319/2.0811.575/2.093

The benchmark evaluates whether an agent can turn observations into a mechanistic hypothesis with the available resources. For general agents, the main table uses the same 300 tasks and 30-minute terminal budget. Shared-backend harness comparisons also match inputs, backend, and skill setting. Analysis, tool use, and valid hypothesis delivery are components of the system being evaluated. Differences in successful delivery under this common budget are therefore part of harness performance. Retaining the full task set gives all configurations a consistent evaluation population, using each task's Empty control when no valid submission is delivered. Making common-valid results the primary table would instead select the evaluation population by the systems' completion outcomes and exclude tasks completed by only one system. Beyond the main-table evaluation, our analyses use paired comparisons under matched conditions, with task sets and relevant experimental settings aligned for each contrast.

With DeepSeek-V4-Flash, Claude Code delivers more valid hypotheses under both skill settings within the same budget (A), a harness-level difference. The skill finding also persists on common-valid tasks (C): skill access does not consistently improve consequence agreement.

AuthorsOpenReview ↗

Q2: Scientific skills under variable anonymization

Transferability is an inclusion criterion in skill collection. The pool was curated for scientific data analysis and hypothesis generation that can transfer across research fields. After an initial research-relevance filter, the LLM reviewed candidate names, descriptions, and available content using four criteria: (i) Scientific research goal: Whether the skill directly supports a research question, scientific data, or scientific conclusions. (ii) Scientific reasoning or data-analysis contribution: Whether it provides substantive analysis, reasoning, evidence integration, or hypothesis-generation operations rather than merely restating or presenting content. (iii) Consistency of purpose and content: Whether the advertised research function is supported by identifiable operations and corresponding inputs and outputs. (iv) Cross-disciplinary applicability: Whether its core operations can transfer across research fields rather than depending entirely on a single field's terminology or procedures. Each criterion was scored 0–4. Retention required at least 2 on every criterion and classification as Hypothesis Generation or Data Analysis.

What was selected and why it applies. SkillRouter-Reranker-0.6B selected 32 skills per task using the task description and public dataset overview, structure, and format. Selected skills were exposed as descriptions, with content available on demand. Actual routed and accessed examples include:

  • data-science-statistical-analysis: data exploration, missingness checks, regression diagnostics, and mixed-effects methods for non-independent observations.
  • hypothesis-generation: competing mechanistic explanations, falsifiable predictions, and specifying the conditions under which predictions should hold. These operations can act on the available condition/response observations and data structure without requiring the original field-specific variable names.

We report confirmed accesses in the recorded runs. Access requires content from a selected skill file or skill instructions delivered by a load invocation. Repeated records of the same call count once. Directory listings are excluded. Each configuration has 300 evaluated tasks.

ConfigurationAccess callsTasks with accessRate
Codex / GPT-5.6 Sol564279/30093.0%
Codex / DeepSeek-V4-Flash5252/30017.3%
Codex / GLM-5.2163113/30037.7%
Claude Code / Claude Sonnet 526280/30026.7%
Claude Code / DeepSeek-V4-Flash4747/30015.7%
Claude Code / GLM-5.2129107/30035.7%

Why a skill-access analysis is appropriate. For each task, all skill-enabled configurations share access to the same 32 skills selected by SkillRouter-Reranker-0.6B. The offered guidance is therefore held fixed across harnesses and models for that task. Each harness/model autonomously decides which skill information to read, whether to load or invoke a skill, and how to use its guidance in the analysis. Within each harness/backend, the with-/without-skills comparison holds task inputs and the execution budget fixed while changing access to this resource. This is an appropriate way to test whether agents benefit from scientific guidance, because selecting, retrieving, and integrating useful information are parts of the capability being evaluated.

We have added the skill-access statistics and clarified the rationale for evaluating skill access in the appendix section E.3. These additions are highlighted in red in the revised manuscript.

Reviewer_VkxcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Introduces a benchmark for mechanistic hypothesis generation. From what I gather it is constructed as follows:

  1. Find a qualitative relationship from a natural science paper.
  2. Create a mathematical model inspired by that relationship.
  3. Create synthetic data from that mathematical model where the inputs to the model are gathered from other papers.
  4. An LLM under evaluation sees only the synthetic data, the mathematical model is latent. It is prompted to do some EDA on the data and create a detailed description of the behavior of the hidden model.
  5. That detailed description is passed to a second LLM interpreter model that is prompted to predict some hidden directional behaviors (e.g., changing X causes Y to go up/down/stay the same).

300 tasks, uses mechanisms derived from 244 papers across 14 fields, uses 150 independently collected empirical datasets to create synthetic data.

Critical strengths

Its claimed advance over earlier benchmarks is bringing together messy real-world inputs, computer-generated outcomes with known mathematical ground truth, and tool-using AI agents whose written explanations are judged by whether they predict unseen outcomes.

Critical weaknesses

The withheld directional questions may not be uniquely answerable (statistically identified) from the public data under the assumptions disclosed to the evaluated agent. Different models could explain those observations equally well (statistically speaking) while predicting different directions under the withheld conditions. Assume we had a ground truth explanation that is learnable purely from statistical trends in the data, if we provide this to the interpreter and it gets high accuracy, the interpreter might be making some implicit assumptions (e.g., Occam's razor) that favor one out of a multiple answers that are possible under different latent models that can all explain the data. Because the authors control the mathematical model, they could ensure that the public data contains information that is sufficient to identify each hidden question. That's lacking in the paper now. In other words, for each hidden question, can you say, "Given this public data, and these basic explicit public assumptions, all hidden mathematical models consistent with this data have the same correct answer to this question." for each hidden question.

Questions for authors

See my comment on identifiability.

2 comments
AuthorsOpenReview ↗

Identifiability of hidden matched comparisons

We sincerely thank you for your time and thoughtful, encouraging feedback. We especially appreciate your recognition of MechHypoBench's combination of real-world observational complexity, known mathematical ground truth, and tool-using agents whose written hypotheses are evaluated through their predictions under unseen conditions. These are central to the motivation and design of our benchmark.

Your question concerns whether public observations and explicit assumptions determine that direction, rather than simply whether the benchmark designer knows the answer. We address this distinction through the construction of the comparisons, the public information retained after withholding, and the assumptions needed for directional inference.

Directional reliability is built into task construction. Controlling the mathematical mechanism is precisely what allows us to select comparisons with stable, mechanism-relevant response changes. Appendix D.3 and Algorithm 3 describe those requirements:

  • Observational stability: the mean noise-free responses determine the ground-truth direction. Both condition groups must contain sufficient observations. The recorded mean difference must agree with that direction and exceed the required magnitude. A check further assesses stability under measurement variation.
  • Parameter robustness: among 32 numerical variants preserving graph structure, coefficient signs and qualitative transition bands, at least 90% must retain the assigned direction and required effect magnitude, with a nonnegative robustness margin.
  • Mechanistic relevance: a directional comparison must agree with the full mechanism on the condition grid and change under at least one simplification, such as removing interactions or linearizing nonlinear relationships. No-change comparisons require absence of a directed dependency, and cancellation alone is insufficient. These requirements are applied before agent evaluation. Thus, the reference directions are supported by the construction's observational and mechanistic checks, rather than justified retrospectively by high QMR/PMR scores.

Withholding comparisons is distinct from withholding the experimental roles. Agents receive public observations of the four experimental semantic roles and the response channels, together with grouping, ordering and auxiliary measurements. The split withholds the complete condition combinations used in a matched comparison. Individual role levels may still occur under other public conditions. After withholding, the constructor records whether the compared role levels remain represented and applies the public marginal baseline for comparison selection and difficulty assignment. This checks retained role level support, rather than assuming that a reliable reference direction automatically implies sufficient public evidence.

A candidate mechanism must account for the response relationships observed across the public conditions. Inferring the direction at a withheld combination additionally depends on assumptions about how those relationships extend across context, history and phase. Your all-compatible-model condition is therefore relative to an explicit model class and a criterion for compatibility under measurement variation. In the current tasks, the role meanings and observed measurements are public. The graph, coefficient signs and transition bands used in the robustness checks are properties of the private mechanism. The result established by the existing checks is comparison-level directional reliability under the stated construction conditions.

Thank you for the clarification.

Reviewer_vmnCOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a benchmark for data-driven mechanistic hypothesis generation by AI agents across 14 scientific fields. This is to address the gap that remains between the generated hypotheses and underlying mechanisms in current literature.

Critical strengths

None

Critical weaknesses

Unclear motivations for MCA/MCNLL metrics used in the paper for evaluation. This is consequential to my assessment because the entire evaluation suite lies on such and the metrics themselves are very LLM-output based which further have ambiguities on rubric definition.

Unclear LLM usage and human validation. The exact rubric used for human review for the candidate papers and for how LLMs were used/primed to generate probabilities over the specified distribution for scientific skill collection seem very unclear. Furthermore, the methodology of defining the metrics above based on the unvalidated outputs of the LLM usage seem to have second/third-order effects that limit the reproducibility of this work.

Questions for authors

Here are my notes for points for improvement and clarifying questions:

  • How do you define a fully specified underlying mechanism?
  • What is the rubric here for the human review of the candidate papers?
  • What was the reason exactly 200 datasets were collected?
  • What was the rubric for how the LLM was used/primed to generate probabilities over the specified distribution for scientific skill collection?
  • Can you please motivate MCA/MCNLL further and validate/ground these metrics in theory?
  • Appendix contains task IDs, csv filenames, etc. that are irrelevant to the publication. Please revise to not have such information.
10 comments

We sincerely thank you for your time, careful reading, and constructive feedback. We particularly appreciate your detailed questions regarding the definition of a fully specified mechanism, the human-review rubrics, and the motivation and validation of MCA/MCNLL. These questions help us clarify the benchmark's design and present its supporting evidence more explicitly. We have also incorporated the corresponding clarifications and additions into the revised manuscript, with the changes marked in blue for ease of reference. We address your questions and concerns point by point below.

AuthorsOpenReview ↗

Q1: Definition and scientific review of the underlying mechanism

By a fully specified underlying mechanism, we mean a fixed paper-derived mechanism M∗M^* with an explicitly specified graph of processes, interactions, and responses, including its node functions, numerical parameters, and response mappings. Given a condition c=(cd,cx,ch,cp)c=(c_d,c_x,c_h,c_p) specifying the driver, context, history, and phase, its noise-free responses can be computed directly. A worked example is given in https://anonymous.4open.science/r/ICLR-2027-benchmark-E1FB/mechanism_case/README.md.

As described in Appendix D.1 and Algorithm 1, reviewed mechanistic claims are organized, with LLM assistance, into a graph. Structural validation checks its dependencies and response mappings and behavioral validation examines its numerical behavior on a canonical condition grid. Human review assesses whether the constructed mechanism preserves the scientific meaning of the reviewed claims.

The mechanism review involved ten reviewers with complementary scientific and methodological backgrounds. Each mechanism was assigned to a domain reviewer assessing source claims and experimental semantic roles, and a method reviewer assessing the graph, numerical stability, and suitability of matched comparisons. A mechanism graph was assessed in five dimensions. (i) Faithfulness to source evidence: whether the reviewed claims, supporting passages, and conditions of applicability are faithfully preserved. (ii) Distinct experimental semantic roles: whether driver, context, history, and phase have distinct, scientifically supported meanings. (iii) Consistency of the mechanism graph: whether the represented processes, interactions, dependencies, and response mappings agree with the reviewed claims. (iv) Falsifiability and mechanistic relevance: whether matched comparisons test conditional interactions, distinguish changes from structurally absent effects, and differentiate the full mechanism from simplified variants. (v) Numerical behavior and task suitability: whether response changes are stable, interactions and nonlinear transformations affect the responses, response channels capture distinct behavior, and experimental role alignment supports task construction. Each dimension is scored 0–4: failure, substantial reconstruction required, revision and reassessment required, meeting the construction requirements, and full support for the criterion, respectively. Acceptance requires at least 3 in every dimension. Mechanisms returned for revision are reconstructed and submitted to the same checks and review again.

The accepted numerical mechanism is used for subsequent observation generation and evaluation. It is not regenerated by an LLM for each evaluated agent or repeated run.

We have clarified the definition of the paper-derived mechanism in Section 3.4. Appendix D.1 now specifies the mechanism review rubric, its scoring scale and acceptance criteria, and the revision and reassessment procedure. These additions and clarifications are marked in blue in the revised manuscript.

AuthorsOpenReview ↗

Q2: Human-review rubric for candidate papers and extracted claims

For candidate papers, human review assessed whether an empirical study offered a scientifically grounded account of how a phenomenon arises and sufficient mechanistic content to support further mechanism construction. The rubric covered four dimensions: (i) Scientific explanatory aim assesses whether the study explains how a phenomenon arises, through which processes, or under which conditions it changes; (ii) Role and process assesses whether the reported processes and conditions provide a meaningful basis for identifying the experimental semantic roles; (iii) Potential for mechanistic structure assesses evidence of interactions, feedback, mediation, nonlinearity, reversals, delays, history dependence, or phase-dependent behavior; (iv) Support in the source study assesses whether the judgment is supported by the article's text, figures, or reported findings. Each dimension was scored from 0 to 4: 0 indicated no relevant mechanistic basis; 1, fragmentary support; 2, sufficient evidence to warrant further mechanism assessment; 3, clear mechanistic and role-related evidence; and 4, multiple mutually supporting lines of evidence. A candidate had to score at least 2 on every dimension.

For LLM-assisted claim extraction, human reviewers checked each claim against its supporting passages in the original paper on three criteria: (i) source support and accurate location; (ii) preservation of applicability conditions, comparisons, qualifications, and scope; and (iii) faithful separation and grouping of claims without introducing unsupported interpretations. Each criterion was scored 0–2: 0 indicated an unsupported or meaning-changing error; 1, a correctable issue; and 2, satisfaction of the criterion. Retention required 2 on all three criteria. Correctable issues required revision and reassessment and an irreparable lack of source support required exclusion.

Both reviews involved a panel of ten reviewers with complementary scientific and methodological backgrounds, with each paper assessed independently by two reviewers matched to its domain and methods. Substantive disagreements were reconsidered before the final decision.

We have expanded Appendix C.2 with the human-review rubric for candidate papers and extracted claims. These additions are highlighted in blue in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.