- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces MechHypoBench, a 300-task benchmark (120 easy, 180 hard) designed to evaluate whether AI agents (Codex, Claude Code) and AI-scientist workflows (AI-Scientist-v2, AutoDiscovery) can generate open-form mechanistic hypotheses from tabular observational data. To construct tasks with known ground truth while preserving empirical data structures (grouping, longitudinal/time ordering, missingness, auxiliary covariates), the authors extract qualitative mechanistic claims from 244 papers across 14 fields, compile each into a fixed numerical DAG ($M^\ast $) built from multilinear interactions, logistic saturation, and exponential decay, and evaluate $M^\ast $ on percentile-ranked columns (Driver, Context, History, Phase) of 150 unrelated real-world datasets to synthesize five response channels (A–E). Agents observe a public subset of rows and propose open-form hypotheses. A frozen LLM interpreter (GPT-5.6 Terra) reads only the submitted hypothesis and predicts directional response changes (decrease, no_change, increase) across 12 withheld one-factor conditional comparisons, scored by Mechanism Consequence Agreement (MCA) and negative log-likelihood (MCNLL). Experiments show that the best agent (Codex + GPT-5.6 Sol) reaches 64.9% easy / 50.8% hard MCA (well below the qualitative and parameterized references at 87.6% / 82.9% and 96.8% / 97.5%), that routed GitHub scientific skills and AI-scientist workflows do not reliably improve MCA, and that supplying precomputed conditional contrasts from the public data (Evidence Summaries) improves MCA by +10.0 to +14.1 percentage points.
Critical strengths
- Consequence-based evaluation with falsification controls and a clean diagnostic ablation: Scoring open-form hypotheses via their implied directional predictions on withheld one-factor conditional comparisons—rather than surface text overlap—is a well-motivated evaluation design, and the
Empty,Shuffled,QMR, andPMRcontrols confirm that the interpreter distinguishes genuine conditional structure from vacuous or mismatched text. More importantly, the diagnostic interventions in Section 4.5 and Appendices E.5–E.6 (Evidence Summaries vs. Structured Records, plus retrospective hypothesis selection in Table 18) yield a clear, useful insight: agents primarily fail to surface conditional contrasts and interaction sign-reversals during exploratory data analysis (+10 to +14 pp when given precomputed conditional summaries), rather than merely failing to retain notes (+2 pp, n.s.).
Critical weaknesses
- Substantial mismatch between the "14 scientific fields / domain mechanistic discovery" framing and the anonymized cross-domain task: Inspection of Appendix D, Appendix F, and the supplementary tasks (
MechHypoBench.zip) shows that tasks graft a mechanism from one field onto an unrelated dataset from another field (e.g., a Toxoplasma gondii signaling mechanism grafted onto a rural child mental-health survey intask_fcc33a74fab2, or a molecular biology mechanism grafted onto an oceanography CTD dataset intask_02ddb9c16878) and strip all domain semantics (Factor Alpha,Factor Beta,Factor Gamma history,Observation Phase->Response Channels A–E, whereBACKGROUND.mdexplicitly states "All names are abstract study roles; external domain interpretations are neither defined nor required"). Consequently, the benchmark actually evaluates abstract nonlinear response-surface / conditional-sign recovery on grouped tabular data, not domain-grounded scientific mechanism discovery. This also makes the "scientific skills" experiment (Section 4.3) largely vacuous by construction, since domain-specific scientific skills from GitHub have no domain semantics to match against anonymized variables. - Confounding of non-termination with hypothesis quality in headline metrics, QMR sign-indeterminacy, and incomplete human-evaluation reporting:
- Across the 12 general-agent configurations, valid output completion rates range from 71.3% to 96.7% (Table 12), and missing/invalid runs are imputed with the
Emptycontrol (16.7% MCA). This conflates harness completion rate with scientific hypothesis quality in the main tables (for example,DeepSeek-V4-Flashjumps from 73.3%/76.7% valid in Codex to 92.5%/90.0% valid in Claude Code, accounting for much of the reported harness gap). - In the supplement (
ORACLE_HANDOFF.md), the Qualitative Mechanism Reference (QMR) is a symbolic structural-equation dump (component_b := b + w1*(a*decay) - w2*(a*b)), yet it achieves only 87.6% (easy) and 82.9% (hard) MCA because competing positive and negative interaction terms have algebraically indeterminate signs without exact numerical coefficients (PMR). - The human evaluation in Section 3.5 (Table 2) reports post-revision means of 4.54–4.97 across "ten reviewers with scientific backgrounds," but omits how many tasks were evaluated, how many raters scored each task, whether raters were independent of the authors, and inter-rater agreement.
- Across the 12 general-agent configurations, valid output completion rates range from 71.3% to 96.7% (Table 12), and missing/invalid runs are imputed with the
Questions for authors
- Conditional-on-valid performance (Clarification): What are the MCA and MCNLL of all 12 general-agent configurations in Table 3 / Table 12 when computed (a) conditional on valid outputs for each configuration and (b) on the common intersection of tasks where compared harnesses/skill settings both produced valid outputs?
- Relevance of scientific skills under variable anonymization (Clarification): Given that
TASK.mdandBACKGROUND.mdreplace all domain variables withFactor Alpha/Beta/GammaandResponse Channels A–E, what kinds of skills from the 4,043 GitHub skill pool were selected bySkillRouter-Reranker-0.6B, and how often did agents actually load or invoke them? - Human evaluation protocol (Clarification): In Section 3.5, did all 10 reviewers evaluate all 300 tasks, were the reviewers external to the author team, what were the pre-revision scores on the 17 reconstructed mechanisms, and what was the inter-annotator agreement?
9 comments
Q1: Performance on valid outputs
All entries separated by / are Easy/Hard. n is the number of included tasks out of 120/180, respectively. MCA is in %, higher is better; lower MCNLL is better. A. Each configuration's valid outputs.
| Configuration | Without skills: n | MCA | MCNLL | With skills: n | MCA | MCNLL |
|---|---|---|---|---|---|---|
| Codex / GPT-5.6 Sol | 116/173 | 66.2/50.6 | 0.899/1.360 | 113/174 | 67.9/52.0 | 0.859/1.298 |
| Codex / DeepSeek-V4-Flash | 88/138 | 50.4/35.6 | 1.624/2.126 | 91/143 | 47.6/33.4 | 1.765/2.247 |
| Codex / GLM-5.2 | 89/125 | 49.3/33.9 | 1.573/2.124 | 94/127 | 49.6/32.7 | 1.497/2.020 |
| Claude Code / Claude Sonnet 5 | 97/153 | 50.1/31.0 | 1.422/1.898 | 101/151 | 46.9/31.1 | 1.523/1.981 |
| Claude Code / DeepSeek-V4-Flash | 111/162 | 51.8/36.0 | 1.587/2.047 | 114/165 | 46.8/33.2 | 1.712/2.204 |
| Claude Code / GLM-5.2 | 93/129 | 54.1/33.5 | 1.380/2.075 | 95/145 | 48.9/32.0 | 1.600/2.145 |
B. Common-valid tasks: harness comparisons.
| Backend | Skills | n | Codex MCA | Claude Code MCA | Codex MCNLL | Claude Code MCNLL |
|---|---|---|---|---|---|---|
| DeepSeek-V4-Flash | off | 83/125 | 51.1/35.6 | 49.9/35.1 | 1.596/2.133 | 1.612/2.085 |
| DeepSeek-V4-Flash | on | 89/130 | 48.0/33.6 | 47.7/33.7 | 1.774/2.229 | 1.708/2.211 |
| GLM-5.2 | off | 69/94 | 48.4/34.8 | 54.8/32.9 | 1.546/2.061 | 1.353/2.109 |
| GLM-5.2 | on | 77/108 | 49.8/32.4 | 47.9/31.6 | 1.493/2.000 | 1.627/2.172 |
C. Common-valid tasks: skills off/on.
| Configuration | n | MCA off | MCA on | MCNLL off | MCNLL on |
|---|---|---|---|---|---|
| Codex / GPT-5.6 Sol | 109/167 | 66.7/51.4 | 67.5/52.4 | 0.890/1.343 | 0.867/1.290 |
| Codex / DeepSeek-V4-Flash | 69/114 | 54.0/34.7 | 50.5/33.8 | 1.516/2.154 | 1.654/2.242 |
| Codex / GLM-5.2 | 72/92 | 48.7/33.6 | 49.8/33.8 | 1.548/2.106 | 1.487/1.952 |
| Claude Code / Claude Sonnet 5 | 88/139 | 48.6/30.7 | 46.0/30.5 | 1.459/1.905 | 1.541/2.006 |
| Claude Code / DeepSeek-V4-Flash | 105/149 | 52.6/35.9 | 46.8/34.1 | 1.568/2.047 | 1.711/2.149 |
| Claude Code / GLM-5.2 | 75/107 | 54.9/33.6 | 49.6/33.6 | 1.319/2.081 | 1.575/2.093 |
The benchmark evaluates whether an agent can turn observations into a mechanistic hypothesis with the available resources. For general agents, the main table uses the same 300 tasks and 30-minute terminal budget. Shared-backend harness comparisons also match inputs, backend, and skill setting. Analysis, tool use, and valid hypothesis delivery are components of the system being evaluated. Differences in successful delivery under this common budget are therefore part of harness performance. Retaining the full task set gives all configurations a consistent evaluation population, using each task's Empty control when no valid submission is delivered. Making common-valid results the primary table would instead select the evaluation population by the systems' completion outcomes and exclude tasks completed by only one system. Beyond the main-table evaluation, our analyses use paired comparisons under matched conditions, with task sets and relevant experimental settings aligned for each contrast.
With DeepSeek-V4-Flash, Claude Code delivers more valid hypotheses under both skill settings within the same budget (A), a harness-level difference. The skill finding also persists on common-valid tasks (C): skill access does not consistently improve consequence agreement.
Q2: Scientific skills under variable anonymization
Transferability is an inclusion criterion in skill collection. The pool was curated for scientific data analysis and hypothesis generation that can transfer across research fields. After an initial research-relevance filter, the LLM reviewed candidate names, descriptions, and available content using four criteria: (i) Scientific research goal: Whether the skill directly supports a research question, scientific data, or scientific conclusions. (ii) Scientific reasoning or data-analysis contribution: Whether it provides substantive analysis, reasoning, evidence integration, or hypothesis-generation operations rather than merely restating or presenting content. (iii) Consistency of purpose and content: Whether the advertised research function is supported by identifiable operations and corresponding inputs and outputs. (iv) Cross-disciplinary applicability: Whether its core operations can transfer across research fields rather than depending entirely on a single field's terminology or procedures. Each criterion was scored 0–4. Retention required at least 2 on every criterion and classification as Hypothesis Generation or Data Analysis.
What was selected and why it applies. SkillRouter-Reranker-0.6B selected 32 skills per task using the task description and public dataset overview, structure, and format. Selected skills were exposed as descriptions, with content available on demand. Actual routed and accessed examples include:
- data-science-statistical-analysis: data exploration, missingness checks, regression diagnostics, and mixed-effects methods for non-independent observations.
- hypothesis-generation: competing mechanistic explanations, falsifiable predictions, and specifying the conditions under which predictions should hold. These operations can act on the available condition/response observations and data structure without requiring the original field-specific variable names.
We report confirmed accesses in the recorded runs. Access requires content from a selected skill file or skill instructions delivered by a load invocation. Repeated records of the same call count once. Directory listings are excluded. Each configuration has 300 evaluated tasks.
| Configuration | Access calls | Tasks with access | Rate |
|---|---|---|---|
| Codex / GPT-5.6 Sol | 564 | 279/300 | 93.0% |
| Codex / DeepSeek-V4-Flash | 52 | 52/300 | 17.3% |
| Codex / GLM-5.2 | 163 | 113/300 | 37.7% |
| Claude Code / Claude Sonnet 5 | 262 | 80/300 | 26.7% |
| Claude Code / DeepSeek-V4-Flash | 47 | 47/300 | 15.7% |
| Claude Code / GLM-5.2 | 129 | 107/300 | 35.7% |
Why a skill-access analysis is appropriate. For each task, all skill-enabled configurations share access to the same 32 skills selected by SkillRouter-Reranker-0.6B. The offered guidance is therefore held fixed across harnesses and models for that task. Each harness/model autonomously decides which skill information to read, whether to load or invoke a skill, and how to use its guidance in the analysis. Within each harness/backend, the with-/without-skills comparison holds task inputs and the execution budget fixed while changing access to this resource. This is an appropriate way to test whether agents benefit from scientific guidance, because selecting, retrieving, and integrating useful information are parts of the capability being evaluated.
We have added the skill-access statistics and clarified the rationale for evaluating skill access in the appendix section E.3. These additions are highlighted in red in the revised manuscript.
We sincerely thank you for your careful reading and constructive feedback. We especially appreciate your recognition of our consequence-based evaluation as “a well-motivated evaluation design” and of our diagnostic interventions as yielding “a clear, useful insight” into the roles of evidence extraction and retention in mechanistic hypothesis generation. We address your questions and concerns point by point below.