acceptodds
Under review as a conference paper at ICLR 2027

Mechanisms Before Methods: Measuring How AI Designs AI Research

Abstract

Can an AI system do more than propose a plausible research idea—can it choose a principle worth pursuing and turn it into a workable method? Existing evaluations either score open-ended proposals, where reliable negative examples are hard to obtain, or end-to-end execution, where scientific judgment is mixed with engineering. We introduce Mechanisms Before Methods (MBM), a benchmark that scores research design before any experiment is run, built from 789 papers. MBM represents each method by its mechanism: the operating principle that explains how it works without revealing its paper-specific implementation. Each paper supports two paired tasks. Mechanism Allocation chooses the documented route among plausible alternatives; Mechanism Realization works out how that route should operate on the target problem. Across ten systems the two abilities rise together (r = 0.857), while within a single paper a correct choice adds under one point of realization quality. Asking a model to draft its own realization before choosing lowers its accuracy, while prewritten realizations raise it by 4.70 points in a matched diagnostic: a plausible plan is not evidence for a choice. Reinforcement learning improves Realization F1 by 3.37 points, and the ability transfers: a short Realization phase improves later Allocation itself. MBM makes pre-experimental research decisions separately measurable, and suggests that learning how a principle works helps decide whether it is worth pursuing.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI research reasoning, mechanism abstraction, benchmark, route selection, mechanism realization, paper-grounded evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_viVoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Mechanisms Before Methods (MBM), a benchmark that evaluates research design before any experiment is run. It is built from 789 published papers. Each method is abstracted into a mechanism card: an operating principle such as optimal transport, MCTS, MPC, ADMM or conformal prediction. A card names the object the mechanism acts on, the transformation, the guiding signal and the applicability conditions, without revealing the paper-specific implementation. The library has 243 reviewed cards.

Each paper comprises of two tasks:

  • Mechanism Allocation: given an anonymized problem statement, rank four cards. One is the source-documented route; the other three are plausible alternatives drawn from mechanisms that other papers pursued. The metric is Top-1 accuracy.
  • Mechanism Realization: given the problem and the documented mechanism, propose the high-level components that bind the principle to the target. An LLM judge (GPT-5.6 Sol) scores the answer against source-derived requirement atoms (10,270 in total, 4,415 marked core) using route precision, core recall and F1.

The authors also claim many insightful findings:

  1. Across ten frontier systems, Allocation Top-1 ranges from 49% to 69%, and the two abilities correlate across systems (r = 0.857). Within a paper, a correct Allocation adds only 0.93 points of Realization F1.
  2. Prewritten realizations attached to all four cards raise Allocation by 4.70 points (10 of 11 systems improve; sign test p = 0.012). Letting a model draft its own realizations lowers accuracy by 5.17 points (one Qwen3.5 model).
  3. Realization-only RL raises F1 by 3.37 points, and a short Realization phase before Allocation training raises Allocation from 44.00% to 45.67% (single run).
Critical strengths
  1. A well-designed intermediate unit and source of negatives for evaluating research design. The mechanism abstraction sits between a topic label (too broad) and a full method (gives away the answer). This makes research routes comparable across papers and lets choice and specification be scored separately on the same paper. Drawing distractors from mechanisms that other papers actually pursued, rather than from generator-invented bad ideas, directly addresses a known weakness of ideation benchmarks: generated negatives leave stylistic artifacts. The paper adds sensible controls: exact position balancing, position/length/TF-IDF baselines near chance (24–27%), no retrieval at test time, and a design in which Realization receives the gold mechanism so that Allocation errors do not propagate. Together, these make MBM a credible, scalable alternative to both proposal rating and end-to-end reproduction.

  2. Controlled diagnostics that go beyond a leaderboard. The paired design supports informative interventions. Prewritten realizations on all four cards improve choice (+4.70 points, 10 of 11 systems). A model's own drafts hurt choice (−5.17 points), and a shuffled-notes arm that holds content and length fixed shows that the drafts are used, not ignored. A panel analysis with system and paper effects separates cross-system correlation (r = 0.857) from the small within-paper link (0.93 points, 95% CI [0.25, 1.62]). These findings are relevant to how research agents should be built, for example whether "write the plan first" helps decisions. They show the benchmark can answer questions a single score cannot.

Critical weaknesses
  1. The validity of benchmark items and labels is under-evidenced. Allocation scores agreement with the documented route among four "plausible" alternatives, and the paper itself acknowledges that other routes may be viable (l. 96–100). The benchmark is only meaningful if, for most items, the problem statement makes the documented route identifiable on principled grounds and the distractors are not equally valid. The main text gives little evidence that this holds:

    • Mostly unverified, model-authored content. Problem statements, cards, distractor screening and references are all model-authored (AI Use Statement). Only "a random 10% of records" of the realization references is confirmed by hand (l. 249–250), and it is unclear whether problem statements or distractor plausibility were human-checked at all.
    • Possible answer leakage. Problem statements are drafted from the source paper by a model that knows the answer, so they may carry wording that leaks the mechanism. Even the Fig. 2 example ("use local and distant graph information with practical computation, while adapting the learned filter") arguably points toward Krylov/spectral methods. A TF-IDF baseline does not rule out such semantic cues.
    • No human reference point. There is no human-expert performance or inter-annotator agreement on Allocation items, so 69% Top-1 cannot be interpreted. It is unclear how far it is from the ceiling set by item ambiguity. The persistent 15–21-point Top-1/Top-2 gap that the authors read as "cannot settle the last comparison" (l. 348–350) is equally consistent with a second distractor being a valid answer, i.e., label noise.
    • Contamination is argued, not measured. The argument rests on the abstract card vocabulary and on 226 papers appearing after "several" systems' cutoffs. The main text never reports accuracy split by publication date relative to each system's cutoff [check appendix], which is the direct test.

    Because MBM's main contribution is the benchmark, establishing that its labels measure research judgment rather than literature recall or item artifacts is essential.

  2. Several headline conclusions rest on thin or non-reusable evidence.

    • Transfer result. The abstract states that "the ability transfers: a short Realization phase improves later Allocation itself." This is a +1.67-point gain, i.e., five items out of 300, from a single run on one model family. The authors concede in l. 454–457 and l. 473–474 that this "marks a direction and not an effect size," but the abstract, the Sec. 7 title and the conclusion state it as a finding.
    • Self-persuasion result. The self-draft result (−5.17 points) is shown for one model only.
    • Realization scoring. Realization uses a single LLM judge that is itself an evaluated system (ranked third on the metric it scores). The ten answers are judged jointly, so scores are cohort-dependent and "would have to be recomputed to admit a new system" (l. 302–303). This undermines MBM's use as a reusable benchmark. The authors also acknowledge lower repeat agreement on which requirements count as core. With a total Realization F1 spread of only ~8 points across systems, the leaderboard's resolution is unclear without a human-judge agreement study.
Questions for authors
  1. Corpus construction. How were the 789 papers collected (venues, search procedure, time range) and selected? How were papers handled whose method does not map onto a single card, either because it introduces a new mechanism or combines several? Does "226 of those papers appeared in 2025 or later" (l. 319–320) refer to the 300-paper evaluation split or to the full corpus?
  2. Distractor construction and difficulty. Who or what screened distractors for plausibility, how is "nearby" defined, and how large was the candidate pool? Please report per-distractor selection rates and the effective number of competitive options per item. Ideally, also report a human annotation of each distractor as (a) clearly inapplicable, (b) plausible but not the documented route, or (c) equally viable. This would quantify both item difficulty and label noise.
  3. Human validation and human baseline. Beyond the 10% check of realization references, were problem statements and distractor sets checked by humans, by whom (domain experts?), and with what agreement? What is human-expert Top-1 accuracy on a sample of Allocation items?
  4. Agreement with the documented route vs. design quality. Is there any evidence that matching the documented route reflects better research design? For example, for items where strong systems pick a non-documented route, do experts judge that route viable or inferior? Could Allocation be scored with multiple acceptable routes where experts agree they are valid?
  5. Leakage from problem statements. How were problem statements prevented from encoding the mechanism, given that they were written by a model that saw the source method? Could you report a "problem-only" probe, such as a model fine-tuned on the training split to predict the card from the problem statement, or the lexical/semantic similarity between the problem and the correct card vs. the distractors?
  6. Contamination. Please report Allocation accuracy per system, split by paper date relative to that system's training cutoff [check appendix].
  7. Prewritten-realization diagnostic (Sec. 6.1). Which two models drafted the realizations, and are they among the evaluated systems? The problem statement and the correct card both derive from the same paper, so the correct card's realization may fit the problem more naturally. Could the +4.70-point gain reflect that leakage rather than "discriminative" information? A control with realizations written without access to the problem statement would help.
  8. Judge reliability and reusability. What is the agreement between GPT-5.6 Sol and human annotators, and between Sol and a different judge family, on precision and core-recall credits? Can Realization be scored per answer (not jointly), so that new systems can be added without rescoring the whole cohort?
  9. Statistical support for Sec. 7. Can you provide multiple seeds and a budget-matched comparator (e.g., the same number of extra Allocation-training steps instead of the Realization phase) for the 44.00% → 45.67% result?
2 comments
AuthorsOpenReview ↗

Reply to viVo (1/2): Q1–Q5 and W1

Thank you for the detailed questions about how we constructed and validated the benchmark.

Q1 (corpus construction). We expanded an existing abstract collection using the arXiv Atom API. From January 2012 to August 20, 2026, we sampled one rotating quarter per year in each of 12 categories (cs.AI, cs.LG, cs.CL, cs.CV, cs.NE, cs.RO, cs.MA, cs.IR, cs.HC, cs.SI, cs.SE, stat.ML), with no venue restrictions. Of 154,267 distinct abstracts, we selected those in which one sentence both named a mechanism and indicated that the authors used it. This yielded 2,646 paper–mechanism pairs across 2,599 papers. At the full-text stage, we looked for evidence that the authors had implemented the mechanism and excluded papers whose approach could not be represented by one card. All 789 papers in the final benchmark came from this candidate pool.

Q1 (the year count in the submission). The 226 papers dated 2025 or later were counted across the full corpus. The submission incorrectly attributed that count to the evaluation split. The original Semantic Scholar data lacked years for 54 of the 789 papers. We filled these gaps using the year of the first arXiv submission. The corpus spans 2012–2026. With complete coverage, 249 of 789 papers are dated 2025 or later, including 104 of the 300 evaluation papers.

Q2 (distractors). We ranked cards in the 243-card library by TF–IDF cosine similarity to the problem statement and its constraints, excluding the documented route. A model screened the top 23 cards, followed by human review. We kept three plausible alternatives that met the constraints and differed from the documented route and from one another. Evaluated-system answers and scores played no role in selection. In the 300-paper evaluation split, each of A–D is correct 75 times. On the full evaluation split, keeping the documented card and replacing the three screened alternatives with random cards raises Top-1 by 35–39 points, to 83–89%, for GPT-5.6 Sol, Claude Sonnet 5 and GPT-5.6 Luna, so most of the difficulty comes from the screened alternatives.

Q3 (human validation and a human baseline). Human reviewers checked a sample of problem statements (Q5) and took part in distractor selection (Q2). In a separate test on 50 items, three domain experts, seeing what the systems see, chose the documented route in 130 of 150 answers (86.7%). Pairs of experts chose the same route in 83.3% of comparisons (125 of 150).

Q4 (agreement with the documented route vs. design quality). In the expert test in General Response 1/2, given only the problem and the cards, experts chose the documented route in 86.7% of answers, slightly more often than two experts agreed with each other (83.3%). We will next ask experts to assess the alternatives that strong systems pick; their judgments would identify items where more than one route deserves credit.

Q5 (leakage from problem statements). Our drafting instructions require the statement to describe the problem and its constraints and to omit the method, the result and identifying details. In a human audit of 78 items drawn from the full benchmark, two statements contained a cue to the paper’s identity (2.6%). We also revised the wording about learned filters in the Figure 2 illustration because it suggested spectral methods.

For the similarity test you suggest, we ranked the four cards by embedding similarity to the problem statement (all-MiniLM-L6-v2, cosine); the documented card ranks first on 92 of 300 items (30.7%, against 25.0% by chance). The ten systems reach 42.0% to 58.0% on the same items, so wording similarity accounts for little of their accuracy.

W1, Top-1/Top-2 gap. We agree the gap cannot by itself tell a hard last comparison from a second valid answer, and the revision drops the sentence that read it as the former.

AuthorsOpenReview ↗

Reply to viVo (2/2): Q6–Q9 and W2

Q6 (contamination). The ten systems have closely spaced training cutoffs. We compared their Mechanism Allocation accuracy on all 789 papers using the same two groups for each system. Of these papers, 674 are dated 2025 or earlier and 115 are dated 2026. Mean Top-1 across systems falls from 61.10% to 55.91% (−5.18 points; differences are computed before rounding). Nine systems score lower on the 2026 papers; GLM 5.3 is essentially unchanged (+0.04 points). Accuracy on the 2026 papers ranges from 45.2% to 60.9%, above the 25% chance baseline for every system. Exposure to older papers and differences in difficulty could both contribute to the gap, and year groups cannot show which papers a system saw in training. The construction appendix reports the year sources and the results for each system.

System≤2025 (674)2026 (115)Δ (pp)
GPT-6 Astra70.62%60.87%-9.75
Claude Opus 567.80%59.13%-8.67
GPT-5.6 Sol63.35%57.39%-5.96
Kimi K362.46%60.87%-1.59
GLM 5.360.83%60.87%+0.04
GPT-5.6 Terra61.13%53.91%-7.21
Claude Sonnet 560.98%53.91%-7.07
GPT-5.6 Luna57.57%51.30%-6.26
HY4 Preview56.08%55.65%-0.43
DeepSeek V4 Pro50.15%45.22%-4.93
Ten-system mean61.10%55.91%-5.18

Q7 (prewritten realizations). GPT-6 Astra and Claude Opus 5, both evaluated systems, drafted and cross-revised the realizations, working from each candidate card alone. The drafts already meet the control you suggest: none was written with the problem statement or the source paper in view, so the drafts carry no information about the problem beyond what the cards already show the selector.

Q8 (judge reliability and reuse). A second judge from another family, Gemini 3.5 Flash, rescored 100 papers with all ten answers in one batch, as in the submission. Its F1 ranking of the ten systems closely matches that of the original judge, GPT-5.6 Sol (Spearman 0.903), and the two judges agree on which system has higher precision in 87.5% of the pairs that differ by more than two points (General Response 2/2). A new system can be added without rescoring the cohort. Scored alone against the recorded judgments, HY4-Flash gets a mean F1 of 59.60, against 59.55 when all eleven are rescored. A human audit confirmed 69 of the 78 sampled core labels in the scoring reference (88.5%).

Q9 (statistical support for Section 7). We ran three seeds per schedule and matched the total number of optimizer updates. For both models, starting with 40 or 80 Realization updates gave slightly higher mean Allocation accuracy than Allocation-only training. General Response 2/2 reports the four comparisons.

W2 (self-drafted realizations). We also tested GPT-5.6 Sol on the same items as Qwen. Drafting a realization for each candidate before choosing lowered its Allocation Top-1 from 52.33% to 51.00%.

Reviewer_RhE6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper is about MBM, a benchmark that tests whether AI can make research design choices before any experiment runs.

To test this, the authors let the AI gets a research problem, then it sees 4 short idea cards where each card names one general idea. The AI is then expected to pick the idea the real paper used, which the authors call 'allocation' Next, the AI is given the correct idea and it must explain how that idea would work for this problem by writing a short essay which is later judged by a judge AI comparing the essay against the real paper. The authors call this step 'realization'

The data include 789 papers and 243 mechanism cards and a 300 papers evaluation split

The insights are that:

  1. strong AI do better at both tasks
  2. picking right does not mean explaining well
  3. picking benefits from a 'how it would work' hint
  4. if the hint is written by the AI itself, picking becomes worse
  5. training an AI on task 2 helped task 1 a little

There were 10 LLMs tested. The models get only the question text. No web search or tools. The LLMs are also responsible for writing the problems and the idea cards and the reference answers. One model was trained with RL on the explain task.

Critical strengths

The idea is intuitive. The amount of papers curated for testing is impressive. The 'LLM's self-written notes hurt while the ones written by other LLMs help' result is interesting. The limit section is honest by admitting the scope is limited in terms of model family, the number of runs and the reliability of the LLM judge.

Critical weaknesses
  1. There are similar works like 'AInstein: Can LLMs Solve Research Problems From Parametric Memory Alone' by Mishra et al., 2025. And the difference between this work and the similar works is not significant enough, or is at least not discussed
  2. AI drafted the reference answers and only 10% were manually checked
  3. There are several terms uncommon in the field of LLM evaluations like 'route', 'atoms', 'binding', 'auditable positive', 'common queue', etc. These make reading the paper difficult.
  4. the finding 'training on Explain improved Pick' comes from training only one model with RL.
Questions for authors
  1. what's the underlying difference that distinguishes this work from similar existing works?
  2. Given the limited reliability of LLM-as-judge, how would you make your evaluation more convincing? Is it possible to provide more proofs like for example the judge agreement rate with human raters?
1 comment
AuthorsOpenReview ↗

Reply to RhE6

Thank you. The revision removes the internal labels you list and defines its terms at first use (W3 below).

W1 / Q1 (the underlying difference from AInstein). MBM scores the choice of a mechanism separately from how it is realized, and the two come apart on individual papers. AInstein asks a model to write a solution and judges it as a whole, including whether it matches the published approach. With no alternatives to choose among, a miss there can mean a wrong principle, a valid different one, or the right one left underspecified. MBM separates the first from the third: Allocation forces a choice among four mechanisms that real papers pursued, and Realization supplies the documented mechanism and scores how fully the design is specified against requirements from the source paper. Scored this way, choosing the documented route goes with only 0.93 more points of Realization F1 on the same paper, after accounting for differences between systems and papers (General Response 1/2).

W2 (model-drafted references with a 10% check). In the 78-paper human audit, reviewers flagged two problem statements for clues to the source paper and three documented cards for giving too much detail. For the scoring reference, reviewers took one core requirement per paper at random, hid its label and confirmed 69 of the 78 as necessary (88.5%).

W3 (uncommon terms). We revised the terminology in the paper and supplement. In the introduction we now define the documented route as the mechanism the source paper’s authors used. We replaced “atom” with “requirement” in the scoring description. Where the appendix used “binding,” it now describes how the mechanism applies to a particular problem. We also removed “auditable positive” and “common queue.” The changes are marked in blue.

W4 (Realization training on one model only). We added Llama-3.1-8B-Instruct as a second family. In the submission, Realization-only RL raises Realization F1 by 3.37 points for Qwen. In both families, mean Allocation accuracy was slightly higher after 40 or 80 updates of Realization training followed by 390 Allocation updates than after the same total number of updates spent entirely on Allocation. General Response 2/2 gives the differences for both settings.

Q2 (making the judge evidence more convincing). A second judge from another family, Gemini 3.5 Flash, rescored 100 papers with all ten answers in one batch, as in the submission. Its F1 ranking of the ten systems closely matches that of the original judge, GPT-5.6 Sol (Spearman 0.903), and the two judges agree on which system has higher precision in 87.5% of the pairs that differ by more than two points (General Response 2/2). The reference both judges score against was itself checked by hand (W2).

Reviewer_ckeZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Mechanisms Before Methods (MBM), an execution-free benchmark built from 789 published papers (with a 300-paper evaluation split) designed to evaluate pre-experimental research design decisions in AI systems. The authors abstract each paper's method into a reusable "mechanism card" (243 reviewed cards in total) that captures the operating principle (target object, transformation, guiding signal, and applicability conditions) while stripping away paper-specific implementation details. Each paper is paired into two decoupled tasks:

  1. Mechanism Allocation: ranking four candidate mechanism cards (one documented route from the source paper and three plausible distractors used in other papers) given the problem statement.
  2. Mechanism Realization: given the problem statement and the ground-truth documented mechanism card, generating the essential target-specific design commitments required before running experiments (scored against 10,270 source-indexed requirement atoms, including 4,415 core items, using GPT-5.6 Sol as an LLM judge).

Evaluating ten frontier LLMs, the authors report three main findings: (1) Allocation and Realization correlate strongly across systems (r=0.857r = 0.857) but dissociate within individual papers (+0.93 F1 points when Allocation is correct); (2) providing prewritten high-quality realizations improves Allocation by +4.70 percentage points on average, whereas asking a model (Qwen3.5-35B-A3B) to draft its own realizations before choosing lowers Allocation accuracy by 5.17 points ("self-persuasion"); and (3) reinforcement learning on Realization improves Realization F1 (+3.37 points) and yields a small transfer gain (+1.67 points, or 5 items on 300 papers, in a single run) to Allocation without test-time realization drafts.

Critical strengths
  1. Clean Task Decoupling and Distractor Construction for Pre-Experimental Design: Separating route selection (Mechanism Allocation) from target-specific specification (Mechanism Realization)—and feeding the ground-truth mechanism into the Realization stage so upstream selection errors do not propagate—is a clean diagnostic design. Drawing negative candidate cards from real mechanisms pursued in other papers (rather than synthetically generating bad ideas) effectively mitigates stylistic shortcut artifacts common in multiple-choice ideation benchmarks.

  2. Interesting Diagnostic Contrast Between External vs. Self-Generated Realizations (§6.1): The controlled comparison showing that externally supplied high-quality realizations improve route selection (+4.70 pp across 10 of 11 models), whereas self-generated realization drafts degrade selection accuracy by 5.17 pp, offers a useful takeaway: fluent, self-generated implementation plans can act as self-persuasion noise rather than discriminative evidence when models evaluate competing research directions.

Critical weaknesses
  1. Limited Conceptual Differentiation from Prior Paper-Decomposition Benchmarks and Questionable Ground-Truth Justification: Decomposing published papers into problem contexts, intermediate inspiration/mechanism units, and concrete method specifications to test retrospective rediscovery is already well-explored in prior benchmarks such as Scideator (Radensky et al., 2024), MOOSE-Chem (Yang et al., 2025), ResearchBench (Liu et al., 2026), and Matter to Mechanism (Sourav et al., 2026). While MBM standardizes a cross-domain card vocabulary and pairs selection with specification, the core task remains retrospective reverse-engineering of a single paper's historical choice. More fundamentally, as the authors acknowledge, the source-documented route mi∗m_i^* is not uniquely optimal, and the three distractor mechanisms are explicitly screened to be plausible under the problem's constraints. Consequently, penalizing a model in Allocation for ranking another plausible mechanism first—without execution—merely measures conformity to one author group's historical bet rather than true scientific judgment. In Realization, even when mi∗m_i^* is fixed, grading solely against the exact design choices of the source paper risks penalizing valid alternative instantiations of the same mechanism. Furthermore, the related work omits relevant execution-grounded research benchmarks such as EXP-Bench, which directly targets paper-grounded experimental design and execution.

  2. Fragile Empirical Support for Key Claims (Single-Run RL Transfer, Model-Generated Benchmark with 10% Human Audit, and Cohort-Coupled LLM Judging):

    • Under-substantiated RL Transfer Claim (§7): A central headline contribution of the paper is that internalizing Realization via RL transfers to improve Allocation. However, this claim rests on a single run of a single model family where Allocation improves from 44.00% to 45.67% on the 300-paper split—an difference of only 5 papers (+1.67 pp), without multi-seed variance, significance testing, or a compute-matched Allocation-only RL baseline.
    • LLM-Generated Benchmark & Missing Human Baseline: The problem statements, mechanism cards, distractor screening, and 10,270 realization requirement atoms are all LLM-drafted, with only a 10% random sample confirmed by hand (and no reported human error rate, inter-annotator agreement, or human expert performance baseline). The authors also note in §8 that judge repeat agreement is lower on which requirements count as "core."
    • Cohort-Coupled Judging by a Participating Model: Realization is scored by GPT-5.6 Sol—which is itself an evaluated model (placing 3rd, while its sibling GPT-6 Astra places 1st/2nd)—by bundling all ten models' answers into a single prompt packet. This makes every model's score dependent on the evaluated cohort (requiring full rescoring whenever a new model is added) and leaves the benchmark vulnerable to within-family stylistic preference bias.
Questions for authors
  1. Clarification on Differentiation and Validity Under Non-Unique Ground Truth: Beyond enforcing a shared card vocabulary across domains, how does MBM fundamentally advance what we learn about AI research capabilities compared to ResearchBench, MOOSE-Chem, and Matter to Mechanism? Specifically, when a model selects a plausible distractor card in Allocation or proposes a valid alternative instantiation of mi∗m_i^* in Realization that diverges from the source paper's specific pipeline, how can an execution-free rubric distinguish a genuinely viable alternative research design from an error? Please also discuss how MBM relates to EXP-Bench in Section 3.

  2. Additional Evidence for the RL Transfer Claim (§7): Can you provide multi-seed results (e.g., ≥3\ge 3 random seeds with mean and standard deviation) and a strictly compute-/token-matched Allocation-only RL baseline to verify whether the +1.67 pp (5-item) gain from Realization-to-Allocation transfer is statistically significant rather than run-to-run noise or a byproduct of additional training steps?

  3. Benchmark Audit Metrics, Human Baseline, and Independent Judge Validation:

    • What were the exact defect rates (e.g., identity leakage in problem statements, overly specific cards, or non-essential details marked as core atoms) and inter-annotator agreement found during the 10% manual audit? Can you report a human expert baseline on a subset of 30–50 items?
    • Does the Realization leaderboard ranking hold if answers are scored independently (one model answer per prompt against the reference atoms, rather than bundling 10 models together) and evaluated by a non-GPT judge (e.g., Claude Opus 5 or an open-weight model)?
  4. Suggestion for Future Work — Mitigating Self-Persuasion via Falsification Prompting: In §6.1, self-generated realizations hurt Allocation because models write fluent, positive plans for all four candidates. Have you tested whether prompting the model to write critical falsification notes (i.e., identifying which problem constraint each mechanism fails to satisfy) rather than constructive realization plans mitigates this self-persuasion drop?

1 comment
AuthorsOpenReview ↗

Reply to ckeZ

Thank you for the precise questions. We answer them in your order; the shared evidence is in the two general comments.

W1 / Q1 (how MBM differs from the work you cite). We agree that decomposing papers is not new; the difference is what gets scored. MOOSE-Chem and ResearchBench treat an inspiration as an input, retrieved and composed with a research background into a hypothesis. Scideator and Matter to Mechanism start from existing papers or proposals, recombining their facets or annotating how an intervention addresses a failure mode. EXP-Bench supplies the method and starter code and scores the experiment an agent runs. In MBM the principle is the output: the model chooses it among four mechanisms used in real papers, and the choice is checked against the one the authors pursued for this problem. Realization then supplies that mechanism, so a wrong choice does not lower the design score.

Scored separately, choice and design come apart. On a given paper, choosing the documented route goes with only 0.93 more points of Realization F1 (General Response 1/2), so how well a model develops a principle says little about whether it would have picked it. Systems also trail people on the choice. Experts choose the documented route in 86.7% of answers, against 49.4–69.2% for the ten systems (W2 / Q3a below; General Response 1/2), and route frequency does not explain system accuracy (General Response 2/2). Because every alternative is a real mechanism that can be realized, MBM can also measure the self-persuasion you describe in Q4. Drafting a realization for every candidate before choosing lowers Qwen's Allocation accuracy by 5.17 points (General Response 2/2).

W1 / Q1 (valid alternatives without execution). Allocation scores agreement with the documented route; it does not treat the alternatives as unworkable. The expert baseline in General Response 1/2 shows that researchers given the same problem and cards mostly choose that route, so it is not an arbitrary standard. In Realization, the supplied principle fixes the scope, and the model decides how to make it work under the problem's constraints. Core recall counts only the operations the mechanism cannot work without on this problem, not every choice the paper happened to make. It accepts paraphrases and established equivalents, so a different instantiation earns core credit wherever it meets a requirement, and designs beyond those requirements would need separate evaluation.

W2 / Q2 (the transfer claim). The original Qwen difference was +1.67 points (five items, one run). Extra Allocation-only updates do not help. From 390 to 470 updates, mean Top-1 stays within 44.1–44.9% for Qwen and 37.2–37.4% for Llama, over three seeds each. With 40 or 80 Realization updates first, both families end slightly higher than the matched Allocation-only runs, which have the same total number of optimizer updates, though not necessarily the same tokens or FLOPs. General Response 2/2 gives the results for each comparison.

W2 / Q3a (audit metrics and a human baseline). Across 78 papers, the human audit flagged two problem statements for identifying cues (2.6%), three documented cards for excessive detail (3.8%) and nine sampled core requirements as replaceable or nonessential (11.5%). In a separate test on 50 Allocation items, three experts chose the documented card in 130 of 150 answers (86.7%). Pairs of experts chose the same route in 83.3% of comparisons (125 of 150).

W2 / Q3b (choice of judge and scoring format). A non-GPT judge, Gemini 3.5 Flash, rescored 100 papers; its F1 ranking of the ten systems closely matches the original judge's (Spearman 0.903), with the same top two systems (General Response 2/2).

Q4 (falsification prompting). Our diagnostic used constructive plans only. Asking the model for the strongest reason against each candidate before choosing is the natural next control, and the revision lists it as future work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response 1/2: what MBM measures and why we use the documented route

Thank you for three careful reviews. You agree on what MBM gets right: the card “sits between a topic label (too broad) and a full method (gives away the answer)” (viVo); the alternatives come “from real mechanisms pursued in other papers (rather than synthetically generating bad ideas)” (ckeZ); and because Realization receives the documented mechanism, “upstream selection errors do not propagate” (ckeZ). You also agree on what the submission had not yet shown: that agreement with the documented route means something, that the Realization scores can be trusted, and that Section 7 supports transfer. We report new evidence on all three questions in these two general comments. Each reviewer reply answers the remaining points.

What MBM adds. Decomposing papers into problems, inspirations and methods is not new (ckeZ). What MBM adds is a score for the decision that comes before a method: given a problem, which of four mechanisms, each used in a real paper, to pursue. Realization then supplies the documented mechanism and asks how to make it work there. A model that knows the answer is optimal transport still has to say which distributions to align, what cost to use and how to use the alignment, and each realization is scored against requirements drawn from its source paper.

What the separate scores show. Choosing and realizing rise together across systems (r = 0.857), yet once system and paper effects are removed, choosing the documented route goes with only 0.93 more points of Realization F1 on the same paper (95% CI [0.25, 1.62]). A model that can work out a mechanism would not necessarily have chosen it, and a single score on the finished proposal cannot tell the two apart. Models also fall well short of people on the choice itself. Given the same problem and four cards, three domain experts choose the documented mechanism in 86.7% of answers, while the ten systems reach 49.4–69.2% across all 789 papers (Table 2). Neither route frequency nor wording similarity accounts for the systems' accuracy (General Response 2/2; reply to viVo, Q5).

What does agreement with the source paper tell us? ckeZ and viVo ask why we score a choice against the mechanism the authors pursued. All four candidates are mechanisms used in real papers, and the three alternatives are screened for plausibility under the problem's constraints. What sets the documented one apart is that the source paper records how it was made to work on this particular problem. The experts each answered the same 50 Allocation items, seeing only the problem statement and the four cards; 130 of their 150 answers chose the documented mechanism. Pairs of experts chose the same route in 83.3% of comparisons (125 of 150). Researchers given the same information mostly converge on the documented route, though other routes may still be viable.

Outside MBM, expert studies also rate the routes researchers chose above model proposals. In IdeaTaste's blinded ratings (Table 3), experts score human papers above LLM proposals on expected success (70.7 vs. 50.4 out of 100), with a much smaller gap in presentation (62.5 vs. 58.2), and the gap is widest for proposals no human paper shares (72.4 vs. 46.6); these are ratings, not outcomes. Likewise, in the Ideation–Execution Gap study, the advantage AI ideas held before execution disappears once experts carry them out.

How the cards limit shortcuts from recall. All four cards use a common vocabulary and omit paper-specific details, so recognizing a paper's wording helps little; a model that remembers the method still has to identify its principle and choose the card that describes it.

Table 2 reports both tasks for the ten evaluated systems on all 789 papers; models trained on MBM are evaluated on the fixed 300-paper evaluation split. Section 3 now covers EXP-Bench, AInstein, IdeaTaste and Chen et al. (2026). We answer the comparisons with specific work in the ckeZ and RhE6 threads. Changes from the submitted version are marked in blue throughout the paper and supplement.

AuthorsOpenReview ↗

General Response 2/2: route frequency, scoring reliability and training results

Route frequency and Allocation accuracy. We split the 300 evaluation papers by how often their documented route appears as an answer among the 390 training papers:

Documented route as a training answerPapersFrequency guessTen systems
never610.0%51.3%
1–4 times9832.7%48.0%
5 or more times14166.7%52.0%

A frequency guess climbs from 0.0% to 66.7% across the three groups, while the ten systems stay flat, averaging only about 0.7 points higher on frequent routes than on never-seen ones, and five of the ten do better on never-seen routes. The same holds on the 756 papers whose documented route is a library card, with frequency counted leaving the paper itself out. Systems score 43–76% on the 21 routes no other paper documents, where a frequency guess scores 0, and their accuracy is only weakly related to route frequency (rank correlation 0.16, 95% CI [0.09, 0.23], against 0.59 for the frequency baseline).

An independent judge. Gemini 3.5 Flash, a judge from outside the ten evaluated systems, rescored Realization on 100 papers. Like the original judge, GPT-5.6 Sol, it received all ten anonymized answers to a paper in one batch. The two judges' F1 rankings of the ten systems have a Spearman correlation of 0.903, agree on 40 of 45 system pairs, and both put Claude Opus 5 first and GPT-6 Astra second.

Pairwise agreement on precision. We treated precision gaps of two points or less as too close to call. On the remaining pairs, the two judges agreed on which system had higher precision 87.5% of the time.

Extending the leaderboard. On 100 papers, we added HY4-Flash, a different model from the HY4 Preview already among the ten, as an eleventh system. Using the existing systems’ recorded judgments to score HY4-Flash alone gave it a mean F1 of 59.60 (8th of 11). Rescoring all eleven systems together, blind to those judgments, gave it 59.55 (9th of 11). Scoring a new system alone against the recorded judgments thus gives nearly the same result as rescoring the whole cohort.

Human audit. For each of 78 randomly drawn papers, reviewers checked the problem statement, the documented mechanism card and one randomly chosen core requirement. With the original labels hidden, they confirmed 69 of the 78 core labels as necessary (88.5%).

FlagRate
Problem statement flagged for a cue to the paper’s identity2 of 78 (2.6%)
Documented card flagged as overly specific3 of 78 (3.8%)
Sampled core requirement flagged as replaceable or nonessential9 of 78 (11.5%)

Self-drafted realizations. For Qwen3.5-35B-A3B, drafting a realization for each candidate before choosing lowered Allocation Top-1 by 5.17 points (95% CI [−9.56, −0.78]). The drop appeared in all six runs, three seeds each for a selector trained with drafts and one trained without, all reading the same drafts. GPT-5.6 Sol, tested on the same items, fell from 52.33% with direct selection to 51.00% with self-drafting.

Section 7. We compared training schedules with the same total number of optimizer updates. In one schedule, 40 or 80 Mechanism Realization updates were followed by 390 Mechanism Allocation updates. In the other, all 430 or 470 updates were spent on Allocation. For both models, we report means over three runs per schedule, each with a different seed for Allocation training. The table gives the difference in mean Allocation Top-1 accuracy between the matched schedules, in points.

Realization → Allocation (updates)Allocation only (updates)Qwen3.5-35B-A3BLlama-3.1-8B-Instruct
40 → 390430+0.83+0.44
80 → 390470+0.78+1.78

Allocation accuracy had largely leveled off by 390 updates of Allocation-only training. In each of the four comparisons, mean Allocation accuracy was slightly higher when training began with Realization.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.