- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces NatureGym, an LLM-agent pipeline that turns a Nature-family paper into a containerized task package. Each package has a task brief, the paper's data, a hidden test set and evaluator, and the paper's reported SOTA score as an anchor. An "information firewall" removes the source method from the package. Applying the pipeline to ~5,500 papers (2022–2025, ten journals) and an evaluation-time calibration step yields NatureBench: 90 tasks, 332 evaluation instances, six scientific domains and 80 distinct primary metrics.
Agents are scored with a SOTA-normalized relative gap , averaged over instances. The headline metrics are Surpass-SOTA () and Match-SOTA (). A post-hoc GPT-5.5 judge invalidates runs with shortcuts (fabrication, hand-written rules, answer recovery, feedback gaming).
Critical strengths
-
A large, cross-disciplinary set of executable tasks anchored to real scientific results. Most agent-on-research benchmarks either reproduce known results (PaperBench, CORE-Bench, ReplicationBench) or optimize on ML-engineering proxies (MLE-bench, PostTrainBench). NatureBench covers omics, protein biology, molecular design, physical modeling and biomedical tasks with each paper's own data and metric, in a containerized environment with a hidden evaluator. The build–verify pipeline is also a reusable artifact: the 36-check package verification, the method firewall (§2.3–2.4, App. B.4), and especially the reproduce-mode audit (App. D, Fig. 6). In that audit, an agent is given the paper and asked to reproduce it, which surfaces package defects that structural checks miss; it led to dropping 45 tasks. If the tasks are released as described, this is a useful substrate for the community, independent of the specific findings.
-
Behavioral evidence that current agents succeed through generic ML rather than domain reasoning, and that more time increases shortcut-taking. The method-pathway analysis (§5.1, Fig. 4) shows that agents tend to recast scientific problems as supervised prediction, and that runs using the same method family as the paper match SOTA more often (37.8% vs. 26.8%). The time-scaling study (App. I) shows that 72h budgets reduce timeouts but not failures, and that Opus 4.7's invalid runs rise from 3 to 7 out of 20. These are useful observations for anyone building autonomous science agents, as long as the measurement concerns below are addressed.
Critical weaknesses
-
"Surpass-SOTA" does not measure discovery relative to the published SOTA, as the experimental setting is not invariable.
- Unlimited feedback on the hidden test set. During the run,
/evaluatescores each submission "on every instance across all reported metrics and returns raw scores, relative gaps, and the running best" (l. 1311–1314). There is no separate validation split for feedback. One run made 258 submissions (l. 1580). Agents can therefore select models, thresholds and ensembles directly on the test set, while the paper's SOTA was (presumably) not tuned on its test set. If the reported score is the best submission (the paper refers to "the running best" and "best @ 220"), there is an additional max-over-noisy-evaluations bias. The validity judge targets explicit "feedback gaming", but ordinary hyperparameter and checkpoint selection on test data is not a shortcut the judge would flag, and it still inflates . This alone could explain a large share of the 23.3% / 57.8% figures. - Anchor attainability is shown for only a minority of tasks. In reproduce mode, where the agent receives the paper itself, at least one model reaches the anchor on only 33/90 tasks (l. 1239–1240). The remaining failures are attributed to "uniform resource budget and agent capability" (l. 261–262). Many source methods were trained for far longer than 4 hours on one RTX 3090/4090. For the other ~57 tasks, falling short of SOTA could reflect compute rather than discovery ability, and "insufficient budget or time" is itself 17.6% of failures (l. 430). The headline "agents rarely match published SOTA" therefore mixes capability with a compute mismatch.
- The anchor is publication-time SOTA, and training-data exposure is not measured. Source papers date from 2022–2025, and the evaluated models were trained in 2026. Many tasks (e.g., Nucleotide Transformer benchmarks, USPTO-480k) have well-known later methods that beat the original numbers. Beating a 2022 anchor with a method published in 2024 is recall, not discovery. Web search is disabled, but no memorization or recall check is reported (e.g., asking each model to name the source paper or method from the task brief). The firewall removes the method from the package, not from the model's weights.
- The threshold depends on metric scale and headroom. For a bounded metric with SOTA 0.9 (e.g., accuracy), requires . For an AUPRC of 0.3, it requires only 0.33. For lower-is-better metrics, . Means as low as (Table 5) show how unstable the normalization is. The paper acknowledges this (App. H) but keeps the 0.1 threshold as the headline metric. Surpass-SOTA counts are therefore not comparable across tasks or domains. A headroom-normalized gap, e.g., , or normalizing between the optional baseline and SOTA (already in the record, l. 132), would be more interpretable.
- Is the evaluation protocol the same as the paper's? In Case 1, a standard ChebNet ensemble beats a Nature BME method by 17.7% AUPRC after retraining on train+val (l. 1535–1541). This is consistent with test-set selection or a protocol mismatch (split, use of validation labels) rather than a stronger method. The paper does not report checks that the agent's protocol matches the one behind each anchor.
- Unlimited feedback on the hidden test set. During the run,
-
The behavioral and comparative claims rest on single runs and unvalidated LLM judgments, and the task set was filtered with the evaluated models.
- No run-to-run variance. Each agent is run once per task. Many adjacent leaderboard entries differ by one task (e.g., 11.1% vs. 10.0%; Table 2), and per-domain results use as few as 5 tasks (Relational: 13 of 16 agents at exactly 60%, i.e. the same 3/5). No confidence intervals or repeated runs are given, so the ranking and the domain "tiers" (§5.2) are not statistically supported.
- The validity judge is a single, unvalidated LLM. GPT-5.5 decides validity, and invalid runs are scored . No agreement with human adjudication (precision/recall on a labeled sample) is reported. GPT-5.5 is also the agent most affected (40 invalid runs; CR 54.4% vs. SR 98.9%). This is either a real behavioral difference or judge bias toward its own family, and the paper cannot tell them apart. The judge's criteria also conflict with the paper's framing. "Rule substitution for learning" invalidates hand-written splice-site rules (Case 2), but motif-based rules grounded in domain knowledge are a legitimate scientific approach. Penalizing them while concluding that agents lack "domain reasoning" (§5.1) is inconsistent.
- Annotation of success and failure modes is not described. The 1,440-run annotation (§5.1) is described as "we annotate" and the AI statement says LLMs assisted "trajectory analysis and interpretation" (l. 489). Who or what labeled method families, success modes and failure layers? With what codebook and what inter-annotator agreement? The paper's main scientific conclusion ("methodological translation rather than scientific invention") depends on these labels.
- Selection effects from calibration. Tasks were dropped based on trajectories from Claude Opus 4.6 (base mode) and Opus 4.6 / DeepSeek-V4-Pro (reproduce mode) (§3.2). Both are also evaluated. Removing ~70 of ~160 tasks (45 in the reproduce audit alone) after observing these models' behavior may bias the final set for or against these models and their harness (Claude Code). It also raises a question about residual defects: if ~44% of verified packages were defective, how many defects remain in the 90? A human-verified random subset would help.
- Harness confound. 13 models use Claude Code, GPT models use Codex CLI, and Gemini uses Gemini CLI. Cross-family comparisons therefore mix model and harness effects.
Questions for authors
- Is there any split between the data used for in-run feedback and the data used for final scoring? Is the final reported score the best submission, the last one, or a designated final submission? What are the Surpass/Match-SOTA rates if each agent is scored only on its final (or first) submission, or on a held-out portion of the test set never exposed through
/evaluate? - For how many of the 90 tasks has it been shown that the source method (author code) reaches the anchor within the benchmark's 4h / single-GPU budget? Can you report results separately for the 33 tasks where reproduce mode reached the anchor?
- For each task, is the anchor still SOTA at the evaluated models' training cutoff? Did you run any memorization check (e.g., asking models to identify the source paper or method from the README)? For the 21 tasks Opus 5 surpasses, how many solutions correspond to methods published after the source paper?
- What is the judge's agreement with human adjudication on a labeled sample, and how are borderline cases (domain-knowledge rules, rule-based post-processing) handled? Does using a different judge model change the GPT-5.5 completion rate?
- Who produced the method-family, success-mode and failure-layer labels in §5.1 (humans, LLMs, or both)? Please give the codebook and inter-annotator agreement.
- Can you rerun a subset (e.g., top 4 agents × 30 tasks × 3 seeds) to estimate run-to-run variance and give CIs for Table 2?
- For Case 1 and other large- cases, does the agent's protocol (splits, use of validation labels, retraining on train+val) match the protocol behind the paper's anchor?
- How do Opus 4.6 and DeepSeek-V4-Pro rank on the dropped tasks versus the retained tasks? Please give exact counts for each funnel step in Table 4.
9 comments
Reply to W2 & Q2: Reproduction difficulty and evaluation budgets
Reproduction difficulty and runtime requirements.
Reproducing a method from a paper requires substantial implementation work. In our audit, we gave Claude Opus 4.6 and DeepSeek-V4-Pro the source papers and asked them to implement, debug, and train the described methods themselves (Section 3.2 and Appendix D). We drew on the reproduction setup in PaperBench, whose results illustrate this difficulty. Its best 12-hour BasicAgent configuration achieved a mean replication score of 21.0% (Section 5.2, Table 4). A failed agent reproduction alone therefore cannot establish that the time budget was insufficient. Accordingly, we use these runs primarily to check task package quality. We compare attempted implementations with the source papers and check the data, evaluators, and anchors for consistency regardless of the reproduction score, complementing the construction checks in Appendix B.4.
Resource allocation and interpretation.
The difficulty and cost of reproducing the published results limited our assessment to resource estimates. We use reported hardware and runtimes where available and otherwise estimate requirements from the data, training procedure, and task. These estimates guide the choice of CPU only or a single RTX 3090, RTX 4090, or A800 GPU, with a four-hour limit for every task (Section 4.1). The main evaluation does not require agents to reproduce the source methods. Agents may use any valid approach within the allotted time and compute, so we just assign each task a resource tier suited to its estimated needs. The main results should be interpreted under the resource conditions in Section 4.1.
Effect of extending the time budget.
To examine how much the time limit contributes, Appendix I compares DeepSeek-V4-Pro and Claude Opus 4.7 at 4, 12, and 72 hours on 20 tasks across six domains, including 16 with prior signs of budget constraints and four near-anchor controls. Timeouts fell from 19 to 13 to 6 out of 40 runs per budget, while Match-SOTA counts were 7, 6, and 7. At 72 hours, 22 of the 33 runs that fell short of Match-SOTA had method-layer failures. The longer budgets reduced timeouts and, for some tasks, exceeded the runtimes reported in the source papers. The overall Match-SOTA count remained roughly constant, suggesting that extending the time limit had little effect on aggregate Match-SOTA performance on these tasks. Choosing and implementing valid methods remained a major challenge.
Main-evaluation results on the requested subset.
The table reports Match / Surpass counts out of the 33 tasks where at least one audit agent met the reproduction-success criterion.
| Configuration | Match / Surpass | Configuration | Match / Surpass |
|---|---|---|---|
| Claude Opus 5 | 27 / 9 | Kimi K2.6 | 19 / 6 |
| Claude Opus 4.7 | 25 / 11 | Qwen 3.7 Max | 16 / 5 |
| Kimi K3 | 20 / 8 | MiniMax-M3 | 18 / 3 |
| GLM-5.2 | 24 / 10 | GLM-5.1 | 18 / 4 |
| Qwen 3.8 Max | 21 / 7 | GPT-5.4 | 20 / 6 |
| Claude Opus 4.6 | 17 / 7 | GPT-5.5 | 17 / 4 |
| GLM-5.3 | 18 / 7 | DeepSeek-V4-Pro | 18 / 3 |
| Gemini 3.5 Flash | 18 / 8 | MiniMax-M2.7 | 10 / 1 |
Overall attainment in the main evaluation was higher on these 33 tasks. Resource limits may contribute to this gap. The groups were defined by audit performance, which also depends on the difficulty of reproducing the source methods and of solving the tasks themselves. The gap alone therefore provides insufficient evidence that the budget was inadequate.
Reply to W3 & Q3: Publication-time anchors, training exposure, and later methods
We select tasks that arise from a core contribution of the source paper and use the metrics on which the paper bases its SOTA claims and with which it demonstrates the effectiveness of its proposed inventions (Sections 2.2 and 3.4, Appendix B.3). The paper's reported score on the selected metric provides a fixed reference for all agents. In NatureBench, discovery refers to an agent's ability to find an effective solution to a scientific task by formulating, implementing, and testing its own approaches within a fixed budget, with the source method withheld. Agents may use any approach that meets the task's validity requirements, including adapting a general-purpose ML method to a specific scientific problem, a type of contribution covered by our paper selection criteria (Appendix B.1). Finding a valid solution that improves on the published score with the source method withheld is therefore a measurable outcome of discovery on the original research task. We have narrowed the claims in the paper to clarify that the main results show how agents perform against the original published scores under our evaluation protocol. These results come from runs with fixed time and compute budgets and repeated score feedback from the final evaluation set. Comparisons with later work or with the current state of the art are outside the scope of this evaluation.
We agree that knowledge of later methods can contribute to improvements. Determining which anchors remained SOTA at each model's training cutoff requires reliable cutoff dates and task-by-task checks of later results under comparable protocols. Determining which Surpass solutions correspond to later publications would require an exhaustive search of the global literature for each task and a comparison of the agents' implementations with those reported in later work. Given that training-data cutoff dates are available for only some models and that conducting the review described above at this scale would be impractical, we examined potential contamination through a source recognition probe and a review of agent trajectories(Appendix M). We also checked the agent-visible README, data description, and data filenames for all 90 tasks and found no full source title or DOI.
For the probe, we asked 16 models to identify the source paper from the agent-visible README and data description, with no tools. A correct full DOI or normalized full title counts as an exact hit.
| Model | Exact DOI | Exact title | Exact hits |
|---|---|---|---|
| GPT-5.5 | 5 | 3 | 5/90 (5.6%) |
| Claude Opus 5 | 17 | 23 | 25/90 (27.8%) |
| Kimi K3 | 1 | 3 | 4/90 (4.4%) |
| DeepSeek-V4-Pro | 5 | 5 | 7/90 (7.8%) |
| Claude Opus 4.6 | 12 | 12 | 14/90 (15.6%) |
| Claude Opus 4.7 | 4 | 4 | 6/90 (6.7%) |
| Gemini 3.5 Flash | 4 | 2 | 5/90 (5.6%) |
| Kimi K2.6 | 1 | 2 | 2/90 (2.2%) |
| GLM-5.3 | 0 | 1 | 1/90 (1.1%) |
| GPT-5.4 | 1 | 1 | 1/90 (1.1%) |
| GLM-5.1 | 0 | 0 | 0/90 (0.0%) |
| GLM-5.2 | 0 | 0 | 0/90 (0.0%) |
| MiniMax-M2.7 | 0 | 0 | 0/90 (0.0%) |
| MiniMax-M3 | 0 | 0 | 0/90 (0.0%) |
| Qwen 3.7 Max | 0 | 0 | 0/90 (0.0%) |
| Qwen 3.8 Max | 0 | 0 | 0/90 (0.0%) |
The hits show that some source papers remain recognizable despite removing explicit identifiers. For Claude Opus 5, Match-SOTA was 52% (13/25) on tasks with confirmed source identification and 60% (39/65) on the remaining tasks. Surpass-SOTA was 32% (8/25) and 20% (13/65), respectively. The groups differ in size and may differ in task difficulty. With Match-SOTA lower and Surpass-SOTA higher in the confirmed-hit group, we report these rates descriptively and draw no conclusion about an overall performance advantage on those tasks.
To assess whether agents reconstructed source methods in their scored solutions, we reviewed the main-evaluation trajectories corresponding to all 70 hits and 30 valid Surpass-SOTA runs on distinct tasks, including all 21 Claude Opus 5 Surpass runs. In one of the 30 high-scoring runs, the solution resembled the source method. The corresponding trajectory contained neither the paper’s full title nor its DOI. Across all solutions we reviewed, we found no evidence that any agent reconstructed a source paper’s workflow. This check addresses the concern about observable reconstruction in solutions. It cannot exclude subtler effects of prior exposure on scores, which we acknowledge in Section 6.1.
The pre- and post-cutoff groups differ in size and domain composition and may also differ in task difficulty. Reliable training cutoff dates are available for only a subset of models. These differences prevent us from separating the effects of training exposure from those of task characteristics, so we do not report the comparison.
Thank you for taking the time to review our paper. We address your comments and questions below and have marked the manuscript revisions in blue for ease of reference.
Reply to W1 & Q1: Test-set feedback and best-submission selection
In-run feedback and final scoring use the same evaluation set. As described in Section 3.4 and Appendix F,
/evaluatereturns scores while the labels and evaluator remain outside the agent's container. We report the highest-scoring submission produced within the run budget, subject to validity review. There is no separate feedback-free test partition or designated final submission.Our goal is to measure how much agents can improve a solution through experimentation. Feedback makes the objective concrete and allows the agent to compare experiments and decide how to use the remaining time. Taking the best result preserves an improvement already achieved if later exploration performs worse. For these reasons, we provide score feedback to guide iteration and report the best score achieved within the run budget, a design also used in RE-Bench .
To quantify the effect of this scoring choice, we compare first, best, and last across all 1,440 configuration–task pairs. Using last instead of best reduces overall Match by 1.53 percentage points and Surpass by 0.63 points. Ten of 16 configurations retain their Surpass counts, including Claude Opus 5 at 21/90. The aggregate counts (first / best / last) are 154 / 423 / 401 for Match and 38 / 139 / 130 for Surpass. The table gives all configurations.
These comparisons show a small aggregate difference between best and last under the existing feedback protocol. To examine how withholding test-set scores during the run affects Match-SOTA and Surpass-SOTA, we compare the main evaluation with format-only feedback for Claude Opus 5 and DeepSeek-V4-Pro on NatureBench-25. Under the format-only protocol, the evaluator returns format checks during the run and scores the locked final submission on the original test set using the same metrics as in the main evaluation.
We also compared attainment task by task. For each configuration, percentages are calculated over all 25 tasks, separately for Match-SOTA and Surpass-SOTA. Gained means the threshold was met only with format-only feedback. Lost means it was met only in the main evaluation. Unchanged means it was met in both conditions or missed in both.
Claude Opus 5 retains 14 of its 17 Match-SOTA results and gains two matches. It retains four of its five Surpass-SOTA results, with the remaining task still meeting Match-SOTA. DeepSeek-V4-Pro gains and loses two Match-SOTA results, leaving the total at five. It retains its original Surpass-SOTA result and gains one more.
On these selected tasks, we found no consistent score advantage for the main protocol. Repeated score feedback can still lead agents to tune their solutions to the evaluation set. The leaderboard compares agent configurations under fixed time and compute budgets with access to scores on the final evaluation set during each run. We have made the leaderboard's scope explicit in the manuscript and interpret the results in the paper under its stated feedback and scoring conditions.