acceptodds
Under review as a conference paper at ICLR 2027

NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?

10 views

Abstract

We introduce NatureBench, a cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems. NatureBench is built on NatureGym, an automated pipeline that constructs a standardized, per-task containerized environment from a source paper, addressing the environment-fragmentation problem that has limited the credibility of prior agent-on-research benchmarks. Evaluating sixteen frontier agent configurations under a strict web-search-disabled protocol, we find that the strongest model surpasses SOTA on only 23.3% of tasks under the criterion. Analysis of method pathways reveals that agents succeed primarily through methodological translation, converting scientific tasks into familiar supervised prediction problems, rather than through genuine scientific invention. Failures are dominated by wrong method choice and judge-identified shortcuts, followed by insufficient compute budget, not by task misunderstanding. We release the benchmark, the NatureGym pipeline, and a public leaderboard with maintainer-side reproduction.

Primary area:
datasets and benchmarks
Keywords:
scientific benchmarks, coding agents, scientific discovery, agent evaluation

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_bnndOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces NatureGym, an LLM-agent pipeline that turns a Nature-family paper into a containerized task package. Each package has a task brief, the paper's data, a hidden test set and evaluator, and the paper's reported SOTA score as an anchor. An "information firewall" removes the source method from the package. Applying the pipeline to ~5,500 papers (2022–2025, ten journals) and an evaluation-time calibration step yields NatureBench: 90 tasks, 332 evaluation instances, six scientific domains and 80 distinct primary metrics.

Agents are scored with a SOTA-normalized relative gap g=dir⋅(m−msota)/∣msota∣g = \mathrm{dir}\cdot(m - m^{\mathrm{sota}})/|m^{\mathrm{sota}}|, averaged over instances. The headline metrics are Surpass-SOTA (g>0.1g > 0.1) and Match-SOTA (g≥0g \ge 0). A post-hoc GPT-5.5 judge invalidates runs with shortcuts (fabrication, hand-written rules, answer recovery, feedback gaming).

Critical strengths
  1. A large, cross-disciplinary set of executable tasks anchored to real scientific results. Most agent-on-research benchmarks either reproduce known results (PaperBench, CORE-Bench, ReplicationBench) or optimize on ML-engineering proxies (MLE-bench, PostTrainBench). NatureBench covers omics, protein biology, molecular design, physical modeling and biomedical tasks with each paper's own data and metric, in a containerized environment with a hidden evaluator. The build–verify pipeline is also a reusable artifact: the 36-check package verification, the method firewall (§2.3–2.4, App. B.4), and especially the reproduce-mode audit (App. D, Fig. 6). In that audit, an agent is given the paper and asked to reproduce it, which surfaces package defects that structural checks miss; it led to dropping 45 tasks. If the tasks are released as described, this is a useful substrate for the community, independent of the specific findings.

  2. Behavioral evidence that current agents succeed through generic ML rather than domain reasoning, and that more time increases shortcut-taking. The method-pathway analysis (§5.1, Fig. 4) shows that agents tend to recast scientific problems as supervised prediction, and that runs using the same method family as the paper match SOTA more often (37.8% vs. 26.8%). The time-scaling study (App. I) shows that 72h budgets reduce timeouts but not failures, and that Opus 4.7's invalid runs rise from 3 to 7 out of 20. These are useful observations for anyone building autonomous science agents, as long as the measurement concerns below are addressed.

Critical weaknesses
  1. "Surpass-SOTA" does not measure discovery relative to the published SOTA, as the experimental setting is not invariable.

    • Unlimited feedback on the hidden test set. During the run, /evaluate scores each submission "on every instance across all reported metrics and returns raw scores, relative gaps, and the running best" (l. 1311–1314). There is no separate validation split for feedback. One run made 258 submissions (l. 1580). Agents can therefore select models, thresholds and ensembles directly on the test set, while the paper's SOTA was (presumably) not tuned on its test set. If the reported score is the best submission (the paper refers to "the running best" and "best @ 220"), there is an additional max-over-noisy-evaluations bias. The validity judge targets explicit "feedback gaming", but ordinary hyperparameter and checkpoint selection on test data is not a shortcut the judge would flag, and it still inflates gg. This alone could explain a large share of the 23.3% / 57.8% figures.
    • Anchor attainability is shown for only a minority of tasks. In reproduce mode, where the agent receives the paper itself, at least one model reaches the anchor on only 33/90 tasks (l. 1239–1240). The remaining failures are attributed to "uniform resource budget and agent capability" (l. 261–262). Many source methods were trained for far longer than 4 hours on one RTX 3090/4090. For the other ~57 tasks, falling short of SOTA could reflect compute rather than discovery ability, and "insufficient budget or time" is itself 17.6% of failures (l. 430). The headline "agents rarely match published SOTA" therefore mixes capability with a compute mismatch.
    • The anchor is publication-time SOTA, and training-data exposure is not measured. Source papers date from 2022–2025, and the evaluated models were trained in 2026. Many tasks (e.g., Nucleotide Transformer benchmarks, USPTO-480k) have well-known later methods that beat the original numbers. Beating a 2022 anchor with a method published in 2024 is recall, not discovery. Web search is disabled, but no memorization or recall check is reported (e.g., asking each model to name the source paper or method from the task brief). The firewall removes the method from the package, not from the model's weights.
    • The g>0.1g > 0.1 threshold depends on metric scale and headroom. For a bounded metric with SOTA 0.9 (e.g., accuracy), g>0.1g > 0.1 requires >0.99> 0.99. For an AUPRC of 0.3, it requires only 0.33. For lower-is-better metrics, g≤1g \le 1. Means as low as gˉall=−10.18\bar g_{\text{all}} = -10.18 (Table 5) show how unstable the normalization is. The paper acknowledges this (App. H) but keeps the 0.1 threshold as the headline metric. Surpass-SOTA counts are therefore not comparable across tasks or domains. A headroom-normalized gap, e.g., (m−msota)/(m⋆−msota)(m - m^{\mathrm{sota}})/(m^{\star} - m^{\mathrm{sota}}), or normalizing between the optional baseline BB and SOTA (already in the record, l. 132), would be more interpretable.
    • Is the evaluation protocol the same as the paper's? In Case 1, a standard ChebNet ensemble beats a Nature BME method by 17.7% AUPRC after retraining on train+val (l. 1535–1541). This is consistent with test-set selection or a protocol mismatch (split, use of validation labels) rather than a stronger method. The paper does not report checks that the agent's protocol matches the one behind each anchor.
  2. The behavioral and comparative claims rest on single runs and unvalidated LLM judgments, and the task set was filtered with the evaluated models.

    • No run-to-run variance. Each agent is run once per task. Many adjacent leaderboard entries differ by one task (e.g., 11.1% vs. 10.0%; Table 2), and per-domain results use as few as 5 tasks (Relational: 13 of 16 agents at exactly 60%, i.e. the same 3/5). No confidence intervals or repeated runs are given, so the ranking and the domain "tiers" (§5.2) are not statistically supported.
    • The validity judge is a single, unvalidated LLM. GPT-5.5 decides validity, and invalid runs are scored g=−1g = -1. No agreement with human adjudication (precision/recall on a labeled sample) is reported. GPT-5.5 is also the agent most affected (40 invalid runs; CR 54.4% vs. SR 98.9%). This is either a real behavioral difference or judge bias toward its own family, and the paper cannot tell them apart. The judge's criteria also conflict with the paper's framing. "Rule substitution for learning" invalidates hand-written splice-site rules (Case 2), but motif-based rules grounded in domain knowledge are a legitimate scientific approach. Penalizing them while concluding that agents lack "domain reasoning" (§5.1) is inconsistent.
    • Annotation of success and failure modes is not described. The 1,440-run annotation (§5.1) is described as "we annotate" and the AI statement says LLMs assisted "trajectory analysis and interpretation" (l. 489). Who or what labeled method families, success modes and failure layers? With what codebook and what inter-annotator agreement? The paper's main scientific conclusion ("methodological translation rather than scientific invention") depends on these labels.
    • Selection effects from calibration. Tasks were dropped based on trajectories from Claude Opus 4.6 (base mode) and Opus 4.6 / DeepSeek-V4-Pro (reproduce mode) (§3.2). Both are also evaluated. Removing ~70 of ~160 tasks (45 in the reproduce audit alone) after observing these models' behavior may bias the final set for or against these models and their harness (Claude Code). It also raises a question about residual defects: if ~44% of verified packages were defective, how many defects remain in the 90? A human-verified random subset would help.
    • Harness confound. 13 models use Claude Code, GPT models use Codex CLI, and Gemini uses Gemini CLI. Cross-family comparisons therefore mix model and harness effects.
Questions for authors
  1. Is there any split between the data used for in-run feedback and the data used for final scoring? Is the final reported score the best submission, the last one, or a designated final submission? What are the Surpass/Match-SOTA rates if each agent is scored only on its final (or first) submission, or on a held-out portion of the test set never exposed through /evaluate?
  2. For how many of the 90 tasks has it been shown that the source method (author code) reaches the anchor within the benchmark's 4h / single-GPU budget? Can you report results separately for the 33 tasks where reproduce mode reached the anchor?
  3. For each task, is the anchor still SOTA at the evaluated models' training cutoff? Did you run any memorization check (e.g., asking models to identify the source paper or method from the README)? For the 21 tasks Opus 5 surpasses, how many solutions correspond to methods published after the source paper?
  4. What is the judge's agreement with human adjudication on a labeled sample, and how are borderline cases (domain-knowledge rules, rule-based post-processing) handled? Does using a different judge model change the GPT-5.5 completion rate?
  5. Who produced the method-family, success-mode and failure-layer labels in §5.1 (humans, LLMs, or both)? Please give the codebook and inter-annotator agreement.
  6. Can you rerun a subset (e.g., top 4 agents × 30 tasks × 3 seeds) to estimate run-to-run variance and give CIs for Table 2?
  7. For Case 1 and other large-gg cases, does the agent's protocol (splits, use of validation labels, retraining on train+val) match the protocol behind the paper's anchor?
  8. How do Opus 4.6 and DeepSeek-V4-Pro rank on the dropped tasks versus the retained tasks? Please give exact counts for each funnel step in Table 4.
9 comments

Thank you for taking the time to review our paper. We address your comments and questions below and have marked the manuscript revisions in blue for ease of reference.

Reply to W1 & Q1: Test-set feedback and best-submission selection

In-run feedback and final scoring use the same evaluation set. As described in Section 3.4 and Appendix F, /evaluate returns scores while the labels and evaluator remain outside the agent's container. We report the highest-scoring submission produced within the run budget, subject to validity review. There is no separate feedback-free test partition or designated final submission.

Our goal is to measure how much agents can improve a solution through experimentation. Feedback makes the objective concrete and allows the agent to compare experiments and decide how to use the remaining time. Taking the best result preserves an improvement already achieved if later exploration performs worse. For these reasons, we provide score feedback to guide iteration and report the best score achieved within the run budget, a design also used in RE-Bench .

To quantify the effect of this scoring choice, we compare first, best, and last across all 1,440 configuration–task pairs. Using last instead of best reduces overall Match by 1.53 percentage points and Surpass by 0.63 points. Ten of 16 configurations retain their Surpass counts, including Claude Opus 5 at 21/90. The aggregate counts (first / best / last) are 154 / 423 / 401 for Match and 38 / 139 / 130 for Surpass. The table gives all configurations.

ConfigurationMatch: first / best / lastSurpass: first / best / last
Claude Opus 520 / 52 / 519 / 21 / 21
Claude Opus 4.716 / 37 / 356 / 14 / 12
Claude Opus 4.616 / 26 / 245 / 10 / 9
GPT-5.55 / 23 / 220 / 4 / 4
GPT-5.411 / 22 / 203 / 6 / 6
Gemini 3.5 Flash7 / 23 / 212 / 9 / 9
Kimi K313 / 30 / 284 / 13 / 12
Kimi K2.66 / 25 / 250 / 7 / 7
DeepSeek-V4-Pro8 / 21 / 202 / 4 / 3
GLM-5.34 / 23 / 221 / 9 / 9
GLM-5.212 / 32 / 322 / 12 / 12
GLM-5.19 / 22 / 171 / 6 / 3
MiniMax-M37 / 25 / 231 / 6 / 6
MiniMax-M2.74 / 11 / 101 / 1 / 0
Qwen 3.8 Max8 / 32 / 320 / 10 / 10
Qwen 3.7 Max8 / 19 / 191 / 7 / 7

These comparisons show a small aggregate difference between best and last under the existing feedback protocol. To examine how withholding test-set scores during the run affects Match-SOTA and Surpass-SOTA, we compare the main evaluation with format-only feedback for Claude Opus 5 and DeepSeek-V4-Pro on NatureBench-25. Under the format-only protocol, the evaluator returns format checks during the run and scores the locked final submission on the original test set using the same metrics as in the main evaluation.

Main evaluationFormat-only feedback
ConfigurationMatch-SOTASurpass-SOTAMatch-SOTASurpass-SOTA
Claude Opus 517/255/2516/254/25
DeepSeek-V4-Pro5/251/255/252/25

We also compared attainment task by task. For each configuration, percentages are calculated over all 25 tasks, separately for Match-SOTA and Surpass-SOTA. Gained means the threshold was met only with format-only feedback. Lost means it was met only in the main evaluation. Unchanged means it was met in both conditions or missed in both.

Match-SOTASurpass-SOTA
ConfigurationGainedLostUnchangedGainedLostUnchanged
Claude Opus 58%12%80%0%4%96%
DeepSeek-V4-Pro8%8%84%4%0%96%

Claude Opus 5 retains 14 of its 17 Match-SOTA results and gains two matches. It retains four of its five Surpass-SOTA results, with the remaining task still meeting Match-SOTA. DeepSeek-V4-Pro gains and loses two Match-SOTA results, leaving the total at five. It retains its original Surpass-SOTA result and gains one more.

On these selected tasks, we found no consistent score advantage for the main protocol. Repeated score feedback can still lead agents to tune their solutions to the evaluation set. The leaderboard compares agent configurations under fixed time and compute budgets with access to scores on the final evaluation set during each run. We have made the leaderboard's scope explicit in the manuscript and interpret the results in the paper under its stated feedback and scoring conditions.

Reply to W2 & Q2: Reproduction difficulty and evaluation budgets

Reproduction difficulty and runtime requirements.

Reproducing a method from a paper requires substantial implementation work. In our audit, we gave Claude Opus 4.6 and DeepSeek-V4-Pro the source papers and asked them to implement, debug, and train the described methods themselves (Section 3.2 and Appendix D). We drew on the reproduction setup in PaperBench, whose results illustrate this difficulty. Its best 12-hour BasicAgent configuration achieved a mean replication score of 21.0% (Section 5.2, Table 4). A failed agent reproduction alone therefore cannot establish that the time budget was insufficient. Accordingly, we use these runs primarily to check task package quality. We compare attempted implementations with the source papers and check the data, evaluators, and anchors for consistency regardless of the reproduction score, complementing the construction checks in Appendix B.4.

Resource allocation and interpretation.

The difficulty and cost of reproducing the published results limited our assessment to resource estimates. We use reported hardware and runtimes where available and otherwise estimate requirements from the data, training procedure, and task. These estimates guide the choice of CPU only or a single RTX 3090, RTX 4090, or A800 GPU, with a four-hour limit for every task (Section 4.1). The main evaluation does not require agents to reproduce the source methods. Agents may use any valid approach within the allotted time and compute, so we just assign each task a resource tier suited to its estimated needs. The main results should be interpreted under the resource conditions in Section 4.1.

Effect of extending the time budget.

To examine how much the time limit contributes, Appendix I compares DeepSeek-V4-Pro and Claude Opus 4.7 at 4, 12, and 72 hours on 20 tasks across six domains, including 16 with prior signs of budget constraints and four near-anchor controls. Timeouts fell from 19 to 13 to 6 out of 40 runs per budget, while Match-SOTA counts were 7, 6, and 7. At 72 hours, 22 of the 33 runs that fell short of Match-SOTA had method-layer failures. The longer budgets reduced timeouts and, for some tasks, exceeded the runtimes reported in the source papers. The overall Match-SOTA count remained roughly constant, suggesting that extending the time limit had little effect on aggregate Match-SOTA performance on these tasks. Choosing and implementing valid methods remained a major challenge.

Main-evaluation results on the requested subset.

The table reports Match / Surpass counts out of the 33 tasks where at least one audit agent met the reproduction-success criterion.

ConfigurationMatch / SurpassConfigurationMatch / Surpass
Claude Opus 527 / 9Kimi K2.619 / 6
Claude Opus 4.725 / 11Qwen 3.7 Max16 / 5
Kimi K320 / 8MiniMax-M318 / 3
GLM-5.224 / 10GLM-5.118 / 4
Qwen 3.8 Max21 / 7GPT-5.420 / 6
Claude Opus 4.617 / 7GPT-5.517 / 4
GLM-5.318 / 7DeepSeek-V4-Pro18 / 3
Gemini 3.5 Flash18 / 8MiniMax-M2.710 / 1

Overall attainment in the main evaluation was higher on these 33 tasks. Resource limits may contribute to this gap. The groups were defined by audit performance, which also depends on the difficulty of reproducing the source methods and of solving the tasks themselves. The gap alone therefore provides insufficient evidence that the budget was inadequate.

Reply to W3 & Q3: Publication-time anchors, training exposure, and later methods

We select tasks that arise from a core contribution of the source paper and use the metrics on which the paper bases its SOTA claims and with which it demonstrates the effectiveness of its proposed inventions (Sections 2.2 and 3.4, Appendix B.3). The paper's reported score on the selected metric provides a fixed reference for all agents. In NatureBench, discovery refers to an agent's ability to find an effective solution to a scientific task by formulating, implementing, and testing its own approaches within a fixed budget, with the source method withheld. Agents may use any approach that meets the task's validity requirements, including adapting a general-purpose ML method to a specific scientific problem, a type of contribution covered by our paper selection criteria (Appendix B.1). Finding a valid solution that improves on the published score with the source method withheld is therefore a measurable outcome of discovery on the original research task. We have narrowed the claims in the paper to clarify that the main results show how agents perform against the original published scores under our evaluation protocol. These results come from runs with fixed time and compute budgets and repeated score feedback from the final evaluation set. Comparisons with later work or with the current state of the art are outside the scope of this evaluation.

We agree that knowledge of later methods can contribute to improvements. Determining which anchors remained SOTA at each model's training cutoff requires reliable cutoff dates and task-by-task checks of later results under comparable protocols. Determining which Surpass solutions correspond to later publications would require an exhaustive search of the global literature for each task and a comparison of the agents' implementations with those reported in later work. Given that training-data cutoff dates are available for only some models and that conducting the review described above at this scale would be impractical, we examined potential contamination through a source recognition probe and a review of agent trajectories(Appendix M). We also checked the agent-visible README, data description, and data filenames for all 90 tasks and found no full source title or DOI.

For the probe, we asked 16 models to identify the source paper from the agent-visible README and data description, with no tools. A correct full DOI or normalized full title counts as an exact hit.

ModelExact DOIExact titleExact hits
GPT-5.5535/90 (5.6%)
Claude Opus 5172325/90 (27.8%)
Kimi K3134/90 (4.4%)
DeepSeek-V4-Pro557/90 (7.8%)
Claude Opus 4.6121214/90 (15.6%)
Claude Opus 4.7446/90 (6.7%)
Gemini 3.5 Flash425/90 (5.6%)
Kimi K2.6122/90 (2.2%)
GLM-5.3011/90 (1.1%)
GPT-5.4111/90 (1.1%)
GLM-5.1000/90 (0.0%)
GLM-5.2000/90 (0.0%)
MiniMax-M2.7000/90 (0.0%)
MiniMax-M3000/90 (0.0%)
Qwen 3.7 Max000/90 (0.0%)
Qwen 3.8 Max000/90 (0.0%)

The hits show that some source papers remain recognizable despite removing explicit identifiers. For Claude Opus 5, Match-SOTA was 52% (13/25) on tasks with confirmed source identification and 60% (39/65) on the remaining tasks. Surpass-SOTA was 32% (8/25) and 20% (13/65), respectively. The groups differ in size and may differ in task difficulty. With Match-SOTA lower and Surpass-SOTA higher in the confirmed-hit group, we report these rates descriptively and draw no conclusion about an overall performance advantage on those tasks.

To assess whether agents reconstructed source methods in their scored solutions, we reviewed the main-evaluation trajectories corresponding to all 70 hits and 30 valid Surpass-SOTA runs on distinct tasks, including all 21 Claude Opus 5 Surpass runs. In one of the 30 high-scoring runs, the solution resembled the source method. The corresponding trajectory contained neither the paper’s full title nor its DOI. Across all solutions we reviewed, we found no evidence that any agent reconstructed a source paper’s workflow. This check addresses the concern about observable reconstruction in solutions. It cannot exclude subtler effects of prior exposure on scores, which we acknowledge in Section 6.1.

The pre- and post-cutoff groups differ in size and domain composition and may also differ in task difficulty. Reliable training cutoff dates are available for only a subset of models. These differences prevent us from separating the effects of training exposure from those of task characteristics, so we do not report the comparison.

Reviewer_ZzHaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NatureBench which evaluates 16 coding agents on 90 containerized scientific ML tasks derived from Nature-family papers, testing whether they can discover methods that match or beat paper-reported results without web search or access to the source method. The best agent surpasses the anchor on 23.3% of tasks and matches it on 57.8%; the authors argue that most successes come from standard ML reformulations rather than scientific invention.

Critical strengths
  1. The paper targets an important and timely question: whether coding agents can make meaningful progress on real scientific ML problems, rather than only reproduce known code or optimize conventional ML benchmarks.
  2. The paper reports more than an aggregate score. The per-domain analysis, success/failure taxonomy, and discussion of shortcut behavior are useful contributions.
  3. The findings are valuable: agents mainly succeed by reframing tasks as conventional supervised learning; failures usually reflect poor method choices, shortcuts, or limited compute not task misunderstanding.
Critical weaknesses

The authors should provide a substantially stronger contamination analysis. For example, a comparison between tasks from more versus less likely contaminated sources; ablations that redact or perturb recognizable identifiers in task descriptions and filenames; and examples showing that high-performing agents did not reconstruct the source method from latent recall.

Questions for authors

How sensitive are rankings to agent randomness, prompt/harness differences, compute budgets, and the g > 0.1 threshold?

4 comments

Thank you for taking the time to review our paper. We address your comments and questions below and have marked the manuscript revisions in blue for ease of reference.

Reply to W1: Contamination analysis

We present each task as a standalone scientific challenge, withhold the source paper and its method, and disable web search (Sections 2.4 and 4.1). The post-hoc judge receives the source paper's title and DOI to check whether the solution uses the paper, its authors' repository, or associated artifacts (Appendix F.1). We checked the agent-visible README, data description, and data filenames for all 90 tasks and found no full source title or DOI.

We agree, however, that these safeguards cannot remove knowledge acquired during training. To investigate this concern, we added a source-recognition probe and a follow-up trace review(Appendix M). For the probe, we asked 16 models to identify the source paper from the agent-visible README and data description, with no tools. A correct full DOI or normalized full title counts as an exact hit.

ModelExact DOIExact titleExact hits
GPT-5.5535/90 (5.6%)
Claude Opus 5172325/90 (27.8%)
Kimi K3134/90 (4.4%)
DeepSeek-V4-Pro557/90 (7.8%)
Claude Opus 4.6121214/90 (15.6%)
Claude Opus 4.7446/90 (6.7%)
Gemini 3.5 Flash425/90 (5.6%)
Kimi K2.6122/90 (2.2%)
GLM-5.3011/90 (1.1%)
GPT-5.4111/90 (1.1%)
GLM-5.1000/90 (0.0%)
GLM-5.2000/90 (0.0%)
MiniMax-M2.7000/90 (0.0%)
MiniMax-M3000/90 (0.0%)
Qwen 3.7 Max000/90 (0.0%)
Qwen 3.8 Max000/90 (0.0%)

The hits show that some source papers remain recognizable despite removing explicit identifiers. For Claude Opus 5, Match-SOTA was 52% (13/25) on tasks with confirmed source identification and 60% (39/65) on the remaining tasks. Surpass-SOTA was 32% (8/25) and 20% (13/65), respectively. The groups differ in size and may differ in task difficulty. With Match-SOTA lower and Surpass-SOTA higher in the confirmed-hit group, we report these rates descriptively and draw no conclusion about an overall performance advantage on those tasks.

To assess whether agents reconstructed source methods in their scored solutions, we reviewed the main-evaluation trajectories corresponding to all 70 hits and 30 valid Surpass-SOTA runs on distinct tasks, including all 21 Claude Opus 5 Surpass runs. In one of the 30 high-scoring runs, the solution resembled the source method. The corresponding trajectory contained neither the paper’s full title nor its DOI. Across all solutions we reviewed, we found no evidence that any agent reconstructed a source paper’s workflow. This check addresses the concern about observable reconstruction in solutions. It cannot exclude subtler effects of prior exposure on scores, which we acknowledge in Section 6.1.

The pre- and post-cutoff groups differ in size and domain composition and may also differ in task difficulty. Reliable training cutoff dates are available for only a subset of models. These differences prevent us from separating the effects of training exposure from those of task characteristics, so we do not report the comparison.

Reply to Q1: Ranking sensitivity to randomness, prompts and harnesses, budgets, and the g>0.1g>0.1 threshold

Run variation.

On NatureBench-25, applying the main evaluation's ranking rule to the three-round means gives the same order for the four leading configurations as Table 2: Claude Opus 5, Claude Opus 4.7, Kimi K3, and GLM-5.2. The sample standard deviations of Match and Surpass rates range from 0 to 4.6 percentage points (Appendix L, Table 14).

ConfigurationRound 1Round 2Round 3Match (%)Surpass (%)
Claude Opus 517 / 516 / 715 / 664.0±4.064.0\pm4.024.0±4.024.0\pm4.0
Claude Opus 4.712 / 312 / 510 / 445.3±4.645.3\pm4.616.0±4.016.0\pm4.0
Kimi K39 / 311 / 410 / 540.0±4.040.0\pm4.016.0±4.016.0\pm4.0
GLM-5.28 / 39 / 39 / 334.7±2.334.7\pm2.312.0±0.012.0\pm0.0

Round columns report Match / Surpass counts out of 25. The final two columns give the mean ±\pm sample SD of each rate across the three rounds.

Prompts and harnesses.

Our comparison unit is the complete coding agent, as specified in Section 4.1. Separating model, prompt, and harness effects would require additional controlled comparisons. Given the cost of evaluation and our focus on complete coding agents, we prioritized evaluating a wider range of agent configurations, leaving the individual effects of the model, prompt, and harness outside the scope of this study. The rankings therefore apply to the tested configurations.

Compute budgets.

On the same 20 tasks, extending the time budget from 4 to 12 to 72 hours reduced timeouts from 19 to 13 to 6 out of 40 runs per budget. The combined Match-SOTA counts were 7, 6, and 7. Under the ranking rule used in the main evaluation, Claude Opus 4.7 led at 4 and 12 hours, and DeepSeek-V4-Pro led at 72 hours by one task. This comparison covers only the two tested configurations on these 20 tasks (Appendix I, Figure 9).

The Surpass threshold.

Surpass-SOTA counts tasks with relative improvements above 10% (g>0.1g>0.1), distinguishing them from smaller positive gaps (Section 3.4). Because agents differ in their distributions of task-level gains, changing the threshold can change their relative rankings by emphasizing different levels of improvement. We recomputed the rankings from the existing scores using g>0.05g>0.05, g>0.10g>0.10, and g>0.15g>0.15 as the Surpass thresholds and using Match-SOTA to break ties. Claude Opus 5 and Claude Opus 4.7 ranked first and second at all three thresholds, while the configurations in third and fourth place varied. We therefore complement Surpass-SOTA with Match-SOTA (g≥0g\geq0, Table 2), the full gap distributions (Figure 8a), and median gaps (Table 6 and Figure 8b), providing a broader view of both attainment frequency and performance relative to the published anchors.

Dear Reviewer ZzHa,

Thank you for taking the time to review our paper. Your constructive comments have helped us improve our work.

As the discussion deadline approaches, we would appreciate hearing whether the clarifications and additional analyses in our rebuttal address your main concerns. We would be happy to clarify any remaining points.

If our response has changed your assessment, we would be grateful if you could consider updating your rating.

Best regards,

The Authors

Dear Reviewer ZzHa,

Thank you again for your review. As the discussion period draws to a close, we wanted to check whether there are any remaining concerns we could address.

We have responded to the points raised in your review in our rebuttal and revised manuscript, adding source-recognition tests and trajectory reviews, examining ranking sensitivity to repeated runs, budgets, and thresholds, and clarifying the role of prompts and harnesses in the tested configurations.

If these responses have addressed your main concerns, we would be grateful if you would consider updating your rating. We remain happy to clarify any outstanding questions.

Best regards,

The Authors

Comments on the paper

4 comments

Thank you for taking the time to review our paper. We address your comments and questions below and have marked the manuscript revisions in blue for ease of reference.

Reply to W1 & Q1: Reproduction failures, task validity, and resource adequacy

There may be a misunderstanding about what the reproduction experiment measures and how to interpret its failure rate.

We understand the concern that reproduction failures could reflect defective tasks or insufficient resources. The reproduction scores alone cannot distinguish these explanations from difficulty implementing the paper. Our audit therefore uses the execution evidence to identify package defects (Section 3.2 and Appendix D).

We examined the 57 tasks for which neither audit agent met the reproduction criterion. In the main evaluation with the source methods withheld, at least one configuration obtained a valid score on 57/57 tasks, achieved Match-SOTA on 31/57, and achieved Surpass-SOTA on 15/57.

What the reproduction experiment tests.

As in PaperBench, agents receive the paper but must implement, debug, and train the method themselves. PaperBench's best 12-hour BasicAgent configuration achieved a mean rubric-based replication score of 21.0% (Section 5.2, Table 4), illustrating the difficulty of this process. Attributing a shortfall to a package defect therefore requires examining the implementation and run evidence alongside the score. We audit each attempt against the paper's preprocessing, model, training, and inference steps, using the trajectory and score to locate the source of the shortfall. Regardless of the score, we check that the task description, data, evaluator, and the published anchor agree.

Evidence for task validity.

We use several complementary checks to assess task quality and verify that each task can be attempted and scored as intended. Construction includes 36 package checks and environment verification (Section 2.4 and Appendix B.4). Across multiple stages of task construction, we extract the information needed to define each task, assemble its data and splits, and identify its evaluation anchor. We verify these details against the source paper and available author artifacts and correct any discrepancies we find. The evaluator is tested with perfect results and malformed inputs, and a simple baseline is run through the documented interface. We also build the environments and verify dependencies. Before finalizing the evaluation set, we use Claude Opus 4.6's base-mode runs to identify cases for inspection, then audit the packages in reproduction mode with Claude Opus 4.6 and DeepSeek-V4-Pro. We review agent trajectories alongside the scores and task packages to check for defects such as missing data, answer leakage, environment failures, and mismatches between the evaluator and the published anchor. We make targeted repairs where feasible and drop tasks with irreparable defects. The authors conduct manual reviews at each stage of task construction and benchmark calibration (Appendices B and D). These procedures support the executability and correctness of the retained tasks.

With the source methods withheld in the main evaluation (Section 4.1), at least one configuration obtained a valid score on every task (90/90), and at least one configuration matched the published anchor on 64/90 tasks.

Resource budget and time scaling.

The difficulty and cost of reproducing the published results limited our assessment to resource estimates. We use reported hardware and runtimes where available and otherwise estimate requirements from the data, training procedure, and task. These estimates guide the choice of CPU only or a single RTX 3090, RTX 4090, or A800 GPU, with a four-hour limit for every task (Section 4.1). The main evaluation does not require agents to reproduce the source methods. Agents may use any valid approach within the allotted time and compute, so we just assign each task a resource tier suited to its estimated needs. The main results should be interpreted under the resource conditions in Section 4.1.

To examine how much the time limit contributes, Appendix I compares DeepSeek-V4-Pro and Claude Opus 4.7 at 4, 12, and 72 hours on 20 tasks across six domains, including 16 with prior signs of budget constraints and four near-anchor controls. Timeouts fell from 19 to 13 to 6 out of 40 runs per budget, while Match-SOTA counts were 7, 6, and 7. At 72 hours, 22 of the 33 runs that fell short of Match-SOTA had method-layer failures. The longer budgets reduced timeouts and, for some tasks, exceeded the runtimes reported in the source papers. The overall Match-SOTA count remained roughly constant, suggesting that extending the time limit had little effect on aggregate Match-SOTA performance on these tasks. Choosing and implementing valid methods remained a major challenge.

Reply to W2 & Q2: Domain imbalance and small-sample comparisons

We agree that five tasks cannot establish a statistically reliable domain ranking. NatureBench's cross-disciplinary scope refers to coverage of six scientific domains, with task availability and verification determining the final composition (Sections 3.1–3.3). The benchmark was not designed to include the same number of tasks in each discipline.

The reported rates describe agent performance on the tasks included in each domain. We revised Section 5.2 to make clear that these results do not establish statistically significant differences between domains or a stable ordering across them.

Reply to W3 & Q3: Validating shortcut judgments and possible judge bias

We added cross-judge comparisons and author audits, holding the evaluated solutions and scores fixed (Appendix F.1). First, GPT-5.5 and Claude Opus 5 judged the NatureBench-25 runs of four solvers: Claude Opus 5, GPT-5.5, Kimi K3, and DeepSeek-V4-Pro. They agreed on 93/100 decisions. The authors reviewed all seven disagreements and agreed with GPT-5.5 on five and Claude Opus 5 on two. Applying the authors’ verdicts to the seven disputed runs changed only GPT-5.5’s results, increasing valid completions from 20 to 22, Match-SOTA from 7 to 9, and Surpass-SOTA from 1 to 2 out of 25 tasks. The ordering of the four configurations was unchanged.

For the GPT-5.5 solver, the authors reviewed all 40 runs marked invalid by the GPT-5.5 judge and all 27 runs on which the three judges disagreed. The two groups overlap by 16 runs, yielding 51 unique cases. The authors upheld 36 of the 40 original invalid verdicts and found 10 of the 11 additional cases valid and one invalid. The table reports each judge's invalid rate across all 90 tasks and agreement with the author audit on the 51 reviewed cases.

JudgeInvalid / 90AgreeDisagree
GPT-5.540/90 (44.4%)46 (90.2%)5
Claude Sonnet 542/90 (46.7%)34 (66.7%)17
Claude Opus 525/90 (27.8%)39 (76.5%)12

GPT-5.5 had the highest agreement with the authors on the audited cases. Claude Sonnet 5, from a different model family, also marked 42/90 runs invalid. These findings suggest that the high invalid rate largely reflects issues documented in the GPT-5.5 solver's runs.

Applying the authors' verdicts to the 51 audited runs while keeping all submissions, raw scores, and unaudited verdicts fixed raises GPT-5.5's valid completions from 49/90 to 52/90, Match-SOTA from 23/90 to 25/90, and Surpass-SOTA from 4/90 to 5/90. These changes are small in absolute terms, and GPT-5.5 would remain 14th out of 16 under the ranking rule in Table 2. To apply the same judging protocol to all agents, we retain the original GPT-5.5 results in the main table.

Dear Reviewer AVMX,

Thank you for taking the time to review our paper. Your constructive comments have helped us improve our work.

As the discussion deadline approaches, we would appreciate hearing whether the clarifications and additional analyses in our rebuttal address your main concerns. We would be happy to clarify any remaining points.

If our response has changed your assessment, we would be grateful if you could consider updating your rating.

Best regards,

The Authors

Dear Reviewer AVMX,

Thank you again for your review. As the discussion period draws to a close, we wanted to check whether there are any remaining concerns we could address.

We have responded to the points raised in your review in our rebuttal and revised manuscript, explaining the reproduction audit, providing further evidence on task validity and resource budgets, clarifying domain comparisons, and validating judge decisions through cross-model comparisons and author audits.

If these responses have addressed your main concerns, we would be grateful if you would consider updating your rating. We remain happy to clarify any outstanding questions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.