acceptodds
Under review as a conference paper at ICLR 2027

RECLAIM: Can Agents Reproduce the Claims of Machine Learning Papers?

Abstract

Reproducing a machine learning paper involves most research steps, from installing software and debugging to running experiments, work that AI agents increasingly do. We introduce RECLAIM, a benchmark of 100 NeurIPS 2025 papers that can be rebuilt yearly from new conferences. For each paper we fix in advance the result to reproduce, what counts as a successful reproduction, and a GPU-hour budget. An agent must reproduce that result using the paper and whatever its authors released. What the authors released decides the difficulty tier. Run-tier releases include code, data, and weights; Retrain-tier releases lack weights, so the agent trains the model; Reimplement-tier releases also lack code, so the agent writes it. A separate language model grades runs from logs and outputs rather than agents' reports. We run four agents once per paper; the best agent in each tier reproduces only 41% of Run-tier papers, 27% at Retrain, and 15% at Reimplement, where every agent does worst. Failed attempts use on average 29% of their budget, so most stop with budget left. The most common agent error is writing the method without checking any part against the paper's numbers, in 63 of 372 graded runs.

Primary area:
datasets and benchmarks
Keywords:
benchmark, LLM agents, paper reproduction, reproducibility, automated research, AI for science, agent evaluation, code generation, failure analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_YGTROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Reproducing Claims in Machine Learning (RECLAIM), which evaluates AI agents on whether they can reproduce released empirical results of NeurIPS 2025 papers. It is extendable to other papers. The work distinguishes between run-tier releases (code, data, weights released), retrain-tier releases (code, data), and reimplement-tier releases (data only) with different results for each, One of the findings suggest that a common failure mode is the agents rewriting codes without comparing results against those in the actual papers.

Critical strengths
  1. The task and evaluation design is sound, explained in detail and reusable.
Critical weaknesses
  1. No frontier models are tested and the agentic harness choice is questionable. The tested models are rather weak, so it is not obvious that any claimed findings/results would generalize. Moreover, the React harness is very minimal, especially for such long paper reproduction tasks. A better setup would be to use better agentic harnesses like ClaudeCode, Codex or miniSWE-agent, which would make the setup more realistic and relevant in practice. A researcher asking AI to reproduce AI/ML paper results, would ask a powerful coding agent, not set up a simple React harness to do so. The corresponding success/failure modes would change significantly if you used a better harness.
  2. Not thorough enough analysis of confounding factors, I suspect the analysis includes a number of confounding factors, e.g. it's not obvious how many of the failures are due to the hardware differences, rather than the fault of the agent. Agent's are also only run once so it's not clear whether some failures were transient or are actually consistent.
Questions for authors
  1. Can you repeat the analysis with stronger frontier models and proper coding harnesses?
  2. Given the network access, did you check whether the agents have "cheated" by e.g. fetching model checkpoints, rather than training from scratch?
  3. Have you checked how many failures were not actually the agent's fault?
3 comments
AuthorsOpenReview ↗

Response to Reviewer YGTR

Thank you for the detailed review and for recognising the benchmark design as sound and reusable. To test each confound you named, we ran three frontier models in two production harnesses.

W1 and Q1: Frontier models and production coding harnesses.

We agree that a benchmark is most useful when it is tested on the strongest models, in the harnesses people use them with. Our new runs show that RECLAIM remains challenging for frontier models. Over the 57 frontier runs, 18 are reproduced (32% [21%, 44%]), against 23 and 20 for DeepSeek-V4-Flash and Muse Spark 1.2 on the same papers (Section 5.2, Appendix B.3). The OpenAI models ran at high reasoning effort and Muse Wonder at xhigh.

Papers reproduced:

ModelHarnessPapersReproducedDeepSeek-V4-FlashMuse Spark 1.2
gpt-6.1-solour harness12544
gpt-6.1-solCodex CLI23487
gpt-6-astraCodex CLI8343
gpt-6-astraour harness6143
Muse Wondermini-SWE-agent6432
Muse Wonderour harness2101
All frontier runs57182320

How the papers were chosen. We preselected 24 papers from all three tiers, half that at least one of our four agents (the open agents below) had reproduced and half that none had, so that the two cases can be read separately.

  • On the 12 papers that an open agent had reproduced, the frontier runs reproduce 14 of 33, against 23 and 20 for DeepSeek-V4-Flash and Muse Spark 1.2.
  • On the 12 papers where all four open agents had failed, 4 of the 24 frontier runs succeed, and they reproduce 2 papers that no open agent had.

Runs where only the harness differs. gpt-6.1-sol finished the same 11 papers in both harnesses, with every other setting as in the main runs. It reproduces 4 in our harness and 3 in Codex CLI, with the same verdict on 8 of the 11, so the two harnesses differ by one reproduced paper.

Of the three harnesses you suggested, we ran Codex CLI and mini-SWE-agent. We did not evaluate Claude Code because our auditor is Claude Sonnet 5, and an auditor grading a model from its own family could be biased. Including it would require an independent auditor from another provider, validated against human grades as in Appendix D, which we leave to future work.

Whether the failure modes and findings change.

Two of our findings carry over to the frontier models, and the main failure mode changes. Run remains the easiest tier (11 of 18, against 3 of 20 at Retrain and 4 of 19 at Reimplement), and most of the GPU grant again goes unused. The main cause of failure becomes stopping early, and reporting a number without a valid measurement becomes less common. Retrain and Reimplement are close here because the sample holds equal numbers of reproduced and unreproduced papers in every tier, and DeepSeek-V4-Flash is level on these papers as well (6 of 20 and 6 of 19).

Interestingly, the frontier models stop even earlier than the open agents. Over the 57 runs, the median frontier run used 5.6% of its GPU grant, against 14.3% for the open agents on the same papers, and the frontier runs that did not reproduce used 2.1%. To test the effect further, we ran a variant that prompts Codex to continue, up to three times, whenever it stops before using half of its GPU grant. On 3 papers it spent more of its grant and received the same scores as the unprompted runs (8, 8 and 7).

To find the reason, we analysed the full transcripts of the 39 failing frontier runs (Appendix G.3). 27 of the 39 never ran the paper's experiment at full scale, and they decided this early, at a median of turn 9:

  • 12 stopped because the code, checkpoint or data they needed is not released, 8 of them at Reimplement, the tier where the agent writes that code.
  • 6 timed a short probe, projected a cost above the grant and stopped. On 2506.08898 (32 H100-hour grant), three runs projected 33 to 51 hours of retraining and stopped, while Muse Spark 1.2 and Muse Wonder in our harness retrained the model within the grant and reproduced the claim (Section 5.3).
  • 5 evaluated a different released checkpoint, and 4 hit build or launch errors.

The other 12 ran the experiment and missed the number. For 11 of the 27, an open agent given the same task text kept working and reproduced the paper, so these papers were within reach of the grant. Failures that report a number without a valid measurement fall to 8 of 39 = 21%, against 110 of 300 = 37% for the open agents, and 14 of 39 = 36% end before producing a number, against 44 of 300 = 15% (46% and 23% for the open agents when Claude Sonnet 5, the same annotator, labels a sample of 50 of their runs).

AuthorsOpenReview ↗

Response to Reviewer YGTR Part 2

W2: Failures due to hardware differences.

To separate failures caused by our aarch64 nodes from failures of the agent, we took every run that failed at build or dependency install and replayed its install commands on x86 nodes. There are 24 such runs, 14 of them on a package with no aarch64 build, and 12 of the 14 have an install command in their transcripts. On x86, 9 of the 12 build (75% [47%, 91%]) and 3 fail again. Our hardware therefore explains 9 failures, or at most 11 of the 372 graded runs if the 2 we could not replay are counted. With all 24 dropped from every denominator, DeepSeek-V4-Flash stays at 13/34 at Run and 9/33 at Retrain, and Muse Spark 1.2 goes from 5/33 to 5/32 at Reimplement, so no best-agent rate moves by more than one percentage point (Section 5.3, Appendix B.2). As a direct test we also ran gpt-6.1-sol and gpt-6-astra in our harness on x86 nodes. They reproduce 4 of 8 model and paper pairs there, against 3 of 8 for the same pairs on our GH200 nodes, and the two scores of every pair are within one point (Appendix B.3).

W3: Agents are only run once.

We agree that one run per cell leaves the results open to noise, so we measured how much a verdict changes when a run is repeated with the same budget.

  • Muse Spark 1.2, 10 Reimplement papers. A second run reaches the same reproduced verdict on 9 of the 10 (4 reproduced in both and 5 in neither). In 9 of the 10 pairs both runs measured the same quantity, and their two values differ by a median of 0.11%.
  • DeepSeek-V4-Flash, 9 papers from all three tiers. We graded 16 repeat runs, and on 7 of the 9 papers every repeat reaches the verdict of the run in the paper.
  • The 8 pairs of W2, run on x86 and GH200 nodes, are repeats as well, and 7 keep their verdict.

Together, the verdict is stable for 23 of 27 model and paper pairs (85% [68%, 94%]). In the four that change, every run scores 7, 8 or 9, on either side of the threshold of 8, so in these pairs a repeat moves a paper only when it already sits at the line. This is still a modest sample, and the revision reports every rate with its 95% Wilson interval (Section 5.2).

Q2: Fetching checkpoints over the network.

Thank you for suggesting this check, which changed two grades. We scanned the run transcripts for model downloads: hf_hub_download, snapshot_download, the Hugging Face CLI, from_pretrained with a repository id, torch.hub, git clone and git lfs of Hugging Face repositories, gdown, and wget or curl of weight files. Downloading released weights is part of the task at Run, so the runs that matter are the 42 that were graded reproduced at Retrain and Reimplement at submission (29 and 13). Of these, 16 matched a pattern, and we read them by hand: 4 were not downloads, 10 fetched only a base model or tokenizer that the method starts from, and 2 evaluated a ready checkpoint of the claimed model (Muse Spark 1.2 on 2508.21046 and MiniMax-M2.7 on 2505.12677). At these two tiers the run has to produce the model itself, so the revision counts both as not reproduced. That leaves 40 (28 and 12), and the best rates stay at 9 of 33 at Retrain and 5 of 33 at Reimplement (Section 5.2, Appendix B.2).

Q3: Failures that were not the agent's fault.

We traced 27 of the 400 cells to causes outside the agent (Sections 5.2 and 5.3, Appendix B.2):

  • 6 cells with no graded run. Four belong to paper 2510.20725, whose runs did not start because of a harness error that is now fixed, and 2 were Muse Spark 1.2 audits that never returned a verdict. The same auditor has now graded those 2, one reproduced and one partial.
  • 21 of the 24 build and dependency failures of W2. These are the 14 aarch64 packages and 7 other environment problems.

A run can also follow the released pipeline and still miss. To test whether such a miss comes from the agent or from the released code, we executed the final commands of 3 such runs again ourselves, with two new seeds and no agent:

  • 2505.17836 (Retrain). Both reruns miss the target by the same amount as the agent, and the three values agree within 0.0007, so the released code produces this miss.
  • 2510.07501 (Retrain). The agent's run and the two reruns range from -0.22 to +0.06 against a target of -0.20, so the agent's near miss is within seed noise.
  • 2506.10351 (Run). Against a target of 93.12%, both reruns (93.65% and 94.15%) land within 0.54 points of the agent's 94.19%.

In all three, the reruns are consistent with the agent's number. We read the misses on 2505.17836 and 2510.07501 as coming from outside the agent, and on 2506.10351 the partial grade came from a protocol deviation the agent disclosed. Only 3 of the 22 papers we examined had a single command chain that could be rerun unchanged, so we report them as cases and do not extrapolate.

We would be glad to discuss any of these results further. Thank you again.

AuthorsOpenReview ↗

Follow-up for Reviewer YGTR

Dear Reviewer YGTR, thank you again for your review. Our response above adds runs of three frontier models in two production harnesses, runs on x86 nodes, and a checkpoint check of every reproduced Retrain and Reimplement run, all of which are in the revised PDF. We would be glad to answer any remaining questions before the discussion closes.

Reviewer_rqBNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RECLAIM, a benchmark of 100 NeurIPS 2025 papers in which an autonomous agent must reproduce one pre-specified empirical claim using the paper and whatever artifacts its authors released.

Critical strengths

The methodological disclosure is excellent. The paper provides prompts, rubric text, model-serving configurations, task tables, compute estimates, run traces, and detailed vignettes. The distinction among faithful misses, unvalidated reimplementations, wrong experiments, stale results, and dependency failures is more actionable than a single binary leaderboard.

Critical weaknesses

Evaluator and evidence validity are also material concerns. Claude Sonnet 5 obtains F1 = 0.82 against one human label per run, but the paper omits the confusion matrix, precision, recall, tier-conditioned errors, multi-human adjudication, and validation of the full 0–10 scale and anti-cheat flags. More seriously, commands.log and trajectory.jsonl are writable by the evaluated agent without an integrity check. An instruction to distrust them does not replace an external append-only or tamper-evident event log.

Questions for authors
  1. Can the authors provide a stratified human-expert reproduction baseline across Run, Retrain, and Reimplement using the same environment and budgets?

  2. How were central claims, minimal configurations, target tuples, and tolerances independently validated, and what was the agreement among annotators?

2 comments
AuthorsOpenReview ↗

Response to Reviewer rqBN

Thank you for the thoughtful review and for your kind words about the methodological disclosure.

W1: Validation of the auditor.

We compared the auditor (Claude Sonnet 5) with human grades on 372 runs. Three human graders scored the runs, one grader per run, without seeing the auditor's score. On the reproduced verdict, the auditor's precision is 6473\frac{64}{73} = 88% and its recall is 6483\frac{64}{83} = 77% (F1 0.82). When the two disagree, the LLM auditor tends to be the stricter one. Of the 28 runs that only one of them marked as reproduced, 19 were from the human grader and 9 from the LLM auditor. On the full 0 to 10 scale, the two differ by at most one point on 324 of the 372 runs (quadratic weighted κ\kappa = 0.89). For the anti-cheat flags, where a high-severity flag sets a run's score to 0, 57 runs received a 0 from at least one grader, and both graders gave a 0 to 42 of them (full matrix in Appendix D).

W2: Agent-writable logs without an integrity check.

Our harness wrote commands.log and trajectory.jsonl into a folder that the sandbox mounted read-write, so we checked whether any agent had modified them. During each run, the harness also sends every tool call to a database outside the cluster. The sandbox holds no credentials for this database, and the database stamps each row with its own clock. Across the 372 graded runs, the 30,790 commands present in both the logs and the database have the same text, no exit code differs, and their timestamps agree within 9 seconds. The only entries found on one side are 221 calls that ended without an exit code (mostly because no GPU could be allocated) and 79 timed-out attempts. In one run the agent overwrote commands.log, which the database shows, and the auditor had already scored that run 1 out of 10. Still, to prevent this, the harness now mounts both files read-only inside the sandbox (Appendix A).

Q1: A stratified human-expert reproduction baseline.

We agree that a human-expert reproduction would be the gold-standard baseline. However, the cost of this would be very high. For example, in PaperBench, ML PhDs reached 41.4% of the replication rubric after 48 hours of work on a paper. At 48 hours per paper, our 100 papers would take about 4,800 expert-hours, which is more than we are able to provide. To still address this point, our revision reports what we can measure in its place:

  • Papers known to be reproducible. At least one of our four agents reproduced 32 of the 100 papers (95% interval 24% to 42%): 16 of 34 at Run, 11 of 33 at Retrain and 5 of 33 at Reimplement. These papers are therefore reproducible in the same environment and budget.
  • Expert targets. Each target is technically an expert result already, since it is the number that the paper's own authors reported from the same code and data.
  • Agent results rerun by us. To confirm that the numbers the agents report are what their code produces, we reran the final commands of 3 partial runs ourselves, with two new seeds and without the agent (Section 5.3). On two papers our reruns repeat the agent's number (on 2506.10351, 93.65% and 94.15% against the agent's 94.19%). On the third, 2510.07501, the result varies across seeds by more than the agent's original miss.

Q2: How claims, targets and tolerances were validated, and annotator agreement.

The revision adds an independent check of every target. We asked another model, gpt-6.1-sol, to compare each target value with its paper and to quote the passage behind each answer. It confirmed our value for 92 of the 100 claims (Appendix E), which is the agreement between our targets and an independent reader. A second check found the frozen number wrong on 8 targets, and when we correct them and rescore their runs, only two verdicts change, both on 2505.23747. Before the dataset was frozen, one author also reviewed every extracted field. For tolerances, 58 of the 62 point-estimate claims use the default of 5%, and the other 4 use the margin stated in their paper. When every run is rescored at 1%, 2%, 5% and 10%, Reimplement keeps the lowest reproduction rate: the best agent reproduces 4, 5, 5 and 5 of its 33 papers, against 9, 12, 14 and 14 of 34 at Run (Appendix F).

We are happy to answer any follow-up questions. Thank you again for the review.

Thanks for the detailed rebuttal. I maintain my score to support this paper.

Reviewer_T29uOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RECLAIM, a benchmark of 100 NeurIPS 2025 papers for evaluating whether autonomous agents can reproduce a paper’s central empirical claim from the artifacts its authors released. The authors evaluate four agents once per paper, yielding 400 agent-paper cells. The best agent reproduces 41% of Run-tier, 27% of Retrain-tier, and 15% of Reimplement-tier papers, with 73/400 reproductions overall. Most runs use far less compute than granted, and the failure analysis identifies unvalidated reimplementation, wrong-artifact measurement, and early stopping as major failure modes.

Critical strengths
  1. The benchmark targets a real and important bottleneck in autonomous research. Unlike many agent benchmarks that provide a prepared environment or a fixed verifier, RECLAIM asks agents to reproduce a published claim under realistic artifact-availability conditions. The tiering by released code, data, and weights is natural and directly measures a capability relevant to autonomous research agents.

  2. The evaluation and diagnostic design is unusually careful. Grading from execution evidence rather than agent self-reports, applying anti-cheat flags, freezing targets and rubrics before evaluation, and providing run-level failure analyses substantially strengthen the empirical contribution.

Critical weaknesses
  1. The reported rates are point estimates from a single run per agent-paper cell. With four agents, one venue/year, and one rollout per cell, run-to-run stochasticity and environment sensitivity could materially affect both agent rankings and tier-level comparisons.

  2. The central measurement relies heavily on an LLM auditor. The reported F1 of 0.82 against human graders is encouraging, but the paper would benefit from a more detailed audit of disagreements, especially near the reproduced/partial boundary and for high-severity integrity flags.

Questions for authors
  1. Can you provide a confusion matrix between the LLM auditor and human graders, especially for scores near the 7/8 boundary and for high-severity anti-cheat flags?
  2. How sensitive are the conclusions to the choice of match target and the default 5% tolerance when the paper does not state one?
2 comments
AuthorsOpenReview ↗

Response to Reviewer T29u

Thank you for your review, and for noting that the task is important and the evaluation design careful.

W1: One run per agent-paper cell.

We agree that one run per cell leaves our results susceptible to noise. To account for this, we now report every reproduction rate in the revised paper with its 95% interval, as in the table below. To measure the variation between runs, we graded a second, independent run of Muse Spark 1.2 on 10 Reimplement papers, made with the same budget as the run in the paper. The two runs reach the same reproduced verdict on 9 of the 10 papers (4 reproduced in both runs and 5 in neither; 95% interval 60% to 98%), and the tenth pair scored 8 and 7.

AgentRun (34 papers)Retrain (33)Reimplement (33)
DeepSeek-V4-Flash13 = 38% [24%, 55%]9 = 27% [15%, 44%]4 = 12% [5%, 27%]
Muse Spark 1.210 = 29% [17%, 46%]8 = 24% [13%, 41%]5 = 15% [7%, 31%]
Qwen3.6-27B5 = 15% [6%, 30%]6 = 18% [9%, 34%]2 = 6% [2%, 20%]
MiniMax-M2.72 = 6% [2%, 19%]5 = 15% [7%, 31%]1 = 3% [1%, 15%]
At least one agent16 = 47% [31%, 63%]11 = 33% [20%, 50%]5 = 15% [7%, 31%]

Comparing the tiers, we find that the reproduction rate at the Run tier is higher than at the Reimplement tier for DeepSeek-V4-Flash (38% against 12%, z = 2.46) and for papers reproduced by at least one agent (47% against 15%, z = 2.81). Every agent also has its lowest reproduction rate at the Reimplement tier. For the agent ranking, DeepSeek-V4-Flash and Muse Spark 1.2 have mean audit scores of 5.13 and 5.14 out of 10, and Qwen3.6-27B and MiniMax-M2.7 have 3.51 and 3.03. To investigate environment sensitivity, we removed the 24 runs that failed on a build or dependency error, and no best-agent reproduction rate moved by more than one percentage point (Section 5.3).

W2: The auditor's disagreements with the human grade.

We observe that when the human grader and the LLM auditor disagree, the auditor tends to be the stricter of the two. In the 28 runs where only one of the two marked the run as reproduced, it was the human grader in 19 and the auditor in 9, and the same direction holds in every tier. The auditor's reproduction rate is therefore slightly below the human's (73 reproduced runs against 83 at submission), and Reimplement has the lowest reproduction rate under both graders (Appendix D).

Q1: Confusion matrix, the 7/8 boundary and high-severity flags.

Against the human grade, the auditor's reproduced verdict has precision 6473\frac{64}{73} = 88% [78%, 93%] and recall 6483\frac{64}{83} = 77% [67%, 85%]. The confusion matrix below condenses the five-way matrix of Appendix D.

Human ↓ / Auditor →ReproducedPartialLowerTotal
Reproduced (8 to 10)6415483
Partial (6, 7)7641788
Lower (0 to 5)27192201
Total7386213372

At the 7/8 boundary, 100 runs received a 7 or an 8 from at least one grader, and the two graders put 75 of them on the same side of it. For high-severity flags, which set a run's score to 0, 57 runs received a 0 from at least one grader and both graders gave a 0 to 42 of them. None of the runs that the human scored 0 was marked as reproduced by the auditor.

Q2: How sensitive the results are to the match targets and the default 5% tolerance.

We rescored every run at tolerances of 1%, 2%, 5% and 10%, and Reimplement has the lowest reproduction rate at each of them. A run matches its target when its result is within a relative tolerance of the value the paper reports, which is 5% by default. The counts at submission are in the table below (Appendix F).

Relative tolerance1%2%5%10%
Best agent, Run (of 34)9121414
Best agent, Retrain (of 33)89912
Best agent, Reimplement (of 33)4555
All agents (of 400)56657382

To validate our match targets, we asked another model, gpt-6.1-sol, to compare each target value with its paper. It confirmed our value for 92 of the 100 claims (Appendix E). A second check found the frozen number wrong on 8 targets, and when we correct them and rescore their runs, only two verdicts change. Both are on one paper (2505.23747), and the best-agent count at Run goes from 14 to 13 of 34.

Please let us know if anything here needs more detail, and thank you again for your time.

AuthorsOpenReview ↗

Follow-up for Reviewer T29u

Dear Reviewer T29u, thank you again for your review. Our response above adds repeat runs for W1 and the grader confusion matrix and tolerance table for your two questions, all of which are in the revised PDF. We would be glad to answer any remaining questions before the discussion closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We thank the three reviewers for their careful and constructive reviews. They found the task important and the evaluation design unusually careful (T29u), the design sound and reusable (YGTR), and the methodological disclosure excellent (rqBN). Following the reviews, the revision adds these analyses:

  • Frontier models in production harnesses (Section 5.2, Appendix B.3). Pooled over 57 runs of gpt-6.1-sol, gpt-6-astra and Muse Wonder in our harness, Codex CLI and mini-SWE-agent, 18 are reproduced, where DeepSeek-V4-Flash and Muse Spark 1.2 reproduce 23 and 20 on the same papers, half of which were sampled because one of our four agents had reproduced them.
  • Uncertainty (Section 5.2). Every rate has a 95% Wilson interval, and repeated runs keep the reproduced verdict on 23 of 27 model and paper pairs, including 9 of the 10 Reimplement papers that Muse Spark 1.2 ran a second time.
  • Auditor validation (Appendix D). Against a human grade on 372 runs, the auditor's reproduced verdict has precision 88% and recall 77%.
  • Log integrity (Appendix A). During each run the harness sends every tool call to a database outside the cluster, which the agents have no credentials to access. The 30,790 commands present in both that record and the agent-writable logs have the same text, and no exit code differs. To be sure, the harness now also mounts both logs read-only inside the sandbox.
  • Targets and tolerance (Appendices E and F). We used gpt-6.1-sol to check every target value against its paper. It supports 92 of the 100, and we corrected the 8 targets whose number a second check found wrong. Reimplement stays the lowest tier at every tolerance from 1% to 10%.
  • Failures outside the agent (Sections 5.2 and 5.3). Removing every build failure moves no best-agent rate by more than one percentage point.

Grade changes since submission. Three changes take the reproduced cells from 73 to 70 of 400:

ChangeCellsReproduced cells
Corrected targets (Appendix E)DeepSeek-V4-Flash and MiniMax-M2.7 on 2505.23747-2
Checkpoint check (Section 5.2)Muse Spark 1.2 on 2508.21046, MiniMax-M2.7 on 2505.12677-2
Two audits that had not returnedMuse Spark 1.2 on 2503.23035 (reproduced) and 2510.08177 (partial)+1

The best agent then reproduces 13 of 34 Run, 9 of 33 Retrain and 5 of 33 Reimplement papers, the ordering of the tiers is unchanged, and papers reproduced by at least one agent go from 34 to 32.

We would be glad to answer further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.