- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Reproducing Claims in Machine Learning (RECLAIM), which evaluates AI agents on whether they can reproduce released empirical results of NeurIPS 2025 papers. It is extendable to other papers. The work distinguishes between run-tier releases (code, data, weights released), retrain-tier releases (code, data), and reimplement-tier releases (data only) with different results for each, One of the findings suggest that a common failure mode is the agents rewriting codes without comparing results against those in the actual papers.
Critical strengths
- The task and evaluation design is sound, explained in detail and reusable.
Critical weaknesses
- No frontier models are tested and the agentic harness choice is questionable. The tested models are rather weak, so it is not obvious that any claimed findings/results would generalize. Moreover, the React harness is very minimal, especially for such long paper reproduction tasks. A better setup would be to use better agentic harnesses like ClaudeCode, Codex or miniSWE-agent, which would make the setup more realistic and relevant in practice. A researcher asking AI to reproduce AI/ML paper results, would ask a powerful coding agent, not set up a simple React harness to do so. The corresponding success/failure modes would change significantly if you used a better harness.
- Not thorough enough analysis of confounding factors, I suspect the analysis includes a number of confounding factors, e.g. it's not obvious how many of the failures are due to the hardware differences, rather than the fault of the agent. Agent's are also only run once so it's not clear whether some failures were transient or are actually consistent.
Questions for authors
- Can you repeat the analysis with stronger frontier models and proper coding harnesses?
- Given the network access, did you check whether the agents have "cheated" by e.g. fetching model checkpoints, rather than training from scratch?
- Have you checked how many failures were not actually the agent's fault?
3 comments
Response to Reviewer YGTR Part 2
W2: Failures due to hardware differences.
To separate failures caused by our aarch64 nodes from failures of the agent, we took every run that failed at build or dependency install and replayed its install commands on x86 nodes. There are 24 such runs, 14 of them on a package with no aarch64 build, and 12 of the 14 have an install command in their transcripts. On x86, 9 of the 12 build (75% [47%, 91%]) and 3 fail again. Our hardware therefore explains 9 failures, or at most 11 of the 372 graded runs if the 2 we could not replay are counted. With all 24 dropped from every denominator, DeepSeek-V4-Flash stays at 13/34 at Run and 9/33 at Retrain, and Muse Spark 1.2 goes from 5/33 to 5/32 at Reimplement, so no best-agent rate moves by more than one percentage point (Section 5.3, Appendix B.2). As a direct test we also ran gpt-6.1-sol and gpt-6-astra in our harness on x86 nodes. They reproduce 4 of 8 model and paper pairs there, against 3 of 8 for the same pairs on our GH200 nodes, and the two scores of every pair are within one point (Appendix B.3).
W3: Agents are only run once.
We agree that one run per cell leaves the results open to noise, so we measured how much a verdict changes when a run is repeated with the same budget.
- Muse Spark 1.2, 10 Reimplement papers. A second run reaches the same reproduced verdict on 9 of the 10 (4 reproduced in both and 5 in neither). In 9 of the 10 pairs both runs measured the same quantity, and their two values differ by a median of 0.11%.
- DeepSeek-V4-Flash, 9 papers from all three tiers. We graded 16 repeat runs, and on 7 of the 9 papers every repeat reaches the verdict of the run in the paper.
- The 8 pairs of W2, run on x86 and GH200 nodes, are repeats as well, and 7 keep their verdict.
Together, the verdict is stable for 23 of 27 model and paper pairs (85% [68%, 94%]). In the four that change, every run scores 7, 8 or 9, on either side of the threshold of 8, so in these pairs a repeat moves a paper only when it already sits at the line. This is still a modest sample, and the revision reports every rate with its 95% Wilson interval (Section 5.2).
Q2: Fetching checkpoints over the network.
Thank you for suggesting this check, which changed two grades. We scanned the run transcripts for model downloads: hf_hub_download, snapshot_download, the Hugging Face CLI, from_pretrained with a repository id, torch.hub, git clone and git lfs of Hugging Face repositories, gdown, and wget or curl of weight files. Downloading released weights is part of the task at Run, so the runs that matter are the 42 that were graded reproduced at Retrain and Reimplement at submission (29 and 13). Of these, 16 matched a pattern, and we read them by hand: 4 were not downloads, 10 fetched only a base model or tokenizer that the method starts from, and 2 evaluated a ready checkpoint of the claimed model (Muse Spark 1.2 on 2508.21046 and MiniMax-M2.7 on 2505.12677). At these two tiers the run has to produce the model itself, so the revision counts both as not reproduced. That leaves 40 (28 and 12), and the best rates stay at 9 of 33 at Retrain and 5 of 33 at Reimplement (Section 5.2, Appendix B.2).
Q3: Failures that were not the agent's fault.
We traced 27 of the 400 cells to causes outside the agent (Sections 5.2 and 5.3, Appendix B.2):
- 6 cells with no graded run. Four belong to paper 2510.20725, whose runs did not start because of a harness error that is now fixed, and 2 were Muse Spark 1.2 audits that never returned a verdict. The same auditor has now graded those 2, one reproduced and one partial.
- 21 of the 24 build and dependency failures of W2. These are the 14 aarch64 packages and 7 other environment problems.
A run can also follow the released pipeline and still miss. To test whether such a miss comes from the agent or from the released code, we executed the final commands of 3 such runs again ourselves, with two new seeds and no agent:
- 2505.17836 (Retrain). Both reruns miss the target by the same amount as the agent, and the three values agree within 0.0007, so the released code produces this miss.
- 2510.07501 (Retrain). The agent's run and the two reruns range from -0.22 to +0.06 against a target of -0.20, so the agent's near miss is within seed noise.
- 2506.10351 (Run). Against a target of 93.12%, both reruns (93.65% and 94.15%) land within 0.54 points of the agent's 94.19%.
In all three, the reruns are consistent with the agent's number. We read the misses on 2505.17836 and 2510.07501 as coming from outside the agent, and on 2506.10351 the partial grade came from a protocol deviation the agent disclosed. Only 3 of the 22 papers we examined had a single command chain that could be rerun unchanged, so we report them as cases and do not extrapolate.
We would be glad to discuss any of these results further. Thank you again.
Follow-up for Reviewer YGTR
Dear Reviewer YGTR, thank you again for your review. Our response above adds runs of three frontier models in two production harnesses, runs on x86 nodes, and a checkpoint check of every reproduced Retrain and Reimplement run, all of which are in the revised PDF. We would be glad to answer any remaining questions before the discussion closes.
Response to Reviewer YGTR
Thank you for the detailed review and for recognising the benchmark design as sound and reusable. To test each confound you named, we ran three frontier models in two production harnesses.
W1 and Q1: Frontier models and production coding harnesses.
We agree that a benchmark is most useful when it is tested on the strongest models, in the harnesses people use them with. Our new runs show that RECLAIM remains challenging for frontier models. Over the 57 frontier runs, 18 are reproduced (32% [21%, 44%]), against 23 and 20 for DeepSeek-V4-Flash and Muse Spark 1.2 on the same papers (Section 5.2, Appendix B.3). The OpenAI models ran at high reasoning effort and Muse Wonder at xhigh.
Papers reproduced:
How the papers were chosen. We preselected 24 papers from all three tiers, half that at least one of our four agents (the open agents below) had reproduced and half that none had, so that the two cases can be read separately.
Runs where only the harness differs. gpt-6.1-sol finished the same 11 papers in both harnesses, with every other setting as in the main runs. It reproduces 4 in our harness and 3 in Codex CLI, with the same verdict on 8 of the 11, so the two harnesses differ by one reproduced paper.
Of the three harnesses you suggested, we ran Codex CLI and mini-SWE-agent. We did not evaluate Claude Code because our auditor is Claude Sonnet 5, and an auditor grading a model from its own family could be biased. Including it would require an independent auditor from another provider, validated against human grades as in Appendix D, which we leave to future work.
Whether the failure modes and findings change.
Two of our findings carry over to the frontier models, and the main failure mode changes. Run remains the easiest tier (11 of 18, against 3 of 20 at Retrain and 4 of 19 at Reimplement), and most of the GPU grant again goes unused. The main cause of failure becomes stopping early, and reporting a number without a valid measurement becomes less common. Retrain and Reimplement are close here because the sample holds equal numbers of reproduced and unreproduced papers in every tier, and DeepSeek-V4-Flash is level on these papers as well (6 of 20 and 6 of 19).
Interestingly, the frontier models stop even earlier than the open agents. Over the 57 runs, the median frontier run used 5.6% of its GPU grant, against 14.3% for the open agents on the same papers, and the frontier runs that did not reproduce used 2.1%. To test the effect further, we ran a variant that prompts Codex to continue, up to three times, whenever it stops before using half of its GPU grant. On 3 papers it spent more of its grant and received the same scores as the unprompted runs (8, 8 and 7).
To find the reason, we analysed the full transcripts of the 39 failing frontier runs (Appendix G.3). 27 of the 39 never ran the paper's experiment at full scale, and they decided this early, at a median of turn 9:
The other 12 ran the experiment and missed the number. For 11 of the 27, an open agent given the same task text kept working and reproduced the paper, so these papers were within reach of the grant. Failures that report a number without a valid measurement fall to 8 of 39 = 21%, against 110 of 300 = 37% for the open agents, and 14 of 39 = 36% end before producing a number, against 44 of 300 = 15% (46% and 23% for the open agents when Claude Sonnet 5, the same annotator, labels a sample of 50 of their runs).