acceptodds
Under review as a conference paper at ICLR 2027

ReproBridge: Building Reproducibility into ML Research Artifacts

Abstract

Reproducing machine learning experiments typically requires reconstructing experimental procedures from information scattered across a paper, its released code, and its supporting artifacts. For coding agents, reproduction success can therefore depend on their ability to fill in missing details, rather than on the released artifacts alone. We ask whether the procedures and decisions recovered during reproduction can be preserved to improve subsequent independent reproduction. We define a standardized artifact design comprising a refined codebase, environment setup, experiment launchers, and a reproduction guide, and introduce ReproBridge, an agent workflow for constructing these refined reproducible artifacts. We also introduce ReproBridge-Bench, with 1,029 tasks from 387 experiments across 36 papers, anchored by five human-verified reference scores per task. Reproducing agents attempt the tasks in fresh sessions using either the raw codebase or the refined artifact; an Auditor separately checks the execution evidence. Across four evaluated models, refined artifacts increase mean reproduction success from 76.4% to 92.6%, with an average 19.7% reduction in downstream API spending. Component ablations examine the role of each part of the design. These findings suggest that preserving experimental procedures and their rationale can ease reproduction while lowering downstream API costs.

Primary area:
datasets and benchmarks
Keywords:
AI scientist, Responsible AI, Reproducibility

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JJ4WOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ReproBridge, which is an agentic framework for creating artifacts from papers and raw codebases that facilitate later efforts for reproducing the results of these papers. The resulting artifacts (REFINED REPRODUCIBLE ARTIFACT, RRA) comprise of a guide, refined code, environments and code launchers. To quantify the advantage offered by this framework, the authors also constructed ReproBridge-Bench, a benchmark comprising of 36 papers drawn from ICML 2026 spotlight papers that evaluates agents' ability to reproduce the results of these papers. On this benchmark, comparing agents that received only the paper and the raw codebase with agents that also received the RRAs generated by Opus 5 using the framework, the authors show that RRA both improves success rates and reduces API cost across four frontier LLMs.

Critical strengths
  1. The authors constructed a large and carefully verified reproduction benchmark, which upon release is useful beyond this paper.
  2. The evaluations are transparent and well designed. Success requires both reproducing results consistent with human-curated reference numbers and passing an evidence audit, which is validated against human labels and shows reasonable reliability.
Critical weaknesses
  1. The performance improvement claim and cost reduction claim are confounded with upstream work. The compute and cost of the upstream work constructing RRAs is not properly taken into account. From the performance perspective, the results fail to distinguish between "preserving recovered decisions and procedures helps" and "receiving completed prior work helps". From the cost perspective, adding the upstream API cost could also change the picture. Additionally, no GPU hours/cost is reported, which could outweigh the API costs.
Questions for authors
  1. Could the authors report the success rate and api costs for a scenario in which the reproducer receives the artifact construction instructions without the artifact?
  2. Could the authors report the GPU hours/cost associated with their experiments, both the reproduction runs and the artifact productions runs. With these numbers, could the authors provide a comprehensive cost comparison?
3 comments

We thank the reviewer for the constructive review and suggested controls. We have submitted a revised manuscript in which new and revised text is marked in blue.

[W1, Q1] Upstream work, performance gains, and the instructions-only control

Upstream work and performance gains. ReproBridge uses guide-aware reproduction: from the outset, the constructor is tasked with recovering procedures and preparing explicit guidance for a subsequent independent reproducer. During investigation, debugging, and execution, it records configuration choices, measurement rules, paper–code discrepancies, and the reasons for adopted solutions (Appendices B.1.1 and D.5).

Table 4 compares this process with ordinary-output reuse. The control passes Opus 5's raw-code execution directory—code, scripts, and logs with result files removed—to a fresh Sonnet 5 session. On the same 12 papers and 287 tasks, success is 73.9% with Raw, 81.2% with ordinary outputs, and 93.0% with RRA. The two handoffs come from separate upstream Opus 5 runs with comparable resource use. The additional 11.8 pp measures the benefit of guide-aware construction and its resulting handoff in this evaluation.

Table 4 also reports component removals from a single evaluation. For a more rigorous evaluation, we repeated the full RRA and all four component-removal conditions in two additional evaluations (Appendix D.5 (revised), Table 21). Removing any of the four components reduces success in all three evaluations, with mean paired losses of 3.2–5.7 pp, as shown in the table below.

ConditionEval. 1Eval. 2Eval. 3Success (%), mean ± SDLoss (pp), mean ± SD
Full RRA93.092.793.092.9 ± 0.2—
Without refined codebase87.886.487.587.2 ± 0.75.7 ± 0.6
Without environment setup88.588.989.288.9 ± 0.44.0 ± 0.4
Without reproduction guide89.289.588.289.0 ± 0.73.9 ± 0.8
Without experiment launchers89.989.289.989.7 ± 0.43.2 ± 0.2

Three evaluations with Sonnet 5 on the same 12 papers and 287 tasks. Loss is full RRA minus the removal condition within each evaluation. SD is the sample standard deviation across evaluations.

Construction instructions without the artifact. Thank you for suggesting this control. We added an instructions-only condition in which the reproducer receives the paper, raw repository, and construction instructions. The tables below compare its success rates, token usage, and downstream API expenditure with Raw and full RRA, reporting means and SDs over three evaluations. Instructions alone improve success over Raw by 1.3–4.5 pp, while providing the RRA adds a further 3.0–21.9 pp over the instructions-only condition.

For every model, the gain from instructions-only to RRA exceeds the gain from Raw to instructions-only. RRA also reduces token usage by 11.2%–32.8% and downstream API expenditure by 13.3%–43.7% relative to instructions-only. These results support the additional value of the paper-specific code, environment, launchers, and recovered decisions preserved in the artifacts.

Success (%)

ModelRawInstructions-onlyRRA
Sonnet 4.672.3 ± 1.176.8 ± 1.391.6 ± 3.4
Sonnet 576.5 ± 2.879.1 ± 1.993.3 ± 1.4
Opus 589.2 ± 1.291.7 ± 0.894.7 ± 2.2
Gemini 3.8 Flash67.6 ± 1.968.9 ± 2.290.8 ± 1.1

Token usage (B)

ModelRawInstructions-onlyRRA
Sonnet 4.68.48 ± 0.389.85 ± 0.556.80 ± 0.76
Sonnet 54.98 ± 0.556.73 ± 0.364.52 ± 0.24
Opus 53.01 ± 0.163.68 ± 0.322.82 ± 0.29
Gemini 3.8 Flash2.24 ± 0.262.06 ± 0.051.83 ± 0.09

Downstream API expenditure (USD)

ModelRawInstructions-onlyRRA
Sonnet 4.63,165 ± 1423,548 ± 2192,512 ± 286
Sonnet 51,726 ± 1751,866 ± 1871,317 ± 102
Opus 53,480 ± 1653,304 ± 3472,865 ± 312
Gemini 3.8 Flash392 ± 41579 ± 67326 ± 16

Entries are mean ± SD across three evaluations. Tokens are in billions (B). Raw and RRA values are from Table 2. Instructions-only provides the paper, raw code, and construction instructions; full RRA additionally provides the constructed artifact.

We have included this discussion and the results in Appendix D.10 (revised), Table 25.

[W1, Q2] Construction and reproduction costs

We have expanded the accounting to report construction and downstream GPU usage alongside API expenditure (Appendix E.2 (revised), Tables 27 and 28). Constructing the 36 RRAs used USD 5,026 in API calls and 989.68 GPU-hours. At Google Cloud's us-central1 A3 Mega on-demand list price of USD 93.40 per eight-GPU node-hour [1], the VM estimate is USD 11,554.60, giving an API-plus-VM construction subtotal of USD 16,580.60. This VM estimate covers GPUs, CPU, memory, and bundled Local SSD.

The tables below provide the per-model GPU-hours and VM estimates. Averaged over the four reproducer models, recorded GPU usage is 780.8 hours with Raw and 847.3 with RRA, corresponding to VM estimates of USD 9,115.41 and USD 9,892.10. For shortcut cases such as copied scores or extrapolation from partial runs, we estimate the omitted computation using human-verified H100 executions. The resulting reference-adjusted workloads average 1,149.6 and 898.5 GPU-hours, with VM estimates of USD 13,421.39 and USD 10,489.54, respectively.

ModelSetupSuccess (%)Audit-invalid / matchedRecorded GPU-hEstimated skipped GPU-hAdjusted GPU-h
Sonnet 4.6Raw72.163/805693.18428.341,121.52
Sonnet 4.6RRA91.823/968818.66119.09937.75
Sonnet 5Raw76.551/838773.95296.771,070.72
Sonnet 5RRA93.50/962914.700.00914.70
Opus 5Raw89.243/961939.81268.861,208.67
Opus 5RRA95.310/991818.6573.23891.88
Gemini 3.8 FlashRaw67.6173/869716.09481.301,197.39
Gemini 3.8 FlashRRA90.66/938837.1212.37849.49

Reproduction outcomes are from Tables 10 and 16. The audit column reports invalidated/numerically matched task outcomes. Adjusted GPU-hours = recorded usage + skipped computation estimated from human-verified H100 runs.

ModelRecorded usage: Raw (USD)Recorded usage: RRA (USD)Reference-adjusted: Raw (USD)Reference-adjusted: RRA (USD)
Sonnet 4.68,092.949,557.9313,093.8510,948.31
Sonnet 59,035.9410,679.2012,500.7510,679.20
Opus 510,972.379,557.8114,111.3310,412.78
Gemini 3.8 Flash8,360.419,773.4513,979.639,917.87

Downstream VM estimates use an A3 Mega reference price of USD 93.40 per eight-GPU node-hour. Reference-adjusted estimates use recorded GPU-hours plus skipped computation estimated from human-verified H100 runs.

Table 2 reports downstream API expenditure averaged over three evaluations, with model-wise reductions of 16.8%–23.7%. For repeated reuse, construction is charged once, followed by the API and VM usage of each attempt. Section 6.6's API accounting charges the full construction API cost to each model-specific reuse scenario, giving break-even at 8, 13, and 9 complete 36-paper rounds for Sonnet 4.6, Sonnet 5, and Opus 5, respectively.

[1] Google Cloud Compute Engine pricing, accelerator-optimized machine types (A3 Mega, us-central1, on-demand list price); accessed November 15, 2026.

Thank you for the additional experiments. The instructions-only control addresses my Q1, and the GPU accounting addresses the reporting part of Q2.

Two concerns remain. First, the comparison that separates preserved decisions from prior upstream work is the ordinary-outputs control, which is still a single evaluation on 12 papers with one reproducer. Repeating it, as was done for the ablations, and quantifying the upstream resources of both handoffs would strengthen the claim.

Second, the new tables show that GPU cost dominates API cost and that recorded GPU usage increases with RRA for three of four models, so total downstream cost rises for them. I suggest scoping the cost claim to API spending explicitly, reporting total cost per successful task, and stating the break-even with GPU included.

Overall I think this is a substantial work that will benefit the community. However, the unresolved confounds and the cost arguments are preventing me from further raising my score. Therefore I am maintaining my rating of 3.

Reviewer_n9mgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ML reproduction is dependent on many variables, such model capability and code discrepancies. To study the effects of different factors, the paper builds a four-component refined reproducible artifact from a paper and raw codebase: refined codebase, environment setup, experiment launchers, and a reproduction guide.

It has also introduced a benchmark (ReproBridge-Bench) across 1,029 tasks from 36 papers with five human-verified reference scores per task. The observation is that the reproduction failure is primarily an information problem, such as sufficient execution information, not a capability problem.

Critical strengths
  1. The problem is significant and the core-hypothesis that model capability or code dependency (similar to so-called harness) are important to the ML reproduction.

  2. The ablated four components are kinda of smart, by decomposing the blackbox into four stages and which stage can represent one particular information it required for reproduction

Critical weaknesses

It is unclear whether the results/observations are depending on what models are used to built the agentic framework? or like the model's particular skills. That's, is the paper identified the drawbacks of existing open-source Code repo or criticizing the code agent (which is responsible to build the agentic framework) for reproducing the code? More specifically, the code/paper repo itself can be well-equipment for a skilled programmer, but not for the code agent; or whether the code agent can itself add some "tricks" to improve the original code's drawbacks or inversely, ignore the debug information already provided.

The quality of the code itself, and the capability of the code agent are kinda entangled.

Questions for authors

See weakness.

2 comments

We thank the reviewer for highlighting the interaction between artifact quality and agent capability. [W1] Artifact quality, agent capability, and the construction model

Artifact quality and agent capability. Reproduction depends jointly on artifact quality and agent capability. Our comparison asks how artifact preparation changes the performance of a fixed reproducer: Raw and RRA use the same model, tasks, success criteria, and 24-hour per-paper budget, with fresh sessions in both conditions (Section 6.1). Five independently executed, human-verified reference observations per task establish the feasibility of the evaluated measurements (Section 4.2). Table 2 reports gains of 5.5–23.2 pp across the four tested reproducers.

Code repairs, environment recovery, and explicit procedural information are complementary parts of the intervention. The component ablations examine their contributions by removing one component while retaining the others. Across three evaluations with Sonnet 5 on 12 papers and 287 tasks (Appendix D.5 (revised), Table 21), each removal lowers success; mean paired losses are 5.7, 4.0, 3.9, and 3.2 pp for code, environment, guide, and launchers, respectively, as shown in the table below.

ConditionEval. 1Eval. 2Eval. 3Success (%), mean ± SDLoss (pp), mean ± SD
Full RRA93.092.793.092.9 ± 0.2—
Without refined codebase87.886.487.587.2 ± 0.75.7 ± 0.6
Without environment setup88.588.989.288.9 ± 0.44.0 ± 0.4
Without reproduction guide89.289.588.289.0 ± 0.73.9 ± 0.8
Without experiment launchers89.989.289.989.7 ± 0.43.2 ± 0.2

Three evaluations with Sonnet 5 on the same 12 papers and 287 tasks. Loss is full RRA minus the removal condition within each evaluation.

Dependence on the construction model. To test dependence on the constructor, we added a non-Claude construction experiment (Appendix D.9 (revised), Table 24). Gemini 4 Argon performs all construction roles to produce RRAs for all 36 papers. After the integrity audit, Sonnet 5 and Gemini 3.8 Flash succeed on 888/1,029 tasks (86.3%) and 871/1,029 tasks (84.6%), respectively, as shown in the table below. This evaluates a different constructor family and is reused by two reproducer families.

The workflow, role responsibilities, and delivery requirements are specified in advance (Section 5.2; Appendix B.1). Humans define these protocols; models carry out investigation, execution, review, and packaging within them. The main experiment uses Opus 5 for construction and evaluates the resulting artifacts with four reproducers (Table 2). The added Gemini 4 experiment applies the same artifact design with a different constructor.

ReproducerRaw successGemini 4 RRA success
Sonnet 5787/1,029 (76.5%)888/1,029 (86.3%)
Gemini 3.8 Flash696/1,029 (67.6%)871/1,029 (84.6%)

Raw success and success with Gemini 4 Argon-constructed RRAs on 36 papers and 1,029 tasks per reproducer, after the integrity audit. Entries give task counts and rates. Raw success values are from the success column of Table 10.

Both reproducers improve over Raw with Gemini 4-constructed RRAs. The benefit therefore does not depend on a particular construction model or its specific skills: the same pre-specified protocol and artifact design yield reusable artifacts with a different model family.

Programmer usability and agent behavior. We agree that an agent can overlook debugging information in a repository that is usable by a skilled programmer. This concerns its ability to locate and apply available information. ReproBridge makes both existing and recovered execution knowledge easier to use through task-specific guidance, documented pitfalls, and executable launchers (Section 5.1; Appendix B.1.1). The change record distinguishes execution fixes, correctness fixes, and convenience changes, so improvements in information accessibility and repairs to an implementation can be inspected separately.

For example, the released TinyImageNet validation loader in Effective Model Pruning labels all 10,000 images as one class. ReproBridge adds a script organizing images by their true labels and a launcher check for prepared data. In paired Gemini 3.8 Flash runs, ResNet-18 validation loss changes from 8.898 to 2.288, compared with the paper's 2.303 (Section 6.3; Appendix D.6). This is a concrete implementation issue that the artifact repairs and documents.

Agent-introduced changes must follow the paper's specified experiment and retain their rationale and execution evidence (Appendix B.1.1). The downstream Auditor traces submitted measurements to executed code and raw outputs, checking for copied values, altered outputs, and substituted computations. Its judgments are assessed against two-person human verification of 818 task–condition outcomes (Section 6.5; Appendix C).

Thanks for the author's response, especially adding different construction models for evaluation.

I keep the positive attitude towards the paper, would be happy to see if it can be accepted.

Reviewer_QP53OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a “refined reproducible artifact” (RRA)with repaired code, environment setup, launchers, and a reproduction guide. ReproBridge is an agent workflow that builds an RRA from a paper and its released code. ReproBridge-Bench has 36 random ICML 2026 Spotlight papers. Five human reproductions per task define a 95% interval, and 1,029 of 1,463 tasks are kept. Across four coding agents, RRAs raise success from 76.4% to 92.6% and cut API cost by 19.7%.

Critical strengths
  • Dedicated benchmark design. 5 human reproductions per task give a principled numerical reference. Papers are randomly sampled, and failure types are reported separately.
  • Integrity audit and useful cases. The Auditor checks execution evidence and is validated by humans. The case studies (e.g., a mislabeled validation loader) show practical value.
Critical weaknesses
  1. No code, data, reference scores, or artifacts are provided, especially the benchmark as a main contribution. The human reproducers are undocumented, who, how long it took, GPU cost. The sample of 36 papers is small and correlated, each task a main-table cell with the ten largest papers contributing 55% of tasks.
  2. The main comparison does not necessarily support the method's contribution.
  • Raw vs. RRA mixes two effects: the code has already been debugged and run once upstream, and the work is packaged in the proposed format. The first effect would help any later reproducer.
  • The fair control already reaches 81.2%. The format itself adds about 12 pp, measured on only 12 papers in a single run. The component ablations (3–5 pp) are within run-to-run noise.
  • The cost savings exclude construction, GPU, and audit costs, and break-even needs 8–13 full rounds. Author self-checking and human reuse, the stated uses, are not evaluated.
  1. Target leakage and hacking are not controlled. The constructor sees the published scores, and some guides state expected result magnitudes (App. C.1). Its code changes are checked only by an agent of the same family, so tuning toward the target may hide inside the artifact. Results depend on the auditor and model family. Opus 5 constructs the artifacts, and three of four reproducers are Claude models. The Claude Auditor flags 0 vs. 43 invalid Opus raw runs, and under it the Opus gain drops to 2.9 pp.
  2. The benchmark is limited in application and the task might be too easy. It covers only reproduction from released code with proved reproduction, not from scratch, and excludes papers with heavy compute or closed models. The strongest model already reaches 89.2% on raw code, so the benchmark is near saturation. A recent large-scale replication of ICML 2026 orals finds that only a small fraction of papers mostly hold up.
Questions for authors
  • report results with a non-Claude constructor, and with standard errors clustered by paper?
  • release benchmark
5 comments

We thank the reviewer for the careful review and constructive feedback. We have submitted a revised manuscript in which new and revised text is marked in blue.

[W1, Q1] Sample size, correlated tasks, and paper-cluster uncertainty

Sample size and task concentration. The 36 papers are retained from a random sample of 50 drawn from the 241 eligible ICML 2026 Spotlight papers (Section 4.1). They span eight topic families and 22 subdomains (Figure 2) and provide 1,029 tasks from 387 experiments, each anchored by five independent human-verified reference reproductions.

Appendix D.2 reports paper-wise comparisons (Figure 9) and equally weighted paper-macro success rates (Table 13), in which each paper contributes 1/36 regardless of its task count. The ten largest papers hold 561 of the 1,029 tasks (54.5%) but carry only 27.8% of the paper-macro estimate. As the table below shows, the paper-macro gains (8.4–19.6 pp) remain comparable to the task-level gains in Table 2 (5.5–23.2 pp), so the improvement is not driven by the largest papers.

Standard errors clustered by paper. We extend this analysis to all three evaluations and report paper-cluster uncertainty in Appendix D.2 (revised), Table 15. For each paper, we first average its RRA-minus-Raw success-rate gain across the three evaluations. The macro estimate then gives each of the 36 papers equal weight, regardless of its task count. Its SE is the sample standard deviation of these 36 paper-level mean gains divided by √36.

The table below reports the paper-macro success rates and mean gains with paper-cluster uncertainty. The 95% CIs are the mean gain ± 1.96 SE. The corresponding task-level (micro) results are reported in Table 2.

Paper-macro success and mean gain across three evaluations

ModelRaw (%)RRA (%)Mean gain (pp)SE (pp)95% CI (pp)
Sonnet 4.669.787.7+18.04.9[8.3, 27.7]
Sonnet 576.190.6+14.55.0[4.7, 24.4]
Opus 587.595.9+8.43.2[2.0, 14.7]
Gemini 3.867.286.8+19.65.3[9.2, 30.1]

For all four models, the paper-macro gains are positive and their 95% CIs exclude zero. The improvement from RRAs therefore holds when every paper counts equally and tasks from the same paper are treated as correlated.

[W1, Q2] Artifact release and human reference documentation

Benchmark and artifact release. We plan to release the benchmark and refined artifacts after acceptance. To make the method inspectable during review, we include representative prompt excerpts and execution-grounded qualitative cases below and in Appendices B.1.1, C.2, and D.6 (revised).

Constructor prompt (excerpt): “Record every fact with refactorkit as it happens, not at the end.” The guide-writing role is instructed: “You write from the record: every sentence you add is traceable to a file, and where the record is silent you write 'not recorded'.”

Reproducer prompt (excerpts): “Do not tune, do not re-run an experiment because the number 'looks wrong', and do not pick the run that came closest. Report what the code produced.” Result submission is constrained by: “Every number goes in with result.py link, which re-reads it out of the file itself. You cannot type a score anywhere.”

Qualitative case: WBMM. The Raw Sonnet 4.6 attempt could not install an H100-compatible mmcv-full stack. The artifact records the compatible dependencies, Ninja prerequisite, and source build for H100 kernels. The downstream session runs six COCO tasks without patching the paper's scientific code. Box AP values are 51.61 and 52.78 against 51.6 and 52.8; mask AP values are 44.78 and 45.64 against 44.8 and 45.6 (Appendix D.6). The prepared environment and launchers preserve the recovered dependency route.

Qualitative case: TinyImageNet validation in Effective Model Pruning. The released loader assigns all 10,000 validation images to one class. The artifact adds an annotation-based preparation script that verifies 200 classes and 10,000 images, writes a preparation receipt, and makes the launcher check that receipt before evaluation. In paired Gemini 3.8 Flash runs, validation loss changes from 8.898 to 2.288, compared with the paper's 2.303 (Appendix D.6). The repair and the preflight check make this data-preparation requirement explicit for later reuse.

Human reproducer details. Five ML PhD students or PhD holders performed the reference reproductions, with approximately 160 person-hours of human work (32 hours / person), excluding experiment runtime. The five reference collections used 5,208.6 GPU-hours. Section 4.2 and Appendix A.3 describe the independent execution and human verification protocol, and these details are added to Appendix A.3 (revised).

[W2] Guide-aware construction, component contributions, and costs

Upstream work and guide-aware construction. ReproBridge uses guide-aware reproduction: from the outset, the constructor is tasked with producing explicit guidance for a later, independent reproducer. While resolving dependencies, debugging, and running experiments, it records execution procedures, configuration choices, metric aggregation, paper–code discrepancies, and the reasons for adopted solutions. The four-component delivery requirements guide what information is recovered and preserved during construction (Appendices B.1.1 and D.5).

Table 4 compares this process with reuse of ordinary reproduction outputs. The control passes Opus 5's raw-code execution directory—code, scripts, and logs with result files removed—to a fresh Sonnet 5 session. On the same 12 papers and 287 tasks, success is 73.9% with Raw, 81.2% with ordinary outputs, and 93.0% with RRA. Both handoffs are produced in separate upstream Opus 5 runs with comparable resource use. The 11.8 pp gain over ordinary-output reuse measures the benefit of guide-aware construction and its resulting handoff in this evaluation.

Component contributions across three evaluations. We added two evaluations of full RRA and all four component-removal conditions on the same 12 papers and 287 tasks with Sonnet 5 (Appendix D.5 (revised), Table 21). The table below reports all three evaluations, their means and sample SDs, and the paired losses relative to full RRA. Removing each component reduces success in every evaluation. In particular, removing the environment while retaining the other components lowers success from 92.9 ± 0.2% to 88.9 ± 0.4%. The paired losses are 4.5, 3.8, and 3.8 pp (mean ± SD: 4.0 ± 0.4 pp), showing a consistent contribution across runs.

ConditionEval. 1Eval. 2Eval. 3Success (%), mean ± SDLoss (pp), mean ± SD
Full RRA93.092.793.092.9 ± 0.2—
Without refined codebase87.886.487.587.2 ± 0.75.7 ± 0.6
Without environment setup88.588.989.288.9 ± 0.44.0 ± 0.4
Without reproduction guide89.289.588.289.0 ± 0.73.9 ± 0.8
Without experiment launchers89.989.289.989.7 ± 0.43.2 ± 0.2

Three evaluations with Sonnet 5 on the same 12 papers and 287 tasks. Loss is full RRA minus the removal condition within each evaluation. SD is the sample standard deviation across evaluations.

Environment recovery also entails substantive work. In the WBMM environment-removal trace, the experiment code and commands remain available, but Sonnet 5 reconstructs a Python 3.10 / PyTorch 2.0 / CUDA 11.8 stack, rebuilds H100 operator kernels, and resolves installation-tool and filesystem issues without patching the paper's source (Appendix D.5). Preserving this executable dependency route avoids repeating the same environment recovery in each downstream attempt.

Construction and downstream costs. Section 6.6 reports USD 5,026 in API expenditure for constructing the 36 RRAs. Construction also used 989.68 GPU-hours on A3 Mega nodes. At Google Cloud's us-central1 on-demand list price of USD 93.40 per eight-GPU node-hour [1], this corresponds to an estimated VM cost of USD 11,554.60, covering GPUs, CPU, memory, and bundled Local SSD, for an API-plus-VM construction subtotal of USD 16,580.60.

The table below presents reproduction success, audit outcomes, and GPU usage together. For shortcut cases such as copied values or extrapolation from partial runs, we estimate the omitted computation using human-verified H100 setups. The table reports recorded usage and reference-adjusted computation estimates separately.

ModelSetupSuccess (%)Audit-invalid / matchedRecorded GPU-hEstimated skipped GPU-hAdjusted GPU-h
Sonnet 4.6Raw72.163/805693.18428.341,121.52
Sonnet 4.6RRA91.823/968818.66119.09937.75
Sonnet 5Raw76.551/838773.95296.771,070.72
Sonnet 5RRA93.50/962914.700.00914.70
Opus 5Raw89.243/961939.81268.861,208.67
Opus 5RRA95.310/991818.6573.23891.88
Gemini 3.8 FlashRaw67.6173/869716.09481.301,197.39
Gemini 3.8 FlashRRA90.66/938837.1212.37849.49

Reproduction outcomes are from Tables 10 and 16. The audit column reports invalidated/numerically matched task outcomes. Adjusted GPU-hours = recorded usage + skipped computation estimated from human-verified H100 runs.

The table below extends the downstream accounting with VM estimates for recorded GPU usage and reference-adjusted workloads.

ModelRecorded usage: Raw (USD)Recorded usage: RRA (USD)Reference-adjusted: Raw (USD)Reference-adjusted: RRA (USD)
Sonnet 4.68,092.949,557.9313,093.8510,948.31
Sonnet 59,035.9410,679.2012,500.7510,679.20
Opus 510,972.379,557.8114,111.3310,412.78
Gemini 3.8 Flash8,360.419,773.4513,979.639,917.87

Downstream VM estimates use an A3 Mega reference price of USD 93.40 per eight-GPU node-hour. Reference-adjusted estimates use recorded GPU-hours plus skipped computation estimated from human-verified H100 runs.

Averaged over the four reproducer models, reference-adjusted computation decreases from 1,149.6 to 898.5 GPU-hours (21.8%). The reported 19.7% saving and the 8–13 reuse-round break-even estimate concern API expenditure; each reuse round covers all 36 papers. The expanded accounting is included in Appendix E.2 (revised), Tables 27 and 28.

[1] Google Cloud Compute Engine pricing, accelerator-optimized machine types (A3 Mega, us-central1, on-demand list price); accessed November 15, 2026.

Author self-checking and human reuse. In the revised manuscript, we have moved author self-checking and human reuse from the Conclusion to the Discussion (Appendix G (revised)), presenting them as potential applications and directions for future evaluation.

[W3, Q1] Published targets, construction model, and Auditor dependence

Published targets and code modifications. The framework uses controls inside construction and reproduction, followed by a separate integrity audit, to reduce opportunities for target-directed editing and result substitution. Published scores remain visible as comparison targets, as in human reproduction; human-reference guidance and hidden evaluation fields are withheld from constructor and Reproducer sessions (Appendix C.1).

During construction, facts, choices, and patches are recorded as work proceeds. The internal Reviewer is instructed to compare the actual code diff with the patch journal and to examine whether choices recorded after a group's first run came from the paper/code or from the observed result. The Reproducer prompt prohibits tuning, rerunning because a score looks wrong, or selecting the closest run; submitted values are re-read from output files through result.py link. These controls are designed to reduce target-directed modifications and substitution before the independent Auditor inspects the execution evidence. The corresponding prompt excerpts are included in Appendices B.1.1 and C.2 (revised).

The downstream Auditor traces submitted scores to execution records, scripts, and raw outputs, checking for copied values, altered outputs, and substituted computations (Section 6.1; Appendix C.3). Two human reviewers cross-verified 818 task–condition outcomes from eight randomly selected papers, including outcomes not flagged by either Auditor. Against these human labels, the primary Codex Auditor achieves 88.6% precision and 79.5% recall (Figure 6; Table 11).

We agree that cross-family review of code changes during artifact construction is a useful extension and will study it in future work.

Non-Claude constructor and reproducers. The original experiments already include a non-Claude Reproducer, Gemini 3.8 Flash: success increases from 67.6% with Raw to 90.8% with RRAs constructed by Opus 5 (+23.2 pp; Table 2). The additional experiment below also tests Gemini 4 as the constructor.

We added a non-Claude construction experiment (Appendix D.9 (revised), Table 24): Gemini 4 Argon performs all construction roles to produce RRAs for all 36 papers, which are then evaluated with Sonnet 5 and Gemini 3.8 Flash. After the integrity audit, the two reproducers succeed on 888/1,029 tasks (86.3%) and 871/1,029 tasks (84.6%), respectively, as shown in the table below.

ReproducerRaw successGemini 4 RRA success
Sonnet 5787/1,029 (76.5%)888/1,029 (86.3%)
Gemini 3.8 Flash696/1,029 (67.6%)871/1,029 (84.6%)

Raw success and success with Gemini 4 Argon-constructed RRAs on 36 papers and 1,029 tasks per reproducer, after the integrity audit. Entries give task counts and rates. Raw success values are from the success column of Table 10.

These results show that a Gemini-family model can also construct reusable artifacts for both Claude and Gemini reproducers.

Auditor dependence. For Auditor sensitivity, Table 10 evaluates the same outcomes under Codex and Claude. RRA improves success under both Auditors for every reproducer; the Opus 5 gain is 6.1 pp under Codex and 2.9 pp under Claude. The primary Codex Auditor detects human-confirmed violations with 79.5% recall, versus 61.5% for Claude, with four false positives for each Auditor on 779 human-verified valid outcomes (Figure 6; Table 11).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.