- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ReproBridge, which is an agentic framework for creating artifacts from papers and raw codebases that facilitate later efforts for reproducing the results of these papers. The resulting artifacts (REFINED REPRODUCIBLE ARTIFACT, RRA) comprise of a guide, refined code, environments and code launchers. To quantify the advantage offered by this framework, the authors also constructed ReproBridge-Bench, a benchmark comprising of 36 papers drawn from ICML 2026 spotlight papers that evaluates agents' ability to reproduce the results of these papers. On this benchmark, comparing agents that received only the paper and the raw codebase with agents that also received the RRAs generated by Opus 5 using the framework, the authors show that RRA both improves success rates and reduces API cost across four frontier LLMs.
Critical strengths
- The authors constructed a large and carefully verified reproduction benchmark, which upon release is useful beyond this paper.
- The evaluations are transparent and well designed. Success requires both reproducing results consistent with human-curated reference numbers and passing an evidence audit, which is validated against human labels and shows reasonable reliability.
Critical weaknesses
- The performance improvement claim and cost reduction claim are confounded with upstream work. The compute and cost of the upstream work constructing RRAs is not properly taken into account. From the performance perspective, the results fail to distinguish between "preserving recovered decisions and procedures helps" and "receiving completed prior work helps". From the cost perspective, adding the upstream API cost could also change the picture. Additionally, no GPU hours/cost is reported, which could outweigh the API costs.
Questions for authors
- Could the authors report the success rate and api costs for a scenario in which the reproducer receives the artifact construction instructions without the artifact?
- Could the authors report the GPU hours/cost associated with their experiments, both the reproduction runs and the artifact productions runs. With these numbers, could the authors provide a comprehensive cost comparison?
3 comments
We have included this discussion and the results in Appendix D.10 (revised), Table 25.
[W1, Q2] Construction and reproduction costs
We have expanded the accounting to report construction and downstream GPU usage alongside API expenditure (Appendix E.2 (revised), Tables 27 and 28). Constructing the 36 RRAs used USD 5,026 in API calls and 989.68 GPU-hours. At Google Cloud's us-central1 A3 Mega on-demand list price of USD 93.40 per eight-GPU node-hour [1], the VM estimate is USD 11,554.60, giving an API-plus-VM construction subtotal of USD 16,580.60. This VM estimate covers GPUs, CPU, memory, and bundled Local SSD.
The tables below provide the per-model GPU-hours and VM estimates. Averaged over the four reproducer models, recorded GPU usage is 780.8 hours with Raw and 847.3 with RRA, corresponding to VM estimates of USD 9,115.41 and USD 9,892.10. For shortcut cases such as copied scores or extrapolation from partial runs, we estimate the omitted computation using human-verified H100 executions. The resulting reference-adjusted workloads average 1,149.6 and 898.5 GPU-hours, with VM estimates of USD 13,421.39 and USD 10,489.54, respectively.
| Model | Setup | Success (%) | Audit-invalid / matched | Recorded GPU-h | Estimated skipped GPU-h | Adjusted GPU-h |
|---|---|---|---|---|---|---|
| Sonnet 4.6 | Raw | 72.1 | 63/805 | 693.18 | 428.34 | 1,121.52 |
| Sonnet 4.6 | RRA | 91.8 | 23/968 | 818.66 | 119.09 | 937.75 |
| Sonnet 5 | Raw | 76.5 | 51/838 | 773.95 | 296.77 | 1,070.72 |
| Sonnet 5 | RRA | 93.5 | 0/962 | 914.70 | 0.00 | 914.70 |
| Opus 5 | Raw | 89.2 | 43/961 | 939.81 | 268.86 | 1,208.67 |
| Opus 5 | RRA | 95.3 | 10/991 | 818.65 | 73.23 | 891.88 |
| Gemini 3.8 Flash | Raw | 67.6 | 173/869 | 716.09 | 481.30 | 1,197.39 |
| Gemini 3.8 Flash | RRA | 90.6 | 6/938 | 837.12 | 12.37 | 849.49 |
Reproduction outcomes are from Tables 10 and 16. The audit column reports invalidated/numerically matched task outcomes. Adjusted GPU-hours = recorded usage + skipped computation estimated from human-verified H100 runs.
| Model | Recorded usage: Raw (USD) | Recorded usage: RRA (USD) | Reference-adjusted: Raw (USD) | Reference-adjusted: RRA (USD) |
|---|---|---|---|---|
| Sonnet 4.6 | 8,092.94 | 9,557.93 | 13,093.85 | 10,948.31 |
| Sonnet 5 | 9,035.94 | 10,679.20 | 12,500.75 | 10,679.20 |
| Opus 5 | 10,972.37 | 9,557.81 | 14,111.33 | 10,412.78 |
| Gemini 3.8 Flash | 8,360.41 | 9,773.45 | 13,979.63 | 9,917.87 |
Downstream VM estimates use an A3 Mega reference price of USD 93.40 per eight-GPU node-hour. Reference-adjusted estimates use recorded GPU-hours plus skipped computation estimated from human-verified H100 runs.
Table 2 reports downstream API expenditure averaged over three evaluations, with model-wise reductions of 16.8%–23.7%. For repeated reuse, construction is charged once, followed by the API and VM usage of each attempt. Section 6.6's API accounting charges the full construction API cost to each model-specific reuse scenario, giving break-even at 8, 13, and 9 complete 36-paper rounds for Sonnet 4.6, Sonnet 5, and Opus 5, respectively.
[1] Google Cloud Compute Engine pricing, accelerator-optimized machine types (A3 Mega, us-central1, on-demand list price); accessed November 15, 2026.
Thank you for the additional experiments. The instructions-only control addresses my Q1, and the GPU accounting addresses the reporting part of Q2.
Two concerns remain. First, the comparison that separates preserved decisions from prior upstream work is the ordinary-outputs control, which is still a single evaluation on 12 papers with one reproducer. Repeating it, as was done for the ablations, and quantifying the upstream resources of both handoffs would strengthen the claim.
Second, the new tables show that GPU cost dominates API cost and that recorded GPU usage increases with RRA for three of four models, so total downstream cost rises for them. I suggest scoping the cost claim to API spending explicitly, reporting total cost per successful task, and stating the break-even with GPU included.
Overall I think this is a substantial work that will benefit the community. However, the unresolved confounds and the cost arguments are preventing me from further raising my score. Therefore I am maintaining my rating of 3.
We thank the reviewer for the constructive review and suggested controls. We have submitted a revised manuscript in which new and revised text is marked in blue.
[W1, Q1] Upstream work, performance gains, and the instructions-only control
Upstream work and performance gains. ReproBridge uses guide-aware reproduction: from the outset, the constructor is tasked with recovering procedures and preparing explicit guidance for a subsequent independent reproducer. During investigation, debugging, and execution, it records configuration choices, measurement rules, paper–code discrepancies, and the reasons for adopted solutions (Appendices B.1.1 and D.5).
Table 4 compares this process with ordinary-output reuse. The control passes Opus 5's raw-code execution directory—code, scripts, and logs with result files removed—to a fresh Sonnet 5 session. On the same 12 papers and 287 tasks, success is 73.9% with Raw, 81.2% with ordinary outputs, and 93.0% with RRA. The two handoffs come from separate upstream Opus 5 runs with comparable resource use. The additional 11.8 pp measures the benefit of guide-aware construction and its resulting handoff in this evaluation.
Table 4 also reports component removals from a single evaluation. For a more rigorous evaluation, we repeated the full RRA and all four component-removal conditions in two additional evaluations (Appendix D.5 (revised), Table 21). Removing any of the four components reduces success in all three evaluations, with mean paired losses of 3.2–5.7 pp, as shown in the table below.
Three evaluations with Sonnet 5 on the same 12 papers and 287 tasks. Loss is full RRA minus the removal condition within each evaluation. SD is the sample standard deviation across evaluations.
Construction instructions without the artifact. Thank you for suggesting this control. We added an instructions-only condition in which the reproducer receives the paper, raw repository, and construction instructions. The tables below compare its success rates, token usage, and downstream API expenditure with Raw and full RRA, reporting means and SDs over three evaluations. Instructions alone improve success over Raw by 1.3–4.5 pp, while providing the RRA adds a further 3.0–21.9 pp over the instructions-only condition.
For every model, the gain from instructions-only to RRA exceeds the gain from Raw to instructions-only. RRA also reduces token usage by 11.2%–32.8% and downstream API expenditure by 13.3%–43.7% relative to instructions-only. These results support the additional value of the paper-specific code, environment, launchers, and recovered decisions preserved in the artifacts.
Success (%)
Token usage (B)
Downstream API expenditure (USD)
Entries are mean ± SD across three evaluations. Tokens are in billions (B). Raw and RRA values are from Table 2. Instructions-only provides the paper, raw code, and construction instructions; full RRA additionally provides the constructed artifact.