ReproBridge: Building Reproducibility into ML Research Artifacts
Abstract
Reproducing machine learning experiments typically requires reconstructing experimental procedures from information scattered across a paper, its released code, and its supporting artifacts. For coding agents, reproduction success can therefore depend on their ability to fill in missing details, rather than on the released artifacts alone. We ask whether the procedures and decisions recovered during reproduction can be preserved to improve subsequent independent reproduction. We define a standardized artifact design comprising a refined codebase, environment setup, experiment launchers, and a reproduction guide, and introduce ReproBridge, an agent workflow for constructing these refined reproducible artifacts. We also introduce ReproBridge-Bench, with 1,029 tasks from 387 experiments across 36 papers, anchored by five human-verified reference scores per task. Reproducing agents attempt the tasks in fresh sessions using either the raw codebase or the refined artifact; an Auditor separately checks the execution evidence. Across four evaluated models, refined artifacts increase mean reproduction success from 76.4% to 92.6%, with an average 19.7% reduction in downstream API spending. Component ablations examine the role of each part of the design. These findings suggest that preserving experimental procedures and their rationale can ease reproduction while lowering downstream API costs.
Then back it, or bet against it.
Related papers
Open the market on this paper to see 7 more related papers.