- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CompoRepro, an automated paper reproduction framework that explicitly identifies which components should be reused, modified, or newly implemented based on their relationships with prior work. Unlike existing approaches that primarily rely on overall performance feedback, CompoRepro further introduces component-level feedback to guide code refinement by comparing reproduced component effects with those reported in the target paper. The authors evaluate their framework on an extended benchmark covering 40 papers across ten tasks in three domains, achieving a lower performance gap than existing paper reproduction frameworks.
Critical strengths
-
The paper evaluates the proposed framework across 40 papers and ten tasks in three domains, demonstrating consistent improvements over existing paper reproduction frameworks.
-
Preserving reusable components while making only the necessary modifications is a promising direction for faithful paper reproduction, as it helps attribute performance differences to the intended methodological changes.
Critical weaknesses
- Insufficient benchmark validation and documentation.
- The authors extend an existing benchmark and evaluate their framework on 40 papers. However, the quality and representativeness of these reproduction targets have not been sufficiently validated. While the proposed method outperforms other automated frameworks on this benchmark, it remains unclear how these results compare with actual human reproduction performance. If a large-scale human evaluation is too costly, the authors could consider a small subset of papers for comparison. Moreover, PaperBench is an established public benchmark for paper reproduction, but the authors neither evaluate on it nor explain why it was excluded.
- The benchmark documentation appears incomplete: Appendix B lists only 36 entries, with some papers appearing in multiple tasks, while Sections B.3.1 and B.3.2 contain no paper lists.
Questions for authors
- Some typo issues: Figure 1 is not cited.
- The paper reports component relation distributions but does not quantify how frequently the proposed strategies lead to actual code modifications during reproduction. How many components are inspected, modified, or reimplemented, and what fraction of these modifications successfully correct implementation errors?
Thank you for the constructive feedback.
W1
Our benchmark primarily includes papers from major conferences released after 2025 with publicly available code. This selection helps ensure that the reproduction targets are representative and executable, while reducing potential data contamination. To further assess benchmark difficulty, we invited two PhD researchers to independently reproduce six recommender-system papers under the same paper, code, and evaluation settings.
The average Performance Gap was 3.28% for human reproduction and 10.19% for CompoRepro. Human reproduction achieved a lower gap overall, showing that the benchmark distinguishes expert reproduction from automated agents while remaining challenging for automated systems.
To evaluate the method on an external benchmark, we further evaluated CompoRepro on a three-paper subset of PaperBench under its standard evaluation protocol. CompoRepro achieved a Replication Score of 32.8%, outperforming Claude 3.5 Sonnet (23.0%) and o1 (26.6%), while remaining below PhD researchers (41.4%). These results show that CompoRepro maintains a consistent advantage on an independent benchmark, while a gap remains between automated and human reproduction.
Lower Performance Gap is better, while higher PaperBench Replication Score is better.
W2
We thank the reviewer for pointing out this documentation omission. The incomplete appendix entries were a writing oversight. We will complete Appendix B in the camera-ready version by listing all target papers and filling in the missing paper lists in Sections B.3.1 and B.3.2.
Q1
We thank the reviewer for pointing out these issues. We will add the missing citation to Figure 1 and correct the remaining typographical errors in the camera-ready version.
Q2
We further quantified the code changes guided by feedback. Across 64 feedback-guided component inspections, 51 existing components were modified and 6 components were reimplemented. The remaining 7 components were left unchanged because the available evidence was insufficient to support a reliable edit. Expert audit confirmed that 50 inspections targeted genuine implementation errors, of which 39 were correctly repaired in subsequent revisions, yielding a repair success rate of 78.0%.