acceptodds
Under review as a conference paper at ICLR 2027

Beyond Retrieval: Component Relations and Feedback for Faithful Paper Reproduction

Abstract

Automated paper reproduction aims to implement methods from research papers and reproduce their reported results. Existing systems retrieve related papers and code to supplement missing implementation details. However, accurate retrieval does not ensure faithful reproduction. Our diagnostic study shows that agents can unnecessarily alter reusable components while failing to implement required changes. Moreover, feedback on overall performance provides limited guidance for identifying and correcting these errors. We therefore propose , which guides code generation through component relations and refines implementations using component-level experimental feedback. On an extended ReproduceBench covering 40 papers across ten tasks in three domains, achieves an average performance gap of 9.26%, reducing the gap by 25.46 percentage points over the strongest baseline. The code will be publicly released upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Automated Paper Reproduction, AI for Science

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_99QGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CompoRepro, an automated paper reproduction framework that explicitly identifies which components should be reused, modified, or newly implemented based on their relationships with prior work. Unlike existing approaches that primarily rely on overall performance feedback, CompoRepro further introduces component-level feedback to guide code refinement by comparing reproduced component effects with those reported in the target paper. The authors evaluate their framework on an extended benchmark covering 40 papers across ten tasks in three domains, achieving a lower performance gap than existing paper reproduction frameworks.

Critical strengths
  1. The paper evaluates the proposed framework across 40 papers and ten tasks in three domains, demonstrating consistent improvements over existing paper reproduction frameworks.

  2. Preserving reusable components while making only the necessary modifications is a promising direction for faithful paper reproduction, as it helps attribute performance differences to the intended methodological changes.

Critical weaknesses
  1. Insufficient benchmark validation and documentation.
    • The authors extend an existing benchmark and evaluate their framework on 40 papers. However, the quality and representativeness of these reproduction targets have not been sufficiently validated. While the proposed method outperforms other automated frameworks on this benchmark, it remains unclear how these results compare with actual human reproduction performance. If a large-scale human evaluation is too costly, the authors could consider a small subset of papers for comparison. Moreover, PaperBench is an established public benchmark for paper reproduction, but the authors neither evaluate on it nor explain why it was excluded.
    • The benchmark documentation appears incomplete: Appendix B lists only 36 entries, with some papers appearing in multiple tasks, while Sections B.3.1 and B.3.2 contain no paper lists.
Questions for authors
  1. Some typo issues: Figure 1 is not cited.
  2. The paper reports component relation distributions but does not quantify how frequently the proposed strategies lead to actual code modifications during reproduction. How many components are inspected, modified, or reimplemented, and what fraction of these modifications successfully correct implementation errors?
1 comment

Thank you for the constructive feedback.

W1

Our benchmark primarily includes papers from major conferences released after 2025 with publicly available code. This selection helps ensure that the reproduction targets are representative and executable, while reducing potential data contamination. To further assess benchmark difficulty, we invited two PhD researchers to independently reproduce six recommender-system papers under the same paper, code, and evaluation settings.

The average Performance Gap was 3.28% for human reproduction and 10.19% for CompoRepro. Human reproduction achieved a lower gap overall, showing that the benchmark distinguishes expert reproduction from automated agents while remaining challenging for automated systems.

MethodMMRecGeneralRecSeqRecAverage
PhD Researcher 16.4%3.8%2.1%4.10%
PhD Researcher 21.7%2.4%3.3%2.47%
Mean of human reproduction4.05%3.10%2.70%3.28%
CompoRepro18.23%8.41%3.94%10.19%

To evaluate the method on an external benchmark, we further evaluated CompoRepro on a three-paper subset of PaperBench under its standard evaluation protocol. CompoRepro achieved a Replication Score of 32.8%, outperforming Claude 3.5 Sonnet (23.0%) and o1 (26.6%), while remaining below PhD researchers (41.4%). These results show that CompoRepro maintains a consistent advantage on an independent benchmark, while a gap remains between automated and human reproduction.

MethodEvaluation subsetReplication Score
Claude 3.5 SonnetThree-paper subset23.0%
o1Three-paper subset26.6%
PhD researchersThree-paper subset41.4%
CompoRepro (Claude 3.5 Sonnet)Three-paper subset32.8%

Lower Performance Gap is better, while higher PaperBench Replication Score is better.

W2

We thank the reviewer for pointing out this documentation omission. The incomplete appendix entries were a writing oversight. We will complete Appendix B in the camera-ready version by listing all target papers and filling in the missing paper lists in Sections B.3.1 and B.3.2.

Q1

We thank the reviewer for pointing out these issues. We will add the missing citation to Figure 1 and correct the remaining typographical errors in the camera-ready version.

Q2

We further quantified the code changes guided by feedback. Across 64 feedback-guided component inspections, 51 existing components were modified and 6 components were reimplemented. The remaining 7 components were left unchanged because the available evidence was insufficient to support a reliable edit. Expert audit confirmed that 50 inspections targeted genuine implementation errors, of which 39 were correctly repaired in subsequent revisions, yielding a repair success rate of 78.0%.

Reviewer_tqJvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CompoRepro, an LLM-based framework for automated paper reproduction. It uses expert-defined component schemas to identify which parts of related implementations should be reused, modified, or newly implemented. It then reproduces the target paper’s ablation experiments and compares component-level performance changes with the reported effects to guide code inspection and refinement.

Critical strengths
  • The paper motivated the proposed approach through a diagnostic study of whether the coding agent correctly utilizes the related work, by assessing two properties, Reliability and Locality. The proposed approach CompRepro naturally follows from the identified failures.

  • Using the target paper’s own ablation results as component-level feedback provides a useful signal for locating implementation errors beyond overall performance. This is a clever idea. The reported gains are substantial and consistent across ten tasks and five runs. Expert evaluation results also support the observed gains.

Critical weaknesses
  • The novelty over DeepCode (Li et al., 2025) should be better elaborated, since component decomposition and reference-code mapping have already been proposed for DeepCode. The main difference seems to be the ablation-based feedback.

  • The paper argues for faithful reproduction. But the main evaluation metric used is performance gap. There seems to be a mismatch between the paper's claim and its evaluation results. It is better clearly define what is faithful reproduction, and conduct evaluations based on component-level fidelity.

Questions for authors
  1. What does the proposed relation analysis add beyond DeepCode’s component decomposition and CodeRAG mapping?

  2. The proposed approach largely relies on component-level feedback obtained from ablation studies from a target paper. What would be the results if the components do not have the ablation or comparison results?

  3. How to distinguish faithful implementation from performance matching? Would it be possible that an incorrectly implemented method achieve a low performance gap?

  4. The paper required expert-defined component schema. Essentially, human experts need to first design a task ontology to tell the model how to decompose an implementation problem. How sensitive are results to different expert schemas?

  5. Could a small-scale evaluation with newer models and Claude Code/Codex test whether the identified failures and the proposed gains remain under comparable budgets?

2 comments

Thank you for the constructive feedback. We have added the analyses and experiments described below to address these concerns.

W1 & Q1

Unlike DeepCode, our method introduces a task-specific schema during relation analysis. The schema defines a common set of functional components for representing the target paper, related implementations, and generated code, and assigns relations such as Reuse, Modify, and New to these aligned components. Although DeepCode also performs component decomposition, its decomposition is primarily prompt-driven and does not provide a shared set of component units for subsequent verification.

This distinction directly affects the feedback. Ablation results describe the effects of specific components, so the agent must first map each component to the corresponding unit in the generated code before constructing a control experiment and inspecting the implementation. Our schema provides this mapping. Without such alignment, component-level feedback cannot be reliably attributed to specific code, limiting the agent’s ability to localize implementation errors and perform subsequent repairs. Consistent with this analysis, the No Schema condition in Figure 6 represents the prompt-driven setting used by DeepCode. Its Performance Gap is 28.30% for recommendation and 29.20% for time series, whereas introducing the task-specific schema reduces these values to 10.19% and 9.92%, respectively.

W2 & Q3

Following the evaluation protocol of ReproduceBench [1], we distinguish performance matching from faithful reproduction. Performance Gap is an execution-level metric that measures the numerical difference between the results of the official and reproduced implementations. Faithful reproduction concerns whether the reproduced implementation follows the target paper's method, parameter configuration, and experimental setup. These objectives can differ. An implementation may achieve a small Performance Gap through a different architecture or additional tuning while violating the target design. Conversely, a faithful implementation may still show a performance difference because of random seeds, hardware, or software dependencies.

We already report both objectives separately in the paper. Table 1 reports the execution-based Performance Gap, while Table 2 reports expert evaluation scores and inter-rater agreement. Figure 5 further evaluates component-level reliability and locality. CompoRepro achieves an average Performance Gap of 9.26% in Table 1, an average expert evaluation score of 8.49, and an inter-rater agreement of κ=0.76\kappa=0.76 in Table 2. Thus, our evaluation does not use Performance Gap as the sole measure of faithful reproduction.

Q2

We further evaluate the setting in which the target paper does not report component-level ablation or comparison results. We remove the corresponding experimental results from the target paper. Without the reported component effects, the system cannot construct evidence-based control experiments for the relevant components and therefore uses overall performance feedback for subsequent refinement.

Under this setting, the Performance Gap increases from 10.19% to 20.39% for recommendation and from 9.92% to 25.46% for time series. This decrease is expected because component-level feedback relies on reported component effects as references. Without these references, the agent becomes less effective at locating and repairing implementation errors. These results show that ablation and comparison results are important information sources for component-level feedback and clarify the scope of our method. Automatically constructing reliable component-level feedback or finding alternative signals when such results are unavailable is an important direction for future work.

References

[1] Xuanle Zhao et al. AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage. ACL 2026. Paper · Code

Q4

As shown in Figure 6, schema design has a substantial effect on reproduction performance. To further evaluate this effect, we added three task-level schemas independently defined by different experts and compared them under the same tasks, models, and execution budgets. The three expert-defined schemas achieve average Performance Gaps of 10.06%, 11.31%, and 12.47% across recommendation and time-series tasks. All outperform the Generic Schema and No Schema settings, which achieve 23.65% and 28.75%, respectively.

These results show that the method benefits from experts' domain knowledge rather than from one particular schema. Although experts may differ in component names and granularity, their schemas identify the core functional components, component dependencies, and key design differences. This supports component alignment across papers, relation analysis, and component-level feedback. We therefore clarify in the paper that reasonable expert-defined schemas provide consistent gains, while their specific designs still affect the final performance.

Schema settingRecommendationTime SeriesAverage
No Schema28.30%29.20%28.75%
Generic Schema20.20%27.10%23.65%
Expert Schema A10.19%9.92%10.06%
Expert Schema B11.27%11.34%11.31%
Expert Schema C12.18%12.76%12.47%

Q5

We conducted an additional evaluation with Codex and GPT-6 Sol on three recommendation tasks. Even with these newer models, agents still omitted component modifications required by the target paper or unnecessarily changed components that should have been reused. Adding component relation analysis and component-level feedback reduced the Performance Gap from 21.8% to 9.1% for GPT-6 Sol and from 24.7% to 10.8% for Codex. It also reduced omitted target-component modifications and unnecessary changes to reusable components. These results suggest that the observed errors are not specific to a single backbone model and that our method provides consistent improvements.

ModelSettingPerformance GapOmitted target-component modificationsUnnecessary changes to reusable components
GPT-6 SolOverall performance feedback21.8%18%15%
GPT-6 SolCompoRepro9.1%8%6%
CodexOverall performance feedback24.7%23%19%
CodexCompoRepro10.8%10%8%
Reviewer_DHwAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes COMPOREPRO, it uses component relations to guide paper reproduction: it identifies which parts of a method should be reused, modified or newly implemented, then it uses component level experimental feedback to refine the generated code. The evaluation covers 40 papers across ten tasks in three domains, and reports an average performance gap of 9.26%. Expert assessments also examine whether the implementations follow the intended method.

Critical strengths
  1. The evaluation treats implementation fidelity as a distinct objective. The distinction between correctly implementing changed components and preserving reused components is useful. The oracle retrieval diagnostic supplies the relevant papers and code, allowing the study to examine what happens after retrieval succeeds. Three experts assess the generated components, while their gold relation labels are withheld from the agents. This provides direct evidence about implementation mistakes that an overall result alone could miss.

  2. The reported gains have support beyond a single performance number. The results are consistent across ten tasks and are reported over five independent runs, using a common model and execution environment. The expert evaluation gives the system a mean score of 8.49 with substantial inter rater agreement, and the component analyses show improvements in both reliability and locality.

Critical weaknesses
  1. The diagnostic value of component feedback is not yet well established. The method uses disagreement with a paper's reported ablation effect to decide which code to inspect. Such disagreement can arise from an implementation error, but also from training variability or interactions with other components. The paper acknowledges this and requires code inspection before revision. What is missing is an evaluation of how reliably that inspection turns the feedback into a correct repair.

  2. The feedback comparison does not isolate the benefit of its information. The comparison starts from the same implementation and uses five refinement rounds, but component level feedback consumes ~1.67 times the execution cost of overall feedback. Matching rounds thus leaves the amount of experimentation unequal. The results establish that the more expensive procedure works better under this protocol, they do not show how much improvement comes from the component specific information rather than the additional experimentation it permits.

  3. The method depends heavily on task specific expert decomposition. Expert defined schemas provide the component categories and functional descriptions on which both relation analysis and feedback depend. Fig 6 shows that these task specific schemas account for a substantial improvement over a generic schema. This makes their construction part of the method's central contribution, yet the evaluation does not establish how well a schema chosen independently of the target papers transfers to new papers or tasks.

Questions for authors
  1. What happens during feedback triggered revisions?

  2. How were the expert schemas separated from evaluation?

1 comment

Thank you for the constructive comments. To address your concerns, we have added an evaluation of the feedback mechanism, a cost-matched comparison of the two feedback strategies, and an analysis of the generalization of the expert-defined schema.

W1 & Q1. To assess whether component-level feedback leads to correct code repairs, we evaluate two aspects. First, we measure whether the feedback helps the agent locate implementation errors, using error localization accuracy. Second, we measure whether the localized code is correctly repaired, using repair success rate. Among 64 feedback guided code refinement audited, 50 were confirmed to real implementation errors, yielding an error localization accuracy of 78.1%. Of these, 39 were correctly repaired in subsequent refinement, yielding a repair success rate of 78.0%. These results show that component-level feedback helps the agent determine whether a specific component is correctly implemented, reducing the difficulty of locating faulty code and providing clear evidence for subsequent refinement.

W2. To compare the two feedback strategies under the same execution cost, we use overall-performance feedback as the baseline and increase its number of refinement rounds to match the total cost of component-level feedback. Since one component-level feedback round costs approximately 1.67 times one overall-feedback round, we use five rounds for component-level feedback and eight rounds for overall-performance feedback. Under the matched cost, component-level feedback achieves Performance Gaps of 16.2% and 18.1% on recommendation and time-series tasks, respectively, compared with 17.2% and 31.5% for overall-performance feedback. These results show that component-level feedback improves reproduction performance under the same execution cost.

W3 & Q2. To evaluate the generalization of expert-defined schemas, we consider two settings. The first tests transfer to new papers within the same task. The second tests extension to a new task within the same domain. For the first setting, we fix the Multimodal Recommendation schema and add two new papers, UniRec [1] and AdaM-Rec [2]. For the second setting, we add Multivariate Time-Series Imputation as a new task in the Time Series domain and evaluate T1 [3] and HELIX [4] using the Time-Series Forecasting schema. In both settings, we compare the expert-defined schema with a generic schema and no schema. The expert-defined schema achieves lower Performance Gap and higher relation Macro-F1 in both settings, with detailed results shown below. These results indicate that the expert-defined functional schema transfers to new papers within the same task, while the schema construction approach extends to new tasks within the same domain. For entirely new domains, such as LLM-based reinforcement learning, component categories, data interfaces, and experimental workflows may differ substantially from those in the current domains. Such settings may require a new schema based on domain knowledge. Automatically constructing schemas or reusing them across domains remains an important direction for future work.

Generalization to New Papers Within the Same Task

Entries report Performance Gap (%) ↓ / relation Macro-F1 (%) ↑.

PaperNo SchemaGeneric SchemaExpert-Defined Schema
UniRec31.8 / 52.423.6 / 63.112.8 / 75.4
AdaM-Rec34.7 / 50.826.1 / 61.814.2 / 73.9

Generalization to a New Task Within the Same Domain

Entries report Performance Gap (%) ↓ / relation Macro-F1 (%) ↑.

PaperNo SchemaGeneric SchemaExpert-Defined Schema
T136.4 / 48.628.3 / 60.116.9 / 72.6
HELIX39.1 / 46.830.5 / 58.918.4 / 71.0

References

[1] Zijie Lei et al. UniRec: Unified Multimodal Encoding for LLM-Based Recommendations. 2026. Paper · Code

[2] Honghao Fu et al. AdaM-Rec: Adaptive Modality Routing for Multimodal Recommendation. NeurIPS 2026. Paper · Code

[3] Dongik Park et al. T1: One-to-One Channel-Head Binding for Multivariate Time-Series Imputation. 2026. Paper · Code

[4] Fengming Zhang et al. HELIX: Hybrid Encoding with Learnable Identity and Cross-dimensional Synthesis for Time Series Imputation. ICML 2026. Paper · Code

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.