acceptodds
Withdrawn from ICLR 2027

PaperCompiler: Faithful Paper-to-Code Generation via Repository-Level Specification Compilation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Faithfully translating research papers into repository-level implementations remains challenging because papers often describe methods at a high level, leave implementation assumptions implicit, and require generated repositories to preserve method logic, evaluation protocols, and cross-file consistency. Despite recent advances in paper-to-code agents, their intermediate outputs are often presented as free-form plans or summaries that downstream coding agents may ignore, reinterpret, or compress, leading to algorithmic simplification and inconsistent repository structure. To address these challenges, we introduce PaperCompiler, a paper-to-code generation framework that compiles paper-grounded evidence into explicit repository-level implementation specifications. PaperCompiler grounds implementation-relevant evidence while preserving source provenance and distinguishing paper-supported, inferred, externally delegated, and unresolved information. The resulting specifications encode non-degradation requirements, ownership assignments, cross-file dependencies, and file-level constraints. Repository generation proceeds under these compiled specifications while retaining flexibility over local engineering choices not fixed by the paper. On 90 papers from Paper2CodeBench, PaperCompiler outperforms strong baselines, achieving a 13.8% relative improvement in reference-based fidelity (3.644.15) and reducing high-severity evaluator critiques from 13.2% to 6.1%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Paper-to-Code Generation, Scientific Reproducibility, Code Generation, Repository Synthesis, AI for Science

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bkrwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PaperCompiler, a framework for generating code repositories from research papers. The main idea is to compile paper-derived information into structured implementation specifications, including explicit requirements, non-degradation constraints, file-level ownership, and cross-file dependencies. The framework consists of three stages: Paper Grounding, Specification Compilation, and Constraint-Guided Repository Generation.

Experiments on 90 papers from Paper2CodeBench show improvements over existing baselines, including a 13.8% relative improvement in reference-based fidelity over PaperCoder. Ablation studies suggest that requirement reconciliation and file-level contracting contribute most to the improvements.

Critical strengths

Consistent empirical improvements. The method achieves improvements over PaperCoder across three evaluation protocols on 90 papers. The ablation study also provides some evidence that requirement reconciliation and file-level contracting are useful components.

Critical weaknesses
  1. Limited evidence for the effectiveness and generalizability of the proposed workflow. The experiments only use o3-mini for code generation. The proposed workflow mainly decomposes the task into smaller steps and introduces explicit specifications, which may primarily compensate for the limited capabilities of a weaker LLM. Moreover, generating accurate specifications is itself challenging, especially identifying forbidden algorithmic substitutions. The paper does not evaluate the correctness of these intermediate specifications or establish whether the method remains beneficial for stronger LLMs. Comparisons with execution-guided reflection and iterative refinement methods are also insufficient.

  2. Evaluation does not adequately reflect actual implementation correctness. All three primary evaluation protocols rely on LLM-based correctness rubrics rather than actual execution or reproduction results. Although reference-based evaluation provides additional information, it still requires an LLM to judge complex implementation differences. The reported improvements are relatively modest in absolute score, and it is unclear whether they translate into meaningful improvements in executable code quality. Without functional tests or experimental reproduction results, the practical effectiveness of the proposed method remains insufficiently demonstrated.

Questions for authors
  1. Generalization to stronger LLMs: Does PaperCompiler still provide meaningful improvements when using stronger code-generation models? Can the authors provide experiments with a stronger backbone to distinguish the benefits of specification compilation from those of task decomposition for weaker models?

  2. Reliability of intermediate specifications: How accurate and complete are the LLM-generated requirements, particularly the forbid constraints? Have the authors examined cases where incorrect or incomplete specifications introduce errors into the generated implementation?

  3. Evaluation protocol and comparison with reflection-based methods: I believe a more reliable evaluation protocol is needed to assess the actual correctness of generated implementations. The current LLM-based rubric scores may not adequately reflect whether the generated code correctly implements the proposed algorithms or reproduces the reported experimental results. Can the authors provide more objective evaluations, such as execution success rates, functional correctness tests, and experimental reproduction accuracy? In addition, how does PaperCompiler compare with execution-guided reflection and iterative refinement methods under a comparable computational budget?

Reviewer_yN1MOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to translate papers into repository-level implementations while preserving methodological requirements and consistency. The authors propose PaperCompiler, which combines Paper Grounding, Specification Compilation, and Constraint-Guided Repository Generation to associate paper-derived requirements with supporting evidence, file-level responsibilities, interfaces, and artifact dependencies. On 90 Paper2CodeBench papers, PaperCompiler improves over PaperCoder across three evaluation protocols, with the largest relative gain of 13.8% in reference-based fidelity.

Critical strengths
  1. The comparison covers both general software-development agents and specialized paper-to-code systems. Including ChatDEV and MetaGPT alongside PaperCoder, AutoP2C, and AutoReproduce provides useful coverage of different workflows and information sources.

  2. The failure-category analysis and ablations is informative. It goes beyond aggregate scores to distinguish algorithmic degradation, missing components, evaluation mismatches, and interface problems, helping identify where improvements and remaining difficulties occur. The ablations examine Reconciliation, Context Slicing, and Contracting, while concrete examples, such as omitted decoder operations in INTR and the unimplemented multimodal inference path in VDC, help illustrate the failures associated with removing components.

Critical weaknesses
  1. The necessity of the Blueprint Construction–Specification Compilation workflow is not clearly supported. The nine-paper ablation removes Reconciliation, Context Slicing, or Contracting, but does not test whether separately constructing the implementation blueprint and implementation specifications is necessary. It also provides limited evidence about how non-degradation requirements, ownership assignments, and cross-file dependencies interact to produce the improvement over PaperCoder.

  2. The correctness of the implementation blueprint and compiled specifications, and their consistency with generated code, are insufficiently evaluated. LLM-based Reconciliation can correct blueprint errors but may also turn correct statements into incorrect contracts. The paper does not evaluate or shows any strategy to limit these potential error corrections, newly introduced errors, or subsequent propagation into code. Their generation-time prompts do not establish reliable compliance. Also, Table 3 reports increased API/schema mismatch critiques and external-protocol hallucination critiques while not sufficiently explaining their connection to the design.

  3. The results do not establish generalization beyond the tested o3-mini and stage-prompt configuration. Other LLM families, more advanced LLMs, and open-source LLMs are not evaluated, leaving unclear whether the benefits persist beyond this particular model–prompt–workflow combination. Repository architectural granularity is also guided by loose qualitative instructions such as coherent responsibilities and reasonable file sizes, without evaluating how decomposition and design choices affect the generated repositories.

Questions for authors
  1. What evidence shows that Reconciliation improves the implementation blueprint and that generated code follows the resulting specifications? Since the mechanism relies on generation-time prompts, providing experimental evidence is important for assessing whether the workflow reduces error propagation and specification–code inconsistency.

  2. Can any experiments establish whether the gains persist beyond o3-mini? Please consider representative backbones from other LLM families, including more advanced or open-source models, and reasonable variations of the stage prompts. These experiments would help establish whether the improvements reflect the proposed workflow rather than the particular model and prompt configuration.

  3. Why should Reference-based evaluation take precedence when Reference-free and P2C-Ex favor removing Context Slicing or Contracting? The conflicting Reference-free, P2C-Ex, and Reference-based ablation results are not sufficiently explained. For example, removing Context Slicing improves Reference-free scores from 4.57 to 4.72 and P2C-Ex from 4.42 to 4.74, while reducing Reference-based scores from 4.38 to 4.11, and removing Contracting produces the same directional disagreement. The explanation that reference-free evaluation rewards superficial completeness needs stronger evidence before these results can substantiate the benefits of retaining both components. Also, for the Table 3 diagnostics, practical label definitions and examples, can the authors further explain about the increased API/schema mismatch and external-protocol hallucination critiques, would make the interpretation clearer.

  4. Higher token cost and computational effort remain confounding factors in the PaperCoder comparison. PaperCompiler uses approximately 1.71M tokens per repository, compared with 0.98M for PaperCoder. AutoP2C’s comparable 1.68M-token usage does not establish that PaperCompiler’s advantage over PaperCoder is attributable to specification compilation rather than the additional computational effort.

Reviewer_x1PROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PaperCompiler, a framework for generating code repositories from machine learning papers with an emphasis on fidelity to the original methods. The authors identify information loss and algorithmic degradation as limitations of existing multi-stage workflows. PaperCompiler derives implementation specifications that record evidence provenance, file ownership, cross-file dependencies, and prohibited simplifications, and uses these specifications to guide repository generation.

Critical strengths
  1. The paper addresses a practical problem: preserving paper-specific implementation requirements throughout a multi-stage generation process. Its focus on faithfulness is valuable beyond assessing the apparent completeness of generated repositories.
  2. The specifications explicitly connect paper-derived requirements to concrete files and interfaces. Recording evidence provenance, requirement ownership, and producer-consumer relationships also provides a useful basis for tracing requirements through the generation process and diagnosing information loss.
Critical weaknesses
  1. Effect of additional reasoning budget. PaperCompiler introduces additional specification-reasoning stages. Section 4.2 reports average token usage of 1.71M per repository for PaperCompiler and 0.98M for PaperCoder on the ten-paper subset. The comparison therefore leaves open how much of the improvement could be obtained by allocating a similar budget to a simpler generation or refinement workflow. Likewise, removing Contracting in Table 4 removes a reasoning step, and token usage for the ablation variants is not reported. Comparisons with AutoP2C and AutoReproduce are informative, but differences in their other mechanisms limit their ability to resolve this question. A budget-matched PaperCoder baseline with additional refinement rounds would help establish the benefit of the proposed workflow at comparable computational cost.
  2. Effect of representation versus requirement content. Reconciliation organizes extracted information into a structured specification while also making behavioral obligations, boundary conditions, and prohibited substitutions explicit. The w/o Reconciliation variant removes non-degradation and method-level requirements, rather than retaining the same requirements in a different representation. This ablation therefore supports the usefulness of the resulting guidance, but does not isolate the contribution of its structured form. A content-matched comparison, using the same evidence and implementation requirements expressed as either free-form or structured specifications, would clarify whether explicit structure provides benefits beyond more detailed implementation guidance.
  3. Limited execution-based validation. Faithfulness is assessed primarily through LLM judges. The authors’ SEABO analysis in Appendix D.4 illustrates a gap between preserving requirements in intermediate specifications and realizing them in the generated code: the runner falls back to a dummy dataset when no real D4RL path is provided, and the offline-RL trainer raises NotImplementedError for state_only=True. The authors appropriately acknowledge the broader validation limitation in Appendix A. Higher judge scores provide evidence of improved model-assessed fidelity, but targeted execution tests or automated checks of core components would help determine how consistently the specified requirements are actually implemented.
Questions for authors
  1. Could the authors compare PaperCompiler with a PaperCoder variant that adds only generation or refinement rounds under a comparable token or compute budget? Could they also report token usage for each ablation variant?
  2. Could the authors compare free-form and structured specifications containing the same implementation requirements, while keeping the downstream generator and generation budget comparable?
  3. Section 3.2 and Appendix E describe the grounding procedure and prompt schema. How is the grounding of the resulting behavioral constraints, boundary conditions, and prohibited substitutions validated? Please also clarify whether Paper2CodeBench evaluation rubrics, evaluator outputs, or scoring feedback were used when developing or tuning these prompts.
  4. Could the authors provide targeted execution tests or automated checks on a representative subset of generated repositories to assess whether method-critical contracts are satisfied?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.