- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PaperCompiler, a framework for generating code repositories from research papers. The main idea is to compile paper-derived information into structured implementation specifications, including explicit requirements, non-degradation constraints, file-level ownership, and cross-file dependencies. The framework consists of three stages: Paper Grounding, Specification Compilation, and Constraint-Guided Repository Generation.
Experiments on 90 papers from Paper2CodeBench show improvements over existing baselines, including a 13.8% relative improvement in reference-based fidelity over PaperCoder. Ablation studies suggest that requirement reconciliation and file-level contracting contribute most to the improvements.
Critical strengths
Consistent empirical improvements. The method achieves improvements over PaperCoder across three evaluation protocols on 90 papers. The ablation study also provides some evidence that requirement reconciliation and file-level contracting are useful components.
Critical weaknesses
-
Limited evidence for the effectiveness and generalizability of the proposed workflow. The experiments only use o3-mini for code generation. The proposed workflow mainly decomposes the task into smaller steps and introduces explicit specifications, which may primarily compensate for the limited capabilities of a weaker LLM. Moreover, generating accurate specifications is itself challenging, especially identifying forbidden algorithmic substitutions. The paper does not evaluate the correctness of these intermediate specifications or establish whether the method remains beneficial for stronger LLMs. Comparisons with execution-guided reflection and iterative refinement methods are also insufficient.
-
Evaluation does not adequately reflect actual implementation correctness. All three primary evaluation protocols rely on LLM-based correctness rubrics rather than actual execution or reproduction results. Although reference-based evaluation provides additional information, it still requires an LLM to judge complex implementation differences. The reported improvements are relatively modest in absolute score, and it is unclear whether they translate into meaningful improvements in executable code quality. Without functional tests or experimental reproduction results, the practical effectiveness of the proposed method remains insufficiently demonstrated.
Questions for authors
-
Generalization to stronger LLMs: Does PaperCompiler still provide meaningful improvements when using stronger code-generation models? Can the authors provide experiments with a stronger backbone to distinguish the benefits of specification compilation from those of task decomposition for weaker models?
-
Reliability of intermediate specifications: How accurate and complete are the LLM-generated requirements, particularly the
forbidconstraints? Have the authors examined cases where incorrect or incomplete specifications introduce errors into the generated implementation? -
Evaluation protocol and comparison with reflection-based methods: I believe a more reliable evaluation protocol is needed to assess the actual correctness of generated implementations. The current LLM-based rubric scores may not adequately reflect whether the generated code correctly implements the proposed algorithms or reproduces the reported experimental results. Can the authors provide more objective evaluations, such as execution success rates, functional correctness tests, and experimental reproduction accuracy? In addition, how does PaperCompiler compare with execution-guided reflection and iterative refinement methods under a comparable computational budget?