acceptodds
Under review as a conference paper at ICLR 2027

Evidence Implementation Alignment for Paper-to-Code Generation

Abstract

Faithfully translating scientific papers into complete code repositories remains challenging even when the relevant algorithmic requirements have already been correctly identified during paper analysis. During coding, these requirements may still be omitted, partially realized, or implemented inconsistently. We refer to this mismatch between identified requirements and their code realizations as the Analysis-to-Code Grounding Gap. To address it, we introduce Evidence Implementation Alignment (EIA), which maintains a unit-level correspondence between paper-grounded implementation requirements and their concrete repository realizations. EIA organizes these requirements into Algorithm Units and verifies their paper support before code generation, then traces the same units into the generated repository and diagnoses their alignment to guide targeted repair. We evaluate EIA on PaperBench CodeDev and Paper2CodeBench using DeepSeek-v4-Flash and Qwen2.5-Coder-32B-Instruct-AWQ. On PaperBench CodeDev, EIA improves over PaperCoder from 52.40% to 63.75% and from 35.98% to 43.48%, respectively. On Paper2CodeBench, EIA achieves the highest average score among the evaluated methods under DeepSeek-v4-Flash and remains close to PaperCoder under Qwen2.5-Coder-32B-Instruct-AWQ. Stage-level ablations and unit-level diagnostics further show incremental gains from both stages and lower measured evidence-supported implementation misalignment after targeted repair.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Paper-to-Code, Repository-Level Code Generation, Large Language Models, Scientific Reproducibility, Evidence-Implementation Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kadGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Evidence Implementation Alignment (EIA) to improve the fidelity of paper-to-code generation. It links paper-derived requirements to code through structured Algorithm Units, verifies them before generation, and traces and repairs mismatches afterwards. Experiments show improvements on PaperBench CodeDev and mixed results on Paper2CodeBench, with most gains coming from the pre-generation stage.

Critical strengths
  • The paper proposed a framework in which they defined Algorithm Units that explicitly connect paper evidence, implementation requirements, and code locations, providing a structured basis for identifying mismatches and guiding targeted repair.

  • The proposed EIA framework achieves good performance on PaperBench, outperforming PaperCoder by about 11 points with DeepSeek-v4-Flash and over 7 points with Qwen2.5-Coder.

Critical weaknesses
  • The paper compared EIA only with PaperCoder and HiRAS. It did not compare with coding-agent systems such as DeepCode, OpenHands, Codex, and Claude Code. These comparisons therefore provide insufficient evidence of EIA’s practical advantage over modern coding agents.

  • The evaluation relies on PaperBench CodeDev scores and reference-based LLM assessments, without reporting whether the generated repositories can execute successfully and reproduce the original papers’ experimental results. As such, it is unclear whether the improved evidence–implementation alignment translates into reliable execution or better reproduction outcomes.

Questions for authors
  1. The backbone used are Qwen2.5-Coder and DeepSeek-v4-Flash. It is unclear whether the reported grounding gap is a limitation of paper-to-code generation or primarily reflects the capabilities of the evaluated backbones. What is the framework's performance if the backbone is replaced by a frontier coding model such as Calude Opus 5.5? Does EIA still show performance gains?

  2. How does EIA compare with existing coding agents, such as DeepCode or OpenHands, under comparable inputs and inference budgets?

  3. Have the generated repositories been executed, and do alignment improvements translate into better reproduction outcomes?

  4. Why is the gain substantially weaker on Paper2CodeBench, particularly with Qwen2.5-Coder?

Reviewer_u9uWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that in paper to code pipelines like PaperCoder [1] the requirements found during analysis get missed or only partly exist in the code (the "Analysis to Code Grounding Gap"). EIA adds two stages -> Stage 1 turns the analysis into Algorithm Units and an LLM labels each VERIFIED, INFERRED or UNRESOLVED before coding. Stage 2 has an LLM trace each unit into the repo and mark it IMPLEMENTED, PARTIAL, MISSING or CONFLICTING. VERIFIED units which are not IMPLEMENTED get their file regenerated.

It is tested on PaperBench CodeDev [2] and Paper2CodeBench with two backbones against PaperCoder and HiRAS [3]. CodeDev improves 52.40 -> 63.75 (DeepSeek) and 35.98 -> 43.48 (Qwen). Paper2CodeBench is 1.81 -> 1.94 (DeepSeek) and about equal (Qwen). Stage 1 gives most of the gain in the ablation and new diagnostics (VMR, VRSR, VRR) show fewer misalignments after repair.

References ->

[1] Seo et al. 2026, Paper2Code / PaperCoder, ICLR 2026, arXiv 2504.17192

[2] Starace et al. 2025, PaperBench, ICML 2025, arXiv 2504.01848

[3] Hong et al. 2026, HiRAS, Findings of ACL 2026

Critical strengths
  1. The problem the paper addresses is real and important which is that code that runs and executes can still be unfaithful to the paper and repairing only units with explicit paper support is a good way to avoid forcing any guesses into the code.

  2. The PaperBench improvement over PaperCoder is consistent across papers and backbones. EIA wins on 17 of 20 papers (DeepSeek-v4-Flash) and 14 of 20 (Qwen2.5-Coder-32B-Instruct-AWQ).

Critical weaknesses
  1. The scores cannot show that the code is more faithful to the paper -> All scores come from LLM judges and the choices are weak. On PaperBench own judge test (Appendix Sec G of the paper) GPT 4o mini gets F1 0.59 and the default o3 mini gets 0.83. PaperBench also calls CodeDev a noisy shortcut and advises several runs per paper [1]. Additionally EIA seems to use one run while PaperBench uses o3 mini as the judge and uses 3 runs per paper [1]. So this evaluation differs from the PaperBench one and its PaperCoder score of 52.40 cannot be compared to the published 45.14 [2] because the model and judge differ. This also means we cannot tell if the baseline is run right, and a gain of a few points on 20 papers with a weaker judge and one run can be noise. Moreover no benchmark where code is run or tested is used, although SciReplicate Bench [3] is cited and ResearchCodeBench [4] also exists which can be used. Also, SA Bench [5] has human labelled units that could check the tracing step.

  2. The novelty is overstated -> ReproAgent [6] already checks each requirement against the code and gives labels close to EIA's four states. RePro [7] already links criteria to code and edits it and PaperCompiler [8] has almost the same evidence labels before coding. The "grounding gap" is the old idea of linking requirements to code (traceability) and none of that work is cited.

References->

[1] Starace et al. 2025, PaperBench, ICML 2025, arXiv 2504.01848

[2] Seo et al. 2026, Paper2Code, arXiv 2504.17192 (v5 PaperBench table)

[3] Xiang et al. 2025, SciReplicate Bench, arXiv 2504.00255

[4] ResearchCodeBench, NeurIPS 2025 Datasets and Benchmarks, arXiv 2506.02314

[5] Hu et al. 2026a, SA Bench, arXiv 2608.24252

[6] Hu et al. 2026b, ReproAgent, arXiv 2608.24291

[7] Zhou et al. 2025, RePro, arXiv 2508.16671

[8] Liu et al. 2026, PaperCompiler, arXiv 2609.02272

Questions for authors

Clarifications ->

  1. How many runs per paper did you use on PaperBench? Do the baselines also get the extra notes file that PaperBench gives with each paper? The ftrl unit takes its 10,000 batch number from it.
  2. How are files chosen for repair and how many repair rounds are run? Does the paper plus the repo fit in the Qwen context or is some of it cut?
  3. On average how many units are there per paper and what share are VERIFIED, INFERRED and UNRESOLVED?
  4. Is the model used for the Blueprint, checking, tracing and repair the same as the coding model? Why use GPT 4o mini as the PaperBench judge when PaperBench itself uses o3 mini as the default judge [1]?
  5. Why is your PaperCoder score different from the published one and why is HiRAS so low?

Suggestions ->

  1. Please run a paired test for Variant B vs C (Stage 1 only vs full EIA, Table 3), and add a baseline with the same token budget.
  2. Please check the tracing by hand (or human based evaluation) on a sample of units. How often are the labels right? Please also run the tracer twice on the same code to see if the labels change.
  3. Please add a benchmark where the code is run or tested, and a judge from another model family. Compare with RePro and DeepCode if you can.
  4. Please also report how many units get worse after repair, not just the total.

References

[1] Starace et al. 2025, PaperBench, ICML 2025, arXiv 2504.01848 (default judge in Sec 4.1, judge test in Table 3)

Reviewer_w3vAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a practical failure in automatic paper-to-code generation: a system can correctly identify an algorithmic requirement during analysis but omit or incorrectly implement it during coding. Its proposed framework, Evidence Implementation Alignment (EIA), maintains a structured checklist of “Algorithm Units.” Each unit links a requirement to supporting paper evidence, its intended code location, and relevant interface constraints. EIA verifies these requirements before generation, then checks their implementation afterward and repairs detected mismatches.

Critical strengths

The paper addresses an important practical problem: requirements correctly identified during paper analysis may still be lost or inconsistently implemented during code generation. The proposed evidence-to-implementation representation makes these failures inspectable and supports targeted repair.

Critical weaknesses

Limited methodological novelty. The core workflow—extracting paper-grounded atomic requirements, verifying their realization in code, localizing discrepancies, and performing targeted repair—substantially overlaps with RePro/SigRef. EIA’s apparent additions are an explicit requirement-to-code correspondence representation, separate evidence and implementation states, and a repair acceptance gate. The paper does not convincingly establish why these additions constitute a substantial methodological advance over existing verification-driven approaches.

Insufficient experimental comparison. The experiments compare against PaperCoder and HiRAS, but omit RePro/SigRef, the most directly relevant baseline for the proposed verification-and-repair mechanism. Although the authors explain that ReproAgent and PaperCompiler appeared shortly before submission, this explanation does not address RePro’s omission.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.