- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Evidence Implementation Alignment (EIA) to improve the fidelity of paper-to-code generation. It links paper-derived requirements to code through structured Algorithm Units, verifies them before generation, and traces and repairs mismatches afterwards. Experiments show improvements on PaperBench CodeDev and mixed results on Paper2CodeBench, with most gains coming from the pre-generation stage.
Critical strengths
-
The paper proposed a framework in which they defined Algorithm Units that explicitly connect paper evidence, implementation requirements, and code locations, providing a structured basis for identifying mismatches and guiding targeted repair.
-
The proposed EIA framework achieves good performance on PaperBench, outperforming PaperCoder by about 11 points with DeepSeek-v4-Flash and over 7 points with Qwen2.5-Coder.
Critical weaknesses
-
The paper compared EIA only with PaperCoder and HiRAS. It did not compare with coding-agent systems such as DeepCode, OpenHands, Codex, and Claude Code. These comparisons therefore provide insufficient evidence of EIA’s practical advantage over modern coding agents.
-
The evaluation relies on PaperBench CodeDev scores and reference-based LLM assessments, without reporting whether the generated repositories can execute successfully and reproduce the original papers’ experimental results. As such, it is unclear whether the improved evidence–implementation alignment translates into reliable execution or better reproduction outcomes.
Questions for authors
-
The backbone used are Qwen2.5-Coder and DeepSeek-v4-Flash. It is unclear whether the reported grounding gap is a limitation of paper-to-code generation or primarily reflects the capabilities of the evaluated backbones. What is the framework's performance if the backbone is replaced by a frontier coding model such as Calude Opus 5.5? Does EIA still show performance gains?
-
How does EIA compare with existing coding agents, such as DeepCode or OpenHands, under comparable inputs and inference budgets?
-
Have the generated repositories been executed, and do alignment improvements translate into better reproduction outcomes?
-
Why is the gain substantially weaker on Paper2CodeBench, particularly with Qwen2.5-Coder?