- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DeepCode is an autonomous framework for generating a repository from a scientific paper or other technical specification. It organizes the workflow into blueprint generation, incremental code generation, and verification/refinement. A pair of analysis agents extracts conceptual and algorithmic specifications; CodeMem retains module summaries, interfaces, and dependencies; CodeRAG conditionally retrieves implementation patterns; and static checks and sandbox execution support repair. The main evaluation uses PaperBench Code-Dev, with three runs on each of 20 papers, reporting a mean score of 74.7%. Additional comparisons cover commercial agents on five papers, AutoP2C on three and five papers, and several backbones and component ablations. The evidence concerns code-development rubric satisfaction; it does not measure successful reproduction of the original papers' empirical results.
Critical strengths
A coherent treatment of cross-file context management. CodeMem preserves public interfaces and dependency information rather than repeatedly carrying full source files, and CodeRAG is gated by the target module's needs (Secs. 3.2.1–3.2.2). The CodeMem ablation in Sec. 4.4 shows large improvements over the stated sliding-window baseline on several tasks. This is consequential because cross-file inconsistency under finite context is a central obstacle in paper-to-repository synthesis. Substantial benchmark coverage and some same-backbone comparisons. Three runs on all 20 Code-Dev papers, with per-paper results in Table 4, are stronger evidence than a few demonstrations. DeepCode also exceeds all listed commercial agents on the five-paper subset, and the three-paper AutoP2C comparison in Table 3 uses the same model family and reports a 77.39% versus 71.59% macro average. These results make the framework practically interesting, even though the controls below need clarification.
Critical weaknesses
The central framework and context-efficiency claims are not isolated by sufficiently specified controls. Most full-benchmark baselines are inherited results using different backbones. The same-backbone commercial and AutoP2C subsets help, but Sec. 4.1 and Appendices A.2–A.4 do not specify comparable generation time/token/tool budgets, context-window sizes, reasoning settings, stopping rules, or the numeric repair limit . Table 4 lists an “Avg. Cost” without units or an accounting definition. Furthermore, the memory ablation compares against simple eviction, rather than a larger-context or competitive summarization control, and blueprint compression is not ablated. Consequently, the reported gains do not establish the strong claim that hierarchical information orchestration is a more effective scaling path than increasing context size, or precisely identify how much improvement is due to the framework rather than execution resources. Executable faithfulness is asserted more strongly than it is independently measured. The quantitative evaluation explicitly excludes post-submission execution and grades Code-Dev leaves through static analysis (Sec. 4.1). The internal verification loop uses tests generated from the system's own blueprint; these tests can share omissions with the generated code. No aggregate independently checked entry-point execution rate or execution/result-match score is reported. Moreover, the loop can terminate at a repair cap, so its output is not necessarily executable in every case (Sec. 3.3.2). This matters because executability and faithful scientific implementation are central motivations, and Sec. 4.4 interprets a modest verification gain as showing that earlier phases already achieved technical correctness. The measured Code-Dev gains support a narrower conclusion.
Questions for authors
Evaluation clarification affecting framework attribution: Please specify the context limit, reasoning mode, time/token/tool budgets, numeric repair cap, and stopping rules for DeepCode and each newly run baseline, including the matched AutoP2C comparison. What are the units and included components of Table 4's cost column? Were the five commercial-comparison papers chosen before observing results? Evidence clarification affecting executable-faithfulness claims: Did an independent evaluator run the submitted entry points in a fresh environment, beyond the agent's own tests? If so, what fraction completed, what was checked, and how often did verification stop at the cap? If not, please distinguish the observed Code-Dev score from established executability and successful empirical reproduction. Claim clarification affecting the information-management contribution: Is there existing evidence comparing CodeMem with a stronger summarization baseline or a larger context at a comparable budget, or isolating blueprint distillation? If not, can the context-scaling conclusion be restricted to the actual sliding-window comparison? Please also give the repetition/uncertainty protocol for the ablation scores.
5 comments
Part2
1. Experimental settings, cost and the five-paper subset (W1 and Q1, continued)
Inherited baselines. The full-benchmark baselines are the results reported by PaperBench and PaperCoder, which is standard practice on this benchmark. The attribution of DeepCode's gain to the framework rests on the same-model comparisons: with Claude Code and Cursor in Table 1, with AutoP2C in Table 3, and with DeepSeek Harness in the new experiment.
Cost (Table 4). The "Avg. Cost" column in Table 4 is the API cost of one DeepCode run in US dollars, covering every model call from planning to verification. It averages USD 8.88 per run and ranges from USD 7.06 to 11.90 across papers. We will add the unit and this definition to the table.
The five papers in Table 1.
- Were they chosen before observing results? The five papers were drawn at random, so their selection could not depend on any system's results.
- Why five papers. The number reflects our compute budget: we gave priority to evaluating DeepCode on all 20 papers and to running each baseline three times per paper, which already amounts to 60 baseline runs on these five papers.
- Diversity of the set. The random draw turned out to be diverse. The papers involve three quite different kinds of research code: training reinforcement-learning agents (fre, rice), implementing numerical algorithms from their mathematical derivations (bam, pinn), and probing and fine-tuning a pretrained language model (mech-u). They are not easy for the baselines either: Claude Code and Cursor score below 0.42 on rice and bam.
- What the comparison shows. DeepCode outperforms all four baselines (Codex, Claude Code, Cursor and AutoP2C) on every one of the five papers (Table 1). If DeepCode and a baseline were equally strong, winning all five papers would happen by chance with probability 1/32.
- What the selection affects. The selection affects only which baselines are compared; DeepCode itself is evaluated on all 20 papers (Table 4).
2. Executable faithfulness (W2 and Q2)
Response to W2 and Q2: We understand the concern about how executability is supported by the evaluation. To answer the question directly: our evaluation follows the PaperBench Code-Dev protocol, which grades the submitted code against the paper's requirements and does not include a separate reproduction run, so no independent evaluator ran the submitted entry points. Within this protocol, the evidence on the quality of the generated code is nevertheless strict, for three reasons.
- Code-Dev grades implementation fidelity requirement by requirement. PaperBench Code-Dev is an official protocol of the benchmark (Starace et al., 2025, Sec. 2.6). Its rubrics are co-developed with the original authors of each paper and contain 3,674 separate code-development requirements across the 20 papers (Starace et al., 2025, Table 7). Each requirement asks whether the submitted code contains a correct implementation of a specific element of the paper, and the judge that grades them is validated against human expert grading on JudgeEval (Starace et al., 2025, Sec. 4.2).
- The generated code is executed during generation. In its verification phase, DeepCode runs each generated project in the sandbox and repairs the errors that execution exposes (Sec. 3.3.2). The effect of this step is visible in the Code-Dev score itself: enabling verification raises the score by 3.7–6.5% on the three papers of the ablation (Sec. 4.4).
- All systems are graded under the same protocol. Every baseline in our comparisons is evaluated with the same Code-Dev rubrics and the same judge, so the comparisons are made on equal terms.
Revisions. We will state clearly in the paper that our scores measure how well the code-development requirements are met, as distinct from an independent execution test and from reproduction of the original results. In Sec. 3.3.2 (L271–272), we will describe the verification loop as an iterative repair procedure that runs for at most J = 50 iterations. In Sec. 4.4 (L458–461), we will describe its contribution in terms of the types of errors it corrects: misspelled variable names, missing dependencies and incorrect command-line arguments.
Part3
3. Memory and blueprint controls, and the context-scaling claim (W1 and Q3)
Response to W1 and Q3: We understand the concern about how far the CodeMem comparison supports the context-scaling claim. The claim rests on two comparisons: a controlled ablation against sliding-window eviction, which is the context management used by PaperBench's own agent scaffolds, and a new comparison with a general-purpose harness that has its own context management. We follow the reviewer's suggestion and state the conclusion in terms of these comparisons.
- Why sliding-window eviction is the control. It is the standard way of handling long contexts on this benchmark: PaperBench's agent scaffolds remove old messages from the context when the context limit is approached (Starace et al., 2025, App. F.1). Our ablation keeps everything else in DeepCode unchanged and replaces CodeMem with this strategy, so it measures what CodeMem adds over the benchmark's default.
- A second reference point. The new comparison with DeepSeek Harness described under point 1 involves a general-purpose harness with its own context management. On the same model, it uses about five times more input tokens per call than DeepCode and obtains a lower score (81.0% against 86.8%).
- Scope of the claim. In the revised paper, the context-scaling conclusion is stated in terms of these comparisons (L90–91 and L472–473).
- Blueprint distillation. Blueprint distillation is not ablated on its own, because the blueprint is the foundation on which the later phases are built: it determines the file plan and the build order (Sec. 3.1.3), CodeMem selects memory entries based on it (Sec. 3.2.1), CodeRAG maps reference code to its target files (Sec. 3.2.2), and static verification checks the repository against its file structure (Sec. 3.3). Removing the blueprint would change several components at once, so a clean ablation would require a new baseline pipeline rather than switching off a single component. We will state this in the paper.
- Repetition and uncertainty. The ablations follow the same principle as the main comparisons: each variant is a full evaluation with three runs per paper under the main configuration. Given our budget, we ran the CodeRAG and verification ablations on the three papers of the PaperBench human-baseline subset and the CodeMem ablation on five randomly selected papers, rather than spreading fewer runs over more papers. Figure 5 reports the mean score per paper, and we will add the run-to-run standard deviations in the revision.
We thank the reviewer again for the detailed questions. Answering them has made the experimental description of the paper more complete. We are happy to provide any further details during the discussion period.
References
[1] Starace et al. PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv:2504.01848, 2025.
[2] Seo et al. Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning. ICLR, 2026.
[3] Lin et al. AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Machine Learning Papers. ICASSP, 2026.
[4] DeepSeek Harness. https://github.com/deepseek-ai/deepseek-harness, 2026.
Part1
We are grateful for an exceptionally careful and detailed review. The questions are precise, and we have used them as a checklist for the revision. We answer each of them below.
1. Experimental settings, cost and the five-paper subset (W1 and Q1)
Response to W1 and Q1: We understand the concern about how the experimental settings are specified and whether they are comparable across systems. We give the full settings below and will add them to Sec. 4.1 and App. A.3. In every comparison, what is under our control is held fixed; the commercial agents ran without any limit from us, while DeepCode ran under fixed caps.
Settings.
Time, token and tool budgets.
Measured resource use (new experiment). To check directly whether DeepCode's gain could come from using more resources, we ran an additional comparison during the rebuttal period with full token logging. DeepCode and DeepSeek Harness, a widely used open-source general-purpose agent harness, were run with the same model, Claude Sonnet 5.5. Given the limited time of the rebuttal period, the comparison covers a five-paper subset, namely the randomly selected papers of our CodeMem ablation, with three runs per paper; all numbers are averages over the three runs. DeepCode scored higher on average (86.8% against 81.0%) while using 73% fewer tokens (44.4M against 165.7M) and 23% less wall-clock time. The per-paper results are in our response to Reviewer LLTk (point 4). Where resources are measured directly, the gain is not obtained by spending more of them.