acceptodds
Under review as a conference paper at ICLR 2027

DeepCode: Open Agentic Coding

Abstract

Recent advances in Large Language Models (LLMs) have enabled the shift from coding assistants to autonomous software engineers. However, high-fidelity document-to-codebase synthesis—such as reproducing scientific papers—remains challenging due to the fundamental conflict between information overload and the finite context constraints of LLMs. In this work, we introduce DeepCode, a fully autonomous framework that addresses this challenge through principled information-flow management. DeepCode maximizes task-relevant signals under strict context budgets via four orchestrated operations: source compression via blueprint distillation, structured indexing using stateful memory, conditional knowledge injection via retrieval-augmented generation, and closed-loop error correction. Extensive evaluations on PaperBench demonstrate that DeepCode achieves state-of-the-art performance among the evaluated automated agents, outperforms leading commercial agents on a 5-paper subset, and exceeds the reported PhD-level human Best@3 baseline on the PaperBench 3-paper subset under the Replication Score metric. Our source code is available at: https://anonymous.4open.science/r/DeepCode-C464.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, LLM Agent, Coding Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fvdgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DeepCode is an autonomous framework for generating a repository from a scientific paper or other technical specification. It organizes the workflow into blueprint generation, incremental code generation, and verification/refinement. A pair of analysis agents extracts conceptual and algorithmic specifications; CodeMem retains module summaries, interfaces, and dependencies; CodeRAG conditionally retrieves implementation patterns; and static checks and sandbox execution support repair. The main evaluation uses PaperBench Code-Dev, with three runs on each of 20 papers, reporting a mean score of 74.7%. Additional comparisons cover commercial agents on five papers, AutoP2C on three and five papers, and several backbones and component ablations. The evidence concerns code-development rubric satisfaction; it does not measure successful reproduction of the original papers' empirical results.

Critical strengths

A coherent treatment of cross-file context management. CodeMem preserves public interfaces and dependency information rather than repeatedly carrying full source files, and CodeRAG is gated by the target module's needs (Secs. 3.2.1–3.2.2). The CodeMem ablation in Sec. 4.4 shows large improvements over the stated sliding-window baseline on several tasks. This is consequential because cross-file inconsistency under finite context is a central obstacle in paper-to-repository synthesis. Substantial benchmark coverage and some same-backbone comparisons. Three runs on all 20 Code-Dev papers, with per-paper results in Table 4, are stronger evidence than a few demonstrations. DeepCode also exceeds all listed commercial agents on the five-paper subset, and the three-paper AutoP2C comparison in Table 3 uses the same model family and reports a 77.39% versus 71.59% macro average. These results make the framework practically interesting, even though the controls below need clarification.

Critical weaknesses

The central framework and context-efficiency claims are not isolated by sufficiently specified controls. Most full-benchmark baselines are inherited results using different backbones. The same-backbone commercial and AutoP2C subsets help, but Sec. 4.1 and Appendices A.2–A.4 do not specify comparable generation time/token/tool budgets, context-window sizes, reasoning settings, stopping rules, or the numeric repair limit JJ. Table 4 lists an “Avg. Cost” without units or an accounting definition. Furthermore, the memory ablation compares against simple eviction, rather than a larger-context or competitive summarization control, and blueprint compression is not ablated. Consequently, the reported gains do not establish the strong claim that hierarchical information orchestration is a more effective scaling path than increasing context size, or precisely identify how much improvement is due to the framework rather than execution resources. Executable faithfulness is asserted more strongly than it is independently measured. The quantitative evaluation explicitly excludes post-submission execution and grades Code-Dev leaves through static analysis (Sec. 4.1). The internal verification loop uses tests generated from the system's own blueprint; these tests can share omissions with the generated code. No aggregate independently checked entry-point execution rate or execution/result-match score is reported. Moreover, the loop can terminate at a repair cap, so its output is not necessarily executable in every case (Sec. 3.3.2). This matters because executability and faithful scientific implementation are central motivations, and Sec. 4.4 interprets a modest verification gain as showing that earlier phases already achieved technical correctness. The measured Code-Dev gains support a narrower conclusion.

Questions for authors

Evaluation clarification affecting framework attribution: Please specify the context limit, reasoning mode, time/token/tool budgets, numeric repair cap, and stopping rules for DeepCode and each newly run baseline, including the matched AutoP2C comparison. What are the units and included components of Table 4's cost column? Were the five commercial-comparison papers chosen before observing results? Evidence clarification affecting executable-faithfulness claims: Did an independent evaluator run the submitted entry points in a fresh environment, beyond the agent's own tests? If so, what fraction completed, what was checked, and how often did verification stop at the cap? If not, please distinguish the observed Code-Dev score from established executability and successful empirical reproduction. Claim clarification affecting the information-management contribution: Is there existing evidence comparing CodeMem with a stronger summarization baseline or a larger context at a comparable budget, or isolating blueprint distillation? If not, can the context-scaling conclusion be restricted to the actual sliding-window comparison? Please also give the repetition/uncertainty protocol for the ablation scores.

5 comments

We are grateful for an exceptionally careful and detailed review. The questions are precise, and we have used them as a checklist for the revision. We answer each of them below.

1. Experimental settings, cost and the five-paper subset (W1 and Q1)

Response to W1 and Q1: We understand the concern about how the experimental settings are specified and whether they are comparable across systems. We give the full settings below and will add them to Sec. 4.1 and App. A.3. In every comparison, what is under our control is held fixed; the commercial agents ran without any limit from us, while DeepCode ran under fixed caps.

Settings.

  • Model and reasoning setting. DeepCode, Claude Code (v2.0.22) and Cursor (v1.7.52) all use Claude Sonnet 4.5 with the same thinking setting. Codex (codex-cli 0.47.0) uses GPT-5-Codex, the model it is optimized for, with high reasoning effort. AutoP2C uses Claude Sonnet 4.5 under the same generation and evaluation conditions as DeepCode.
  • Context window and environment. All systems use the model's default context window, which is 200K tokens for Claude Sonnet 4.5, and all are run in the same sandbox described in Sec. 4.1.
  • Repetitions. Every system, and every ablation variant, is run three times on each paper. The new comparison described below follows the same protocol.
  • Stopping rule and repair cap. After code generation, DeepCode runs the project in the sandbox. If execution succeeds, it stops. Otherwise, it analyzes the error trace, applies fixes using language-server tools and tries again, up to a maximum of J = 50 iterations (Sec. 3.3.2, Eq. 3).

Time, token and tool budgets.

  • DeepCode: time and steps. Code generation stops after 800 agent steps or two hours, whichever comes first, and verification stops after at most 50 iterations, as described above. These limits are identical for every paper.
  • DeepCode: tokens. Each model call can generate at most 8,192 tokens, and the working context never exceeds the model's 200K-token window. Total token usage is bounded by the step and time limits and is reported as API cost (see "Cost" below).
  • DeepCode: tools. DeepCode uses only the tools listed in App. A.7 (Table 6).
  • DeepCode in practice. DeepCode stays well within this budget: on the five papers of our CodeMem ablation, a complete run with Claude Sonnet 4.5 took 58 minutes 36 seconds on average and at most 1 hour 17 minutes.
  • Commercial agents. They were run in their standard agent mode with their own built-in tools, and we did not impose any step, time or token limit on them; each ran until it finished the task on its own. We consider this the appropriate setting for comparing agent frameworks, because deciding when a task is complete is itself part of what a framework does. It also means that any difference in resources favors the baselines rather than DeepCode.

Measured resource use (new experiment). To check directly whether DeepCode's gain could come from using more resources, we ran an additional comparison during the rebuttal period with full token logging. DeepCode and DeepSeek Harness, a widely used open-source general-purpose agent harness, were run with the same model, Claude Sonnet 5.5. Given the limited time of the rebuttal period, the comparison covers a five-paper subset, namely the randomly selected papers of our CodeMem ablation, with three runs per paper; all numbers are averages over the three runs. DeepCode scored higher on average (86.8% against 81.0%) while using 73% fewer tokens (44.4M against 165.7M) and 23% less wall-clock time. The per-paper results are in our response to Reviewer LLTk (point 4). Where resources are measured directly, the gain is not obtained by spending more of them.

1. Experimental settings, cost and the five-paper subset (W1 and Q1, continued)

Inherited baselines. The full-benchmark baselines are the results reported by PaperBench and PaperCoder, which is standard practice on this benchmark. The attribution of DeepCode's gain to the framework rests on the same-model comparisons: with Claude Code and Cursor in Table 1, with AutoP2C in Table 3, and with DeepSeek Harness in the new experiment.

Cost (Table 4). The "Avg. Cost" column in Table 4 is the API cost of one DeepCode run in US dollars, covering every model call from planning to verification. It averages USD 8.88 per run and ranges from USD 7.06 to 11.90 across papers. We will add the unit and this definition to the table.

The five papers in Table 1.

  • Were they chosen before observing results? The five papers were drawn at random, so their selection could not depend on any system's results.
  • Why five papers. The number reflects our compute budget: we gave priority to evaluating DeepCode on all 20 papers and to running each baseline three times per paper, which already amounts to 60 baseline runs on these five papers.
  • Diversity of the set. The random draw turned out to be diverse. The papers involve three quite different kinds of research code: training reinforcement-learning agents (fre, rice), implementing numerical algorithms from their mathematical derivations (bam, pinn), and probing and fine-tuning a pretrained language model (mech-u). They are not easy for the baselines either: Claude Code and Cursor score below 0.42 on rice and bam.
  • What the comparison shows. DeepCode outperforms all four baselines (Codex, Claude Code, Cursor and AutoP2C) on every one of the five papers (Table 1). If DeepCode and a baseline were equally strong, winning all five papers would happen by chance with probability 1/32.
  • What the selection affects. The selection affects only which baselines are compared; DeepCode itself is evaluated on all 20 papers (Table 4).

2. Executable faithfulness (W2 and Q2)

Response to W2 and Q2: We understand the concern about how executability is supported by the evaluation. To answer the question directly: our evaluation follows the PaperBench Code-Dev protocol, which grades the submitted code against the paper's requirements and does not include a separate reproduction run, so no independent evaluator ran the submitted entry points. Within this protocol, the evidence on the quality of the generated code is nevertheless strict, for three reasons.

  • Code-Dev grades implementation fidelity requirement by requirement. PaperBench Code-Dev is an official protocol of the benchmark (Starace et al., 2025, Sec. 2.6). Its rubrics are co-developed with the original authors of each paper and contain 3,674 separate code-development requirements across the 20 papers (Starace et al., 2025, Table 7). Each requirement asks whether the submitted code contains a correct implementation of a specific element of the paper, and the judge that grades them is validated against human expert grading on JudgeEval (Starace et al., 2025, Sec. 4.2).
  • The generated code is executed during generation. In its verification phase, DeepCode runs each generated project in the sandbox and repairs the errors that execution exposes (Sec. 3.3.2). The effect of this step is visible in the Code-Dev score itself: enabling verification raises the score by 3.7–6.5% on the three papers of the ablation (Sec. 4.4).
  • All systems are graded under the same protocol. Every baseline in our comparisons is evaluated with the same Code-Dev rubrics and the same judge, so the comparisons are made on equal terms.

Revisions. We will state clearly in the paper that our scores measure how well the code-development requirements are met, as distinct from an independent execution test and from reproduction of the original results. In Sec. 3.3.2 (L271–272), we will describe the verification loop as an iterative repair procedure that runs for at most J = 50 iterations. In Sec. 4.4 (L458–461), we will describe its contribution in terms of the types of errors it corrects: misspelled variable names, missing dependencies and incorrect command-line arguments.

3. Memory and blueprint controls, and the context-scaling claim (W1 and Q3)

Response to W1 and Q3: We understand the concern about how far the CodeMem comparison supports the context-scaling claim. The claim rests on two comparisons: a controlled ablation against sliding-window eviction, which is the context management used by PaperBench's own agent scaffolds, and a new comparison with a general-purpose harness that has its own context management. We follow the reviewer's suggestion and state the conclusion in terms of these comparisons.

  • Why sliding-window eviction is the control. It is the standard way of handling long contexts on this benchmark: PaperBench's agent scaffolds remove old messages from the context when the context limit is approached (Starace et al., 2025, App. F.1). Our ablation keeps everything else in DeepCode unchanged and replaces CodeMem with this strategy, so it measures what CodeMem adds over the benchmark's default.
  • A second reference point. The new comparison with DeepSeek Harness described under point 1 involves a general-purpose harness with its own context management. On the same model, it uses about five times more input tokens per call than DeepCode and obtains a lower score (81.0% against 86.8%).
  • Scope of the claim. In the revised paper, the context-scaling conclusion is stated in terms of these comparisons (L90–91 and L472–473).
  • Blueprint distillation. Blueprint distillation is not ablated on its own, because the blueprint is the foundation on which the later phases are built: it determines the file plan and the build order (Sec. 3.1.3), CodeMem selects memory entries based on it (Sec. 3.2.1), CodeRAG maps reference code to its target files (Sec. 3.2.2), and static verification checks the repository against its file structure (Sec. 3.3). Removing the blueprint would change several components at once, so a clean ablation would require a new baseline pipeline rather than switching off a single component. We will state this in the paper.
  • Repetition and uncertainty. The ablations follow the same principle as the main comparisons: each variant is a full evaluation with three runs per paper under the main configuration. Given our budget, we ran the CodeRAG and verification ablations on the three papers of the PaperBench human-baseline subset and the CodeMem ablation on five randomly selected papers, rather than spreading fewer runs over more papers. Figure 5 reports the mean score per paper, and we will add the run-to-run standard deviations in the revision.

We thank the reviewer again for the detailed questions. Answering them has made the experimental description of the paper more complete. We are happy to provide any further details during the discussion period.

References

[1] Starace et al. PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv:2504.01848, 2025.

[2] Seo et al. Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning. ICLR, 2026.

[3] Lin et al. AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Machine Learning Papers. ICASSP, 2026.

[4] DeepSeek Harness. https://github.com/deepseek-ai/deepseek-harness, 2026.

Reviewer_JeP3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduce DeepCode, which converts research papers into repositories using structured implementation blueprints, compressed code memory, external code retrieval, and iterative verification. It reports strong PaperBench Code-Dev scores, including advantages over commercial coding agents.

Critical strengths

I find the problem important and challenging. I find the design approach sensible. Encouraging empirical results.

Critical weaknesses

The central metric does not establish experimental reproduction. Section 4.1 uses static SimpleJudge grading. Without post-submission reproduction; high scores therefore do not demonstrate that the original scientific results were recovered.

Limited methodological novelty. Planning, summarization, retrieval, and repair are familiar agent components, and “information-flow management” is primarily an organizing interpretation.

Comparisons and ablations are not comprehensive. Commercial and human comparisons use small subsets; the CodeMem baseline uses simple sliding-window eviction, and the paper does not establish a comprehensive matched-budget comparison.

Questions for authors

I find it difficult to identify the novelty of the techniques and the "Why" behind the scientific development. For now, the paper is more like a combination of several engineering techniques that are emerging in "agent" design and I cant identify novel ideas in the paper.

6 comments

We thank the reviewer for the frank and helpful review, and for finding the problem important, the design sensible and the results encouraging. We address each point below.

1. Scope of the evaluation metric (W1)

Response to W1: We understand the concern about what the Code-Dev score establishes. Code-Dev is a strict, fine-grained evaluation of whether a paper has been faithfully implemented, and this is exactly the capability that DeepCode targets. Three facts about the protocol support this.

  • It is an official PaperBench protocol with author-approved rubrics. PaperBench Code-Dev is released by the PaperBench authors as part of the benchmark (Starace et al., 2025, Sec. 2.6). Its rubrics are co-developed with the original authors of each paper, and they break each paper down into requirements small enough for an expert to check one by one.
  • It checks implementation fidelity requirement by requirement. Across the 20 papers, Code-Dev grades 3,674 separate code-development requirements (Starace et al., 2025, Table 7). Each one asks whether the submitted code contains a correct implementation of a specific element of the paper, such as a model component, a training procedure or an experimental configuration. A repository cannot score well merely by looking plausible or being well organized; it has to implement what the paper specifies.
  • Its judge is validated against human experts. Grading is performed by PaperBench's SimpleJudge, whose decisions the PaperBench authors validated against human expert grading on a dedicated benchmark, JudgeEval (Starace et al., 2025, Sec. 4.2).

Code-Dev is also the protocol used by prior paper-to-code work such as PaperCoder, and every system in our comparisons is graded under it, so all comparisons are made on equal terms. It matches our contribution directly: DeepCode changes how the code is written, and Code-Dev evaluates that code. Reproducing the numerical results of a paper additionally depends on running the experiments within a compute budget, which reflects hardware and training time as well as the code. Our claims are stated in terms of the Code-Dev Replication Score (Abstract; Sec. 5), and we will describe the protocol and its scope more fully in Sec. 4.1.

2. Novelty and the reasoning behind the design (W2 and Q)

Response to W2 and Q: We understand the concern about where the novelty of DeepCode lies, given that planning, summarization, retrieval and repair appear in recent agent systems. We appreciate the opportunity to make it explicit. The novelty lies in one design principle and in the three mechanisms that follow from it. Below we explain what is new, and then why it works.

What is new: the design principle. Existing paper-to-code systems, such as PaperCoder and AutoP2C, are organized around the stages of a pipeline: planning, analysis and generation (App. A.1.2). DeepCode starts from a different question: given a long and fragmented paper and a limited context window, what information should the model see at each step? We regard this as a design principle, because it determines what each component does. It forces decisions that a stage-based pipeline does not need to make, namely what to discard, what to keep, and when to bring information in. These decisions lead to three mechanisms that differ from common practice.

(1) The blueprint replaces the paper rather than supplementing it. After the planning phase, the raw paper is removed from the model's context, and every later step works from the blueprint alone (Sec. 3.1.3). The blueprint is built by two agents that read the paper through a section index: one extracts the overall architecture, and the other extracts the exact equations and hyperparameters. A planning agent then reconciles the two and returns to the paper whenever something is ambiguous.

(2) The code memory stores interface contracts rather than summaries of the conversation history. Generated code is not kept in the context. Instead, each finished file is recorded as a structured entry that describes its purpose, its typed public interface, the modules it imports, and the modules expected to depend on it (Sec. 3.2.1). Recording the expected dependents allows a file's interface to be designed with its later users in mind. When the next file is generated, only the entries relevant to it are retrieved.

(3) Retrieval is tied to the blueprint and used only when needed. Standard retrieval-augmented generation looks up code by similarity to a query. CodeRAG instead maps reference repositories to specific target files of the blueprint before generation begins, and during generation a gate decides for each file whether external reference code is needed at all (Sec. 3.2.2, Eq. 2).

2. Novelty and the reasoning behind the design (W2 and Q, continued)

Why it works. Three pieces of evidence point to the same explanation.

(1) With the same model, the framework makes a large difference. Commercial coding agents also plan, summarize, retrieve and repair. If DeepCode were simply a combination of these capabilities, giving all systems the same model should largely close the gap. This is not what we observe: with the same model, Claude Code and Cursor score 0.587 and 0.584, while DeepCode scores 0.861 (Table 1).

(2) The ablation shows where the difference comes from. When CodeMem is replaced by a sliding window that drops older messages, the score on three of the five papers falls from 0.70–0.92 to 0.33–0.43. As reported in Sec. 4.4, the cause is dependency truncation: class and function definitions written early in the run are dropped from the context before the code that depends on them is generated. The model is the same in both settings, so the failure comes from what the model is shown, not from what it is able to do. Similarly, when the model lacks implementation knowledge, CodeRAG fills the gap: on Gemini-2.5-Flash it improves the score by 41–71%.

(3) A new measurement makes this concrete. During the rebuttal period, we ran DeepCode and DeepSeek Harness, a widely used open-source general-purpose agent harness, with the same model, Claude Sonnet 5.5, on the five papers of our CodeMem ablation, with three runs per paper (a subset, because of the limited time of the rebuttal period). DeepSeek Harness sends about 197K input tokens to the model per call on average, while DeepCode sends about 38K, and DeepCode still obtains the higher average score (86.8% against 81.0%; the details are in our response to Reviewer LLTk, point 4). Giving the model less, but more relevant, information did not hurt the result; it improved it.

The lesson. Each of DeepCode's mechanisms targets one such failure. The blueprint keeps irrelevant parts of the paper out of the context, CodeMem keeps earlier interfaces visible, CodeRAG brings in reference code only when a file needs it, and verification catches the errors that remain. We believe this lesson extends beyond DeepCode: in long-horizon code generation, what information reaches the model at each step matters as much as how well the model reasons. In the revision, we will rewrite Section 3 to contrast each mechanism with prior practice and present this lesson as the main takeaway of the paper.

3. Comprehensiveness of comparisons and ablations (W3)

Response to W3: We understand the concern about the coverage of our comparisons and ablations. We address the three parts of this point in turn.

  • Small subsets. The subsets limit only the comparisons with other systems; DeepCode itself is evaluated on all 20 papers with three runs each (Table 4). The 3-paper human comparison uses the only subset on which PaperBench reports a human baseline (Starace et al., 2025, Table 9). For the other comparisons and the ablations, our compute budget forced a choice between covering more papers and controlling the conditions strictly. We chose strict control, because the purpose of these comparisons is to separate the contribution of the framework from that of the model, and this requires running the systems on the same model.
  • Matched budget. In the comparisons with Claude Code, Cursor and AutoP2C, all systems use the same model and the same papers, and each is run three times per paper. DeepCode runs under a fixed budget that is identical for every paper: code generation stops after 800 agent steps or two hours, whichever comes first. The commercial agents, by contrast, were not limited by us at all; they ran in their standard mode until they finished the task on their own. Any difference in resources therefore favors the baselines rather than DeepCode. The new comparison with DeepSeek Harness mentioned above also logs the resources on both sides: DeepCode uses 73% fewer tokens and 23% less time while scoring higher. The full settings are given in our response to Reviewer fvdg.
  • The CodeMem baseline. Sliding-window eviction is the standard way of handling long contexts on this benchmark: PaperBench's own agent scaffolds remove old messages when the context limit is approached (Starace et al., 2025, App. F.1). Our ablation keeps everything else in DeepCode unchanged and replaces CodeMem with this strategy, so it cleanly measures what CodeMem adds over the benchmark's default. The paper's conclusion on CodeMem is stated for this comparison (L90–91 and L472–473). The new comparison with DeepSeek Harness adds a second reference point: a general-purpose harness with its own context management uses about five times more input tokens per call than DeepCode and scores lower.

We thank the reviewer for the candid questions on novelty, which helped us present the design reasoning of DeepCode more clearly. We are happy to answer any further questions.

References

[1] Starace et al. PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv:2504.01848, 2025.

[2] Seo et al. Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning. ICLR, 2026.

[3] Lin et al. AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Machine Learning Papers. ICASSP, 2026.

[4] DeepSeek Harness. https://github.com/deepseek-ai/deepseek-harness, 2026.

Reviewer_LLTkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DeepCode, a framework for document-to-codebase synthesis. DeepCode first builds a hierarchical implementation blueprint, then generates files while maintaining structured summaries of module purpose, public interfaces, and dependency edges. It also augments this process with blueprint-aligned retrieval of external implementation patterns and finishes with static checks, sandbox execution, language-server-based repairs, and dynamic functional verification. On PaperBench Code-Dev, DeepCode achieves a replication score of 74.7 and substantially outperforms prior coding agents, even exceeding the reported PhD-level human baseline. Extensive ablation studies verify the value of each module in DeepCode.

Critical strengths
  • The paper proposes a well-designed system for high-fidelity document-to-codebase synthesis, which has significant practical value.
  • The architecture of DeepCode is well aligned with the failure modes of long coding tasks. High-level planning, persistent dependency tracking, selective use of external knowledge, and execution feedback together effectively improve the ability of LLMs to generate large, complex codebases.
  • The experimental result that DeepCode outperforms PhD-level human experts at paper replication is particularly impressive.
Critical weaknesses
  • The experiments are only conducted on 3 or 5 paper subsets of PaperBench. The conclusions may heavily depend on paper selection and fail to demonstrate the generality of DeepCode.
  • The framework is complicated and has many components. This raises a concern that such an elaborate harness built on many human priors may be unnecessary as LLMs get stronger. It would be beneficial if the authors conducted experiments on stronger frontier LLMs (e.g., Claude Opus 5) to check whether the gains remain compared to other widely used harnesses (e.g., Codex, Claude Code) with carefully curated skills.
  • (Minor) Figures 1 and 3 overlap heavily and could be merged into one figure.
Questions for authors

It would be more informative if the paper could compare the time and token costs of DeepCode with other harnesses.

6 comments

We thank the reviewer for the constructive review and the encouraging comments on our results. Following the reviewer's suggestions, we ran a new experiment during the rebuttal period that compares DeepCode with another harness on a stronger model and measures time and token cost.

1.Response to W1:

We understand the concern: if the evidence for DeepCode came only from three or five papers, the conclusions could depend on which papers were chosen. This is not the case for DeepCode's own results. DeepCode was evaluated on all 20 papers of PaperBench Code-Dev, with three independent runs per paper, and the headline score of 74.7 is the average over all 60 runs (Table 4). These results are also stable across runs: on 19 of the 20 papers, the standard deviation over the three runs is at most 0.09. We acknowledge that the main text did not make this sufficiently clear, since Table 4 currently sits in the appendix, and we will move Table 4 into the main text.

The 3-paper and 5-paper subsets appear only in the comparisons with other systems, and each has its own reason.

Human experts (3 papers). PaperBench reports its human expert baseline only on these three papers (Starace et al., 2025, Table 9), so a comparison with human experts is not possible on any other paper.

Commercial agents (5 papers).

  • Why five papers. Running Codex, Claude Code and Cursor is costly, and we spent our budget on two priorities: evaluating DeepCode on the full benchmark, where our main claims are made, and running each commercial agent three times per paper on the same papers and sandbox, with Claude Code and Cursor using the same model as DeepCode. Covering all 20 papers at this standard was beyond our budget.
  • How they were chosen. The five papers were drawn at random. They turned out to involve quite different kinds of research code: training reinforcement-learning agents (fre, rice), implementing numerical algorithms from their mathematical derivations (bam, pinn), and analyzing and fine-tuning a pretrained language model (mech-u).
  • What the comparison shows. DeepCode outperforms every commercial agent on each of the five papers (Table 1), which indicates that its advantage is not tied to one particular type of task. The new experiment under point 4 adds a head-to-head comparison on two further papers (all-in-one and test-time-model-adaptation), with the same outcome.

Response to W2:

We agree that it is an important question for any agent framework: if a harness mainly compensates for the weaknesses of current models, stronger models could make it unnecessary. Our view is that models and harnesses advance together rather than replace each other, and the new experiment we ran on a stronger model supports this view. We explain our reasoning in five steps.

(1) A stronger model still needs a harness. The language model is the reasoning engine, but a complex task also requires decisions about how the work is organized: what to do first, what information to keep, and how to check the result. This is the role of the harness. In the development of coding agents, each improvement in model capability has led people to give the model larger tasks, from completing a single line, to writing a function, to building an entire repository, and each of these steps has come with a new harness designed for it. This is also why model providers keep building and updating their own harnesses, such as Claude Code and Codex, alongside their models. In our view, a stronger model raises what a good harness can achieve rather than removing the need for one, and the practical question is which harness design best suits a given class of tasks.

(2) DeepCode's mechanisms come from the task, not from the weaknesses of a particular model. DeepCode is designed around the structure of one class of tasks: turning a research paper into a complete repository. This task requires reading a long and often fragmented document, writing many files that depend on each other, and producing code that runs. DeepCode addresses each of these needs with a dedicated mechanism:

  • the blueprint turns the paper into a concrete implementation plan;
  • CodeMem records each file's interface and dependencies, so that later files remain consistent with earlier ones;
  • the verification loop runs the code and repairs errors.

These needs come from the task itself, so they do not disappear when the model becomes stronger.

(3) Evidence on the model used in the paper (Claude Sonnet 4.5). DeepCode, Claude Code and Cursor use the same strong model, Claude Sonnet 4.5 with thinking, so the difference between them lies in the harness. Claude Code is Anthropic's own harness for this model, yet it scores 0.587 on the five papers, compared with 0.861 for DeepCode . On the same model, replacing CodeMem with a simple sliding-window history also lowers the score on all five papers tested.

Whether the gains hold with stronger models and curated harnesses (W2, continued)

(4) New evidence on a stronger model (Claude Sonnet 5.5). Following the reviewer's suggestion, we ran a new experiment during the rebuttal period. We compared DeepCode with DSH (DeepSeek Harness), a widely used open-source general-purpose agent harness, with both systems running on Claude Sonnet 5.5. Because the rebuttal period is short, we ran it on a five-paper subset, reusing the randomly selected papers of our CodeMem ablation so that no new selection was involved, with three runs per paper. The full results are given under point 4; two observations are relevant here:

  • DeepCode becomes stronger with the stronger model. On these five papers, its average score rises from 78.3% with Claude Sonnet 4.5 (Table 4) to 86.8% with Claude Sonnet 5.5, in line with the backbone study in Sec. 4.3.
  • Its advantage over a general-purpose harness remains. On the same stronger model, DSH averages 81.0% and DeepCode 86.8%. DeepCode scores higher on four of the five papers, while using about 73% fewer tokens.

The experiment covers five of the 20 papers, so we present it as evidence on this subset rather than as a full-benchmark result. It points in the same direction as our view: the stronger model lifts DeepCode rather than replacing it.

(5) Curated skills. We see skills as complementary to DeepCode rather than as an alternative. A skill adds a procedure to a general-purpose agent, and DeepCode's blueprint-and-memory workflow is itself the kind of procedure that could be packaged as a skill. In Table 1, no system, DeepCode included, was tuned for individual papers; each used a single fixed configuration across all five. A comparison with commercial agents equipped with curated skills on the newest models would be a valuable extension. It was not feasible within the rebuttal period, and we will note it as future work.

Finally, we agree with the principle behind the concern: a harness should not hard-code what the model can already do on its own. DeepCode's components are designed around the structure of the task rather than the weaknesses of a particular model, which is why we expect them to remain useful as models continue to advance.

3.Response to the minor comment:

We thank the reviewer for this careful observation. We agree that Figures 1 and 3 largely present the same comparison. Following the reviewer's suggestion, we will merge them into a single figure in the revised version, which we will upload during the discussion period, and we will use the space saved to bring the per-paper results of Table 4 into the main text.

4.Response to Q:

We thank the reviewer for this suggestion. We agree that cost matters as much as accuracy in practice, and we ran a new experiment during the rebuttal period to provide this comparison. On the same model, DeepCode obtains a higher score than a general-purpose harness while using 73% fewer tokens and 23% less time.

Setup.

  • Baseline. A cost comparison needs complete token records for every system. The commercial agents in Table 1 were used through subscriptions, which do not provide per-run token records, so those runs cannot be used for this purpose. We therefore compared DeepCode with DSH (DeepSeek Harness), a widely used open-source general-purpose agent harness that logs every model call.
  • Papers. Because the rebuttal period is short, we could not run the full benchmark for both systems, so we used a subset of five papers. To avoid making a new selection, we reused the five papers of our CodeMem ablation, which were randomly selected (Sec. 4.4).
  • Protocol. Both systems use Claude Sonnet 5.5, receive the same papers and are run three times per paper, and both are graded with the same Code-Dev protocol. We report the Replication Score, the wall-clock generation time and the token usage.

Results. All values are averages over the three runs; the last row gives the average score and the totals of the per-paper times and token counts.

Followed by Q:

PaperScore (%): DeepSeek HarnessScore (%): DeepCodeTime: DeepSeek HarnessTime: DeepCodeTotal tokens: DeepSeek HarnessTotal tokens: DeepCode
fre84.6583.8937m54s34m19s52.8M13.2M
all-in-one77.5083.9223m53s31m58s23.1M9.1M
rice73.5383.6224m41s32m26s24.2M10.6M
mech-u92.9697.0454m11s18m26s32.5M4.3M
test-time-model-adaptation76.2385.7244m00s24m14s33.1M7.1M
Average / Total80.9786.843h04m40s2h21m23s165.7M44.4M

Summed over the five papers, DeepSeek Harness made 836 API calls with 164.6M input tokens and 1.10M output tokens, and DeepCode made 1,112 API calls with 42.7M input tokens and 1.64M output tokens.

What the results show.

  • Tokens. DeepCode uses 73% fewer tokens than DeepSeek Harness in total (44.4M against 165.7M), and fewer on every paper.
  • Time. DeepCode takes 23% less time in total and is faster on three of the five papers.
  • Score. DeepCode's average score is higher (86.84% against 80.97%): it leads on four papers and is within one point on the fifth (fre).
  • Why the cost is lower. The token saving follows from the design. DeepCode in fact makes more model calls than DeepSeek Harness and writes more output, but each call carries a much smaller context: about 38K input tokens per call on average, compared with about 197K for DeepSeek Harness. This is consistent with a general-purpose harness carrying much of its accumulated history from call to call, whereas DeepCode gives the model only the blueprint and the memory entries relevant to the file being written.

Cost in the main experiments. For the main experiments with Claude Sonnet 4.5, Table 4 reports DeepCode's API cost: USD 8.88 per run on average across all 20 papers, ranging from USD 7.06 to 11.90 per paper. The unit is currently missing from Table 4, and we will add it.

Scope. This comparison covers five of the 20 papers, so it should be read as a result on this subset rather than as a full-benchmark estimate. We will add it to the revised paper with this scope stated.

We thank the reviewer again for the questions on generality and cost. They led us to run the new comparison above, which we believe makes the paper more complete. We are happy to provide any further clarification during the discussion period.

References

[1] Starace et al. PaperBench: Evaluating AI's Ability to Replicate AI Research. arXiv:2504.01848, 2025.

[2] DeepSeek Harness. https://github.com/deepseek-ai/deepseek-harness, 2026.

Comments on the paper

1 comment

Dear Reviewers, thank you for your thoughtful reviews. As the authors have now responded, please review their responses and the other reviews, and leave follow-up comments on whether the concerns have been addressed. Thank you for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.