acceptodds
Withdrawn from ICLR 2027

DR4Code-Bench: Benchmarking Research Experience Reuse Across Software Evolution

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Related changes in evolving software repositories often revisit the same concepts, creating opportunities to reuse earlier repository investigation. Whether such reuse improves coding success, reduces repeated investigation, or both requires direct evaluation. We introduce , an executable benchmark for evaluating pre-solution Investigation Report reuse across concept-linked software evolution tasks. It contains 88 executable tasks from 10 Python repositories organized into 22 concept-propagation chains. Each task is independently reconstructed and validated. Agents investigate the current repository before coding; only their Investigation Reports are carried across tasks, while prior agent patches, research traces, and oracle feedback are withheld. We evaluate six models on the 66 non-anchor tasks under No-History, Full-History, Recent-1, and Top-1 conditions, using a shared research-to-coding protocol and scoped outcome evaluator. Single-run correctness differences vary across models and policies. Three additional runs for DeepSeek-V4-Flash and Qwen3.8-Flash show that the direction of observed correctness gains and the strongest fixed policy can change across executions. In the six-model single-run evaluation, the reuse policies require 2.1–2.8 fewer research calls per episode on average across models and tasks, with small mean changes in coding calls. The reductions persist when comparisons are restricted to paired episodes in which research completes under both conditions. Repeated-run analyses show more consistent research-call reductions for DeepSeek than for Qwen. These results distinguish savings in repository investigation from improvements in final coding correctness and establish DR4Code-Bench as a setting for measuring both.

Primary area:
datasets and benchmarks
Keywords:
LLM-Based Coding Agents, Repository Investigation, Agent Memory, Software Evolution

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KeqVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces DR4Code-Bench, a benchmark designed to evaluate the ability of coding agents to reuse historical investigation reports during the software evolution process. The benchmark organizes historical modifications from real-world repositories into "conceptual propagation chains" and reconstructs individual tasks in isolation. It permits only the transfer of investigation reports—generated prior to the actual coding phase—across tasks, thereby isolating the influence of prior code patches, complete investigation trajectories, and evaluation feedback.

Critical strengths
  1. The paper introduces an executable benchmark for studying cross-task investigation reuse in coding agents through concept-linked changes from real repositories.
  2. The experiments cover multiple models and history-selection policies, with repeated runs and sensitivity analyses examining both coding correctness and resource use.
Critical weaknesses
  1. Insufficient evidence of benchmark representativeness and quality control. Although the paper validates task executability, it provides limited information on the initial candidate pool, retention rates at each filtering stage, and consistency of human review. This makes it difficult to assess potential selection bias and the consistency of task-description quality.

  2. The evaluation does not sufficiently establish the intended capability. Historical reports contain both conceptual knowledge and localization cues such as file paths, but their effects are not evaluated separately. It therefore remains unclear whether the benefits primarily reflect conceptual knowledge reuse or reduced file search, limiting the interpretation of the results.

  3. The practical relevance and applicability of the task setting remain unclear. The benchmark uses strictly filtered concept-propagation chains and restricts cross-task information transfer and interaction between investigation and coding. However, it does not sufficiently explain which practical development needs these constraints address or what additional research value they offer over approaches such as reusing investigation summaries in continuous conversations. Missing candidate-selection statistics, direct comparisons with these approaches, and evaluations with existing coding agents further limit assessments of representativeness and generalizability.

Questions for authors
  1. How do the authors assess whether the models encountered the public code or patches during training, and how might such contamination affect the evaluation?
  2. Why does the benchmark require strict separation between investigation and coding, report-only transfer, and restricted modification scopes? How representative are these constraints of real-world development?
  3. Why was report reuse not evaluated with existing coding agents such as SWE-agent? Are the observed effects specific to the proposed two-phase workflow?
  4. Are the benefits of historical reports primarily driven by conceptual knowledge or localization cues such as file paths? Have these information sources been evaluated separately?
  5. How common are tasks that benefit from reusing pre-solution investigation reports in real-world development? Could the authors report the initial candidate pool and retention rates at each filtering stage to help assess representativeness?
  6. Compared with retaining investigation summaries in a continuous conversation, what practical needs and additional research value are addressed by independently reconstructing repository states and transferring only investigation reports?
Reviewer_KSCVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies whether reusing pre-solution investigation reports across related software evolution tasks can reduce repeated repository exploration or improve coding success. The authors introduce DR4Code-Bench, an executable benchmark of 88 tasks (22 anchor tasks and 66 evaluation tasks) across 10 Python repositories, organized into 22 concept-propagation chains where only structured investigation reports are transferred across tasks. Using a custom two-phase research-to-coding protocol, the authors evaluate six models on the 66 history-bearing tasks under No-History, Full-History, Recent-1, and Top-1 reuse policies, with three additional runs on two models.

Critical strengths

The paper is well written, and the experiments cover six models with additional repeated runs and detailed analyses.

Critical weaknesses

My primary concern is that the benchmark's evaluation setting, including both its problem setup and the agent workflow it prescribes, is heavily detached from how real coding agents operate in practice, and the benchmark itself is too small and currently unavailable for quality review.

First, the custom two-phase agent setup does not reflect realistic coding agent workflows. In practice, real coding agents such as Claude Code and Codex interleave exploration, editing, and test execution: they modify code, run tests to get feedback, and run grep or inspect additional files whenever new questions arise during implementation. In contrast, the benchmark evaluates agents under a restricted two-phase protocol that artificially separates read-only research from coding. During research, the agent only has three basic tools (grep_repo, list_files, and read_file) without shell or Git access. Worse still, during the coding phase, repository search and directory browsing are completely disabled, the agent cannot run any tests or execute code to verify its changes, file edits must be done by overwriting entire files with write_file, and the evaluator discards any edits outside pre-selected function or class scopes. Moreover, the paper neither tests real agents like Claude Code or Codex that search and edit iteratively, nor compares against simple practical baselines such as a static project context file (e.g., AGENTS.md). Under a realistic agent workflow where the agent can freely grep, read files, and run tests while coding, it is unclear whether reusing these pre-solution reports provides any real benefit, and thus whether conclusions drawn from this benchmark transfer to real coding agents.

Second, the benchmark scale is very small. After excluding the 22 anchor tasks, only 66 tasks are actually used for evaluation, and only 44 of them have more than one historical report to compare different selection policies.

Questions for authors

Could the authors clarify why the benchmark adopts such a restricted two-phase setup (e.g., strictly separating research from coding, disabling search and test execution during coding, and discarding edits outside pre-selected scopes) rather than evaluating realistic coding agents (such as Claude Code or Codex) that iteratively search, edit, run tests, and use project context files like AGENTS.md?

Reviewer_H1W1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a new benchmark, i.e. DR4Code-Bench, to evaluate whether reusing history knowledge (i.e., investigation reports) can improve coding task quality. It consists of 88 coding tasks across 10 repositories. Experiments with six models show that reusing historical knowledge can help reduce the investigation effort, but the accuracy improvements is inconsistent for different models and runs.

Critical strengths
  1. The studied problem is important for real-world use, and the benchmark design is also reasonable, which separates prior investigation experience from coding solutions and feedback.

  2. The experiments cover different models and conduct multiple repeated runs to obtain reliable analysis results.

Critical weaknesses
  1. The novelty is quite limited compared to existing works like SWE-ContextBench and SWE-Bench-CL, which already studied reusing prior knowledge for swe tasks. I do not see big difference from SWE-ContextBench besides modeling the reused information to pre-solution investigation reports.

  2. The benchmark is relatively small, with only 88 tasks from 10 repos. And the language coverage is also bad (only python).

  3. The accuracy improvements are not consistent across models or repeated runs, making it hard to draw strong conclusions about the benefit of memory reuse.

  4. The ablation is not comprehensive enough, it does not provide the key ablation to show whether the gains come from actual knowledge or simply knowing which files to look at. There is no comparison against stronger memory retrieval or summarization methods beyond only Full-History, Recent-1, and Top-1.

Questions for authors

Refer to weakness part

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.