- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces DR4Code-Bench, a benchmark designed to evaluate the ability of coding agents to reuse historical investigation reports during the software evolution process. The benchmark organizes historical modifications from real-world repositories into "conceptual propagation chains" and reconstructs individual tasks in isolation. It permits only the transfer of investigation reports—generated prior to the actual coding phase—across tasks, thereby isolating the influence of prior code patches, complete investigation trajectories, and evaluation feedback.
Critical strengths
- The paper introduces an executable benchmark for studying cross-task investigation reuse in coding agents through concept-linked changes from real repositories.
- The experiments cover multiple models and history-selection policies, with repeated runs and sensitivity analyses examining both coding correctness and resource use.
Critical weaknesses
-
Insufficient evidence of benchmark representativeness and quality control. Although the paper validates task executability, it provides limited information on the initial candidate pool, retention rates at each filtering stage, and consistency of human review. This makes it difficult to assess potential selection bias and the consistency of task-description quality.
-
The evaluation does not sufficiently establish the intended capability. Historical reports contain both conceptual knowledge and localization cues such as file paths, but their effects are not evaluated separately. It therefore remains unclear whether the benefits primarily reflect conceptual knowledge reuse or reduced file search, limiting the interpretation of the results.
-
The practical relevance and applicability of the task setting remain unclear. The benchmark uses strictly filtered concept-propagation chains and restricts cross-task information transfer and interaction between investigation and coding. However, it does not sufficiently explain which practical development needs these constraints address or what additional research value they offer over approaches such as reusing investigation summaries in continuous conversations. Missing candidate-selection statistics, direct comparisons with these approaches, and evaluations with existing coding agents further limit assessments of representativeness and generalizability.
Questions for authors
- How do the authors assess whether the models encountered the public code or patches during training, and how might such contamination affect the evaluation?
- Why does the benchmark require strict separation between investigation and coding, report-only transfer, and restricted modification scopes? How representative are these constraints of real-world development?
- Why was report reuse not evaluated with existing coding agents such as SWE-agent? Are the observed effects specific to the proposed two-phase workflow?
- Are the benefits of historical reports primarily driven by conceptual knowledge or localization cues such as file paths? Have these information sources been evaluated separately?
- How common are tasks that benefit from reusing pre-solution investigation reports in real-world development? Could the authors report the initial candidate pool and retention rates at each filtering stage to help assess representativeness?
- Compared with retaining investigation summaries in a continuous conversation, what practical needs and additional research value are addressed by independently reconstructing repository states and transferring only investigation reports?