- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies edit-induced dependency mismatch: cases where a coding-agent edit creates new obligations for existing consumers, but dependent code paths are not updated. The authors audit 2,879 DeepSWE runs, report 473 terminal diagnosis runs, analyze recovery events, and validate a small number of cases through local replay. They also provide targeted examples from SWE-bench Verified and FeatureBench.
The problem is real and relevant: task tests can pass while important dependent paths remain broken. However, the paper is stronger as an exploratory audit than as a validated bug-finding or measurement paper. My main concerns are unclear novelty over incomplete patches/change-impact analysis, limited behavioral validation of the 473 diagnoses, under-specified replay selection, lack of external confirmation, and an introduction that foregrounds many counts before clearly stating the main insight.
Critical strengths
-
The paper studies a real and potentially important failure mode: coding-agent patches can pass the original task tests while leaving dependent consumers unsynchronized.
-
The edit–obligation–consumer–diagnosis framing is a useful way to make manual diagnoses more structured and auditable.
-
The authors performed a large manual audit over 2,879 DeepSWE runs, with multiple reviewers and explicit evidence categories.
-
Some concrete examples, especially the Kysely case, are compelling illustrations of evaluation blind spots in existing benchmarks.
Critical weaknesses
- Unclear novelty over existing notions of incomplete patches and change-impact failures.
The paper introduces "edit-induced dependency mismatch" and formalizes it through an edit–obligation–consumer–diagnosis chain. While this is a useful audit vocabulary, the paper does not clearly establish how the phenomenon differs from existing notions such as incomplete patches, regression bugs, migration incompleteness, multi-location repair failures, or change-impact failures. Many examples appear to follow the familiar pattern that an agent modifies one part of a program but fails to update dependent locations. The paper would be stronger if it clarified whether the contribution is a new bug category, a measurement framework, an evaluation critique, or a systematic empirical study of known patch-incompleteness phenomena in coding-agent trajectories.
- The large diagnosis set is weakly validated, and the replay selection procedure is under-specified.
The paper's most prominent empirical result is the identification of 473 terminal diagnosis runs in DeepSWE. However, only five DeepSWE cases are validated through local replay and synchronization repair. This creates a substantial gap between the scale of the reported diagnosis set and the scale of behavioral confirmation. The remaining cases may be plausible human diagnoses, but the paper does not estimate how many would survive reproduction, repair validation, independent review, or external confirmation. As a result, the 473 number should not be interpreted as a count of confirmed bugs.
This concern is amplified by the under-specified replay selection procedure. The authors state that replay cases were selected based on localizability and executability, but this does not provide enough information to assess representativeness or selection bias. Without a clear account of how the authors moved from 473 terminal diagnosis runs to five replay cases, the replay results are best interpreted as illustrative examples rather than validation of the full diagnosis set.
- Additional benchmark examples do not substitute for external confirmation.
The SWE-bench Verified and FeatureBench examples are useful as existence evidence, but they do not fully address the validation gap. Prior work has already shown that SWE-bench-style evaluations can contain underspecified tasks, insufficient tests, and hidden behavioral failures. Therefore, showing a few additional passing patches with hidden failures is interesting, but not by itself strong external validation of the proposed diagnosis framework. More convincing evidence would include confirmation by upstream developers, benchmark maintainers, independent evaluators, or accepted regression tests / issue reports / pull requests.
- The paper's organization obscures the main contribution and insight.
The abstract and introduction are dominated by counts, case names, evidence categories, and audit statistics before the paper clearly establishes its central novelty or main empirical insight. The reader sees many numbers early on 2,879 runs, 473 diagnoses, 37 tasks, 85 consumer relations, 177 execution-supported cases, 296 static cases, 370 recovery events, five replays, and additional benchmark examples. But it is not immediately clear what broader claim these numbers support. This makes the paper read more like a collection of audit artifacts than a sharply argued research contribution.
Questions for authors
-
What is the precise novelty over incomplete patches and regression-bug analysis?
-
How many of the 473 diagnoses are behaviorally reproduced or repair-validated?
-
How were the five replay cases selected, and are they representative?
-
How many diagnoses received external confirmation from developers, maintainers, or independent evaluators?
Thank you for your feedback. We have addressed all the issues and incorporated the changes into the revised manuscript, which now states the scope of the study more clearly. We have added ablation and control experiments and a lightweight proof-of-concept module.
W1 / Q1: Empirical contribution and practical significance. This paper is a systematic empirical diagnosis and measurement study of coding agents. It establishes a verifiable evidence chain linking “edit–adaptation obligation–existing consumer–diagnosis” and separately records terminal omissions, recovery within trajectories, and original task scores. Its value lies in revealing specific differences among these dimensions.
First, even in the strict subset in which all terminal events have undisputed scope and execution support, 75 runs that passed the original evaluation still contain consumer defects. Second, 370 runs contain recovery events, but 83 of them still have other terminal omissions, indicating that “recovery occurred” and “propagation was complete at the end” need to be measured separately. Third, local synchronization controls connect the diagnoses to program behavior: both the original Kysely patch and the version with a one-line synchronization fix pass all 276 original evaluation checks, while the same additional consumer-path checks improve from 0/6 to 6/6. Three patches that passed the original evaluation on SWE-bench Verified also provide execution-based counterexamples.
These findings give practitioners concrete targets to inspect: which adaptation obligations an edit introduces, which existing consumers remain unsynchronized, and which combinations of paths the original tests do not cover. The ICLR reviewer guidelines also explicitly recognize contributions to empirical knowledge.
Within this systematic audit, the process cases also reveal a specific gap in checking: even when the relevant code has already been presented, an agent may still incorrectly assess the consumer obligations introduced by an edit. In response to the suggestion of a lightweight proof of concept, the revised manuscript therefore adds three prompt-intervention experiments to test the effect of explicitly revisiting dependency obligations using the edit information already available. The experiments continue from the same checkpoint, holding fixed the full context and code state at the start of continuation and matching the token-budget cap, while varying only the prompt (inserted after each edit). We use the same DeepSWE tasks, mismatch criteria, and budget rules as in the original manuscript, with 2,879 runs in each group.
All three groups use the same mismatch criteria; the 290 baseline runs here correspond to the strict subset with undisputed scope in the original manuscript. The baseline uses the archived runs from the original manuscript. The general-check and dependency-review groups resume from the same checkpoint in each corresponding original run, holding fixed the starting context and code state and matching the budget.
The third group uses the following prompt:
The dependency-review group has a lower percentage of runs with mismatches than both the normal-continuation and general-check groups. Compared with the baseline, it has 221 fewer runs with mismatches, a relative reduction of 76.2%, with the percentage decreasing from 10.1% to 2.4%, alongside a slight 1% improvement in the overall task success rate. This supports the conclusion that, given the same initial information and code state, explicitly re-examining dependencies and adaptation obligations after edits can improve propagation completeness under the measured conditions. This result provides an actionable mitigation direction for the specific judgment gap identified above; the conclusion is limited to the reported propagation outcomes.