- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how deletion and preservation instructions affect the performance of large language models (LLMs) on code repair tasks in ShyBench. It further analyzes editing behaviors and code commits from real-world software projects.
Critical strengths
-
The paper explores how editing instructions influence code repair.
-
Comparative experiments across multiple models provide useful evidence, and the subsequent evaluation considers the risk of over-deletion.
Critical weaknesses
-
The correctness metric conflates functional correctness with compliance with specific editing requirements enforced by meta-tests. As acknowledged in Appendix E, some patches are functionally correct but are marked as failures because they preserve existing code. Since the saved records do not include the generated code or the specific failed tests, it is impossible to determine from these records what proportion of the reported performance drop stems from functional errors versus failure to meet editing requirements. Functional test results and structural check results should be reported separately.
-
The preservation prompt not only asks the model to keep existing code but also explicitly requires fixes to be implemented by adding code rather than deleting it. However, all tasks in ShyBench are centered on deleting or simplifying code. Consequently, the observed trend is unsurprising, and the experiments do not separate the effects of the preservation constraint from the requirement for additive repairs. While the empirical contribution has some value, it is somewhat limited. Further clarification is needed regarding the specific insights gained from the cross-model experiments and the experiments with scope-limited preservation prompts.
Questions for authors
-
Table 2 reports the proportion of passing patches that use additive edits within predefined task categories. This does not directly establish that the patches actually use wrapping or downstream patching. Could the authors verify these behaviors or limit the conclusions to additive-editing rates?
-
The shyness rate uses the number of patches that passed the tests as the denominator. However, the preservation instruction itself changes the overall pass rate. For GPT-5-mini, the pass rate drops from approximately 94% to 55%, while the shyness rate increases from 13% to 26%. The proportion of all outputs that pass the tests and use additive edits therefore changes from approximately 12% to 14%. Could the authors report the proportions of different types of edits across all outputs, along with their respective test-pass rates? This would help distinguish changes in editing behavior from changes in the denominator.
-
The criteria for determining over-deletion on the BM tasks are insufficiently clear. Phase 2 needs to distinguish deletions that violate preservation requirements from permitted cleanup operations. Which requirements are protected, and how are violations checked?
-
The paper does not report results for the original deletion instruction on the BM tasks, so its over-deletion cost remains unclear. Could the authors provide BM results for this same instruction to allow a direct comparison with preservation?
-
For Llama 4, each BM condition involves only 60 runs. A single change in outcome corresponds to approximately 1.7 percentage points, exceeding the gate’s 1-percentage-point tolerance. Furthermore, the differences between the six candidates and blanket preservation are not statistically significant after Holm correction. Do the current results support a stable trade-off between correctness and safety? Could the authors include additional independent BM tasks and report confidence intervals for the differences in over-deletion rates?
-
The main text and Appendix D.2 require an average correctness recovery of at least 75% across models, but the caption for Figure 3 requires all three model points to fall within the gate region, effectively requiring each model to reach at least 75%. Which criterion is actually applied?
-
Appendix D.2 notes that the number of BM tasks was expanded from the six originally preregistered tasks to twelve before the pilot was completed. Were any results examined before this expansion, and how were the new tasks selected? Are the conclusions consistent between the original six tasks and the expanded set?