- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies four interfaces for diffusion-based code editing: whole-file rewriting, search-and-replace, locate-then-infill, and token-level editing. Experiments on CanItEdit reveal a gap between infilling with gold edit locations and editing with predicted locations. Controlled scaffold interventions suggest that both coverage and boundary precision matter: missing required regions prevents necessary changes, while overly broad regions increase the burden of regenerating unchanged code. Additional experiments examine source conditioning, known-gap infilling, and sentence-level editing.
Critical strengths
-
The scaffold interventions help explain failures within the evaluated system. Holding the filler and decoding procedure fixed, jointly repairing coverage and boundaries improves multi-span pass@1 from 10.6% to 21.8%, compared with 25.4% using gold scaffolds. Expanding gold regions also substantially reduces performance despite preserving coverage. These experiments provide a useful breakdown of localization errors.
-
The paper is transparent about its experimental limitations. It distinguishes autonomous systems from oracle diagnostics and acknowledges differences between training recipes. The matched source-conditioning experiment also provides evidence that retaining the original code helps with multi-span infilling.
Critical weaknesses
-
The comparison with stronger models and current editing methods is insufficient. The central experiments rely on DiffuCoder-based implementations whose performance is relatively limited. The appendix includes stronger model reference results, including Stable-DiffCoder and autoregressive models, but these systems are not evaluated through the same interface and scaffold experiments. Likewise, alternative adaptive-length infilling methods are discussed without a direct experimental comparison. Consequently, the paper does not establish whether its main bottlenecks remain important for stronger systems or primarily reflect the limitations of the selected implementation. For a study asking how diffusion models should edit code, this missing comparison substantially limits its practical relevance.
-
The experimental design does not support a clear recommendation among interfaces. The autonomous systems differ in initialization, training settings, and decoding procedures, so their scores cannot isolate the effect of the interface. Meanwhile, the central boundary-sensitivity analysis uses a single filler and length mechanism. Even with gold locations, this filler achieves only 27.1% overall pass@1 and produces syntax errors on 27.1% of examples. It remains unclear whether the large penalty from wider regions would persist with a stronger filler or training that includes imperfect scaffolds. The controlled results are informative for this system, but the broader conclusions need stronger evidence.
Questions for authors
-
Can you evaluate a stronger diffusion code model under the same interfaces and scaffold perturbations? Does the coverage–boundary tradeoff remain comparably severe?
-
How does the chosen block-decision infiller compare with more advanced adaptive-length infilling methods under the same gold and predicted locations?
-
Can you provide a matched comparison with a strong autoregressive editing system, using the same evaluation set and comparable editing interfaces?
3 comments
Q2. Block-decision versus adaptive-length infilling
Released adaptive-length model (Sec. 6.4). We evaluate DreamOn-7B on the same 105 descriptive instances, supplying gold locations and gold span lengths as initial budgets (DreamOn remains adaptive during decoding). It reaches 19.0%, versus 29.5% for our block-decision infiller, a paired difference of 10.5 pp (95% CI [2.9, 18.1]). DreamOn is not fine-tuned on our editing data and does not see the original source, so this is not a training-matched comparison. However, its 0.0% on spans above 64 tokens shows that adaptive length alone does not solve long-span infilling.
Frozen-weight block-width ablation. Widening only the first decoding block from 16 to 32 tokens improves 17–32-token spans in all nine evaluated checkpoints and 33–64-token spans in eight of nine, at a cost on short spans. Our pre-specified analysis of spans above 64 tokens over six checkpoints shows no reliable gain. Increasing denoising steps from 64 to 256 produces identical completions on all 210 instances. Block granularity is therefore a useful knob for medium-length spans, but widening blocks does not resolve the long-span failures.
Q3. Comparison with an autoregressive editor
Autoregressive locator with the same infiller (Sec. 5.3). We fine-tuned Qwen3-8B with LoRA for two epochs on the same locator data and fed its predicted regions to the same code infiller. It reaches 11.9% overall and 7.7% on multi-span instances, versus 27.1% and 25.4% with gold regions; the three diffusion locators reach 11.9–13.8% overall. The composition gap therefore also arises with an autoregressive locator.
Autoregressive search-and-replace on the same evaluation set (210 instances, one greedy sample; Sec. 5.2):
| Metric | Qwen3-8B | Diffusion search-and-replace |
|---|---|---|
| pass@1 | 15.2% (32/210) | 23.3% |
| Parse success | 92.9% (195/210) | approximately 89% |
| Edit application success | 45.2% (95/210) | approximately 75% |
Under the same interface, our diffusion editor outperforms Qwen3-8B by 8.1 pp. Qwen3-8B usually produces parseable edits, but fewer than half apply to the source, so application failure is a major obstacle for it. Training and model differences remain, so we read this as a same-interface comparison and do not draw a general diffusion-versus-autoregressive ranking from it. For context, Qwen3-8B whole-file rewriting reaches 51.4% on the 105 descriptive instances, a different interface and subset.
A matched autoregressive infiller under the same gold-region and perturbation protocols would be a valuable further comparison.
In summary, the composition gap appears with four locator backbones (including an autoregressive one) and in prose editing; coverage-preserving robustness training softens loose boundaries slightly but cannot repair missing regions; a released adaptive-length infiller shows the same long-span failure; and under the same search-and-replace interface our diffusion editor outperforms Qwen3-8B. We would be grateful if the reviewer reconsidered the score in light of these results, and are happy to clarify anything further.
Thank you for your detailed response. While I appreciate the authors' clarifications, I still have concerns regarding the limited significance of the contribution. Therefore, I will maintain my original score.
Thank you for the precise distinction between interface-level findings and the strength of the particular infiller, and for valuing the scaffold interventions and our transparency about limitations. We address this distinction with new experiments: coverage-preserving robustness training, a released adaptive-length infiller, an autoregressive locator, and an autoregressive search-and-replace editor.
W1 / Q1. Do the bottlenecks persist beyond the selected implementation?
We have not yet evaluated a second independently fine-tuned code infiller under the full perturbation grid, and the revised limitations (Sec. 7) now state this. Three new checks show that the two main bottlenecks are not specific to our implementation:
The exact boundary-sensitivity magnitudes may still differ for a stronger infiller.
W2. Would training on imperfect scaffolds reduce the boundary penalty?
It helps slightly for loose boundaries, but cannot repair missing regions. We ran two experiments that target different kinds of imperfection.
Training with coverage-preserving perturbations (new; Sec. 6.3). We trained an infiller on gold scaffolds augmented with one-line expansions and extra false-positive spans, and compared it with a control trained with the same two-epoch recipe (overall pass@1):
The expansion penalty relative to clean gold shrinks from 11.9 to 10.0 pp, a directional improvement from a single training comparison (same-recipe runs vary by up to 4.7 pp overall). Missing-region failures remain severe for both models: when code outside the holes is immutable, the infiller cannot edit an excluded region.
Training on predicted scaffolds (Sec. 6.3). Before submission, we also continued training the infiller on scaffolds predicted by the search-and-replace locator. Hybrid-multi multi-span pass@1 fell from 10.6% to 5.6%, and the all-routed variant from 9.2% to 1.4%. The training data explain this result: only 37% of the predicted training scaffolds could be completed into the reference program, and 56% missed a required region, so the model was mostly trained on infeasible targets.
Together, these results confirm that both coverage and boundary precision matter: robustness training can soften loose boundaries a little, but coverage errors require better localization.