acceptodds
Under review as a conference paper at ICLR 2027

How Should Diffusion Language Models Edit Code?

Abstract

Code editing requires a model to decide *where* to make changes, generate the new content, and preserve everything else. We study how masked diffusion language models divide these responsibilities across four editing interfaces: whole-file rewriting, search-and-replace, locate-then-infill, and token-level editing. Experiments on CanItEdit reveal a *composition gap*: diffusion models can generate coordinated changes when the correct edit locations are supplied, but much of this capability is lost when those locations must be predicted. Access to the intact original code helps the model fill multiple edit regions, yet does not resolve the difficulty of selecting those regions. By varying the editable regions while holding the generation model and decoding procedure fixed, we identify two distinct requirements for successful editing: covering every required change and placing precise boundaries around it. Missing a required region prevents the corresponding change, while widening regions to ensure coverage can sharply reduce success by requiring unchanged code to be regenerated. A sentence-level Wiki editing probe shows the same qualitative gap between supplied and predicted locations beyond code. These findings show why strong infilling capability alone does not ensure reliable editing: the interface must expose all required changes while limiting regeneration of unchanged code.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, code editing, masked diffusion, edit interfaces, in-place infilling, code generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q86UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies four interfaces for diffusion-based code editing: whole-file rewriting, search-and-replace, locate-then-infill, and token-level editing. Experiments on CanItEdit reveal a gap between infilling with gold edit locations and editing with predicted locations. Controlled scaffold interventions suggest that both coverage and boundary precision matter: missing required regions prevents necessary changes, while overly broad regions increase the burden of regenerating unchanged code. Additional experiments examine source conditioning, known-gap infilling, and sentence-level editing.

Critical strengths
  1. The scaffold interventions help explain failures within the evaluated system. Holding the filler and decoding procedure fixed, jointly repairing coverage and boundaries improves multi-span pass@1 from 10.6% to 21.8%, compared with 25.4% using gold scaffolds. Expanding gold regions also substantially reduces performance despite preserving coverage. These experiments provide a useful breakdown of localization errors.

  2. The paper is transparent about its experimental limitations. It distinguishes autonomous systems from oracle diagnostics and acknowledges differences between training recipes. The matched source-conditioning experiment also provides evidence that retaining the original code helps with multi-span infilling.

Critical weaknesses
  1. The comparison with stronger models and current editing methods is insufficient. The central experiments rely on DiffuCoder-based implementations whose performance is relatively limited. The appendix includes stronger model reference results, including Stable-DiffCoder and autoregressive models, but these systems are not evaluated through the same interface and scaffold experiments. Likewise, alternative adaptive-length infilling methods are discussed without a direct experimental comparison. Consequently, the paper does not establish whether its main bottlenecks remain important for stronger systems or primarily reflect the limitations of the selected implementation. For a study asking how diffusion models should edit code, this missing comparison substantially limits its practical relevance.

  2. The experimental design does not support a clear recommendation among interfaces. The autonomous systems differ in initialization, training settings, and decoding procedures, so their scores cannot isolate the effect of the interface. Meanwhile, the central boundary-sensitivity analysis uses a single filler and length mechanism. Even with gold locations, this filler achieves only 27.1% overall pass@1 and produces syntax errors on 27.1% of examples. It remains unclear whether the large penalty from wider regions would persist with a stronger filler or training that includes imperfect scaffolds. The controlled results are informative for this system, but the broader conclusions need stronger evidence.

Questions for authors
  1. Can you evaluate a stronger diffusion code model under the same interfaces and scaffold perturbations? Does the coverage–boundary tradeoff remain comparably severe?

  2. How does the chosen block-decision infiller compare with more advanced adaptive-length infilling methods under the same gold and predicted locations?

  3. Can you provide a matched comparison with a strong autoregressive editing system, using the same evaluation set and comparable editing interfaces?

3 comments

Thank you for the precise distinction between interface-level findings and the strength of the particular infiller, and for valuing the scaffold interventions and our transparency about limitations. We address this distinction with new experiments: coverage-preserving robustness training, a released adaptive-length infiller, an autoregressive locator, and an autoregressive search-and-replace editor.

W1 / Q1. Do the bottlenecks persist beyond the selected implementation?

We have not yet evaluated a second independently fine-tuned code infiller under the full perturbation grid, and the revised limitations (Sec. 7) now state this. Three new checks show that the two main bottlenecks are not specific to our implementation:

Check (Sec. 5.3, 6.5)Gold regionsPredicted regions
Locator backbone (multi-span pass@1, same infiller): DiffuCoder-7B / LLaDA-8B / iLLaDA-8B25.4%12.0% / 10.6% / 11.3%
Autoregressive locator: Qwen3-8B (multi-span pass@1, same infiller)25.4%7.7%
Prose editing with Dream-7B, FineEdit Wiki (edit F1, 64 examples)0.9270.376
  • The composition gap appears with every locator, including an autoregressive one, and in a different domain with a different base model.
  • The long-span difficulty appears in the released DreamOn-7B and off-the-shelf DiffuCoder-7B-Instruct, even with gold locations and oracle initial length budgets (DreamOn-7B: 0.0% above 64 tokens; see Q2 and our response to sxyZ, Q3).

The exact boundary-sensitivity magnitudes may still differ for a stronger infiller.

W2. Would training on imperfect scaffolds reduce the boundary penalty?

It helps slightly for loose boundaries, but cannot repair missing regions. We ran two experiments that target different kinds of imperfection.

Training with coverage-preserving perturbations (new; Sec. 6.3). We trained an infiller on gold scaffolds augmented with one-line expansions and extra false-positive spans, and compared it with a control trained with the same two-epoch recipe (overall pass@1):

Scaffold conditionControlAugmented trainingΔ (pp)
Gold24.8%25.2%+0.4
Expand ±1 line12.9%15.2%+2.3
Add a false-positive span21.9%23.3%+1.4
Drop one required span2.4%3.3%+0.9

The expansion penalty relative to clean gold shrinks from 11.9 to 10.0 pp, a directional improvement from a single training comparison (same-recipe runs vary by up to 4.7 pp overall). Missing-region failures remain severe for both models: when code outside the holes is immutable, the infiller cannot edit an excluded region.

Training on predicted scaffolds (Sec. 6.3). Before submission, we also continued training the infiller on scaffolds predicted by the search-and-replace locator. Hybrid-multi multi-span pass@1 fell from 10.6% to 5.6%, and the all-routed variant from 9.2% to 1.4%. The training data explain this result: only 37% of the predicted training scaffolds could be completed into the reference program, and 56% missed a required region, so the model was mostly trained on infeasible targets.

Together, these results confirm that both coverage and boundary precision matter: robustness training can soften loose boundaries a little, but coverage errors require better localization.

Q2. Block-decision versus adaptive-length infilling

Released adaptive-length model (Sec. 6.4). We evaluate DreamOn-7B on the same 105 descriptive instances, supplying gold locations and gold span lengths as initial budgets (DreamOn remains adaptive during decoding). It reaches 19.0%, versus 29.5% for our block-decision infiller, a paired difference of 10.5 pp (95% CI [2.9, 18.1]). DreamOn is not fine-tuned on our editing data and does not see the original source, so this is not a training-matched comparison. However, its 0.0% on spans above 64 tokens shows that adaptive length alone does not solve long-span infilling.

Frozen-weight block-width ablation. Widening only the first decoding block from 16 to 32 tokens improves 17–32-token spans in all nine evaluated checkpoints and 33–64-token spans in eight of nine, at a cost on short spans. Our pre-specified analysis of spans above 64 tokens over six checkpoints shows no reliable gain. Increasing denoising steps from 64 to 256 produces identical completions on all 210 instances. Block granularity is therefore a useful knob for medium-length spans, but widening blocks does not resolve the long-span failures.

Q3. Comparison with an autoregressive editor

Autoregressive locator with the same infiller (Sec. 5.3). We fine-tuned Qwen3-8B with LoRA for two epochs on the same locator data and fed its predicted regions to the same code infiller. It reaches 11.9% overall and 7.7% on multi-span instances, versus 27.1% and 25.4% with gold regions; the three diffusion locators reach 11.9–13.8% overall. The composition gap therefore also arises with an autoregressive locator.

Autoregressive search-and-replace on the same evaluation set (210 instances, one greedy sample; Sec. 5.2):

MetricQwen3-8BDiffusion search-and-replace
pass@115.2% (32/210)23.3%
Parse success92.9% (195/210)approximately 89%
Edit application success45.2% (95/210)approximately 75%

Under the same interface, our diffusion editor outperforms Qwen3-8B by 8.1 pp. Qwen3-8B usually produces parseable edits, but fewer than half apply to the source, so application failure is a major obstacle for it. Training and model differences remain, so we read this as a same-interface comparison and do not draw a general diffusion-versus-autoregressive ranking from it. For context, Qwen3-8B whole-file rewriting reaches 51.4% on the 105 descriptive instances, a different interface and subset.

A matched autoregressive infiller under the same gold-region and perturbation protocols would be a valuable further comparison.


In summary, the composition gap appears with four locator backbones (including an autoregressive one) and in prose editing; coverage-preserving robustness training softens loose boundaries slightly but cannot repair missing regions; a released adaptive-length infiller shows the same long-span failure; and under the same search-and-replace interface our diffusion editor outperforms Qwen3-8B. We would be grateful if the reviewer reconsidered the score in light of these results, and are happy to clarify anything further.

Thank you for your detailed response. While I appreciate the authors' clarifications, I still have concerns regarding the limited significance of the contribution. Therefore, I will maintain my original score.

Reviewer_sxyZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript introduces an important issue of how masked diffusion language models(MDLMs) can effectively perform code editing tasks in LLMs. The author defined and evaluated MDLMs in four categories editing interfaces: whole-file rewriting, search-and-replace, locate-then-infill, and token-level editing. The author evaluated four editing interfaces on the CanItEdit dataset based on the fine-tuned DiffuCoder, and found that Search-and-replace (str_replace) performed the best in autonomous editing. However, in the multi-span editing task, the Gold scaffold + old-visible infill reached 25.4%, while the Hybrid-multi based on predicted positions was only 10.6%, revealing a significant composition gap. And it was experimentally verified that MDLMs require precise code editing positions to have good filling ability, and coverage and boundary precision are two key factors limiting the performance of autonomous editing.

Critical strengths
  1. Meaningful research: Explored the difficulties faced by diffusion language models in migrating from known location code filling to autonomous code editing, and discussed and validated the important determining factors of coverage and boundary precision in the editing area.
Critical weaknesses
  1. The organization and clarity of writing need improvement.
  2. There are confounding factors in the experimental comparison. Although the author acknowledges in Appendix I that different editing interfaces use different training and decoding configurations, these differences still affect the fairness of interface performance comparison, making it difficult for experimental results to fully reflect the advantages and disadvantages of the editing interface itself.
Questions for authors
  1. Fairness of the experiment: Although the author acknowledges in Appendix I that there are differences in training and decoding configurations among different editing interfaces, can the author provide stricter control experiments or further analyze how these differences affect the performance of the four editing interfaces?
  2. Attribution of Composition Gap: Among the 142 multi-span editing samples in Hybrid multi, only 53 entered the filling stage, while the rest of the samples rolled back to str_replace. Can the author report the performance of these two sets of samples separately to further clarify the impact of routing and fallback mechanisms on the observed Composition Gap?
  3. Localization and generation capabilities: Even if the correct Gold edit regions are provided, old-visible infilling performs well in multi-span editing tasks pass@1 It is only 25.4%. Can the author further analyze the main reasons for the failure of the remaining examples, especially the limitations in the generation stage when accurate editing positions have already been provided?
3 comments

Thank you for recognizing the coverage and boundary-precision analysis as meaningful, and for the detailed questions on experimental fairness, routing, and the gold-region ceiling. We show below that the central claims rest on interventions that fix the infiller and decoder, that routing does not explain the composition gap, and that long spans account for most remaining gold-region failures.

W1. Writing and clarity

In the revised PDF we (i) state the analysis contribution and its scope in the abstract and introduction (Sec. 1), (ii) organize Sec. 6 around the composition gap and its diagnostics, (iii) removed internal run names and file paths from the main text, and (iv) clarify the supervision pipeline in Sec. 3: localization coordinates come from deterministic line diffs, teacher-generated text provides only rationales, and at inference edit locations are predicted without access to the reference edit.

Q1. Fairness of training and decoding across interfaces

We agree that the cross-interface comparison in Table 1 cannot isolate the effect of interface choice. Our central conclusions therefore rest on controlled interventions that hold the infiller and decoder fixed, and the Table 1 caption now presents its autonomous rows as operating points rather than a causal interface ranking.

Controlled evidence for the central claims (multi-span instances, problem-clustered paired bootstrap):

ComparisonΔ pass@1 (pp)Paired 95% CI
Fixed infiller: predicted regions → coverage and boundaries repaired+11.2[4.9, 17.6]
Matched-recipe training: source-visible minus source-hidden infiller+11.3[3.5, 19.7]
System level: gold-region system minus Hybrid-multi+14.8[5.6, 24.6]

The first row is a direct scaffold intervention on the same weights. The second compares two infillers trained with the same recipe, differing only in whether the original source is visible. The third additionally involves Hybrid-multi's fallback routing, which we analyze in Q2. CIs resample underlying problems, keeping their descriptive and lazy variants together (5,000 replicates).

Shared and differing settings. All four systems start from DiffuCoder-7B-Instruct and train on EditPackFT Python data. The infiller, search-and-replace model, and locator use a learning rate of 1×10⁻⁵, 10% warmup, batch size 64, and 3–5 epochs; rewrite uses 2×10⁻⁵, batch size 128, and four epochs. Sec. 4 now summarizes these settings (full recipes in App. J). The main comparison uses each run's final checkpoint, and Sec. 4 discloses that the rewrite canvas was selected from 256, 512, and 768 tokens based on test performance.

Sensitivity within each interface is small. Checkpoint choice changes pass@1 by about 3 pp or less, with CIs that include zero (Sec. 5.1):

Within-interface comparisonΔ pass@1 (pp)Paired 95% CI
Rewrite: intermediate (step 428) minus final (step 856)+1.0[−2.9, +4.8]
Locate-then-infill: intermediate (step 1290) minus final (step 1935)−0.5[−5.2, +4.8]
Search-and-replace: intermediate (step 303) minus final (step 1515)−2.9[−7.6, +1.4]

The exception is output-length allocation for search-and-replace: fixed-slot and block-append configurations differ by 14.3 pp (95% CI [8.6, 20.5]), and Sec. 5.1 now reports this dependence alongside the main result.

Between the reported configurations, search-and-replace exceeds rewrite by 11.9 pp (95% CI [6.7, 17.1]) and locate-then-infill by 10.5 pp ([5.2, 16.7]), whereas locate-then-infill and rewrite are statistically indistinguishable (1.4 pp, [−3.3, 6.2]). Checkpoint effects (about 3 pp or less) are much smaller than these gaps, so the ranking is stable across checkpoints; because recipes still differ across interfaces, we present it as a ranking of these specific configurations.

Q2. Routing: only 53 of 142 multi-span instances enter the infiller

Routing does not explain the composition gap. The branch breakdown (Sec. 5.1) is:

Hybrid-multi branchInstancesSolvedpass@1
Routed to infiller53815.1%
Search-and-replace fallback8977.9%
Combined1421510.6%

The router sends better-localized instances to the infiller (gold-span recall 0.60 versus 0.32 in the fallback group), so the two branches are not directly comparable. Even within the routed branch, only 19 of 53 instances cover every required gold span.

To remove routing entirely, we evaluate a predicted-region baseline that sends all 142 instances to the infiller. It also reaches 10.6%, and repairing its coverage and boundaries raises it to 21.8%, close to the 25.4% with gold regions. The gap therefore comes from localization errors, and the fallback mechanism does not account for it. (Hybrid-multi and this baseline happen to share the same aggregate score but are different procedures.)

Q3. Why is gold-region multi-span performance only 25.4%?

Gold locations solve where to edit, but the model must still generate the contents, and long spans dominate the remaining failures (Sec. 6.4).

Gold edit structureInstancespass@1
Longest span ≤16 tokens2871.4%
Longest span 17–32 tokens4042.5%
Longest span 33–64 tokens6822.1%
Longest span >64 tokens746.8%
Multi-span, all spans ≤32 tokens3253.1%
Multi-span, at least one span >64 tokens549.3%

The first four rows partition all 210 instances; the last two are subsets of the 142 multi-span instances. When all spans are short, multi-span infilling succeeds 53.1% of the time, so span length, more than the number of holes, drives the low score. Edit complexity and cross-region dependencies likely also contribute.

Other diffusion checkpoints show the same decline. On the 105 descriptive instances, with gold locations and oracle initial length budgets:

Longest gold spanInstancesOur infillerDreamOn-7BDiffuCoder-Instruct
≤16 tokens1471.4%64.3%64.3%
17–32 tokens2050.0%40.0%45.0%
33–64 tokens3423.5%8.8%11.8%
>64 tokens378.1%0.0%5.4%
All10529.5%19.0%22.9%

All three models decline sharply with span length, and our infiller is the strongest in every bin. These checkpoints are not trained on our editing data, but the shared pattern shows that the long-span difficulty is not limited to our implementation.

Why prior infilling scores look higher. DreamOn evaluates single-gap HumanEval-Infilling and SantaCoder-FIM, and FlexMDM's code evaluation uses single-line HumanEval infilling. PILL's main evaluations also use single contiguous gaps, with a multi-span MBPP extension in its appendix, which we now acknowledge in App. H. CanItEdit instead combines instructions, multiple regions, and long replacements, so its scores are not directly comparable with these benchmarks.

Q3 continued. Additional training and decoding experiments

To find the cause of the long-span failures, we ran approximately twenty additional fine-tuning runs and a series of frozen-weight decoding ablations (Sec. 6.4; full list in App. E):

Hypothesized limitationTested interventionOutcome
Too little exposure to heavily masked inputsChange time reweighting; force all-mask startsNo reliable long-span gain; all-mask starts cost 5.7 pp overall
Too little continuation supervisionRaise continuation-block exposure from 0.32 to 0.42>64-token pass@1: 6.8% → 5.4%
Too many block transitionsTrain and decode with 64-token blocksNo reliable long-span gain; short spans worsen
Poor stopping calibrationReweight end-of-span loss (incl. a repeated run and a lower-weight control)No reliable long-span gain
Initial block budget too smallTrain with wider initial budgetsNo reliable long-span gain
Sparse target supervisionAdd context-denoising losses at different weightsLower validation loss, no task gain
Decoding configurationVary block width, fill order, step count, confidence thresholdNo reliable long-span gain; wider first block helps medium spans
Wrong length decisionsLock the gold number of blocks at decoding26.2% overall versus 27.1% baseline

None of these straightforward fixes produced a reliable long-span gain in this setting. In particular, giving the model the gold number of blocks does not help, so correcting block-count decisions alone does not resolve long spans. Nor is the regime unseen in training: exact match on EditPackFT training examples falls from 37.7% within 16 tokens to 0.0% above 64, although 18% of training spans exceed 64 tokens. The evaluation is sensitive enough to detect real effects: widening the first decoding block improves 17–32-token spans in all nine evaluated checkpoints and 33–64-token spans in eight of nine. Five same-recipe control runs range from 22.4% to 27.1% overall, which is why we do not report small single-run changes as gains. As a negative control, a retrain with no change matches its baseline within 0.33 pp in all three length buckets.

Why a lower auxiliary loss did not improve editing. Only 3.6% of tokens in an infiller training example carry the fill loss. Context denoising reduced validation loss from 0.157 to 0.107 without improving editing. A probe shows that the added tokens are easy: cross-entropy is 0.466 on scaffold tokens and 0.363 on source tokens versus 0.987 on fill tokens (top-1 accuracy 92.2%, 94.4%, and 80.6%). Because the intact-source prefix duplicates much of the scaffold, the auxiliary objective can largely be solved by copying. We present this as a diagnostic hypothesis; denser, non-redundant supervision or larger-scale training is a promising direction for long-span editing.


In summary, the central claims are supported by fixed-infiller interventions with problem-clustered CIs, and within-interface sensitivity is small; a baseline that routes all instances to the infiller shows that routing does not explain the composition gap; and the gold-region ceiling is driven by long spans, a difficulty that released diffusion checkpoints share and that persists across about twenty training and decoding interventions. We would be grateful if the reviewer reconsidered the score in light of these results, and are happy to clarify anything further.

Reviewer_GADmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates code editing by diffusion language models, where the models are required to return a working revision based on source code. This paper further analyzes four types of edit interfaces, including whole-file rewriting, search-and-replace, locate-then-infill, and token-level editing. Experimental results show a composition gap that models can generate coordinated changes with correct edit location, while losing the ability if the locations need to be predicted.

Critical strengths
  • This paper formulates and evaluates four editing interfaces in masked diffusion language models, which is comprehensive.
  • This paper shows an interesting composition gap that models lose ability if the editing locations need to be predicted.
  • This paper conducts extensive experiments to analyze why oracle advantages disappear.
Critical weaknesses
  • Generalization ability to different models.

This paper conducts experiments by fine-tuning DiffuCoder-Instruct (line 238). Could these findings generalize to the group of diffusion language models as claimed in the title?

  • Unclear result presentation and comparison.

Table 1 shows results for different experiments using specific training recipes in this paper, e.g., different editing interfaces. How do these numbers compare with other published diffusion language models?

Moreover, this paper evaluates results with descriptive and lazy instruction variants. How do different instructions influence the key conclusions?

  • Influence of entire source code length.

This paper shows that models perform worse when the editing locations need to be predicted. How about results with different lengths for the entire source file? For example, if the source file is very short, will it be easier to localize the editing location than a longer source file?

Questions for authors

Please refer to the weaknesses.

4 comments

Thank you for finding the comparison of four interfaces comprehensive and the composition gap interesting, and for the concrete suggestion to analyze source-file length. Below we show that the gap holds beyond the DiffuCoder locator, place our absolute scores in context and explain why they are low, and report the source-file-length analysis.

W1. Do the findings generalize beyond DiffuCoder?

For localization, yes: every locator we tested, diffusion or autoregressive, leaves a large gap to gold regions. All locators below feed the same fine-tuned code infiller (Sec. 5.3):

LocatorOverall pass@1Multi-span pass@1
DiffuCoder-7B13.3%12.0%
LLaDA-8B11.9%10.6%
iLLaDA-8B13.8%11.3%
Qwen3-8B (autoregressive)11.9%7.7%
Gold regions27.1%25.4%

On multi-span instances, predicted regions lose 13–18 pp relative to gold regions with every backbone. Because the gap also appears with Qwen3-8B, it reflects the difficulty of localization itself.

The gap also transfers to another domain and base model. On sentence-level FineEdit Wiki editing with models trained from Dream-7B, edit F1 is 0.927 with gold regions versus 0.376 with predicted regions (64-example probe; Sec. 6.5).

For generation, the long-span difficulty also appears in other diffusion checkpoints. The released DreamOn-7B and off-the-shelf DiffuCoder-7B-Instruct, given gold locations and oracle initial length budgets, show the same decline with span length (table in our response to sxyZ, Q3).

Our fine-tuned code infiller is based on DiffuCoder; evaluating a second independently fine-tuned infiller is future work, and the revised introduction and limitations (Sec. 1, Sec. 7) now state this scope.

W2. How do the numbers compare with published models, and how do instruction styles affect the conclusions?

Published references. The original CanItEdit report (Cassano et al., Table 3) gives the following autoregressive results (Sec. 5.2):

Autoregressive modelDescriptive pass@1Lazy pass@1
CodeLlama-7B-Instruct32.8%23.5%
DeepSeekCoder-6.7B-Instruct41.0%31.7%
EditCoder-6.7B48.3%39.3%
Ours: diffusion search-and-replace27.6%19.0%
Ours: gold-region infiller29.5%24.8%

These references estimate pass@1 from 20 samples, whereas we use one greedy sample, so they serve as context and are not matched comparisons. Our diffusion editors are not yet competitive with strong autoregressive editors; our paper aims to identify why.

Our own evaluation. On the 105 descriptive instances, autoregressive whole-file rewriting reaches 72.4% with GPT-4o, 51.4% with Qwen3-8B, and 48.6% with Qwen2.5-Coder-7B-Instruct. Under the same search-and-replace interface on all 210 instances, however, our diffusion editor outperforms Qwen3-8B (23.3% versus 15.2%), and Qwen3-8B's edits often fail to apply to the source (details in our response to Q86U, Q3).

Why the scores are low (Sec. 5.2). Three causes compound:

  • The backbone starts behind. DiffuCoder-7B-Instruct trails its autoregressive parent Qwen2.5-Coder-7B-Instruct on HumanEval (72.0 versus 90.2) and MBPP (75.1 versus 83.9), as reported by Gong et al., and every interface built on it inherits this deficit.

  • Whole-file rewriting is limited by length and generation quality. With a 512-token rewrite canvas, 56 of 210 target files exceed the canvas, and all 56 fail; the other 154 instances reach 15.6%. A 768-token canvas yields 10.0% overall, so truncation is not the only cause.

  • Long spans are hard to infill, even at known locations. Only 28 of 210 instances have all edited spans within 16 tokens. Gold-region pass@1 falls from 71.4% in this group to 6.8% when the longest span exceeds 64 tokens, and released diffusion checkpoints show the same pattern.

The pipeline itself is sound (Sec. 6.1). On HumanEval-Infilling, our setup reaches 93.6% single-line accuracy with the off-the-shelf model and 94.6% after fine-tuning, with 61.9% on multi-line infilling. Fine-tuning is also effective on CanItEdit: one infiller run rises from 9.0% after one epoch to 27.1% after two. We therefore attribute the low CanItEdit scores to the difficulty of instruction-conditioned, multi-region, long-span editing.

Instruction styles (Sec. 5.2). On the multi-span subset:

SystemDescriptiveLazy
Gold regions + source-visible infiller26.8%23.9%
Hybrid-multi (predicted regions)14.1%7.0%
Search-and-replace14.1%8.5%

The composition gap appears under both instruction styles and widens under lazy instructions (from 12.7 pp to 16.9 pp), while the gold-region system drops only 2.9 pp. This is consistent with less specific instructions making localization harder, although instruction style may also affect generation and routing.

W3. Does source-file length affect localization?

Yes, for longer edits. We stratified instances by longest gold-span length and source-file length:

Longest gold spanSource-file lengthInstancesAll gold spans coveredGold-region pass@1
≤32 tokens≤32 lines3857.9%57.9%
≤32 tokens>32 lines3056.7%50.0%
>32 tokens≤32 lines6841.2%10.3%
>32 tokens>32 lines7421.6%17.6%

For longer edits, complete coverage roughly halves in longer files (from 41.2% to 21.6%), whereas gold-region infilling shows no consistent decline across the same bins. Source-file length thus mainly hurts localization, as the reviewer anticipated. For short edits, coverage is similar across file lengths. This is a post-hoc analysis with coarse bins, and CanItEdit files are short (median 32 lines); we added it to Sec. 6.2 with these qualifications. Repository-scale files are a natural next test.


In summary, the composition gap holds across four locator backbones (including an autoregressive one), both instruction styles, and a prose-editing domain; the low absolute scores trace to long-span generation, which also affects released diffusion checkpoints, while sanity checks confirm that the pipeline works; and source-file length mainly hurts localization. We would be grateful if the reviewer reconsidered the score in light of these results, and are happy to clarify anything further.

Thanks to the authors for the rebuttal and additional experiments. While the responses provide analysis of the low scores, it remains unclear to what extent the identified findings generalize to stronger diffusion-based editing systems. Considering this factor and the overall contribution, I would like to maintain my score.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Summary of Updates

We thank all reviewers for their constructive feedback. Reviewers find the problem meaningful (sxyZ), the study of four editing interfaces comprehensive (GADm), and the composition gap between supplied and predicted edit regions interesting (GADm, sxyZ). They note that the scaffold interventions explain where failures come from (GADm, Q86U): with the infiller fixed, repairing coverage and boundaries raises multi-span pass@1 from 10.6% to 21.8%, close to 25.4% with gold regions (Q86U). Q86U also values our transparency and the matched source-conditioning experiment.

Why this problem matters. Diffusion models can jointly denoise several regions while conditioning on the surrounding code. On CanItEdit, the median edit inserts only 24% as many tokens as the target file contains, so keeping unchanged code fixed could avoid most of the regeneration that rewriting requires. We ask what prevents this in practice, and use controlled interventions to separate choosing where to edit from generating what to write. (We measure accuracy; we do not measure wall-clock speed.)

We address the concerns with additional experiments and a revised PDF, in which the titles of new or substantially revised parts are highlighted in blue; section numbers below refer to it. The main updates are:

1. The composition gap holds across locators, instruction styles, and domains (GADm, sxyZ, Q86U). Gold regions beat Hybrid-multi by 14.8 pp (95% CI [5.6, 24.6]) on multi-span CanItEdit, and the gap persists in every setting we tested:

Setting (multi-span unless noted)GoldPredicted
Locator: DiffuCoder-7B / LLaDA-8B / iLLaDA-8B / Qwen3-8B (autoregressive)25.4%12.0% / 10.6% / 11.3% / 7.7%
Descriptive / lazy instructions (Hybrid-multi)26.8% / 23.9%14.1% / 7.0%
FineEdit Wiki prose editing, Dream-7B (edit F1)0.9270.376

Since an autoregressive locator shows the same gap, the difficulty lies in localization itself (Sec. 5.3, 6.5).

2. Routing does not explain the gap (sxyZ). A predicted-region baseline that routes all 142 instances to the infiller also reaches 10.6%; repairing its coverage and boundaries raises it to 21.8% (Sec. 5.1).

3. Coverage and boundary precision are both required (Q86U, sxyZ). With the infiller fixed, dropping one required region reduces multi-span pass@1 from 25.4% to 6.3%, and widening regions by one line reduces it to 7.7%. Training on coverage-preserving perturbed scaffolds shrinks the expansion penalty only from 11.9 to 10.0 pp and does not recover missing regions (pass@1 with a dropped region changes only from 2.4% to 3.3%; Sec. 6.3).

4. The low absolute scores stem from long-span generation (GADm, sxyZ, Q86U). With gold regions, pass@1 falls from 71.4% (longest span ≤16 tokens) to 6.8% (>64 tokens). Released DreamOn-7B and off-the-shelf DiffuCoder-7B-Instruct, given gold locations and oracle length budgets, decline the same way (DreamOn-7B: 0.0% above 64 tokens). As a sanity check, our pipeline reaches 94.6% single-line and 61.9% multi-line on HumanEval-Infilling. About twenty further training runs and decoding ablations gave no reliable long-span gain, although a wider first decoding block helped 17–32-token spans in all nine checkpoints (Sec. 6.4).

5. Comparisons with autoregressive and adaptive-length editors (Q86U, GADm).

Same evaluation setOurs (diffusion)Other
Search-and-replace, 210 instances, vs. Qwen3-8B23.3%15.2%
Gold-region infilling, 105 descriptive, vs. DreamOn-7B29.5%19.0%

Only 45.2% of Qwen3-8B's edits apply to the source, versus approximately 75% for ours; we also add published CanItEdit results for context (Sec. 5.2).

6. Source-file length mainly affects localization (GADm). For edits over 32 tokens, complete gold-span coverage drops from 41.2% in files of at most 32 lines to 21.6% in longer files, while gold-region infilling shows no consistent decline (Sec. 6.2).

7. Fairness and writing (sxyZ, Q86U). We consolidated all training and decoding settings (Sec. 4), showed that checkpoint and canvas choices change pass@1 by about 3 pp or less (Sec. 5.1), and added problem-clustered paired bootstrap CIs. The central claims rest on the fixed-infiller interventions above. The introduction now states the analysis framing, without internal run names.

In summary, the composition gap holds across four locators, both instruction styles, and prose editing; it stems from coverage and boundary errors in the predicted regions; and the remaining gold-region failures trace to long-span generation, which also affects released diffusion checkpoints. A second independently fine-tuned code infiller under the full perturbation grid remains future work, as we now state in Sec. 7. We hope these updates address the reviewers' concerns and are happy to clarify anything further.

Dear Reviewers and Area Chair,

Thank you again for your time and thoughtful feedback. We have provided point-by-point responses and updated the manuscript, with the main changes summarized in our general response.

As the discussion period draws to a close, we would appreciate any feedback on whether our responses address your concerns. Please let us know if any points remain unclear or require further clarification; we would be happy to follow up.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.