acceptodds
Under review as a conference paper at ICLR 2027

Learning from Failures: Verified Edit Distillation for Variable-Length Diffusion Language Models

Abstract

Variable-length diffusion language models insert and delete tokens as they generate, so their outputs are not tied to a fixed set of positions. This matters when the model is to learn from its own failed output. A corrected version of a failure usually differs from it in length, so the two do not share positions, and the standard masked SFT objective, which supervises tokens at fixed positions, can only treat the correction as a whole new response; it cannot express which tokens of the failure to keep and which to change. We propose Verified Edit Distillation (VEDI), which obtains such corrections as execution-verified teacher repairs of the model's own failures. VEDI aligns each failure with its repair, accounting for insertions and deletions, and jointly trains one backbone to predict where to edit and to reconstruct only the edited region given the preserved context. Inference uses the original sampler without a teacher or an additional repair stage. On two 7B models, FlexMDM and DreamOn, VEDI improves over Base on all five benchmarks, with gains of up to 12.7 HumanEval+ points on DreamOn. Trained on the same failures and repairs, full-rewrite SFT recovers only part of this gain: VEDI exceeds it by 6.3 and 5.0 HumanEval+ points on FlexMDM and DreamOn and matches or exceeds it on MBPP+. Controlled comparisons show that the gain requires a real student failure, a repair written for that failure rather than a reference solution, and edit-location supervision that reaches the shared backbone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion language models, masked diffusion, code generation, knowledge distillation, post-training, execution feedback

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dYC5OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes VEDI, a post-training method for variable-length diffusion language models that learns from the model's own failed code. A teacher repairs failed programs sampled on TACO, and execution tests keep only verified repairs. VEDI aligns each failure with its repair, including insertions and deletions, and trains one backbone to predict where to edit and to reconstruct only the edited regions. At inference, the model uses its original sampler with no teacher, auxiliary selector, or extra repair stage. On FlexMDM and DreamOn (both 7B), VEDI improves over the base model on all five benchmarks (TACO, HumanEval, HumanEval+, MBPP, MBPP+) and exceeds full-rewrite SFT trained on the same failures and repairs by 6.3 and 5.0 HumanEval+ points.

Critical strengths

The paper is clearly written, and its components serve one coherent idea: keep what is right in a failed response and focus supervision on the changes it needs. The full-rewrite baselines use the same failures and verified repairs, and the controls (synthetic failures, reference targets, random reconstruction regions, selector gradients) make a convincing case that the gain comes from real failures, failure-specific repairs, and edit-location supervision. The method also adds no cost at inference.

Critical weaknesses
  1. Unclear dependence on variable-length modeling. Aligned repair supervision, targeted reconstruction, and the auxiliary edit-location loss could also be applied to a fixed-length dLLM such as LLaDA or Dream. Without that controlled comparison, the experiments establish the value of edit-structured supervision but not that variable-length modeling is needed. The paper should also separate the benefit of aligning responses of different lengths from the benefit of the backbone's native insert and delete operations.

  2. Incremental novelty. The method combines established ideas from repair distillation, edit supervision, and auxiliary training. This is a minor concern given the clear formulation and the supporting evidence, but the paper should state its contribution relative to these lines of work more precisely.

Overall, the paper presents a clear, focused, and well-supported contribution. The novelty is incremental, but the coherent method and meaningful ablations make the work valuable, and I lean toward acceptance with the minor revisions above.

Questions for authors
  1. Would the same supervision give comparable gains on a fixed-length dLLM (e.g., LLaDA or Dream) trained on the same failures and repairs? If not tested, please clarify that the experiments establish the value of edit-structured supervision while leaving the necessity of variable-length modeling open.
  2. How much of the gain comes from aligning responses of different lengths, and how much from the native insert and delete operations of FlexMDM and DreamOn?
4 comments

Dear Reviewer dYC5,

Thank you for your thoughtful review and clear suggestions. We summarize our responses and the corresponding revisions below. New or changed text in the revised paper is shown in blue. We also corrected the size of FlexMDM, which is 8B, not 7B as the original paper stated; no result changes.

ConcernMain responseRevision locations
W1 / Q1. Dependence on variable-length modelingAdded two fixed-length backbones with the same edit supervision.Sec. 5.2; App. B.5 (Table 13); Sec. 6.3
W2. Relation to prior workStated the relation to repair distillation, edit supervision, and auxiliary training more precisely.Sec. 2.2; Sec. 2.3
Q2. Gain from alignment vs. native operationsAdded an ablation without alignment; compared the gain on DreamOn and its fixed-length counterpart, Dream-Coder.Sec. 6.1 (Table 2); Sec. 5.2; App. B.5 (Table 13)

 

W1 / Q1. Dependence on variable-length modeling

We added the fixed-length comparison you suggested, on two masked diffusion code models with no length-changing operations: Dream-Coder 7B, the code model of the Dream family, and DiffuCoder 7B. The selector loss and the edit-region reconstruction loss are unchanged. Because the method uses the model's own failures, the data were rebuilt from each model's failures. The teacher, the repair procedure, and the training and evaluation protocol follow the other backbones.

Table R1. Fixed-length backbones. Pass@1 (%), mean ± std over three seeds.

BackboneMethodTACOHumanEvalHumanEval+MBPPMBPP+
Dream-CoderBase4.8663.4158.5474.6061.90
Reference SFT4.23 ± 0.2560.98 ± 3.2357.72 ± 3.5770.63 ± 0.2658.02 ± 0.55
Repair SFT5.60 ± 0.4361.38 ± 2.5457.93 ± 2.6673.90 ± 0.6760.67 ± 1.00
VEDI6.48 ± 0.1268.09 ± 2.1463.41 ± 1.0673.99 ± 1.2561.20 ± 0.61
DiffuCoderBase4.9873.7872.5670.3757.67
Reference SFT4.65 ± 0.6371.14 ± 1.2768.09 ± 0.9374.43 ± 1.8662.61 ± 1.30
Repair SFT5.48 ± 1.3271.95 ± 1.0667.68 ± 1.0675.40 ± 0.7063.67 ± 1.59
VEDI6.72 ± 0.4577.24 ± 0.3573.37 ± 0.3576.46 ± 0.7065.34 ± 0.53

On both models VEDI is the highest trained method on all five benchmarks. It exceeds Repair SFT on HumanEval+ by 5.48 points on Dream-Coder and 5.69 on DiffuCoder. On HumanEval and HumanEval+ the full-rewrite methods (Reference SFT and Repair SFT) fall below Base, whereas VEDI rises above it. On Dream-Coder, every trained method is below Base on MBPP and MBPP+, and VEDI loses the least (0.61 and 0.70 points).

A variable-length backbone is therefore not necessary for this gain. As you suggest, these results support the value of edit-structured supervision and do not show that variable-length modeling is required. The revision adds them in Section 5.2 and Appendix B.5 (Table 13). The separation between alignment and the native operations is discussed under Q2 below.

 

W2. Relation to prior work

We agree. VEDI builds on these three lines of work, and the revised Section 2 states the relation to each more precisely (Sections 2.2 and 2.3).

Prior methods either learn the corrected program as a complete rewrite or apply the predicted edits at inference. VEDI derives its edit targets from execution-verified repairs of the student's own failures and uses them only as training targets. We also state that the EditSelector follows auxiliary-task training.

[continued below]

Q2. Gain from alignment and from the native operations

Gain from alignment. We trained a variant that builds the edit labels without alignment, by comparing the tokens at the same position. Everything else is the same as VEDI.

Table R2. Alignment ablation. Pass@1 (%), mean ± std over three seeds.

TACOHumanEval+MBPP+
FlexMDM
VEDI-Min w/o alignment2.14 ± 0.2043.75 ± 1.5153.15 ± 1.59
VEDI-Min3.54 ± 0.2851.04 ± 0.6558.05 ± 0.77
DreamOn
VEDI-Min w/o alignment3.32 ± 0.1452.44 ± 2.2060.58 ± 0.95
VEDI-Min4.65 ± 0.4060.87 ± 1.1163.58 ± 0.81

Without alignment, the score drops on all three benchmarks. On HumanEval+ the drop is 7.29 points on FlexMDM and 8.43 on DreamOn.

The labels show why. When a repair inserts or removes a token, every later token shifts position. Without alignment, these tokens receive a REMASK label although they did not change. The share of failed-program tokens labeled KEEP falls from 74.5% to 39.1% on FlexMDM and from 73.4% to 34.0% on DreamOn.

Gain from the native insert and delete operations. These operations are part of the backbone and cannot be switched off within a model. We therefore use the closest comparison available. The DreamOn checkpoint we use was fine-tuned from Dream-Coder 7B to learn [expand] and [delete]. The two models have different Base scores, so we compare how far VEDI exceeds Repair SFT within each model. Repair SFT is trained on the same failures and repairs, without edit supervision.

Table R3. VEDI − Repair SFT, pass@1 points.

BackboneHumanEval+MBPP+
Dream-Coder (no length-changing operations)+5.48+0.53
DreamOn+8.02+3.88

The gain appears without the operations, but it is larger on DreamOn.

However, we cannot attribute this difference entirely to the operations. DreamOn went through additional fine-tuning, and the training data were built from each model's own failures.

The alignment ablation is in Section 6.1 (Table 2) of the revision, and the Dream-Coder results are in Section 5.2 and Appendix B.5 (Table 13).

Thank you again for the clear and helpful suggestions. We would be glad to clarify any remaining points.

Thank you for your reply. The VEDI's contribution is clear to me now

Dear Reviewer dYC5,

Thank you for your reply and for raising your score. Your suggestions on the fixed-length comparison and on the relation to prior work made the contribution clearer, and we will keep these additions in the final version.

Reviewer_Y7WkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a post-training method called Verified Edit Distillation (VEDI) for variable-length diffusion language models, aiming to enhance code generation capabilities by enabling models to learn local fixes from their own erroneous outputs.

Critical strengths

The research question has clear motivation and a certain degree of novelty. The method demonstrates strong overall coherence.

Critical weaknesses
  1. The mechanistic link between editorial oversight and final generation capabilities remains insufficiently clear. There is a certain difference between the conditional tasks learned during training and the actual reasoning tasks: during the training phase, the model receives the correct context partially retained and reconstructs the repair content under the condition of known editing positions; The inference stage model needs to directly complete the entire program from the initial generated state. Therefore, it is currently unclear whether the performance improvement means that the model has truly learned transferable editing capabilities, or whether local reconstruction training and auxiliary editing position supervision have improved the representation learning of the backbone network.
  2. Reference is TACO's built-in human solution, with a variety of styles (the lambda template in Fig 10 is an example). Repair was written by the 32B teacher under the prompt for "small change", which naturally fits the style of students. So Repair is much better than Reference and cannot be directly attributed to the fact that the repair was written for this failure. Three baselines are missing, and the data volume needs to be aligned: (a) The teacher does not look at failures, solves problems from scratch, and conducts SFT after verification; (b) Align the failure to the teacher solution edit written from scratch; (c) Perform RFT/STaR (oversampling) using students' own correct samples.
  3. Repair SFT+Draft uses explicit repair prompts during training, but performs regular code generation during testing, which differs from training tasks to some extent. Suggest adding a more consistent comparison between training input and supervision forms to more accurately verify the independent contribution of editing supervision.
  4. Only about 35% -41% of failed programs received effective teacher repair, and the majority of failed samples were not used for repair supervision. The author lacks analysis of successful and failed repair samples, making it difficult to determine whether the method primarily benefits from easily repairable errors.
  5. Although a large number of repairs involve length changes, DreamOn's local reconstruction does not directly supervise the [expand] and [delete] states. The paper lacks independent ablation of insertion, deletion, and replacement operations, making it difficult to clarify the actual role of variable length mechanisms in performance improvement.
  6. The experiment only involved two 7B code models and a single teacher. At the same time, error sampling, teacher generated multiple repair candidates, and validation all incur additional costs, but the paper lacks a complete analysis of data construction costs and cross task validation.
Questions for authors
  1. Can the author explain more directly how editing position supervision improves the original diffusion generation process, as EditSelectors are discarded during the inference phase? Can this improvement be validated by generating error types or changes in internal representations?
  2. Repair SFT+Draft receives explicit repair prompts during the training phase, but no longer receives failed programs during testing. Can the author provide a more consistent comparison of training and reasoning conditions to ensure that the performance differences do indeed come from the editing supervision mechanism?
  3. Only about 35% -41% of failed programs can receive verified teacher fixes. Is there a systematic difference in error type, difficulty, or editing distance between successfully fixed and failed fixed programs? Will this affect the generalization ability of the method?
  4. For DreamOn, local reconstruction does not directly supervise the [expand] and [delete] states. So how does the editorial supervision proposed in this article improve the ability of variable length generation? Can you provide ablation experiments divided by insertion, deletion, and replacement operations?
  5. How much does it cost for VEDI teachers to repair data construction? Does this method still have advantages compared to adding regular SFT data or other post training strategies under the same total computational budget?

Current score is based on the proposed concerns and problems, I think they can be solved in the rebuttal process. I expect a clear rebuttal, and willing to imcrease my score based on a good rebuttal.

7 comments

Dear Reviewer Y7Wk,

Thank you for your detailed review. We summarize our responses and the corresponding revisions below. New or changed text in the revised paper is shown in blue.

ConcernMain responseRevision locations
W1 / Q1. How edit supervision improves generationA linear probe on frozen backbones shows that the selector gradient changes the backbone representation.Sec. 6.1 (Table 3b); Sec. 6.3
W2. Who wrote the targetAdded Teacher-Solution SFT, Teacher-Solution Edit, and RFT on matched data; revised the interpretation.Sec. 4.2; Sec. 5.1; App. B.4 (Table 9)
W3 / Q2. Inputs of Repair SFT + DraftRepair SFT is the comparison with matching inputs; the alignment ablation changes only the supervision.Sec. 4.2; Sec. 6.1 (Table 2)
W4 / Q3. Repaired vs. unrepaired failuresCompared them by difficulty, error type, and tests passed; reported gains by difficulty.App. C.3 (Fig. 10, Table 14); Sec. 6.3
W5 / Q4. Per-operation ablationAdded the per-operation ablation and two fixed-length backbones.Sec. 6.1 (Table 2); Sec. 5.2; App. B.5 (Table 13)
W6 / Q5. Cost and scopeAdded a cost breakdown and a budget-matched baseline.App. B.4 (Table 10); Sec. 6.3

 

W1 / Q1. How edit supervision improves generation

The model does not edit at inference. The EditSelector is discarded, but during training its gradient changes the backbone that predicts tokens, and this change remains at inference. Of the two readings you describe, our evidence supports the second: the auxiliary edit-location supervision changes the backbone representation. As you suggested, we verified this through internal representations.

Probe. We froze each backbone, fed it failed programs written by the model, and trained a linear probe on the hidden states to predict the edit label of each token. A higher score means that the representation carries more information about where to edit. The failures come from held-out problems.

Table R1. AUPRC of a linear probe on frozen backbone states.

BackboneMethodMeanREMASKDELETEINSERT
FlexMDMBase0.2280.2340.3240.126
Repair SFT0.2460.2390.3510.148
Detached VEDI0.2480.2460.3460.153
VEDI0.3180.3310.4070.216
DreamOnBase0.3100.2690.5100.151
Repair SFT0.3130.2750.5170.147
Detached VEDI0.3140.2770.5170.149
VEDI0.4080.3500.6280.246

On both backbones, Repair SFT and Detached VEDI stay close to Base. The score rises only for VEDI, and it rises for all three operations. VEDI exceeds Detached VEDI in mean AUPRC by 0.070 on FlexMDM and 0.094 on DreamOn.

Detached VEDI shows that this difference comes from the selector. It is trained on the same data, with the same edit labels and the same reconstruction loss as VEDI; only the selector gradient does not reach the backbone. The representation is therefore changed by the selector gradient, not by the reconstruction loss.

States during generation. The results above use complete programs as input, whereas during generation only part of the tokens are revealed. On DreamOn, we replaced 25%, 50%, and 75% of the input tokens with masks and ran the same probe. The difference between VEDI and Detached VEDI remains: 0.076, 0.071, and 0.062.

At each step, a masked diffusion model predicts the tokens at masked positions from the representations of the tokens already revealed. When those representations change, the predictions for the remaining tokens can change as well. This is the path through which a selector that is discarded at inference can affect generation.

Effect on generation. Blocking the selector gradient also lowers HumanEval+ by 6.38 points on FlexMDM and 4.06 on DreamOn (Table 2).

These results show that the selector gradient makes edit locations more readable in the backbone and that removing it lowers generation accuracy. However, we do not show directly how generation uses this information, and we state this in the limitations (Section 6.3). The probe results are in Section 6.1 (Table 3b).

[continued below]

W2. Who wrote the target

As you point out, Reference and Repair differ not only in whether the target was written for the failure, but also in who wrote it. We added the three baselines you requested, on both backbones.

  • (a) Teacher-Solution SFT (T-S SFT). The same teacher solves the problem from scratch, without seeing the failure. The solution is verified by execution in the same way as a repair and used as a full-response target.
  • (b) Teacher-Solution Edit (T-S Edit). The student failure is aligned with that teacher solution, and the VEDI objective is applied.
  • (c) RFT. The model is trained only on the student's own passing programs, oversampled to the same number of updates.

Data. Not every failure with a verified repair also has a verified teacher solution. T-S SFT, T-S Edit, Repair SFT, and VEDI are therefore trained on the failures that have both (5,104 of 6,432 on FlexMDM; 5,424 of 6,600 on DreamOn), together with the same passing student programs as in Table 1, for the same number of updates. Repair SFT and VEDI are retrained on this subset, so their scores differ slightly from Table 1.

Table R2. Target source and training objective. Pass@1 (%), mean over three seeds. All five benchmarks with standard deviations are in Table 9.

BackboneMethodTargetObjectiveTACOHumanEval+MBPP+
FlexMDM(c) RFTstudent's passing programsSFT2.4442.3552.78
(a) T-S SFTteacher solutionSFT2.6344.0554.25
Repair SFTteacher repairSFT2.8742.5353.70
(b) T-S Editteacher solutionEdit2.6942.5454.54
VEDIteacher repairEdit3.2249.8957.63
DreamOn(c) RFTstudent's passing programsSFT3.7850.2059.52
(a) T-S SFTteacher solutionSFT2.2455.8960.93
Repair SFTteacher repairSFT3.8653.6661.99
(b) T-S Editteacher solutionEdit2.5752.8560.58
VEDIteacher repairEdit4.3659.1561.82

The results show two things.

First, under full rewriting, which target is better depends on the benchmark. T-S SFT is higher on HumanEval+ and Repair SFT is higher on TACO on both backbones, and MBPP+ differs by backbone. Repair SFT is thus not consistently above T-S SFT. We therefore cannot attribute the advantage of Repair SFT over Reference SFT to the repair being written for the failure. As you point out, it can be explained by the target being written by the teacher. We revised this interpretation in Section 5.1.

Second, under edit supervision, the target needs to be a repair of the failure. T-S Edit and VEDI share the teacher, the objective, and the training failures; they differ only in whether the teacher saw the failure. VEDI is higher on all three benchmarks on both backbones, by 7.35 HumanEval+ points on FlexMDM and 6.30 on DreamOn.

VEDI is above all three added baselines on TACO, HumanEval+, and MBPP+ on both backbones. Compared with T-S SFT, the strongest of the three on HumanEval+, VEDI is higher by 5.84 points on FlexMDM and 3.26 on DreamOn. These results are in Appendix B.4 (Table 9).

 

W3 / Q2. Inputs of Repair SFT + Draft

Repair SFT + Draft asks what happens when SFT can see the student's failed draft, since Repair SFT does not see the draft during training.

As you point out, this baseline has different inputs in training and inference. This difference cannot be removed by changing the prompt. No failed draft exists at inference, so any method that takes the draft as input has different inputs in training and inference. Section 4.2 now states that every method receives only the problem at inference.

The comparison with matching inputs is Repair SFT. It is trained on the same failures and repairs as VEDI and uses the same input in training and inference. VEDI-Min exceeds Repair SFT on HumanEval+ by 7.09 points on FlexMDM and 8.02 on DreamOn (Table 1). This is larger than the difference from Repair SFT + Draft (6.34 and 5.03 points).

The contribution of edit supervision is also confirmed by a comparison that keeps the input fixed and changes only the supervision. VEDI with edit labels built without alignment has the same input and training procedure as VEDI, but its HumanEval+ score is lower by 7.29 points on FlexMDM and 8.43 on DreamOn, which brings it to the level of Repair SFT (Section 6.1, Table 2).

[continued below]

W4 / Q3. Repaired vs. unrepaired failures

We compared repaired and unrepaired failures by difficulty, error type, and distance from a correct program. We re-executed every failure in the training split (18,374 on FlexMDM; 16,279 on DreamOn) on the full test suites. The teacher repaired 6,432 and 6,600 of them.

Difficulty. The largest difference is difficulty. The table gives the share of failures for which the teacher produced a verified repair.

Table R3. Share of failures with a verified repair, by problem difficulty.

DifficultyFlexMDMDreamOn
Easy42.8%53.4%
Medium33.6%38.4%
Medium-hard26.5%27.2%
Hard17.6%19.7%
Very hard21.1%18.2%

The teacher repairs failures on easy problems more often.

Error type and distance from a correct program. The table compares the student's failed programs, split by whether the teacher later repaired them.

Table R4. Repaired / unrepaired failures.

FlexMDMDreamOn
Wrong answer (%)61.4 / 57.465.7 / 62.0
Syntax error (%)18.7 / 20.910.9 / 14.2
Runtime error (%)12.2 / 12.720.6 / 21.1
Tests passed, mean (%)10.3 / 6.511.4 / 8.1

Differences in error type are small. Unrepaired failures have no repair, so repair scope cannot be compared directly; we compare the share of tests passed instead. Repaired failures pass slightly more tests.

Repaired failures are not limited to programs that were almost correct, however: 75.8% (FlexMDM) and 72.3% (DreamOn) of them pass no test at all, and 87.4% and 80.7% come from problems that the student did not solve in any rollout.

Generalization. To see whether the concentration of repairs on easy problems confines the gain to easy problems, we split the TACO test problems by difficulty.

Table R5. TACO pass@1 (%) by difficulty, mean over three seeds.

DifficultyProblemsFlexMDM BaseFlexMDM VEDI-MinDreamOn BaseDreamOn VEDI-Min
Easy1827.2810.266.5910.62
Medium1730.582.313.473.47
Medium-hard1580.471.581.273.38
Hard1530.161.911.962.83
Very hard1370.180.241.461.70
All8031.933.543.114.65

The gain is not confined to easy problems. On hard problems, FlexMDM rises from 0.16 to 1.91 and DreamOn from 1.96 to 2.83. There are two exceptions: DreamOn does not change on medium problems, and on very hard problems there is almost no gain on either backbone. The gain also appears on HumanEval and MBPP, where the teacher produced no repairs (Table 1).

A limitation of the method is that 60% to 65% of failures yield no training signal, and that these failures come more often from hard problems. Section 6.3 now states that verified repairs are skewed toward easier problems and that VEDI changes little on the hardest ones, and the analysis above is in Appendix C.3 (Figure 10, Table 14).

[continued below]

Reviewer_Ru4VOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

VEDI trains variable-length diffusion language models using execution-verified teacher repairs of actual student failures. It aligns failure–repair pairs and jointly supervises edit locations and local reconstruction through a shared backbone. Inference retains the original sampler without an additional repair stage. Experiments on two 7B backbones show consistent improvements over their base models. The contribution lies in combining failure-specific verified repairs with edit-structured supervision, rather than edit-based training itself.

Critical strengths
  1. The method distinguishes learning a successful endpoint from learning what should change in a particular failure. Alignment, edit-location supervision, and local reconstruction form a coherent training strategy.
  2. Results on both FlexMDM and DreamOn support practical value. VEDI-Min exceeds Repair SFT + Draft by 6.34 and 5.03 HumanEval+ points, respectively, while retaining ordinary inference without a teacher or repair stage.
  3. Reference-target controls, selector-gradient ablations, alignment comparisons, and retention analyses provide useful evidence beyond aggregate accuracy. The paper also acknowledges limitations of some controls and mixed results between selection variants.
Critical weaknesses
  1. The closest full-rewrite comparison needs clarification. Repair SFT + Draft uses a failed candidate and revision instruction during training, while the evaluation description concerns ordinary generation. Whether corresponding drafts are supplied at inference, and how any mismatch affects performance, remains unclear.
  2. The necessity of variable-length modeling is insufficiently isolated. Insertion-aware alignment is useful, but this does not establish that a variable-length backbone is necessary. A fixed-canvas control with comparable edit supervision would clarify the source of improvement.
  3. Some component-necessity claims exceed the controls. Synthetic corruption changes the failure distribution and lacks DELETE targets, while random-region supervision retains substantial overlap with true edits. These comparisons support the design without fully isolating its causal ingredients.
  4. Statistical uncertainty is insufficiently reported for close comparisons. Table 1 provides three-seed means without dispersion, despite differences as small as 0.18 MBPP+ points between VEDI-Min and Repair SFT + Draft on DreamOn.
Questions for authors

See Weaknesses.

4 comments

Dear Reviewer Ru4V,

Thank you for your careful review. We summarize our responses and the corresponding revisions below. New or changed text in the revised paper is shown in blue. We also corrected the size of FlexMDM, which is 8B, not 7B as the original paper stated; no result changes.

ConcernMain responseRevision locations
W1. Inference input of Repair SFT + DraftNo draft is supplied at inference for any method; Repair SFT is the comparison without the mismatch.Sec. 4.2
W2. Need for a variable-length backboneAdded two fixed-length backbones with the same edit supervision.Sec. 5.2; App. B.5 (Table 13)
W3. Controls vs. necessity claimsTested whether missing DELETE targets explain Synthetic Corruption; softened the wording.Abstract; Sec. 1; Sec. 6.1 (Table 2)
W4. Statistical uncertaintyAdded standard deviations over three seeds.Table 1

 

W1. Inference input of Repair SFT + Draft

No drafts are supplied at inference. Every method, including Repair SFT + Draft, receives only the problem prompt and generates in a single pass. The original paper did not state this clearly, and we now state it in Section 4.2.

Repair SFT + Draft therefore sees the failed program in training but not at inference. The effect of this mismatch cannot be isolated. Removing it means either supplying a draft at inference, which changes the task, or dropping the draft from training, which is Repair SFT. Repair SFT is thus the comparison without the mismatch: it uses the same input in training and inference and is trained on the same failures and repairs as VEDI. VEDI-Min exceeds it on HumanEval+ by 7.09 points on FlexMDM and 8.02 on DreamOn, and on MBPP+ by 3.42 and 3.88 points (Table 1).

 

W2. Need for a variable-length backbone

We added the fixed-canvas control you suggested, on two masked diffusion models with no length-changing operations: Dream-Coder 7B and DiffuCoder 7B. The selector loss and the edit-region reconstruction loss are unchanged. The data were rebuilt from each model's own failures, and the training and evaluation protocol follows the other backbones.

Table R1. Fixed-length backbones. Pass@1 (%), mean ± std over three seeds.

BackboneMethodTACOHumanEvalHumanEval+MBPPMBPP+
Dream-CoderBase4.8663.4158.5474.6061.90
Reference SFT4.23 ± 0.2560.98 ± 3.2357.72 ± 3.5770.63 ± 0.2658.02 ± 0.55
Repair SFT5.60 ± 0.4361.38 ± 2.5457.93 ± 2.6673.90 ± 0.6760.67 ± 1.00
VEDI6.48 ± 0.1268.09 ± 2.1463.41 ± 1.0673.99 ± 1.2561.20 ± 0.61
DiffuCoderBase4.9873.7872.5670.3757.67
Reference SFT4.65 ± 0.6371.14 ± 1.2768.09 ± 0.9374.43 ± 1.8662.61 ± 1.30
Repair SFT5.48 ± 1.3271.95 ± 1.0667.68 ± 1.0675.40 ± 0.7063.67 ± 1.59
VEDI6.72 ± 0.4577.24 ± 0.3573.37 ± 0.3576.46 ± 0.7065.34 ± 0.53

On both models VEDI is the highest trained method on all five benchmarks. It exceeds Repair SFT on HumanEval+ by 5.48 points on Dream-Coder and 5.69 on DiffuCoder. On HumanEval and HumanEval+ the full-rewrite methods fall below Base, whereas VEDI rises above it. On Dream-Coder, every trained method is below Base on MBPP and MBPP+, and VEDI loses the least (0.61 and 0.70 points).

A variable-length backbone is therefore not necessary for this gain. These results are in Section 5.2 and Appendix B.5 (Table 13).

[continued below]

W3. Controls and necessity claims

We agree that what the controls support should be distinguished from the necessity of each component.

Synthetic Corruption. This control keeps the repair and replaces the student's real failure with an artificially corrupted version of that repair. Changing the source of the failure is the purpose of the control, so the difference in failure distribution is intended. As you note, the corruption has no DELETE targets. It follows the corruption process of the diffusion model, which only removes or masks tokens and never adds any (Appendix A.4). Adding tokens would require an arbitrary choice of which tokens to insert and where, and the resulting errors would not resemble real failures.

We checked whether the missing DELETE supervision explains the gap. We trained VEDI on real failures with only the selector's DELETE supervision removed, so this variant receives no DELETE supervision either.

Table R2. Synthetic Corruption vs. VEDI without DELETE supervision. Pass@1 (%), mean ± std over three seeds.

BackboneMethodTACOHumanEvalHumanEval+MBPPMBPP+
FlexMDMSynthetic Corruption2.63 ± 0.2644.82 ± 0.9741.35 ± 1.2663.45 ± 2.4253.90 ± 2.41
VEDI w/o DELETE supervision3.00 ± 0.5352.10 ± 0.9348.29 ± 0.7866.07 ± 0.5056.31 ± 0.43
VEDI-Min3.54 ± 0.2854.42 ± 0.7051.04 ± 0.6567.50 ± 0.9458.05 ± 0.77
DreamOnSynthetic Corruption3.15 ± 0.6851.22 ± 1.8347.92 ± 1.3769.75 ± 1.0759.88 ± 0.55
VEDI w/o DELETE supervision4.19 ± 0.1459.55 ± 1.7656.71 ± 0.6174.69 ± 0.5561.90 ± 0.79
VEDI-Min4.65 ± 0.4064.84 ± 2.8860.87 ± 1.1172.75 ± 0.7063.58 ± 0.81

Without DELETE supervision, VEDI loses 2.75 (FlexMDM) and 4.16 (DreamOn) HumanEval+ points. It still stays 6.94 and 8.79 points above Synthetic Corruption, and it is above Synthetic Corruption on all five benchmarks. Most of the gap therefore remains after DELETE supervision is removed, so the missing DELETE supervision alone does not explain it.

One difference remains that this check cannot remove. The synthetic input never contains tokens that need to be deleted, because the corruption only removes or masks tokens. The check therefore narrows the explanation but does not fully isolate the effect of a real failure. The row without DELETE supervision is in the new ablation table (Table 2, Section 6.1).

Random-Region Reconstruction. This control keeps the number of reconstructed tokens and the number and lengths of spans, and moves only their positions at random. We did not force the random spans to avoid the true edit regions, and the token-level overlap is 69% on DreamOn and 64% on FlexMDM. It is therefore a position control matched in reconstruction amount and span structure, not a condition that removes all information about the true edit positions. Section 6.1 states this limitation.

Revision. In the abstract, the introduction, and Section 6.1, we replaced statements that a component is required ("must") with statements of what the gain depends on in our controlled comparisons.

 

W4. Statistical uncertainty

We agree. Table 1 in the revision now reports the standard deviation over three seeds for every trained method. The values for the two rows you mention are below.

Table R3. DreamOn, Pass@1 (%), mean ± std over three seeds.

MethodTACOHumanEvalHumanEval+MBPPMBPP+
Repair SFT + Draft3.90 ± 0.0859.55 ± 1.2755.84 ± 1.0175.57 ± 0.9363.40 ± 0.55
VEDI-Min4.65 ± 0.4064.84 ± 2.8860.87 ± 1.1172.75 ± 0.7063.58 ± 0.81

The 0.18-point difference on MBPP+ is well within the seed variation, and we do not claim an advantage there; Section 5.2 states that the two are indistinguishable on MBPP+ and that Repair SFT + Draft is higher on MBPP. The advantage on DreamOn is on TACO, HumanEval, and HumanEval+.

Thank you again for the constructive comments. We would be glad to clarify any remaining points.

Thank you for the substantial additional experiments and careful clarification. The Repair SFT comparison reduces my concern about attributing the gains to the draft-conditioned baseline’s input mismatch. The fixed-length experiments are particularly useful: they strengthen the evidence for the supervision strategy while clarifying that a variable-length backbone is not necessary.

The additional DELETE ablation narrows the explanation without fully isolating the effect of real failures, as you acknowledge. Reporting seed variation and treating the small MBPP+ difference as inconclusive also appropriately addresses my statistical concern.

I retain my score of 3 (Weak accept), with increased confidence. The contribution is better supported, although task-dependent gains and remaining limitations of the controls keep my recommendation at weak accept.

Dear Reviewer Ru4V,

Thank you for your reply and for your increased confidence. Your suggestions on the fixed-length control and on reporting seed variation made the evidence stronger, and we will keep these additions in the final version.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

Dear Area Chair and Reviewers,

We thank all reviewers for the careful reviews and the discussion. For convenience, we summarize the changes in the revised paper. New or changed text is shown in blue.

ChangeRaised byLocation
Two fixed-length backbones (Dream-Coder 7B, DiffuCoder 7B) with the same edit supervisionRu4V W2; dYC5 W1/Q1; Y7Wk W5Sec. 5.2; App. B.5 (Table 13)
Teacher-Solution SFT, Teacher-Solution Edit, and RFT on matched data; revised interpretation of repair vs. reference targetsY7Wk W2Sec. 5.1; App. B.4 (Table 9)
Ablations of alignment and of each edit labeldYC5 Q2; Y7Wk W3, W5; Ru4V W3Sec. 6.1 (Table 2)
Linear probe on frozen backbone statesY7Wk W1/Q1Sec. 6.1 (Table 3b)
Repaired vs. unrepaired failures; gains by difficultyY7Wk W4App. C.3 (Fig. 10, Table 14)
Cost breakdown and budget-matched baselineY7Wk W6App. B.4 (Table 10)
Standard deviations over three seedsRu4V W4Table 1 and all new pass@1 tables
Inference input stated for all methodsRu4V W1; Y7Wk W3Sec. 4.2
Necessity wording replaced by what the gain depends onRu4V W3Abstract; Sec. 1; Sec. 6.1
Relation to repair distillation, edit supervision, and auxiliary trainingdYC5 W2Sec. 2.2; Sec. 2.3

We also corrected the size of FlexMDM to 8B; no result changes.

Thank you again for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.