- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes VEDI, a post-training method for variable-length diffusion language models that learns from the model's own failed code. A teacher repairs failed programs sampled on TACO, and execution tests keep only verified repairs. VEDI aligns each failure with its repair, including insertions and deletions, and trains one backbone to predict where to edit and to reconstruct only the edited regions. At inference, the model uses its original sampler with no teacher, auxiliary selector, or extra repair stage. On FlexMDM and DreamOn (both 7B), VEDI improves over the base model on all five benchmarks (TACO, HumanEval, HumanEval+, MBPP, MBPP+) and exceeds full-rewrite SFT trained on the same failures and repairs by 6.3 and 5.0 HumanEval+ points.
Critical strengths
The paper is clearly written, and its components serve one coherent idea: keep what is right in a failed response and focus supervision on the changes it needs. The full-rewrite baselines use the same failures and verified repairs, and the controls (synthetic failures, reference targets, random reconstruction regions, selector gradients) make a convincing case that the gain comes from real failures, failure-specific repairs, and edit-location supervision. The method also adds no cost at inference.
Critical weaknesses
-
Unclear dependence on variable-length modeling. Aligned repair supervision, targeted reconstruction, and the auxiliary edit-location loss could also be applied to a fixed-length dLLM such as LLaDA or Dream. Without that controlled comparison, the experiments establish the value of edit-structured supervision but not that variable-length modeling is needed. The paper should also separate the benefit of aligning responses of different lengths from the benefit of the backbone's native insert and delete operations.
-
Incremental novelty. The method combines established ideas from repair distillation, edit supervision, and auxiliary training. This is a minor concern given the clear formulation and the supporting evidence, but the paper should state its contribution relative to these lines of work more precisely.
Overall, the paper presents a clear, focused, and well-supported contribution. The novelty is incremental, but the coherent method and meaningful ablations make the work valuable, and I lean toward acceptance with the minor revisions above.
Questions for authors
- Would the same supervision give comparable gains on a fixed-length dLLM (e.g., LLaDA or Dream) trained on the same failures and repairs? If not tested, please clarify that the experiments establish the value of edit-structured supervision while leaving the necessity of variable-length modeling open.
- How much of the gain comes from aligning responses of different lengths, and how much from the native insert and delete operations of FlexMDM and DreamOn?
4 comments
Q2. Gain from alignment and from the native operations
Gain from alignment. We trained a variant that builds the edit labels without alignment, by comparing the tokens at the same position. Everything else is the same as VEDI.
Table R2. Alignment ablation. Pass@1 (%), mean ± std over three seeds.
| TACO | HumanEval+ | MBPP+ | |
|---|---|---|---|
| FlexMDM | |||
| VEDI-Min w/o alignment | 2.14 ± 0.20 | 43.75 ± 1.51 | 53.15 ± 1.59 |
| VEDI-Min | 3.54 ± 0.28 | 51.04 ± 0.65 | 58.05 ± 0.77 |
| DreamOn | |||
| VEDI-Min w/o alignment | 3.32 ± 0.14 | 52.44 ± 2.20 | 60.58 ± 0.95 |
| VEDI-Min | 4.65 ± 0.40 | 60.87 ± 1.11 | 63.58 ± 0.81 |
Without alignment, the score drops on all three benchmarks. On HumanEval+ the drop is 7.29 points on FlexMDM and 8.43 on DreamOn.
The labels show why. When a repair inserts or removes a token, every later token shifts position. Without alignment, these tokens receive a REMASK label although they did not change. The share of failed-program tokens labeled KEEP falls from 74.5% to 39.1% on FlexMDM and from 73.4% to 34.0% on DreamOn.
Gain from the native insert and delete operations. These operations are part of the backbone and cannot be switched off within a model. We therefore use the closest comparison available. The DreamOn checkpoint we use was fine-tuned from Dream-Coder 7B to learn [expand] and [delete]. The two models have different Base scores, so we compare how far VEDI exceeds Repair SFT within each model. Repair SFT is trained on the same failures and repairs, without edit supervision.
Table R3. VEDI − Repair SFT, pass@1 points.
| Backbone | HumanEval+ | MBPP+ |
|---|---|---|
| Dream-Coder (no length-changing operations) | +5.48 | +0.53 |
| DreamOn | +8.02 | +3.88 |
The gain appears without the operations, but it is larger on DreamOn.
However, we cannot attribute this difference entirely to the operations. DreamOn went through additional fine-tuning, and the training data were built from each model's own failures.
The alignment ablation is in Section 6.1 (Table 2) of the revision, and the Dream-Coder results are in Section 5.2 and Appendix B.5 (Table 13).
Thank you again for the clear and helpful suggestions. We would be glad to clarify any remaining points.
Thank you for your reply. The VEDI's contribution is clear to me now
Dear Reviewer dYC5,
Thank you for your reply and for raising your score. Your suggestions on the fixed-length comparison and on the relation to prior work made the contribution clearer, and we will keep these additions in the final version.
Dear Reviewer dYC5,
Thank you for your thoughtful review and clear suggestions. We summarize our responses and the corresponding revisions below. New or changed text in the revised paper is shown in blue. We also corrected the size of FlexMDM, which is 8B, not 7B as the original paper stated; no result changes.
W1 / Q1. Dependence on variable-length modeling
We added the fixed-length comparison you suggested, on two masked diffusion code models with no length-changing operations: Dream-Coder 7B, the code model of the Dream family, and DiffuCoder 7B. The selector loss and the edit-region reconstruction loss are unchanged. Because the method uses the model's own failures, the data were rebuilt from each model's failures. The teacher, the repair procedure, and the training and evaluation protocol follow the other backbones.
Table R1. Fixed-length backbones. Pass@1 (%), mean ± std over three seeds.
On both models VEDI is the highest trained method on all five benchmarks. It exceeds Repair SFT on HumanEval+ by 5.48 points on Dream-Coder and 5.69 on DiffuCoder. On HumanEval and HumanEval+ the full-rewrite methods (Reference SFT and Repair SFT) fall below Base, whereas VEDI rises above it. On Dream-Coder, every trained method is below Base on MBPP and MBPP+, and VEDI loses the least (0.61 and 0.70 points).
A variable-length backbone is therefore not necessary for this gain. As you suggest, these results support the value of edit-structured supervision and do not show that variable-length modeling is required. The revision adds them in Section 5.2 and Appendix B.5 (Table 13). The separation between alignment and the native operations is discussed under Q2 below.
W2. Relation to prior work
We agree. VEDI builds on these three lines of work, and the revised Section 2 states the relation to each more precisely (Sections 2.2 and 2.3).
Prior methods either learn the corrected program as a complete rewrite or apply the predicted edits at inference. VEDI derives its edit targets from execution-verified repairs of the student's own failures and uses them only as training targets. We also state that the EditSelector follows auxiliary-task training.
[continued below]