- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
FlexDLLM alternates parallel diffusion with short autoregressive repairs. A trained diffusion adapter emits ASK tokens at predicted failure locations. A second adapter generates a repair conditioned on the denoising state. Teacher annotations and repair targets train the two stages. Experiments use LLaDA-8B-Instruct on six benchmarks.
Critical strengths
-
Selective sequential repair is a concrete hybrid design. The shared backbone avoids keeping a second large inference model resident. Hardware measurements include adapter switching.
-
The oracle analysis identifies a useful repair opportunity. Learned triggers outperform random and low-confidence triggers. Results span mathematical reasoning, code, and multiple-choice tasks.
Critical weaknesses
-
The comparisons do not fully isolate selective repair from teacher supervision. Vanilla SFT helps, but its data and targets are not specified well enough to establish a matched distillation control. A model trained on the same repaired trajectories could explain part of the gain. All experiments use LLaDA 1.0. Six benchmarks broaden task coverage, but cannot establish transfer across model families or generations. Compatibility with newer block-diffusion and editing models remains untested. A matched comparison with CoDiLA [1] is also missing.
-
The central analysis and reported configurations need clarification. Failure spans are labeled by o3 without a reported annotation audit. Table 3 uses the external teacher refiner. Its 85.1 score is not the deployed method’s 79.6 score. Repair limits also differ across descriptions: capacity 10 in Section 3.1 versus patch budget 30 in Table 5.
Questions for authors
-
What happens when SFT uses the identical teacher-corrected trajectories and training budget? Please compare RG-Diff alone and the complete method. Do the repair gains persist on a newer backbone under the same supervision budget? If not tested, what limits the claimed applicability?
-
Which refiner, patch capacity, and training configuration produced each main result? Please reconcile MBPP 51.2 in Table 1 with 54.2 in Section 4.1.
-
How accurate are the failure-span labels under independent checking? How sensitive are the learned trigger and gains to annotation errors?
-
The comparison with training-free correction methods is incomplete. WINO [2] revises suspicious drafts. ReMix [3] refines continuous token states. What advantage does learned AR repair provide under matched block sizes and generation limits? CoDiLA [1] is the closest hybrid comparator. The discussion should also distinguish joint prediction in CoDD [4] and dependency-aware selection in DEMASK [5]. Quality, total latency, and supervision costs should be separated.
[1] Locally Coherent Parallel Decoding in Diffusion Language Models
[2] Wide-In, Narrow-Out: Revokable Decoding for Efficient and Effective DLLMs
[3] Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inference
[4] Breaking the Factorization Barrier in Diffusion Language Models
[5] Dependency-Guided Parallel Decoding in Discrete Diffusion Language Models
7 comments
Replying to Response to Reviewer ih8k [Part 2]
W1 (2) & Q1 (2). Transfer to other backbones and newer block-diffusion/editing models, and comparison with CoDiLA.
Response: We thank the reviewer for this suggestion. During the rebuttal, we added experiments on two new backbones, together with a matched comparison with CoDiLA.
[Experiment on new model family: Dream-Coder-7B.] We follow the same pipeline as in the paper, and train FlexDLLM from scratch on Dream-Coder's own decoding trajectories. RG-Diff data: Dream-Coder generates answers for the first 10K Python problems in Magicoder-OSS-Instruct (4 tokens per step, length 256), and we record the decoding process. o3 then locates the first error and gives a fix. This gives 5,279 RG-Diff training samples. SC-AR data: We decode 2,000 problems with the trained RG-Diff adapter, and Qwen3-4B writes a patch for each <ASK>. This gives 5,994 patches to train the SC-AR adapter. The adapter sizes and training settings are the same as in the paper.
We test on HumanEval/HumanEval+ (generation length 768) and MBPP/MBPP+ (length 512), with 4 and 8 tokens per step. The baseline is the original Dream-Coder with the same decoder but no adapters. CoDiLA results are taken from its paper.
As shown below, FlexDLLM again performs best on Dream-Coder, and outperforms CoDiLA in 7 of the 8 settings under the same conditions. This is because FlexDLLM's find-and-fix framework is general and does not depend on a specific model architecture.
| Tokens per step | Method | HumanEval | HumanEval+ | MBPP | MBPP+ |
|---|---|---|---|---|---|
| 4 | Dream-Coder | 24.4 | 24.4 | 33.8 | 30.0 |
| 4 | CoDiLA | 51.8 | 48.8 | 55.6 | 45.5 |
| 4 | FlexDLLM | 58.2 | 55.3 | 62.6 | 54.3 |
| 8 | Dream-Coder | 8.5 | 8.5 | 14.0 | 12.5 |
| 8 | CoDiLA | 29.9 | 28.7 | 40.0 | 27.0 |
| 8 | FlexDLLM | 31.3 | 28.1 | 42.4 | 31.6 |
[Experiment on Newer block-diffusion model with self-editing: LLaDA2.1-mini.] To test compatibility with newer block-diffusion and editing models, we run experiments on LLaDA2.1-mini.
- Data and training: The model generates answers for all 7,473 GSM8K training problems and the first 5,000 Python problems in Magicoder, using the official Quality mode, and we record the trajectories (including every self-edit). After o3 annotation, we get 398 (GSM8K) and 1,687 (Magicoder) RG-Diff training samples. RG-Diff is trained with a global batch size of 96 (4 epochs on Magicoder). Qwen3-4B then writes 1,095 and 4,996 patches, respectively, to train SC-AR with a global batch size of 128 for 3 epochs.
- Decoder: LLaDA2.1-mini decodes differently from LLaDA 1.0: it generates from left to right in blocks of 32 tokens, and it edits tokens that were already written (self-editing). To work with this, we embed FlexDLLM's
<ASK>trigger and repair into block-wise generation, with four rules: (1)<ASK>can only be emitted when filling a masked position, and self-editing cannot turn an existing token into<ASK>; (2) after the SC-AR patch is written back at the<ASK>position, all positions to its right are reset to masks and regenerated; (3) after the patch is written back, the original block structure is kept, and decoding resumes from the block where the patch ends; (4) the patch region is locked and cannot be changed by self-editing, so the repair is not overwritten. - LoRA placement: Since LLaDA2.1-mini uses an MoE architecture, we add LoRA only to the attention layers and leave the expert layers unchanged.
We test on GSM8K and HumanEval with generation length 256, and compare LLaDA2.1-mini with only its built-in editing against built-in editing + FlexDLLM. As shown below, FlexDLLM works well with this newest block-diffusion model with self-editing, and clearly improves both math and coding (+9.0 on GSM8K, +10.4 on HumanEval). This is because the find-and-fix skill learned by FlexDLLM fits well into different dLLM decoding frameworks, and it complements the model's built-in editing.
| Method | GSM8K | HumanEval |
|---|---|---|
| LLaDA2.1-mini (built-in editing only) | 70.3 | 48.8 |
| LLaDA2.1-mini (built-in editing + FlexDLLM) | 79.3 | 59.2 |
We have added these results to Appendix F.5 (Tables 11 and 12). Thanks for the suggestion.
Replying to Response to Reviewer ih8k [Part 3]
W2 (1) & Q3. Quality of the o3 failure-span labels, and sensitivity to label errors.
Response: We thank the reviewer for pointing this out. During the rebuttal, we manually audited a random sample of o3's failure-span labels, and tested how sensitive FlexDLLM is to annotation errors.
[Manual audit of the labels.] We select 150 problems from the training set (50 math, 50 code, and 50 commonsense), and manually check whether o3's labeled failure span and failure step are correct. We find no clear labeling errors. For 19 problems (12.7%), the labeled failure step is one decoding step earlier or later than the true one, but the labeled span itself is still basically correct.
[Experiment: Sensitivity to annotation errors.] We add noise to the "where to emit <ASK>" labels in the Stage I training data, with two types of noise:
- Position shift: we randomly move the
<ASK>position of every sample left or right by 2, 4, 8, or 16 tokens. This simulates labels that are "roughly right but not precise". - Random replacement: we randomly pick 10%, 20%, or 40% of the samples and move their
<ASK>to a completely random position. This simulates labels at "the wrong place".
For each setting, we retrain RG-Diff with exactly the same configuration as the original, and report the accuracy and the average number of <ASK> calls per question on GSM8K.
| Label noise | None | Shift ±2 | Shift ±4 | Shift ±8 | Shift ±16 | Random 10% | Random 20% | Random 40% |
|---|---|---|---|---|---|---|---|---|
| GSM8K accuracy | 79.6 | 78.7 | 80.2 | 78.1 | 76.4 | 78.2 | 76.5 | 72.8 |
It can be seen that small position shifts (up to ±8 tokens) barely change the accuracy. When part of the labels are randomly wrong, the accuracy stays fairly stable at 10–20% noise and drops more at 40%. This shows that FlexDLLM is robust to small annotation errors. Therefore, the ±2-token shifts in only 12.7% of o3's labels have a negligible effect on the final performance: even shifting every label by ±2 tokens only changes the accuracy from 79.6 to 78.7.
We have added these experiments and analyses to Appendix F.4 (Table 10).
W2 (2). Table 3's 85.1 uses the external refiner, not the deployed method (79.6).
Response: We thank the reviewer for pointing this out. The 85.1 in Table 3 is the result when the model emits <ASK> by itself, but the patches are written by the external Qwen3-4B-Instruct. SC-AR is distilled from this model, so 85.1 can be seen as the upper bound of our method. Table 3 uses Qwen as the refiner for all rows because it compares how well different strategies choose the <ASK> positions. For a fair comparison, all strategies use the same model to write patches after <ASK>, so any difference comes only from where <ASK> is placed.
The deployed FlexDLLM uses only one model, and switches between two LoRAs for diffusion decoding and patch repair. Its GSM8K result is 79.6. This is our main result and how the method is used in practice. We have clarified this in the captions of Table 1 and Table 3, and in Appendix E (Table 6).
W2 (3). Inconsistent repair limits.
Response: We thank the reviewer for catching this inconsistency. We have checked the details: the patch length is 10 in both training and inference. The 30 in the appendix was a typo, and we have fixed it in the revised version (Table 5 and Appendix D.2.2).
Replying to Response to Reviewer ih8k [Part 1]
Response: We thank the reviewer for this insightful question and suggestion. To clarify the source of our performance gains, we conducted new experiments to explicitly disentangle the contribution of FlexDLLM's core mechanism from the supervision signal.
[Experiment: Disentangling FlexDLLM and Supervision.] Following the reviewer's constructive suggestion, we added two new baselines during the rebuttal:(1) Matched SFT: We train a baseline directly on the exact same teacher-corrected trajectories (from o3 and Qwen3) used by FlexDLLM, under an identical training budget (batch size 96, 200 steps).(2) RG-Diff alone: We run inference with the trained RG-Diff adapter but strictly block the
<ASK>token (setting its logit to −∞). Consequently, the SC-AR repair is never triggered, reducing the decoding process to that of the original LLaDA.From the result above, we can draw two conclusions: (1) RG-Diff alone degrades performance. When the model is restricted to only identifying error locations without the ability to correct them, its performance drops even below the original LLaDA baseline. This confirms that merely knowing where an error happens is useless without the capacity to fix it. (2) Matched SFT yields limited gains. Passively feeding the model correct trajectories is insufficient. Under this setting, the model never learns how to proactively discover and rectify its own errors during inference. These results show that every component of FlexDLLM is needed. FlexDLLM gets consistent gains because the model learns both to find errors and to fix them.
In addition, to reduce the reliance on external models, we also built an external-model-free version of FlexDLLM during the rebuttal. It uses simple rules to find and fix errors, and still brings large gains. Please see our response to Reviewer inWy (W1 (1) and W1 (2)) for details. We have added these experiments and analyses to Appendix F.1 and F.2 (Tables 7 and 8).