- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents the Semi-Continuous Diffusion Language Model (SC-dLLM), a framework for converting pretrained autoregressive (AR) LLMs into blockwise continuous-state diffusion generators with a small additional training budget. Within each block, every unresolved position evolves from the mask embedding to its clean token embedding along a Brownian bridge. Training follows a two-stage curriculum: a continuous endpoint initialisation with complementary masking, followed by a mixture of endpoint updates and tokenwise-independent bridge corruption, all under the same-position cross-entropy objective with the pretrained vocabulary head. At inference, the predicted mean embedding drives a bridge SDE, and confidence-based early commitment discretises positions once they become sufficiently certain. Empirically, with approximately 1.1B additional training tokens, SC-dLLM achieves comparable performance to Fast-dLLM v2 and the same-data AR fine-tuning controls, with the most consistent gains on code generation.
Critical strengths
- Converting a pretrained AR backbone into a continuous-state generator with only approximately 1.1B tokens is practically appealing.
- The method is simple and easy to implement. Intermediate states are sampled analytically from dataset tokens and trained with a plain cross-entropy objective, avoiding additional velocity-regression losses.
Critical weaknesses
-
Limited Novelty: My main concern lies in the actual contribution of this paper. The overall framework is highly similar to Fast-dLLM v2 [1]: the AR initialisation, blockwise attention, complementary masking, sub-block decoding and confidence-based early commitment all follow [1]. The main difference is replacing the discrete masked states within the active block with continuous embeddings, which is adopted from Masked Language Flow Models [2]. In this regard, SC-dLLM reads as applying [2] to the AR-to-block-diffusion conversion recipe of [1].
-
Clarity of the Training Objective: The presentation of the core training objective is difficult to follow. In Eq. 5, and are only described verbally as "Equation 4 under continuous endpoint and independent-time corruption", while the corresponding corruption distributions are deferred to Appendix A.2 and the schedule of to Appendix A.3. In particular:
- The "continuous endpoint" branch only uses with complementary masking, i.e., it is essentially the discrete masked block-diffusion objective of Fast-dLLM v2. Calling it "continuous" is somewhat misleading.
I recommend writing both losses explicitly as expectations over their corruption distributions in the main text, together with a training algorithm, so that the objective is self-contained.
-
Unclear Benefit of Continuous States: Given that replacing discrete masks with continuous embeddings is the main difference from Fast-dLLM v2, it is crucial to demonstrate in which aspects the continuous states actually help. However, this remains unclear:
- The improvements over Fast-dLLM v2 in Table 1 are not significant, and the Fast-dLLM v2 numbers are published results obtained with different training data. More rigorous ablation studies are needed to justify the effectiveness of the continuous states. For example, a discrete counterpart trained with the same data and recipe (e.g., the Stage-I-only model decoded with the standard remasking sampler) would provide a controlled comparison.
- A potential advantage of continuous states is that the states of uncommitted positions can evolve during generation and carry partially resolved information across steps, whereas in masked diffusion an uncommitted position always remains a mask. This could be beneficial, yet no experiment examines this property in depth. For instance, the authors could (i) reset the uncommitted states to the mask embedding after every evaluation, which removes the carried information while keeping everything else unchanged, and (ii) analyse how the predictions of uncommitted positions evolve over the denoising steps before commitment.
Without such analyses, it is hard to tell whether the gains come from the continuous states or from the data and training recipe.
-
Evaluation Protocol Concerns:
- The threshold is selected based on the nine-score average in Figure 4, which appears to be computed on the same benchmarks reported in Table 1. This amounts to tuning on the test set. Notably, at (the setting of the published Fast-dLLM v2 results), SC-dLLM achieves 59.8, which is below Fast-dLLM v2 (60.3). For a fair comparison, Fast-dLLM v2 should also undergo a comparable confidence threshold search under the same setting as Figure 4.
[1] Wu, Chengyue, et al. "Fast-dLLM v2: Efficient block-diffusion LLM." ICLR (2026).
[2] Azangulov, Iskander, et al. "Masked language flow models." arXiv preprint arXiv:2606.27617 (2026).
Questions for authors
- How often do positions actually traverse interior continuous states before being committed? Could you report statistics such as the fraction of tokens committed at the first evaluation (, i.e., directly from the mask embedding) and the average number of evaluations per token? If most tokens are committed after only a few steps under , the sampler would behave close to a discrete masked decoder.
- With and unnormalised embeddings (Appendix A.2), what is the scale of the bridge noise relative to the distances between token embeddings? An ablation over , including where the bridge reduces to a deterministic interpolation, would help clarify the role of stochasticity.
- Training samples times independently across positions (Eq. 8), whereas at inference, positions follow the scheduled clock in Eq. 10 and committed positions jump directly to . Is there a train–test mismatch in the joint distribution of times within a block? How sensitive is the performance to the choice of in Eq. 10?
- For a fair comparison, could you report Fast-dLLM v2 with its threshold selected in the same way as SC-dLLM
6 comments
Response to Reviewer AgaP
Q1: Interior-state use and prediction evolution
A4:
We added trajectory measurements and sampling ablations to examine continuous refinement and early commitment.
-
Use of continuous states. At 7B and , we decode S=8 tokens per sub-block using at most K=32 denoising steps. Confident predictions commit immediately; the remaining positions retain continuous states for further refinement (Section 4.4, new Figure 5, Table A8).
Task Commit at first denoising step Receive continuous updates Mean denoising steps per token HumanEval 67.24% 32.76% 5.19 MBPP 39.79% 60.21% 10.02 GSM8K 62.95% 37.05% 4.72 The counts include the first denoising step from MASK. Subsequent denoising steps read the continuous states updated between predictions.
-
Prediction changes. At , 17.02%–37.47% of positions change their top-1 prediction before commitment, and over 90% of predictions made one denoising step before commitment agree with the final token. Predictions evolve and stabilize during refinement. Resetting unresolved states to MASK reduces performance, linking this retained information to generation quality (Section 4.4, Figure 5, Table A4; Section B.4, Figure A2, Table A7).
-
Early commitment. Lowering from 1.0 to 0.9 reduces the mean number of denoising steps between prediction stabilization and commitment to 1.65–3.19 on HumanEval and MBPP. A prediction is stable once it remains unchanged through commitment (Section 4.4, Table A7).
-
Denoising steps. At , increasing K from 8 to 32 substantially improves performance, with little further gain at K=64. Additional denoising steps allow continuous states to refine before commitment (Section B.3, Figure A1).
-
Sub-block size. Increasing S from 8 to 32 at raises the fraction receiving continuous updates to over 80% on both HumanEval and MBPP (Section B.6, Table A11).
-
Position within the block. At , early commitment reduces continuous state updates across all four 8-token sub-blocks within each 32-token block (Section B.6, Table A12).
Q2: Noise scale and σ=0
A5:
We added noise measurements and ablations to quantify its magnitude and performance contribution.
- Noise relative to embeddings. At and , accumulated noise is 5.86–7.21× the distance from the predicted token embedding to its nearest neighbor. Noise also exceeds token embedding norms and the distance to MASK, showing that these intermediate states are dominated by noise rather than MASK (Section B.5, Figure A3, Table A10).
- Training noise ablation. At 1.5B and , improves the average by 7.28 points over . Stronger noise reduces performance, supporting the selected scale (Section 4.4; Section B.2, Table A5).
- Inference noise ablation. At , removing noise while retaining continuous states lowers the average at both 1.5B and 7B. Noise thus contributes during generation itself (Section 4.4; Section B.1, Table A4).
- Noise during refinement. The deterministic update moves toward the average token embedding weighted by predicted probabilities. Noise dominates the updates in the middle of denoising and approaches the deterministic update's magnitude near completion (Section B.5, Figure A4).
Q3: Training times and inference clocks
A6:
We updated the blockwise training setting for interior denoising: tokens within a block share one sampled denoising time t, while different blocks sample t independently (Sections 3.2 and A.2–A.4). Endpoint reconstruction uses masks and clean tokens. At inference, controls each position's progression from t=0 toward t=1, and committed positions move to t=1.
- Inference clock. SC-dLLM maintains similar performance across different configurations, demonstrating robustness to the inference clock. The sweep tests increasing, decreasing, or remaining constant across positions, as well as a wider range, with , a budget of K=32 denoising steps per sub-block, and S=8 tokens per sub-block (Section B.3, Figure A1).
- Training mixture. The 1:1 mixture of Endpoint reconstruction and interior denoising gives the highest average, supporting training on both noisy continuous states and the clean-token context provided by committed positions (Section B.2, Table A5).
Response to Reviewer AgaP
W4 / Q4: Comparable threshold selection
A7:
-
Main comparison. Updated Table 1 uses for both methods. SC-dLLM achieves a 7B average of 63.89 versus official Fast-dLLM v2's 60.3 and Fast-dLLM v2-FT's 61.0, establishing the advantage at the published baseline threshold (Sections 4.1–4.2).
-
Matched threshold sweep. We evaluated SC-dLLM and official Fast-dLLM v2 over {0.3, 0.5, 0.7, 0.9, 1.0}. At the shared , SC-dLLM achieves higher HumanEval Plus and MBPP Plus scores and faster decoding on all three tasks. SC-dLLM at also outperforms Fast-dLLM v2 at in average (62.45 vs. 60.3) and throughput, supporting the choice of (Figure 6, Section 4.5; protocol in Section A.8; Tables 1 and A4).
The table below compares the 7B models at the shared :
Benchmark Score gain (points) Throughput ratio HumanEval Plus +4.88 1.45× MBPP Plus +6.61 1.21× GSM8K +2.65 1.30× -
Prediction stability. At , predictions stabilize at denoising step 10.48 on average on HumanEval and 12.06 on MBPP. During denoising steps 24–31, predictions at 94.45%–97.87% of unresolved positions remain unchanged from the preceding step. commits stable predictions earlier, saving denoising steps while unresolved positions continue to refine (Section 4.4, Figure 5; Section B.4, Figure A2, Table A7).
Response to Reviewer AgaP
We thank the reviewer for recognizing the practical conversion budget and the simplicity of token reconstruction.
W1: Limited novelty
A1:
SC-dLLM provides an efficient AR-to-continuous conversion procedure. Three new experiments establish the value of its design:
Our conversion procedure provides substantially higher performance and efficiency than this combination. Throughput compares SC-dLLM-7B at τ=0.9 with the 1.5B combination on the same H200, batch size 256, and timing protocol (Section 4.4, Section B.2, Tables 1 and A6).
W2: Explicit training objectives and endpoint terminology
A2:
W3: Benefit of carrying continuous states
A3:
We added three controlled comparisons to isolate the benefit of continuous states:
Average scores: