acceptodds
Under review as a conference paper at ICLR 2027

Into the Continuum: Efficient Semi-Continuous Diffusion LLM

Abstract

Continuous diffusion language models retain partially resolved token information across denoising steps. We study whether pretrained autoregressive (AR) LLMs can acquire this generation mechanism with limited additional data and training. Our efficiency focus is this conversion budget; cross-model inference speed remains to be measured. We present Semi-Continuous Diffusion Language Model (SC-dLLM), the first low-data conversion of pretrained AR LLMs to blockwise continuous-state generation. Each block starts from masks, evolves through continuous states, and yields discrete tokens. A brief continuous endpoint initialization prepares the backbone for same-position prediction. The main adaptation then trains on independently corrupted intermediate states while retaining the vocabulary prediction objective. During generation, token predictions guide stochastic updates and determine when confident positions are committed. Training uses analytically sampled states without an online teacher or unrolled generation trajectories. The 7B conversion uses approximately 1.1B additional training tokens, about of the 45.2B effective tokens reported for ELF's OpenWebText training. Across code generation, mathematics, instruction following, and knowledge-intensive QA, SC-dLLM reaches nine-score averages of 45.2 at 1.5B and 61.2 at 7B, exceeding the reported AR-initializer and Fast-dLLM v2 averages at both scales. At 7B, all four HumanEval and MBPP Base/Plus scores exceed both the same-data AR fine-tuning control and Fast-dLLM v2. These results establish the feasibility of adapting AR backbones to blockwise continuous generation with a small additional training budget using the pretrained vocabulary head.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Language Model, Continuous Diffusion, Data Efficiency, Training Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AgaPOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents the Semi-Continuous Diffusion Language Model (SC-dLLM), a framework for converting pretrained autoregressive (AR) LLMs into blockwise continuous-state diffusion generators with a small additional training budget. Within each block, every unresolved position evolves from the mask embedding to its clean token embedding along a Brownian bridge. Training follows a two-stage curriculum: a continuous endpoint initialisation with complementary masking, followed by a mixture of endpoint updates and tokenwise-independent bridge corruption, all under the same-position cross-entropy objective with the pretrained vocabulary head. At inference, the predicted mean embedding drives a bridge SDE, and confidence-based early commitment discretises positions once they become sufficiently certain. Empirically, with approximately 1.1B additional training tokens, SC-dLLM achieves comparable performance to Fast-dLLM v2 and the same-data AR fine-tuning controls, with the most consistent gains on code generation.

Critical strengths
  • Converting a pretrained AR backbone into a continuous-state generator with only approximately 1.1B tokens is practically appealing.
  • The method is simple and easy to implement. Intermediate states are sampled analytically from dataset tokens and trained with a plain cross-entropy objective, avoiding additional velocity-regression losses.
Critical weaknesses
  • Limited Novelty: My main concern lies in the actual contribution of this paper. The overall framework is highly similar to Fast-dLLM v2 [1]: the AR initialisation, blockwise attention, complementary masking, sub-block decoding and confidence-based early commitment all follow [1]. The main difference is replacing the discrete masked states within the active block with continuous embeddings, which is adopted from Masked Language Flow Models [2]. In this regard, SC-dLLM reads as applying [2] to the AR-to-block-diffusion conversion recipe of [1].

  • Clarity of the Training Objective: The presentation of the core training objective is difficult to follow. In Eq. 5, Lend\mathcal{L}_{\text{end}} and Liid\mathcal{L}_{\text{iid}} are only described verbally as "Equation 4 under continuous endpoint and independent-time corruption", while the corresponding corruption distributions are deferred to Appendix A.2 and the schedule of λs\lambda_s to Appendix A.3. In particular:

    • The "continuous endpoint" branch only uses ti∈{0,1}t_{i} \in \{0, 1\} with complementary masking, i.e., it is essentially the discrete masked block-diffusion objective of Fast-dLLM v2. Calling it "continuous" is somewhat misleading.

    I recommend writing both losses explicitly as expectations over their corruption distributions in the main text, together with a training algorithm, so that the objective is self-contained.

  • Unclear Benefit of Continuous States: Given that replacing discrete masks with continuous embeddings is the main difference from Fast-dLLM v2, it is crucial to demonstrate in which aspects the continuous states actually help. However, this remains unclear:

    • The improvements over Fast-dLLM v2 in Table 1 are not significant, and the Fast-dLLM v2 numbers are published results obtained with different training data. More rigorous ablation studies are needed to justify the effectiveness of the continuous states. For example, a discrete counterpart trained with the same data and recipe (e.g., the Stage-I-only model decoded with the standard remasking sampler) would provide a controlled comparison.
    • A potential advantage of continuous states is that the states of uncommitted positions can evolve during generation and carry partially resolved information across steps, whereas in masked diffusion an uncommitted position always remains a mask. This could be beneficial, yet no experiment examines this property in depth. For instance, the authors could (i) reset the uncommitted states to the mask embedding mm after every evaluation, which removes the carried information while keeping everything else unchanged, and (ii) analyse how the predictions of uncommitted positions evolve over the denoising steps before commitment.

    Without such analyses, it is hard to tell whether the gains come from the continuous states or from the data and training recipe.

  • Evaluation Protocol Concerns:

    • The threshold τ=0.9\tau=0.9 is selected based on the nine-score average in Figure 4, which appears to be computed on the same benchmarks reported in Table 1. This amounts to tuning on the test set. Notably, at τ=1.0\tau=1.0 (the setting of the published Fast-dLLM v2 results), SC-dLLM achieves 59.8, which is below Fast-dLLM v2 (60.3). For a fair comparison, Fast-dLLM v2 should also undergo a comparable confidence threshold search under the same setting as Figure 4.

[1] Wu, Chengyue, et al. "Fast-dLLM v2: Efficient block-diffusion LLM." ICLR (2026).

[2] Azangulov, Iskander, et al. "Masked language flow models." arXiv preprint arXiv:2606.27617 (2026).

Questions for authors
  • How often do positions actually traverse interior continuous states before being committed? Could you report statistics such as the fraction of tokens committed at the first evaluation (k=0k=0, i.e., directly from the mask embedding) and the average number of evaluations per token? If most tokens are committed after only a few steps under τ=0.9\tau=0.9, the sampler would behave close to a discrete masked decoder.
  • With σ=0.2\sigma=0.2 and unnormalised embeddings (Appendix A.2), what is the scale of the bridge noise relative to the distances between token embeddings? An ablation over σ\sigma, including σ→0\sigma \to 0 where the bridge reduces to a deterministic interpolation, would help clarify the role of stochasticity.
  • Training samples times independently across positions (Eq. 8), whereas at inference, positions follow the scheduled clock in Eq. 10 and committed positions jump directly to t=1t=1. Is there a train–test mismatch in the joint distribution of times within a block? How sensitive is the performance to the choice of ρj\rho_j in Eq. 10?
  • For a fair comparison, could you report Fast-dLLM v2 with its threshold selected in the same way as SC-dLLM
6 comments
AuthorsOpenReview ↗

Response to Reviewer AgaP

We thank the reviewer for recognizing the practical conversion budget and the simplicity of token reconstruction.

W1: Limited novelty

A1:

SC-dLLM provides an efficient AR-to-continuous conversion procedure. Three new experiments establish the value of its design:

  1. Direct combination with MLFM.
  • MLFM setup. Following MLFM’s official configuration, we continued training the converted Fast-dLLM v2-1.5B checkpoint, already a masked diffusion model (MDM), for 1.1B tokens on our data.

Our conversion procedure provides substantially higher performance and efficiency than this combination. Throughput compares SC-dLLM-7B at τ=0.9\tau=0.9 with the 1.5B combination on the same H200, batch size 256, and timing protocol (Section 4.4, Section B.2, Tables 1 and A6).

MetricHumanEvalMBPP
SC-dLLM-1.5B Plus score41.5041.70
Fast-dLLM v2-1.5B + MLFM Plus score8.107.03
Throughput ratio (SC-dLLM-7B / combination-1.5B)20.79×20.01×
  1. Endpoint initialization. With the same subsequent 1:1 mixture of Endpoint reconstruction and interior denoising, endpoint initialization improves the 1.5B average by 3.67 points. Training the AR model to reconstruct tokens at their original positions improves its adaptation to continuous inputs (Sections 3.2 and 4.4, Table A5).
  2. Training mixture. Equal proportions of Endpoint reconstruction and interior denoising achieve the highest average. This supports combining continuous reconstruction with the clean-token context used after commitment (Section B.2, Table A5).

W2: Explicit training objectives and endpoint terminology

A2:

  1. Training objectives. Endpoint reconstruction follows Fast-dLLM v2's masked-token cross-entropy loss. Interior denoising uses the same loss to reconstruct tokens from noisy continuous inputs (Section 3.2, Equations (4)–(5)).
  2. Endpoint initialization. “Endpoint” denotes t=0 (mask embedding) and t=1 (clean-token embedding), while interior denoising trains on intermediate states with 0<t<10<t<1 (Sections 3.1–3.2).
  3. Manuscript changes. We added explicit formulas for both training losses in Equations (9)–(10) and Algorithm 1 in Section A.3. Updated Section A.4 specifies 1,000 updates of Endpoint reconstruction followed by a 1:1 mixture with interior denoising.

W3: Benefit of carrying continuous states

A3:

We added three controlled comparisons to isolate the benefit of continuous states:

  1. Matched training. SC-dLLM outperforms Fast-dLLM v2-FT with the same Qwen initialization, data, and 1.1B-token budget (Sections 4.1–4.2, updated Table 1).
  2. State reset. Resetting unresolved states to MASK after each denoising step lowers the average at both scales. This directly establishes the benefit of retaining intermediate information (Section 4.4, Table A4).
  3. Discrete sampling. SC-dLLM also performs better with continuous sampling than with Fast-dLLM v2's discrete sampler, which keeps unresolved positions masked between denoising steps (Section 4.4, Table 2; Section B.1, Table A4).

Average scores:

Model / samplerτ\tau1.5B7B
SC-dLLM1.046.863.89
Fast-dLLM v2-FT1.045.361.0
Continuous0.946.0462.45
Reset to MASK0.943.9561.14
Continuous1.046.8663.81
Discrete1.044.7362.97
AuthorsOpenReview ↗

Response to Reviewer AgaP

Q1: Interior-state use and prediction evolution

A4:

We added trajectory measurements and sampling ablations to examine continuous refinement and early commitment.

  1. Use of continuous states. At 7B and τ=0.9\tau=0.9, we decode S=8 tokens per sub-block using at most K=32 denoising steps. Confident predictions commit immediately; the remaining positions retain continuous states for further refinement (Section 4.4, new Figure 5, Table A8).

    TaskCommit at first denoising stepReceive continuous updatesMean denoising steps per token
    HumanEval67.24%32.76%5.19
    MBPP39.79%60.21%10.02
    GSM8K62.95%37.05%4.72

    The counts include the first denoising step from MASK. Subsequent denoising steps read the continuous states updated between predictions.

  2. Prediction changes. At τ=0.9\tau=0.9, 17.02%–37.47% of positions change their top-1 prediction before commitment, and over 90% of predictions made one denoising step before commitment agree with the final token. Predictions evolve and stabilize during refinement. Resetting unresolved states to MASK reduces performance, linking this retained information to generation quality (Section 4.4, Figure 5, Table A4; Section B.4, Figure A2, Table A7).

  3. Early commitment. Lowering τ\tau from 1.0 to 0.9 reduces the mean number of denoising steps between prediction stabilization and commitment to 1.65–3.19 on HumanEval and MBPP. A prediction is stable once it remains unchanged through commitment (Section 4.4, Table A7).

  4. Denoising steps. At τ=1.0\tau=1.0, increasing K from 8 to 32 substantially improves performance, with little further gain at K=64. Additional denoising steps allow continuous states to refine before commitment (Section B.3, Figure A1).

  5. Sub-block size. Increasing S from 8 to 32 at τ=0.9\tau=0.9 raises the fraction receiving continuous updates to over 80% on both HumanEval and MBPP (Section B.6, Table A11).

  6. Position within the block. At τ=0.9\tau=0.9, early commitment reduces continuous state updates across all four 8-token sub-blocks within each 32-token block (Section B.6, Table A12).

Q2: Noise scale and σ=0

A5:

We added noise measurements and ablations to quantify its magnitude and performance contribution.

  1. Noise relative to embeddings. At σ=0.2\sigma=0.2 and τ∈[0.4,0.6)\tau \in [0.4,0.6), accumulated noise is 5.86–7.21× the distance from the predicted token embedding to its nearest neighbor. Noise also exceeds token embedding norms and the distance to MASK, showing that these intermediate states are dominated by noise rather than MASK (Section B.5, Figure A3, Table A10).
  2. Training noise ablation. At 1.5B and τ=1.0\tau=1.0, σ=0.2\sigma=0.2 improves the average by 7.28 points over σ=0\sigma=0. Stronger noise reduces performance, supporting the selected scale (Section 4.4; Section B.2, Table A5).
  3. Inference noise ablation. At τ=0.9\tau=0.9, removing noise while retaining continuous states lowers the average at both 1.5B and 7B. Noise thus contributes during generation itself (Section 4.4; Section B.1, Table A4).
  4. Noise during refinement. The deterministic update moves toward the average token embedding weighted by predicted probabilities. Noise dominates the updates in the middle of denoising and approaches the deterministic update's magnitude near completion (Section B.5, Figure A4).

Q3: Training times and inference clocks

A6:

We updated the blockwise training setting for interior denoising: tokens within a block share one sampled denoising time t, while different blocks sample t independently (Sections 3.2 and A.2–A.4). Endpoint reconstruction uses masks and clean tokens. At inference, ρ\rho controls each position's progression from t=0 toward t=1, and committed positions move to t=1.

  1. Inference clock. SC-dLLM maintains similar performance across different ρ\rho configurations, demonstrating robustness to the inference clock. The sweep tests ρ\rho increasing, decreasing, or remaining constant across positions, as well as a wider range, with τ=1.0\tau=1.0, a budget of K=32 denoising steps per sub-block, and S=8 tokens per sub-block (Section B.3, Figure A1).
  2. Training mixture. The 1:1 mixture of Endpoint reconstruction and interior denoising gives the highest average, supporting training on both noisy continuous states and the clean-token context provided by committed positions (Section B.2, Table A5).
AuthorsOpenReview ↗

Response to Reviewer AgaP

W4 / Q4: Comparable threshold selection

A7:

  1. Main comparison. Updated Table 1 uses τ=1.0\tau=1.0 for both methods. SC-dLLM achieves a 7B average of 63.89 versus official Fast-dLLM v2's 60.3 and Fast-dLLM v2-FT's 61.0, establishing the advantage at the published baseline threshold (Sections 4.1–4.2).

  2. Matched threshold sweep. We evaluated SC-dLLM and official Fast-dLLM v2 over {0.3, 0.5, 0.7, 0.9, 1.0}. At the shared τ=0.9\tau=0.9, SC-dLLM achieves higher HumanEval Plus and MBPP Plus scores and faster decoding on all three tasks. SC-dLLM at τ=0.9\tau=0.9 also outperforms Fast-dLLM v2 at τ=1.0\tau=1.0 in average (62.45 vs. 60.3) and throughput, supporting the choice of τ=0.9\tau=0.9 (Figure 6, Section 4.5; protocol in Section A.8; Tables 1 and A4).

    The table below compares the 7B models at the shared τ=0.7\tau=0.7:

    BenchmarkScore gain (points)Throughput ratio
    HumanEval Plus+4.881.45×
    MBPP Plus+6.611.21×
    GSM8K+2.651.30×
  3. Prediction stability. At τ=1.0\tau=1.0, predictions stabilize at denoising step 10.48 on average on HumanEval and 12.06 on MBPP. During denoising steps 24–31, predictions at 94.45%–97.87% of unresolved positions remain unchanged from the preceding step. τ=0.9\tau=0.9 commits stable predictions earlier, saving denoising steps while unresolved positions continue to refine (Section 4.4, Figure 5; Section B.4, Figure A2, Table A7).

Reviewer_x92yOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SC-dLLM, a method for converting pretrained autoregressive language models into semi-continuous diffusion language models. It retains the pretrained token embeddings and vocabulary head, introduces time conditioning, and trains on masked endpoints and Gaussian-corrupted intermediate embedding states using a token reconstruction objective. During blockwise generation, unresolved positions undergo continuous bridge updates, while confident predictions are committed as discrete tokens and provide context for subsequent predictions. Experiments on Qwen2.5-Instruct models at 1.5B and 7B demonstrate conversion with approximately 1.1B additional non-padding token exposures, competitive downstream performance, and improvements in code generation. The paper also evaluates the quality–throughput trade-off associated with early commitment.

Critical strengths

1. A practical, low-budget route from AR models to semi-continuous generation.

The method reuses pretrained embeddings and the vocabulary head while introducing continuous intermediate states under a token-level reconstruction objective. Demonstrating this conversion at both 1.5B and 7B with limited additional training is a useful contribution: it provides an accessible way to investigate continuous-state language generation without training a model from scratch.

2. Code-generation improvements supported by a same-data AR control.

The 7B model improves all four reported code scores over the same-data AR fine-tuning control. For example, HumanEval Base/Plus increases from 53.5/50.2 to 65.7/61.2. This provides evidence that the complete conversion recipe offers benefits beyond simply exposing the AR model to additional training data, even though the contribution of the continuous mechanism remains unresolved.

Critical weaknesses

1. The additional benefit of continuous-state generation over discrete block diffusion remains unclear.

The experiments establish that the complete conversion recipe works, but do not adequately isolate the contribution of continuous-state evolution. The method combines Gaussian bridge corruption, soft embedding feedback, a training curriculum, blockwise conditioning, and confidence-based commitment. Without a controlled discrete counterpart or targeted ablations, it is difficult to determine which components account for the observed gains.

The closest discrete baseline, Fast-dLLM v2, is represented by published results rather than a matched rerun. Section 4.1 also states that its main results use τ=1.0, whereas SC-dLLM uses τ=0.9. Thus, differences in commitment policy further complicate attribution. I recognize that the paper positions its main result as conversion feasibility; nevertheless, the lack of controlled evidence limits what can be learned about the value of its central continuous-state design.

2. The inference-efficiency evaluation lacks essential configuration and timing details.

Figure 4 reports throughput without specifying the inference GPU model and count, DP/TP configuration, or throughput aggregation procedure. The eight H200 GPUs listed in Appendix A.1 describe the training configuration, not an explicit inference setup. These omissions make the absolute throughput difficult to reproduce or interpret.

Appendix A.4 further states that completed nonterminal blocks require an additional computation to construct their final KV cache. This computation is excluded from the denoising-evaluation count, but the paper does not clarify whether its time is included in Figure 4. This does not establish that the measurement omitted the cost; it leaves the accounting unresolved. Moreover, the reported 2.54× speedup compares early commitment against the same model's full schedule, rather than against a discrete baseline under matched hardware and timing conditions. The current evidence therefore does not establish a practical inference-efficiency advantage over discrete block diffusion.

Questions for authors
  1. Do you have a discrete block-diffusion control using the same initialization, training data and budget, block/sub-block configuration, and a comparably tuned commitment policy? Are there ablations that distinguish the contribution of Gaussian noise from that of continuous soft-state feedback? Such evidence would help determine whether the gains can be attributed to the continuous mechanism.

  2. What GPU model and count, numerical precision, DP/TP configuration, and per-replica batch size were used for Figure 4? Is the reported throughput measured per request, per replica, or aggregated across replicas?

  3. What are the exact timing boundaries for Figure 4? In particular, are prompt prefill and block-boundary KV recomputation/cache updates included? Please clarify the contribution of KV recomputation to generation time and whether a discrete baseline has been measured under the same hardware and timing protocol.

3 comments
AuthorsOpenReview ↗

Response to Reviewer x92y

We thank the reviewer for recognizing the practical value of AR-to-semi-continuous conversion and the same-data AR comparison.

Q1 & W1: Matched discrete control, state feedback, and noise

A1:

We added matched controls and ablations to evaluate the contributions of continuous states and Gaussian noise.

  1. Matched discrete control. SC-dLLM outperforms Fast-dLLM v2-FT with the same Qwen initialization, data, 1.1B-token budget, and threshold (Sections 4.1–4.2, updated Table 1).
  2. State reset. Resetting unresolved positions to MASK after each denoising step reduces performance at both scales. Retaining intermediate information therefore benefits generation (Section 4.4, Table A4).
  3. Noise removal. Removing Gaussian noise (σ=0\sigma=0) during training and inference while retaining continuous states reduces performance, showing that noise improves continuous denoising (Tables A4–A5).

Average scores at τ=0.9\tau=0.9:

Model / samplerτ\tau1.5B7B
Continuous0.946.0462.45
Reset to MASK0.943.9561.14
Continuous without noise0.944.4861.23
  1. Noise magnitude. At σ=0.2\sigma=0.2 and t∈[0.4,0.6)t\in [0.4,0.6), accumulated noise is 5.86–7.21× the distance from the predicted token embedding to its nearest neighbor. Noise also exceeds token embedding norms and the distance to MASK, showing that these intermediate states are dominated by noise rather than MASK (Section B.5, Figure A3, Table A10).
  2. Threshold selection. We evaluated SC-dLLM and official Fast-dLLM v2 over the same thresholds {0.3, 0.5, 0.7, 0.9, 1.0}. Figure 6 now reports this comparison under the common protocol in Section A.8.

Q2 & W2: Hardware, precision, parallelism, and aggregation

A2:

We have specified the inference configuration and throughput aggregation in Section A.8:

  1. Hardware and precision. SC-dLLM-7B and Fast-dLLM v2-7B each use one NVIDIA H200, BF16 parameters and forward computation, DP=1, TP=1, and batch size 256 on all three benchmarks.
  2. Aggregation. Throughput is the single GPU's total generated-token rate across the batch.
  3. Presentation. Updated Figure 4 reports performance and throughput separately for HumanEval Plus, MBPP Plus, and GSM8K.

Q3 & W2: Timing boundaries, cache overhead, and baseline speed

A3:

  1. Timing. The timer covers denoising/decoding and excludes prompt prefill and block-boundary KV cache recomputation. Section A.8 now states these boundaries explicitly.

  2. Cache overhead. After each block except the last, we run a forward pass on its generated tokens to update the KV cache (Section A.5). With 8-token sub-blocks and τ=0.9\tau=0.9, prefill and cache recomputation together account for 5.24% of total forward-pass time on HumanEval and 2.94% on MBPP (Section B.6, Table A11).

  3. Discrete baseline. SC-dLLM-7B and Fast-dLLM v2-7B each use one H200, BF16, batch size 256, and the timing protocol above. SC-dLLM achieves higher HumanEval Plus/MBPP Plus performance and throughput than official Fast-dLLM v2 at the shared τ=0.9\tau=0.9. Compared with Fast-dLLM v2 at τ=1\tau=1.0, it also achieves a higher average and faster decoding on all three benchmarks. (Figure 6; Section 4.5; Table 1; Table A4)

    SC-dLLM at τ=0.9\tau=0.9 versus Fast-dLLM v2 at τ=1.0\tau=1.0, both at 7B:

    MetricComparison
    Average (SC-dLLM vs. Fast-dLLM v2)62.45 vs. 60.3
    HumanEval throughput ratio1.52×
    MBPP throughput ratio1.12×
    GSM8K throughput ratio1.53×

I thank the authors for the thorough rebuttal and the substantial set of new experiments. The revision directly addresses the two concerns in my review, and I am raising my score to 3 (weak accept).

W1 (contribution of continuous states). The matched Fast-dLLM v2-FT control (same Qwen initialization, data, 1.1B-token budget, and threshold) and the targeted ablations resolve my main concern. The reset-to-MASK experiment isolates the benefit of carrying intermediate information (7B average 62.45 -> 61.14 at tau=0.9), and the sigma=0 ablation separates the contribution of stochastic noise (62.45 -> 61.23). Together with the continuous-vs-discrete sampler comparison on the same model, these establish that the continuous mechanism genuinely helps, rather than the gains being attributable to the data or recipe alone. I also appreciate the three-seed evaluation and the matched threshold sweep.

W2 (inference-efficiency measurement). The inference configuration is now fully specified (one H200, BF16, DP=1/TP=1, batch size 256, single-GPU aggregate throughput), the timing boundaries are explicit (denoising/decoding only, excluding prompt prefill and block-boundary KV recomputation, quantified at 5.24%/2.94% of forward time), and a discrete baseline is measured under the same hardware and protocol. This resolves the reproducibility and accounting concerns I raised.

Remaining caveats (why weak accept rather than higher). My acceptance is tempered by three points that the rebuttal clarifies but does not remove. (1) Novelty: the continuous-state mechanism is the Brownian-bridge marginal of MLFM, and the blockwise layout and commitment policy follow Fast-dLLM v2; the contribution is primarily the low-data AR-to-continuous conversion recipe, which is useful but a combination of existing components. (2) The continuous-state gain is modest (~1-2 points), and a majority of tokens commit at the first denoising step (67.24% on HumanEval, 62.95% on GSM8K), so the mechanism actively refines only a minority of positions. (3) The reported throughput is batch-256 serving aggregate excluding prefill/KV recomputation, not single-request latency; the efficiency claim should be read within this scope.

These are limitations of scope and significance rather than soundness. The conversion is practical and low-budget, the mechanism is now validated by controlled experiments, and the evaluation is transparent. On balance I consider the paper a useful contribution and raise my score to 3.

Thank you for the careful reassessment and the updated rating. We appreciate your recognition that the controlled experiments establish the contribution of continuous states and that the revised evaluation resolves the measurement concerns. We provide further evidence on the three remaining points below.

1. Novelty over Fast-dLLM v2 and MLFM

SC-dLLM converts an AR LLM into a continuous dLLM using 1.1B non-padding tokens, or 1.5B packed token positions (Section A.1). In token positions, this is approximately 1/30 of TESS 2(arXiv:2502.13917)'s 45B, 1/43 of CADD(arXiv:2510.01329)'s 65B, and 1/67 of MLFM(arxiv:2606.27617)'s 100B adaptation budget.

  1. Direct combination with MLFM. We continued training the converted Fast-dLLM v2-1.5B checkpoint, already a masked diffusion model (MDM), for 1.1B non-padding tokens on our data using MLFM’s official configuration. Our inference uses MLFM’s official code and matches its official throughput. The comparison below shows that our conversion design achieves higher performance and throughput within this budget (Section 4.4, Section B.2, Tables 1 and A6).

    MetricHumanEvalMBPP
    SC-dLLM-1.5B Plus score41.5041.70
    Fast-dLLM v2-1.5B + MLFM Plus score8.107.03
    Throughput ratio (SC-dLLM-7B / combination-1.5B)>20×>20×
  2. Endpoint reconstruction. Initialization before the 1:1 training mixture improves the 1.5B average by 3.67 points. This prepares the AR model to reconstruct tokens at their original positions (Section 4.4, Table A5).

  3. Training mixture. The 1:1 mixture of Endpoint reconstruction and interior denoising gives the highest average, supporting training on noisy continuous states and clean-token context (Section B.2, Table A5).

2. Practical contribution and early commitment

  1. Performance and efficiency. At τ=0.9\tau=0.9, continuous sampling improves the average over reset-to-MASK by 2.09 points at 1.5B and 1.31 points at 7B (Section B.1, Table A4). Our additional 7B measurements show 19.5%–35.5% fewer denoising steps and 1.27–1.32× throughput. Continuous states improve performance with less computation.
TaskSamplerScore (%)Mean denoising stepsTokens/s
HumanEval PlusContinuous60.9886.62961.40
HumanEval PlusReset-to-MASK59.20107.62732.90
MBPP PlusContinuous58.9980.49864.74
MBPP PlusReset-to-MASK57.32114.92680.84
GSM8KContinuous82.4798.671171.44
GSM8KReset-to-MASK81.70152.92885.38
  1. Early commitment token categories. In our analysis, the three categories below represent simple structural or fixed-format tokens, such as whitespace, punctuation, operators, and keywords like def. Together, they account for more than 86% of first-step commits. Early commitment lets these predictable tokens finish while uncertain tokens needed to construct the solution continue to refine. Entries are percentages of first-step commits.
Token categoryHumanEvalMBPPGSM8K
Punctuation/operators32.7331.8730.01
Whitespace/indentation/newlines16.4916.4915.06
Identifier/keywords39.0938.5943.02
Sum88.3186.9588.09
  1. Continued refinement. At τ=0.9\tau=0.9 and τ=1.0\tau=1.0, token-change curves show predictions evolving and stabilizing before commitment (Section 4.4, Figure 5; Section B.4, Figure A2). Together with the category analysis above and the performance loss from resetting unresolved positions to MASK, this supports continuous refinement of words and identifiers whose predictions remain uncertain as the model constructs the solution.

  2. Stable predictions. Tokens that stabilize early at τ=1.0\tau=1.0 show the same structural and fixed-format patterns as those committed early at τ=0.9\tau=0.9. At τ=1.0\tau=1.0, predictions wait another 21.40 denoising steps on average on HumanEval and 19.87 on MBPP after stabilizing. Early commitment reduces this repeated computation (Section 4.4, Figure 5, Table A7).

Both thresholds use full continuous denoising. The threshold accelerates generation by avoiding repeated computation on stable predictions: confident tokens finish early while unresolved predictions continue to refine.

3. Throughput measurement

We report batch-256 aggregate denoising/decoding throughput on one H200. Prefill and block-boundary KV recomputation are calculated separately: together, they account for 5.24% of forward time on HumanEval and 2.94% on MBPP (Sections A.8 and B.6). We will compare the time spent on prefill, denoising, and block-boundary KV recomputation in detail.

Reviewer_umW9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SC-dLLM, which converts a pretrained AR LLM (Qwen2.5-Instruct at 1.5B and 7B) into a blockwise diffusion model whose unresolved positions evolve through continuous states. Each block starts from mask embeddings, moves along a Brownian bridge toward the predicted token embedding, and commits a position once its predicted confidence exceeds a threshold. Training first reconstructs masked tokens at the two endpoints and then mixes in independently corrupted interior states, both with the same token cross-entropy, using about 1.1B tokens. Empirical results on nine scores over code, math, instruction following and knowledge QA show averages of 45.2 at 1.5B and 61.2 at 7B, slightly above Fast-dLLM v2 (45.0 and 60.3), with the largest gains in code generation.

Critical strengths
  1. The conversion budget is small (about 1.1B tokens), and the paper includes a same-data AR fine-tuning control, which is the right reference for the additional data exposure.
  2. The paper is candid about its limitations, including the task-dependent capability loss and the unmeasured inference speed.
Critical weaknesses
  1. The novelty is limited. The bridge construction follows MLFM (Azangulov et al., 2026), the blockwise attention and complementary masking layout follow Fast-dLLM v2, the removal of token shift follows Efficient-DLM, and confidence-based commitment follows both MLFM and Fast-dLLM v2. MLFM also already uses the same noise scale (σ=0.2\sigma = 0.2), the same token cross-entropy and the same posterior-mean embedding as the bridge endpoint, so what remains is AR initialization, the blockwise layout and independent per-position times. More importantly, the paper does not show that the continuous states matter. The natural control is the same recipe on the same data without the interior bridge states, decoded as discrete block diffusion. Without it, the gains over Fast-dLLM v2 may come from the training data or the recipe rather than the continuous states, which the paper itself acknowledges is not isolated. There are also no ablations of σ\sigma, the two-stage curriculum, the number of denoising steps or the sub-block size.

  2. The evaluation does not support the main claims:

    i. The gains over the AR model rest on weak reference numbers. The AR initializer row, copied from Fast-dLLM v2, reports HumanEval 51.2, MBPP 57.7 and GSM8K 71.4 for Qwen2.5-7B, far below the official Qwen2.5-7B-Instruct results (84.8, 79.2 and 91.6). The paper does not re-evaluate the initializer under its own protocol, where its AR control already reaches GSM8K 82.9. The authors' own same-data AR control is similarly low on code (HumanEval 53.5). The code gains, which drive the average, may therefore reflect the evaluation protocol rather than the conversion. At 7B, against the same-data AR control, SC-dLLM is lower on all five non-code scores (e.g., MATH 65.0 vs 69.3, IFEval 62.2 vs 70.2). At 1.5B it is lower on five of nine scores, including both MBPP scores, and the average margin is only 0.5.

    ii. The average favors the paper's strengths. The four code columns take four of the nine weights, and MMLU and GPQA are scored by likelihood at the masked endpoint, without running the continuous sampler, so two of the nine scores do not test the proposed generation. The 7B margin over Fast-dLLM v2 (61.2 vs 60.3) also depends on τ=0.9\tau = 0.9, which is selected on the same benchmarks (Figure 4, one generation seed). At τ=1.0\tau = 1.0, the setting of Fast-dLLM v2's published results, the average is 59.8, below Fast-dLLM v2 and equal to the AR control.

    iii. Inference efficiency is not measured against any baseline, as the paper acknowledges. Without early commitment, each 8-token sub-block uses up to 32 evaluations, i.e., up to 4 full-block forward passes per token versus one for AR decoding, and throughput is reported only for SC-dLLM itself. The data-efficiency comparison in the abstract (1/41 of ELF's tokens) compares the conversion of a pretrained 7B model with a model trained from scratch, which is not meaningful. Counted as packed tokens, as Fast-dLLM v2 counts, the conversion uses about 1.5B tokens, slightly more than Fast-dLLM v2's 1.31B.

Questions for authors
  1. What do the original Qwen2.5-Instruct models score under the authors' own evaluation harness, and why are the AR code and GSM8K numbers far below the official results?
  2. How does the same recipe perform on the same data without the interior continuous states, decoded as discrete block diffusion?
  3. What is the throughput or latency of SC-dLLM against Qwen2.5-7B-Instruct and Fast-dLLM v2 on the same hardware?
  4. What is the variance over generation seeds and training runs for the average, given the 0.9-point margin over Fast-dLLM v2?
  5. (Suggestion) How can τ be chosen without using the test benchmarks?
  6. Why does the full continuous schedule (τ=1.0\tau = 1.0) score below early commitment (59.8 vs 61.4), if the continuous states carry useful information?
5 comments
AuthorsOpenReview ↗

Response to Reviewer umW9

We thank the reviewer for the detailed review and for highlighting the small conversion budget and the same-data AR control.

W1. Method Novelty.

A1.

Directly combining Fast-dLLM v2 with Masked Language Flow Models (MLFM) under the same 1.1B-token conversion budget gives substantially lower performance and throughput than SC-dLLM. This control demonstrates the value of adapting the pretrained token predictor to evolving continuous states. Throughput compares SC-dLLM-7B at τ=0.9 with the 1.5B combination on the same H200, batch size 256, and timing protocol. (Section 4.4, Section B.2, Tables 1 and A6)

  • MLFM setup. Following MLFM’s official configuration, we continued training the converted Fast-dLLM v2-1.5B checkpoint, already a masked diffusion model (MDM), for 1.1B tokens on our data.
MetricHumanEvalMBPP
SC-dLLM-1.5B Plus score41.5041.70
Fast-dLLM v2-1.5B + MLFM Plus score8.107.03
Throughput ratio (SC-dLLM-7B / combination-1.5B)20.79×20.01×

Q1 & W2(i). AR comparison and evaluation protocol.

A2.

  1. We evaluate Qwen2.5-FT, Fast-dLLM v2 and SC-dLLM under the same protocol, following the official Fast-dLLM v2 configuration for all benchmarks. (Sections A.6–A.7).
  2. Qwen2.5-FT is Qwen2.5-Instruct fine-tuned on the same conversion data as SC-dLLM, retaining the AR next-token objective.
  3. Under this shared protocol, SC-dLLM achieves a higher average than the same-data AR control at both scales. At 7B, gains extend to GSM8K and MMLU alongside HumanEval Plus and MBPP Plus. We have updated our Table 1, now SC-dLLM uses τ=1.0 in this comparison:

Q2 & W1. Discrete control and Ablation.

A3.

The following controls and ablations establish the value of interior continuous states.

  1. Discrete control. SC-dLLM outperforms Fast-dLLM v2-FT at both scales with the same Qwen initialization, conversion data, and 1.1B-token budget (Sections 4.1–4.2, Table 1).
  2. Reset to MASK. Resetting unresolved states to MASK after each denoising step lowers the average at both scales. Retaining continuous information directly improves generation (Section 4.4, Table A4).
  3. σ\sigma Ablation. At σ=0.2\sigma=0.2 adds 7.28 points over zero noise (σ=0\sigma=0). This establishes the value of learning reconstruction from noisy continuous inputs (Section 4.4, Table A5).

Average scores:

Model / samplerτ\tau1.5B7B
SC-dLLM1.046.863.89
Fast-dLLM v2-FT1.045.361.0
Continuous1.046.863.8
Discrete1.044.762.9
  1. Initialization. At 1.5B, endpoint initialization improves the average by 3.67 points. It trains the model to reconstruct tokens at their original positions before continuous training (Section 4.4, Table A5).
  2. Training mixture. Equal proportions of Endpoint reconstruction and interior denoising give the highest 1.5B average, supporting training on both noisy continuous inputs and clean-token context (Section B.2, Table A5).
  3. Denoising steps. As shown in Figure A1, Increasing the denoising steps from 8 to 32 substantially improves performance, establishing the benefit of sufficient continuous refinement (Section B.3, Figure A1).
  4. Sub-block size. We also shows the ablations on sub-block size in Figure A1. Smaller groups let completed tokens guide subsequent generation (Section B.3, Figure A1).

Q3 & W2(iii). Baseline speed and token accounting.

A4.

  1. SC-dLLM at τ=0.9 achieves a higher average than Fast-dLLM v2 at τ=1.0 (62.45 vs 60.3), and higher throughput than both baselines below (Tables 1 and A4; Section 4.5, Figure 6). All three models use one H200 with bf16. Both diffusion models use batch size 256. This time excludes prefill and block-boundary KV recomputation (Section A.8).
  2. The conversion uses approximately 1.1B non-padding tokens, equivalent to 1.5B packed tokens; Fast-dLLM v2 reports 1.31B packed tokens. The abstract now reports the conversion budget directly. Figure 3 counts additional conversion tokens for AR-initialized models and pretraining tokens for scratch-trained models (Sections 4.1 and 4.3).

Throughput at 7B, with SC-dLLM at τ=0.9 and Fast-dLLM v2 at τ=1.0:

TaskSC-dLLM tokens/sFast-dLLM v2 tokens/sQwen2.5-7B-FT tokens/s
HumanEval932.61612.44504.19
MBPP846.14754.54546.16
GSM8K1171.44765.67473.62

Q4. Random Seed & Variance.

A5.

We have updated our Table 1 with three-seed evaluation, now SC-dLLM achieves a 3.6-point average advantage over Fast-dLLM v2 at 7B. We evaluate one trained checkpoint per scale with generation seeds 2718, 2719, and 2720. At 7B, generation-task standard deviations range from 0.2 to 1.2 points; MMLU and GPQA use deterministic likelihood scoring. (Table 1, Section A.7)

AuthorsOpenReview ↗

Response to Reviewer umW9

Q5. Threshold selection.

A6.

  1. The main comparison now fixes τ=1.0 for fair comparison. We also evaluate SC-dLLM and official Fast-dLLM v2 on the same grid {0.3, 0.5, 0.7, 0.9, 1.0}, reported in Figure 6. For deployment, select τ against a quality or latency target on held-out development data and fix it before testing (Section 4.5).

W2(ii). Benchmark weighting and threshold selection.

A7.

  1. Table 1 follows Fast-dLLM v2's weighting and evaluation protocol, with τ=1.0 for the diffusion models.
  2. After averaging Base/Plus within HumanEval and MBPP and then averaging the seven benchmarks, SC-dLLM still outperforms Fast-dLLM v2 at both scales. The results below are computed from Table 1 (scoring details in Section A.7).
  3. For MMLU and GPQA, we score candidate answers at the masked answer position (Section A.7).
Model scaleSC-dLLMFast-dLLM v2-FT
1.5B47.5045.84
7B63.2160.67

Q6. τ=1.0\tau=1.0 performance and early commitment.

A8.

  1. τ\tau controls commitment timing. All thresholds use full continuous denoising. τ\tau controls when tokens are committed. At τ=1.0\tau=1.0, predictions at 94.45%–97.87% of unresolved positions remain unchanged between successive denoising steps 24–31. After predictions stabilize, commitment still takes another 21.40 denoising steps on average on HumanEval and 19.87 on MBPP. In contrast, at τ=0.9\tau=0.9, only 17.02%–37.47% of positions change their predictions during refinement, and over 90% of predictions before commitment agree with the final token. Early commitment saves these repeated denoising steps while unresolved positions continue to refine. (Section 4.4, Figure 5; Section B.4, Figure A2, Table A7).
  2. We have updated the results under the training setting in Section 3.2. Continuous sampling with τ=1.0\tau=1.0 now achieves the highest average below, outperforming discrete sampling. These comparisons establish the performance benefit of continuous refinement while early commitment accelerates decoding (Section 4.4, Table A4).

Average scores at 7B:

Samplerτ\tauAverage
Continuous1.063.81
Continuous0.962.45
Discrete1.062.97
AuthorsOpenReview ↗

A Gentle Reminder

Dear Reviewer umW9,

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal and revised manuscript. We have conducted additional experiments and analyses to address all of your concerns regarding novelty, continuous-state contributions, and evaluation fairness.

We would greatly appreciate your feedback on whether our responses have addressed your concerns. Please let us know if any questions remain, and we would be happy to provide further clarification.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank the reviewers for their careful evaluation and constructive feedback. Our revision addresses the concerns regarding novelty, mechanisms, and evaluation.

Summary of Updates:

  • Final Result: We updated Table 1 with our latest evaluation results. SC-dLLM now achieves averages of 46.8 at 1.5B and 63.89 at 7B.
  • Novelty & Contribution: We added matched discrete baselines and a direct Fast-dLLM v2 + MLFM control, demonstrating the benefit of our AR-to-continuous conversion procedure.
  • Role of Continuous States: We added experiments that reset intermediate states to MASK, alongside ablations and denoising trajectory analysis. These experiments show that retaining and refining continuous states before commitment improves generation.
  • Performance & Efficiency: We added H200 comparisons against official Fast-dLLM v2 and Qwen2.5-7B-FT, showing higher overall performance and faster decoding.
  • Threshold Selection: Updated Table 1 uses τ=1.0\tau=1.0 for both SC-dLLM and Fast-dLLM v2, and Figure 6 compares both methods across thresholds. SC-dLLM at τ=0.9\tau=0.9 achieves a higher average and faster decoding than Fast-dLLM v2 at τ=1.0\tau=1.0.
  • Early Commitment: All thresholds use full continuous denoising. τ\tau only controls when tokens are committed. Early commitment saves computation after predictions stabilize. Our new analysis of early commitment token categories shows that about 86% of first-step commits are simple structural or fixed-format tokens, allowing predictable tokens to finish early while uncertain tokens receive further denoising.

In the revised manuscript, red marks changes made in response to reviewer comments, and blue marks our updates to settings and wording.

We welcome further questions and would appreciate your reconsideration of the scores in light of these additional experiments and analyses.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.