acceptodds
Under review as a conference paper at ICLR 2027

SwiftDLM: From Trajectory Imitation to Self-Exploration in Diffusion Language Models

Abstract

Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive models due to their potential for parallel decoding. However, existing open-source dLLMs require substantial decoding steps to maintain generation quality, thereby limiting inference efficiency. While recent training-based acceleration methods reduce decoding steps, they do not explicitly enforce consistency between intermediate decoding states or explore new trajectories through reinforcement learning. To address these limitations, we propose SwiftDLM, a unified two-stage training framework that substantially reduces decoding steps while maintaining or even improving generation performance. In the first stage, we employ consistency distillation on intermediate trajectory states to achieve skip-step generation. In the second stage, we leverage reinforcement learning to reward trajectories exhibiting both accuracy and efficiency, enabling self-exploration of superior decoding paths. Extensive evaluations demonstrate that our method outperforms existing dLLM acceleration approaches. When applied to LLaDA, our method achieves a 8.9 speedup on GSM8K while improving accuracy by 6 percentage points, and delivers an 11.3 speedup on MBPP while maintaining competitive performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
dLLM, Consistency Distillation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_h4NCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SwiftDLM combines cross-state consistency distillation along teacher trajectories with reinforcement learning that jointly considers correctness and decoding steps, forming a two-stage post-training framework for masked diffusion language models. Experiments on LLaDA and Dream demonstrate improved decoding speed and generation quality under few-step decoding.

Critical strengths

1. Practically relevant problem with strong efficiency gains across models and tasks.

The paper targets a key bottleneck in dLLM inference: reducing decoding steps without sacrificing generation quality. Experiments on LLaDA and Dream across math reasoning and code generation show consistent gains. On LLaDA/GSM8K, SwiftDLM improves accuracy from 74.7% to 81.6% over dParallel while reducing average decoding steps from 30 to 25, demonstrating clear practical value.

2. Well-motivated two-stage design with complementary effects validated by ablation.

The paper compares Stage I, Stage II, and the full method, and also applies the same Stage II RL to a dParallel checkpoint. Although SwiftDLM Stage I starts with lower accuracy than dParallel, it achieves better accuracy and efficiency after the same RL stage, suggesting that consistency distillation provides a more suitable initialization for subsequent reinforcement learning.

Critical weaknesses

W1. The novelty of the RL component is not sufficiently established compared with prior efficiency-aware RL approaches.

The second stage optimizes decoding trajectories using a reward that jointly considers generation quality and decoding efficiency. However, the general idea of incorporating efficiency objectives into RL optimization has been explored in prior works. Prior works such as LightningRL and dUltra have explored RL formulations where generation quality and efficiency considerations are jointly optimized. A comparison with RL-based dLLM acceleration approaches would significantly strengthen the contribution claim.

W2. The evaluation does not fully characterize practical inference efficiency.

The paper evaluates efficiency using decoding steps, latency, and speedup. While these metrics are useful, they do not fully characterize dLLM inference efficiency.

Since SwiftDLM explicitly optimizes decoding trajectories and encourages fewer denoising steps, it is important to verify whether the reduction in decoding steps translates into improved effective throughput. Metrics such as tokens-per-second (TPS), tokens-per-forward-pass (TPF), or similar throughput-oriented measurements would provide a more complete evaluation of practical acceleration.

Questions for authors

Q1: The RL stage appears closely related to prior efficiency-aware RL approaches. Could the authors clarify the key novelty of SwiftDLM's RL formulation compared with methods such as dUltra and LightningRL? Specifically, which component is responsible for the main gains?

Q2: The paper reports latency and decoding steps as efficiency metrics. Could the authors provide throughput-oriented metrics such as TPS/TPF?

Reviewer_pgCDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SwiftDLM accelerates masked dLLMs (LLaDA-8B, Dream-7B) in two stages. Stage 1 fine-tunes the model with LoRA on teacher trajectories that unmask one token per step. For two states Ti→Tj\mathcal{T}_i \to \mathcal{T}_j in the same block, it applies cross-entropy on the tokens unmasked between them. It also adds a stop-gradient KL that pulls the predictions at Ti\mathcal{T}_i toward those at Tj\mathcal{T}_j on tokens that are still masked. Stage 2 runs GRPO with confidence-threshold rollouts. Its reward multiplies correctness by an efficiency factor that grows as the number of steps shrinks, and it evaluates token likelihoods at the recorded intermediate states. With τ=0.9\tau=0.9, L=256L=256 and block size 3232, LLaDA on GSM8K goes from 256 to 25 steps while accuracy rises from 75.6%75.6\% to 81.6%81.6\%.

Critical strengths

The paper provides an useful recipe with distillation followed by RL with a multiplicative step-aware reward gives a better accuracy-efficiency trade-off than compared baselines, at confidence thresholds tested (Table 7). The ablations are clear to show where the gains come from (Tables 3a, 5, 8).

Critical weaknesses

Scope confusion. Stage 1 is trajectory distillation from the teacher's unmasking order, similar to CDLM (Kim et al., MLSys 2026), d3LLM (Qian et al., ICML 2026) and dParallel (Chen et al., ICLR 2026). Stage 2 computes token likelihoods along the actual inference trajectory, which has been studied in TraceRL (Wang et al., ICLR 2026). It rewards correctness jointly with fewer decoding steps, which is DiFFPO (Zhao et al., NeurIPS 2025), and its multiplicative reward is taken from Jazbec et al. (arXiv 2025). CDLM, TraceRL and DiFFPO are neither cited nor compared. A combination can still be a contribution if the combination itself is useful for solving new challenges. Here, it is concerning: most of both the speedup and the accuracy gain come from the RL stage, while Stage 1 adds only a modest improvement on top.

Comparison fairness. On its own, Stage 1 is worse than both dParallel and d3LLM in the paper's own Table 3a (+referring back to baselines in Table 1). It is reported only on LLaDA GSM8K. Stage 1, which adds exactly the intermediate-state consistency that the paper says prior distillation (like d3LLM) lacks, is not shown to be the strongest distillation method in the comparison.

Insufficient Distillation + RL baselines. The paper also never says whether Stage 1 is meant to be sufficient on its own. Whether it's meant to be compared against distillation only dLLM, or distillation + RL. Instead, it quietly reframes Stage I as a warm start for RL (Appendix F.1). Meanwhile, RL from vanilla LLaDA already beats every baseline without Stage 1 (Table 3a). The baselines it is compared against use no RL at all. The only like-for-like comparison applies the same RL on top of dParallel (Table 5). There, the gap shrinks to under 4 points and 7 steps, and that comparison should be the headline.

Questions for authors

(1) To make fair baseline comparisons, please apply Stage 2 on top of d3LLM and CDLM checkpoints, as done for dParallel in Table 5.

(2) Distillation contribution: Please also report Stage 1 alone on all benchmarks and on Dream. If Stage 1 helps only as a warm start, the paper's central claim about prior distillation does not hold.

(3) What does the step reward actually do? Which tokens get unmasked is fixed by thresholding confidence at τ\tau, so the gradient never propagates there. Please report correctness-only RL (α=0\alpha=0), evaluated at τ=0.9\tau=0.9. If most of the speedup remains, the step reward is secondary and the speedup comes from RL sharpening confidence.

(4) How well does the speedup transfer? Please evaluate the GSM8K-trained model on code and the code-trained model on math, plus one non-reasoning task. Please also report seeds or confidence intervals, since several gaps (e.g., 0.30.3 points on HumanEval) amount to less than one problem.

Reviewer_d5vkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SwiftDLM combines trajectory imitation and reinforcement learning to improve the accuracy-latency trade-off of masked diffusion language models. Stage I applies cross-entropy and consistency losses to pairs of intermediate states from block-wise teacher trajectories. Stage II uses a GRPO-style update conditioned on recorded trajectory states, with a multiplicative reward combining correctness, formatting, and decoding efficiency. Stage I uses mixed mathematics/code data; Stage II produces separate mathematics and code checkpoints for each backbone, trained on GSM8K and an AceCode subset, respectively.

Experiments evaluate LLaDA-8B-Instruct and Dream-7B-Instruct on GSM8K, MATH, HumanEval, and MBPP using generation length 256 and block size 32. On LLaDA/GSM8K, SwiftDLM achieves 81.6% accuracy in 25 steps and 1.1 seconds, versus 75.6% in 256 steps and 9.8 seconds for the original model, and 74.7% in 30 steps and 1.3 seconds for dParallel. Stage, initialization, reward, and threshold ablations mainly use LLaDA/GSM8K to examine the combined training recipe.

Critical strengths
  1. The combined method achieves a useful accuracy-latency improvement across two backbones. In Table 1, LLaDA/GSM8K improves from dParallel's 74.7% at 1.3 seconds to 81.6% at 1.1 seconds; against the original LLaDA model, this is a 6.0-point gain and an 8.9-fold speedup. On MBPP it reports 42.4% at 2.2 seconds, versus 41.8% at 24.9 seconds for the original model. Table 2 extends the efficiency improvement to Dream: GSM8K reaches 83.6% at 1.2 seconds, versus 82.8% at 11.7 seconds. Accuracy does not improve in every setting, but the results provide substantive evidence for practical acceleration under the evaluated decoder and workload.

  2. The ablations provide meaningful evidence for the interaction between consistency training and RL. In Table 3a, adding consistency to Stage I improves GSM8K accuracy from 71.9% to 73.5% and reduces steps from 53 to 42. After Stage II, the full initialization reaches 81.6% in 25 steps versus 78.8% in 30 steps for CE-only initialization. Crucially, Appendix F.1 applies the same Stage II data, reward, and optimization configuration to dParallel: it reaches 77.8% in 32 steps, versus 81.6% in 25 for the proposed initialization. This addresses the simple explanation that any existing accelerated model would obtain the same benefit from additional RL. Appendix F.3 also shows an advantage over dParallel across multiple confidence thresholds.

Critical weaknesses
  1. The evidence for transferable benefits of the two-stage design is concentrated in one model-task setting. The paper evaluates both LLaDA and Dream, but Stage II trains separate mathematics and code models (Section 5.1 and Appendix C). Thus, the four-benchmark tables establish results for domain-specific checkpoints, not joint capability retention in one final checkpoint. This is a valid evaluation of domain specialization, but its scope should remain explicit. More importantly, the stage ablations, matched dParallel-plus-Stage-II control, reward comparisons, and hyperparameter studies all use LLaDA/GSM8K (Table 3 and Appendix F). These are meaningful controls, yet do not establish whether the same component interactions explain the Dream and code results. Similarly, Appendix D compares against d1/wd1 without isolating the intermediate-context estimator under matched initialization and reward. The method is promising as a combined recipe, while its broader component-level explanation needs a focused replication beyond LLaDA/GSM8K. Exact dataset mixtures, subset identifiers, and checkpoint-selection criteria would also help interpret the reported transfer.

  2. The practical accuracy-cost comparison needs fuller specification and broader validation. All reported evaluations use generation length 256, block size 32, and batch size one. This matches the explicitly block-wise training design, but leaves transfer to other lengths or decoding schedules untested. d3LLM is evaluated with a shared decoder; caching choices and native-versus-shared execution are not fully specified for all baselines. Evaluation sampling and aggregation details are also insufficient, and no paired uncertainty or independent training-run variability is reported for small margins such as 40.4% versus 40.1% on LLaDA/HumanEval. For training, the exact correctness/format reward values, step normalizer, and weighted-sum coefficients are not given, limiting reproduction of the reward comparison despite the useful training-time accounting in Appendix C.

Questions for authors
  1. Please provide dataset counts by source, subset identifiers, and the validation protocol used to select hyperparameters and final checkpoints. Given the stated separate Stage II models, can you show the Stage I, mathematics-Stage-II, and code-Stage-II checkpoints on all four benchmarks for one backbone? This would distinguish within-domain gains from cross-domain retention without requiring an additional training method.

  2. Does the consistency-initialization advantage over dParallel followed by the same Stage II persist on a code task or on Dream? One controlled replication would substantially strengthen the two-stage explanation. If an estimator-specific advantage is intended, a comparison with a representative single-step estimator using the same initialization, reward, and training data would separately test the value of intermediate contexts; please include training cost as well as final quality and latency.

  3. What are the numerical correctness/format rewards, the definition and value of N_step, and the coefficients used for the weighted-sum ablation? For Tables 1-2, please specify decoder/cache configurations, timing boundaries, evaluation temperature, completions per problem, and score aggregation. Paired problem-level uncertainty and repeated training for a representative main ablation would help distinguish stable effects from small point-estimate differences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.