acceptodds
Under review as a conference paper at ICLR 2027

S²I-dLLM: From Sparsity and Specialization to Adaptive Acceleration in Multimodal Diffusion LLMs

Abstract

Multimodal diffusion large language models (dLLMs) integrate tasks across modalities within a shared bidirectional Transformer and a masked diffusion objective, yet how the computational demands of different tasks are organized within the shared network and evolve during denoising remains insufficiently understood. In this work, we analyze the computational demands of multimodal dLLMs from two perspectives, network structure and token representation evolution, characterizing how different tasks sparsely depend on shared computation and how tokens with different roles differ in representational stability and update requirements. Building on this analysis, we propose S²I-dLLM, a training-free, task-aware acceleration framework that adaptively allocates computation according to the demands of different tasks and how these demands change throughout denoising. S²I-dLLM reduces network channels and selects layers to execute according to task requirements, while restoring necessary computation as denoising progresses to control the accumulation of approximation errors. At the token level, the framework uses representational stability to guide feature reuse and prediction states to adjust the timing of output refinement and commitment, thereby coordinating per-step computational cost with generation progress. Experimental results demonstrate that S²I-dLLM effectively improves the inference efficiency of multimodal dLLMs while preserving task performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion large language models, efficient inference, training-free acceleration, feature caching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_r72KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

S2I-dLLM is a training-free, task-aware accelerator for unified masked-diffusion models. Two diagnostics motivate it: tasks rank FFN channels differently, increasingly with depth, and evolve representations differently (Figure 1); target tokens drift more than conditioning tokens, with spatially correlated image drift in deep layers (Figure 3). Structurally, a Wanda-style saliency (Eq 2) keeps a per-task FFN prefix and a cosine criterion (Eq 4) bypasses layers. Token scheduling refreshes only high-drift image features, mixes MMU embeddings, and commits confident text tokens (App B.2). A final full-compute phase covers the last β of denoising (Eq 8). On MMaDA-8B-MixCoT (RTX 4090) it reports 2.39-2.43x on T2I (Table 1b) and, with token scheduling only, 15.12-30.04x on MMU (Table 1a) and 3.92-10.48x on text (Tables 2 and 6c), plus cross-model transfer (Table 3).

Critical strengths
  1. The task-dependent-compute premise is measured, and a matched control at equal channel budget favors it. Channel-importance rankings of T2I and the two language-output tasks decorrelate with depth (Figure 1a), and at matched retained-channel counts a task-specific ranking scores above a shared one on all three control subsets (GSM8K 56.25 vs 50.78, Table 8a, L1094-1096) at similar latency. The subsets are small (128 GSM8K questions), but GenEval and AI2D-lite move the same way. FlashU already states this premise for unified models; the credit here is for testing it on a masked-diffusion model, where text is also generated by iterative denoising.

  2. The depth-resolved analysis of image-target drift is a fresh diagnostic, and a small control favors the design it motivates. Image-target drift becomes spatially correlated in deeper layers, measured by Moran's I against a flat shuffled control (Figure 3b), and FlashU reports no such analysis. The paper turns it into depth-dependent refresh, token units in shallow layers and two-dimensional blocks in deep ones (App B.2.1, L880-883), which scores 0.5521 against 0.5208 for token-wise refresh on the 48-prompt GenEval control subset at similar latency (Table 8c). This is the most defensible novel piece of the paper.

  3. The evaluation covers all three task types of a unified model and transfers to two other backbones. Task-routed configurations run every MMU, T2I, and text/code benchmark in Tables 1, 2, and 6 on MMaDA-8B-MixCoT against existing accelerators on the same backbone; recalibrated per model without training, S2I-dLLM reaches 2.07-4.71x on Dynin-Omni and Lumina-DiMOO on fixed subsets (Table 3). Latency is per-request wall-clock time at batch size 1 on an RTX 4090, with the timing protocol (App A.3) and hyperparameters (Table 5) disclosed. I weight this breadth because a unified model exists to serve all three tasks from one backbone.

Critical weaknesses
  1. The headline MMU and text speedups come mainly from per-task recipes of the compared baselines, not from the structural and adaptive parts the paper proposes. Those runs use "full FFN width and all layers with token scheduling and prefix caching" (App B.4, L982-983): ReMix-style mixing for MMU (Eq 11) and Fast-dLLM-style commitment for text (Eq 12). Where adaptive layer selection is isolated, refreshing the skip set instead of freezing the first probe's set cuts latency by only 1.5-1.9%, with no score gain (Table 8b, L1097-1099). A component ablation on the full MMU and text benchmarks, with the structural half on, would show what the framework itself adds.

  2. The structural half closely follows Flash-Unified, published in CVPR 2026 Findings and compared in no table. FlashU (Ke et al., 2026, arXiv:2603.15271) already ranks FFN neurons by activation L2 norm times summed absolute weight (the form of Eq 2), skips layers by input-output cosine recomputed at fixed intervals (cf. Eq 4-5), and switches between pruned and full FFNs by timestep (cf. Eq 8). Its introduction also claims "the first systematic dissection of the internal computational mechanisms of unified models", and Figure 1(b) follows its analysis (L1184). Running FlashU's structural parts on MMaDA-8B-MixCoT would show what this half adds, and L54 should credit that dissection.

  3. Two headline T2I cells take different values in different tables for the same configuration. DPG generation is timed at 8.90s (2.43x) in Table 1b but 9.64s (2.24x) in Table 6b, although ImageReward "evaluates the generated DPG images without additional image generation" (App A.3, L821-822). Only S2I and dLLM-Cache move, each by the same amount, while Vanilla, Few-step Vanilla, and ML-Cache are identical; the fixed 35-call GenEval score also differs between Table 1b and Table 11. Reporting each table from one run with fixed seeds, or stating per cell which run it comes from, costs no experiments.

Questions for authors
  1. Can you compare Flash-Unified with your channel pruning and layer skipping alone on MMaDA-8B-MixCoT, in speed and quality? If not, what prevents running it on MMaDA?
  2. On the full MMU and text benchmarks, what do channel selection and layer skipping add on top of the token-only configuration with prefix caching? Figure 9 (L1473) reports one such run, on a single AI2D example.
  3. With your prefix caching, commitment thresholds (0.80 and 0.85, L983), and BF16 confidence (L815-816), how do ReMix on MMU and Fast-dLLM on text compare with S2I-dLLM in NFE, latency, and accuracy?
  4. Which runs produce the DPG latencies of 8.90s (Table 1b) and 9.64s (Table 6b), and the GenEval scores 0.5489 (Table 1b) and 0.5281 (Table 11)? How do 57.21 and 57.63 (Table 8b) arise from 96 and 128 questions?
  5. S2I-dLLM reaches 1.21-1.38x in the lighter 50-step development setting (Table 10). Can you time dLLM-Cache and ML-Cache at 35 NFE, to compare the 2.43x DPG speedup at matched calls and show the step-subsampling share?
Reviewer_2UyaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies computational redundancy in multimodal diffusion LLMs from two perspectives: task-dependent structural sparsity and token representation dynamics. Based on these observations, the authors propose S²I-dLLM, a training-free acceleration framework combining task-aware channel selection, adaptive layer skipping, token-level scheduling, and computation recovery.

Experiments on MMaDA-8B-MixCoT and two additional backbones show substantial inference speedups across multimodal understanding, text reasoning, and image generation, with generally moderate but task-dependent quality changes. The main MMU and text results use token-only acceleration, while image generation evaluates the combined framework.

Critical strengths
  1. Well-motivated task-aware acceleration. The analysis of channel importance and token dynamics provides useful evidence that different tasks have distinct computational requirements despite sharing the same backbone. The task-specific channel ranking results in Table 8 further support this motivation.

  2. Strong empirical efficiency gains. The method achieves substantial latency reductions across multiple tasks, with comparisons against relevant baselines. The ablations in Table 4 also show that structural sparsification and feature reuse contribute to the overall speedup. Cross-model experiments provide additional evidence of applicability.

Critical weaknesses
  1. Limited validation of the full framework. Although the paper emphasizes jointly optimizing structural and token-level computation, the main MMU and text experiments use token-only acceleration. This makes it difficult to assess the benefit of the complete framework across tasks. For image generation, the speedup also comes partly from reducing the number of denoising steps. A matched-NFE comparison would help isolate the contribution of computational sparsification.

  2. Some experimental findings need clarification. The GenEval score differs between Table 1b (0.5489) and the fixed 35-call setting in Table 11 (0.5281), without a clear explanation of the configuration differences. Moreover, Table 8 shows only modest latency gains from adaptive layer selection over a fixed skip set, with slightly lower scores. These results leave some uncertainty about the benefit of the adaptive components. Reporting variability across runs would also strengthen the evaluation.

Questions for authors

Could the authors provide full-benchmark results for the joint structural and token-level framework on MMU and text reasoning, to clarify its benefits beyond token-only acceleration?

Could the authors explain the GenEval discrepancy between Tables 1b and 11, and provide a matched-35-NFE vanilla comparison to better isolate per-step computational savings?

Appendix A.3 reports different confidence-computation precisions for the text baselines (FP64) and S²I-dLLM (BF16). How much does this difference affect the reported latency comparisons?

Reviewer_jX4COpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes S²I-dLLM, a training-free acceleration framework for multimodal diffusion LLMs. It combines task-specific channel pruning, layer skipping, token scheduling, feature caching, and computation recovery. Experiments on MMaDA and two additional backbones report substantial speedups with task-dependent quality trade-offs. The main MMU and text results use token scheduling and caching without structural sparsification.

Critical strengths
  • Task- and token-level diagnostics provide useful evidence of computational redundancy in multimodal diffusion models.

  • The framework achieves practical acceleration across understanding, reasoning, and image generation, with cross-model evaluations and component ablations.

Critical weaknesses
  • Timing comparisons contain confounding implementation differences. Appendix A.3 states that Vanilla, dLLM-Cache, and Fast-dLLM compute text confidence in FP64, whereas S²I-dLLM uses BF16. Confidence thresholds also differ. Precision-matched comparisons and quality–latency sweeps are needed to establish the incremental algorithmic speedup.

  • The central joint contribution is incompletely validated. Main MMU/text results disable channel pruning and layer skipping, so their headline speedups do not validate the complete framework. Table 8 provides limited support for adaptive layer selection or task-specific refinement. Moreover, the closely related Flash-Unified already combines task-specific pruning and dynamic layer skipping, making a clearer controlled comparison important.

Questions for authors
  • How much advantage remains with matched confidence-computation precision and comparable quality–latency operating points?

  • Can the complete joint framework outperform token-only variants and an adapted FlashU-style structural baseline under matched decoding schedules?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.