acceptodds
Withdrawn from ICLR 2027

ITC-MoE: Importance-guided Token-aware Compression for MoE Diffusion Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Mixture-of-Experts (MoE) Diffusion Language Models (DLMs) offer flexible parallel decoding and increased model capacity, but their large number of expert parameters incurs substantial computation and storage costs. Existing low-rank MoE compression methods largely rely on static factorization and fixed rank allocation, which overlook the distinctive properties of MoE DLMs. Specifically, we identify two properties: cross-mode non-uniform redundancy, where parameter redundancy and sensitivity to rank truncation vary across the input, output, and expert modes, and token-wise utilization variation, where hot and cold tokens exhibit distinct spectral characteristics and expert activation patterns. To address these challenges, we propose ITC-MoE, an Importance-guided Token-aware Compression framework for MoE DLMs. ITC-MoE consists of two complementary components. First, Importance-guided Adaptive Tucker Compression (IATC) incorporates activation and gradient importance into expert weight transformation, jointly factorizes expert weights across multiple modes, and adaptively allocates ranks under a fixed parameter budget. Second, Token-aware Compensation and Routing (TCR) applies lightweight low-rank compensation to compression-sensitive hot tokens and restricts the candidate expert set for cold tokens with concentrated routing patterns. By jointly adapting compression capacity and inference execution to both parameter redundancy and token-wise variation, ITC-MoE substantially reduces the computation and storage costs of MoE DLMs while preserving their generation quality. For example, on SDAR-30B-A3B-Chat-b32, ITC-MoE maintains an accuracy of 96.33% on MultiArith under a 30% compression budget, while achieving up to a 7.22× end-to-end speedup.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts Compression, Diffusion Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gnbAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents ITC-MoE, an importance-aware and token-adaptive compression framework for Mixture-of-Experts Diffusion Language Models (MoE DLMs), aiming to reduce model storage requirements and inference costs. The authors identify two key characteristics of MoE DLMs: (1) non-uniform low-rank redundancy across different weight dimensions, and (2) distinct feature spectral distributions and expert activation patterns between hot and cold tokens. Building on these observations, the authors introduce two main components: (1) Importance-guided Adaptive Tucker Compression (IATC), which leverages activation and gradient statistics to guide the Tucker decomposition of expert weights, adaptively allocates compression ranks across different dimensions under a fixed parameter budget, and reduces redundant computation through shared factors; and (2) Token-aware Compensation and Routing (TCR), which applies low-rank residual compensation to compression-sensitive hot tokens while restricting the candidate expert set for cold tokens to reduce inference overhead. Experiments on SDAR and LLaDA2.0-mini demonstrate that ITC-MoE achieves higher inference speedups than existing compression methods, with a maximum reported speedup of 7.22× under certain configurations, while largely preserving model accuracy.

Critical strengths
  1. Well-motivated method design: The authors provide empirical observations on weight spectral distributions, dimension-wise compression sensitivity, and differences in expert activation patterns between hot and cold tokens. These observations provide a reasonable foundation for the proposed method.

  2. Effective integration of model compression and inference optimization: Beyond reducing parameter counts through Tucker decomposition, the proposed method leverages shared factors to enable computational reuse across experts, aiming to translate model compression into practical inference speedups.

Critical weaknesses
  1. Potential significant degradation in model accuracy: The proposed method appears to incur substantial accuracy degradation on several benchmarks. Accuracy is largely preserved only on relatively simple datasets, such as MultiArith, SingleOp, and SingleQ, while noticeable performance drops are observed even on moderately challenging benchmarks such as MBPP and GSM8K. These results raise concerns about the practical applicability of the proposed method, particularly for more complex tasks.

  2. Insufficient analysis and ablation studies of end-to-end speedups: The paper lacks a detailed explanation of the substantial inference speedups reported in Table 2, particularly why ITC-MoE significantly outperforms existing methods at comparable compression ratios. Furthermore, Table 3 reports several-fold speedups with a compression ratio of only 3%, which appears surprisingly large and warrants further investigation. The paper would benefit from additional ablation studies and a more detailed breakdown of the sources of inference acceleration to substantiate these results.

Questions for authors
  1. Could the authors provide additional evaluation results on more challenging benchmarks, such as HumanEval, MATH-500, and AIME, to better assess the effectiveness of the proposed method on complex tasks?

  2. Could the authors provide more detailed ablation studies and explanations for the inference speedups reported in Tables 2 and 3? In particular, please address the concerns raised in Weakness 2 regarding the substantial speedups over existing methods at comparable compression ratios and the surprisingly large acceleration achieved with only 3% compression. Clarifying these results is important for assessing the validity of the reported improvements and will directly influence my overall rating.

  3. The paper mentions the use of a calibration set but does not provide sufficient details about its construction. Could the authors clarify how the calibration set was constructed and whether it was strictly separated from the evaluation datasets to avoid potential data leakage?

Reviewer_sgMVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper formulates the Tucker Compression for MoE dLLLMs acceleration in a token-aware way, and proposed the ITC-MoE method. Combining with TEAM, it claimed achieving notable speedup over existing baselines.

Critical strengths
  • The design is motived by observations, and there are ablation studies for each of the components (IATC, learnable rank trucation, TCR); the overall developing logic is very clear.
Critical weaknesses
  • What is the speedup attributing to? In Line 286, it shows this strategy achieves up to 2.18X speedup, while in Table-2, ITC-MoE overall achieves 4x-7X speedup, what is the additional speedup coming from? Is it from TEAM? If it is TEAM, what if other methods combined with TEAM?
  • What is the speedup measured over? Huggingface transformer implementation or inference engine? If it is just HF, can the speedup be applied to inference engine like SGLANG? I am curious the method itself is designed less elegant and too complex from the inference angle. I doubt that the speedup may be not that bigger when applied to inference engine.
  • In Table-2, why D2−MoED^2-MoE only gets 0.8x relative to baseline? Is the configuration fair?
  • The hyper-parameters like λ\lambda, τh\tau_h are tuned and selected on benchmark-set, this is potentially unfair for compared method like TD-MoE.
Questions for authors

check the comments in "weakness"

Reviewer_ufEkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Existing compression methods for MoE diffusion language models fail to exploit two types of non-uniformity simultaneously: first, expert weights exhibit varying redundancy and compression sensitivity across the expert, input, and output dimensions; second, different tokens have distinct representation complexity and expert utilization patterns during denoising. Therefore, the authors aim to reduce model storage and inference overhead while preserving generation quality via importance-guided multi-dimensional low-rank compression, together with differentiated compensation and routing for hot/cold tokens.

Critical strengths
  • S1. Perspective targeting the unique characteristics of MoE DLMs and practical application value: Unlike traditional autoregressive models, MoE DLMs suffer from frequent repeated expert invocations and memory access bottlenecks during iterative parallel denoising. By jointly considering static parameter compression and dynamic token states, this paper delivers tangible engineering and academic value for optimizing the deployment and throughput of diffusion large models.

  • S2. Comprehensive empirical motivation with method design closely aligned with observations: Figure 2 systematically analyzes the low-rank geometric structure of experts, cross-expert CKA similarity, modality truncation sensitivity, and the divergence in utilization rates of Hot/Cold tokens. These observations provide intuitive and sufficient design foundations for importance weighting, adaptive rank allocation, and candidate expert restriction.

  • S3. Organic integration of low-rank decomposition and underlying inference operator optimization: Breaking the limitation of merely theoretical parameter reduction, the proposed Shared-factor Execution strategy reuses input dimensionality reduction across experts for Up/Gate projections and performs a single output after low-rank aggregation of Down projections. It effectively mitigates the split multiplication latency drawback commonly seen in low-rank decomposition.

  • S4. Remarkable end-to-end acceleration potential with targeted ablation validations: Evaluations across backbone models (SDAR and LLaDA2.0-mini) and tasks (code and mathematical reasoning) achieve an end-to-end speedup of up to 7.22×. Meanwhile, ablation experiments are designed for importance weighting, learnable rank truncation and residual compensation to verify the effectiveness of each component in preserving model capacity.

Critical weaknesses
  • W1. The independent improvement of IATC over TD-MoE and the exclusivity of its execution optimization are questionable. Existing ablation studies only compare naive unweighted decomposition and fixed-rank allocation, failing to decouple the independent contribution of IATC relative to TD-MoE. Meanwhile, it remains unclear whether Shared-factor Execution based on the standard Tucker structure is exclusive to this work and whether it can directly boost TD-MoE.

  • W2. The incremental value of TCR over TEAM and the necessity of Hot-only compensation lack verification. The method is built directly on the TEAM framework but is not benchmarked against its native routing. In addition, ablation experiments only verify the benefits of introducing residuals, and fail to demonstrate that the online dynamically executed "Hot-only compensation" outperforms "full-token compensation", which can be directly fused into weights offline.

  • W3. There is a magnitude discrepancy in the reported 4–7× end-to-end speedup gain, with no closed-loop decomposition. The reported local speedups (up to 2.18×2.18\times for Shared-factor and up to 13.0%13.0\% for constrained routing) cannot account for the overall 4–7× speedup in terms of magnitude. Furthermore, it cannot reasonably explain why nearly double the speedup over the underlying TEAM framework is achieved with merely 3% parameter compression.

Questions for authors
  • Q1. Controlled comparison between IATC and TD-MoE: With TCR and inference execution strategy fixed uniformly, what is the standalone performance gain of IATC over TD-MoE? Can TD-MoE also achieve similar acceleration via Shared-factor Execution?

  • Q2. Trade-off between the standalone benefits of TCR and compensation overhead: What advantages does Centroid-based Routing have over the native routing of TEAM in terms of fidelity, memory access volume and latency? Does Hot-only compensation truly outperform offline-fusable Token-agnostic compensation in the final end-to-end accuracy-latency trade-off?

  • Q3. Component-wise latency ablation and overhead accounting: Could you provide an end-to-end latency breakdown table that incrementally adds each component from the Dense Baseline and TEAM to form the full model, and quantify the online overhead introduced by hot token compensation to explain the source of the high speedup under a 3% compression ratio?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.