- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents ITC-MoE, an importance-aware and token-adaptive compression framework for Mixture-of-Experts Diffusion Language Models (MoE DLMs), aiming to reduce model storage requirements and inference costs. The authors identify two key characteristics of MoE DLMs: (1) non-uniform low-rank redundancy across different weight dimensions, and (2) distinct feature spectral distributions and expert activation patterns between hot and cold tokens. Building on these observations, the authors introduce two main components: (1) Importance-guided Adaptive Tucker Compression (IATC), which leverages activation and gradient statistics to guide the Tucker decomposition of expert weights, adaptively allocates compression ranks across different dimensions under a fixed parameter budget, and reduces redundant computation through shared factors; and (2) Token-aware Compensation and Routing (TCR), which applies low-rank residual compensation to compression-sensitive hot tokens while restricting the candidate expert set for cold tokens to reduce inference overhead. Experiments on SDAR and LLaDA2.0-mini demonstrate that ITC-MoE achieves higher inference speedups than existing compression methods, with a maximum reported speedup of 7.22× under certain configurations, while largely preserving model accuracy.
Critical strengths
-
Well-motivated method design: The authors provide empirical observations on weight spectral distributions, dimension-wise compression sensitivity, and differences in expert activation patterns between hot and cold tokens. These observations provide a reasonable foundation for the proposed method.
-
Effective integration of model compression and inference optimization: Beyond reducing parameter counts through Tucker decomposition, the proposed method leverages shared factors to enable computational reuse across experts, aiming to translate model compression into practical inference speedups.
Critical weaknesses
-
Potential significant degradation in model accuracy: The proposed method appears to incur substantial accuracy degradation on several benchmarks. Accuracy is largely preserved only on relatively simple datasets, such as MultiArith, SingleOp, and SingleQ, while noticeable performance drops are observed even on moderately challenging benchmarks such as MBPP and GSM8K. These results raise concerns about the practical applicability of the proposed method, particularly for more complex tasks.
-
Insufficient analysis and ablation studies of end-to-end speedups: The paper lacks a detailed explanation of the substantial inference speedups reported in Table 2, particularly why ITC-MoE significantly outperforms existing methods at comparable compression ratios. Furthermore, Table 3 reports several-fold speedups with a compression ratio of only 3%, which appears surprisingly large and warrants further investigation. The paper would benefit from additional ablation studies and a more detailed breakdown of the sources of inference acceleration to substantiate these results.
Questions for authors
-
Could the authors provide additional evaluation results on more challenging benchmarks, such as HumanEval, MATH-500, and AIME, to better assess the effectiveness of the proposed method on complex tasks?
-
Could the authors provide more detailed ablation studies and explanations for the inference speedups reported in Tables 2 and 3? In particular, please address the concerns raised in Weakness 2 regarding the substantial speedups over existing methods at comparable compression ratios and the surprisingly large acceleration achieved with only 3% compression. Clarifying these results is important for assessing the validity of the reported improvements and will directly influence my overall rating.
-
The paper mentions the use of a calibration set but does not provide sufficient details about its construction. Could the authors clarify how the calibration set was constructed and whether it was strictly separated from the evaluation datasets to avoid potential data leakage?