- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a post-training quantization method CLASP for mixture-of-experts (MoE) language models that combines cross-layer shared low-rank correction with inference-side system optimization. The work is motivated by two key empirical findings: first, the activation-weighted dominant subspaces of experts overlap significantly across transformer layers, with 71% of experts having their nearest subspace neighbor in a different layer; second, experts activated by the same token tend to concentrate in a small number of subspace clusters. Based on these observations, CLASP groups experts across all layers using spectral clustering on Grassmannian principal-angle distances, then learns a shared low-rank basis for each cluster while retaining per-expert private projection factors. This basis sharing reduces the storage cost of correction factors and allows a higher correction rank under a fixed bit budget. For inference, the paper proposes a device-resident factor pool, batched correction computation, and CUDA stream overlap to minimize execution overhead. Experiments across four MoE models show that CLASP achieves state-of-the-art perplexity at 2.15–2.21 bits per quantized parameter, and delivers up to 4.9× end-to-end generation speedup over the FP16 baseline along with 4.3×–5.4× weight memory reduction.
Critical strengths
-
The core idea of cross-layer subspace pooling is supported by both empirical evidence and formal analysis. The paper rigorously proves the basis-invariance of the Grassmannian distance metric, ensuring the validity of the clustering criterion. It also derives the activation-weighted reconstruction objective from the expected output error, and establishes that cross-layer partitioning strictly enlarges the feasible solution space compared to layer-constrained sharing, providing a clear theoretical lower bound on fitting performance. This stands out from many heuristic quantization methods.
-
The work delivers not only a compression algorithm but also a fully optimized inference pipeline, making it practically relevant for deployment. The ablation studies are carefully controlled: they isolate the contribution of subspace-aware clustering by comparing against random assignment, quantify the impact of correction rank and cluster size, and break down the speedup from each execution optimization (batching, graph capture, stream overlap). The evaluation covers both language modeling quality and system-level performance, offering a complete picture of the tradeoffs.
Critical weaknesses
-
The activation-weighted objective uses a diagonal approximation of the input covariance matrix with a hand-tuned exponent α, which is an engineering heuristic rather than a formally justified target. The paper does not provide an error bound for this approximation, nor does it analyze how the approximation error behaves as model depth, expert count, or feature dimension grows. Additionally, the size-balancing step applied after spectral clustering is a post-hoc adjustment that deviates from the optimal clustering objective, with no theoretical analysis of its impact on reconstruction quality.
-
All four evaluated MoE models (Qwen1.5-MoE-2.7B, Mixtral-8x7B, Qwen3-30B-A3B, DeepSeek-MoE-16B) are relatively dated and limited to mid parameter scales (up to 30B). None of the latest-generation MoE architectures—such as larger-scale models with hundreds of billions of parameters, or models with advanced routing mechanisms like token-choice routing or expert specialization diversification—are tested. This limits the ability to assess whether the cross-layer subspace overlap phenomenon and CLASP’s effectiveness generalize to modern MoE designs.
-
Among MoE-specific compression methods, only TileQ is compared as a shared low-rank counterpart, while other peer methods with similar shared decomposition designs, including MoE-SVD, TD-MoE, and KBVQ-MoE, are absent. Without these comparisons, the competitiveness of the cross-layer subspace pooling paradigm against alternative sharing strategies cannot be fully validated. Additionally, the TileQ results are taken directly from published work instead of being reimplemented under the same code framework and calibration pipeline, introducing uncontrolled variables from implementation differences and calibration settings that further weaken the rigor of the comparison.
Questions for authors
-
The activation-scale exponent α is fixed at 2.4 across experiments. Is there a theoretical justification for this specific value, and has a systematic sensitivity analysis of α been performed? Furthermore, we wonder if the diagonal activation approximation may introduce non-negligible bias in larger, more deeply specialized MoE models, and if so, how this would affect the cross-layer pooling strategy.
-
All experiments are conducted on a set of relatively early and mid-sized MoE models, with no evaluation on more recent or larger MoE architectures. Could you elaborate on the rationale for choosing these particular models, and share any results or insights regarding the generalization of CLASP to state-of-the-art MoE models with more complex expert structures and routing policies?