acceptodds
Under review as a conference paper at ICLR 2027

CLASP: Cross-Layer Active-Subspace Pooling for Quantized Mixture-of-Experts

Abstract

Low-rank corrections can recover accuracy lost to aggressive post-training quantization of mixture-of-experts (MoE) language models. Most low-bit MoE correction methods fit their factors independently or within individual layers. We find substantial overlap between experts' activation-weighted dominant subspaces across layers. We use this overlap in Cross-Layer Active-Subspace Pooling (CLASP), which clusters experts by the Grassmannian distance between their active subspaces and fits a shared low-rank basis to each cluster. Sharing reduces basis storage and allows a larger correction rank at a fixed bit budget. In the evaluated Qwen models, experts activated by the same token also tend to belong to a few clusters. CLASP stores the shared factors in a device-resident pool and batches correction operations, with stream overlap reducing execution overhead. At 2.15–2.21 bits per quantized parameter, CLASP achieves the lowest perplexity among the compared quantizers on four MoE models. Under the same measurement protocol and hardware, end-to-end generation is up to faster than the FP16 baseline. Our source code will be available after acceptance.

Primary area:
optimization
Keywords:
Mixture-of-Experts (MoE), Post-Training Quantization (PTQ), Low-Rank Correction, Cross-Layer Subspace Sharing, Active-Subspace Pooling, Grassmannian Clustering, Efficient LLM Inference

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_TF7UOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a post-training quantization method CLASP for mixture-of-experts (MoE) language models that combines cross-layer shared low-rank correction with inference-side system optimization. The work is motivated by two key empirical findings: first, the activation-weighted dominant subspaces of experts overlap significantly across transformer layers, with 71% of experts having their nearest subspace neighbor in a different layer; second, experts activated by the same token tend to concentrate in a small number of subspace clusters. Based on these observations, CLASP groups experts across all layers using spectral clustering on Grassmannian principal-angle distances, then learns a shared low-rank basis for each cluster while retaining per-expert private projection factors. This basis sharing reduces the storage cost of correction factors and allows a higher correction rank under a fixed bit budget. For inference, the paper proposes a device-resident factor pool, batched correction computation, and CUDA stream overlap to minimize execution overhead. Experiments across four MoE models show that CLASP achieves state-of-the-art perplexity at 2.15–2.21 bits per quantized parameter, and delivers up to 4.9× end-to-end generation speedup over the FP16 baseline along with 4.3×–5.4× weight memory reduction.

Critical strengths
  1. The core idea of cross-layer subspace pooling is supported by both empirical evidence and formal analysis. The paper rigorously proves the basis-invariance of the Grassmannian distance metric, ensuring the validity of the clustering criterion. It also derives the activation-weighted reconstruction objective from the expected output error, and establishes that cross-layer partitioning strictly enlarges the feasible solution space compared to layer-constrained sharing, providing a clear theoretical lower bound on fitting performance. This stands out from many heuristic quantization methods.

  2. The work delivers not only a compression algorithm but also a fully optimized inference pipeline, making it practically relevant for deployment. The ablation studies are carefully controlled: they isolate the contribution of subspace-aware clustering by comparing against random assignment, quantify the impact of correction rank and cluster size, and break down the speedup from each execution optimization (batching, graph capture, stream overlap). The evaluation covers both language modeling quality and system-level performance, offering a complete picture of the tradeoffs.

Critical weaknesses
  1. The activation-weighted objective uses a diagonal approximation of the input covariance matrix with a hand-tuned exponent α, which is an engineering heuristic rather than a formally justified target. The paper does not provide an error bound for this approximation, nor does it analyze how the approximation error behaves as model depth, expert count, or feature dimension grows. Additionally, the size-balancing step applied after spectral clustering is a post-hoc adjustment that deviates from the optimal clustering objective, with no theoretical analysis of its impact on reconstruction quality.

  2. All four evaluated MoE models (Qwen1.5-MoE-2.7B, Mixtral-8x7B, Qwen3-30B-A3B, DeepSeek-MoE-16B) are relatively dated and limited to mid parameter scales (up to 30B). None of the latest-generation MoE architectures—such as larger-scale models with hundreds of billions of parameters, or models with advanced routing mechanisms like token-choice routing or expert specialization diversification—are tested. This limits the ability to assess whether the cross-layer subspace overlap phenomenon and CLASP’s effectiveness generalize to modern MoE designs.

  3. Among MoE-specific compression methods, only TileQ is compared as a shared low-rank counterpart, while other peer methods with similar shared decomposition designs, including MoE-SVD, TD-MoE, and KBVQ-MoE, are absent. Without these comparisons, the competitiveness of the cross-layer subspace pooling paradigm against alternative sharing strategies cannot be fully validated. Additionally, the TileQ results are taken directly from published work instead of being reimplemented under the same code framework and calibration pipeline, introducing uncontrolled variables from implementation differences and calibration settings that further weaken the rigor of the comparison.

Questions for authors
  1. The activation-scale exponent α is fixed at 2.4 across experiments. Is there a theoretical justification for this specific value, and has a systematic sensitivity analysis of α been performed? Furthermore, we wonder if the diagonal activation approximation may introduce non-negligible bias in larger, more deeply specialized MoE models, and if so, how this would affect the cross-layer pooling strategy.

  2. All experiments are conducted on a set of relatively early and mid-sized MoE models, with no evaluation on more recent or larger MoE architectures. Could you elaborate on the rationale for choosing these particular models, and share any results or insights regarding the generalization of CLASP to state-of-the-art MoE models with more complex expert structures and routing policies?

Reviewer_egsSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CLASP, a post-training quantization method for MoE models. It clusters experts across layers based on activation-weighted subspace similarity, shares low-rank correction bases within clusters, and optimizes inference through batched corrections and CUDA stream overlap. Experiments on four MoE models show improved perplexity and up to 4.9× speedup over FP16.

Critical strengths
  • Cross-layer subspace sharing is an interesting direction for reducing redundant low-rank correction parameters.
  • This paper is overall well-presented with RQs and figures
Critical weaknesses
  • The paper misses several highly relevant methods, including MiLo, MoE-SVD, and GlowQ. In particular, MiLo and GlowQ are closely related to low-rank correction and factor sharing. Without these comparisons, the improvement over existing methods is not convincing.
  • The paper does not compare cross-layer clustering against an otherwise identical layer-restricted clustering baseline. The random-clustering ablation cannot establish whether crossing layer boundaries is actually responsible for the improvements.
  • The reported 4.9× speedup is relative to FP16 rather than competitive quantized inference baselines. Moreover, the ablation indicates that most execution gains come from batching, making the efficiency contribution of cross-layer sharing unclear.
  • Despite lower perplexity, CLASP reduces Mixtral's average zero-shot accuracy from 70.92% (TileQ) to 67.09%. On Qwen1.5, it also slightly underperforms TileQ in average accuracy
Questions for authors

See weaknesses

Reviewer_S9SfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is targeting to reduce the storage and inference cost by utilizing low-rank corrections in quantized MoE models. To do so, the authors study the activation-weighted per-expert subspaces, and observe that similar subspaces can appear across different experts on different layers. Based on this observation, this paper proposes CLASP, which groups expert across layers using Grassmannian distance and performs a joint low-rank decomposition within each group. Experts in the same group share a low-rank basis, while keeping their own private factors and quantized residuals. To make inference efficient, the authors further introduce a shared basis pool, batched correction execution, and stream overlap between the quantized backbone and the low-rank correction.

Experiments include Qwen1.5-MoE, Qwen3-MoE, Mixtral, and DeepSeek-MoE. The authors report lower perplexity across all four models compared to the listed quantization baselines, while the average downstream accuracy is not consistently better. The system evaluation shows lower weight memory usage and faster end-to-end inference compared to FP16.

Critical strengths

The motivation and the overall sharing idea make sense. Sharing low-rank bases across experts is a reasonable way to reduce repeated storage and use the saved budget for a larger correction rank. The problem is practical and matches the actual needs of inference. Most of the mathematical formulation also makes sense for this problem.

Critical weaknesses
  1. The mathematical connection between Grassmannian clustering and the subsequent joint SVD is not clear to me. Grassmannian distance is used to group experts by their subspaces, and then a joint SVD is performed within each group. However, the paper does not clearly explain why this grouping should lead to better quantization than other grouping methods. The paper also does not clearly isolate how much improvement comes from cross-layer sharing. Although TileQ is a non-cross-layer baseline, this comparison cannot directly prove CLASP's cross-layer gain. Especially considering that TileQ has higher average accuracy on Mixtral, I am not convinced about how much practical improvement this mathematical design actually gives.
  2. The core overlap observation lacks direct validation across different MoE architectures. The cross-layer overlap observation is only measured on Qwen3-30B-A3B, which does not have shared experts. However, this overlap seems closely related to the motivation for using shared experts. I am concerned that this observation may not be true in models with shared experts, such as DeepSeek-MoE. Since in Table 1, the average accuracy gain on DeepSeek-MoE is also very small, I believe the paper should directly check whether the same observation holds in these architectures.
Questions for authors
  1. The system throughput evaluation only compares with FP16. Can the authors also compare throughput with other quantization methods at similar bit per weight?
  2. In Table 1, Qwen1.5 and Mixtral both have lower average accuracy than TileQ. Can the authors analyze why? Given the motivation for cross-layer sharing, why does the expected benefit not appear in these results?
Reviewer_6yPGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CLASP compresses MoE models using cross-layer shared low-rank corrections and a two-bit vector-quantized residual backbone. It clusters experts by principal-angle distances between activation-weighted dominant subspaces, fits a shared basis and expert-specific factors, and batches correction operations at inference. Experiments cover four MoE models, five zero-shot tasks, perplexity, and generation efficiency. The reported perplexity results are promising, while downstream accuracy gains vary by model.

Critical strengths
  1. Promising low-bit perplexity across four architectures. Table 1 reports lower WikiText-2 perplexity than its listed quantizers on all four models. CLASP's reported perplexities also remain below the published TileQ vector-quantization variant, although this remains a cross-paper comparison with different accounting conventions.

  2. Useful controls and an implementable correction path. Table 4 compares similarity-based and random grouping at matched settings, supporting an assignment benefit beyond additional correction capacity. Table 2 separates batching, graph capture and stream overlap, with identical greedy outputs across these execution variants. It identifies batching as the main measured execution improvement.

Critical weaknesses
  1. The closest baseline is incompletely specified and does not reflect its stronger variant. CLASP uses vector quantization, but most TileQ entries in Table 1 match TileQ's scalar variant.

  2. The benefit of cross-layer selection is not directly isolated. Table 4 establishes that the chosen grouping outperforms random assignment, but does not compare cross-layer sharing with a feasible layer-constrained alternative under a common budget.

  3. The strongest speedup depends on the reference implementation. Section 5.3 explicitly notes that the Qwen3 FP16 baseline visits all 6,144 expert entries per token. Table 2 usefully isolates batching and scheduling, and the paper acknowledges their contribution. However, the reported 4.86x does not establish the speedup over an FP16 implementation using comparable active-expert dispatch, or isolate the runtime benefit of the shared pool.

Questions for authors
  1. Which source version and TileQs/TileQv row supplies each TileQ entry in Table 1? Please reconcile the Qwen1.5 HellaSwag value and revise the five-task comparisons using the appropriate variants and accounting conventions.

  2. On Qwen3, can you compare cross-layer and layer-constrained sharing using the same VQ backbone, rank, group size and matched storage budget, reporting perplexity and downstream accuracy? Its 128 experts per layer make the default group size feasible in both conditions. Please also state the within-cluster ordering used before Algorithm 1's size-based splitting.

  3. For Qwen1.5, which tensors and denominator produce the 2.15-bit Table 1 and 2.01-bit no-correction Table 3 entries? Please specify whether attention and the shared expert are included, the shared expert's stored precision, and how nominal rates relate to exported bytes. For implementation clarity, does Algorithm 1 line 14 use the effective dequantized INT8 factors stored at line 11?

  4. Please identify the FP16 backend/version and GPU for each Figure 4 result. For Qwen3, can you provide prefill/decode times using comparable active-expert dispatch, or a profiler breakdown quantifying traversal overhead? A pool-specific measurement under the fixed low-bit execution path would also clarify its contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.