acceptodds
Under review as a conference paper at ICLR 2027

ShadowMoE: Graph-Conditioned Shadow Expert Representations for Mixed-Precision MoE Quantization

Abstract

Sparse mixture-of-experts (MoE) models expose a structured trace of computation through token-level routing: experts are not used independently, but repeatedly cooperate across layers. We ask whether this routing topology contains information about expert vulnerability to model compression. We introduce ShadowMoE, a topology-aware representation framework that converts token-level top-k routing traces into weighted cross-layer expert graphs and learns relational representations of individual experts. A graph-restricted encoder propagates information only along observed routing paths, while leakage-controlled structural objectives test whether the learned representation captures non-local routing information. We further probe whether these representations predict expert-level quantization residuals and use them to construct budget-constrained mixed-precision policies. On Qwen3-30B-A3B and Mixtual V0.1, ShadowMoE achieves a strict link-prediction AUC of 0.9432, compared with 0.8378 for a graph-free MLP, and its expert representations explain R2 = 0.5405 and 0.5953 of FP8 and INT4 local quantization residuals, respectively. At an average 5.0-bit budget, the resulting INT4/FP8 allocation retains 99.43% and 98.33% of FP8+activation quantized accuracy on the two models, respectively. Importantly, independently estimated expert risks do not reliably compose into scheme-level degradation: the correlation between predicted local risk and complete-scheme loss is only. 0.0580. This negative result suggests that compression sensitivity in MoEs is relational rather than purely expert-wise. ShadowMoE therefore provides both a practical compression mechanism and a framework for studying how routing topology encodes functional dependencies among experts. Anonymous artifacts: https://anonymous.4open.science/r/topology_0723-70B3.

Primary area:
reinforcement learning
Keywords:
Mixed-precision quantization, Graph Transformer, Expert-Aware Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_M9oVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes using a GNN to study inter-layer routing dependencies in MoEs. The proposed method is reported to accurately predict routing trajectories. The learned expert embeddings are used to allocate bits to routed experts in a mixed precision setting where expert weights receive 8- or 4-bits. The proposed method is compared to prior quantization schemes.

Critical strengths
  • Expert interactions across layers remains an underexplored topic. Establishing a better understanding of the inter-layer relations between routed experts is a well motivated perspective for improving MoE compression.
Critical weaknesses
  • The method is poorly defined. Several terms lack precise definitions such as “router scores” (raw router logits, softmax logits, or post top-k renormalization?), “contribution ratio”, “contribution entropy”, “PageRank-like statistics”, and “normalized risk prior”. It is generally unclear how the optimization described in Section 3.6 is converted into discrete precision assignments for experts. I was unable to find implementation of the optimization described in Section 3.6 in the provided supplementary material.
  • The experimental details and design are lacking. How the baseline methods were extended to a 5-bit setting is unclear. Training and quantization settings are also poorly described. Results use a single seed. Calibration data and splits are not defined. Controlled ablations are absent which would establish whether the learned graph structure adds value beyond the simple aggregate expert routing statistics. Calibration on Qwen3 is restricted to a few thousand tokens, leaving 17.8% of experts without any observations. The second model is named “Mixtual V0.1,” but no checkpoint link or citation is provided. This appears to be a simple misspelling of Mixtral, but the authors explicitly claim it is a “Mixtral-style model”.
Questions for authors
  • What is the provenance of the calibration data used?
  • Please confirm the optimization procedure. How are Eq. 13 and 3 used to allocate the precision? The supplied supplemental material contains a completely different heuristic using a weighted combination of activation, routing, and graph statistics (stage4_graph_encoder.py, L199) then assigns precision by sorting those scores (stage6_allocate_bits.py L162). This heuristic does not appear to be described in the paper.
Reviewer_yhKNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cross-layer relationships between experts in adjacent layers and represents these relationships as topology graphs, using this information as a reference for their sensitivity to mixed-precision quantization. The goal is to construct a mixed-precision quantization scheme with INT4 and FP8. The authors record token-level routing traces to construct links between experts in adjacent layers that process the same token. The resulting representations are combined with quantization features and activation statistics for quantization.

The link prediction experiment shows that the graph encoder can predict connections between two experts. The experimental section also evaluates some simple downstream tasks on Qwen3-30B-A3B-base and Mixtual V0.1. The main comparison evaluates the proposed method against other quantization methods at an average 5-bit setting. The accuracy drop is acceptable, but the ablation study is not complete enough. They use SGLang for the implementation, but the deployment details are not fully reported. These issues are discussed in the weaknesses and questions below.

Overall, the paper is clearly presented, but some implementation details are missing.

Critical strengths
  • S1: I think this paper provides a useful graph-based perspective on measuring relationships between experts. The method for learning relationships between experts in adjacent layers is supported by strict held-out link prediction experiments. The authors remove selected edges and use the graph encoder to predict connections between experts, obtaining a high link-prediction AUC. The graph encoder achieves a better result than the graph-free MLP baseline. These results provide empirical support for learning cross-layer relationships between experts from routing topology.
  • S2: It looks to me that the result is impressive in mixed precision. The accuracy drop is very small.
Critical weaknesses
  • W1: It appears that the pipeline connecting the graph and mixed-precision allocation is not very easy to follow. How is the output of the graph used as the initial input for mixed-precision allocation? Although the graph encoder performs well at predicting connections between two experts in adjacent layers, how these relationships are used in mixed-precision allocation is not fully explained. Whether using the graph would change the current results requires further study. For example, if they do not use the graph and only use quantization vulnerability to allocate the precision for each expert, would that be worse or better?
  • W2: Also, the authors compare against only MxMoE as a mixed-precision MoE quantization baseline. The coverage of allocation methods is limited. QuaRot and GPTQ are primarily quantization methods, not mixed-precision allocation methods, so their expert bit-width assignments are not specified. If the authors use only a naive bit-width allocation method, the comparison is not fair to these two baselines. So I think more comparison baselines that were initially designed for mixed-precision bit-width assignment should be compared in the main experiments.
  • W3: Furthremore, this method uses a very small calibration set to collect relationships between experts, but the authors report that about 18% of experts do not have observed routing representations. How are these experts handled in the graph, and how are bit widths assigned to these unobserved experts? The initialization and bit-width assignment for these missing experts should be stated.
  • W4: The experiments cover only two model families and six commonsense, reading comprehension, and QA tasks, which are relatively simple for these advanced MoE models. No math, coding, or other complex tasks are included. These evaluation tasks could therefore be too simple to validate whether the method is effective for compressing or quantizing MoE models. Advanced MoE models may have very little accuracy drop on these tasks. However, if we change to smaller or less advanced models and use more complex QA benchmarks, the accuracy may be hard to preserve, but that is the real application scenario. Therefore, a comprehensive evaluation is recommended, including more model families, larger model sizes, and difficult and complex tasks, as well as downstream tasks from different domains such as math, coding, or multimodal tasks.
  • W5: It seems that the actual compression benefits are not quantified. Average bit width is an effective way to make a fair comparison, but it does not measure the actual implementation or deployment efficiency of these compression methods. So, actual compression measurements are recommended to be reported, such as model size after compression, peak memory, or actual throughput.
Questions for authors
  • Q1: How does the topology of the graph actually affect expert vulnerability prediction and mixed-precision allocation? I wonder how the topology serves as the predictor's input and how it participates in precision choices.
  • Q2: How was the static policy in Table 1 produced? Were differentiable policy search and Gumbel–Softmax used for the main table? How were the INT4 and FP8 expert counts determined to match the average 5.0-bit budget? What was the actual procedure used in the main experiments?
  • Q3: What allocation policies were used for the QuaRot and GPTQ baselines? As I mentioned above, if the authors use only a very naive bit-width allocation method for these two methods, it is not fair to them because they do not provide an official mixed-precision allocation method. It would be better to compare against other mixed-precision methods rather than methods with a single quantization bit width.
  • Q4: What is the quantization setting for the main experiments: weight-only or W/A? If both weights and activations are quantized to INT4 or FP8, does the target GPU natively support the required operations? The authors report that they use Ampere, but it does not support native FP8 Tensor Core operations. So, I wonder what “+Act” means in the main table and whether the weights are dequantized for computation.
  • Q5: The loss used for local risk measurement may not reliably reflect the final quantization setting. Each time, only one expert is perturbed rather than all experts, so other unquantized layers and experts may compensate for the quantization error. It is therefore hard to infer the joint effect of quantizing multiple experts from these individual measurements.
  • Q6: According to the method description, the method constructs one graph for each prompt separately before pooling. Why not construct a single global graph instead using all these prompts? Is there any evidence that constructing a graph for each prompt can provide more precise estimates than constructing a global graph? I did not find a clear design motivation for this.
Reviewer_KCZJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

ShadowMoE asks whether MoE routing topology tells us how vulnerable each expert is to compression. Token-level top-k routing traces are turned into per-prompt weighted directed graphs over (layer, expert) nodes, where an edge accumulates the product of router scores between adjacent layers. A graph transformer that attends only over observed predecessors is trained with held-out link prediction and masked feature reconstruction, and its states are pooled into a 256-d embedding per expert. These embeddings are probed against per-expert pseudo-quantization residuals and, together with local quantization features, used to initialize a differentiable INT4/FP8 precision policy at an average budget of 5.0 bits.

Critical strengths
  1. The paper has a clear and testable hypothesis with leakage control: Using cross-layer routing as relational context for experts is a fresh angle on MoE compression. The authors also caught and discarded a leaky link-prediction protocol in which a router-only baseline reached the same AUC. I appreciate this kind of care in the evaluation.
Critical weaknesses
  1. The end-task gain is not attributed to topology: The allocator's risk prior combines z_e with pseudo-quantization features and activation statistics (Section 3.5), and the local residual is measured directly for every expert. The paper never shows what happens when precision is allocated from the measured residuals alone, when z_e is removed or replaced with the graph-free MLP embedding, or when an MxMoE-style frequency/importance score is combined with the residuals. Several numbers suggest topology adds little: the risk predictor, high-risk retrieval recall@10 is 0.29, and scheme-level calibration. As it stands, the main claim that routing topology encodes compression vulnerability is not supported.

  2. Narrow evaluation and questionable baselines: All end-task results are zero-shot, likelihood-scored commonsense multiple-choice tasks on a base model. Compression damage in MoEs tends to show up on generative math and code tasks, which multiple-choice benchmarks hide [4], and no GSM8K, MATH, HumanEval or perplexity results are given. Some baselines also look misconfigured. INT4+Act drops to near chance (Winogrande 0.5375, ARC-e 0.3413 on Qwen3), GPTQ W4A16 at "avg 5.0" is below what 4-bit GPTQ usually reaches on a 30B MoE, and it is unclear how a W4A16 method or QuaRot is placed at an average 5.0-bit budget at all. ShadowMoE at 5.0 bits also beats FP8 on Winogrande (0.7324 vs. 0.7269), which is hard to reconcile with quantizing part of the experts to INT4. Finally, no memory or latency numbers are reported (Sections 4.3 and B.2), so the practical value of the policy is unknown.

  3. Missing related work: Apart from MxMoE, the paper does not discuss MoE-specific compression with per-expert bit allocation, such as MC-MoE [1], which assigns mixed precision per expert from activation frequency and importance via linear programming, or MoE quantization benchmarks [5]. It should also discuss [4], which shows that activation frequency, router score and weight magnitude are poor importance proxies in MoEs and that importance concentrates in a small set of intermediate dimensions, and [6], which shows that rarely activated experts still hold knowledge the model needs. The second point matters here because 1,096 of the 6,144 Qwen3 experts never appear in the 200-prompt trace corpus and get a zero embedding. For the allocation step itself, Hessian-based mixed-precision methods such as HAWQ-V2 [2] are a direct alternative to the learned policy, and earlier MoE quantization work such as QMoE [3] and MoQE [7] should be discussed as well.

References:

[1] Wei Huang et al. MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More. ICLR 2025.

[2] Zhen Dong et al. HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks. NeurIPS 2020.

[3] Elias Frantar and Dan Alistarh. QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models. MLSys 2024.

[4] Haoze He, Xinkai Zou et al. Less is MoE: Trimming Experts in Domain-Specialist Language Models. EMNLP 2026.

[5] Pingzhi Li et al. Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark. arXiv:2406.08155, 2024.

[6] Haoze He et al. Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning. COLM 2026.

[7] Young Jin Kim, Raffy Fahim, Hany Hassan Awadalla. Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness. arXiv:2310.02410, 2023.

Questions for authors
  1. What retention does the 5.0-bit allocator reach if the risk prior uses only the measured local residuals, with no z_e, and if z_e comes from the graph-free MLP instead? If either matches ShadowMoE, the topology representation is not what drives the result. (Clarification; this could change my rating.)

  2. How are QuaRot and GPTQ W4A16 placed at an average 5.0-bit budget, and which experts get FP8 in those baselines? Why does INT4+Act fall to near-random accuracy?

  3. How does the precision policy treat the 1,096 experts that never appear in the traces, and how many of them end up in INT4?

  4. Could the authors report GSM8K or MATH, a code benchmark, and WikiText perplexity for BF16, FP8, MxMoE and ShadowMoE on Qwen3-30B-A3B?

  5. Suggestion: what does the full pipeline cost (trace collection, graph training, residual labelling, policy search) compared with measuring per-expert residuals and allocating greedily? This decides whether the topology route is worth its complexity in practice.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.