- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes using a GNN to study inter-layer routing dependencies in MoEs. The proposed method is reported to accurately predict routing trajectories. The learned expert embeddings are used to allocate bits to routed experts in a mixed precision setting where expert weights receive 8- or 4-bits. The proposed method is compared to prior quantization schemes.
Critical strengths
- Expert interactions across layers remains an underexplored topic. Establishing a better understanding of the inter-layer relations between routed experts is a well motivated perspective for improving MoE compression.
Critical weaknesses
- The method is poorly defined. Several terms lack precise definitions such as “router scores” (raw router logits, softmax logits, or post top-k renormalization?), “contribution ratio”, “contribution entropy”, “PageRank-like statistics”, and “normalized risk prior”. It is generally unclear how the optimization described in Section 3.6 is converted into discrete precision assignments for experts. I was unable to find implementation of the optimization described in Section 3.6 in the provided supplementary material.
- The experimental details and design are lacking. How the baseline methods were extended to a 5-bit setting is unclear. Training and quantization settings are also poorly described. Results use a single seed. Calibration data and splits are not defined. Controlled ablations are absent which would establish whether the learned graph structure adds value beyond the simple aggregate expert routing statistics. Calibration on Qwen3 is restricted to a few thousand tokens, leaving 17.8% of experts without any observations. The second model is named “Mixtual V0.1,” but no checkpoint link or citation is provided. This appears to be a simple misspelling of Mixtral, but the authors explicitly claim it is a “Mixtral-style model”.
Questions for authors
- What is the provenance of the calibration data used?
- Please confirm the optimization procedure. How are Eq. 13 and 3 used to allocate the precision? The supplied supplemental material contains a completely different heuristic using a weighted combination of activation, routing, and graph statistics (stage4_graph_encoder.py, L199) then assigns precision by sorting those scores (stage6_allocate_bits.py L162). This heuristic does not appear to be described in the paper.