- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a heterogeneous analog-digital mapping strategy for Mixture-of-Experts (MoE) models. The key idea is to execute noise-sensitive experts on digital hardware while keeping the remaining experts on analog units. The authors introduce MaxNNScore, a weight-norm-based metric for identifying sensitive experts, and provide theoretical analysis suggesting improved noise tolerance under selective digital mapping. Experiments on three MoE models and seven benchmarks show substantial accuracy recovery with a relatively small fraction of parameters mapped to digital.
Critical strengths
-
MaxNNScore relies only on model weights and requires no retraining or calibration data. This makes the method easy to apply to large MoE models, where more expensive sensitivity estimation can be impractical.
-
The method achieves substantial accuracy recovery across three MoE architectures and generally performs competitively with or better than existing expert selection baselines. The results suggest that expert-level mapping is a useful direction for improving analog inference.
Critical weaknesses
-
The theoretical analysis relies on a highly simplified single-layer MoE model with fixed down-projection weights and a synthetic data distribution. It is unclear whether the derived relationship between expert weight norms and noise sensitivity extends to pretrained Transformer MoEs. In addition, the proof of Lemma 4.1 appears to rely on asymptotic upper and lower bounds to establish a strict ordering of expert scores, which requires further justification. These issues weaken the paper's central claim of theoretically justified expert selection.
-
The experiments show that MaxNNScore is competitive with the noise-aware ENS baseline, but its advantage is limited or inconsistent on some models. The paper does not directly establish how well MaxNNScore correlates with actual expert noise sensitivity. Furthermore, the energy and latency results are based on analytical estimates rather than measured hardware performance. This makes the practical advantages less conclusive.
Questions for authors
-
Can the authors provide quantitative evidence that MaxNNScore correlates with actual expert-level accuracy degradation under analog noise? This would help establish whether the metric measures noise sensitivity rather than simply weight magnitude.
-
What is the computational cost of MaxNNScore compared with ENS? Given their similar performance on DeepSeekMoE, quantifying the selection overhead would help clarify the practical advantage of the proposed method.
5 comments
(Response to Weakness 2) On the estimates of energy efficiency and latency values
Measuring the energy consumption and latency values in real analog in-memory hardware requires getting access to sufficient in-memory computing resources and configuring the in-memory computing system according to the need. Configuring such systems for sole research purposes usually falls beyond the scope. Therefore, it is a common practice in the analog in-memory computing of neural networks literature to report the estimated values [6,7,8]. Note that our energy efficiency and latency estimates are based on the latency and energy consumption values measured on a standard AIMC system; see Appendix C for details. Furthermore, the energy efficiency and latency values for full-precision and digital compression baselines are upper- and lower-bound estimates, which reinforce the practical advantage of the proposed mapping.
(Response to Question 2) On the computational cost of MaxNNScore compared with ENS
The computational cost of ENS is incomparably higher than that of the proposed MaxNNScore
ENS requires computing expert output separately for each layer over multiple random seeds for a sufficiently large calibration dataset. Therefore, the computational cost grows quadratically with the number of layers (i.e., complexity, where is the total number of layers in the model). On the other hand, MaxNNScore requires a very insignificant amount of compute required to calculate the neuron norms and does not require inference on calibration data.
We provide the cost of evaluating ENS and MaxNNScore for DeepSeekMoE in the following table. The costs are evaluated on a 32 GB V100 GPU. ENS of each expert is evaluated over 8 random seeds for a randomly selected 2048 samples from WikiText-2.
| Method | Compute cost (GPU-hours) |
|---|---|
| MaxNNScore | 0.006 |
| ENS | 16 |
References
[1] Bu, Dake, Wei Huang, Taiji Suzuki, Ji Cheng, Qingfu Zhang, Zhiqiang Xu, and Hau-San Wong. "Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples." In International Conference on Machine Learning, pp. 4642-4695. PMLR, 2024.
[2] Arora, Sanjeev, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. "Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks." In International conference on machine learning, pp. 322-332. PMLR, 2019.
[3] Allen-Zhu, Zeyuan, and Yuanzhi Li. "Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning." In The Eleventh International Conference on Learning Representations. 2023.
[4] Chen, Zixiang, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. "Towards understanding mixture of experts in deep learning." arXiv preprint arXiv:2208.02813 (2022).
[5] Chowdhury, Mohammed Nowaz Rabbani, Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Yu Chen. "Patch-level routing in mixture-of-experts is provably sample-efficient for convolutional neural networks." In International Conference on Machine Learning, pp. 6074-6114. PMLR, 2023.
[6] Büchel, Julian, Athanasios Vasilopoulos, William Andrew Simon, Irem Boybat, HsinYu Tsai, Geoffrey W. Burr, Hernan Castro et al. "Efficient scaling of large language models with mixture of experts and 3D analog in-memory computing." Nature Computational Science 5, no. 1 (2025): 13-26.
[7] Büchel, Julian, Iason Chalas, Giovanni Acampa, An Chen, Omobayode Fagbohungbe, Hsinyu Tsai, Kaoutar El Maghraoui, Manuel Le Gallo, Abbas Rahimi, and Abu Sebastian. "Analog foundation models." Advances in Neural Information Processing Systems 38 (2026): 61179-61228.
[8] Jain, Shubham, Hsinyu Tsai, Ching-Tzu Chen, Ramachandran Muralidhar, Irem Boybat, Martin M. Frank, Stanisław Woźniak et al. "A heterogeneous and programmable compute-in-memory accelerator architecture for analog-AI using dense 2-D mesh." IEEE Transactions on Very Large Scale Integration (VLSI) Systems 31, no. 1 (2022): 114-127.
Thank you for the detailed response and additional experiments. The expert-level sensitivity analysis provides useful evidence for the effectiveness of MaxNNScore, and the computational cost comparison helps clarify its advantage over ENS. I appreciate the authors' efforts to address these concerns.
My main remaining concern is the choice of experimental models. The evaluation is based on relatively old MoE architectures, which makes it difficult to assess how well the proposed method generalizes to more recent and larger-scale MoE models. Given the rapid development of MoE architectures, I believe experiments on newer models would be important to establish the practical relevance of the method.
We thank the reviewer for their valuable comments and suggestions. We provide our responses below.
(Response to Weakness 1) On the limitations and assumptions of theoretical analysis and the practical implications in pretrained transformer MoEs
Our theoretical setup is state-of-the-art in MoE training dynamic literature
Our theoretical simplification reflects the current frontier of theoretical analysis of neural-network training dynamics and generalization analysis. Rigorous characterization of feature-learning dynamics and generalization remains challenging even for standard deep neural networks, due to the coupled, nonconvex evolution of representations across layers. Thus, existing theoretical analyses on training dynamics and generalization of neural networks are largely restricted to shallow networks and simplified data distributions [1-3]. Therefore, the existing theoretical studies of MoE models similarly adopt simplified expert architectures, routing mechanisms, optimization procedures, and data models [4–5]. The simplification in this paper follows the setup of these state-of-the-art theoretical analyses.
The simplification of the setup allows us to quantify the overall effect of analog noise on model accuracy to justify the rigorousness of the proposed method
A direct relationship between the noise sensitivity of expert output and the MaxNNorm can be established without taking any assumptions on input data or expert weights. See the following theorem that we developed in response to reviewer 8LXF.
Theorem: Let us denote Wd×m as an expert projection matrix and x as its input. Let us define C:=∑u=13cu. Then the root-mean-square output error satisfies
(E[∣∣y^−y∣∣22])1/2≤m(C∣∣x∣∣1+cs)MaxNNorm(W)
where y^ and y are the noisy and noise-free outputs, respectively.
However, the above result does not characterize the effect of analog noise of different experts on the model’s overall generalization performance.
To characterize the model’s overall generalization performance under analog noise, we first need to characterize how much each expert contributes to the model output for different inputs and how much these contributions are affected by the analog noise. Our data model assumptions and the simplified network architecture facilitate this theoretical characterization and allow us to formally quantify the overall noise tolerance of different experts in order to maintain the original generalization performance as presented in Theorem 4.2.
Justification of the strict ordering of MaxNNScore in Lemma 4.1
From the proof of Lemma K.1 in appendix, it is evident that the asymptotic bounds of MaxNNScore for both groups experts learning more-frequent and less-frequent tokens maintains same hidden constant for the given selection of T. Therefore, the MaxNNScore of the experts learning more frequent tokens is at least α1−2α times higher as shown in the proof. Now, as α<1/4, the strict ordering in Lemma 4.1 is justified.
(Response to Weakness 2 and Question 1) On the correlation of MaxNNScore and the accuracy-level experts’ noise sensitivity
We extensively verify the strong correlation between the experts’ MaxNNScore and their noise effect on the model’s overall accuracy
We conduct an extensive experiment to validate the high sensitivity of the model accuracy to the analog noise of the top MaxNNScore experts. To measure the sensitivity of model accuracy to the analog noise of each expert, we only map the expert of interest in analog, while the rest of the experts are mapped in digital. We measure the average accuracy under this mapping for each expert separately on the benchmark LLM tasks. The results are presented in Figure 6-13 of this anonymous link for OLMoE:
https://storage.googleapis.com/anonymous_link/accuracy_level_expert_sensitivity.pdf
As observed in these figures, the experts yielding noticeable accuracy degradation are also the top MaxNNScore experts. In contrast, the noise-aware ENS baseline, which directly measures the experts’ output degradation under analog noise, fails to include some of these critical experts (e.g., expert 27 in layer 4) in its top-k ranking, as the metric ignores the experts’ contributions to model accuracy. Note that using the accuracy-level sensitivity itself for expert ranking is prohibitively expensive, as the process requires separate repetitive evaluation of accuracy over multiple random seeds for each expert of the model. For example, our experiment on OLMoE requires 3840 GPU-hours on V100 GPUs. In contrast, the proposed MaxNNScore-based identification does not require any inference on calibration data.