acceptodds
Under review as a conference paper at ICLR 2027

Expert-Level Analog-Digital Mapping of Mixture-of-Experts with Theoretical Generalization Guarantees

Abstract

Mixture-of-Experts (MoE) models enable scalable learning by activating only a subset of parameters per input, but remain memory-bound due to frequent movement of expert weights. Analog in-memory computing (AIMC) offers a promising solution by performing matrix–vector multiplications (MVMs) directly within memory, reducing data movement and improving energy efficiency. However, existing AIMC deployments map all MVMs uniformly to the analog units, leading to significant accuracy degradation for large MoE models due to hardware non-idealities. In this paper, we show that noise sensitivity varies significantly across MVM operations, with dense modules (such as attention layers) and certain experts being particularly vulnerable to analog noise. We then introduce a theoretically grounded metric, the maximum neuron norm score (MaxNNScore), to identify noise-sensitive experts. Our theoretical results prove that selectively mapping high-MaxNNScore experts to the digital units significantly improves the tolerable noise level while maintaining generalization compared to uniform analog execution. Empirical evaluations on large-scale MoE models, including DeepSeekMoE (16B), Qwen-1.5 MoE (14B), and OLMoE (7B), demonstrate that our approach recovers near full-precision performance (within 3-8% drop) over several benchmark LLM tasks while mapping only 11-29% of parameters to digital.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Analog In-memory Computing, Mixture-of-Experts, Theoretical Generalization Guarantee

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_sjxKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a heterogeneous analog-digital mapping strategy for Mixture-of-Experts (MoE) models. The key idea is to execute noise-sensitive experts on digital hardware while keeping the remaining experts on analog units. The authors introduce MaxNNScore, a weight-norm-based metric for identifying sensitive experts, and provide theoretical analysis suggesting improved noise tolerance under selective digital mapping. Experiments on three MoE models and seven benchmarks show substantial accuracy recovery with a relatively small fraction of parameters mapped to digital.

Critical strengths
  1. MaxNNScore relies only on model weights and requires no retraining or calibration data. This makes the method easy to apply to large MoE models, where more expensive sensitivity estimation can be impractical.

  2. The method achieves substantial accuracy recovery across three MoE architectures and generally performs competitively with or better than existing expert selection baselines. The results suggest that expert-level mapping is a useful direction for improving analog inference.

Critical weaknesses
  1. The theoretical analysis relies on a highly simplified single-layer MoE model with fixed down-projection weights and a synthetic data distribution. It is unclear whether the derived relationship between expert weight norms and noise sensitivity extends to pretrained Transformer MoEs. In addition, the proof of Lemma 4.1 appears to rely on asymptotic upper and lower bounds to establish a strict ordering of expert scores, which requires further justification. These issues weaken the paper's central claim of theoretically justified expert selection.

  2. The experiments show that MaxNNScore is competitive with the noise-aware ENS baseline, but its advantage is limited or inconsistent on some models. The paper does not directly establish how well MaxNNScore correlates with actual expert noise sensitivity. Furthermore, the energy and latency results are based on analytical estimates rather than measured hardware performance. This makes the practical advantages less conclusive.

Questions for authors
  • Can the authors provide quantitative evidence that MaxNNScore correlates with actual expert-level accuracy degradation under analog noise? This would help establish whether the metric measures noise sensitivity rather than simply weight magnitude.

  • What is the computational cost of MaxNNScore compared with ENS? Given their similar performance on DeepSeekMoE, quantifying the selection overhead would help clarify the practical advantage of the proposed method.

5 comments

We thank the reviewer for their valuable comments and suggestions. We provide our responses below.

(Response to Weakness 1) On the limitations and assumptions of theoretical analysis and the practical implications in pretrained transformer MoEs

Our theoretical setup is state-of-the-art in MoE training dynamic literature

Our theoretical simplification reflects the current frontier of theoretical analysis of neural-network training dynamics and generalization analysis. Rigorous characterization of feature-learning dynamics and generalization remains challenging even for standard deep neural networks, due to the coupled, nonconvex evolution of representations across layers. Thus, existing theoretical analyses on training dynamics and generalization of neural networks are largely restricted to shallow networks and simplified data distributions [1-3]. Therefore, the existing theoretical studies of MoE models similarly adopt simplified expert architectures, routing mechanisms, optimization procedures, and data models [4–5]. The simplification in this paper follows the setup of these state-of-the-art theoretical analyses.

The simplification of the setup allows us to quantify the overall effect of analog noise on model accuracy to justify the rigorousness of the proposed method

A direct relationship between the noise sensitivity of expert output and the MaxNNorm can be established without taking any assumptions on input data or expert weights. See the following theorem that we developed in response to reviewer 8LXF.

Theorem: Let us denote Wd×m\mathbf{W}^{d\times m} as an expert projection matrix and x\mathbf{x} as its input. Let us define C:=∑u=13cuC:=\sum_{u=1}^3c_u. Then the root-mean-square output error satisfies

(E[∣∣y^−y∣∣22])1/2≤m(C∣∣x∣∣1+cs)MaxNNorm⁡(W)\left(\mathbb{E}\left[\left|\left|\hat{\mathbf{y}}-\mathbf{y}\right|\right|_2^2\right]\right)^{1/2}\le\sqrt{m}\left(C||\mathbf{x}||_1+c_s\right)\operatorname{MaxNNorm}(\mathbf{W})

where y^\hat{\mathbf{y}} and y\mathbf{y} are the noisy and noise-free outputs, respectively.

However, the above result does not characterize the effect of analog noise of different experts on the model’s overall generalization performance.

To characterize the model’s overall generalization performance under analog noise, we first need to characterize how much each expert contributes to the model output for different inputs and how much these contributions are affected by the analog noise. Our data model assumptions and the simplified network architecture facilitate this theoretical characterization and allow us to formally quantify the overall noise tolerance of different experts in order to maintain the original generalization performance as presented in Theorem 4.2.

Justification of the strict ordering of MaxNNScore in Lemma 4.1

From the proof of Lemma K.1 in appendix, it is evident that the asymptotic bounds of MaxNNScore for both groups experts learning more-frequent and less-frequent tokens maintains same hidden constant for the given selection of TT. Therefore, the MaxNNScore of the experts learning more frequent tokens is at least 1−2αα\frac{1-2\alpha}{\alpha} times higher as shown in the proof. Now, as α<1/4\alpha<1/4, the strict ordering in Lemma 4.1 is justified.

(Response to Weakness 2 and Question 1) On the correlation of MaxNNScore and the accuracy-level experts’ noise sensitivity

We extensively verify the strong correlation between the experts’ MaxNNScore and their noise effect on the model’s overall accuracy

We conduct an extensive experiment to validate the high sensitivity of the model accuracy to the analog noise of the top MaxNNScore experts. To measure the sensitivity of model accuracy to the analog noise of each expert, we only map the expert of interest in analog, while the rest of the experts are mapped in digital. We measure the average accuracy under this mapping for each expert separately on the benchmark LLM tasks. The results are presented in Figure 6-13 of this anonymous link for OLMoE:

https://storage.googleapis.com/anonymous_link/accuracy_level_expert_sensitivity.pdf

As observed in these figures, the experts yielding noticeable accuracy degradation are also the top MaxNNScore experts. In contrast, the noise-aware ENS baseline, which directly measures the experts’ output degradation under analog noise, fails to include some of these critical experts (e.g., expert 27 in layer 4) in its top-k ranking, as the metric ignores the experts’ contributions to model accuracy. Note that using the accuracy-level sensitivity itself for expert ranking is prohibitively expensive, as the process requires separate repetitive evaluation of accuracy over multiple random seeds for each expert of the model. For example, our experiment on OLMoE requires 3840 GPU-hours on V100 GPUs. In contrast, the proposed MaxNNScore-based identification does not require any inference on calibration data.

(Response to Weakness 2) On the estimates of energy efficiency and latency values

Measuring the energy consumption and latency values in real analog in-memory hardware requires getting access to sufficient in-memory computing resources and configuring the in-memory computing system according to the need. Configuring such systems for sole research purposes usually falls beyond the scope. Therefore, it is a common practice in the analog in-memory computing of neural networks literature to report the estimated values [6,7,8]. Note that our energy efficiency and latency estimates are based on the latency and energy consumption values measured on a standard AIMC system; see Appendix C for details. Furthermore, the energy efficiency and latency values for full-precision and digital compression baselines are upper- and lower-bound estimates, which reinforce the practical advantage of the proposed mapping.

(Response to Question 2) On the computational cost of MaxNNScore compared with ENS

The computational cost of ENS is incomparably higher than that of the proposed MaxNNScore

ENS requires computing expert output separately for each layer over multiple random seeds for a sufficiently large calibration dataset. Therefore, the computational cost grows quadratically with the number of layers (i.e., O(L2)O(L^2) complexity, where LL is the total number of layers in the model). On the other hand, MaxNNScore requires a very insignificant amount of compute required to calculate the neuron norms and does not require inference on calibration data.

We provide the cost of evaluating ENS and MaxNNScore for DeepSeekMoE in the following table. The costs are evaluated on a 32 GB V100 GPU. ENS of each expert is evaluated over 8 random seeds for a randomly selected 2048 samples from WikiText-2.

MethodCompute cost (GPU-hours)
MaxNNScore0.006
ENS16

References

[1] Bu, Dake, Wei Huang, Taiji Suzuki, Ji Cheng, Qingfu Zhang, Zhiqiang Xu, and Hau-San Wong. "Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samples." In International Conference on Machine Learning, pp. 4642-4695. PMLR, 2024.

[2] Arora, Sanjeev, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. "Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks." In International conference on machine learning, pp. 322-332. PMLR, 2019.

[3] Allen-Zhu, Zeyuan, and Yuanzhi Li. "Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning." In The Eleventh International Conference on Learning Representations. 2023.

[4] Chen, Zixiang, Yihe Deng, Yue Wu, Quanquan Gu, and Yuanzhi Li. "Towards understanding mixture of experts in deep learning." arXiv preprint arXiv:2208.02813 (2022).

[5] Chowdhury, Mohammed Nowaz Rabbani, Shuai Zhang, Meng Wang, Sijia Liu, and Pin-Yu Chen. "Patch-level routing in mixture-of-experts is provably sample-efficient for convolutional neural networks." In International Conference on Machine Learning, pp. 6074-6114. PMLR, 2023.

[6] Büchel, Julian, Athanasios Vasilopoulos, William Andrew Simon, Irem Boybat, HsinYu Tsai, Geoffrey W. Burr, Hernan Castro et al. "Efficient scaling of large language models with mixture of experts and 3D analog in-memory computing." Nature Computational Science 5, no. 1 (2025): 13-26.

[7] Büchel, Julian, Iason Chalas, Giovanni Acampa, An Chen, Omobayode Fagbohungbe, Hsinyu Tsai, Kaoutar El Maghraoui, Manuel Le Gallo, Abbas Rahimi, and Abu Sebastian. "Analog foundation models." Advances in Neural Information Processing Systems 38 (2026): 61179-61228.

[8] Jain, Shubham, Hsinyu Tsai, Ching-Tzu Chen, Ramachandran Muralidhar, Irem Boybat, Martin M. Frank, Stanisław Woźniak et al. "A heterogeneous and programmable compute-in-memory accelerator architecture for analog-AI using dense 2-D mesh." IEEE Transactions on Very Large Scale Integration (VLSI) Systems 31, no. 1 (2022): 114-127.

Thank you for the detailed response and additional experiments. The expert-level sensitivity analysis provides useful evidence for the effectiveness of MaxNNScore, and the computational cost comparison helps clarify its advantage over ENS. I appreciate the authors' efforts to address these concerns.

My main remaining concern is the choice of experimental models. The evaluation is based on relatively old MoE architectures, which makes it difficult to assess how well the proposed method generalizes to more recent and larger-scale MoE models. Given the rapid development of MoE architectures, I believe experiments on newer models would be important to establish the practical relevance of the method.

Reviewer_KDJhOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a method to mitigate the accuracy degradation of large MoE language models on noisy accelerators. The authors map all dense linear layers to digital cores and rank routed experts using the maximum neuron norm score (MaxNNScore) which is the product of maximum column l2l_2 norms across an expert's linear layers.

Critical strengths

Practical metric. Unlike noise-aware retraining or the proposed calibration-based ENS baseline, MaxNNScore is purely easy and fast to calculate.

Critical weaknesses

The paper tackles an important, practical hardware problem for scaling MoE inference on analog accelerators and proposes a remarkably simple, zero-cost metric (MaxNNScore) that works very well empirically. However, it is not entirely clear to me why MaxNNScore beats activation frequency.

Questions for authors

Why do you believe MaxNNScore correlates with token frequency in practice? Why does the empirical activation frequency baseline fail so dramatically (Table 3) if frequent activation is the root cause of high MaxNNScore?

3 comments

We thank the reviewer for their valuable comments and suggestions. We provide our responses below.

(Response to Weakness 1 and Question 1) Clarity on the difference between expert activation frequency and the frequency of appearance of individual tokens in data

We want to clarify the difference between expert activation frequency and the frequency of appearance of individual tokens in data. This will help us to explain the discrepancy in performance between the MaxNNScore-based metric and the activation frequency-based metric.

Expert activation frequency measures how frequently an expert is being activated over all tokens in the data. On the other hand, the expert with a large MaxNNScore learns tokens that individually appear with high frequencies in the data. We further elaborate the difference with a toy example.

Suppose we have two experts E1E_1 and E2E_2 and four tokens x1,x2,x3,x4x_1, x_2, x_3, x_4 with frequencies of their appearance 0.1, 0.2, 0.3, 0.4, respectively. Suppose the tokens x1,x2,x3x_1, x_2, x_3 are being routed to expert E1E_1 and the token x4x_4 is being routed to expert E2E_2. In that case, the expert activation frequencies of E1E_1 and E2E_2 are 0.6 and 0.4, respectively. However, according to our theory (Lemma 4.1), expert E2E_2 will have a larger MaxNNScore than E1E_1, as it’s learned a token that possesses a larger frequency of appearance for a single token.

Why does learning high-frequency tokens indicate a high MaxNNScore, in practice?

Our training dynamic analysis explains why an expert learning high-frequency tokens exhibits a large MaxNNScore. Specifically, we theoretically show that the neurons inside the experts that learn these high-frequency tokens receive large batch-gradient updates during training, thanks to their high frequency of appearance. Therefore, the neuron norms grow quickly to higher levels, exhibiting higher MaxNNScores. In practice, for large MaxNNScore-experts of MoE LLMs, we observe that the neurons with the largest norms are indeed being activated by the tokens (i.e., exhibit high activation outputs for these tokens) that can be found frequently in language. Similarly, for the small MaxNNScore-experts of MoE LLMs, we observe that the neurons with the largest norms are being activated by rare tokens found in language. See Figure 5 of the original manuscript for such evidence.

Why does MaxNNScore succeed while activation frequency fails dramatically?

The analog noise model in Section 2.2 clearly indicates that the experts with large weight magnitude inject a high degree of noise into the system. By definition, MaxNNScore captures these experts for digital mapping, while the activation frequency fails to capture them.

Dear Reviewer KDJh,

Thank you again for your thoughtful review and for taking the time to read our rebuttal. As the deadline for the author–reviewer discussion period approaches (November 18, AoE), we kindly ask whether our responses have adequately addressed your concerns.

If there are any remaining concerns or points that would benefit from further clarification, we would be very happy to address them before the discussion period concludes.

Thank you again for your time and valuable feedback.

Best regards,

Authors

AuthorsOpenReview ↗

Look forward to re-evaluation

Dear Reviewer KDJh,

As the author discussion period is coming to an end, we kindly invite you to review our rebuttal and let us know whether our responses have adequately addressed your concerns. If so, would you consider updating your review? Many thanks.

Best regards, Authors

Reviewer_8LXFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MaxNNScore, which estimates each expert’s sensitivity to analog hardware noise using the product of the maximum neuron weight norms across its projection matrices. Based on this score, noise-sensitive experts are mapped to digital computing units, while less-sensitive experts are mapped to analog computing units, with dense MVMs also executed digitally. The authors further provide theoretical analysis and simulation results to demonstrate the benefits of this strategy in terms of inference accuracy, noise tolerance, and energy efficiency.

Critical strengths
  1. The paper addresses an important practical challenge, as hardware noise is an inherent issue in analog in-memory computing (AIMC). Investigating the varying noise sensitivities of MoE experts and proposing an expert-level mapping strategy is well motivated.

  2. The proposed MaxNNScore relies solely on pretrained model weights and introduces minimal computational overhead, making it relatively easy to implement and deploy.

  3. The paper evaluates three MoE models across seven benchmark tasks and compares the proposed method against several baselines, such as activation frequency, demonstrating promising experimental results.

Critical weaknesses
  1. Although the authors establish a relationship between weight norms and analog noise bounds under certain assumptions, they do not directly demonstrate a monotonic relationship between MaxNNScore and actual expert-level noise sensitivity. Therefore, MaxNNScore appears to remain a weight-based heuristic.

  2. Noise sensitivty of expert is defined as the product of the MaxNNScore across the three linear projection matrices of each expert. However, the paper does not sufficiently justify why this multiplicative formulation is appropriate or preferable to alternative aggregation strategies.

  3. The theoretical analysis relies on simplified noise distribution assumptions, but it remains unclear how closely these assumptions match the noise distributions observed in real AIMC hardware. The authors should provide further evidence or discussion to validate the applicability of their theoretical noise model to practical hardware settings.

Questions for authors

Existing weight-only, calibration-free quantization methods for MoE models have proposed metrics to estimate expert importance or sensitivity to quantization noise. How does MaxNNScore differs from these metrics and provide empirical comparisons to demonstrate its advantages under comparable settings.

6 comments

We thank the reviewer for their valuable comments and suggestions. We provide our responses below.

(Response to Weakness 1) On the monotonic relationship of the proposed metric and the expert noise sensitivity

We establish a monotonic relationship between the upper bound of the output noise sensitivity of the expert projection matrices and their MaxNNorm without taking any assumption on the input and weight distributions

The analog noise model described in Section 2.2 clearly indicates the dependency of analog-noise level on the expert weight magnitude. Here we provide a formal monotonic relationship between the upper bound of noise sensitivity of the expert projection matrix and its MaxNNorm. Note that to establish the relationship, we do not need any assumptions on the input and weight distributions. We also do not need any independence assumptions among different analog noise sources. The formal theorem is given below.

Theorem: Let us denote Wd×m\mathbf{W}^{d\times m} as an expert projection matrix and x\mathbf{x} as its input. Let us define C:=∑u=13cuC:=\sum_{u=1}^3c_u. Then the root-mean-square output error satisfies

(E[∣∣y^−y∣∣22])1/2≤m(C∣∣x∣∣1+cs)MaxNNorm⁡(W)\left(\mathbb{E}\left[\left|\left|\hat{\mathbf{y}}-\mathbf{y}\right|\right|_2^2\right]\right)^{1/2}\le\sqrt{m}\left(C||\mathbf{x}||_1+c_s\right)\operatorname{MaxNNorm}(\mathbf{W})

where y^\hat{\mathbf{y}} and y\mathbf{y} are the noisy and noise-free outputs, respectively.

Proof:

For any weight Wj,iW_{j,i} of the projection matrix, the programming noise magnitude, σj,i≤c0Wmax+∑u=13cuWmax=CWmax\sigma_{j,i}\le c_0W_{max}+\sum_{u=1}^3c_uW_{max}=CW_{max}.

Now, Wmax=∣∣W:,i∣∣∞≤∣∣W:,i∣∣2≤MaxNNorm⁡(W)W_{max}=\left|\left|\mathbf{W}_{:,i}\right|\right|_{\infty}\le \left|\left|\mathbf{W}_{:,i}\right|\right|_2\le \operatorname{MaxNNorm}(\mathbf{W}).

Therefore, σj,i≤CMaxNNorm⁡(W)\sigma_{j,i}\le C\operatorname{MaxNNorm}(\mathbf{W}).

Similarly, the system noise magnitude of neuron ii, σs,i≤CMaxNNorm⁡(W)\sigma_{s,i}\le C\operatorname{MaxNNorm}(\mathbf{W}).

Now, output of the ii-th neuron, y^i=∑j=1d(Wj,i+ΔWj,i)xj+ϵi\hat{y}_i=\sum_{j=1}^d(W_{j,i}+\Delta W_{j,i})x_{j} + \epsilon_i, where ΔWj,i\Delta W_{j,i} and ϵi\epsilon_i are the weight-programming system noise instances associated with neuron ii.

Therefore, error in the ii-th neuron output, ei=y^i−yi=∑j=1dΔWj,ixj+ϵie_i=\hat{y}_i-y_i=\sum_{j=1}^d\Delta W_{j,i}x_j + \epsilon_{i}.

Now, using Minkowski's inequality,

(E[ei2])1/2≤∑j=1d∣xj∣σj,i+σs,i≤MaxNNorm⁡(W)(C∣∣x∣∣1+cs)(\mathbb{E}\left[e_i^2\right])^{1/2}\le \sum_{j=1}^d|x_j|\sigma_{j,i}+\sigma_{s,i}\le \operatorname{MaxNNorm}(\mathbf{W})(C||\mathbf{x}||_1+c_s).

Now, ∣∣y^−y∣∣22=∑i=1mei2||\hat{\mathbf{y}}-\mathbf{y}||_2^2=\sum_{i=1}^m e_i^2.

Therefore, E[∣∣y^−y∣∣22]=≤m(C∣∣x∣∣1+cs)2MaxNNorm⁡(W)\mathbb{E}\left[ ||\hat{\mathbf{y}}-\mathbf{y}||_2^2\right]=\le m \left(C||\mathbf{x}||_1+c_s\right)^2 \operatorname{MaxNNorm}(\mathbf{W}), which completes the proof.

Our theoretical generalization results, presented in Theorem 4.2, go beyond the output noise-sensitivity analysis of the expert projection matrices, and quantify the overall effect of analog noise on model accuracy to justify the rigorousness of the proposed method

The noise sensitivity analysis presented above only characterizes the noise magnitude of different experts. It does not characterize the effect of analog noise on the model’s overall generalization performance.

To characterize the model’s overall generalization performance under analog noise, we first need to characterize how much each expert contributes to the model output for different inputs and how much these contributions are affected by the analog noise. Our data model assumptions and training dynamic analysis facilitate this theoretical characterization and allow us to formally quantify the overall noise tolerance of different experts in order to maintain the original generalization performance as presented in Theorem 4.2.

(Continued Response to Weakness 1) On the monotonic relationship of the proposed metric and the expert noise sensitivity

We extensively verify the relationship between the experts’ MaxNNScore and their noise effect on the model’s overall accuracy We conduct an extensive experiment to validate the high sensitivity of the model accuracy to the analog noise of the top MaxNNScore experts. To measure the sensitivity of model accuracy to the analog noise of each expert, we only map the expert of interest in analog, while the rest of the experts are mapped in digital. We measure the average accuracy under this mapping for each expert separately on the benchmark LLM tasks. The results are presented in Figure 6-13 of this anonymous link:

https://storage.googleapis.com/anonymous_link/acc_level_exprt_sensitivity.pdf

As observed in these figures, the experts yielding noticeable accuracy degradation are also the top MaxNNScore experts. In contrast, the noise-aware ENS baseline, which directly measures the experts’ output degradation under analog noise, fails to include some of these critical experts (e.g., expert 27 in layer 4) in its top-k ranking, as the metric ignores the experts’ contributions to model accuracy.

Note that using the accuracy-level sensitivity itself for expert ranking is prohibitively expensive, as the process requires separate repetitive evaluation of accuracy over multiple random seeds for each expert of the model. For example, our experiment on OLMoE requires 3840 GPU-hours of V100s. In contrast, the proposed MaxNNScore-based identification does not require any inference on calibration data.

(Response to Weakness 2) On the choice of taking the product of maximum neuron norms to evaluate MaxNNScore

We multiply the MaxNNorms of up, gate, and down projection matrices, as their effects are not independent

The up, gate, and down projection matrices’ outputs are interdependent. The outputs of the gate and the up projection matrices are multiplied together, which is the input of the down projection matrix. Therefore, severe degradation of one of the outputs due to analog noise would significantly degrade the output of the other. Therefore, the product of the three norms is a natural choice. We further evaluate the product-based aggregation against two other aggregation strategies:

I. Sum-based aggregation, i.e., ∑p∈up, down, gateMaxNNorm⁡(Wp)\sum_{p\in\\{\text{up, down, gate}\\}}\operatorname{MaxNNorm}(\mathbf{W}_{p})

II. Weighted-product-based aggregation, i.e., [MaxNNorm⁡(Wup)]α×[MaxNNorm⁡(Wdown)]β×[MaxNNorm⁡(Wgate)]γ\left[\operatorname{MaxNNorm}(\mathbf{W}_{\text{up}})\right]^\alpha\times\left[\operatorname{MaxNNorm}(\mathbf{W}_{\text{down}})\right]^\beta\times\left[\operatorname{MaxNNorm}(\mathbf{W}_{\text{gate}})\right]^\gamma, where α+β+γ=1\alpha+\beta+\gamma=1

The average accuracy on the benchmark LLM tasks for OLMoE with 25% digital experts is given below for the three aggregation methods.

MethodAvg. Accuracy (%)
Product-based (Ours)52.51 ± 0.16
Sum-based52.38 ± 0.15
Weighted-product (α=0.2,β=0.2\alpha=0.2,\beta=0.2)24.06 ± 0.19
Weighted-product (α=0.2,β=0.4\alpha=0.2,\beta=0.4)52.67 ± 0.16
Weighted-product (α=0.2,β=0.6\alpha=0.2,\beta=0.6)52.66 ± 0.16
Weighted-product (α=0.4,β=0.2\alpha=0.4,\beta=0.2)51.63 ± 0.11
Weighted-product (α=0.4,β=0.4\alpha=0.4,\beta=0.4)52.12 ± 0.19
Weighted-product (α=0.6,β=0.2\alpha=0.6,\beta=0.2)51.14 ± 0.22

As we can see, all three methods perform similarly (with proper selection of hyperparameters in the weighted product-based case). This is because of the high similarity (over 0.97 rank correlation among them) of their selected digital expert sets.

(Response to Weakness 3) On the practicality of analog noise model

Our original manuscript already includes a detailed discussion on the practicality of the analog noise model. The discussion is provided in Appendix H, and we point to this discussion in Section 6 (Conclusion) of the main paper.

Our analog noise model is empirically derived from practical AIMC devices

The third-degree polynomial model for analog programming noise is widely adopted in the analog in-memory computing (AIMC) literature [1,2,3]. The coefficients of this model are obtained by fitting experimental measurements from practical AIMC devices. Likewise, the widely used AIHWKIT simulator [4] models programming noise for multiple AIMC technologies using third-degree polynomial fits derived from device characterization data. In our experiments (Section 5), we instantiate the noise model by setting the coefficients (c0,c1,c2,c3c_0, c_1, c_2, c_3) according to the polynomial fitted to measurements from a state-of-the-art PCM-based AIMC chip [5].

The effect of device-specific properties, including cell resolution, is captured through the noise coefficients.

Device-specific characteristics, such as the NVM cell resolution, influence the fitted coefficients of the polynomial noise model [4]. Therefore, rather than assuming a particular cell resolution, in Figure 4 of our original manuscript, we systematically sweep the noise model coefficients to emulate a broad range of analog noise levels arising from different device technologies and internal device properties. As shown in the figure, the proposed analog-digital mapping consistently outperforms all competing baselines across this range of noise conditions, demonstrating that our conclusions are not tied to a specific analog hardware implementation.

(Response to Question 1) Comparing with expert-selection baseline used for measuring experts’ quantization noise sensitivity

In our original manuscript, we already compare with a weight-only, calibration-free expert-selection baseline, namely lower router norm, used for measuring experts’ quantization noise sensitivity [6]. The detailed results are provided in Table 3 of our original manuscript. The technique significantly underperforms in identifying analog noise-sensitive experts. This is because of the fact that the fundamental nature of quantization noise and analog noise is different. The quantization noise depends on the weight range and distribution, while the analog noise solely depends on the weight magnitude.

Reviewer_MpZWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduced an in memory compute paradigm with analog circuits where it performs matrix multiplications of transformer modules in mixture of experts that has lower noise. It has identified that noise could get amplified with the inherent noise of analog operations if the matrix has higher l2 norm. So, this framework selects which matrices to be multiplied by using l2 norm as the metric of decision.

Critical strengths
  1. Theoretical justification: The paper has some solid mathematical reasoning behind their metric.
  2. Strong Result on Benchmarks: The paper has some solid result on widely recognized benchmarks.
  3. Illustration: Diagram and written descriptions were clear and easy to follow.
Critical weaknesses
  1. Weak Benchmark: Benchmarks used are MCQ type which means the network needs to be able to select an option from a lot of choices. This makes the evaluation very weak and is suitable for just LLM as a judge problem.
  2. Technique is not strong: The technique identifies via a weak heuristic of looking at L2 norm. Question is why not learn scaling factors to counteract the high norm and use it for all matrices.
  3. Strong baselines absent: The paper did not rigorously compare against other baseline. The paper relied on simple MCQ tasks to prove the strength of the technique. Although the math is solid, it can not override the importance of an automatic approach of learning scaling factor over choosing a subset of weights for it which makes the paper’s central contribution questionable.
Questions for authors

Please see above.

6 comments

We thank the reviewer for their valuable comments and suggestions. We provide our responses below.

(Response to Weaknesses 1 and 3) Evaluation on stronger benchmarks

Our analog-digital mapping maintains performance on stronger reasoning benchmarks that require longer generation

Before presenting our new results on stronger reasoning tasks, we want to clarify the difficulty of evaluating large foundation models on existing analog in-memory computing simulators. The analog in-memory computing simulators, such as the AIHWKIT-lightning [1] used in this paper, are extremely slow, as the simulation process needs to sample many noise instances on-the-fly, e.g., sampling system noise for each neuron of each expert in each decoding step. Therefore, the simulation time increases to an unbearable level for large models such as MoE LLMs and for longer generation tasks. For example, completing only one run of Qwen-1.5 MoE (14B) on the GSM8K test dataset (1319 samples) takes 65 hours on 4 V100 GPUs (32 GB each). Moreover, for a proper evaluation, one needs to evaluate over multiple random seeds (8 in our case). The previous largest evaluation of analog foundation models (Phi-3-mini-4k-instruct, 3.8B) is found in [2], which excluded system noise from their evaluation and mostly restricted their study to MCQ benchmarks (except GSM8K). To the best of our knowledge, our paper provides the largest end-to-end evaluation of large foundation LLMs (e.g., OLMoE (7B), Qwen-1.5 MoE (14B), and DeepSeekMoE (16B)) against analog in-memory noise, including ADC, DAC, weight-programming, and system noise.

We now present results evaluating the proposed analog-digital mapping on two stronger reasoning benchmarks, GSM8K (a mathematical reasoning benchmark) and HumanEval (a code generation benchmark), which require generating significantly longer responses. Due to the difficulty of evaluating on such benchmarks, as described above, we restrict the evaluation to the first 100 samples of GSM8K (7.5% of total samples) and the first 50 samples of HumanEval (30% of total samples). For these tasks, we evaluate the supervised finetuned version of DeepSeekMoE (DeepSeekMoE-Chat), Qwen-1.5 MoE, and the OLMoE-Instruct, which is post-trained via reinforcement learning with verifiable reward (RLVR). Since it has been established in our original manuscript that other expert selection baselines significantly underperform compared to our method, except ENS, we only compare with ENS for these two new benchmarks. The results are presented below.

(Continued Response to Weaknesses 1 and 3) Evaluation on stronger benchmarks

ModelMethodDigital expert (%)GSM8K Accuracy (%)HumanEval Pass@1 (%)
DeepSeekMoE-Chat (16B)FP-161003066
DeepSeekMoE-Chat (16B)All-analog05.63 ± 0.3313.75 ± 1.87
DeepSeekMoE-Chat (16B)MaxNNScore (Ours)3.12523.75 ± 1.2856.25 ± 1.03
DeepSeekMoE-Chat (16B)MaxNNScore (Ours)6.2525.25 ± 1.1853 ± 1.00
DeepSeekMoE-Chat (16B)MaxNNScore (Ours)12.5026 ± 1.1555.25 ± 1.96
DeepSeekMoE-Chat (16B)ENS3.12522 ± 1.0052.5 ± 2.26
DeepSeekMoE-Chat (16B)ENS6.2522 ± 1.0054.5 ± 0.91
DeepSeekMoE-Chat (16B)ENS12.5024.13 ± 2.1653 ± 1.36
Qwen-1.5 MoE (14B)FP-161003364
Qwen-1.5 MoE (14B)All-analog02.5 ± 0.380.25 ± 0.25
Qwen-1.5 MoE (14B)MaxNNScore (Ours)3.3337.125 ± 1.651.25 ± 1.0
Qwen-1.5 MoE (14B)MaxNNScore (Ours)6.6736.75 ± 1.3252.5 ± 1.5
Qwen-1.5 MoE (14B)MaxNNScore (Ours)13.3338 ± 0.9755.5 ± 1.35
Qwen-1.5 MoE (14B)ENS3.3337.25 ± 1.8153.75 ± 1.83
Qwen-1.5 MoE (14B)ENS6.6740.75 ± 1.3654.5 ± 1.05
Qwen-1.5 MoE (14B)ENS13.3338.125 ± 1.4754.5 ± 1.59
OLMoE-Instruct (7B)FP-161006656
OLMoE-Instruct (7B)All-analog05 ± 0.810.25 ± 0.25
OLMoE-Instruct (7B)MaxNNScore (Ours)6.2544.88 ± 1.3341.5 ± 1.76
OLMoE-Instruct (7B)MaxNNScore (Ours)12.5048.38 ± 1.1241 ± 1.73
OLMoE-Instruct (7B)MaxNNScore (Ours)2550.63 ± 0.7448.25 ± 1.03
OLMoE-Instruct (7B)MaxNNScore (Ours)5059.88 ± 1.5348.75± 1.51
OLMoE-Instruct (7B)ENS6.2538.88 ± 1.2210.75 ± 1.51
OLMoE-Instruct (7B)ENS12.546.88 ± 0.934.25 ± 1.87
OLMoE-Instruct (7B)ENS2550.25 ± 1.2832.5 ± 1.99
OLMoE-Instruct (7B)ENS5056.25 ± 1.3239.25 ± 1.51

The above results confirm that the complexity of the reasoning tasks and the longer generation lengths do not invalidate the accuracy gain of the proposed MaxNNScore-based analog-digital mapping that is observed in the simpler benchmarks. Specifically, mapping all experts in analog significantly degrades the models’ reasoning ability, while our proposed mapping needs only a very few of the experts (at most 13%) to be mapped in digital to recover most of the performance degradation (at least 70%). While ENS performs similarly to our method in DeepSeekMoE and Qwen-1.5 MoE (within the statistical margin of error of each other), our method significantly outperforms ENS in OLMoE/OLMoE-Instruct, which is the most vulnerable to analog noise among the three models presented in this paper.

(Response to Weaknesses 2 and 3) Comparison against weight-scaling baseline

Since analog noise scales with the weight magnitude, one obvious intuition is to downscale the weights to minimize the analog noise. However, downscaling weights requires upscaling the inputs so that the ideal outputs of the analog MVMs are unchanged. Upscaling inputs increases the input ranges and hence it may increase the quantization noise in DAC. In other words, weight-scaling is essentially an approach to balance a trade-off between the DAC noise and the analog device noise.

Learning weight-scaling factors is prohibitively expensive for large MoE foundation LLMs

One approach to balancing the trade-off is to learn the scaling factors for each matrix or even for each neuron. Indeed, hardware-aware retraining learns those factors inherently during the retraining process [2]. However, learning such parameters through retraining is computationally prohibitive, particularly for large MoE models. The process requires acquiring a training dataset that represents the original pretraining data so that the original representation of the pretrained model is preserved. Therefore, state-of-the-art analog hardware-aware retraining for foundation models relies on knowledge distillation from a stronger teacher model, requiring repeated generation of synthetic training data prior to retraining [2]. In addition, hardware-aware retraining introduces extra training-time hyperparameters that must be tuned beyond those of conventional training, further increasing the computational cost. To the best of our knowledge, the largest reported demonstration of analog hardware-aware retraining is for the 3.8B-parameter Phi-3-mini-4k-instruct model [2], which is substantially smaller than the smallest MoE model evaluated in our work (OLMoE-7B). The paper reported 230 hours of retraining only to achieve accuracy below 3.5% of the full-precision digital inference on benchmarks that mostly consist of MCQ-type questions, despite excluding system noise from the evaluation. Taking these challenges into account, we believe our method contributes by providing a retraining-free alternative for deploying large foundation MoE LLMs in analog in-memory hardware. Moreover, our analog-digital mapping can still be applied to further accuracy improvements of hardware-aware retrained models, as it does not alter the underlying mechanism motivating our expert ranking strategy (i.e., Lemma 4.1). However, conducting computationally expensive retraining for the MoE models for a baseline comparison and to measure the further accuracy improvement of the retrained models is beyond the scope of the current work, and we leave this investigation to future work.

Retraining-free weight-scaling significantly underperforms in terms of accuracy gain compared to our method

Another retraining-free approach is to treat these scaling factors as hyperparameters and calibrating these hyperparameters on a calibration dataset. Indeed, the strategy has been implemented in previous analog in-memory computing literature. Specifically, [3] proposed a scaling-factor calibration strategy, where the scaling factor for each neuron W:,i\mathbf{W}_{:,i} is evaluated as, si=max(∣xi∣)λmax(W:,i)1−λ\mathcal{s}_{i}=\frac{\text{max}(|\mathbf{x}_i|)^{\lambda}}{\text{max}(\mathbf{W}_{:,i})^{1-\lambda}}, so that the new scaled neuron weights are siW:,i\mathcal{s}_{i}\mathbf{W}_{:,i} and the new scaled input is 1sixi\frac{1}{\mathcal{s}_{i}}\mathbf{x}_i. Here, λ\lambda controls the trade-off between the DAC and the analog device noise. The paper proposed to estimate max(∣xi∣)\text{max}(|\mathbf{x}_i|) over a calibration dataset.

We reimplemented the scaling strategy on OLMoE for various values of λ\lambda using C4 as the calibration dataset and evaluated the rescaled model against analog noise on the 7 benchmarks presented in our original manuscript. The results are presented in the following table.

MethodAvg. accuracy (%)
FP-1661.03
All-analog (0% experts in digital)36.03
MaxNNScore (6.25% experts in digital)47.10 ± 0.17
MaxNNScore (25% experts in digital)52.51 ± 0.16
Weight-scaling (λ=0.05\lambda=0.05)38.99 ± 0.18
Weight-scaling (λ=0.1\lambda=0.1)38.93 ± 0.15
Weight-scaling (λ=0.2\lambda=0.2)38.37 ± 0.15
Weight-scaling (λ=0.4\lambda=0.4)36.96 ± 0.24
Weight-scaling (λ=0.6\lambda=0.6)35.17 ± 0.11
Weight-scaling (λ=0.8\lambda=0.8)32.61 ± 0.01

As the results suggest, weight-scaling falls significantly short in terms of accuracy gain compared to our method. This implies that there exist highly noise-sensitive experts in the model, whose noise levels can not be suppressed by weight rescaling. Our method provides a theoretically provable strategy to identify these experts for digital mapping to achieve a significant accuracy gain.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A summary of rebuttal revision

We thank the AC and all the reviewers for their valuable comments and suggestions. We really appreciate the reviewers' acknowledgements of the motivation, theoretical, and empirical justification of our work. All the reviewers appreciated the practicality and the ease of implementation ability of our method for identifying analog noise-sensitive experts.

We now list the key questions asked by the reviewers:

  1. Evaluation on stronger benchmarks (Reviewer MpZW\color{red}{\text{MpZW}})

  2. Comparison against weight-scaling baseline for minimizing analog device noise (Reviewer MpZW\color{red}{\text{MpZW}})

  3. Theoretical and empirical justification of the direct relationship between MaxNNScore (our proposed metric) and the accuracy-level expert-noise sensitivity (Reviewer 8LXF\color{red}{\text{8LXF}}, sjxK\color{red}{\text{sjxK}})

  4. Justification of taking the product of MaxNNorm⁡\operatorname{MaxNNorm}s of expert projection matrices to evaluate MaxNNScore⁡\operatorname{MaxNNScore} (Reviewer 8LXF\color{red}{\text{8LXF}})

  5. Practicality of the analog noise model considered in this paper. (Reviewer 8LXF\color{red}{\text{8LXF}})

  6. Computational cost comparison between MaxNNScore and the empirical metric, namely Expert Noise Sensitivity (ENS) (Reviewer sjxK\color{red}{\text{sjxK}})

We addressed these questions with additional theoretical and empirical results and included these results in the revised 10-page manuscript (all the revisions are highlighted in blue\color{blue}{\text{all the revisions are highlighted in blue}}). We provide a summary of these results below.

  1. New results added on stronger benchmarks. We added additional results on stronger benchmarks, namely GSM8K (math reasoning) and HumanEval (code generation), requiring longer generation. The additional results follow the same trend found in the results presented in the original manuscript. This justifies the robustness of our proposed method against complex reasoning that requires longer generation. The additional results can be found in Table 2 of our revised manuscript.

  2. Weight rescaling is not an alternative; empirical evidence is provided. We further compare our method against a retraining-free weight-scaling baseline, found in previous analog in-memory computing literature. Our method significantly outperforms the weight-scaling baseline, justifying the inability of weight rescaling to be an alternative to our method. The results can be found in Table 3 of our revised manuscript.

  3. Direct relationship is validated. We provide an additional theorem (Theorem L.1) to establish a direct relationship between MaxNNorm and the upper bound of noise sensitivity of expert projection matrices. We noted that the new theorem does not require any assumption on training data or on the expert's weight distribution. However, we further argue that the new theorem cannot be a replacement for our key theoretical results in Theorem 4.1, as it characterizes the analog noise effect of different expert groups on the model's overall accuracy. We further provide extensive empirical evidence (Appendix I) to validate the strong correlation between the accuracy-level sensitivity of the experts and their MaxNNScore.

  4. Justification for product-based design. An empirical justification is provided in Appendix F for the selection of the product of MaxNNorm for the MaxNNScore evaluation.

  5. Our analog noise model is fitted on practical device data. Our original manuscript already includes a discussion on the practicality of our analog noise model (Appendix J). Specifically, our noise model is constructed using the data collected from state-of-the-art analog in-memory devices.

  6. Computational cost is insignificant. We finally added a computational cost comparison between our method and the empirical ENS metric (Table 4). Specifically, we show that the computational cost is insignificant in our method (0.006 vs. 16 GPU hours) compared to ENS.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.