- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work derives an expert-level quantization error upper bound and adopts the L2 norm of expert outputs as a theoretically grounded importance metric. It constructs a two-stage offline ILP-based bit allocation pipeline, which optimizes activation bit assignment prior to weight bit assignment, and further incorporates online adaptive activation bit adjustment during inference. To tackle the outlier issue originating from the product activations within MoE FFN modules, the method integrates SmoothQuant and Randomized Hadamard Transform (RHT) as preprocessing operations. Built upon these components, this work realizes mixed-precision post-training quantization for MoE large language models. Extensive experiments demonstrate that the proposed approach outperforms existing state-of-the-art methods substantially under the aggressive W4A4 quantization setting.
Critical strengths
-
It provides a theoretically guaranteed expert-level quantization error upper bound and uses the L2 norm of expert outputs as a stable importance metric to guide ILP-based mixed-precision bit allocation.
-
The pipeline combines outlier suppression via SmoothQuant and RHT with lightweight online adaptive activation bit adjustment, achieving strong performance even under the aggressive W4A4 quantization setting.
Critical weaknesses
-
The importance metric only models independent quantization error for individual experts and neglects coupling or interactions among experts. The derived error upper bound is obtained under the independence assumption. The online adaptive scheme solely relies on input norms and does not model the dynamic variation of expert importance across inference processes (e.g., Chain-of-Thought reasoning and different generation steps).
-
Although the L2-norm based importance metric requires forward passes over a calibration dataset and thus remains data-dependent, it exhibits higher rank correlation across distribution shifts compared with routing-frequency-based heuristics. However, the metric cannot fully eliminate the bias introduced by the distribution of calibration samples.
Questions for authors
-
The error upper bound is derived under the independence assumption across experts. Have the authors quantitatively measured the magnitude of cross-expert coupling error term and analyzed its impact on the tightness of the bound in experiments?
-
The L2-based importance metric relies on calibration data. Could the authors provide end-to-end downstream results to show how much performance degrades when there exists a large distribution gap between the calibration set and the test set?
-
The online adaptive activation bit allocation uses input norm as a proxy for output norm. Is the linear relationship between input norm and expert output norm consistently maintained across all benchmarks and model scales, especially under W4A4 low-bit quantization?
4 comments
Response (2/3)
Question 1: The error upper bound is derived under the independence assumption across experts. Have the authors quantitatively measured the magnitude of cross-expert coupling error term and analyzed its impact on the tightness of the bound in experiments?
Response 2: Additional experiments show that coupling is small relative to individual expert errors in the tested models. The quantitative analysis of the experiment results are as follows. We measured cross-expert error coupling in Qwen1.5-MoE and DeepSeek-V2-Lite using RTN with the W4A4 configuration. Let denote each expert’s contribution to the quantization error and the cross-expert error coupling within each MoE layer:
We evaluated all 24 layers of Qwen1.5-MoE and all 26 MoE layers of DeepSeek-V2-Lite on MathQA and WikiText-2. We report the signed and absolute coupling contributions relative to the sum of individual expert error terms as and , respectively. “Overall” is calculated by summing the numerators and denominators separately across both datasets, then taking their ratio.
| Model | Dataset | Signed ratio | Absolute ratio |
|---|---|---|---|
| Qwen1.5-MoE | MathQA | +2.19% | 3.63% |
| WikiText-2 | +0.58% | 2.04% | |
| Overall | +1.15% | 2.61% | |
| DeepSeek-V2-Lite | MathQA | +0.074% | 0.172% |
| WikiText-2 | +0.079% | 0.347% | |
| Overall | +0.077% | 0.259% |
Overall, cross-expert coupling accounts for 1.15% and 0.077% of the sum of individual expert errors in Qwen1.5-MoE and DeepSeek-V2-Lite, respectively. The corresponding absolute ratios are only 2.61% and 0.259%, respectively. These results show that individual expert errors dominate the overall quantization error in both models, while cross-expert coupling has a minor effect. This supports using the expert-level error bound in Eq. (4) to approximately estimate the overall MoE quantization error.
Weaknesses 2: Although the L2-norm based importance metric requires forward passes over a calibration dataset and thus remains data-dependent, it exhibits higher rank correlation across distribution shifts compared with routing-frequency-based heuristics. However, the metric cannot fully eliminate the bias introduced by the distribution of calibration samples.
Question 2: The L2-based importance metric relies on calibration data. Could the authors provide end-to-end downstream results to show how much performance degrades when there exists a large distribution gap between the calibration set and the test set?
Response 3: The L2-norm-based importance metric remains sensitive to the calibration data distribution, but produces more consistent expert rankings across domains than routing frequency. MxQBound also better preserves downstream performance under cross-domain calibration. Specifically:
- Ranking stability across datasets. Figs. 9–11 report average Spearman rank correlations of 0.60–0.75 across the three models, compared with 0.15–0.32 for routing frequency.
- Impact of domain differences. Rankings obtained using C4 and HumanEval as separate calibration sets have lower correlations of 0.26–0.50, indicating that importance estimation remains sensitive to domain differences.
- Downstream performance. Table 1 reports results with calibration on C4 and evaluation on HumanEval. Under W4A4, MxQBound’s HumanEval Pass@1 drops by 5.48, 3.05, and 7.31 percentage points from FP16 on DeepSeek-V2-Lite, Qwen1.5-MoE-A2.7B, and Mixtral-8×7B, respectively. Under the same calibration setting, CodeQuant incurs larger drops of 6.12, 18.91, and 22.56 percentage points. Thus, MxQBound better preserves downstream performance on all three models despite the substantial domain gap between calibration and evaluation.
Response (3/3)
Question 3: The online adaptive activation bit allocation uses input norm as a proxy for output norm. Is the linear relationship between input norm and expert output norm consistently maintained across all benchmarks and model scales, especially under W4A4 low-bit quantization?
Response 3: We use changes in input norms to estimate changes in output norms and adjust activation bit-widths online. Appendix E supports this proxy relationship. The global bit budget constrains bit allocation but does not change the input norms used for the current adjustment. Specifically:
- Input–output norm relationship. After training, each expert’s weights are fixed, which also fixes its input–output mapping. Therefore, we hypothesize that each expert acts as a relatively stable amplifier of the input signal, with approximately proportional input and output norms. Appendix E tests this hypothesis across three models and nine datasets. The median ratio of the input-norm scaling factor to the output-norm scaling factor ranges from 0.9748 to 1.0232, close to 1. Both scaling factors are measured relative to their respective calibration baselines. These results support our hypothesis and allow us to estimate changes in expert importance from input norms before computing the expert outputs.
- Role of the bit budget. During online adaptation, we first increase or decrease the activation bit-widths of selected experts based on changes in their input norms relative to calibration baselines. We then adjust the bit-widths of other experts to offset the resulting change in bit cost and satisfy the global budget. The global budget corresponding to W4A4 constrains the final bit allocation, while input norms are computed before this adjustment and remain unaffected by it.
Dear Reviewer uoRG,
Thank you again for your valuable feedback and for taking the time to review our submission. As the discussion deadline approaches, please let us know if you have further concerns for clarification.
We would greatly appreciate your continued engagement in the discussion.
Best regards,
Authors.
Response (1/3)
Thank you for the valuable feedback.
Response 1: We apply the Cauchy–Schwarz inequality to cross-expert error coupling and derive an overall MoE error bound from the expert-level bounds without assuming independent errors. We also use current inputs to estimate changes in expert importance and adjust activation bit-widths online as inputs vary during generation. We explain these points below.
Theoretical Analysis of Cross-Expert Error Coupling. With routing weights se held fixed, let δye=y^e−ye denote the quantization error of expert e. The MoE output error is δyMoE=∑eseδye. The MoE quantization error can be expanded as follows: ΔMoE≤∑e∣se∣2∥δye∥22+2∑e<f∣se∣∣sf∣∥δye∥2∥δyf∥2=(∑e∣se∣∥δye∥2)2. Using the Cauchy–Schwarz inequality, we obtain: ΔMoE=∥∑eseδye∥22=individual expert errorse∑se2∥δye∥22+cross-expert error coupling2e<f∑sesf⟨δye,δyf⟩. This bound covers the worst case of aligned expert errors without assuming independence. The expert-level error bound derived in Section 3.2 is ∥δye∥2≤(ϵw,e+ϵa,e)Ie, where Ie denotes the L2 norm of the expert output, and ϵw,e and ϵa,e denote the weight and activation quantization error factors, respectively. Substituting this bound into the preceding inequality yields ΔMoE≤[e∑∣se∣(ϵw,e+ϵa,e)Ie]2.
Thus, the expert-level error bound in Section 3.2 also yields an upper bound on the overall MoE output error without assuming independence across expert errors.
Tracking Dynamic Expert Importance and Adapting Bit-Widths
We adjust each expert’s activation bit-width online to reflect changes in expert importance during inference. Specifically:
The rationale and validation of using input norms as a proxy for output norms can be found in our response to your Question 3.