acceptodds
Under review as a conference paper at ICLR 2027

MxQBound: Efficient Mixed-Precision Quantization for MoE-LLMs Guided by Expert Importance and Error Bound Guarantees

Abstract

While mixed-precision quantization reduces the memory overhead of Mixture-of-Experts (MoE)-Large Language Models (LLMs), existing methods often mis-estimate expert importance, causing suboptimal precision allocation and severe performance degradation under aggressive quantization. We propose MxQBound, an efficient and robust Mixed-precision Quantization framework for MoE-LLMs, guided by holistic assessment of expert importance and equipped with quantization error Bound guarantees. With only a single forward pass, MxQBound directly assesses each expert’s importance from its output norms, supported by a provable upper bound on quantization error. Guided by the importance assessment, MxQBound jointly optimizes the per-expert bit-widths of weights and activations to minimize quantization errors under an overall bit-allocation budget. It further adapts each expert’s activation bit-width according to its input strength during inference with negligible overhead. Extensive experiments on diverse MoE-LLMs and benchmarks show that, under comparable compression ratios, MxQBound outperforms state-of-the-art (SOTA) methods by 4.08-22.85% in model performance and 2.2-141× in computational efficiency. Under aggressive W4A4, MxQBound incurs an average accuracy drop of only 15.07% relative to the full-precision baseline, whereas SOTA methods collapse to near-zero accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, Large Language Model Quantization, Post-Training Quantization, Mixed-Precision Quantization, Weight-and-Activation Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uoRGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work derives an expert-level quantization error upper bound and adopts the L2 norm of expert outputs as a theoretically grounded importance metric. It constructs a two-stage offline ILP-based bit allocation pipeline, which optimizes activation bit assignment prior to weight bit assignment, and further incorporates online adaptive activation bit adjustment during inference. To tackle the outlier issue originating from the product activations within MoE FFN modules, the method integrates SmoothQuant and Randomized Hadamard Transform (RHT) as preprocessing operations. Built upon these components, this work realizes mixed-precision post-training quantization for MoE large language models. Extensive experiments demonstrate that the proposed approach outperforms existing state-of-the-art methods substantially under the aggressive W4A4 quantization setting.

Critical strengths
  1. It provides a theoretically guaranteed expert-level quantization error upper bound and uses the L2 norm of expert outputs as a stable importance metric to guide ILP-based mixed-precision bit allocation.

  2. The pipeline combines outlier suppression via SmoothQuant and RHT with lightweight online adaptive activation bit adjustment, achieving strong performance even under the aggressive W4A4 quantization setting.

Critical weaknesses
  1. The importance metric only models independent quantization error for individual experts and neglects coupling or interactions among experts. The derived error upper bound is obtained under the independence assumption. The online adaptive scheme solely relies on input norms and does not model the dynamic variation of expert importance across inference processes (e.g., Chain-of-Thought reasoning and different generation steps).

  2. Although the L2-norm based importance metric requires forward passes over a calibration dataset and thus remains data-dependent, it exhibits higher rank correlation across distribution shifts compared with routing-frequency-based heuristics. However, the metric cannot fully eliminate the bias introduced by the distribution of calibration samples.

Questions for authors
  1. The error upper bound is derived under the independence assumption across experts. Have the authors quantitatively measured the magnitude of cross-expert coupling error term and analyzed its impact on the tightness of the bound in experiments?

  2. The L2-based importance metric relies on calibration data. Could the authors provide end-to-end downstream results to show how much performance degrades when there exists a large distribution gap between the calibration set and the test set?

  3. The online adaptive activation bit allocation uses input norm as a proxy for output norm. Is the linear relationship between input norm and expert output norm consistently maintained across all benchmarks and model scales, especially under W4A4 low-bit quantization?

4 comments
AuthorsOpenReview ↗

Response (1/3)

Thank you for the valuable feedback.

Weaknesses 1: The importance metric only models independent quantization error for individual experts and neglects coupling or interactions among experts. The derived error upper bound is obtained under the independence assumption. The online adaptive scheme solely relies on input norms and does not model the dynamic variation of expert importance across inference processes (e.g., Chain-of-Thought reasoning and different generation steps).

Response 1: We apply the Cauchy–Schwarz inequality to cross-expert error coupling and derive an overall MoE error bound from the expert-level bounds without assuming independent errors. We also use current inputs to estimate changes in expert importance and adjust activation bit-widths online as inputs vary during generation. We explain these points below.

Theoretical Analysis of Cross-Expert Error Coupling. With routing weights ses_e held fixed, let δye=y^e−ye\delta y_e=\hat y_e-y_e denote the quantization error of expert ee. The MoE output error is δyMoE=∑eseδye.\delta y_{\mathrm{MoE}}=\sum_e s_e\delta y_e. The MoE quantization error can be expanded as follows: ΔMoE≤∑e∣se∣2∥δye∥22+2∑e<f∣se∣∣sf∣∥δye∥2∥δyf∥2=(∑e∣se∣∥δye∥2)2.\Delta_{\mathrm{MoE}}\leq\sum_e |s_e|^2\Vert\delta y_e\Vert_2^2+2\sum_{e\lt f}|s_e||s_f|\Vert\delta y_e\Vert_2\Vert\delta y_f\Vert_2=\left(\sum_e |s_e|\Vert\delta y_e\Vert_2\right)^2. Using the Cauchy–Schwarz inequality, we obtain: ΔMoE=∥∑eseδye∥22=∑ese2∥δye∥22⏟individual expert errors+2∑e<fsesf⟨δye,δyf⟩⏟cross-expert error coupling.\Delta_{\mathrm{MoE}}=\left\Vert\sum_e s_e\delta y_e\right\Vert_2^2=\underbrace{\sum_e s_e^2\Vert\delta y_e\Vert_2^2}_{\text{individual expert errors}}+\underbrace{2\sum_{e\lt f}s_es_f\langle\delta y_e,\delta y_f\rangle}_{\text{cross-expert error coupling}}. This bound covers the worst case of aligned expert errors without assuming independence. The expert-level error bound derived in Section 3.2 is ∥δye∥2≤(ϵw,e+ϵa,e)Ie,\Vert\delta y_e\Vert_2\leq(\epsilon_{w,e}+\epsilon_{a,e})I_e, where IeI_e denotes the L2 norm of the expert output, and ϵw,e\epsilon_{w,e} and ϵa,e\epsilon_{a,e} denote the weight and activation quantization error factors, respectively. Substituting this bound into the preceding inequality yields ΔMoE≤[∑e∣se∣(ϵw,e+ϵa,e)Ie]2.\boxed{\Delta_{\mathrm{MoE}}\leq\left[\sum_e |s_e|(\epsilon_{w,e}+\epsilon_{a,e})I_e\right]^2}.

Thus, the expert-level error bound in Section 3.2 also yields an upper bound on the overall MoE output error without assuming independence across expert errors.

Tracking Dynamic Expert Importance and Adapting Bit-Widths

We adjust each expert’s activation bit-width online to reflect changes in expert importance during inference. Specifically:

  • Offline bit allocation uses output norms to determine each expert’s baseline importance and bit-width.
  • Online bit adaptation uses the current input norm relative to its calibration baseline to estimate changes in importance and adjust the activation bit-width (Eq. 13). The same expert can therefore use different activation bit-widths during inference, without explicitly distinguishing CoT stages or generation steps.

The rationale and validation of using input norms as a proxy for output norms can be found in our response to your Question 3.

AuthorsOpenReview ↗

Response (2/3)

Question 1: The error upper bound is derived under the independence assumption across experts. Have the authors quantitatively measured the magnitude of cross-expert coupling error term and analyzed its impact on the tightness of the bound in experiments?

Response 2: Additional experiments show that coupling is small relative to individual expert errors in the tested models. The quantitative analysis of the experiment results are as follows. We measured cross-expert error coupling in Qwen1.5-MoE and DeepSeek-V2-Lite using RTN with the W4A4 configuration. Let DD denote each expert’s contribution to the quantization error and CC the cross-expert error coupling within each MoE layer:

D=se2∥δye∥22,C=2∑e<fsesf⟨δye,δyf⟩.D=s_e^2\Vert\delta y_e\Vert_2^2,\qquad C=2\sum_{e\lt f}s_es_f\langle\delta y_e,\delta y_f\rangle.

We evaluated all 24 layers of Qwen1.5-MoE and all 26 MoE layers of DeepSeek-V2-Lite on MathQA and WikiText-2. We report the signed and absolute coupling contributions relative to the sum of individual expert error terms as ∑C/∑D\sum C/\sum D and ∑∣C∣/∑D\sum\lvert C\rvert/\sum D, respectively. “Overall” is calculated by summing the numerators and denominators separately across both datasets, then taking their ratio.

ModelDatasetSigned ratio ∑C/∑D\sum C/\sum DAbsolute ratio ∑∣C∣/∑D\sum\lvert C\rvert/\sum D
Qwen1.5-MoEMathQA+2.19%3.63%
WikiText-2+0.58%2.04%
Overall+1.15%2.61%
DeepSeek-V2-LiteMathQA+0.074%0.172%
WikiText-2+0.079%0.347%
Overall+0.077%0.259%

Overall, cross-expert coupling accounts for 1.15% and 0.077% of the sum of individual expert errors in Qwen1.5-MoE and DeepSeek-V2-Lite, respectively. The corresponding absolute ratios are only 2.61% and 0.259%, respectively. These results show that individual expert errors dominate the overall quantization error in both models, while cross-expert coupling has a minor effect. This supports using the expert-level error bound in Eq. (4) to approximately estimate the overall MoE quantization error.

Weaknesses 2: Although the L2-norm based importance metric requires forward passes over a calibration dataset and thus remains data-dependent, it exhibits higher rank correlation across distribution shifts compared with routing-frequency-based heuristics. However, the metric cannot fully eliminate the bias introduced by the distribution of calibration samples.

Question 2: The L2-based importance metric relies on calibration data. Could the authors provide end-to-end downstream results to show how much performance degrades when there exists a large distribution gap between the calibration set and the test set?

Response 3: The L2-norm-based importance metric remains sensitive to the calibration data distribution, but produces more consistent expert rankings across domains than routing frequency. MxQBound also better preserves downstream performance under cross-domain calibration. Specifically:

  • Ranking stability across datasets. Figs. 9–11 report average Spearman rank correlations of 0.60–0.75 across the three models, compared with 0.15–0.32 for routing frequency.
  • Impact of domain differences. Rankings obtained using C4 and HumanEval as separate calibration sets have lower correlations of 0.26–0.50, indicating that importance estimation remains sensitive to domain differences.
  • Downstream performance. Table 1 reports results with calibration on C4 and evaluation on HumanEval. Under W4A4, MxQBound’s HumanEval Pass@1 drops by 5.48, 3.05, and 7.31 percentage points from FP16 on DeepSeek-V2-Lite, Qwen1.5-MoE-A2.7B, and Mixtral-8×7B, respectively. Under the same calibration setting, CodeQuant incurs larger drops of 6.12, 18.91, and 22.56 percentage points. Thus, MxQBound better preserves downstream performance on all three models despite the substantial domain gap between calibration and evaluation.
AuthorsOpenReview ↗

Response (3/3)

Question 3: The online adaptive activation bit allocation uses input norm as a proxy for output norm. Is the linear relationship between input norm and expert output norm consistently maintained across all benchmarks and model scales, especially under W4A4 low-bit quantization?

Response 3: We use changes in input norms to estimate changes in output norms and adjust activation bit-widths online. Appendix E supports this proxy relationship. The global bit budget constrains bit allocation but does not change the input norms used for the current adjustment. Specifically:

  • Input–output norm relationship. After training, each expert’s weights are fixed, which also fixes its input–output mapping. Therefore, we hypothesize that each expert acts as a relatively stable amplifier of the input signal, with approximately proportional input and output norms. Appendix E tests this hypothesis across three models and nine datasets. The median ratio of the input-norm scaling factor to the output-norm scaling factor ranges from 0.9748 to 1.0232, close to 1. Both scaling factors are measured relative to their respective calibration baselines. These results support our hypothesis and allow us to estimate changes in expert importance from input norms before computing the expert outputs.
  • Role of the bit budget. During online adaptation, we first increase or decrease the activation bit-widths of selected experts based on changes in their input norms relative to calibration baselines. We then adjust the bit-widths of other experts to offset the resulting change in bit cost and satisfy the global budget. The global budget corresponding to W4A4 constrains the final bit allocation, while input norms are computed before this adjustment and remain unaffected by it.

Dear Reviewer uoRG,

Thank you again for your valuable feedback and for taking the time to review our submission. As the discussion deadline approaches, please let us know if you have further concerns for clarification.

We would greatly appreciate your continued engagement in the discussion.

Best regards,

Authors.

Reviewer_By7UOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MxQBound, a mixed-precision quantization method for MoE models. It estimates expert importance from output norms, uses a local quantization-error bound to guide per-expert precision allocation, and adjusts activation precision according to the input during inference. Across several MoE models and bit-width budgets, it generally achieves better performance than the evaluated baselines.

Critical strengths
  1. The paper is clearly organized, and the motivation for its expert-importance metric is easy to follow.
  2. The evaluation covers multiple MoE architectures and precision budgets. The comparisons include relevant MoE quantization baselines, including MxMoE and CodeQuant.
  3. MxQBound offers a good accuracy–compression trade-off and substantially reduces offline quantization time.
Critical weaknesses
  1. The paper does not report deployed inference measurements. Table 2 measures offline quantization time, while the reported compression ratios do not establish actual GPU memory savings or inference speedups. These measurements matter particularly for mixed per-expert precision and online activation adaptation.
  2. The offline activation allocation is constrained by an average bit budget, but the paper does not establish whether the subsequent online adaptation also preserves that budget or report the resulting distribution of activation bit-widths.
Questions for authors
  1. Could you report measured model storage, peak GPU memory, latency, and throughput using the deployed quantized implementation? How are 3-bit and 5-bit values packed and executed, and is the mix-precision quantization format compatible with grouped GEMM operations?
  2. After online activation adaptation, what are the realized average and peak activation bit-widths? Is the target activation-bit budget enforced during inference?
  3. For models capable of these tasks, could you extend the evaluation to more challenging reasoning benchmarks, such as AIME and GPQA Diamond? These tasks with long reasoning are more sensitive to compression, and are closer to real applications.
8 comments
AuthorsOpenReview ↗

Response (1/3)

Thank you for the valuable feedback.

Weaknesses 1: The paper does not report deployed inference measurements. Table 2 measures offline quantization time, while the reported compression ratios do not establish actual GPU memory savings or inference speedups. These measurements matter particularly for mixed per-expert precision and online activation adaptation.

Question 1: Could you report measured model storage, peak GPU memory, latency, and throughput using the deployed quantized implementation? How are 3-bit and 5-bit values packed and executed, and is the mix-precision quantization format compatible with grouped GEMM operations?

Response 1: We have added deployment measurements for three MoE models. The results confirm reductions in model size and peak GPU memory usage, along with faster prefill. Below, we present the deployment measurements and describe our mixed-precision storage and execution.

  • Deployment Measurements

    We compare MxQBound with FP16 in terms of model size, peak GPU memory, prefill latency, decode latency, and decode throughput under the same hardware, software, and inference workload. All measurements are performed on a single NVIDIA RTX A6000 GPU. The quantized models use W4A4 quantization with a group size of 64 and 128 calibration samples. The batch size is 1, with 32 input tokens and 64 generated tokens. We report averages over five measurement runs. Model size includes unquantized modules, packed weights, and quantization parameters. Peak GPU memory usage is the maximum observed during prefill or decoding. The results are shown in the table below.

    ModelMethodModel size (GB)Peak GPU memory (GB)Prefill latency (ms)Decode latency (ms/token)Decode throughput (tokens/s)
    Qwen1.5-MoE-A2.7BFP1628.6328.66374.1299.73.34
    Ours10.9413.52342.0277.03.61
    Relative to FP162.62×2.12×1.09×1.08×1.08×
    Phi-mini-MoEFP1615.3015.40589.2382.12.62
    Ours6.717.52353.8328.63.04
    Relative to FP162.28×2.05×1.67×1.16×1.16×
    DeepSeek-V2-LiteFP1631.4131.48627.4349.42.86
    Ours10.9713.99466.0352.52.84
    Relative to FP162.86×2.25×1.35×0.99×0.99×

    The “Relative to FP16” denotes FP16/MxQBound for model size, memory, and latency, and MxQBound/FP16 for throughput. Higher values are better. Compared with FP16, our method achieves 2.28–2.86× compression in model size and a 2.05–2.25× reduction in peak GPU memory through compact low-bit storage of expert weights. MxQBound allocates bit-widths according to expert importance and quantization error to preserve accuracy under a given bit budget. It achieves 1.09–1.67× prefill speedups, with decoding performance similar to that of FP16. Prefill benefits from batched matrix computation, whereas online rotation, unpacking, and scheduling overhead limit decode speedups.

  • Mixed-Precision Storage and Execution

    • 3/5-bit storage and execution. We tightly pack 3-bit and 5-bit weights into int32. Triton kernels unpack and dequantize the weights during GEMM, with FP32 accumulation. We apply a randomized Hadamard transform (RHT) to the activations, quantize them in groups at each expert’s current activation bit-width, and dequantize them to FP16 before GEMM.
    • Grouped GEMM. We group experts by weight bit-width and matrix shape, then process the groups using ragged grouped GEMM. Since activations enter GEMM in FP16, Stage III changes only the activation quantization operation, without requiring regrouping. The end-to-end measurements include grouping, scheduling, and online activation processing overhead.
AuthorsOpenReview ↗

Response (2/3)

Weaknesses 2: The offline activation allocation is constrained by an average bit budget, but the paper does not establish whether the subsequent online adaptation also preserves that budget or report the resulting distribution of activation bit-widths.

Question 2: After online activation adaptation, what are the realized average and peak activation bit-widths? Is the target activation-bit budget enforced during inference?

Response 2: We combine online bit-width adaptation with budget balancing to keep the average activation bit-width close to the target. Under W4A4, the measured average ranges from 3.994 to 4.013 bits, while the peak is 12 or 16 bits. We describe the budget-balancing mechanism and present the activation bit-width statistics below.

  • Online Bit Adaptation and Budget Maintenance. Online adaptation starts from the offline activation bit allocation and proceeds in three steps.
    1. First, discrete bit-width mapping. We adjust the offline activation bit-width using the ratio of the current input norm to its calibration mean in Eq. (13). We increase the bit-width by one or two levels when the ratio exceeds 1.1 or 1.2, respectively, and decrease it by one or two levels when the ratio falls below 0.9 or 0.8, respectively. These thresholds are based on the means and standard deviations in Appendix E. A one-level adjustment moves to an adjacent value in the ordered candidate set {2,3,4,5,6,8,12,16}\{2,3,4,5,6,8,12,16\}. For example, increasing an 8-bit activation by one or two levels gives 12 or 16 bits, respectively.
    2. Next, average bit-width calculation. For each layer, Stage III averages the activation bit-widths used by the experts participating in adaptation over 16 consecutive tokens: bˉa(t)=1∣Et∣∑e∈Etbea(t).\bar b^a(t)=\frac{1}{|\mathcal E_t|}\sum_{e\in\mathcal E_t}b_e^a(t).
      Here, tt denotes the current adaptation batch, Et\mathcal E_t is the set of participating experts in the layer, and bea(t)b_e^a(t) is the activation bit-width used by expert ee.
    3. Finally, online budget balancing. We adjust other experts’ activation bit-widths in the same layer to close the gap between the average activation bit-width and the budget target. We offset the added cost of bit-width increases with the savings from bit-width decreases. Adjustments that cannot be balanced are skipped.
  • Realized Activation Bit-Width Statistics. We recorded activation bit-widths during inference for DeepSeek-V2-Lite, Qwen1.5-MoE, and Mixtral under W4A4 on WikiText-2, C4, and BoolQ. The average bit-width is calculated as described above. The peak is the highest quantization bit-width used during inference. Stay, Up, and Down denote the proportions of tokens whose activation bit-width remains unchanged, increases, or decreases relative to the offline allocation, respectively.
ModelDatasetAverage (bits)Peak (bits)StayUpDown
DeepSeek-V2-LiteWikiText-23.9961695.3%2.4%2.3%
C43.9971694.8%2.7%2.5%
BoolQ4.0071697.2%1.7%1.1%
Qwen1.5-MoEWikiText-24.0041695.5%2.3%2.2%
C44.0011694.0%3.3%2.6%
BoolQ4.0131696.0%2.6%1.4%
MixtralWikiText-24.0011699.9%0.1%0.0%
C43.9961299.6%0.0%0.4%
BoolQ3.9941699.7%0.1%0.2%

We can observe that the average activation bit-width ranges from 3.994 to 4.013 bits. The maximum absolute deviation from the 4-bit target is 0.013 bits. 94.0%-99.9% of tokens retain their default activation bit-width (Stay), while 0.0%-3.3% and 0.0%-2.6% use higher (Up) and lower (Down) bit-widths, respectively. Online bit adaptation thus reallocates precision for only a small fraction of inputs, allowing some activations to use 12 or 16 bits while keeping the average activation bit-width close to the target. We describe the budget-balancing mechanism in our response to Weakness 2.

AuthorsOpenReview ↗

Response (3/3)

Question 3: For models capable of these tasks, could you extend the evaluation to more challenging reasoning benchmarks, such as AIME and GPQA Diamond? These tasks with long reasoning are more sensitive to compression, and are closer to real applications.

Response 3: We have added evaluations of Qwen3-30B-A3B-Instruct-2507 on AIME2025 (I+II, 30 questions) and GPQA-Diamond (198 questions), comparing MxQBound with MxMoE and CodeQuant under W4A4 and W4A8. All methods use the same evaluation settings, with a maximum generation length of 4096 tokens and one response per question. The table below reports accuracy (pass@1).

MethodConfigurationAIME2025 (%)GPQA-Diamond (%)
FP16—53.351.3
CodeQuantW4A40.021.2
W4A80.025.3
MxMoEW4A40.022.7
W4A80.028.8
MxQBoundW4A426.740.9
W4A846.748.5

On AIME2025, MxQBound achieves 26.7% accuracy at W4A4 and 46.7% at W4A8, while CodeQuant and MxMoE score 0.0% under both configurations. At W4A8, MxQBound is only 6.6 percentage points below FP16. On GPQA-Diamond, MxQBound achieves 40.9%-48.5%, compared with 21.2%-25.3% for CodeQuant and 22.7%-28.8% for MxMoE. These results indicate that MxQBound better preserves multi-step reasoning performance on challenging tasks. Its expert-importance-aware bit allocation, activation outlier suppression, and online activation bit adaptation help reduce quantization error.

Reviewer_tZCyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MxQBound uses expert output norms to guide mixed-precision quantization of MoE language models. The paper derives a local error bound to motivate bit allocation. The pipeline combines smoothing, Hadamard transforms and clipping with separate weight and activation allocation. It also adjusts activation precision during inference. Experiments cover six MoE models, perplexity and seven downstream tasks. The reported results show improved accuracy and lower quantization preparation cost.

Critical strengths
  1. The low-bit results are promising. Under W4A4, Table 1 reports average scores of 50.65 on Qwen1.5-MoE and 58.40 on Mixtral. CodeQuant scores 41.23 and 47.65, respectively. The evaluation covers six MoE models and includes weight-only quantization (Appendix H). This breadth supports the practical relevance of the method.
  2. The calibration procedure is lightweight. Expert importance statistics are collected in one pass over the calibration set. Table 2 reports 15–52 minutes for quantization preparation. Appendices B and C provide calibration stability and pruning analyses. These results support the usefulness of an inexpensive expert-importance metric.
Critical weaknesses
  1. The allocation objective needs a clearer connection to the error analysis. Fixing activation error in Equation (6) leaves a cross term that still depends on weight precision. Under a shared budget, this term can affect allocation across experts. Equation (10) may be justified as a looser bound or a separable surrogate, but this step is not explained. The scope also needs clarification: Section 3.2 uses a local linear approximation, while Section 3.3.1 analyzes the down projection. Stating which quantized components the bound covers would resolve this ambiguity.
  2. The budget accounting after online adaptation is not clear. Section 2.1 and Equation (9) define activation budgets. Appendix E also explains input-norm averaging. However, the averaging of activation bit-widths after Equation (13) and discrete mapping remains unclear. Algorithm 1 does not specify how the online budget is maintained or measured. This matters for interpreting the nominal W4A4 and W4A8 comparisons. It is a reporting gap, not evidence that evaluations exceeded their budgets.
Questions for authors
  1. Please clarify whether the error bound covers the locally linearized expert, the down projection, or all quantized expert projections.
  2. Is Equation (10) a separable surrogate or a further relaxation of Equation (6)? Please explain the omitted cross term.
  3. After Equation (13) and discrete mapping, how is the achieved average activation bit-width computed? Please specify the mapping rule and weighting across experts, tokens or activation elements. Do W4A4 and W4A8 denote offline targets or achieved online averages?
4 comments

Thank you for the valuable feedback.

Weaknesses 1: The allocation objective needs a clearer connection to the error analysis. Fixing activation error in Equation (6) leaves a cross term that still depends on weight precision. Under a shared budget, this term can affect allocation across experts. Equation (10) may be justified as a looser bound or a separable surrogate, but this step is not explained. The scope also needs clarification: Section 3.2 uses a local linear approximation, while Section 3.3.1 analyzes the down projection. Stating which quantized components the bound covers would resolve this ambiguity.

Question 1: Please clarify whether the error bound covers the locally linearized expert, the down projection, or all quantized expert projections.

Question 2: Is Equation (10) a separable surrogate or a further relaxation of Equation (6)? Please explain the omitted cross term.

Response 1:

  • What does the error bound cover? The error bound covers all quantized expert projections. Specifically:
    • First, WW in Section 3.2 denotes the combined effective gain of the gate, up, and down projections for a given input. Eq (4) uses this model to analyze the quantization error in the entire expert’s output.
    • Second, the experiments in Section 2.4 show that quantizing zz introduces more error than the gate/up activations, and Wdown,eW_{\mathrm{down},e} contributes more than the gate/up weights. This holds across all three evaluated models (Fig 3 and 4).
    • Section 3.3.1 therefore focuses on the down projection, Ye=Wdown,ezY_e=W_{\mathrm{down},e}z, as shown in Eqs (5) and (6). We use the down-projection error to guide expert-level bit allocation and quantize the expert’s other projections at the selected bit-widths.
  • Why can the cross term be omitted? By expanding the bound (Eq (6)), we obtain: (ϵw+ϵa)2Ie2=(ϵw2+2ϵwϵa+ϵa2)Ie2.(\epsilon_w+\epsilon_a)^2I_e^2=(\epsilon_w^2+2\epsilon_w\epsilon_a+\epsilon_a^2)I_e^2. Applying 2ϵwϵa≤ϵw2+ϵa22\epsilon_w\epsilon_a\leq\epsilon_w^2+\epsilon_a^2 gives (ϵw+ϵa)2Ie2≤2(ϵw2+ϵa2)Ie2.(\epsilon_w+\epsilon_a)^2I_e^2\leq2(\epsilon_w^2+\epsilon_a^2)I_e^2. With the activation configuration fixed, the activation contribution 2∑eϵa,e2Ie22\sum_e\epsilon_{a,e}^2I_e^2 in the above inequality is constant during weight allocation. Therefore, by dropping this constant term and the factor 22, the corss term can be omitted and the objective for weight bit allocation can be transformed into Eq (10).

Weaknesses 2: The budget accounting after online adaptation is not clear. Section 2.1 and Equation (9) define activation budgets. Appendix E also explains input-norm averaging. However, the averaging of activation bit-widths after Equation (13) and discrete mapping remains unclear. Algorithm 1 does not specify how the online budget is maintained or measured. This matters for interpreting the nominal W4A4 and W4A8 comparisons. It is a reporting gap, not evidence that evaluations exceeded their budgets.

Question 3: After Equation (13) and discrete mapping, how is the achieved average activation bit-width computed? Please specify the mapping rule and weighting across experts, tokens or activation elements. Do W4A4 and W4A8 denote offline targets or achieved online averages?

Response 2: Online bit adaptation builds on the offline bit-widths and maintains the activation bit budget through discrete mapping, average bit-width calculation, and budget balancing. The steps are as follows.

  1. First, discrete bit-width mapping. We adjust the offline activation bit-width using the ratio of the current input norm to its calibration mean in Eq (13). We increase the bit-width by one or two levels when the ratio exceeds 1.1 or 1.2, respectively, and decrease it by one or two levels when the ratio falls below 0.9 or 0.8, respectively. These thresholds are based on the means and standard deviations in Appendix E. A one-level adjustment moves to an adjacent value in the ordered candidate set {2,3,4,5,6,8,12,16}\{2,3,4,5,6,8,12,16\}. For example, increasing an 8-bit activation by one or two levels gives 12 or 16 bits, respectively.
  2. Next, average bit-width calculation. For each layer, Stage III averages the activation bit-widths used by the experts participating in adaptation over 16 consecutive tokens: bˉa(t)=1∣Et∣∑e∈Etbea(t).\bar b^a(t)=\frac{1}{|\mathcal E_t|}\sum_{e\in\mathcal E_t}b_e^a(t). Here, tt denotes the current adaptation batch, Et\mathcal E_t is the set of participating experts in the layer, and bea(t)b_e^a(t) is the activation bit-width used by expert ee.
  3. Finally, online budget balancing. We adjust other experts’ activation bit-widths in the same layer to close the gap between the average activation bit-width and the budget target. We offset the added cost of bit-width increments with the savings from bit-width decrement. Adjustments that cannot be balanced are skipped. Thus, A4 and A8 in W4A4 and W4A8 denote the target average activation bit-widths for both offline allocation and online adaptation.

Dear Reviewer tZCy,

Thank you again for your valuable feedback and for taking the time to review our submission. As the discussion deadline approaches, please let us know if you have further concerns for clarification.

We would greatly appreciate your continued engagement in the discussion.

Best regards,

Authors.

Thank you for the authors' response. The authors have clarified the scope of the error analysis and explained the online bit-width adaptation mechanism. However, some concerns remain. First, the explanation for omitting the term in Equation (6) is insufficient. The cross term may still depend on weight precision. Second, although the authors have provided more details about budget control, the calculation of the average bit-width is still not sufficiently clear. Overall, the rebuttal addresses some of my concerns, but several issues remain. Therefore, I maintain my original rating.

AuthorsOpenReview ↗

Response (1/1)

Question 1: Thank you for the authors' response. The authors have clarified the scope of the error analysis and explained the online bit-width adaptation mechanism. However, some concerns remain. First, the explanation for omitting the term in Equation (6) is insufficient. The cross term may still depend on weight precision. Second, although the authors have provided more details about budget control, the calculation of the average bit-width is still not sufficiently clear. Overall, the rebuttal addresses some of my concerns, but several issues remain. Therefore, I maintain my original rating.

Thank you for clarifying your remaining concerns.

Cross term in Eq. (6). We derive Eq. (10) by applying the following looser upper bound:

\leq 2(\epsilon_{w,e}^2+\epsilon_{a,e}^2)I_e^2.$$ With the activation configuration fixed, the activation term in this bound is constant during weight allocation. Dropping this constant term and the common factor of two gives the surrogate objective in Eq. (10) under the same weight budget. To assess how the cross term affects allocation under a shared budget, we can compare Eq. (10) with an objective that retains the cross term. Both variants would use the same activation configuration, candidate weight bit-widths, and weight budget. We would report which experts receive different bit-widths and how downstream performance differs. **However, we received this follow-up shortly before the discussion deadline and cannot complete the additional experiment in the time remaining. We can report the results in the revised manuscript.** **Online average activation bit-width.** For budget balancing within each layer, Stage III averages the activation bit-widths of the experts participating in the current adjustment. Each participating expert processes 16 tokens, so each expert contributes equally to this average. Across the full evaluation, we weight the activation bit-widths actually used after discrete mapping and budget balancing by the corresponding token counts: $$\bar b^a=\frac{\sum_{e,t}n_{e,t}b_e^a(t)}{\sum_{e,t}n_{e,t}}.$$ Here, $b_e^a(t)$ is the final activation bit-width used by expert $e$ in batch $t$, and $n_{e,t}$ is the number of tokens that expert processes at that bit-width. A token routed to multiple experts counts once for each expert. A4 and A8 in W4A4 and W4A8 denote the target average activation bit-widths. We will clarify the per-layer budget calculation and the average across the full evaluation in Section 3.3.3 and Algorithm 1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Reply

We sincerely thank all the reviewers for their effort, feedback, and comments. We are pleased that the reviewers recognized MxQBound’s favorable accuracy–compression trade-off (tZCy, By7U), lightweight calibration and efficient offline quantization (tZCy, By7U), and mixed-precision allocation guided by expert importance and error analysis (uoRG).

We invite each reviewer to consult our individual responses, where we address all points raised in detail. In this general reply, we summarize the main changes made in response to the reviewers’ feedback:

  • Error bounds and bit allocation (tZCy)
    We clarified which quantized components the error bound covers and its connection to the down-projection analysis. We also derived the separable surrogate objective in Eq. (10) by bounding the activation–weight error cross term in Eq. (6).

  • Cross-expert error coupling (uoRG)
    We derived an overall MoE error bound without assuming independent expert errors. Additional measurements show that cross-expert coupling is small relative to the sum of individual expert errors

  • Online activation bit adaptation (tZCy, By7U)
    We clarified discrete bit-width mapping, budget balancing, and online average bit-width calculation. We also reported activation bit-width statistics during inference, showing that the average remains close to the target while selected activations receive higher precision.

  • Deployment efficiency (By7U)
    We added measurements of model size, peak GPU memory, latency, and throughput, and explained mixed-precision storage and execution. We also compared MxQBound with optimized FP16 baselines and extended the measurements to longer sequences and larger batches.

  • Challenging reasoning tasks (By7U)
    We added evaluations on AIME 2025 and GPQA Diamond, including a thinking-capable checkpoint tested on benchmark subsets with a 64K-token generation budget. We report accuracy, output lengths, and truncation rates to assess the effects of quantization on long reasoning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.