- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Context-Robust, a training-free activation calibration method for MoE language models. CR compares channelwise activation ranges collected from original and prefixed inputs under natural routing, aggregates their differences into a site-level statistic, and uses it as a lower bound on conventional Absmax quantization steps. Experiments on Qwen3.5-35B-A3B and Gemma-4-26B-A4B show improved average downstream performance across several weight configurations, with particularly large improvements on generation tasks. Additional analyses suggest that CR reduces clipping and severe degradation at the cost of coarser quantization, rather than uniformly improving approximation to the full-precision model.
Critical strengths
- This paper addresses a relevant challenge in MoE quantization that activation statistics depend on the limited token populations routed to individual experts.
- The method requires no retraining and produces fixed inference-time scales, making it potentially straightforward to incorporate into existing calibration pipelines.
- The method improves average downstream performance on two MoE architectures across multiple weight configurations, with substantial gains on HumanEval.
Critical weaknesses
- The geometric discussion motivates using contextual variation, but the authors do not explain why the median range discrepancy should directly determine a scalar quantization-step floor. Here should have more analysis or explaination.
- It seems that the improvements are strongly concentrated in a few tasks, like for Qwen W16A8, HumanEval improves by 32.93 points but the other six tasks improve by only limited points. Can the authors explain this in detail?
- How does CR compare with Absmax when both methods observe exactly the same original and prefixed inputs under the same token budget?
- I'm curous about if the advantage persist against a stronger expert-aware activation-quantization baseline under comparable conditions.
Questions for authors
Please refer to the weakness. I will be willing to adjust my score if the authors can address my concerns.
8 comments
Weakness 3
How does CR compare with Absmax when both methods observe exactly the same original and prefixed inputs under the same token budget?
We ran this control. All rules below are built from the same 2,048 forward passes (the 1,024 SkyPile inputs that give , and the 1,024 prefixed inputs; 1,322,321 forward tokens), re-collected for this purpose:
- Absmax: Original Absmax method with 1,024 SkyPile inputs.
- Absmax-union: Absmax method with both 1,024 SkyPile inputs and the 1,024 prefixed inputs.
- CR: Our method.
- Absmax-extra: Absmax given the same forward-token budget as additional independent, unprefixed SkyPile text.
We do not calibrate Absmax on the prefixed executions alone: they contain only 32 + 32 distinct targets, so the union is the variant that gives Absmax every execution CR sees. CR observes only the target tokens of the prefixed executions, and so do the rules below, with one addition: Absmax-union that also observes the prefix tokens, which is its most favorable setting and matches the token budget of Absmax-extra.
| Rule | HumanEval | MMLU | HellaSwag | BoolQ | Mean ΔPPL | P99 ΔPPL |
|---|---|---|---|---|---|---|
| Absmax | 24.39 (40) | 79.39 | 69.53 | 89.75 | 0.493 | 4.68 |
| Absmax-union, without prefix tokens | 24.39 (40) | 79.30 | 69.73 | 89.26 | 0.493 | 4.66 |
| Absmax-union, with prefix tokens | 29.27 (48) | 79.39 | 70.90 | 87.11 | 0.489 | 4.94 |
| Absmax-extra (equal token budget) | 31.71 (52) | 82.13 | 70.70 | 88.57 | 0.273 | 3.01 |
| CR | 68.90 (113) | 85.35 | 72.66 | 86.33 | 0.439 | 2.69 |
With the same inputs, CR improves on Absmax-union by 44.51 pp on HumanEval, 95% CI [35.98, 53.05] (paired bootstrap, 10,000 resamples; exact McNemar ), by 6.05 pp on MMLU and by 2.93 pp on HellaSwag, while BoolQ is 2.93 pp lower. Giving Absmax the prefixed executions changes HumanEval by 0.00 pp [−4.27, 4.27] (40 -> 40 of 164); even when it also observes the prefix tokens, Absmax-union reaches only 29.27% (48/164). Giving Absmax the same token budget of independent data reaches 31.71% (52/164), still 37.20 pp below CR. The same holds on the two code benchmarks we added:
| Method | MBPP+ (378) | LiveCodeBench test6 (175) |
|---|---|---|
| Absmax | 24.87 (94) | 13.71 (24) |
| Absmax-union | 29.37 (111) | 13.14 (23) |
| CR | 69.05 (261) | 34.86 (61) |
The gain therefore comes from the step rule and not from the additional executions. The reason is structural: extra executions enter Absmax only through a maximum, and the union changes just 6.6% of the channel ranges (10.6% when prefix tokens are also observed). CR does not use the prefixed runs as more range samples; it uses the difference between two range estimates, and this floor sets 98.9% of the step elements and raises the median step 9.56x (App. B).
We gained inference effienciency advantage from EAQuant (a stronger baseline with dynamic activation-quantization), and our accuracy is approximately the same.
Weakness 4
I'm curious about if the advantage persist against a stronger expert-aware activation-quantization baseline under comparable conditions.
In fact, about activation quant, there are no too many other methods, so we just made experiments on EBSS (MoEQuant) and EAQuant. We group activation-quantization baselines into static ones (fixed scales after calibration: Absmax, EBSS, CR) and dynamic ones (scales computed per token at inference: EAQuant), and added EAQuant-MoE as the stronger expert-aware baseline. All methods quantize the same routed experts under W16A8 and share one evaluation pipeline. EAQuant runs from its official code with expert-aware smoothing, rotation/permutation and per-token dynamic INT8 scales, calibrated on the same 1,024 SkyPile inputs as Absmax; its router-alignment component is not enabled (routers stay in FP16 for all methods). "EBSS data" is Absmax calibrated on 256 EBSS-sampled texts with a similar number of tokens as other calibration datasets; it fell below Absmax on HumanEval and was not continued.
| Method | Type | HumanEval (164) | MBPP+ (378) | LiveCodeBench test6 (175) | MMLU | HellaSwag | BoolQ |
|---|---|---|---|---|---|---|---|
| FP16 | / | 67.07 (110) | 65.87 (249) | 35.43 (62) | 85.45 | 73.14 | 86.62 |
| Absmax | static | 24.39 (40) | 24.87 (94) | 13.71 (24) | 79.39 | 69.53 | 89.75 |
| EBSS data + Absmax | static | 18.29 (30) | not run | not run | 72.46 | 66.89 | 87.99 |
| CR (ours) | static | 68.90 (113) | 69.05 (261) | 34.86 (61) | 85.35 | 72.66 | 86.33 |
| EAQuant-MoE | dynamic | 71.95 (118) | 57.41 (217) | 34.29 (60) | 85.64 | 73.44 | 87.40 |
Accuracy. Against the static baselines the advantage persists and is large. Against EAQuant, CR is comparable rather than better: EAQuant is higher on HumanEval (+3.1 pp, within sampling error for 164 problems), MMLU (+0.3), HellaSwag (+0.8) and BoolQ (+1.1), and CR is higher on MBPP+ (+11.6 pp).
What CR offers instead. EAQuant computes scales per token and applies rotation/permutation transforms at inference. CR changes only the value of a static per-channel step, so it is a drop-in replacement for Absmax with identical inference cost, and among static calibrations it closes most of the gap to FP16. With W4A8 integer kernels on one A100 (batch 1, 64 output tokens per request):
| Prompts | Method | TTFT (ms) ↓ | TPOT (ms/token) ↓ | Output tokens/s ↑ | Peak GPU memory (GiB) ↓ |
|---|---|---|---|---|---|
| SkyPile x1,024 | FP16 | 205.1 | 47.32 | 20.09 | 64.95 |
| SkyPile x1,024 | W4A8, GPTQ + CR (static) | 400.0 | 51.89 | 17.44 | 21.00 |
| SkyPile x1,024 | W4A8, EAQuant (dynamic) | 498.3 | 78.29 | 11.79 | 22.02 |
| HumanEval | FP16 | 190.5 | 48.36 | 19.77 | 64.85 |
| HumanEval | W4A8, GPTQ + CR (static) | 263.0 | 52.14 | 18.04 | 20.89 |
| HumanEval | W4A8, EAQuant (dynamic) | 283.9 | 78.92 | 12.18 | 21.50 |
The static path has 48% higher throughput than the full EAQuant pipeline (TPOT 51.9 vs 78.3 ms and 52.1 vs 78.9 ms). Relative to FP16, W4A8 lowers peak memory 3.1x but, with our current kernels and GPU, does not improve latency (throughput −13.2% and −8.7%); this is a property of the W4A8 backend, not of CR. In the simulated-quantization harness used for the accuracy runs, the full MBPP+ evaluation took 2.0 h for CR (21.0 tokens/s) and 20.3 h for EAQuant (3.0 tokens/s).
Response to average advantage: We add more datasets and these show same advantage to Humaneval
We thank the reviewer for the detailed questions. Weakness 1 complicated, we will discuss it at the last.
Unless stated otherwise, new results are for Qwen3.5-35B-A3B under W16A8 from a unified re-run in which calibration ranges were re-collected and all methods were generated and judged in one pipeline. HumanEval uses all 164 problems; MMLU, HellaSwag and BoolQ use fixed 1,024-item subsets. Absolute HumanEval values therefore differ from Table 1 (Table 3 in new revision, FP16 64.63 -> 67.07, Absmax 21.34 -> 24.39, CR 54.27 -> 68.90).
Weakness 2
We agree with this reading of Table 1 (Table 3 in new revision). For Qwen W16A8, CR − Absmax is +5.07 pp averaged over seven tasks but +0.42 pp without HumanEval. Below we show that the gain belongs to long-form generation rather than to one benchmark, explain why, and discuss the decreases.
(1) Two additional code-generation benchmarks. We added MBPP+ (378 problems) and LiveCodeBench (test6 release, 175 problems dated 2025-01 to 2025-04), which also addresses the remark that the benchmark set is dated.
CR improves over Absmax by 44.5, 44.2 and 21.1 pp and is within sampling error of FP16 on all three benchmarks. EAQuant is a stronger, dynamic baseline; its evaluation is about 10x slower based on its open source code. The same pattern appears on the second architecture: on Gemma-4 (Table 2, Table 4 in new revision, FP16 weights) both generation tasks improve, GSM8K 17.19 -> 43.36 and HumanEval 10.37 -> 33.54.
The submission contained a single code benchmark because one configuration then took more than 5 hours to evaluate; after optimizing the pipeline, a static-quantization run takes about 2 hours.
(2) Why long-form generation benefits most. Under Absmax every activation above the calibration maximum is clipped. CR's floor removes most of these events at the cost of a coarser grid (Table 16, Table 18 in new revision, HumanEval: clipped down-input entries 0.239% -> 0.015%, clipped gate/up entries 0.068% -> 0.012%, gate/up relative RMSE 0.053 -> 0.131).