acceptodds
Under review as a conference paper at ICLR 2027

Context-Robust: Context-Informed Activation Calibration for Mixture-of-Experts Language Models

Abstract

Sparse routing makes activation calibration in Mixture-of-Experts (MoE) models inherently router-conditioned: each expert observes only the calibration tokens routed to it, so a small activation range observed in one calibration realization does not necessarily indicate a stable small range. Consequently, conventional Absmax calibration can select overly fine quantization steps that are sensitive to the routed calibration population. We introduce **Context-Robust (CR)**, a training-free activation calibration method that uses contextualized natural-routing executions to probe routed range instability. CR measures channelwise routed-range discrepancies between original and contextualized calibration collections, aggregates them into a site-level routed range-instability signal, and uses this signal as a one-sided floor on channelwise Absmax quantization steps. The resulting steps remain fixed during inference. Across Qwen3.5-35B-A3B and Gemma-4-26B-A4B, CR consistently improves average downstream performance over Absmax across FP16, AWQ, GPTQ, and SmoothQuant weight configurations, while exhibiting a repeatable task-selective preservation pattern. Controlled W16A8 diagnostics further show substantially reduced sample-level degradation dispersion and upper-tail failures while keeping mean degradation close to Absmax. These results show that router-conditioned range instability provides useful calibration information beyond activation magnitude for MoE activation quantization.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Experts, activation quantization, post-training quantization, model compression, efficient inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_WNG2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Context-Robust, a training-free activation calibration method for MoE language models. CR compares channelwise activation ranges collected from original and prefixed inputs under natural routing, aggregates their differences into a site-level statistic, and uses it as a lower bound on conventional Absmax quantization steps. Experiments on Qwen3.5-35B-A3B and Gemma-4-26B-A4B show improved average downstream performance across several weight configurations, with particularly large improvements on generation tasks. Additional analyses suggest that CR reduces clipping and severe degradation at the cost of coarser quantization, rather than uniformly improving approximation to the full-precision model.

Critical strengths
  1. This paper addresses a relevant challenge in MoE quantization that activation statistics depend on the limited token populations routed to individual experts.
  2. The method requires no retraining and produces fixed inference-time scales, making it potentially straightforward to incorporate into existing calibration pipelines.
  3. The method improves average downstream performance on two MoE architectures across multiple weight configurations, with substantial gains on HumanEval.
Critical weaknesses
  1. The geometric discussion motivates using contextual variation, but the authors do not explain why the median range discrepancy should directly determine a scalar quantization-step floor. Here should have more analysis or explaination.
  2. It seems that the improvements are strongly concentrated in a few tasks, like for Qwen W16A8, HumanEval improves by 32.93 points but the other six tasks improve by only limited points. Can the authors explain this in detail?
  3. How does CR compare with Absmax when both methods observe exactly the same original and prefixed inputs under the same token budget?
  4. I'm curous about if the advantage persist against a stronger expert-aware activation-quantization baseline under comparable conditions.
Questions for authors

Please refer to the weakness. I will be willing to adjust my score if the authors can address my concerns.

8 comments
AuthorsOpenReview ↗

Response to average advantage: We add more datasets and these show same advantage to Humaneval

We thank the reviewer for the detailed questions. Weakness 1 complicated, we will discuss it at the last.

Unless stated otherwise, new results are for Qwen3.5-35B-A3B under W16A8 from a unified re-run in which calibration ranges were re-collected and all methods were generated and judged in one pipeline. HumanEval uses all 164 problems; MMLU, HellaSwag and BoolQ use fixed 1,024-item subsets. Absolute HumanEval values therefore differ from Table 1 (Table 3 in new revision, FP16 64.63 -> 67.07, Absmax 21.34 -> 24.39, CR 54.27 -> 68.90).

Weakness 2

It seems that the improvements are strongly concentrated in a few tasks, like for Qwen W16A8, HumanEval improves by 32.93 points but the other six tasks improve by only limited points. Can the authors explain this in detail?

We agree with this reading of Table 1 (Table 3 in new revision). For Qwen W16A8, CR − Absmax is +5.07 pp averaged over seven tasks but +0.42 pp without HumanEval. Below we show that the gain belongs to long-form generation rather than to one benchmark, explain why, and discuss the decreases.

(1) Two additional code-generation benchmarks. We added MBPP+ (378 problems) and LiveCodeBench (test6 release, 175 problems dated 2025-01 to 2025-04), which also addresses the remark that the benchmark set is dated.

Method (W16A8)HumanEval (164)MBPP+ (378)LiveCodeBench test6 (175)
FP1667.07 (110)65.87 (249)35.43 (62)
Absmax24.39 (40)24.87 (94)13.71 (24)
CR (ours)68.90 (113)69.05 (261)34.86 (61)
EAQuant (dynamic)71.95 (118)57.41 (217)34.29 (60)

CR improves over Absmax by 44.5, 44.2 and 21.1 pp and is within sampling error of FP16 on all three benchmarks. EAQuant is a stronger, dynamic baseline; its evaluation is about 10x slower based on its open source code. The same pattern appears on the second architecture: on Gemma-4 (Table 2, Table 4 in new revision, FP16 weights) both generation tasks improve, GSM8K 17.19 -> 43.36 and HumanEval 10.37 -> 33.54.

The submission contained a single code benchmark because one configuration then took more than 5 hours to evaluate; after optimizing the pipeline, a static-quantization run takes about 2 hours.

(2) Why long-form generation benefits most. Under Absmax every activation above the calibration maximum is clipped. CR's floor removes most of these events at the cost of a coarser grid (Table 16, Table 18 in new revision, HumanEval: clipped down-input entries 0.239% -> 0.015%, clipped gate/up entries 0.068% -> 0.012%, gate/up relative RMSE 0.053 -> 0.131).

  • A multiple-choice item is a single decision, so its accuracy can move by at most the per-decision flip rate, in either direction.
  • A program survives only if a long run of consecutive tokens is right. With a per-token deviation probability pp, the probability that KK tokens stay on the FP16 path is about (1−p)K(1-p)^K, so a per-token difference is amplified exponentially. On HumanEval reference tokens (Table 15, Table 17 in new revision), teacher-forced top-1 disagreement with FP16 is 8.36% for Absmax and 2.92% for CR; the mean first divergence moves from token 6.3 to 15.5, and the greedy continuation matches FP16 exactly on 4/164 versus 37/164 prompts. Generated programs are several hundred tokens long (about 400 on MBPP+ in our runs), so this per-token gap becomes a large pass-rate gap.

Weakness 3

How does CR compare with Absmax when both methods observe exactly the same original and prefixed inputs under the same token budget?

We ran this control. All rules below are built from the same 2,048 forward passes (the 1,024 SkyPile inputs that give MM, and the 1,024 prefixed inputs; 1,322,321 forward tokens), re-collected for this purpose:

  • Absmax: Original Absmax method with 1,024 SkyPile inputs.
  • Absmax-union: Absmax method with both 1,024 SkyPile inputs and the 1,024 prefixed inputs.
  • CR: Our method.
  • Absmax-extra: Absmax given the same forward-token budget as additional independent, unprefixed SkyPile text.

We do not calibrate Absmax on the prefixed executions alone: they contain only 32 + 32 distinct targets, so the union is the variant that gives Absmax every execution CR sees. CR observes only the target tokens of the prefixed executions, and so do the rules below, with one addition: Absmax-union that also observes the prefix tokens, which is its most favorable setting and matches the token budget of Absmax-extra.

RuleHumanEvalMMLUHellaSwagBoolQMean ΔPPLP99 ΔPPL
Absmax24.39 (40)79.3969.5389.750.4934.68
Absmax-union, without prefix tokens24.39 (40)79.3069.7389.260.4934.66
Absmax-union, with prefix tokens29.27 (48)79.3970.9087.110.4894.94
Absmax-extra (equal token budget)31.71 (52)82.1370.7088.570.2733.01
CR68.90 (113)85.3572.6686.330.4392.69

With the same inputs, CR improves on Absmax-union by 44.51 pp on HumanEval, 95% CI [35.98, 53.05] (paired bootstrap, 10,000 resamples; exact McNemar p=6.4×10−18p=6.4\times10^{-18}), by 6.05 pp on MMLU and by 2.93 pp on HellaSwag, while BoolQ is 2.93 pp lower. Giving Absmax the prefixed executions changes HumanEval by 0.00 pp [−4.27, 4.27] (40 -> 40 of 164); even when it also observes the prefix tokens, Absmax-union reaches only 29.27% (48/164). Giving Absmax the same token budget of independent data reaches 31.71% (52/164), still 37.20 pp below CR. The same holds on the two code benchmarks we added:

MethodMBPP+ (378)LiveCodeBench test6 (175)
Absmax24.87 (94)13.71 (24)
Absmax-union29.37 (111)13.14 (23)
CR69.05 (261)34.86 (61)

The gain therefore comes from the step rule and not from the additional executions. The reason is structural: extra executions enter Absmax only through a maximum, and the union changes just 6.6% of the channel ranges (10.6% when prefix tokens are also observed). CR does not use the prefixed runs as more range samples; it uses the difference between two range estimates, and this floor sets 98.9% of the step elements and raises the median step 9.56x (App. B).

AuthorsOpenReview ↗

We gained inference effienciency advantage from EAQuant (a stronger baseline with dynamic activation-quantization), and our accuracy is approximately the same.

Weakness 4

I'm curious about if the advantage persist against a stronger expert-aware activation-quantization baseline under comparable conditions.

In fact, about activation quant, there are no too many other methods, so we just made experiments on EBSS (MoEQuant) and EAQuant. We group activation-quantization baselines into static ones (fixed scales after calibration: Absmax, EBSS, CR) and dynamic ones (scales computed per token at inference: EAQuant), and added EAQuant-MoE as the stronger expert-aware baseline. All methods quantize the same routed experts under W16A8 and share one evaluation pipeline. EAQuant runs from its official code with expert-aware smoothing, rotation/permutation and per-token dynamic INT8 scales, calibrated on the same 1,024 SkyPile inputs as Absmax; its router-alignment component is not enabled (routers stay in FP16 for all methods). "EBSS data" is Absmax calibrated on 256 EBSS-sampled texts with a similar number of tokens as other calibration datasets; it fell below Absmax on HumanEval and was not continued.

MethodTypeHumanEval (164)MBPP+ (378)LiveCodeBench test6 (175)MMLUHellaSwagBoolQ
FP16/67.07 (110)65.87 (249)35.43 (62)85.4573.1486.62
Absmaxstatic24.39 (40)24.87 (94)13.71 (24)79.3969.5389.75
EBSS data + Absmaxstatic18.29 (30)not runnot run72.4666.8987.99
CR (ours)static68.90 (113)69.05 (261)34.86 (61)85.3572.6686.33
EAQuant-MoEdynamic71.95 (118)57.41 (217)34.29 (60)85.6473.4487.40

Accuracy. Against the static baselines the advantage persists and is large. Against EAQuant, CR is comparable rather than better: EAQuant is higher on HumanEval (+3.1 pp, within sampling error for 164 problems), MMLU (+0.3), HellaSwag (+0.8) and BoolQ (+1.1), and CR is higher on MBPP+ (+11.6 pp).

What CR offers instead. EAQuant computes scales per token and applies rotation/permutation transforms at inference. CR changes only the value of a static per-channel step, so it is a drop-in replacement for Absmax with identical inference cost, and among static calibrations it closes most of the gap to FP16. With W4A8 integer kernels on one A100 (batch 1, 64 output tokens per request):

PromptsMethodTTFT (ms) ↓TPOT (ms/token) ↓Output tokens/s ↑Peak GPU memory (GiB) ↓
SkyPile x1,024FP16205.147.3220.0964.95
SkyPile x1,024W4A8, GPTQ + CR (static)400.051.8917.4421.00
SkyPile x1,024W4A8, EAQuant (dynamic)498.378.2911.7922.02
HumanEvalFP16190.548.3619.7764.85
HumanEvalW4A8, GPTQ + CR (static)263.052.1418.0420.89
HumanEvalW4A8, EAQuant (dynamic)283.978.9212.1821.50

The static path has 48% higher throughput than the full EAQuant pipeline (TPOT 51.9 vs 78.3 ms and 52.1 vs 78.9 ms). Relative to FP16, W4A8 lowers peak memory 3.1x but, with our current kernels and GPU, does not improve latency (throughput −13.2% and −8.7%); this is a property of the W4A8 backend, not of CR. In the simulated-quantization harness used for the accuracy runs, the full MBPP+ evaluation took 2.0 h for CR (21.0 tokens/s) and 20.3 h for EAQuant (3.0 tokens/s).

Reviewer_tZ83OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Context-Robust proposes a training-free activation-calibration method for MoE quantization. It compares activation ranges from original and prefixed inputs under natural routing, aggregates their discrepancies into a site-level instability signal, and uses this signal to impose a lower bound on quantization step sizes. Experiments across two MoE architectures and multiple weight configurations show improved average downstream scores over Absmax. Supporting analyses suggest that its main benefit is reducing severe degradation rather than uniformly lowering quantization error.

Critical strengths

Simple integration with minimal inference-time changes. Contextual probing is confined to calibration; the method requires no fine-tuning, routing modifications, or online scale adjustment, making it compatible with static quantization workflows.

Useful analysis of severe degradation. CR reduces the 99th percentile of sample-level perplexity degradation from 4.61 to 2.67. Matched-step controls further suggest that uniform step enlargement alone does not explain the gains.

Critical weaknesses

A fully calibration-data-matched control is missing. CR uses 1,024 additional prefixed executions, but the paper does not compare against Absmax recalibrated using those same executions. This leaves the contributions of broader calibration coverage and the proposed step-size rule insufficiently separated.

A key evaluation has a potential comparability issue. HumanEval accounts for most of the average improvement on Qwen, but the archived CR and Absmax batches used different execution environments: Docker versus an isolated local Python process. A matched-environment rerun is needed to rule out this potential confound.

Questions for authors

Please explain the concerns in weakness. I am happy to reconsider my score if authors could commit some convinced experiment results or comments.

4 comments
AuthorsOpenReview ↗

We have carefully considered the two weaknesses you raised and added the corresponding experiments.

We thank the reviewer for the two concrete requests; both experiments are done. New results are for Qwen3.5-35B-A3B under W16A8. HumanEval uses all 164 problems (greedy decoding, at most 512 new tokens); MMLU, HellaSwag and BoolQ use fixed 1,024-item subsets; ΔPPL is measured on 1,024 held-out SkyPile texts against the FP16 reference of the same run.

Weakness 1

A fully calibration-data-matched control is missing. CR uses 1,024 additional prefixed executions, but the paper does not compare against Absmax recalibrated using those same executions.

We ran this control. All rules below are built from the same 2,048 forward passes (the 1,024 SkyPile inputs that give MM, and the 1,024 prefixed inputs; 1,322,321 forward tokens), re-collected for this purpose:

  • Absmax: Original Absmax method with 1,024 SkyPile inputs.
  • Absmax-union: Absmax method with both 1,024 SkyPile inputs and the 1,024 prefixed inputs.
  • CR: Our method.
  • Absmax-extra: Absmax given the same forward-token budget as additional independent, unprefixed SkyPile text.

We do not calibrate Absmax on the prefixed executions alone: they contain only 32 + 32 distinct targets, so the union is the variant that gives Absmax every execution CR sees. CR observes only the target tokens of the prefixed executions, and so do the rules below, with one addition: Absmax-union that also observes the prefix tokens, which is its most favorable setting and matches the token budget of Absmax-extra.

RuleHumanEvalMMLUHellaSwagBoolQMean ΔPPLP99 ΔPPL
Absmax24.39 (40)79.3969.5389.750.4934.68
Absmax-union, without prefix tokens24.39 (40)79.3069.7389.260.4934.66
Absmax-union, with prefix tokens29.27 (48)79.3970.9087.110.4894.94
Absmax-extra (equal token budget)31.71 (52)82.1370.7088.570.2733.01
CR68.90 (113)85.3572.6686.330.4392.69

With the same inputs, CR improves on Absmax-union by 44.51 pp on HumanEval, 95% CI [35.98, 53.05] (paired bootstrap, 10,000 resamples; exact McNemar p=6.4×10−18p=6.4\times10^{-18}), by 6.05 pp on MMLU and by 2.93 pp on HellaSwag, while BoolQ is 2.93 pp lower. Giving Absmax the prefixed executions changes HumanEval by 0.00 pp [−4.27, 4.27] (40 -> 40 of 164); even when it also observes the prefix tokens, Absmax-union reaches only 29.27% (48/164). Giving Absmax the same token budget of independent data reaches 31.71% (52/164), still 37.20 pp below CR. The same holds on the two code benchmarks we added:

MethodMBPP+ (378)LiveCodeBench test6 (175)
Absmax24.87 (94)13.71 (24)
Absmax-union29.37 (111)13.14 (23)
CR69.05 (261)34.86 (61)

The gain therefore comes from the step rule and not from the additional executions. The reason is structural: extra executions enter Absmax only through a maximum, and the union changes just 6.6% of the channel ranges (10.6% when prefix tokens are also observed). CR does not use the prefixed runs as more range samples; it uses the difference between two range estimates, and this floor sets 98.9% of the step elements and raises the median step 9.56x (App. B).

Weakness 2

HumanEval accounts for most of the average improvement on Qwen, but the archived CR and Absmax batches used different execution environments: Docker versus an isolated local Python process. A matched-environment rerun is needed.

We ran two checks.

(1) Same completions, one sandbox. All 13 archived configurations x 164 completions (2,132 in total) were re-judged in a single environment (Bubblewrap sandbox, Python 3.13.15, standard library only). One verdict of 2,132 changed: SmoothQuant + CR on HumanEval/81.

MethodPassed: archived -> re-judgedRe-judged (%)
FP16106 -> 10664.63
Absmax35 -> 3521.34
AWQ + Absmax38 -> 3823.17
GPTQ + Absmax34 -> 3420.73
SmoothQuant + Absmax36 -> 3621.95
CR89 -> 8954.27
AWQ + CR87 -> 8753.05
GPTQ + CR96 -> 9658.54
SmoothQuant + CR89 -> 8853.66
EBSS31 -> 3118.90
AWQ + EBSS31 -> 3118.90
GPTQ + EBSS28 -> 2817.07
SmoothQuant + EBSS34 -> 3420.73

CR − Absmax (W16A8) stays at +32.93 pp, 95% CI [24.39, 42.07], exact McNemar p=9.0×10−11p=9.0\times10^{-11}.

(2) Fresh generation, one pipeline. With re-collected calibration ranges and all methods generated and judged in the same pipeline, FP16 passes 110/164, Absmax 40/164 and CR 113/164 (W1 above). The counts differ from the archive because the ranges were re-collected and the completions regenerated.

The mismatch in the submission was logistical: HumanEval generation is long, the two archived batches were produced at different times, and we lacked the compute to migrate them before the deadline. We will correct Table 1 (Table 3 in new revision, SmoothQuant + CR HumanEval 54.27 -> 53.66, average 69.41 -> 69.32) and replace the mixed-environment note in Appendix H with the single-sandbox protocol.

Thanks for detailed response. I believe these evidence will strengthen the persuasiveness of the paper. I also advise authors to revise the paper at the same time. I decide to raise my score.

AuthorsOpenReview ↗

Thanks for your comments, we will revise our paper later.

We thank the reviewer for the positive feedback and for raising the score. Following your advice, we are revising the paper together with the response.

Specifically, we will:

  1. add the calibration-data-matched control (Absmax-union and Absmax-extra) to the experiments section, with the full results in a new appendix;
  2. update Table 1 and Appendix H with the single-sandbox HumanEval we reran;
  3. add the MBPP+ and LiveCodeBench results to Tables 1, 2;

[Updated]

We have uploaded the revised PDF; all changes addressing the reviews are shown in blue.

  1. Calibration-data-matched control: Section 5.4 and the new Appendix L (Table 25).
  2. Single-sandbox HumanEval: re-judged values in Table 3 (Table 1 in the submission); protocol in Appendix H.
  3. MBPP+ and LiveCodeBench: new main-text Table 1 and Appendix M (Table 26).

Thank you again for the constructive comments, these do help us improve the paper.

Reviewer_gZuuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose Context-Robust, which is a training-free activation calibration method for MoE-base LLMs. The proposed method estimates expert activation-range instability using contextualized inputs and incorporates it as a lower bound on Absmax quantization steps. Experimental results on Qwen3.5 and Gemma-4 demonstrate improvements.

Critical strengths
  • The paper addresses an meaningful research problem in MoE-based LLM activation quantization and proposes a simple yet effective training-free calibration method;

  • The paper provide extensive experiments and detailed analysis experiments, which demonstrates the effectiveness of the proposed method;

  • The benchmark evaluation is outdated;

Critical weaknesses
  • The performance improvements are task-dependent. The average gains are largely driven by HumanEval, while several other benchmarks show performance decrease;

  • The practical efficiency benefits are not sufficiently demonstrated. The evaluation primarily relies on simulated activation quantization without real low-precision inference kernels or even end-to-end efficiency measurements. The additional calibration overhead is also insufficiently characterized.

Questions for authors

See weakness part.

5 comments
AuthorsOpenReview ↗

Response to average advantage: We add more datasets and these show same advantage to Humaneval

We thank the reviewer for the careful reading.

Weakness 1

The performance improvements are task-dependent. The average gains are largely driven by HumanEval, while several other benchmarks show performance decrease.

We agree with this reading of Table 1 (Table 3 in new revision). For Qwen W16A8, CR − Absmax is +5.07 pp averaged over seven tasks but +0.42 pp without HumanEval. Below we show that the gain belongs to long-form generation rather than to one benchmark, explain why, and discuss the decreases.

(1) Two additional code-generation benchmarks. We added MBPP+ (378 problems) and LiveCodeBench (test6 release, 175 problems dated 2025-01 to 2025-04), which also addresses the remark that the benchmark set is dated.

Method (W16A8)HumanEval (164)MBPP+ (378)LiveCodeBench test6 (175)
FP1667.07 (110)65.87 (249)35.43 (62)
Absmax24.39 (40)24.87 (94)13.71 (24)
CR (ours)68.90 (113)69.05 (261)34.86 (61)
EAQuant (dynamic)71.95 (118)57.41 (217)34.29 (60)

CR improves over Absmax by 44.5, 44.2 and 21.1 pp and is within sampling error of FP16 on all three benchmarks. EAQuant is a stronger, dynamic baseline; its evaluation is about 10x slower based on its open source code. The same pattern appears on the second architecture: on Gemma-4 (Table 2, Table 4 in new revision, FP16 weights) both generation tasks improve, GSM8K 17.19 -> 43.36 and HumanEval 10.37 -> 33.54.

The submission contained a single code benchmark because one configuration then took more than 5 hours to evaluate; after optimizing the pipeline, a static-quantization run takes about 2 hours.

(2) Why long-form generation benefits most. Under Absmax every activation above the calibration maximum is clipped. CR's floor removes most of these events at the cost of a coarser grid (Table 16, Table 18 in new revision, HumanEval: clipped down-input entries 0.239% -> 0.015%, clipped gate/up entries 0.068% -> 0.012%, gate/up relative RMSE 0.053 -> 0.131).

  • A multiple-choice item is a single decision, so its accuracy can move by at most the per-decision flip rate, in either direction.
  • A program survives only if a long run of consecutive tokens is right. With a per-token deviation probability pp, the probability that KK tokens stay on the FP16 path is about (1−p)K(1-p)^K, so a per-token difference is amplified exponentially. On HumanEval reference tokens (Table 15, Table 17 in new revision), teacher-forced top-1 disagreement with FP16 is 8.36% for Absmax and 2.92% for CR; the mean first divergence moves from token 6.3 to 15.5, and the greedy continuation matches FP16 exactly on 4/164 versus 37/164 prompts. Generated programs are several hundred tokens long (about 400 on MBPP+ in our runs), so this per-token gap becomes a large pass-rate gap.
AuthorsOpenReview ↗

We evaluate inference efficiency on real kernel environment on 64 tok, including FP16, static W4A8, dynamic W4A8 (EAQuant)

Weakness 2

The practical efficiency benefits are not sufficiently demonstrated. The evaluation primarily relies on simulated activation quantization without real low-precision inference kernels or even end-to-end efficiency measurements. The additional calibration overhead is also insufficiently characterized.

What CR costs at inference. CR changes only the value of the calibrated step. CR and Absmax produce the same artifact (one fixed step per channel per site), so CR has exactly the inference cost of static Absmax: no prefixes, no routing changes and no online scale computation. The efficiency profile below is therefore that of the static W4A8 backend, shared by every static method.

End-to-end measurement with integer kernels. We ran GPTQ W4 weights + CR INT8 activations with static integer GEMM kernels on one A100-80GB (batch 1, 64 output tokens per request, FP16 measured in the same run), and compared with the full EAQuant W4A8 pipeline (dynamic per-token scales with rotation/permutation transforms).

PromptsMethodTTFT (ms) ↓TPOT (ms/token) ↓Output tokens/s ↑Peak GPU memory (GiB) ↓
SkyPile x1,024FP16205.147.3220.0964.95
SkyPile x1,024W4A8, GPTQ + CR (static)400.051.8917.4421.00
SkyPile x1,024W4A8, EAQuant (dynamic)498.378.2911.7922.02
HumanEvalFP16190.548.3619.7764.85
HumanEvalW4A8, GPTQ + CR (static)263.052.1418.0420.89
HumanEvalW4A8, EAQuant (dynamic)283.978.9212.1821.50
  • Peak GPU memory is 3.1x lower than FP16 (64.9 -> 21.0 GiB).
  • With our current kernels on an A100 there is no latency gain: throughput is 13.2% (SkyPile) and 8.7% (HumanEval) lower than FP16, and TTFT is 1.95x and 1.38x higher. It is a property of the W4A8 backend, not of CR.
  • Against the full EAQuant pipeline, the static path has 48% higher throughput (TPOT 51.9 vs 78.3 ms on SkyPile, 52.1 vs 78.9 ms on HumanEval) with 0.6–1.0 GiB less memory.

Calibration overhead. CR adds one forward-only pass over the 1,024 prefixed inputs; the unprefixed ranges reuse the Absmax forward passes.

Item (one A100)Cost
Absmax range collection (1,024 inputs, 576k tokens)3,121 s
CR additional pass (1,024 prefixed inputs, 746k tokens)3,230 s
Step construction on CPU (median and max)16.5 s
Peak GPU memory during collection64.94 GiB (same as FP16 inference)
For reference: GPTQ weight calibration4,765 s
For reference: EAQuant collection + fitting1,912 + 3,674 s

CR therefore costs about 2x the Absmax range collection (+54 minutes, one-time), with no gradients and no additional GPU memory.

Evaluation cost in the simulated-quantization harness. For completeness, the full MBPP+ evaluation (378 problems, A100) took:

MethodPassWall-clockOutput tokensTokens/s
FP16249/37801:58:16164,14723.58
Absmax W16A894/37802:15:47162,47921.11
CR W16A8261/37802:00:32149,00121.01
EAQuant W16A8217/37820:17:04217,3852.98

CR and Absmax run at the same speed. Beyond the fixed 64-token setting, EAQuant's overhead is paid at every decoding step, so the absolute latency gap grows linearly with output length. In our full MBPP+ runs (simulated quantization), the per-token slowdown (about 7x) matched the fixed-64 measurement (6.9x).

Dear Reviewer,

The authors have posted their rebuttal. Please read it and let them know whether it addresses your concerns, updating your review and score if appropriate, so that they have a chance to address any remaining ones.

Thank you, Area Chair

Comments on the paper

1 comment
AuthorsOpenReview ↗

We have uploaded a new revision

Summary of Revisions

We thank all reviewers for their constructive feedback. We have uploaded a revised PDF; changes that address the reviews are shown in blue. The main changes are listed below, followed by a map from the table numbers used in our responses to the revised numbering.

New experiments and analyses

  1. Code-generation benchmarks and an expert-aware dynamic baseline. Table 1 (main text) now compares FP16, Absmax, CR, and EAQuant-MoE under W16A8 on HumanEval, MBPP+, LiveCodeBench, MMLU, HellaSwag, and BoolQ; the LiveCodeBench run for EAQuant, still pending at the time of our response, is now complete. CR stays on par with FP16 on all three code benchmarks, and its accuracy is comparable to EAQuant while its scales remain static. Full results, including problem counts and the EBSS-data baseline, are in Appendix M (Table 26).

  2. Task dependence of the gains. Tables 3 and 4 now include a non-coding average, and Section 5.2 explains why the code benchmarks are reported separately: long generations compound per-token errors, and code benchmarks are too costly to run in every configuration. CR has the best non-coding and seven-task averages in every weight configuration on both models.

  3. Calibration-data-matched control. Section 5.4 and Appendix L (Table 25) compare CR with Absmax calibrated on the union of the original and prefixed executions (with and without prefix tokens) and on an equal token budget of independent text. Every Absmax variant stays far below CR on HumanEval (CR − Absmax-union: +44.51 points, 95% CI [35.98, 53.05], McNemar p = 6.4×10⁻¹⁸), and Absmax-union also stays far below CR on MBPP+ and LiveCodeBench.

  4. HumanEval execution environment. All HumanEval completions are now judged in a single Bubblewrap sandbox (Appendix H). Only one of 2,132 verdicts changed; Table 3 reports the re-judged values (SmoothQuant + CR: 54.27 → 53.66).

  5. End-to-end efficiency and calibration cost. Table 2 (main text) and Appendix N (Tables 27–28) report time to first token, time per output token, throughput, and peak memory with static integer GEMM kernels for W4A8 GPTQ + CR, compared with FP16 and the EAQuant pipeline, together with a breakdown of the calibration cost.

  6. From the median range discrepancy to a scalar step floor. Section 3 is rewritten as a geometric motivation plus the clipping–rounding trade-off, and the floor strength γ is exposed as an explicit coefficient (Section 4.3). The new Appendix K (Figures 4–8, Tables 23–24) adds a conceptual figure, a floor-strength sweep with a validation-selected γ evaluated downstream, a fixed-input analysis of the rounding/clipping trade-off, site-assignment controls, and an ablation of the aggregation statistic.

Corrections and presentation

  • Table 9 (Table 7 in the submission): the γ = 0.5 row contained a transcription error and has been corrected; the accompanying text is updated.
  • The abstract, introduction, and conclusion are reorganized to reflect the new results; the method is unchanged.
  • Experimental descriptions in Section 5 and the appendices now explain how to read each table and what it shows, instead of repeating table values in the text.

Table numbering

Two new tables appear at the start of Section 5, so the original tables are renumbered:

In our responsesIn the revisionContent
Table 1Table 3Qwen3.5-35B-A3B, weight configurations
Table 2Table 4Gemma-4-26B-A4B, weight configurations
Table 4Table 6Representation displacement under prefixes
Table 7Table 9Floor-strength sensitivity (γ)
Table 15Table 17HumanEval prediction drift
Table 16Table 18HumanEval clipping statistics
newTables 1–2W16A8 code/multiple-choice comparison; end-to-end efficiency
newTables 23–28γ validation, aggregation ablation, data-matched control, code baselines, efficiency, calibration cost

New appendices: K (scalar step floor), L (calibration-data-matched control), M (code benchmarks and expert-aware baselines), N (inference efficiency and calibration cost).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.