- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper targets post‑training activation quantization for Transformer large‑language models, observing that uniform‑bit activation quantization ignores the differing functional roles inside decoder models, where different internal activation sites exhibit very different sensitivity to quantization noise. The authors propose Role‑Aware Reliability (RAR) allocation framework: it builds a functional risk atlas by individually quantizing each layer‑role activation site and measuring resulting next‑token distribution distortion via KL divergence against clean model outputs. Based on this atlas, the RAR‑log optimization assigns mixed bit‑widths under a fixed cost‑weighted average bit budget, allocating higher precision to high‑risk activation sites. RAR‑log can be combined with SmoothQuant‑style activation smoothing into the main Smooth‑RAR‑log policy; post‑smoothing recalibration further rebuilds risk atlas after smoothing to capture residual quantization risk.
Critical strengths
- This paper directly measures end‑to‑end next‑token distribution distortion caused by quantizing each individual layer‑role site. Empirically the paper demonstrates activation magnitude/outlier metrics are only weak proxies for true downstream prediction damage.
- RAR does not replace SmoothQuant‑style smoothing; instead it addresses residual quantization risk after smoothing mitigates outlier‑driven numerical errors.
- The authors perform extensive evaluations: covering diverse architectures (GPT‑2, OPT, TinyLlama, Pythia, Qwen, Mistral, OLMo‑2), both fake quantization and real packed‑integer activation kernels, combined with W8 weight quantization, long‑context setting, and multiple‑choice downstream tasks.
Critical weaknesses
- While Smooth‑RAR‑log consistently improves over smoothing‑only baseline, unsmoothed vanilla RAR‑log produces severe degradation for Mistral‑7B. The paper attributes this to outlier‑exposed high‑risk sites, but there is no clear diagnostic or mitigation to predict which models can work well without smoothing. It remains unclear what intrinsic architectural properties make a model benefit greatly from role‑aware allocation (e.g. Qwen) versus cases where vanilla RAR‑log fails.
- Post‑smoothing recalibration brings only marginal overall LM improvement despite substantial site‑level risk reranking; it is not fully clear under what scenarios recalibration delivers large practical gains, which limits actionable guidance for practitioners.
- The paper only evaluates models up to 7B; it lacks analysis on how atlas building scales to larger models (e.g. 34B+, 70B). Also, the current pipeline requires model‑specific hook configuration for different Transformer variants, raising engineering integration burden when deploying to diverse existing LLM inference stacks.
Questions for authors
- Regarding the Mistral‑7B failure of unsmoothed RAR‑log: can you identify measurable pre‑calibration model statistics (e.g. role‑wise outlier leverage, activation statistics) that could predict when vanilla RAR‑log will perform poorly without smoothing?
- Post‑smoothing recalibration substantially reranks site‑level risk but delivers only small aggregate LM improvement. Could you elaborate why large shifts in individual‑site risk rankings translate to only modest end‑task gains?
- Your evaluation is capped at 7B‑parameter models. Could you comment on expected scaling behavior of the functional‑risk atlas construction for models ≥30B?
- In the role‑neutralization experiment for Qwen, protecting QK routing sites is critical for good performance. Do you observe consistent patterns on which functional roles are universally high‑risk across diverse architectures, or are high‑risk roles fundamentally highly model‑specific?
- In real‑world inference deployment, KV‑cache quantization is a major bottleneck. The paper focuses on transient layer‑role activations rather than persistent KV cache. How can the RAR risk‑atlas methodology be extended to KV‑cache mixed‑precision quantization?
6 comments
At the submitted six-bit budget, more than fifty assignments change per Qwen checkpoint (Appendix G.4 and Figure 10). The key question is how much risk lies at those changed sites. The table uses residual single-site KL measured at the common four-bit reference precision:
| Checkpoint | Sites whose bits change after recalibration | Residual single-site risk at unchanged-bit sites | Residual risk at sites remaining at 8 bits |
|---|---|---|---|
| Qwen2.5-7B | 55 / 252 | 93.58% | 81.85% |
| Qwen2.5-7B-Instruct | 64 / 252 | 92.87% | 80.59% |
The sites whose bits change account for less than 8% of the measured residual risk. More than 92% lies at sites with unchanged bits, and over 80% at sites that keep the maximum eight bits. The base atlas already makes the important protection decisions; recalibration mostly adjusts the smaller ones. This explains how many assignments can change while overall quality stays similar.
With fewer bits, getting those priorities right matters more. A tighter budget leaves less precision to share among important sites. We test this directly by changing only the atlas used for allocation at six- and five-bit budgets. The ablation uses BF16 weights, eager attention, eight training blocks for risk estimation, and 256-token evaluation blocks: 128 per corpus for the first three models and 64 for Qwen3.5. Positive reductions favor the residual atlas.
| Model | Average bits | Base atlas | Residual atlas | NLL reduction [95% CI] |
|---|---|---|---|---|
| Qwen2.5-7B | 6 | 2.6311 | 2.6341 | −0.0031 [−0.0068, +0.0006] |
| Qwen2.5-7B | 5 | 2.7006 | 2.6918 | +0.0089 [+0.0036, +0.0143] |
| Mistral-7B | 6 | 2.2097 | 2.2086 | +0.0011 [−0.0012, +0.0034] |
| Mistral-7B | 5 | 2.2662 | 2.2551 | +0.0112 [+0.0067, +0.0156] |
| OLMo-2-7B | 6 | 2.5397 | 2.5304 | +0.0093 [+0.0070, +0.0115] |
| OLMo-2-7B | 5 | 2.6551 | 2.6158 | +0.0393 [+0.0321, +0.0470] |
| Qwen3.5-9B | 6 | 2.4226 | 2.4192 | +0.0034 [+0.0001, +0.0071] |
| Qwen3.5-9B | 5 | 2.4832 | 2.4715 | +0.0117 [+0.0041, +0.0189] |
All four models gain more at five bits, and every five-bit improvement has a positive CI. OLMo's gain grows from 0.0093 at six bits to 0.0393 at five bits, and Mistral improves by 0.0112 at five bits. Mistral's and Qwen2.5's six-bit intervals span zero, while their five-bit intervals favor recalibration. The result gives a clear use case: remeasuring risk helps preserve quality as the storage budget tightens. CIs use 5,000 paired resamples of eight-block text clusters, with policies fixed.
The same pattern can affect downstream decisions. In the full six-task harness evaluation, Pythia at five bits improves from 43.26% to 45.52% when switching from the base to the residual atlas: +2.26 percentage points [1.32, 3.14]. Both atlases use FP32 offline measurement and BF16 inference, so this comparison isolates recalibration.
Practical guidance: for a new smoothed deployment, estimate scales first and build one residual atlas. This measures the quantizer that will actually run and requires a single atlas scan. If a base atlas already exists, the budget sweep shows a clear reason to rebuild when targeting tighter compression. Rebuild after changing the quantizer or preprocessing, and select candidate policies on calibration text.
W3 / Q3: Scaling to 30B–70B and reducing integration work
The same risk measurement and allocation procedure improves quality at both 32B and 72B. We also measured its calibration cost. Both checkpoints use BF16 weights, SDPA, exactly six average activation bits, and 64 held-out 256-token blocks from each of WikiText-2 and C4.
| Model | Clean | Smooth uniform 6 | Base-atlas RAR | Residual-atlas RAR | Reduction vs uniform [95% CI] |
|---|---|---|---|---|---|
| Qwen2.5-32B-Instruct | 2.2698 | 2.3020 | 2.2899 | 2.2879 | +0.0142 [+0.0097, +0.0185] |
| Qwen2.5-72B | 2.0619 | 2.1353 | 2.0934 | 2.0885 | +0.0468 [+0.0411, +0.0521] |
At 72B, residual-atlas RAR improves NLL by 0.0468 [0.0411, 0.0521] over smoothing alone and removes 63.8% of its remaining mean degradation. The model has 80 layers and 720 measured sites. Intervals use 5,000 paired bootstrap draws of eight-block text clusters within each corpus.
Atlas construction uses a clean reference prediction and one perturbed forward evaluation per site. For sites and calibration blocks, the serial work is forward evaluations: clean evaluations plus site interventions. Site measurements are independent and can be distributed across workers. With four 256-token training blocks, measured risk-loop times are:
| Checkpoint | Sites | Hardware arrangement | Risk-loop time |
|---|---|---|---|
| Qwen2.5-7B | 252 | One GH200 | 30.7 s |
| Qwen2.5-32B-Instruct | 576 | One GH200 | 138.7 s |
| Qwen2.5-72B | 720 | Seven GH200 workers, site-partitioned model replicas with CPU weight offload | 180.4 s, maximum worker time |
The 72B residual-atlas loop takes 180.0 seconds. The complete base-plus-residual dispatch takes 398.9 seconds, including worker loading and reference computation; scale estimation takes another 29.0 seconds. These are seven-worker wall times with CPU offload. A new smoothed deployment needs only the residual scan. Scaling therefore follows the number of sites and the cost of a model forward pass, with independent site measurements providing a direct route to parallelization.
The model adapter identifies tensors; shared RAR code measures risk and assigns precision. The adapter maps each functional site to its activation tensor and actual width. An automatic module-name/shape mapping audit reproduces the hand-configured maps on 16/16 tested checkpoints, including standard Qwen, Mistral, OLMo, Llama, TinyLlama, and hybrid Qwen3.5/3.8 architectures. The hybrid models use an architecture-specific boundary profile. Once those sites are mapped, the same atlas construction and allocation rule select precision automatically.
The calibration output is a static precision/scale map. In the packed KV implementation, it selects a compiled kernel variant per layer, so calibration hooks add no decoding-time work. The architecture-specific boundary profiles, site counts, and module-name/shape mapping audit are documented in Appendix D.12.
Q4: A shared allocation principle across architectures
Across architectures, prediction risk concentrates in a small set of roles, and the same allocation rule protects them. In every checkpoint below, the two leading roles account for more than half of the summed single-site risk. RAR identifies these concentrations with one output-KL criterion and allocates precision with one cost-aware rule. The four-bit base-atlas results (Section 5; Appendix G.5, Table 43) show this shared pattern:
| Checkpoint | Leading roles and share of summed single-site KL risk |
|---|---|
| GPT-2 | Pre-attention normalization: 31.1%; MLP hidden: 20.1% |
| OPT-125M | Pre-MLP normalization: 69.7%; MLP hidden: 17.0% |
| Pythia-6.9B | MLP output: 34.2%; pre-MLP normalization: 25.8% |
| Qwen2.5-7B | Q/K routing: 48.0%; MLP output: 18.9% |
| Mistral-7B | MLP output: 31.2%; pre-attention normalization: 28.9% |
| OLMo-2-7B | MLP output: 47.0%; attention output: 26.6% |
The common explanation is that risk reflects both the rounding error at a site and how that error changes the final prediction. The submitted diagnostics show that Qwen's early Q/K sites remain sensitive after smoothing reduces their numerical error, whereas Mistral's leading normalization and MLP risks fall strongly with smoothing. RAR measures both effects through the same final-output criterion. The leading role can change with the checkpoint and quantizer; the procedure for identifying and protecting it stays the same.
Matched-budget controls demonstrate the value of this shared rule. Under identical smoothing, boundaries, and exact six-bit budgets, protecting every Q/K site at eight bits gives NLL 2.5805/2.2633/2.8029 on Qwen/Mistral/OLMo; residual-atlas RAR achieves 2.5115/2.2084/2.5326. Protecting normalization sites instead also trails RAR on all three. The atlas therefore turns a consistent principle—protect the computations whose errors matter most per unit cost—into effective assignments across architectures, without manually choosing a preferred role for each model.
Q5: Extension to persistent KV-cache mixed precision
RAR applies the same allocation rule to persistent KV caches. Each layer contributes separate key-cache and value-cache sites: K determines which stored tokens attention reads, and V supplies their information. We quantize one cache site, measure the final next-token KL, and allocate K/V precision under a persistent-byte budget:
Here, is the number of cached tokens; are the actual K/V widths, including grouped-query attention; are their assigned bits; and counts scale metadata. Dividing by eight converts bits to bytes. All layer caches remain live together, so this sum directly measures persistent storage for one sequence. Batch storage scales with the number of resident sequences.
Thank you for highlighting the distinction between numerical error and prediction damage. One principle connects our results: reduce quantization error, then allocate precision according to its effect on predictions. A training-selected intervention suppresses Mistral's loss spikes with every bit assignment fixed; an atlas-only ablation shows when more accurate risk measurements improve allocation. The same procedure improves quality at 72B and extends to a persistent KV implementation with measured memory and decoding benefits. NLL is in nats/token; lower is better. Both base-atlas and residual-atlas RAR apply smoothing: the former reuses risks measured before smoothing, while the latter measures the remaining risk after smoothing. Brackets report paired 95% CIs.
W1 / Q1: Mistral: mechanism, early diagnosis, and remedy
Mistral provides a controlled explanation for combining smoothing with RAR. RAR already assigns eight bits to the site identified by the training diagnostic. Smoothing that site sharply reduces its quantization error and suppresses the held-out loss spikes, with every bit assignment fixed. This isolates the value of reducing numerical error at a site that already receives high precision.
In the submitted diagnostic (Section 4.2; Appendix E.4), raw RAR improves 62 of the first 64 WikiText-2 blocks; two spikes reverse the mean. We investigate those spikes with training-only diagnosis and a fixed-bit intervention.
1. Screen for outliers before building the atlas.
A clean BF16 pass on 32 training blocks across eleven checkpoints gives Mistral the largest MLP-hidden max-absolute-value/RMS ratio: 341.5, versus 27.6–264.9 for the other ten. This ratio measures how extreme the largest activation is relative to the typical activation size at that role. Its largest max/RMS ratio across all sites, 784.1, is inside the other models' range (415.4–1,524.4). Measuring outliers by role therefore exposes a warning sign hidden by the whole-model maximum. The next check tests whether those outliers harm predictions under the chosen policy.
2. Test the complete allocation before held-out evaluation.
Quantizing all sites together reveals accumulated errors and rare loss spikes. A 64-block training pilot, disjoint from scale and atlas estimation, gives the following native-FP16/eager results:
3. Isolate the remedy by holding every bit fixed.
On the worst training-pilot block, we temporarily restore each of Mistral's 288 sites to native precision and select the site with the largest loss reduction:
L01.mlp_out, already assigned 8 bits. We freeze this training-selected site and smooth only that site for evaluation. Every RAR bit assignment stays fixed at an exact six-bit average.Evaluation uses all 128 held-out 256-token blocks per corpus with native FP16 weights and eager attention, matching the failure diagnostic. Smoothing the selected site reduces its relative quantization-error energy by 111.6×. Held-out NLL improves by 0.1276 [0.0066, 0.2834] on WikiText-2 and 0.0934 [0.0098, 0.2176] on C4 (paired evaluation-block 95% intervals). The largest excess loss falls from 5.7339 to 0.6050. Full smoothing improves both mean and tail loss further.
Both components make a measurable contribution. The fixed-bit intervention shows how smoothing improves a site already protected by RAR. In the separate matched six-bit allocator comparison (BF16 weights, SDPA, 128 held-out 256-token blocks per corpus), Mistral's mean NLL improves from 2.2430 for smooth uniform to 2.2084 for residual-atlas RAR: precision placement adds quality after smoothing. Together, these controls explain the benefit of Smooth-RAR and supply a practical diagnostic. Use Smooth-RAR by default; when considering an unsmoothed policy, screen role-wise outliers and check its mean and worst-block loss on separate training text before held-out evaluation.
W2 / Q2: Why reranking can leave quality stable, and when recalibration helps
Recalibration helps when it changes the bits at sites that still need protection. If the important sites already receive eight bits, changing the order of lower-risk sites can alter many assignments with little effect on quality. A rank change can also leave the assigned integer bit-width unchanged.