acceptodds
Under review as a conference paper at ICLR 2027

Role-Aware Quantization for Reliable Transformer Inference

Abstract

Quantization is widely used to reduce Transformer memory usage and accelerate inference. Prior work has commonly used uniform activation quantization, assigning the same bit-width across activations. However, Transformer components perform distinct functions, including query–key routing, value transport, feed-forward computation, and normalization. We show that these functional roles differ in sensitivity to quantization error and therefore require different precision levels to preserve predictions. To allocate precision efficiently under a fixed budget, we introduce Role-Aware Reliability (RAR) allocation. RAR quantizes one layer–role activation site at a time and measures the resulting change in the next-token distribution using Kullback–Leibler (KL) divergence. The resulting functional risk atlas pinpoints where low precision most strongly distorts predictions. Building the functional risk atlas for a 7B checkpoint requires only a small sample of calibration text and approximately one to two minutes. Our allocation rule, RAR-log, uses the atlas to assign higher precision to high-risk sites and lower precision to low-risk sites under a fixed cost-weighted average bit-width budget. By accounting for how quantization errors in different roles affect predictions, RAR complements SmoothQuant-style activation smoothing. Our theoretical analysis explains how quantization errors from different sites combine and proves bounds on the resulting changes in model predictions. With the same target precision budget, the combined policy, Smooth-RAR-log, removes an average of 94.1% of the language-modeling degradation caused by uniform 6-bit activations, compared with 87.4% for smoothing alone. The combined policy improves on smoothing alone in all 24 paired model–corpus evaluations, covering three corpora and eight Transformer checkpoints ranging from 124M to 7B parameters. Even with packed activations and 8-bit quantized weights, Smooth-RAR-log lowers next-token prediction loss compared with smoothing alone across the evaluated models. Smooth-RAR-log also improves average accuracy on multiple-choice benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Model Compression, Post-Training Quantization, Activation Quantization, Mixed-Precision Quantization, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qa1vOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper targets post‑training activation quantization for Transformer large‑language models, observing that uniform‑bit activation quantization ignores the differing functional roles inside decoder models, where different internal activation sites exhibit very different sensitivity to quantization noise. The authors propose Role‑Aware Reliability (RAR) allocation framework: it builds a functional risk atlas by individually quantizing each layer‑role activation site and measuring resulting next‑token distribution distortion via KL divergence against clean model outputs. Based on this atlas, the RAR‑log optimization assigns mixed bit‑widths under a fixed cost‑weighted average bit budget, allocating higher precision to high‑risk activation sites. RAR‑log can be combined with SmoothQuant‑style activation smoothing into the main Smooth‑RAR‑log policy; post‑smoothing recalibration further rebuilds risk atlas after smoothing to capture residual quantization risk.

Critical strengths
  1. This paper directly measures end‑to‑end next‑token distribution distortion caused by quantizing each individual layer‑role site. Empirically the paper demonstrates activation magnitude/outlier metrics are only weak proxies for true downstream prediction damage.
  2. RAR does not replace SmoothQuant‑style smoothing; instead it addresses residual quantization risk after smoothing mitigates outlier‑driven numerical errors.
  3. The authors perform extensive evaluations: covering diverse architectures (GPT‑2, OPT, TinyLlama, Pythia, Qwen, Mistral, OLMo‑2), both fake quantization and real packed‑integer activation kernels, combined with W8 weight quantization, long‑context setting, and multiple‑choice downstream tasks.
Critical weaknesses
  1. While Smooth‑RAR‑log consistently improves over smoothing‑only baseline, unsmoothed vanilla RAR‑log produces severe degradation for Mistral‑7B. The paper attributes this to outlier‑exposed high‑risk sites, but there is no clear diagnostic or mitigation to predict which models can work well without smoothing. It remains unclear what intrinsic architectural properties make a model benefit greatly from role‑aware allocation (e.g. Qwen) versus cases where vanilla RAR‑log fails.
  2. Post‑smoothing recalibration brings only marginal overall LM improvement despite substantial site‑level risk reranking; it is not fully clear under what scenarios recalibration delivers large practical gains, which limits actionable guidance for practitioners.
  3. The paper only evaluates models up to 7B; it lacks analysis on how atlas building scales to larger models (e.g. 34B+, 70B). Also, the current pipeline requires model‑specific hook configuration for different Transformer variants, raising engineering integration burden when deploying to diverse existing LLM inference stacks.
Questions for authors
  1. Regarding the Mistral‑7B failure of unsmoothed RAR‑log: can you identify measurable pre‑calibration model statistics (e.g. role‑wise outlier leverage, activation statistics) that could predict when vanilla RAR‑log will perform poorly without smoothing?
  2. Post‑smoothing recalibration substantially reranks site‑level risk but delivers only small aggregate LM improvement. Could you elaborate why large shifts in individual‑site risk rankings translate to only modest end‑task gains?
  3. Your evaluation is capped at 7B‑parameter models. Could you comment on expected scaling behavior of the functional‑risk atlas construction for models ≥30B?
  4. In the role‑neutralization experiment for Qwen, protecting QK routing sites is critical for good performance. Do you observe consistent patterns on which functional roles are universally high‑risk across diverse architectures, or are high‑risk roles fundamentally highly model‑specific?
  5. In real‑world inference deployment, KV‑cache quantization is a major bottleneck. The paper focuses on transient layer‑role activations rather than persistent KV cache. How can the RAR risk‑atlas methodology be extended to KV‑cache mixed‑precision quantization?
6 comments

Thank you for highlighting the distinction between numerical error and prediction damage. One principle connects our results: reduce quantization error, then allocate precision according to its effect on predictions. A training-selected intervention suppresses Mistral's loss spikes with every bit assignment fixed; an atlas-only ablation shows when more accurate risk measurements improve allocation. The same procedure improves quality at 72B and extends to a persistent KV implementation with measured memory and decoding benefits. NLL is in nats/token; lower is better. Both base-atlas and residual-atlas RAR apply smoothing: the former reuses risks measured before smoothing, while the latter measures the remaining risk after smoothing. Brackets report paired 95% CIs.

W1 / Q1: Mistral: mechanism, early diagnosis, and remedy

Mistral provides a controlled explanation for combining smoothing with RAR. RAR already assigns eight bits to the site identified by the training diagnostic. Smoothing that site sharply reduces its quantization error and suppresses the held-out loss spikes, with every bit assignment fixed. This isolates the value of reducing numerical error at a site that already receives high precision.

In the submitted diagnostic (Section 4.2; Appendix E.4), raw RAR improves 62 of the first 64 WikiText-2 blocks; two spikes reverse the mean. We investigate those spikes with training-only diagnosis and a fixed-bit intervention.

1. Screen for outliers before building the atlas.

A clean BF16 pass on 32 training blocks across eleven checkpoints gives Mistral the largest MLP-hidden max-absolute-value/RMS ratio: 341.5, versus 27.6–264.9 for the other ten. This ratio measures how extreme the largest activation is relative to the typical activation size at that role. Its largest max/RMS ratio across all sites, 784.1, is inside the other models' range (415.4–1,524.4). Measuring outliers by role therefore exposes a warning sign hidden by the whole-model maximum. The next check tests whether those outliers harm predictions under the chosen policy.

2. Test the complete allocation before held-out evaluation.

Quantizing all sites together reveals accumulated errors and rare loss spikes. A 64-block training pilot, disjoint from scale and atlas estimation, gives the following native-FP16/eager results:

Mistral policyMean excess NLLMaximum block excess NLL
Uniform 6, unsmoothed0.20500.4165
RAR, unsmoothed0.24315.3471
Smooth uniform 60.05690.1097
Same RAR bits + smoothing0.02050.0694

3. Isolate the remedy by holding every bit fixed.

On the worst training-pilot block, we temporarily restore each of Mistral's 288 sites to native precision and select the site with the largest loss reduction: L01.mlp_out, already assigned 8 bits. We freeze this training-selected site and smooth only that site for evaluation. Every RAR bit assignment stays fixed at an exact six-bit average.

Policy, identical RAR bit assignmentsWikiText-2 NLLC4 NLLMaximum excess NLL across both corpora
Clean reference2.06272.31290
Unsmoothed RAR2.28012.47025.7339
Smooth only the training-selected site2.15252.37680.6050
Smooth all sites2.08432.33390.2505

Evaluation uses all 128 held-out 256-token blocks per corpus with native FP16 weights and eager attention, matching the failure diagnostic. Smoothing the selected site reduces its relative quantization-error energy by 111.6×. Held-out NLL improves by 0.1276 [0.0066, 0.2834] on WikiText-2 and 0.0934 [0.0098, 0.2176] on C4 (paired evaluation-block 95% intervals). The largest excess loss falls from 5.7339 to 0.6050. Full smoothing improves both mean and tail loss further.

Both components make a measurable contribution. The fixed-bit intervention shows how smoothing improves a site already protected by RAR. In the separate matched six-bit allocator comparison (BF16 weights, SDPA, 128 held-out 256-token blocks per corpus), Mistral's mean NLL improves from 2.2430 for smooth uniform to 2.2084 for residual-atlas RAR: precision placement adds quality after smoothing. Together, these controls explain the benefit of Smooth-RAR and supply a practical diagnostic. Use Smooth-RAR by default; when considering an unsmoothed policy, screen role-wise outliers and check its mean and worst-block loss on separate training text before held-out evaluation.

W2 / Q2: Why reranking can leave quality stable, and when recalibration helps

Recalibration helps when it changes the bits at sites that still need protection. If the important sites already receive eight bits, changing the order of lower-risk sites can alter many assignments with little effect on quality. A rank change can also leave the assigned integer bit-width unchanged.

At the submitted six-bit budget, more than fifty assignments change per Qwen checkpoint (Appendix G.4 and Figure 10). The key question is how much risk lies at those changed sites. The table uses residual single-site KL measured at the common four-bit reference precision:

CheckpointSites whose bits change after recalibrationResidual single-site risk at unchanged-bit sitesResidual risk at sites remaining at 8 bits
Qwen2.5-7B55 / 25293.58%81.85%
Qwen2.5-7B-Instruct64 / 25292.87%80.59%

The sites whose bits change account for less than 8% of the measured residual risk. More than 92% lies at sites with unchanged bits, and over 80% at sites that keep the maximum eight bits. The base atlas already makes the important protection decisions; recalibration mostly adjusts the smaller ones. This explains how many assignments can change while overall quality stays similar.

With fewer bits, getting those priorities right matters more. A tighter budget leaves less precision to share among important sites. We test this directly by changing only the atlas used for allocation at six- and five-bit budgets. The ablation uses BF16 weights, eager attention, eight training blocks for risk estimation, and 256-token evaluation blocks: 128 per corpus for the first three models and 64 for Qwen3.5. Positive reductions favor the residual atlas.

ModelAverage bitsBase atlasResidual atlasNLL reduction [95% CI]
Qwen2.5-7B62.63112.6341−0.0031 [−0.0068, +0.0006]
Qwen2.5-7B52.70062.6918+0.0089 [+0.0036, +0.0143]
Mistral-7B62.20972.2086+0.0011 [−0.0012, +0.0034]
Mistral-7B52.26622.2551+0.0112 [+0.0067, +0.0156]
OLMo-2-7B62.53972.5304+0.0093 [+0.0070, +0.0115]
OLMo-2-7B52.65512.6158+0.0393 [+0.0321, +0.0470]
Qwen3.5-9B62.42262.4192+0.0034 [+0.0001, +0.0071]
Qwen3.5-9B52.48322.4715+0.0117 [+0.0041, +0.0189]

All four models gain more at five bits, and every five-bit improvement has a positive CI. OLMo's gain grows from 0.0093 at six bits to 0.0393 at five bits, and Mistral improves by 0.0112 at five bits. Mistral's and Qwen2.5's six-bit intervals span zero, while their five-bit intervals favor recalibration. The result gives a clear use case: remeasuring risk helps preserve quality as the storage budget tightens. CIs use 5,000 paired resamples of eight-block text clusters, with policies fixed.

The same pattern can affect downstream decisions. In the full six-task harness evaluation, Pythia at five bits improves from 43.26% to 45.52% when switching from the base to the residual atlas: +2.26 percentage points [1.32, 3.14]. Both atlases use FP32 offline measurement and BF16 inference, so this comparison isolates recalibration.

Practical guidance: for a new smoothed deployment, estimate scales first and build one residual atlas. This measures the quantizer that will actually run and requires a single atlas scan. If a base atlas already exists, the budget sweep shows a clear reason to rebuild when targeting tighter compression. Rebuild after changing the quantizer or preprocessing, and select candidate policies on calibration text.

W3 / Q3: Scaling to 30B–70B and reducing integration work

The same risk measurement and allocation procedure improves quality at both 32B and 72B. We also measured its calibration cost. Both checkpoints use BF16 weights, SDPA, exactly six average activation bits, and 64 held-out 256-token blocks from each of WikiText-2 and C4.

ModelCleanSmooth uniform 6Base-atlas RARResidual-atlas RARReduction vs uniform [95% CI]
Qwen2.5-32B-Instruct2.26982.30202.28992.2879+0.0142 [+0.0097, +0.0185]
Qwen2.5-72B2.06192.13532.09342.0885+0.0468 [+0.0411, +0.0521]

At 72B, residual-atlas RAR improves NLL by 0.0468 [0.0411, 0.0521] over smoothing alone and removes 63.8% of its remaining mean degradation. The model has 80 layers and 720 measured sites. Intervals use 5,000 paired bootstrap draws of eight-block text clusters within each corpus.

Atlas construction uses a clean reference prediction and one perturbed forward evaluation per site. For SS sites and RR calibration blocks, the serial work is R(S+1)R(S+1) forward evaluations: RR clean evaluations plus RSRS site interventions. Site measurements are independent and can be distributed across workers. With four 256-token training blocks, measured risk-loop times are:

CheckpointSitesHardware arrangementRisk-loop time
Qwen2.5-7B252One GH20030.7 s
Qwen2.5-32B-Instruct576One GH200138.7 s
Qwen2.5-72B720Seven GH200 workers, site-partitioned model replicas with CPU weight offload180.4 s, maximum worker time

The 72B residual-atlas loop takes 180.0 seconds. The complete base-plus-residual dispatch takes 398.9 seconds, including worker loading and reference computation; scale estimation takes another 29.0 seconds. These are seven-worker wall times with CPU offload. A new smoothed deployment needs only the residual scan. Scaling therefore follows the number of sites and the cost of a model forward pass, with independent site measurements providing a direct route to parallelization.

The model adapter identifies tensors; shared RAR code measures risk and assigns precision. The adapter maps each functional site to its activation tensor and actual width. An automatic module-name/shape mapping audit reproduces the hand-configured maps on 16/16 tested checkpoints, including standard Qwen, Mistral, OLMo, Llama, TinyLlama, and hybrid Qwen3.5/3.8 architectures. The hybrid models use an architecture-specific boundary profile. Once those sites are mapped, the same atlas construction and allocation rule select precision automatically.

The calibration output is a static precision/scale map. In the packed KV implementation, it selects a compiled kernel variant per layer, so calibration hooks add no decoding-time work. The architecture-specific boundary profiles, site counts, and module-name/shape mapping audit are documented in Appendix D.12.

Q4: A shared allocation principle across architectures

Across architectures, prediction risk concentrates in a small set of roles, and the same allocation rule protects them. In every checkpoint below, the two leading roles account for more than half of the summed single-site risk. RAR identifies these concentrations with one output-KL criterion and allocates precision with one cost-aware rule. The four-bit base-atlas results (Section 5; Appendix G.5, Table 43) show this shared pattern:

CheckpointLeading roles and share of summed single-site KL risk
GPT-2Pre-attention normalization: 31.1%; MLP hidden: 20.1%
OPT-125MPre-MLP normalization: 69.7%; MLP hidden: 17.0%
Pythia-6.9BMLP output: 34.2%; pre-MLP normalization: 25.8%
Qwen2.5-7BQ/K routing: 48.0%; MLP output: 18.9%
Mistral-7BMLP output: 31.2%; pre-attention normalization: 28.9%
OLMo-2-7BMLP output: 47.0%; attention output: 26.6%

The common explanation is that risk reflects both the rounding error at a site and how that error changes the final prediction. The submitted diagnostics show that Qwen's early Q/K sites remain sensitive after smoothing reduces their numerical error, whereas Mistral's leading normalization and MLP risks fall strongly with smoothing. RAR measures both effects through the same final-output criterion. The leading role can change with the checkpoint and quantizer; the procedure for identifying and protecting it stays the same.

Matched-budget controls demonstrate the value of this shared rule. Under identical smoothing, boundaries, and exact six-bit budgets, protecting every Q/K site at eight bits gives NLL 2.5805/2.2633/2.8029 on Qwen/Mistral/OLMo; residual-atlas RAR achieves 2.5115/2.2084/2.5326. Protecting normalization sites instead also trails RAR on all three. The atlas therefore turns a consistent principle—protect the computations whose errors matter most per unit cost—into effective assignments across architectures, without manually choosing a preferred role for each model.

Q5: Extension to persistent KV-cache mixed precision

RAR applies the same allocation rule to persistent KV caches. Each layer contributes separate key-cache and value-cache sites: K determines which stored tokens attention reads, and V supplies their information. We quantize one cache site, measure the final next-token KL, and allocate K/V precision under a persistent-byte budget:

BKV=∑ℓT(dK,ℓbK,ℓ+dV,ℓbV,ℓ)8+Bscales.B_{\mathrm{KV}} = \sum_{\ell}\frac{T\left(d_{K,\ell}b_{K,\ell} + d_{V,\ell}b_{V,\ell}\right)}{8} + B_{\mathrm{scales}}.

Here, TT is the number of cached tokens; dK,dVd_K,d_V are the actual K/V widths, including grouped-query attention; bK,bVb_K,b_V are their assigned bits; and BscalesB_{\mathrm{scales}} counts scale metadata. Dividing by eight converts bits to bytes. All layer caches remain live together, so this sum directly measures persistent storage for one sequence. Batch storage scales with the number of resident sequences.

Reviewer_tw2iOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Role-Aware Reliability (RAR), a mixed-precision activation quantization framework for Transformer inference. RAR measures the sensitivity of each layer-role activation site using the KL divergence between clean and quantized next-token distributions, then allocates precision under a fixed bit-width budget. Combined with activation smoothing, the method improves language modeling quality over uniform quantization and smoothing alone across eight Transformer checkpoints.

Critical strengths
  1. Effective sensitivity-guided precision allocation. The proposed method provides a simple framework for allocating activation precision based on its impact on model predictions. The consistent improvements over uniform precision and smoothing alone support its effectiveness.

  2. Comprehensive empirical analysis. The paper evaluates multiple model architectures and downstream tasks, with detailed ablations, theoretical analysis, and failure-case studies that strengthen the empirical findings.

Critical weaknesses
  1. Limited algorithmic novelty. The core approach largely follows established sensitivity-aware mixed-precision quantization: measuring site-level sensitivity and allocating bit-widths accordingly. Using output KL divergence and grouping activation sites by functional role does not appear to introduce a fundamentally new allocation mechanism. Moreover, the strong performance of layer-only allocation raises questions about the necessity of explicit role awareness.

  2. Insufficient evidence of practical inference efficiency. Although the paper evaluates packed activations and quantized weights, it does not demonstrate end-to-end inference acceleration or memory improvements in a realistic serving system. The cost-weighted average bit-width does not fully capture the execution overhead of heterogeneous precision, including packing, dequantization, and kernel compatibility. Consequently, the practical benefits remain unclear.

Questions for authors
  1. What fundamentally distinguishes RAR from existing sensitivity-based mixed-precision allocation methods beyond the choice of sensitivity metric and activation granularity? Can the authors provide a direct comparison with stronger sensitivity-aware allocation baselines under identical budgets?

  2. Can the authors quantify the actual end-to-end latency, throughput, and memory benefits of RAR compared with uniform precision and smoothing-based baselines? How does heterogeneous bit-width execution affect these benefits?

  3. Given the competitive performance of layer-only allocation, how much improvement can be attributed specifically to functional-role awareness rather than general layer-wise sensitivity estimation?

5 comments

Thank you for recognizing the effectiveness and breadth of the study. RAR makes precise activation placement practical: one forward-only measurement per site supports multiple budgets, and controlled experiments show why those bits must reach the right computations. We isolate this benefit with matched KL, Hessian, and Fisher criteria, fixed layer budgets, and stronger quantizers. The results connect the method's contribution to three practical gains: inexpensive calibration, better quality at the same storage budget, and greater inference capacity with faster KV-cache decoding. Unless stated otherwise, NLL is in nats/token, averaged over the WikiText-2 and C4 corpus means; lower is better. Brackets report paired 95% CIs.

W1 / Q1: What distinguishes RAR from existing sensitivity-based allocation?

RAR turns measured prediction damage at individual activation sites into an efficient, reusable precision policy. The same rule applies across models: measure the effect of rounding, account for each site's storage cost, and allocate bits under the requested budget. Three parts make this contribution concrete:

  1. Measure what rounding does to the final prediction. We quantize one attention, MLP, or normalization site at a time and measure the change in the next-token distribution. The measurement captures the rounding error produced by the chosen quantizer and its propagation through later computation. Measuring again after smoothing identifies the computations that still need precision. HAWQ-V2 (https://arxiv.org/abs/1911.03852) uses curvature for activation allocation, and APTQ (https://arxiv.org/abs/2402.14866) uses attention-output reconstruction and Hessian information for weight allocation. RAR obtains its activation signal through forward evaluations. RippleKV (https://arxiv.org/abs/2608.08684) also measures output KL, using noise in the value cache to allocate retained-token counts across layers; RAR measures actual activation rounding to allocate bits across attention, MLP, and normalization sites.

  2. Measure at one precision; allocate at multiple budgets. RAR measures each site at a common reference precision, normally four bits, and uses the measured damage and tensor width to choose its bit-width under the requested budget. ZeroQ (https://arxiv.org/abs/2001.00281) collects KL costs at candidate layer precisions; TWLA's ILA-AMP (https://arxiv.org/abs/2606.13054) measures layer-level activation NLL costs and adjacent-layer interactions. GAMMA (https://arxiv.org/abs/2605.18475) learns reusable weight-precision preferences through hidden-state reconstruction. RAR reuses its measured activation atlas across budgets, with no gradient computation or learned allocation parameters. The comparison below quantifies the quality retained using one reference precision per site instead of six.

  3. Verify that the right computation receives the bits. Dewage et al. (https://arxiv.org/abs/2609.26173) characterize sensitivity differences among attention-projection weights. RAR turns measured activation sensitivity across the full block into a budgeted policy, then tests the decision itself. We hold the estimator and solver fixed while changing the allocation granularity; separately, we keep every layer's budget and bit histogram fixed while shuffling assignments. The gains survive both controls. They also persist with KL, Hessian, and Fisher criteria, establishing within-layer placement as a useful allocation principle across sensitivity estimators.

Controlled comparison of allocation criteria. We adapt the established criteria to the same activation problem: identical boundaries, smoothing scales, calibration text, candidate bits 3–8, exact integer solver, and 6.000 average bits. This design isolates the information used to place precision. The Hessian control uses actual activation NLL Hessian-vector products (eight probes on each of four training blocks), separately from the Fisher controls. The KL-curve controls measure either a whole layer jointly or every site at all six candidate precisions. Evaluation uses BF16 weights, SDPA, and 128 held-out 256-token blocks from each of WikiText-2 and C4; scales use 32 training blocks.

PolicyQwen2.5-7BMistral-7BOLMo-2-7B
Unquantized reference2.49842.18792.5141
Smooth uniform 62.99422.24302.5755
Direct whole-layer KL curves2.52482.21462.5706
HAWQ-V2-style Hessian trace × error2.51152.22022.5695
Fisher trace × error2.51212.20812.5312
Diagonal Fisher2.51052.21522.5381
Direct site-KL curves, six reference precisions/site2.51042.20782.5310
Smooth-RAR-log, reused base atlas2.51402.20982.5384
Smooth-Recalib-RAR-log, one reference precision/site2.51152.20842.5326

One reference precision retains nearly all the quality of the six-precision scan. The residual-atlas RAR policy is within 0.0017 nat/token of the six-point site-KL policy on every model, using one sixth as many site–precision measurements. Paired text-cluster CIs for these small NLL differences span zero. RAR also matches the rounded Hessian result on Qwen and improves on it by 0.0118 [0.0089, 0.0148] on Mistral and 0.0368 [0.0330, 0.0406] on OLMo. The base atlas is measured before smoothing and reused; the residual atlas is measured after smoothing. These intervals use 5,000 paired resamples of eight-block clusters within each corpus, with the policies fixed.

The calibration saving is substantial. On the same eager backend, the sensitivity pass takes 30.1/42.6/41.5 seconds for RAR versus 392.5/540.1/632.7 seconds for the Hessian control on Qwen/Mistral/OLMo: 12.7–15.2× faster, with no backward computation. Both exclude loading, scale estimation, and solving.

Separate A100 profiles directly compare the one- and six-reference scans using the same implementation, eight training blocks, and frozen smoothing scales: 49.6/60.7/67.6 seconds versus 298.2/363.9/404.4 seconds on Qwen/Mistral/OLMo, a measured 5.98–6.01× reduction in calibration time. These are one timing run per method and model, excluding loading, input transfer, shared scale estimation, and allocation. Together, the quality and cost results show why RAR's economical measurement matters: detailed activation placement remains affordable without measuring a full bit-response curve at every site.

The theory connects finite activation errors to prediction reliability and explains when allocation pays off. CASA (https://arxiv.org/abs/2609.25916) derives a continuous allocation rule for a Hessian-based weight-error model. RAR develops this connection for activation rounding and final next-token predictions: Appendix C analyzes how local errors propagate and interact, and bounds joint output distortion when sites are quantized in execution order under the stated downstream-sensitivity assumptions. It also relates output KL to the margin required to change the top prediction.

For allocation, the additive risk model predicts that one extra bit reduces a site's distortion by a factor of four. Proposition 1 gives the optimal continuous allocation for that model:

mc∗=clip⁡[mmin⁡,mmax⁡](τ+12log⁡2(Acκc)).m_c^{*} = \operatorname{clip}_{[m_{\min},m_{\max}]}\left(\tau + \frac{1}{2}\log_2\left(\frac{A_c}{\kappa_c}\right)\right).

Here, mc∗m_c^{*} is the proposed precision at site cc; AcA_c is its measured prediction damage at the common reference precision; κc\kappa_c is its activation width, hence the cost of one extra bit; and τ\tau shifts the assignments to meet the budget. clip enforces the supported range. A site with four times the damage per unit cost receives one additional bit before clipping. For example, a narrow routing tensor can deserve more bits than a wide MLP tensor. The experiments use the same integer solver for all criteria to isolate their allocation decisions.

Appendix C.3 quantifies the gain over uniform precision through the spread of damage per unit cost in the continuous additive model. The same principle explains Smooth-RAR: reduce rounding error, then spend precision where the remaining error changes predictions most. Smoothing acts on the error; the atlas measures its effect on the final prediction. The following controlled ablations test the resulting assignments with all sites quantized together.

W1 / Q3: How much gain comes specifically from functional-role placement?

Resolving precision within a layer improves quality even when the sensitivity estimator, optimizer, and budget are fixed. Three controlled ablations establish this contribution: allow separate precisions using the same measurements, preserve each layer's spending while shuffling its assignments, and repeat the comparison with a stronger quantizer. Together they test whether the right computations receive the available bits.

The submitted 95.5%/96.8% recovery result corresponds to absolute NLL 2.6080 versus 2.5920 (clean: 2.5389): RAR removes 23.2% of the layer-score policy's remaining degradation. That baseline averages sensitivity within a layer but still permits different bits through site-specific costs. The new ablations sharpen the comparison by enforcing one bit-width per layer or fixing each layer's complete bit histogram.

Across 16 checkpoints' four-bit base atlases, 54–95% of the variation in the allocation score occurs within layers. This is the activation-width-weighted variance of log⁡2(risk/width)\log_2(\text{risk}/\text{width}), the score that determines relative precision. A layer average discards that variation. The experiments below test the prediction benefit of retaining it.

1. Does role-level allocation help when the sensitivity measurements are identical?

One policy assigns a single precision per layer; the other assigns precision independently to layer–role sites. We hold the sensitivity measurements, candidate bits, exact budget, and solver fixed, changing only the permitted allocation granularity. The table reports the mean over Qwen2.5-7B, Mistral-7B, and OLMo-2-7B.

Fixed sensitivity criterionBudgetLayer-tied NLLLayer–role NLLReduction
Single-reference KL62.44012.41750.0226
Six-point KL curve62.43752.41640.0211
Hessian trace × error62.45682.43370.0231
Fisher trace × error62.46752.41710.0504
Single-reference KL52.60762.47470.1329
Six-point KL curve52.65012.46880.1814
Hessian trace × error52.65792.56840.0895
Fisher trace × error52.84122.49700.3442

All 24 per-model comparisons improve, and every paired 95% CI excludes zero (5,000 resamples of eight-block text clusters, with policies fixed). Because the estimator and optimizer are unchanged within each comparison, the improvement isolates the value of allowing different precisions within a layer. It persists across KL, Hessian, and Fisher criteria; the mean reduction grows at the tighter five-bit budget for every criterion. Evaluation measures final-model NLL on held-out text with all sites quantized together.

We repeated the single-reference six-bit comparison with three four-block calibration subsamples, using seeds 11, 23, and 47. Scales and held-out evaluation text stay fixed. The following values are mean ± sample SD across those subsamples; the last column is computed from the paired reduction in each subsample.

ModelLayer-tied NLLLayer–role RAR NLLPaired NLL reduction
Qwen2.5-7B2.5251 ± 0.00022.5111 ± 0.00050.0140 ± 0.0006
Mistral-7B2.2230 ± 0.00102.2081 ± 0.00070.0149 ± 0.0015
OLMo-2-7B2.5675 ± 0.00302.5316 ± 0.00110.0359 ± 0.0039

All nine model/subsample comparisons improve. Each mean reduction exceeds nine times its sample SD across calibration subsamples.

2. Does the choice of site matter when each layer spends exactly the same bits?

Five fixed permutations exchange assignments among equal-width sites within the same layer. They preserve every layer's budget and bit histogram, as well as the global budget. Only the sites receiving those bits change. We evaluate on a reserved suffix of WikiText-2 and C4: 128 additional 256-token blocks per corpus, BF16 weights, eager attention, and an exact six-bit budget.

ModelWhole-layer KLRARShuffled placement, mean ± SDReduction vs shuffle mean [95% CI]
Qwen2.5-7B2.77312.76192.7674 ± 0.0042+0.0055 [+0.0035, +0.0075]
Mistral-7B2.26062.25512.2905 ± 0.0034+0.0354 [+0.0329, +0.0380]
OLMo-2-7B2.63852.60312.6782 ± 0.0121+0.0751 [+0.0698, +0.0804]
Qwen3.5-9B2.64742.63522.6410 ± 0.0008+0.0058 [+0.0042, +0.0074]
Qwen3.8-27B2.51292.50872.5093 ± 0.0002+0.0006 [−0.0011, +0.0025]
Qwen2.5-32B2.48602.47912.4903 ± 0.0007+0.0112 [+0.0092, +0.0131]

RAR has lower mean NLL than all 30 shuffled policies, with positive CIs against the shuffle mean on five models; Qwen3.8's CI includes zero. All six whole-layer comparisons have positive CIs. The shuffle control attributes the gain to which sites receive precision while preserving each layer's spending and mix of bit-widths. SD measures variation across the five shuffles; CIs measure evaluation-text uncertainty using 5,000 paired resamples of eight-block clusters.

3. Does placement still help after improving the quantizer?

With signed-Hadamard-128 rotation and group-64 quantization at exactly four payload bits, RAR improves over direct whole-layer KL allocation by 0.3093 [0.2926, 0.3267], 0.0148 [0.0102, 0.0193], and 0.0113 [0.0074, 0.0153] on Qwen/Mistral/OLMo. The quantizer and scale metadata are identical within each comparison; the whole-layer control receives six-point KL curves and RAR uses a single reference precision. Thus the placement benefit persists after rotation and finer quantization groups reduce numerical error.

We have added these controlled ablations in Section 4.2 and Appendix D.10, explicitly distinguishing the submitted layer-score baseline from the stricter one-bit-width-per-layer control.

Reviewer_T7qhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Role-Aware Reliability, a mixed-precision activation quantization framework. It measures the next-token distribution change caused by quantizing individual layer-role activation sites, then allocates precision using a risk model. The method improves language-modeling quality and downstream accuracies.

Critical strengths
  1. The paper provides a useful sensitivity study with calibration, same-budget allocation controls, granularity ablations.
  2. Smooth-RAR-log improves NLL across evaluations.
Critical weaknesses
  1. The technical advance over existing sensitivity-based mixed-precision allocation is not clear in the current version. The paper’s layer-only baseline is already competitive: it recovers 95.5% of uniform-6-bit NLL degradation versus 96.8% for layer-role allocation. (By the way, I feel like it’s better to use the NLL number directly instead of using percentage for Table 1?) To better establish the value of the proposed formulation, the authors might need to compare with established allocators and simple role-protection policies.
  2. In Table 2, downstream accuracy gains are concentrated in the two Qwen checkpoints. Comparisons with finer-grained activation quantizers and rotation-based methods are helpful to determine whether RAR remains beneficial beyond the quantization setup evaluated here. Also, the width-weighted average bit budget does not directly account for activation lifetimes, peak memory, or execution overhead. A concrete deployment scenario will be helpful to establish whether heterogeneous 3-8 bit activation policies improve the accuracy/memory/latency tradeoff after packing and reconstruction costs are included.
Questions for authors
  1. How does RAR compare with established sensitivity-based allocators under matched budgets? Also, what is the technical contribution here?
  2. Do the gains persist with per-channel/per-group activation quantization or rotation-based preprocessing, using comparable boundaries and resource accounting?
  3. Qwen2.5 is an old model series. Can the authors show the results on the Qwen3.5/3.8 series?
  4. What deployment scenario is targeted, and what end-to-end memory or latency benefit remains after packing and dequantization overhead?
5 comments

Thank you for highlighting the sensitivity study and consistent NLL improvements. Our new controlled ablations establish a consistent result: at a fixed quantizer and storage budget, choosing which computations receive precision improves quality. This benefit holds across KL, Hessian, and Fisher criteria, and remains after rotation and finer quantization. The same allocation principle also improves downstream accuracy beyond Qwen and enables larger long-prompt batches at near-BF16 NLL and faster decoding with smaller persistent KV caches.

Base-atlas RAR is Smooth-RAR-log: smooth activations and reuse the original risk measurements. Residual-atlas RAR is Smooth-Recalib-RAR-log: measure risk after smoothing. NLL is in nats/token; lower is better. Unless specified otherwise, NLL averages the WikiText-2 and C4 corpus means. Brackets report paired 95% confidence intervals (CIs).

W1 / Q1: Technical contribution and stronger allocation baselines

RAR uses one allocation rule: give more precision to sites where rounding causes greater prediction damage per unit of storage. A layer contains several computations: Q/K selects which tokens to attend to, V carries their information, and the MLP transforms features. RAR measures each site's effect on the final prediction and accounts for the number of activation values that an extra bit must store. The rule is shared across models; the measured risks determine the assignments.

The technical contribution is a reusable activation-risk atlas and an allocation rule that turn one reference measurement per site into a policy for multiple budgets. The atlas measures the chosen quantizer's effect on predictions, including after smoothing or rotation. Under the additive risk model, the analysis derives the continuous allocation rule and quantifies the benefit of heterogeneous precision. The controlled ablations then establish why keeping these measurements separate within a layer matters, with all sites quantized together. Smoothing and rotation reduce rounding error; RAR identifies where the remaining error still warrants extra precision.

The closest allocation precedents are HAWQ-V2 (https://arxiv.org/abs/1911.03852), which estimates Hessian-trace sensitivity for weights and activations; ZeroQ (https://arxiv.org/abs/2001.00281), which measures output KL at candidate layer precisions; and TWLA's ILA-AMP (https://arxiv.org/abs/2606.13054), which combines layer-level activation NLL costs with adjacent-layer interactions. RAR measures activation sites across attention, MLP, and normalization, then tests how much quality is gained from retaining their separate measurements.

Attention-aware work also provides a useful comparison: APTQ (https://arxiv.org/abs/2402.14866) uses attention-output reconstruction and Hessian information for weight quantization, while Dewage et al. (https://arxiv.org/abs/2609.26173) study how attention-projection weight errors affect perplexity. RAR turns a full-block activation diagnosis into a budgeted policy and validates the placement effect with all sites quantized together.

Controlled allocator ablations. We designed these experiments to identify which part of the method produces the gain. Checkpoints, quantization sites, smoothing scales, calibration text, candidate bits 3–8, and the exact integer solver are shared. Every quantized policy receives exactly 6.000 width-weighted bits. The table compares sensitivity criteria and placement constraints under this common protocol; the paired ablations below then vary one factor at a time. Evaluation uses BF16 weights, SDPA, and 128 held-out 256-token blocks per corpus. Scales use 32 training blocks; sensitivity uses four.

Allocation policyQwen2.5-7BMistral-7BOLMo-2-7B
Unquantized reference2.49842.18792.5141
Smooth uniform 62.99422.24302.5755
Layer-score allocation2.51652.21412.5487
One bit-width per layer, summed single-site KL2.52492.22242.5731
Direct whole-layer KL curves, six candidate bits2.52482.21462.5706
Protect Q/K at 8 bits, reallocate remaining budget2.58052.26332.8029
Protect normalization sites at 8 bits, reallocate remaining budget9.19462.22072.6489
Local reconstruction-error allocation2.54082.53562.8278
HAWQ-V2-style Hessian-trace allocation2.51152.22022.5695
Fisher-trace allocation2.51212.20812.5312
Diagonal-Fisher allocation2.51052.21522.5381
Direct site-level KL curves, six candidate bits2.51042.20782.5310
Smooth-RAR-log, base atlas2.51402.20982.5384
Smooth-Recalib-RAR-log, single-reference residual atlas2.51152.20842.5326

We adapt established allocation criteria to a common activation problem so that their information can be compared directly. The HAWQ-V2-style control weights quantization error by the magnitude of the activation-NLL Hessian trace, estimated with eight probes on each of four training blocks. Fisher estimators are separate controls. The ZeroQ-style controls directly measure output KL at all six precisions, either for a whole layer jointly or for each site. The shared exact solver removes optimizer differences from this comparison.

RAR obtains fine-grained allocation quality with an inexpensive forward-only scan. It lowers NLL relative to every layer-level and fixed-role-protection control above, matches the Hessian control to four decimal places on Qwen, and improves on it by 0.0118 [0.0089, 0.0148] on Mistral and 0.0368 [0.0330, 0.0406] on OLMo. The RAR scan takes 30.1/42.6/41.5 seconds, versus 392.5/540.1/632.7 seconds for Hessian estimation on the same eager backend: a 12.7–15.2× speedup in sensitivity estimation. Both timings exclude loading, scale estimation, and allocation.

Reducing the reference precisions from six to one leaves small observed NLL gaps to the full site-KL scan: 0.0011 [−0.0009, 0.0033], 0.0006 [−0.0015, 0.0026], and 0.0016 [−0.0002, 0.0034] on Qwen/Mistral/OLMo (RAR minus six-point NLL). All three CIs include zero. NLL intervals use 5,000 paired bootstrap draws of eight-block text clusters within each corpus, with the calibrated policies fixed.

Fix the estimator; change only the precision resolution. Allowing independent layer–role precisions improves held-out NLL in all 24 model × criterion × budget comparisons: three models, four criteria (single-reference KL, six-point KL, Hessian trace, and Fisher trace), and two budgets. Each pair shares the same sensitivity measurements, solver, and exact budget; every paired 95% CI favors layer–role allocation. For single-reference KL, mean NLL falls from 2.4401 to 2.4175 at six bits and 2.6076 to 2.4747 at five bits. The placement benefit therefore transfers across established sensitivity criteria. Repeating the six-bit comparison with three calibration subsamples (seeds 11, 23, and 47) gives paired NLL reductions of 0.0140 ± 0.0006, 0.0149 ± 0.0015, and 0.0359 ± 0.0039 on Qwen/Mistral/OLMo, respectively (mean ± sample SD across subsamples). All nine comparisons improve.

Fix every layer's budget; change only which sites receive its bits. Five fixed permutations exchange assignments among equal-width sites in each layer, preserving its budget and bit histogram. On a reserved evaluation suffix of WikiText-2 and C4, RAR has lower mean NLL than all 30 shuffled policies across six checkpoints. The paired CI against the mean shuffle is positive on five checkpoints; Qwen3.8's interval includes zero. Together, the fixed-estimator and fixed-layer-budget ablations establish the value of role-level placement, with RAR providing an efficient way to measure the required information.

Absolute NLL also clarifies the submitted 95.5%/96.8% result. Across its 15 matched model–corpus pairs, clean, layer-score, and layer–role NLL are 2.5389, 2.6080, and 2.5920. Thus, layer–role allocation removes 23.2% of the layer-score policy's remaining mean degradation. The submitted layer-score baseline already allows some within-layer bit variation because it retains site-specific width costs. The new controls impose one precision per layer, making the comparison explicit. We now report absolute NLL first in Table 1 and distinguish the layer-score and layer-tied controls in Section 4.2 and Appendix D.10.

W2 / Q2: Finer quantizers and rotation

RAR improves quality after finer quantization and rotation, with clear gains on all three models at four bits. We test this through component ablations: uniform and RAR share the same sites, transforms, scales, and exact payload-bit budget, and we remeasure the atlas for each quantizer. This isolates the additional value of precision placement after preprocessing. Evaluation uses BF16 weights, eager attention, eight training blocks for risk estimation, and 128 held-out 256-token blocks per corpus. Grouping and rotation arms use no smoothing. Each cell reports uniform → RAR NLL.

QuantizerAverage payload bitsQwen2.5-7BMistral-7BOLMo-2-7B
Static per-channel63.2802 → 2.65572.3953 → 2.28532.9207 → 2.8481
Group-64412.1596 → 2.85342.3139 → 2.27572.6337 → 2.6271
Group-6465.8742 → 2.65882.1941 → 2.19222.5275 → 2.5207
Hadamard-128411.0199 → 3.36292.6057 → 2.43573.1209 → 2.8854
Hadamard-12863.7978 → 2.62052.2150 → 2.19682.5339 → 2.5241
Hadamard-128 + group-6449.3324 → 2.79552.2687 → 2.25242.5944 → 2.5831
Hadamard-128 + group-6462.8995 → 2.58702.1906 → 2.19082.5171 → 2.5171

Role-level placement also improves on a strong layer-level allocator under the combined quantizer. At four bits, the whole-layer control receives a six-precision KL scan, while RAR uses one reference precision per site:

Hadamard-128 + group-64, four-bit budgetWhole-layer KL curvesLayer–role RARNLL reduction, paired 95% CI
Qwen2.5-7B3.10482.79550.3093 [0.2926, 0.3267]
Mistral-7B2.26722.25240.0148 [0.0102, 0.0193]
OLMo-2-7B2.59442.58310.0113 [0.0074, 0.0153]

CIs use 5,000 paired resamples of eight-block text clusters. All three intervals favor RAR. With the quantizer, scales, weights, and budget fixed, this directly demonstrates the value of within-layer allocation after both rotation and grouping.

Tighter budgets make role-level allocation valuable even after both transforms. At six bits, group-64 gives positive paired improvements on Mistral, 0.0019 [0.0005, 0.0034], and OLMo, 0.0068 [0.0051, 0.0085]. Adding rotation brings these models close to clean NLL, with additional RAR changes of −0.0002 [−0.0013, 0.0009] and 0.0000 [−0.0010, 0.0010]. At four bits, all three models benefit under that combined transform. The low-bit results establish the additional value of allocation when stronger compression makes precision scarce.

On Qwen, this translates into better quality with less storage: four-bit RAR with both transforms achieves 2.7955 NLL, versus 2.8995 for uniform six bits, with 30.8% fewer encoded boundary bytes. Accounting includes the FP32 group scale: 0.5 bit/value at group size 64, so the two formats use 4.5 and 6.5 bits/value. The deployment experiments below additionally measure actual live GPU memory.

The rotation ablation applies a fixed signed Hadamard transform at each site and inverts it before the next operation, keeping weights and boundaries fixed. Full QuaRot (https://arxiv.org/abs/2404.00456) rotates weights, activations, and caches; SpinQuant (https://arxiv.org/abs/2405.16406) learns rotations. Our comparison uses controlled preprocessing components, not complete reproductions of those pipelines. Holding the transform fixed makes the additional contribution of RAR directly measurable.

W2: Downstream benefits beyond Qwen

The expanded evaluation provides two forms of evidence beyond Qwen: significant gains on OLMo and Pythia, and positive mean gains across five non-Qwen models under stronger quantizers. We first evaluated six complete zero-shot tasks using lm-evaluation-harness 0.4.10: ARC-Easy, ARC-Challenge, HellaSwag, BoolQ, OpenBookQA, and WinoGrande (18,627 examples per policy, 512-token input limit). Accuracy is the unweighted mean across tasks. Intervals use paired example bootstraps; realized average bits differ from targets by at most 0.0057 bit.

Model / average-bit targetSmooth uniform (%)Base-atlas RAR (%)Residual-atlas RAR (%)Residual gain, percentage points [95% CI]
Qwen3.5-9B / 663.5863.8564.08+0.50 [−0.05, +1.06]
Qwen3.5-9B / 561.2762.2262.85+1.58 [+0.70, +2.39]
Qwen3.8-27B / 564.0164.9764.66+0.65 [−0.14, +1.39]
Mistral-7B / 562.4461.9562.52+0.08 [−0.63, +0.84]
OLMo-2-7B / 560.5261.2361.65+1.14 [+0.22, +1.99]
Llama-3.1-8B / 561.1160.5060.77−0.34 [−1.15, +0.50]
Pythia-6.9B / 6, FP32 atlas46.3847.3047.59+1.21 [+0.34, +2.04]
Pythia-6.9B / 5, FP32 atlas41.7843.2645.52+3.74 [+2.80, +4.70]

At five bits, residual-atlas RAR improves OLMo by 1.14 percentage points and Pythia by 3.74 points, with positive paired CIs. Accuracy records whether the selected answer changes, so its response can differ from NLL; the Mistral and Llama intervals span zero in this evaluation. The broader four-bit experiment below tests downstream transfer across models under two stronger quantizers.

The Pythia control identifies an effective offline calibration choice. FP32 atlas measurement yields 47.59%/45.52% accuracy at six/five bits, compared with 46.47%/42.26% using a BF16 atlas. This one-time refinement preserves BF16 inference and the same activation-bit targets.

We also tested Mistral-7B, OLMo-2-7B, Pythia-6.9B, Llama-3.1-8B, and Mistral-Nemo-12B at four bits. This separate quantizer ablation uses a common likelihood scorer, a 1,024-token limit, the same six tasks, and the first 3,000 HellaSwag examples (11,585 examples per policy). Tasks and models receive equal weight; the paired bootstrap preserves shared-example dependence across models.

Four-bit quantizer, five non-Qwen modelsMean RAR accuracy gain95% CIModels with positive mean gain
Per-token group-128+2.55 percentage points[+2.12, +3.02]5/5
Signed-Hadamard + per-token quantization+0.78 percentage points[+0.35, +1.24]3/5

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.