acceptodds
Under review as a conference paper at ICLR 2027

Test-Time Adaptation of Quantized ViTs via Single-Pass Quantizer-Aligned Recalibration

Abstract

Post-training quantization is a standard route to fitting vision transformers (ViTs) into edge compute and memory budgets, yet quantized models become especially brittle under distribution shift. Test-time adaptation (TTA) addresses such shifts without labels, but most existing approaches are poorly aligned with the constraints of quantized inference. Prevailing TTA methods recover accuracy through backpropagation, while backprop-free methods often still incur overhead from extra forward passes or parameter updates, and lightweight feature- or logit-level methods recover only part of the loss. Across these approaches, a quantization-specific failure mode that amplifies the drop is not directly targeted: under shift, activations occupy frozen quantizers' calibrated ranges differently, distorting their *code distribution*. We propose *Quantizer-Aligned Recalibration* (QuAR), a *single-pass* TTA method tailored to quantized ViTs that neither backpropagates nor updates any model parameters. QuAR recalibrates activations at the input to a frozen quantizer, mapping the test stream's running per-channel statistics back toward the source calibration. On ImageNet-C with ViT-B, QuAR achieves the highest mean accuracy among state-of-the-art backprop-free TTA methods at 3-, 4-, 6- and 8-bit weight/activation precision, outperforming the strongest baseline by points at 8 bits and at 3 bits, with % lower latency and a memory overhead of only MB (% of peak inference memory). Analysis and diagnostics trace the gain to a reduced per-channel mismatch at these quantizers, which restores the code distribution the baselines leave unchanged or distort further. A single fixed configuration remains ahead across continual streams, non-i.i.d. label shift, seven out-of-distribution suites, and three other backbones.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
test-time adaptation, distribution shift, quantized neural networks, vision transformers

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qQjcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes QuAR for unsupervised test-time adaptation of quantized vision transformers. It applies a per-channel affine correction at the activation-quantizer inputs in the last four transformer blocks, using source statistics and running target statistics. A global branch also corrects the CLS representation at the classifier input. Model weights and quantizer parameters remain frozen, and adaptation requires neither backpropagation nor additional forward passes. The authors report accuracy improvements across several bit-widths and distribution-shift benchmarks, with ablations on the recalibration scope and matching strength.

Critical strengths

QuAR outperforms the compared backpropagation-free adaptation methods across multiple bit-widths on ImageNet-C, and the advantage persists on the full test set. Results on other datasets and backbones further support the empirical value of the design.

Critical weaknesses

[W1] - The conceptual contribution is limited, and the mechanism analysis does not fully explain the accuracy gains

The core operation matches per-channel target activation means and scales to source statistics, which is closely related to existing activation-statistics alignment methods. The main distinctions are its placement at late quantizer inputs and the combination of partial matching with online estimation.

The authors argue that distribution shift contracts some channels, reducing the number of quantization codes they occupy, and that restoring the source scale recovers resolution. This is plausible, but the connection to classification accuracy remains incomplete. Given the reported small changes in clipping and quantization-error magnitude, bringing the code distribution closer to the source does not by itself explain why decisions improve.

The reduced classification margin under shift explains why quantized models become more vulnerable, but does not establish how QuAR repairs that vulnerability. Full matching restores more of the code distribution yet achieves lower accuracy, further showing that this diagnostic alone cannot explain performance. Evidence on how recalibration changes classification margins or harmful quantization-induced prediction flips would help separate general feature-alignment gains from gains specific to quantization. Because the method itself is straightforward, the incomplete explanation limits my assessment of its contribution.

[W2] - The accuracy-efficiency trade-off under actual quantized deployment has not been established

Resource-constrained edge deployment is a central motivation, but both the server and Jetson experiments use floating-point simulation of quantization rather than actual integer kernels. Running this implementation on Jetson demonstrates that it executes on an edge device; it does not establish that the same accuracy gains and relative overhead hold under real low-bit inference.

QuAR requires activation corrections to remain active during inference while target statistics are updated online. The reductions and transformations at multiple interfaces may add memory traffic and kernel-launch overhead, or interfere with existing operator fusion. Once integer kernels accelerate the backbone, these operations may account for a larger fraction of inference time. A single forward pass therefore does not establish low deployment overhead.

At least one real W4A4 or W8A8 implementation, with end-to-end accuracy and latency measurements, is needed to assess the claimed deployment benefits.

Questions for authors
  1. How does recalibration change classification margins or harmful quantization-induced prediction flips?

  2. Can you report accuracy and end-to-end latency under real W4A4 or W8A8 execution, including statistics updates and correction, against baselines on the same stack?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

[Weakness 1-1] Margins, harmful flips, and what is quantization-specific

We thank the reviewer for pinpointing the decision-level evidence that links the analysis to accuracy, and we have now measured it. We agree that a restored code distribution does not by itself explain the decisions, so we measured them. Running the full-precision and the quantized network over the same images, each method applied to the quantized one, we count harmful flips (correct in full precision, wrong once quantized) and helpful ones (the reverse), with the median logit margin (top-1 minus top-2) of the adapted quantized network (Table G).

Table G. Harmful and helpful flips and median logit margin per method (W6A6, 15 corruptions, seed 0).

methodtop-1harmful flips (%)helpful flips (%)median logit margin (quantized network)
No Adapt47.6947.6911.7811.783.893.891.311.31
NEO51.3151.319.689.685.425.421.401.40
QuAR (Global-only)51.1751.179.859.855.445.441.421.42
QuAR (Local-only)53.4853.489.199.197.097.091.551.55
QuAR (full)54.36\mathbf{54.36}8.97\mathbf{8.97}7.757.751.61\mathbf{1.61}

Recalibration at the quantizer inputs removes a quarter of the harmful flips and returns the margin to its full-precision value. Quantization lowers the median margin from 1.611.61 in full precision to 1.311.31 (Table 34 reports the mean, 2.052.05); recalibration restores all of it (1.611.61) and removes a quarter of the harmful flips (11.7811.78% →\to 8.978.97% of images, −2.8-2.8 points), while the head-level methods, NEO and QuAR (Global-only), acting after every quantizer, recover about a third of the margin (1.401.40–1.421.42) and fewer flips; per corruption the reduction tracks the gain (Appendix Q). This is why small changes in clipping and error magnitude are enough. A flip depends on where the error sits against the margin, and recalibration leaves the error nearly unchanged but moves the operating point (Section 2.5).

The quantization-specific part of the gain is the excess over what the same correction gives the full-precision network, and Table G shows it as harmful flips. Applied to the full-precision network, the correction gains +3.8+3.8 points; applied to the quantized network, +6.1+6.1 to +7.6+7.6 (Fig. 1a). The first number is what per-channel statistics alignment buys on its own; the excess is what restoring the quantizer's operating point buys on top, and it exists only at the quantizer input, since the same affine after the quantizer or on the residual stream falls below No Adapt at W3A3 and W4A4 (Table 38). Table G splits the same way. Helpful flips add +3.9+3.9 points, the general-alignment part, against the +3.8+3.8 of the full-precision network; harmful flips drop −2.8-2.8 points, the quantization-specific excess.


[Weakness 1-2] Why full matching restores more codes yet is less accurate

The gain the correction applies is an estimate, and full matching applies all of its error. The correction rescales each channel by σs/σt\sigma_s/\sigma_t with σt\sigma_t estimated from the stream, so the gain carries an error, 2.32.3–2.82.8% after a LayerNorm and 9.99.9% and 42.642.6% at the post-attention and post-GELU inputs (Table 26, Appendix K). The strength λ\lambda sets how much of the correction, and of that error, each activation receives. The code diagnostic averages over tokens and batches, so it sees the restoration, which only improves with λ\lambda, but not the per-activation error; accuracy sees both, and since the injected error grows faster than the restoration the best strength is partial (the shrinkage of Corollary V.2). Our paper fixes λ=0.5\lambda=0.5 once; Table H shows both sides.

Table H. Local-branch match strength λ\lambda (W6A6, seed 0; code restoration pooled over the 16 local inputs and the head input).

λ\lambda000.250.250.50.50.750.7511
top-151.1751.1753.7753.7754.3654.3654.1654.1653.5653.56
harmful flips (%)9.859.858.738.738.978.979.439.4310.1410.14
median logit margin (quantized network)1.421.421.551.551.611.611.601.601.551.55
code distribution restored (%)8.88.829.529.551.751.774.474.484.384.3

Code restoration rises with every step of λ\lambda, 8.88.8% to 84.384.3% (the 8.88.8% at λ=0\lambda=0 is the global branch's share at the head input), while top-1, harmful flips and the margin peak around the deployed λ=0.5\lambda=0.5 and worsen again at λ=1\lambda=1 (−0.8-0.8 top-1 and +1.2+1.2 points of harmful flips). The diagnostic shows where the method acts; the flips and the margin show what it does to the decisions, which is why they track accuracy and the diagnostic does not. The revision adds one sentence to Section 2.5 and both tables to Appendix Q.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

[Weakness 1-3] The conceptual contribution

W1-1 and W1-2 add decision-level evidence, harmful flips and margins across methods and λ\lambda. These results sharpen our contribution, a quantization-specific failure mode and a correction aimed at it directly. Under shift, activations leave the ranges the frozen quantizers were calibrated on, which costs the contracted channels resolution and lets the unchanged quantization error flip more decisions. Existing quantized TTA acts after these quantizers or upstream of them; QuAR corrects the activation at the input of each frozen quantizer in the late blocks. The per-channel affine form is shared with statistics alignment, but the target, a frozen quantizer's operating point, is not; what the paper adds is the evidence that its gain is quantization-specific and an analysis of where and how much to correct. The distinctions the reviewer names, namely placement, online estimation and partial matching, are the mechanism.

  • Only the late quantizer inputs carry the quantization-specific gain. Moved after the quantizer or onto the residual stream, the same affine gains 1.51.5 and 2.32.3 points less at W6A6 (Table 38), and applied to all twelve blocks it falls below No Adapt (Table 5), because the quantized path amplifies an early correction's residual 4.8×4.8\times where the full-precision path does not (0.6×0.6\times; Appendix G).
  • Pooling the moments over the stream is what the post-nonlinearity inputs need. A block has four quantizer inputs, two after a LayerNorm and two after a nonlinearity (attention, GELU). The post-LayerNorm pair gains the same with pooled or batch moments; the post-nonlinearity pair adds 2.22.2 points only with pooled moments, in line with one batch estimating their gain 44 to 1717 times worse than at the post-LayerNorm pair (Tables 25 and 26, Appendix K).
  • Partial matching is optimal because the estimated gain carries error. Matching fully applies the whole estimated gain and with it the whole estimation error, so accuracy peaks at a partial strength (W1-2).

[Weakness 2] Real quantized deployment

We now have the measurement the reviewer asks for (Table I). The ViT-B/16 is exported to a QDQ graph; accuracy is read on ONNX Runtime int8 and end-to-end latency on the Orin Nano with TensorRT INT8, whose engine reproduces the ONNX Runtime accuracies of the unadapted model and QuAR (52.2152.21 and 58.7158.71 against 52.2452.24 and 58.7958.79; Appendix U).

Table I. Real W8A8 on integer kernels (ImageNet-C, 15 corruptions; one run, seed 0). Baselines run their own code on the integer graphs; latency includes each method's host-side work.

W8A8 on integer kernelstop-1latency, batch 64batch 16
No Adapt52.2452.24275.6275.6 ms (1.00×1.00\times)88.788.7 ms (1.00×1.00\times)
T3A52.2152.21807.9807.9 ms (2.93×2.93\times)603.4603.4 ms (6.81×6.81\times)
FOZO54.5254.52561.2561.2 ms (2.04×2.04\times)179.0179.0 ms (2.02×2.02\times)
FOA56.2956.29593.3593.3 ms (2.15×2.15\times)211.6211.6 ms (2.39×2.39\times)
ZOA56.7656.76609.7609.7 ms (2.21×2.21\times)208.9208.9 ms (2.36×2.36\times)
NEO56.7556.75282.1282.1 ms (1.02×1.02\times)89.689.6 ms (1.01×1.01\times)
QuAR58.79\mathbf{58.79}333.9333.9 ms (1.21×\mathbf{1.21\times})105.3105.3 ms (1.19×\mathbf{1.19\times})
  • The accuracy gain transfers. QuAR remains the most accurate method, its gain (+6.55+6.55) tracks the fake-quant gain (+6.08+6.08) and the baselines keep their accuracy order; every method sits slightly lower in absolute terms because ONNX Runtime's PTQ is plainer than PTQ4ViT's.
  • The single-pass overhead transfers. QuAR runs at 1.191.19–1.21×1.21\times the unadapted engine, against 2.022.02–2.39×2.39\times for the two-pass methods (1.071.07–1.08×1.08\times in floating-point simulation).
  • Fusion and memory traffic. The profile shows the GEMMs unchanged (146.7146.7 vs 145.4145.4 ms, batch 64), so the Q/DQ fusion is intact; the corrections alone are free (0.40.4 ms without the moment reductions). What the method adds is the memory traffic of the online statistics, 3030 ms in the token-sum GEMMs and 2222 ms in the pointwise kernels that square and write their outputs, plus a 1.31.3 ms host update; kernels account for 5353 of the 5757 added ms, so launch overhead is at most 44 ms.

[Question 1] Margins and flips

We thank the reviewer for the question and added this measurement (Table G). Recalibration at the quantizer inputs removes a quarter of the harmful flips (11.7811.78% →\to 8.978.97%) and returns the median logit margin to its full-precision value (1.31→1.611.31\to1.61). See our replies to Weakness 1-1 and 1-2 for details.


[Question 2] Real W8A8 execution against baselines on the same stack

We implemented W8A8 on a real integer stack, with the statistics update and correction included and every forward-only baseline on the same stack (Table I). QuAR is the most accurate (58.7958.79, 2.02.0 above ZOA) at 1.191.19–1.21×1.21\times against the two-pass baselines' 2.022.02–2.39×2.39\times. See our reply to Weakness 2 for details.

Thank you for the additional experiments. The real W8A8 results address much of my deployment concern. The margin and prediction-flip analyses strengthen the mechanism explanation, but do not fully isolate quantization-specific gains from general feature alignment. Given the additional evidence, I will raise my score to 3 (weak accept).

We thank the reviewer for the careful reading and constructive feedback, as well as for raising the score. We would welcome any further questions during the discussion period.

Reviewer_8rhwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Quantizer-Aligned Recalibration (QuAR), a single-pass test-time adaptation method for quantized Vision Transformers. The key observation is that distribution shifts change the per-channel activation statistics entering frozen quantizers, altering their code usage and potentially reducing their effective resolution. Based on this, QuAR applies partial affine recalibration before the activation quantizers in the last four transformer blocks, together with a full recalibration of the final CLS representation. The method uses source statistics and running target statistics without updating model weights or quantizer parameters. Experiments across multiple bit-widths, datasets, and architectures show consistent improvements over the evaluated backpropagation-free adaptation methods, with low overhead in the authors’ fake-quantization implementation.

Critical strengths

1.The paper is well motivated and identifies a useful intervention point for adapting quantized models. In particular, the observation that activation drift changes how frozen quantization grids are occupied provides an intuitive motivation for correcting activations before quantization. The placement ablations support this choice, showing that applying comparable corrections after quantization or at block outputs produces smaller gains.

2.The proposed method is simple and computationally economical. Both branches use closed-form affine transformations and running statistics, allowing adaptation within a single forward pass without backpropagation, parameter optimization, or storing previous samples. The component ablations also show that the local branch provides substantial improvements beyond final-feature recalibration alone.

3.The empirical evaluation is extensive. Beyond the main ImageNet-C results, the paper includes full-test-set evaluation, continual adaptation, class-concentrated streams, additional datasets and backbones, alternative quantizers, and asymmetric bit-widths. The experiments with held-out corruption types and the sensitivity studies strengthen the evidence that the gains are not restricted to one narrowly selected configuration.

Critical weaknesses

1.The comparison with the closely related AQR method does not fully isolate the sources of QuAR’s improvement. AQR estimates activation quantiles from the current batch, whereas QuAR uses running statistics accumulated across batches. Table 14 shows that replacing QuAR’s local running statistics with current-batch statistics reduces accuracy by approximately two percentage points, suggesting that the estimator choice contributes substantially to performance. Although the paper includes placement ablations, it remains unclear how much of the advantage over AQR comes from correcting activations at quantizer inputs versus using more stable target statistics.

2.Some theoretical claims appear stronger than the stated assumptions justify. Proposition 1 derives an information bound by treating a variance-based effective range as a bound on the number of possible quantizer outputs. However, a contraction in standard deviation does not generally reduce support cardinality for sub-Gaussian distributions with nonzero tails. Similarly, treating the granular quantization error as an input-independent Δ2/12\Delta^2/12 requires additional approximation assumptions. The analysis provides useful intuition, but its connection to the observed accuracy improvements is less conclusive than parts of the presentation suggest.

3.The practical deployment claims are not fully supported by the implementation evaluated. Appendices R and U clarify that the experiments use floating-point quantization and dequantization rather than an integer-kernel inference stack. The reported overhead ratios may therefore change when the underlying inference is substantially accelerated. The statement that backpropagation-based methods cannot fit the Jetson also depends on batch size: Appendix S shows that several methods fit at batch size 16. This limits the scope of the demonstrated deployment advantage.

Questions for authors

1.Could the authors provide a more controlled comparison with AQR that matches the correction locations and accounts for the difference between per-batch and accumulated target statistics? In particular, how much of QuAR’s advantage remains when the methods use comparably stable statistical estimates? This would help clarify the contribution of quantizer-input placement beyond the benefits of pooling statistics across batches.

2.Is the code-level starvation result intended as an effective-resolution approximation or a strict information-theoretic bound? If the latter, what additional assumptions justify translating the standard-deviation ratio into a bound on the number of possible quantizer outputs? It would also help to clarify which conclusions about accuracy follow from the analysis beyond the assumed local relationship between task loss and deviation from the source operating point.

3.Given that the reported measurements use fake quantization, what evidence supports extending the efficiency conclusions to an optimized integer inference backend? For the Jetson comparison, could the authors report the accuracy–latency trade-off when each method uses a batch size that fits the device? This would help establish whether QuAR remains the most attractive option under the actual memory constraint.

7 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/3)

[Weakness 1] AQR, placement versus statistics

We thank the reviewer for posing the AQR comparison as a measurable question, and we ran the controlled comparison it asks for. The advantage over AQR comes from quantizer inputs that AQR leaves uncorrected, and inside the blocks it needs both the site and pooled statistics, a combination QuAR's mean-and-scale match can use and AQR's quantile map does not. AQR's "after-LayerNorm" tensors are the quantizer inputs of qkv and fc1, so the site question is which quantizer inputs; QuAR adds the post-attention proj and post-GELU fc2 inputs. Table C varies the site and the estimator independently, with AQR also run at QuAR's blocks 8–11 to match the locations as asked.

Table C. Site ×\times estimator study against AQR (W6A6, batch 64, 3 seeds; No Adapt 47.6947.69).

correctionblocksquantizer inputspooled over the stream (QuAR's estimator)batch at hand (AQR's estimator)
AQR as published6–11 (AQR's)qkv, fc1 (AQR's)43.03±0.0343.03\pm0.0351.37±0.0451.37\pm0.04
AQR quantile map8–11 (QuAR's)qkv, fc1 (AQR's)47.70±0.0347.70\pm0.0352.36±0.0452.36\pm0.04
AQR quantile map8–11 (QuAR's)all four (QuAR's)47.78±0.1247.78\pm0.1238.10±0.1038.10\pm0.10
QuAR (Local-only)8–11 (QuAR's)qkv, fc1 (AQR's)51.18±0.0151.18\pm0.0151.28±0.0551.28\pm0.05
QuAR (Local-only)8–11 (QuAR's)all four (QuAR's)53.42±0.0453.42\pm0.0451.25±0.0351.25\pm0.03
QuAR (full)8–11 (QuAR's)all four (QuAR's)54.35±0.02\mathbf{54.35\pm0.02}52.37±0.0152.37\pm0.01

The two points of Table 14 that the reviewer cites come from the two added inputs, not from stable statistics in general. Without the global branch, Table 14's 52.37→54.3552.37\to54.35 is 51.25→53.4251.25\to53.42. At AQR's two inputs pooling changes QuAR by 0.10.1 (51.28→51.1851.28\to51.18); the two added inputs give +2.2+2.2 with pooled moments (51.18→53.4251.18\to53.42) and nothing per batch (51.28→51.2551.28\to51.25). Table D points to the reason. A single batch estimates the per-channel gain well after a LayerNorm but 4 to 17 times worse at the two added inputs, so a per-batch correction carries that error into the activations there, the gain-error term of Corollary V.2, which pooling averages out.

Table D. Error of the per-channel gain estimated from one batch, relative to the whole stream (blocks 8–11, W6A6, No Adapt, seed 0, median over channels).

quantizer inputqkvfc1projfc2
followsLayerNormLayerNormattentionGELU
corrected byAQR and QuARAQR and QuARQuAR onlyQuAR only
per-batch gain error2.32.3%2.82.8%9.99.9%42.642.6%

Of the two corrections, only the mean-and-scale match combines the two. Given QuAR's inputs and pooled statistics, AQR's quantile map stays at the unadapted level (47.7847.78 against 47.6947.69); pooling alone leaves it at 47.7047.70 at QuAR's blocks and drops it to 43.0343.03 at its own, 4.74.7 below No Adapt, and the inputs alone drop it to 38.1038.10. A quantile map is built around the batch's own extremes, so the per-batch estimator is the one it is designed for.

The +3.0+3.0 over AQR as published comes in three parts. QuAR's blocks give +1.0+1.0 (52.3652.36), its local branch at all four inputs with pooled moments +1.1+1.1 (53.4253.42), and its global branch at the head's quantizer input +0.9+0.9 (54.3554.35), so +2.0+2.0 remains over AQR even at QuAR's blocks. Tables C and D go to Appendix K.

AuthorsOpenReview ↗

Official Comment by Authors (2/3)

[Weakness 2] Proposition 1 and the Δ2/12\Delta^2/12 term

We thank the reviewer for the close reading of Proposition 1; the revision restates it accordingly. Proposition 1 is an effective-resolution statement, and in that form it holds. We agree that the support cardinality of a sub-Gaussian variable does not contract with its standard deviation; what contracts is the number of levels carrying all but ε\varepsilon of the mass, a distinction the revision now draws.

  • The frozen bb-bit grid keeps its 2b2^b levels; LtL_t counts those carrying all but ε\varepsilon of the channel's mass. Splitting on the in-range event gives the exact entropy bound H(Q(xc))≤log⁡2Lt,c+h(ptail)+ptailb,H\bigl(Q(x_c)\bigr)\le\log_2L_{t,c}+h(p_{\text{tail}})+p_{\text{tail}}b, with hh the binary entropy and ptail=εp_{\text{tail}}=\varepsilon the mass outside the Lt,cL_{t,c} levels. Under (A1), the sub-Gaussian standardised shape, ε\varepsilon can be small while Lt,cL_{t,c} grows only as log⁡(1/ε)\sqrt{\log(1/\varepsilon)}, so the last two terms stay small. With Ls,cL_{s,c} counted the same way at the source, Lt,c/Ls,c=σt,c/σs,cL_{t,c}/L_{s,c}=\sigma_{t,c}/\sigma_{s,c} up to rounding, so log⁡2(Ls,c/Lt,c)\log_2(L_{s,c}/L_{t,c}) is the bcb_c of the paper.
  • Δ2/12\Delta^2/12 is the high-resolution (Bennett) approximation. The revision adds it as an explicit assumption (A3), the high-resolution regime Δ≪σ\Delta\ll\sigma. The analysis uses one consequence, that the error power is set by the frozen step and does not move with the shift, measured directly as ∥eℓ∥\Vert e_\ell\Vert 6773→66006773\to6600 under shift at W6A6.

The analysis predicts directions and the site of the intervention; the magnitudes are measured. From (A1)–(A3), (i) the mean-and-scale match restores the calibrated code occupancy (Lemma V.1); (ii) a contracted channel loses effective resolution by a channel-dependent amount that no global scale undoes (Proposition 1); (iii) to first order a prediction flips once the propagated quantization error exceeds the unquantized margin (Proposition V.1). Measured, the margin shrinks from 3.783.78 to 2.052.05 under shift while the error remains unchanged (Table 34), and recalibration at the quantizer inputs removes a quarter of the harmful flips and returns the median logit margin to its full-precision value (Table 35, Appendix Q). This is now in Appendix V, with one sentence in Section 2.5.


[Weakness 3-1] Real W8A8 execution

To test whether the floating-point ratios transfer, we measured on a real integer stack. The gains transfer, QuAR +6.55+6.55 over the unadapted model, and the full method runs at 1.191.19–1.21×1.21\times the unadapted TensorRT engine with the GEMM fusion intact, well below the two-pass baselines at 2.022.02–2.39×2.39\times.

We export the same ViT-B/16 to an ONNX QDQ graph with symmetric 8-bit activations, per-channel 8-bit weights and percentile calibration on 128128 training images, and run ONNX Runtime's int8 kernels for accuracy and TensorRT INT8 on the Jetson Orin Nano for latency; the TensorRT engine reproduces these accuracies on the board (52.2152.21 and 58.7158.71). QuAR needs no custom kernel. Each correction is a per-channel Mul/Add before a QuantizeLinear node, the token sums are graph outputs, and the host updates the moments between batches; with identity moments the graph reproduces the plain QDQ logits bit for bit.

Table E. Real W8A8 on integer kernels (ImageNet-C, 15 corruptions; one run, seed 0). Baselines run their own code on the integer graphs with source statistics recomputed there; latency includes each method's host-side work per batch (T3A's is its support selection).

W8A8 on integer kernelstop-1latency, batch 64batch 16
No Adapt52.2452.24275.6275.6 ms (1.00×1.00\times)88.788.7 ms (1.00×1.00\times)
T3A52.2152.21807.9807.9 ms (2.93×2.93\times)603.4603.4 ms (6.81×6.81\times)
FOZO54.5254.52561.2561.2 ms (2.04×2.04\times)179.0179.0 ms (2.02×2.02\times)
FOA56.2956.29593.3593.3 ms (2.15×2.15\times)211.6211.6 ms (2.39×2.39\times)
ZOA56.7656.76609.7609.7 ms (2.21×2.21\times)208.9208.9 ms (2.36×2.36\times)
NEO56.7556.75282.1282.1 ms (1.02×1.02\times)89.689.6 ms (1.01×1.01\times)
QuAR58.79\mathbf{58.79}333.9333.9 ms (1.21×\mathbf{1.21\times})105.3105.3 ms (1.19×\mathbf{1.19\times})

The accuracy gain transfers. The +6.55+6.55 compares with +6.08+6.08 in Table 1, and QuAR leads ZOA by 2.02.0 (+2.3+2.3 there); every method sits slightly lower in absolute terms because ONNX Runtime's PTQ is plainer than PTQ4ViT's.

The single-pass overhead transfers. With the host update, the full method runs at 1.21×1.21\times the unadapted engine at batch 64 and 1.19×1.19\times at batch 16, against 1.07×1.07\times and 1.08×1.08\times in the fake-quant harness (Table 40). The GEMMs are untouched (146.7146.7 vs 145.4145.4 ms at batch 64) and the corrections are free (0.40.4 ms); the added 5757 ms is the memory traffic of the online statistics (breakdown in Appendix U). These numbers are now in Appendix U, with a sentence each in Sections 2.4 and 3.1.

AuthorsOpenReview ↗

Official Comment by Authors (3/3)

[Weakness 3-2] The Jetson comparison at the batch each method fits

We thank the reviewer for pointing out that the Jetson claim depends on the batch size, and we measured each method at the largest batch it fits. A built engine provides no gradient, so TENT, EATA, SAR and SPA cannot adapt on it at any batch size; they need a floating-point copy of the network beside it, the harness Table F measures with a straight-through gradient, while the forward-only methods run on the engine itself (Table E). What we correct is the memory sentence. Section 3.6's claim that none of the four backprop-based methods runs on the board holds at the protocol batch (64) but lacks the batch qualifier of Appendix S, since at 16 TENT, EATA and SAR fit; the revision adds it there and to the "deployable" column of Table 7.

Table F. Jetson Orin Nano at batch 16 (top-1 W6A6, 3 seeds; latency and peak memory measured on the board; ✓ = runs on integer kernels (measured in Table E), ✕ = needs a backward pass; label shift = the online label-shift stream of Section 3.2 at the protocol batch).

batch 16runs on integer kernelstop-1latency / No Adaptpeak memory (MB)
No Adapt✓47.8447.841.000×1.000\times909909
T3A✓47.95±0.0547.95\pm0.051.465×1.465\times918918
FOA✓50.17±0.2150.17\pm0.212.061×2.061\times916916
FOZO✓49.98±0.2649.98\pm0.262.013×2.013\times916916
ZOA✓51.76±0.2051.76\pm0.201.998×1.998\times911911
NEO✓51.42±0.0251.42\pm0.021.000×1.000\times909909
TENT✕50.26±0.5550.26\pm0.551.890×1.890\times33003300
EATA✕53.63±0.0853.63\pm0.081.933×1.933\times33063306
SAR✕50.25±0.2250.25\pm0.223.777×3.777\times33003300
SPA (batch 4; OOM at 16)✕61.07±0.0761.07\pm0.07; 52.63±4.2452.63\pm4.24 under label shift5.85×5.85\times33503350
QuAR✓54.35±0.01\mathbf{54.35\pm0.01}; 54.73±0.0754.73\pm0.07 under label shift1.084×1.084\times910910

At batch 16 QuAR is the most accurate method that runs on the board, 0.70.7 above EATA at 1.08×1.08\times against 1.93×1.93\times the unadapted latency and 3.6×3.6\times less memory, and 2.62.6 above the best engine-executable baseline (ZOA). SPA, the one method above QuAR in Table F, needs three forward passes, a backward pass and a full-precision predictor per batch, so it has no integer-engine form; at batch 4, its largest completed batch on the board, it costs 5.85×5.85\times the latency and 4.5×4.5\times the memory of No Adapt against QuAR's 1.16×1.16\times and 1.01×1.01\times, and under label shift it falls below QuAR. This table is now in Appendix S.


[Question 1] How much of the advantage remains with comparably stable estimates

All of it, since AQR gains nothing from stable estimates. With pooled estimates for both at AQR's sites, QuAR leads by 3.53.5 (51.1851.18 vs 47.7047.70), because pooling removes AQR's gain (unadapted 47.6947.69), and even AQR's best number, per batch at QuAR's blocks (52.3652.36), stays 2.02.0 below QuAR (54.3554.35). Beyond pooling, placement contributes the +2.2+2.2 of the two post-nonlinearity inputs, which, of the two corrections, only the pooled mean-and-scale match uses (Tables C and D).


[Question 2] Approximation or strict bound; which accuracy conclusions follow

It is an effective-resolution approximation, whose strict form is the entropy bound in our reply to W2 (Appendix V), with h(ε)h(\varepsilon) and εb\varepsilon b kept small by (A1) and Δ2/12\Delta^2/12 as the explicit assumption (A3). What follows for accuracy is directional, the three statements listed there; the magnitudes, including the link to accuracy, are measured, as Section 2.5 now states.


[Question 3] Evidence for integer backends; the trade-off at a batch that fits

Table E, real W8A8 on integer kernels with the statistics update included, reproduces the gains at 1.191.19–1.21×1.21\times latency, GEMM fusion intact, and the TensorRT engine reproduces the ONNX Runtime accuracies on the board (52.2152.21 and 58.7158.71 against 52.2452.24 and 58.7958.79, 15-corruption mean). At the batch that fits (Table F) QuAR is the most accurate method on the board at 1.08×1.08\times the latency and the unadapted memory; SPA, the only more accurate method, completes only up to batch 4 at 5.85×5.85\times and, like every backprop-based method, cannot adapt on an integer engine.

Reviewer_1daUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a highly efficient online method for adapting a quantized ViT by recalibrating the quantizer statistics at run-time, where distribution shifts may occur. The idea is very sensible, i.e., it ensures that the distribution of values is recalibrated to the ranges the original quantized network assumes. The paper proposes to do this in an online fashion: the mean and variance of each incoming batch is computed and (partially) recalibrates the activations that go into the frozen quantizers (of the later blocks and globally for CLS). The results are evaluated fairly thoroughly.

Critical strengths

I quite like the core idea of this paper. It's very sensible to recalibrate activations such that they lie in the expected ranges of the quantizers.

The paper shows quite a few results, which are quite good, and has a rather large evaluation in the appendix.

Critical weaknesses

The writing is unclear in places. Some of it is due to grammer, e.g.,

  • Line 17: "Across these approaches, a quantization-specific failure mode that amplifies the drop is not directly targeted" ?
  • Line 78: "These methods act on either side of the quantizer inputs rather than at them, downstream at the feature or logit level and upstream on prompts or model parameters" ?
  • Line 207 "older batches are down-weighted" but line 210 says: "By default we use a forgetting-free cumulative estimate, [...], so all test batches weigh equally." -> Eq. 2 seems to show exponential decay, so Line 210 can't be true.
  • Section 2.5: I found the entire section difficult to follow.

I appreciate that adapting only later blocks works better, though I can't find an intuitive explanation as to why that is. Based on Table 5, it seems that using up to 7 blocks works ok, but using all 12 blocks leads to a large accuracy drop.

Questions for authors

I would love to hear from the authors why adapting the earliest blocks makes the method effectively fail (Tab. 5 "all 12 blocks").

It would be good to see how the method performs on in-distribution data. Does it maintain the original performance?

I appreciate the online label shift experiment. It's a bit surprising that the method is still stable, given that each batch's statistics presumably change considerably. This should be the method's worst-case scenario. Why isn't it?

2 comments
AuthorsOpenReview ↗

Official Comment by Authors (1/2)

[Weakness 1] Writing

We thank the reviewer for the careful reading. Following the suggestions, we have revised the manuscript to state each flagged passage more clearly.

Line 17. Now reads "None of these approaches directly targets the quantization-specific failure mode that amplifies the drop. Under shift, activations occupy the frozen quantizers' calibrated ranges differently, distorting their code distribution."

Line 78. Now reads "These methods adapt features or logits downstream of the quantizers, and prompts or model parameters upstream of them, but never the quantizer inputs themselves."

Lines 207 and 210 (Eq. 2). Line 210 is correct; the two looked inconsistent because of how line 207 was worded. Eq. 2 is the general update, and its default step αt=1/(nℓ+1)\alpha_t=1/(n_\ell+1) shrinks with every batch, which makes it not an exponential decay but exactly the recursive form of the running mean, μt(n)=μs+∑i=1nμ^in+1,\boldsymbol\mu_t^{(n)}=\frac{\boldsymbol\mu_s+\sum_{i=1}^{n}\widehat{\boldsymbol\mu}_i}{n+1}, so no batch is down-weighted and the source moments count as one pseudo-batch. The same recursion becomes an exponential moving average only with a constant α\alpha, which our paper uses only for continually evolving domains (Section 3.2); line 207 described that variant's behaviour as if it were the default. The revision drops "down-weighted" from line 207 and adds the closed form of Eq. 2 under its default αt\alpha_t (the running mean above) right after the equation; the constant-α\alpha EMA remains the continual-stream choice of Section 3.2.

Section 2.5. We restructured it. The claims are unchanged, but the section now runs as three numbered steps, each one claim and one measurement, with the three "why" paragraphs after them. (1) Shift displaces the per-channel operating point at the quantizer input. (2) On a frozen grid a displaced point forfeits code resolution. (3) The quantization error, unchanged by the shift, flips more decisions because the shift shrinks the margin. The three "why" paragraphs (late blocks, low precision, partial match) then draw the design choices from those steps, and the assumptions are listed once at the top. Proposition 1 stays in the body, restated as an effective-resolution statement under the assumptions listed at the top of the section, the added (A3) among them; the formal statements and proofs stay in Appendix V.

AuthorsOpenReview ↗

Official Comment by Authors (2/2)

[Weakness 2] Why adapting the earliest blocks fails

Our paper gives the reason in Section 2.5, under "Why the late blocks", and Section 3.4 ties it to Table 5, but in two sentences and only from the cost side, which makes it easy to miss. We now add two measurements that make both sides concrete, block by block (Table A).

Table A. Per block, No Adapt, W6A6, seed 0 (mismatch relative to the source scale; amplification measured on clean ImageNet; QuAR's blocks in bold).

block01234567891011
mismatch0.230.230.260.260.280.280.320.320.260.260.240.240.230.230.200.200.21\mathbf{0.21}0.25\mathbf{0.25}0.35\mathbf{0.35}0.22\mathbf{0.22}
channels losing at least half a bit (%)7.07.07.27.26.06.07.37.36.06.08.98.98.98.910.510.517.0\mathbf{17.0}30.7\mathbf{30.7}24.0\mathbf{24.0}10.8\mathbf{10.8}
residual amplification (quantized path)4.784.783.973.975.725.725.605.604.634.633.853.852.962.962.282.281.67\mathbf{1.67}1.13\mathbf{1.13}0.82\mathbf{0.82}0.43\mathbf{0.43}

The early blocks have no more mismatch to undo than the late ones, and far less of the quantization-specific kind (code-resolution loss). The mismatch is the same size everywhere (0.230.23–0.320.32 at blocks 0–3, 0.210.21–0.350.35 at 8–11), while the channels whose contraction forfeits code resolution (Proposition 1) concentrate in blocks 8–10 (1717–3131% against 66–77% at blocks 0–4).

And correcting the early blocks costs the most. A correction is never exact, since its scale is a finite-sample estimate (Proposition V.2), and the residual reaches the logits amplified by the path after it, K>ℓK_{>\ell} (Corollary V.1). Through the quantized path that amplification is 4.784.78 at block 0 against 0.430.43 at block 11, an 11×11\times spread; the full-precision composition of Fig. 6 spans the same ratio at an eighth of the level (0.610.61 to 0.0540.054), so the re-rounding by every downstream quantizer lifts the whole profile, and only on the quantized path is an early residual genuinely amplified.

New runs fill Table 5's gap between 7 blocks and all 12 and show the consequence block by block (Table B).

Table B. Trailing-block depth sweep (Local-only, W6A6, 3 seeds; deployed scope in bold).

trailing blocks4 (8–11)7 (5–11)8 (4–11)9 (3–11)10 (2–11)11 (1–11)12 (0–11)
top-153.42±0.04\mathbf{53.42\pm0.04}53.35±0.0253.35\pm0.0251.86±0.0651.86\pm0.0647.35±0.0947.35\pm0.0942.00±0.0442.00\pm0.0435.88±0.1235.88\pm0.1231.93±0.0731.93\pm0.07

Accuracy is flat down to block 5, where the amplification is still below 44, and from block 4 on every added block costs accuracy (−1.5-1.5, −4.5-4.5, −5.4-5.4, −6.1-6.1, −4.0-4.0 points). Each added early block injects a residual amplified 44–66 times into everything after it. "All 12 blocks" is the end of this curve, not a different regime. The revision states both sides in that paragraph of Section 2.5 and adds both tables to Appendix G.


[Question 1] Earliest blocks

Answered under W2 above. The early blocks have far less code-resolution loss to undo, and a residual placed there is amplified about 11×11\times more through the quantized path than at the last blocks, so every added early block costs accuracy (Table B).


[Question 2] In-distribution data

Yes, the original performance is maintained to within half a point, as our paper reports in Appendix N (Table 29) and summarises in the conclusion. On clean ImageNet validation (the same 5k subset) the deployed configuration changes accuracy against No Adapt by −0.14-0.14 (W3A3), +0.08+0.08 (W4A4), −0.08-0.08 (W6A6) and −0.42-0.42 (W8A8).


[Question 3] Why online label shift is not the worst case

Because QuAR corrects each batch with a running mean pooled over the whole stream so far, not with that batch's statistics, as Section 3.2 of our paper notes. The mean is source-initialised and weighs every batch equally, so a class-skewed batch contributes only its share of an average that converges to the all-class moments of the whole stream; the local branch, whose output feeds the remaining quantizers, also leaves the current batch out of its own correction, and λ=0.5\lambda=0.5 halves whatever bias remains. The scenario the reviewer expects is what happens when the pooling is removed. With the moments of the batch at hand (Table 14), accuracy under label shift falls from 54.7254.72 to 42.3942.39 at imbalance ratio 10310^3, from 54.8254.82 to 23.8923.89 at 5×1055{\times}10^5, and from 54.2754.27 to 38.1138.11 at batch size 1. The deployed estimator also matches a two-pass oracle that sees the whole stream first (54.7254.72 vs 54.7454.74 at 10310^3, 54.8254.82 vs 54.6654.66 at 5×1055{\times}10^5), so under label shift the running estimate already does as well as moments taken from the whole stream.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their constructive feedback.

We have addressed each point with clarifications and substantial new experiments during the rebuttal period: (i) real integer-kernel inference results, (ii) a prediction-level analysis of the quantization-induced errors, (iii) controlled comparisons with AQR, (iv) a per-block analysis of where recalibration should act, and (v) the Jetson trade-off at the largest batch each method fits, eight new tables (A–H) given in the individual replies. Following the reviewers' suggestions, we also made the assumptions of Proposition 1 explicit, restructured Section 2.5 into three steps, clarified the online statistics of Eq. 2, and qualified the batch-size statement of the deployment claim.

Detailed responses follow for each reviewer; the new tables are added to the revised manuscript, all changes are marked in blue, and all section, table and appendix numbers below refer to the revised manuscript.

Dear Reviewer,

I hope this message finds you well. I am writing to kindly remind you that the authors’ rebuttal has been posted and is now available for your review in the system.

We would greatly appreciate it if you could review the rebuttal and provide your assessment at your earliest convenience, as your input is important for us to move forward with a timely decision.

Thank you very much for your time and continued support.

ACs

As only 24 hours remain in the author rebuttal period, I sincerely invite the reviewers to review the author's response and promptly raise any remaining questions or indicate whether you believe the author has addressed the issues you have raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.