acceptodds
Under review as a conference paper at ICLR 2027

Inverse Affine Rectification for Low-Bit Post-Training Quantized Vision Transformers

Abstract

Low-bit post-training quantization (PTQ) enables efficient deployment of vision transformers but can leave systematic distortions in the outputs of the resulting quantized model. We introduce Inverse Affine Rectification (InvAR), a simple post-hoc output rectification method for fixed quantized models that requires no rerunning of quantization optimization. InvAR uses paired logits from the full-precision (FP) and quantized (Q) models on a small set of unlabeled calibration images. The key idea is to model quantization-induced output distortion as a forward mapping from FP logits to Q logits. Rather than directly predicting FP logits from Q logits, InvAR estimates this distortion with a classwise affine model and analytically inverts it to rectify the quantized outputs. This distinction is important because, in the presence of residual variation, the least-squares Q-to-FP predictor generally differs from the inverse of the fitted FP-to-Q affine map. Using identical checkpoints and calibration images, vanilla InvAR and direct regression yield average Top-1 accuracy changes of +1.33 and percentage points, respectively, across PTQ methods, ViT architectures, and bit widths. Direct inversion, however, can amplify estimation errors when calibration data are limited, especially for affine slopes near zero. InvAR therefore uses uncertainty-guided shrinkage to reduce classwise deviations from a shared affine reference before inversion. Once paired logits are collected, fitting is non-iterative and requires operations for calibration images and classes. At inference, InvAR adds only one multiplication and one addition per prediction output. Experiments on ImageNet-1K and COCO demonstrate overall improvements across multiple PTQ methods, ViT and CNN architectures, bit widths, calibration budgets, and vision tasks. Our code will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Post-training Quantization, Model Compression

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_yQhTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This work proposes InvAR, a post-hoc output rectification method for low-bit quantized vision models. Instead of directly regressing full-precision (FP) logits on quantized (Q) logits, InvAR estimates a classwise FP-to-Q affine distortion and analytically inverts it. To stabilize inversion with limited calibration data, it shrinks unreliable classwise deviations toward a shared affine reference. Experiments on ImageNet-1K show consistent accuracy gains when InvAR is applied to models quantized with FIMA-Q, LS-ViT, and APHQ-ViT, with additional results on CNNs and COCO.

Critical strengths
  1. Novel perspective on output rectification. In prior work, output-level rectification has been performed in the Q-to-FP direction. Reversing this idea and performing the rectification in the FP-to-Q direction is a novel perspective.

  2. Compatibility with existing PTQ methods. The proposed method can be combined with other PTQ methods.

Critical weaknesses
  1. Missing theoretical justification for FP-to-Q inversion. While the paper empirically shows that estimating the FP-to-Q distortion and inverting it outperforms direct Q-to-FP regression, it provides no theoretical justification for this advantage.

  2. Missing comparisons with existing rectification methods. The authors apply vanilla InvAR and InvAR to models quantized with FIMA-Q, LS-ViT, and APHQ-ViT, but the only competing baseline is ForwardAffine, a direct Q-to-FP regression constructed by the authors. However, CAT already rectifies the outputs of quantized models in a post-hoc manner, and quantization error compensation methods such as QwT and QwT-v2 can also serve as baselines for output rectification when applied to the final head outputs ZqZ^q. To demonstrate the effectiveness of InvAR, the authors should apply these methods to the same quantized models and compare them directly with InvAR.


Minor weakness

Presentation and clarity. Overall, the manuscript does not feel well organized, which makes it difficult to follow. First, the caption of Figure 1 is too long, and the figure itself is so small that the caption asks readers to zoom in. Second, Section 3.3 introduces several statistical components, such as HC3 variance estimates and orthogonal residual shrinkage (HC3-ORS), in a dense manner without sufficient intuition, and key details are deferred to Appendix A. Third, there is a typographical error in the Introduction ("Sections 4.2 and 4.2"). More broadly, the writing reads more like a technical report than a research paper. The method is described largely through procedural steps and implementation rules, such as the slope floor and the numerical thresholds in Appendix A, while the motivation behind the design choices in Section 3.3 is discussed only briefly. I encourage the authors to revise the manuscript to improve its overall organization and readability.

Questions for authors

Q1. The study appears to be overly focused on rectifying class logits (in image classification, and object detection and instance segmentation). However, I think that extending quantization methods to dense prediction tasks is a key challenge in recent quantization research. Can the authors generalize the proposed method to dense prediction tasks?

Q2. Can the authors provide a theoretical justification for why the FP-to-Q approach necessarily outperforms the Q-to-FP approach?

Q3. Can the authors apply CAT, as well as QwT and QwT-v2 applied to the final head outputs ZqZ^q, to models quantized with FIMA-Q, LS-ViT, and APHQ-ViT, and evaluate the effectiveness of InvAR through a direct comparison with these methods?

Q4. The experimental settings for Table 11 do not seem to be specified in detail. I would ask the authors to clarify them (e.g., bit-width).

7 comments

Response 1/3: Theoretical justification for FP-to-Q inversion (W1/Q2)

Thank you for the careful review and for recognizing the FP-to-Q perspective as novel. References are to the revised manuscript; items marked new were added in the revision.

W1 / Q2. The advantage of FP-to-Q inversion is shown only empirically. Why does it necessarily outperform Q-to-FP regression?

Short answer. The revision now derives how the two directions differ for classification and states sufficient conditions under which inversion preserves decisions that direct regression changes. This is not a universal guarantee: if all classes share the same rcr_c and FP calibration mean, both maps rank classes identically, and with very scarce data unshrunk inversion is noisier, which InvAR's shrinkage addresses (Section 3.3). In the evaluated settings, the experiments below consistently favor inversion.

Theory. Beyond the slope attenuation in the original Section 3.2, the new Proposition 1 establishes the exact finite-sample relation

T^cDir(z)=rc2T^cInv(z)+(1−rc2)Zˉcfp\hat T_c^{\mathrm{Dir}}(z)=r_c^2\hat T_c^{\mathrm{Inv}}(z)+(1-r_c^2)\bar Z_c^{\mathrm{fp}}

for OLS with intercepts in both directions, nonzero sample variances, and no regularization, slope clipping, or residual-distribution assumption. Here rc>0r_c>0 is the paired-logit sample correlation and Zˉcfp\bar Z_c^{\mathrm{fp}} the FP calibration mean. Direct regression thus replaces a fraction 1−rc21-r_c^2 of each class's image-specific inverse output with that class's calibration mean. Classification depends on cross-class rankings, and because rcr_c varies across classes, this contraction can reorder classes even while reducing logit MSE. Appendix A.1, Eqs. (9)-(10), gives sufficient conditions under which inversion preserves a correct FP decision that direct regression changes.

Empirical evidence (identical checkpoints and calibration images):

  • Direct regression reduces accuracy in all 84 second-order ViT configurations, and vanilla InvAR outperforms it in each (Table 10).
  • Direct regression has lower validation FP-logit MSE than inversion (0.73 vs. 1.32) but a lower mean Top-1 gain (-1.04 vs. +1.37 pp; new Table 11).
  • Among the three same-form maps in new Table 3, mean gains rise as contraction decreases: -1.04 pp (rc2r_c^2, direct regression), +0.74 pp (rcr_c, mean-variance matching), and +1.37 pp (no contraction, vanilla InvAR).

Response 2/3: Comparisons with CAT, QwT, and QwT-v2 on the same models (W2/Q3)

W2 / Q3. The only baseline, ForwardAffine, is constructed by the authors. Compare InvAR directly with CAT, and with QwT and QwT-v2 applied to the final outputs, on the same quantized models.

As requested, new Table 3 applies CAT, QwT, and QwT-v2 to the final outputs of 27 configurations, each run with three random seeds: FIMA-Q, LS-ViT, and APHQ-ViT on DeiT-S, ViT-S, and Swin-S at W2/A3, W3/A3, and W4/A4. All rectifiers share the frozen checkpoints and the original 1,024 unlabeled calibration images. Entries are mean Top-1 gains (pp) over the unrectified checkpoints; Foldable indicates whether a rectifier folds into a compatible quantized head without changing integer weights (Section 3.4).

Output rectifierW2/A3W3/A3W4/A4MeanExtra paramsFoldable
CAT (K=64)-1.44-1.24-0.63-1.103KC3KCNo
QwT-logit-1.30-0.79-0.18-0.76C2+CC^2{+}CNo
QwT-feature+1.27+0.11+0.03+0.47dC+CdC{+}CNo
QwT-v2 (final outputs) = ForwardAffine-2.05-0.88-0.19-1.042C2CYes
Vanilla InvAR+3.25+0.75+0.12+1.372C2CYes
InvAR+3.25+0.75+0.11+1.372C2CYes

Both InvAR variants have higher mean gains than all external baselines in Table 3 at every bit-width. The strongest competitor, QwT-feature, additionally uses head-input features, stores about 190-385x more rectification parameters (d=384d=384 or 768768), and is not foldable, yet gains 0.90 pp less on average.

Why ForwardAffine is a meaningful baseline. We implement ForwardAffine ourselves, but it corresponds to existing output rectifiers: within the input-independent classwise affine form used throughout the paper (closed-form fitting, analytic inversion, 2C2C coefficients; Section 3.1, Appendix G), it is the classwise least-squares fit inside CAT and coincides with QwT-v2's channelwise compensation applied to final outputs, so the two share one row above. Comparing it with inversion therefore isolates the fitting direction while holding form, parameters, and data fixed.

Adaptations (Appendix C.2). With the official CAT implementation, every tested K/PCA setting reduces mean accuracy, including K=1 (Table 9; LS-ViT on three architectures). As you suggested, all methods act on the final outputs, and our conclusions are limited to this output-level setting.

Response 3/3: Presentation, dense prediction (Q1), and detection settings (Q4)

Presentation. Figure 1 is too small and its caption too long; Section 3.3 is dense and lacks intuition; the paper reads like a technical report; and there is a typo.

We revised the manuscript along the four points you raised:

  • Figure 1: now full-width, with a shorter caption that no longer asks readers to zoom in.
  • Section 3.3: before Eqs. (4)-(6), it now explains why near-zero slope estimates destabilize inversion, why the shared reference preserves the Q model's rankings, and how estimated uncertainty controls retention; the HC3-ORS acronym is removed.
  • Organization: Section 3 presents motivation before procedure, with the slope-attenuation derivation moved to Appendix A; Section 4 is organized around the research questions, with experiments grouped under each.
  • Typo: the duplicated "Sections 4.2 and 4.2" is corrected.

Q1. Can the method be generalized to dense prediction tasks beyond class logits?

In the two settings we tested, yes:

  • Pixelwise semantic segmentation (Section 4.6; Appendix F, Table 20). For RepQ-ViT-quantized Swin-T/S UPerNet on ADE20K, InvAR is fitted on 256 unlabeled images with variances estimated per image, because pixels within an image are correlated (Appendix F.2). InvAR improves mIoU at W4/A4 by 2.08/1.05, whereas direct regression reduces it by 5.81/4.84. Both inverse variants outperform the quantized model and direct regression on all 12 checkpoints (two models, W4/A4 and W6/A6, three seeds).
  • Continuous outputs beyond class logits (Appendix E, Table 19). For W4/A4 Mask R-CNN (LS-ViT and APHQ-ViT; Swin-T/S), rectifying raw box offsets alone improves box AP more than rectifying class logits alone in all four configurations (+0.8 to +2.0 vs. +0.3 to +1.1), and rectifying both is best (+1.2 to +3.0 box AP, +0.6 to +1.6 mask AP).

Q4. Please specify the experimental settings of Table 11 (e.g., bit-width).

Original Table 11 is now Table 18. All models are W4/A4 Mask R-CNN or Cascade Mask R-CNN with Swin-T/S backbones, quantized with LS-ViT, FIMA-Q, or APHQ-ViT; "+Joint" denotes InvAR on both classification and box heads. InvAR is fitted on 256 unlabeled COCO train2017 images and evaluated on all 5,000 val2017 images, with detector weights and quantization parameters fixed. FP and Q outputs are paired on the Q model's proposals; Appendix E gives the pairing for both detectors.

We hope the revision addresses your concerns, and we would appreciate your taking it into account in your assessment. If any question remains, we would be glad to address it and further improve the manuscript.

Reviewer_AY4fOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a post-hoc output rectification method for quantized model, called Inverse Affine Rectification (InvAR). InvAR models quantization-induced output distortion as a forward mapping from full precision logits to quantized logits. It also proposes uncertainty adaptive shrinkage to move unreliable correction parameters toward a shared, more stable correction. Experimental results on ViT and CNN demonstrates overall improvement across classification, object detection and segmentation tasks.

Critical strengths
  1. InvAR is simple yet effective. Instead of directly learning a Q-to-FP mapping, InvAR models the quantization-induced distortion through FP-to-Q regression and analytically inverts the fitted affine transformation. The authors theoretically show that direct Q-to-FP regression suffers from slope attenuation under residual noise, while the inverse mapping avoids this attenuation. The experimental results further support this design choice, demonstrating consistent Top-1 accuracy improvements over direct regression.

  2. The uncertainty-adaptive shrinkage provides a principled way to improve the stability of inverse regression. Since inversion can amplify estimation errors when classwise slopes approach zero, InvAR adaptively shrinks uncertain class-specific parameters toward a shared affine reference. This design reduces the risk of unstable corrections while retaining reliable class-specific adjustments. Moreover, the shared reference provides a ranking-preserving fallback, making the method particularly appealing when calibration data are limited.

Critical weaknesses

1.Classwise affine logit transformations are already established in post-hoc calibration methods. Regression inversion and shrinkage toward pooled estimates are well-established statistical techniques. The paper should more clearly distinguish its method from existing logit calibration and regularized regression approaches.

Questions for authors

The proposed approach builds on established ideas, including classwise affine calibration, regression inversion, and statistical shrinkage. While their combination appears effective for post-hoc quantization correction, the paper could better clarify its novelty relative to existing calibration and regularized regression techniques.

3 comments

Response: Novelty relative to logit calibration and regularized regression

Thank you for recognizing the effectiveness of inversion and uncertainty-adaptive shrinkage. Below, we clarify these distinctions with references to the relevant sections, tables, and equations in the revised manuscript.

W1 / Q. Classwise affine logit maps, regression inversion, and shrinkage toward pooled estimates are established. How does InvAR differ from existing logit calibration and regularized regression?

Our contribution is to formulate quantization-output rectification as distortion estimation and inversion, and to stabilize that inversion under limited calibration data. Our approach builds on established statistical techniques.

  1. Why estimate distortion. Existing output rectifiers (CAT; QwT-v2 at final outputs) regress FP logits on Q logits to minimize reconstruction error, which need not yield better task performance. We instead estimate the FP-to-Q distortion and invert it to correct for its affine component. Our logit analysis shows that the direct fit contracts classes unequally and can thereby reorder them.
  2. What this choice requires. Inversion divides by an estimated slope and becomes unstable with few calibration images; InvAR's shrinkage is designed for this instability.

Fitting direction. For unregularized OLS fits with positive paired-logit correlation rcr_c, Proposition 1 (Section 3.2; added in this revision) gives the finite-sample identity

T^cDir(z)=rc2T^cInv(z)+(1−rc2)Zˉcfp.\hat T_c^{\mathrm{Dir}}(z)=r_c^2\hat T_c^{\mathrm{Inv}}(z)+(1-r_c^2)\bar Z_c^{\mathrm{fp}}.

Direct regression thus contracts each class toward its own FP calibration mean by a class-dependent factor, which can reorder classes. On identical checkpoints and calibration images, direct fitting attains lower FP-logit MSE yet lower accuracy than inversion (Table 11; added in this revision). Among the three same-form maps in Table 3 (added in this revision), mean gains rise as contraction decreases (-1.04, +0.74, and +1.37 pp).

Relation to logit calibration.

MethodDataGoalFitted map
Temperature scalingLabeled validationConfidence (NLL)Shared positive scale; Top-1 unchanged
Vector scalingLabeled validationConfidence (NLL)Classwise affine on logits
Dirichlet calibrationLabeled validationConfidence (NLL)Full-matrix affine on log-probabilities
CAT; QwT-v2 at final outputsUnlabeled FP/Q pairsRecover FP outputsDirect Q-to-FP
InvARUnlabeled FP/Q pairsAffine-distortion correctionFP-to-Q distortion, then inverse

A shared positive map, as in temperature scaling, cannot change Top-1 accuracy. InvAR's fully shrunk reference is exactly such a map and gives zero gain at every budget (Table 13; added in this revision), so InvAR's gains come from class-specific distortion estimates.

Relation to regularized regression. Direct regression never divides by a fitted slope; inversion does, through ac=1/α~ca_c=1/\tilde\alpha_c, which adds an instability specific to near-zero slope estimates. Consistent with this, in six LS-ViT configurations without shrinkage (Table 12; added in this revision), inversion loses more than direct fitting at N=32N=32 (-21.84 vs. -10.91 pp) but overtakes it at N=1024N=1024 (+1.86 vs. -1.42 pp). InvAR therefore adapts shrinkage to inversion in three ways (Section 3.3):

  • What is shrunk. Ridge toward zero on forward slopes would amplify 1/α^c1/\hat\alpha_c, and on direct Q-to-FP slopes would add to their contraction. InvAR instead shrinks centered scale and mean deviations of the forward FP-to-Q distortion before inversion, so unreliable slopes move toward a positive pooled slope.
  • Toward what. The center is a pooled distortion whose inverse is a shared positive affine map, so full shrinkage leaves Q decisions unchanged, while partial shrinkage retains reliable class-specific rectification.
  • How much. Retention is set per class and component from HC3 heteroskedasticity-robust variances, without validation tuning or cross-validated penalties.

In our comparison, ridge-regularized QwT-logit method yields a smaller mean accuracy gain than InvAR (-0.76 vs. +1.37 pp; Table 3, added in this revision). Within InvAR, class-matched retention outperforms shuffled retention (Table 13, added in this revision), and the choice of variance estimator matters: on LS-ViT-quantized DeiT-S at W3/A3 and N=32N=32, accuracy changes are -21.72 pp without shrinkage, -2.80 pp with homoskedastic shrinkage, and +0.17 pp with HC3 shrinkage. The latter two settings differ only in the variance estimator (Table 14).

We hope this clarifies the novelty. If any question remains, we would be glad to address it and further improve the manuscript.

Thank you for the response. My concern has been adequately addressed. I will keep the initial positive score.

We’re glad our response clarified InvAR’s contribution and its connection to established statistical methods. Thank you for raising this question.

Reviewer_Vo2KOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Inverse Affine Rectification (InvAR), a lightweight post-hoc output correction method for post-training quantized models. Using paired full-precision and quantized logits from a small unlabeled calibration set, InvAR estimates classwise affine distortion in the FP-to-Q direction and analytically inverts the fitted mapping, rather than directly regressing FP outputs from Q outputs. To mitigate unstable inversion under limited calibration data, the method combines HC3 variance estimates with empirical Bayes shrinkage, shrinking unreliable class-specific deviations toward a shared affine reference. Fitting requires neither backpropagation nor rerunning quantization optimization, while deployment applies only classwise scaling and shifting. Experiments on ImageNet-1K and COCO cover multiple quantization methods, architectures, and calibration budgets, reporting improvements in classification, object detection, and instance segmentation.

Critical strengths
  1. The method is lightweight and practically useful. InvAR requires only a small unlabeled calibration set and corrects the outputs of a frozen quantized model without rerunning quantization optimization or performing backpropagation. Its small parameter count and low inference overhead make it easy to combine with existing PTQ methods.

  2. The controlled comparison of fitting directions is insightful. Using the same quantized models, calibration data, and classwise affine parameterization, the authors compare direct Q-to-FP regression with FP-to-Q fitting followed by inversion. The substantial difference in classification performance demonstrates that the choice of estimation target can significantly affect accuracy even when the deployment form remains identical.

  3. The broad experimental coverage supports the method’s applicability. The experiments cover multiple PTQ methods, ViT and CNN architectures, bit widths, and calibration budgets, and extend to object detection and instance segmentation. Paired comparisons before and after correction on the same frozen checkpoints, together with results across multiple random seeds, strengthen the empirical evidence for the reported gains.

Critical weaknesses
  1. The theoretical analysis of fitting direction does not sufficiently explain the advantage in classification performance.

    Section 3.2 models the distortion from full-precision (FP) outputs XX to quantized (Q) outputs YY as Y=αX+β+εY=\alpha X+\beta+\varepsilon, and shows that the slope of direct Q→FP regression is generally smaller than the inverse scale 1/α1/\alpha. However, this result primarily establishes that the two estimation targets differ; it does not imply that direct regression is biased with respect to its own prediction target. One could likewise formulate a direct prediction model X=α′Y+β′+ηX=\alpha'Y+\beta'+\eta and accurately estimate its population regression coefficient. Could the authors further explain why treating the FP→Q affine relationship as a distortion to be inverted is more beneficial for classification than directly predicting FP outputs?

  2. The comparison of fitting directions does not sufficiently address the influence of the loss function.

    Section 4.3 shows that InvAR remains effective when output-distribution or ranking supervision is incorporated into quantization reconstruction. However, this experiment does not replace a direct comparison of different post-hoc affine correction objectives on the same frozen model. Given that methods such as APHQ use KL divergence between output distributions for classification distillation, could the authors compare the following four settings: direct Q→FP regression with MSE, FP→Q fitting with MSE followed by inversion, direct Q→FP regression with KL divergence, and FP→Q fitting with KL divergence followed by inversion?

  3. The affine correction can be folded into the quantized classification head; its deployment advantages could be stated more precisely.

    The abstract describes InvAR’s inference overhead as one additional multiplication and one addition per output. However, when the classification head uses per-output-channel uniform quantization and retains a floating-point bias, the classwise correction zccorr=aczcq+bcz_c^{corr}=a_cz_c^q+b_c can be implemented by updating the quantization scale and head bias: sc′=acsc,dc′=acdc+bc.s'_c=a_cs_c,\qquad d'_c=a_cd_c+b_c. This folding leaves the integer weights unchanged, requires no additional weight rounding, and eliminates the need for a separate output-correction operator.

    I suggest that the authors explicitly describe this equivalent implementation. If supported, the paper could make the stronger point that InvAR can be absorbed into the existing scale and bias computations of the classification head without additional inference operators, rather than merely introducing a small computational overhead. This would better communicate the method’s deployment advantages.

  4. The trade-off between a more general matrix correction and requantization after folding merits investigation.

    The current method restricts output distortion to a classwise diagonal affine transformation and does not explicitly model cross-class interactions. Could the authors investigate a more general matrix affine model, fit and invert it, fold the correction into the final classification head, and then requantize the head at its original bit width? Provided that the head retains the same shape and execution scheme, this approach would likewise require neither an additional output-correction operator nor separate storage of the correction matrix. Unlike a diagonal transformation, however, a general matrix transformation typically cannot be implemented solely by updating channelwise scales: it mixes weight rows and requires requantization, potentially introducing additional quantization error.

    I suggest comparing three settings: the current diagonal InvAR, a matrix correction retained as a separate high-precision transformation, and the same matrix correction folded into the head and requantized. This would distinguish the benefits of cross-class modeling from the cost of requantization, establish whether greater expressiveness can outweigh parameter-estimation and requantization errors, and further substantiate the choice of a diagonal transformation.

Questions for authors

Please refer to the weaknesses above.

4 comments

Response 1/2: Fitting direction, classification, and objective (W1-W2)

Thank you for the detailed and constructive review. The revision adopts all four suggestions. References are to the revised manuscript.

W1. Slope attenuation shows only that the two estimation targets differ, not that direct regression is biased for its own target. Why is inverting the FP-to-Q distortion better for classification?

We agree that, for i.i.d. calibration images, direct Q-to-FP regression consistently estimates the best affine predictor for its own MSE target; Appendix A.1 now states this explicitly. Our claim concerns accuracy: lower per-class reconstruction error need not yield higher accuracy, because classification depends on cross-class rankings.

Proposition 1 (Section 3.2; added in this revision) shows how the two maps differ. For OLS with intercepts in both directions, nonzero sample variances, and positive paired-logit correlation rcr_c (no regularization, shrinkage, slope clipping, or residual-distribution assumption),

T^cDir(z)=rc2T^cInv(z)+(1−rc2)Zˉcfp,\hat T_c^{\mathrm{Dir}}(z)=r_c^2\hat T_c^{\mathrm{Inv}}(z)+(1-r_c^2)\bar Z_c^{\mathrm{fp}},

where Zˉcfp\bar Z_c^{\mathrm{fp}} is the FP calibration mean. Direct regression thus replaces a fraction 1−rc21-r_c^2 of each class's image-specific inverse output with that class's calibration mean. Because rcr_c varies across classes, this contraction is class-dependent and can reorder classes; if all classes shared the same rcr_c and FP calibration mean, the two maps would rank classes identically. Appendix A.1, Eqs. (9)-(10), gives sufficient conditions: inversion preserves a correct FP decision when every FP margin exceeds the sum of the two classes' inversion-error bounds, whereas direct regression changes it when its class-dependent contraction reduces some margin by more than that margin plus the rc2r_c^2-scaled bounds. Outside these conditions, either map can be more accurate, so which prevails is an empirical question:

  • Direct regression reduces accuracy in all 84 second-order ViT configurations, while vanilla InvAR is better in every one (Table 10).
  • Among the three same-form maps in Table 3 (added in this revision), mean gains increase as contraction decreases: -1.04 (rc2r_c^2, direct regression), +0.74 (rcr_c, mean-variance matching), and +1.37 pp (no contraction, vanilla InvAR).

W2. Output supervision during PTQ does not replace a post-hoc objective comparison. Compare direct and inverted fitting under MSE and KL on the same frozen models.

We agree. Appendix C.3 (added in this revision) compares fitting direction and objective on the same frozen checkpoints and calibration images. The evaluation covers 27 configurations, each run with three seeds: FIMA-Q, LS-ViT, and APHQ-ViT on DeiT-S, ViT-S, and Swin-S at W2/A3, W3/A3, and W4/A4. With pT(v)=softmax(v/T)p_T(v)=\mathrm{softmax}(v/T) and an affine map FF, direct KL minimizes DKL(pT(Zfp)∥pT(F(Zq)))D_{KL}(p_T(Z^{fp})\|p_T(F(Z^q))) and deploys F(Zq)F(Z^q); inverse KL minimizes DKL(pT(Zq)∥pT(F(Zfp)))D_{KL}(p_T(Z^q)\|p_T(F(Z^{fp}))) and deploys F−1(Zq)F^{-1}(Z^q). All four settings share the same offset constraint and slope floor, and KL fits are solved to convergence. Mean results at T=1T=1 (Table 11):

Fitting/deploymentTop-1 gain over Q (pp)Validation FP-logit MSE
Direct MSE-1.040.73
Inverse MSE+1.371.32
Direct KL-3.693.42
Inverse KL-2.121.63

Inversion yields higher mean accuracy under both objectives. Direct MSE has lower FP-logit MSE but lower accuracy than inverse MSE, so better reconstruction need not improve classification. MSE also outperforms KL at T=1T=1 in both directions. In a sensitivity study (Table 12, added in this revision; six LS-ViT configurations at W2/A3 and W3/A3), inversion outperforms direct fitting under MSE and all three KL temperatures at N=256N=256 and 10241024; at N≤128N\le128, unshrunk inversion can fall behind, which is the instability InvAR's shrinkage addresses (Section 3.3).

On KL versus MSE. In both directions the fitted map enters the second KL argument, so the gradient for class cc's coefficients is weighted by the difference between its two softmax probabilities. That difference is negligible on images where both arguments assign the class little mass. With a sharp 1,000-class softmax at T=1T=1, many images per class can fall in this regime, whereas MSE weights every calibration image equally. Consistent with this, both KL directions improve with TT at every budget in Table 12. KL distillation during PTQ reconstruction, as in APHQ, instead updates weights shared by all classes.

Response 2/2: Head folding and matrix rectification (W3-W4)

W3. The classwise correction can be folded into the head's quantization scale and bias without changing integer weights; please state this equivalent implementation.

We agree; the submission reported only the standalone cost (2C2C coefficients and one multiply-add per class), which understates InvAR's deployment efficiency on compatible heads. The abstract, introduction, Figure 2, Section 3.4, and Section 4.5 now state the folded form: with sc′=acscs'_c=a_cs_c and dc′=acdc+bcd'_c=a_cd_c+b_c, a per-output-channel quantized head with floating-point bias and no incompatible subsequent output quantization computes the rectified logits exactly, leaving integer weights and zero points unchanged and requiring no new weight rounding or additional inference operators. As with batch-normalization folding, this is an exact reparameterization; in our experiments, folded and standalone heads give identical predictions.

W4. Compare diagonal InvAR, a standalone high-precision matrix correction, and the same matrix folded into the head and requantized, to separate cross-class modeling from requantization cost.

Added in this revision. Appendix C.6 reports the suggest comparison on the same 27 configurations at N=1024N=1024. All variants fit the FP-to-Q distortion and deploy its inverse. The diagonal map is InvAR's classwise form used throughout the paper, Zcrec=acZcq+bcZ^{\mathrm{rec}}_c=a_cZ^{\mathrm q}_c+b_c, fitted here without shrinkage; the full matrix replaces the per-class slopes with a C×CC\times C slope matrix (C2+CC^2+C coefficients versus 2C2C). The requantized variant keeps the original head bit-width and uses each PTQ method's own weight-scale search and rounding. Mean Top-1 gains over the unrectified checkpoints (Table 15):

Deploy formMean gain (pp)
Diagonal+1.37
Standalone full-matrix+1.71
Full matrix, folded and requantized+1.10

The results confirm the trade-off you anticipated. Kept in high precision, cross-class modeling improves the mean by 0.34 pp over the diagonal map. However, requantizing to the original head bit-width costs 0.61 pp and leaves the matrix 0.27 pp below the diagonal map. These results support the diagonal map when the rectifier fold into a head at its original bit-width, while a matrix rectifier is an option when a high-precision head or a standalone C×CC\times C layer is acceptable. Fitting direction matters for the matrix form as well: in six LS-ViT configurations, inverse full-matrix fitting outperforms direct full-matrix fitting under matched fitting rules (+3.44 pp on average; Table 16, added in this revision).

Thank you again for these suggestions, which have strengthened the paper. If any question remains, we would be glad to address it and further improve the manuscript.

Thank you to the authors for their detailed rebuttal and the additional experiments/clarifications provided. The authors have thoroughly addressed all of my concerns and questions. Therefore, I will maintain my initial positive score and strongly recommend the acceptance of this paper.

Thank you for your strong recommendation for acceptance! We especially appreciate your suggestion to include the folded form, which highlights a practical advantage of InvAR.

Reviewer_m6qyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper addresses the accuracy lost after low-bit post-training quantization of vision models and asks whether it can be recovered by correcting only the outputs of the fixed quantized model.
  • It proposes InvAR, which fits a per-class scale and offset that maps full-precision outputs to quantized outputs on a few unlabeled images, shrinks unreliable per-class fits toward a shared one, and applies the inverse map to the quantized outputs.
  • The method achieves consistent accuracy gains across several quantization methods, models and bit widths at almost no cost, with the largest gains at the lowest bit widths.
Critical strengths
  1. The comparison of fitting direction is a clean and useful finding. With the same form and data, directly predicting full-precision outputs always hurts accuracy,
Critical weaknesses
  1. Simple alternative corrections are not compared. Maps such as a per-class offset only or per-class mean and variance matching are missing, so the only baseline is a direct regression built by the authors that always hurts accuracy, and it is unclear whether the inverse fit itself is what helps.
  2. The practical benefit is limited. The gains are large only with 2-bit weights, become small at 3 bits and nearly vanish at 4 bits,
Questions for authors
  1. How does InvAR compare with simple per-class mean and variance matching on the same checkpoints?
  2. How much of the gain comes from the per-class offset alone versus the per-class scale?
4 comments

Response 1/2: Simple rectifiers and the role of inverse fitting (W1, Q1, Q2)

Thank you for highlighting the fitting-direction comparison and for asking for simple baselines. The revision adds every rectifier you suggested, fitted without labels on the same frozen checkpoints and 1,024 calibration images. The evaluation covers 27 configurations, each run with three seeds: FIMA-Q, LS-ViT, and APHQ-ViT on DeiT-S, ViT-S, and Swin-S at W2/A3, W3/A3, and W4/A4. References are to the revised manuscript.

W1 / Q1. Simple corrections such as per-class offset or mean-variance matching are missing, and the only baseline, a direct regression built by the authors, always hurts accuracy. Is the inverse fit itself what helps?

Short answer. InvAR outperforms every simple rectifier, including mean–variance matching (+1.37 vs. +0.74 pp on average; +3.25 vs. +1.85 pp at W2/A3).

Mean-variance matching aligns each class's Q-logit mean and standard deviation with the FP values; offset-only adds the FP-minus-Q mean difference; scale-only fits and inverts a no-intercept FP-to-Q slope. Mean Top-1 gains (pp) over the unrectified checkpoints (Table 3; added in this revision):

Rectifierκc\kappa_cW2/A3W3/A3W4/A4Mean
Offset-only–+0.64+0.09+0.01+0.25
Scale-only–+0.22+0.11+0.01+0.11
Direct regressionrc2r_c^2-2.05-0.88-0.19-1.04
Mean-variance matchingrcr_c+1.85+0.35+0.01+0.74
Vanilla InvAR11+3.25+0.75+0.12+1.37

Why the inverse fit helps. Direct regression, mean-variance matching, and vanilla InvAR share the classwise scale-and-shift form, 2C2C parameters, and data. For positive paired-logit correlation rcr_c, Appendix A.1, Eq. (8) (added in this revision), shows that they differ only in how strongly they contract the inverse map toward each class's FP calibration mean Zˉcfp\bar Z_c^{\mathrm{fp}}:

T^c(z)=κc T^cInv(z)+(1−κc) Zˉcfp,κc∈{rc2, rc, 1}.\hat T_c(z)=\kappa_c\,\hat T_c^{\mathrm{Inv}}(z)+(1-\kappa_c)\,\bar Z_c^{\mathrm{fp}},\qquad \kappa_c\in\{r_c^2,\ r_c,\ 1\}.

Across these three methods in Table 3, mean accuracy increases as contraction decreases at every bit-width. Mean-variance matching is thus a partially contracted inverse, and vanilla InvAR exceeds it by 0.63 pp on average.

Why direct regression is the main baseline. We implement this baseline ourselves, but it corresponds to existing output rectifiers. Within the input-independent classwise affine form used throughout the paper (closed-form fitting, analytic inversion, 2C2C coefficients; Section 3.1, Appendix G), it is the classwise least-squares fit inside CAT and coincides with QwT-v2's channelwise compensation applied to final outputs (Appendix C.2). Comparing it with inversion therefore isolates the fitting direction. Published adapters in their own forms, evaluated on the same checkpoints, reach at most +0.47 pp (Table 3).

Q2. How much of the gain comes from the per-class offset alone versus the per-class scale?

Neither component alone reproduces the joint gain. As standalone rectifiers, offset-only and scale-only gain +0.64 and +0.22 pp at W2/A3, each well below the joint gain (+3.25 pp; Table 3), and on ADE20K offset-only reduces mIoU at W4/A4 (Table 20). Within InvAR (Table 13, added in this revision; W2/A3 and W3/A3), at N=1024N=1024, retaining only the classwise scale deviations from the shared reference gives +1.41 pp, retaining only the mean deviations gives +0.61 pp, and retaining both gives +2.00 pp. Scale deviations thus contribute more at this budget, while mean deviations contribute more at N≤128N\le128.

Response 2/2: Practical value beyond 2-bit weights (W2)

W2. Gains are large only with 2-bit weights, small at 3 bits, and nearly vanish at 4 bits.

InvAR’s practical benefit extends beyond 2-bit weights. On the same quantized models, InvAR achieves larger mean accuracy gains than competing rectifiers across various bit-widths (Table 3). Feature-conditioned QwT likewise shows smaller gains at higher bit-widths. However, InvAR delivers larger gains at W4/A4 with other PTQ methods and on other vision tasks, as the following results demonstrate:

1. At W4/A4, the gain depends on the PTQ method and task.

W4/A4 settingMetricInvAR gainSource
FIMA-Q / LS-ViT / APHQ-ViT, 7 ViTs (ImageNet)Top-1 (pp)+0.10Table 5
RepQ-ViT / AdaLog, 7 ViTs (ImageNet)Top-1 (pp)+0.63 / +0.20Table 7
Mask R-CNN, Swin-T/S, 3 PTQ methods (COCO)box / mask AP+0.9 to +3.0 / +0.2 to +1.6Table 18
UPerNet, Swin-T/S, RepQ-ViT (ADE20K)mIoU+2.08 / +1.05Table 20 (added in this revision)

For RepQ-ViT and AdaLog, every model shows a positive mean gain, reusing their original 32 calibration images.

2. Gains below 4 bits are consistent. InvAR improves accuracy in every W2/A3, W2/A4, and W3/A3 configuration under all three seeds (63 configurations, 189 runs; Section 4.2), with W3/A3 gains up to +1.29 pp. When LS-ViT reconstruction is strengthened with output-distribution or ranking supervision, InvAR still improves every W3/A3 checkpoint, by +0.36 to +0.98 pp (Table 17).

3. The cost is seconds. At W2/A3, output collection and InvAR fitting take an estimated 2.34-5.90 s (RTX 3090) on the same unlabeled calibration images, against 2.32-4.28 h for LS-ViT reconstruction (Table 4), with no backpropagation and no rerun of PTQ. In a compatible quantized head, the rectifier folds into the existing scales and biases without adding inference operators (Section 3.4; added in this revision).

We hope these results address your concerns, and we would appreciate your taking them into account in your assessment. If any question remains, we would be glad to address it and further improve the manuscript.

Thank you for the detailed answers! The new comparisons in the responses including offset-only, scale-only and mean–variance matching on the same checkpoints, address part of my concern (W1, Q1, Q2), and the added results on dense prediction are helpful.

However, I'm still not fully convinced on the theory side. Appendix A.1 shows how the inverse, direct regression and mean–variance matching differ, and when they lead to different decisions, but it doesn't really show why the inverse do better in general. That part still rests on the experiments, and the offset-only correction isn't covered by the analysis.

Before I finalize my score, i would like to consider opinions of other reviewers and AC.

Thank you for acknowledging that the new baseline comparisons address part of your concern and that the dense-prediction results are helpful.

We agree that the analysis does not show inversion to be more accurate in general, and the paper does not claim this. In the manuscript revised after our initial response, Appendix A.1 shows when either inversion or contraction can better preserve the FP ordering of a class pair: under the assumptions of Proposition 2, a map whose margin shift exceeds the FP margin is less likely than inversion to preserve this ordering, whereas a shared contraction with zero margin shift can preserve it more often than inversion. Empirically, with 32 calibration images, vanilla InvAR falls below mean–variance matching and offset-only (Table 8), reflecting the instability of inverting noisy slope estimates that InvAR's shrinkage addresses (§3.3). Conversely, direct regression is optimal for FP-logit MSE, yet this does not carry over to accuracy (Table 11).

What the analysis establishes. Under the affine distortion model, if the residual has zero mean conditional on the class's FP logit and that logit has positive variance, the population inverse is the unique classwise affine correction that recovers the FP logit in conditional mean; if the residuals have zero mean conditional on the full FP vector, it also recovers every pairwise FP margin in conditional mean (Appendix A.1). The population counterparts of the other maps in Eq. (8) can shift these conditional mean margins; Proposition 2 gives the pairwise ranking result stated above.

Offset-only is in the same family. Appendix A.1 ("A common form") writes offset-only with κc=α^c\kappa_c=\hat\alpha_c, the fitted FP-to-Q slope. Offset-only aligns each class's Q mean with its FP mean but leaves the Q scale unchanged, so relative to vanilla InvAR it multiplies the inverse score's deviation from the class's FP mean by α^c\hat\alpha_c. This factor is below one in 80,993 of 81,000 classwise fits and varies across classes in every checkpoint (Appendix A.2). Thus offset-only, like direct regression and mean–variance matching, can pull competing scores toward their respective means by different amounts.

How do classwise scaling factors affect the gaps? For this follow-up, we re-evaluated the 81 checkpoints of Table 3, replacing either the classwise factors or the class means by their average within each checkpoint. The factor acts on the actual inverse score, including its residual error. Entries are mean Top-1 gaps to vanilla InvAR in percentage points, averaged over checkpoints in this re-evaluation (§4.3, Appendix A.2); rcr_c is the paired FP–Q logit correlation for class cc.

Comparison mapκc\kappa_cOriginalShared κ\kappa, class means keptShared class mean, κc\kappa_c kept
Direct regressionrc2r_c^22.410.292.03
Mean–variance matchingrcr_c0.620.090.49
Offset-onlyα^c\hat\alpha_c1.120.130.95

Sharing the factors across classes closes 85–88% of the mean accuracy gap for each method, compared with 15–21% when sharing the means. Even for offset-only, which already matches every class mean, the gap falls from 1.12 to 0.13 pp when its relative scale is shared. The gaps also follow the analysis across bit-widths: by Proposition 1, direct regression coincides with inversion when rc=1r_c=1, and as mean rcr_c rises from 0.69 to 0.80 to 0.90 from W2/A3 to W4/A4, its gap to vanilla InvAR narrows from 5.28 to 1.66 to 0.29 pp (Appendix A.2).

On these 81 checkpoints, sharing the scaling factors substantially narrows the mean accuracy gaps for all three maps, including offset-only. This diagnostic rescales both the FP deviations from their class means and the inversion errors, so it does not isolate the margin shift in Proposition 2.

Thank you for prompting us to clarify what the theory establishes about inversion and to extend the analysis to offset-only correction. We hope these additions address your remaining concerns.

Comments on the paper

2 comments

Dear Reviewer,

I hope this message finds you well. I am writing to kindly remind you that the authors’ rebuttal has been posted and is now available for your review in the system.

We would greatly appreciate it if you could review the rebuttal and provide your assessment at your earliest convenience, as your input is important for us to move forward with a timely decision.

Thank you very much for your time and continued support.

ACs

As only 24 hours remain in the author rebuttal period, I sincerely invite the reviewers to review the author's response and promptly raise any remaining questions or indicate whether you believe the author has addressed the issues you have raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.