- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how post-training quantization affects the robustness of Vision Transformers under distribution shifts. The authors evaluate several quantization methods across different architectures, model sizes, pretraining strategies, and datasets. They show that preserving clean ImageNet accuracy does not necessarily preserve performance under distribution shift. The paper also introduces a prediction-level model based on classification margins and quantization sensitivity to explain why certain samples are more affected by quantization. The experimental results provide useful insights into the robustness of low-precision Vision Transformers.
Critical strengths
-
The paper addresses an important and practically relevant problem. Most quantization methods focus on preserving clean accuracy, while their behavior under distribution shifts receives less attention. The authors provide a useful analysis showing why clean accuracy alone may not be sufficient for evaluating quantized models.
-
The experimental study is quite comprehensive. The authors consider multiple quantization methods, architectures, model sizes, activation precisions, and pretraining strategies. The inclusion of ImageNet-C, ImageNet-R, and ImageNet-A helps examine different types of distribution shifts.
-
I find the proposed margin-based analysis particularly interesting. Rather than only reporting accuracy degradation, the authors attempt to explain quantization fragility through classification margins and sensitivity to quantization errors. The reported correlations and sample-level analysis provide encouraging evidence for this explanation.
Critical weaknesses
-
While the proposed prediction-level model is interesting, I would like to see a stronger validation of its individual components. The model relies on a first-order Taylor approximation and a Gaussian approximation of accumulated quantization errors. The reported correlations are promising, but it is not entirely clear how much improvement comes from considering directional quantization sensitivity compared with simpler measures such as classification margins or quantization-error magnitude alone.
-
The analysis of calibration mismatch could be explored further. The authors show that activation-distribution differences are relatively small and that mixed calibration provides limited improvements. However, the activation statistics mainly describe overall distribution differences, while a small number of outliers or highly sensitive layers could still contribute significantly to the observed degradation. It would be useful to examine whether certain layers or activation regions are more responsible for the robustness loss. Also, the ImageNet-C experiments consider only four corruption types at severity 5, which limits the conclusions across corruption severities.
-
I noticed a few inconsistencies in the reported experimental results that should be clarified. For example, the W4A4 AdaLog accuracy for Swin-Base under Gaussian noise is 30.068% in Table 1, whereas Appendix Table 4 reports 41.978%. There are also some non-monotonic results when increasing activation precision, particularly for DeiT-Small. These may be reporting issues or differences in experimental configurations, but an explanation would help make the results more reliable.
Questions for authors
-
How much additional predictive information does directional quantization sensitivity provide compared with the full-precision classification margin alone? Could the authors compare the proposed model against simpler predictors based on margin, quantization-error magnitude, and their combination? I think this would help establish the contribution of the proposed formulation.
-
The paper suggests that activation-statistics mismatch alone cannot explain the robustness degradation. Could the authors provide a layer-wise analysis of quantization sensitivity or examine whether a small number of highly sensitive layers contribute disproportionately to the degradation? It would also be interesting to see whether the observed behavior remains consistent across different corruption severities.
-
Could the authors clarify the differences between the results reported in Table 1 and Appendix Table 4, particularly for Swin-Base under Gaussian noise? Also, why does increasing activation precision sometimes lead to worse performance for certain configurations, such as DeiT-Small? Are these differences related to calibration, quantizer optimization, or experimental variability?
We thank the reviewer for the time and effort to review our paper and providing valuable comments.
Comparison Against Simpler Predictors
We agree that directional sensitivity should be evaluated against simpler measures. We compared our model against four baselines in the same setting as our post-hoc model.
Values report Spearman / Pearson correlation across seven datasets.
Our model achieves higher correlations in all three pooled settings. At fixed W4A4, Spearman correlation increases from 0.6820 for raw classification margins to 0.8109.
Table 3 provides complementary sample-level evidence. Within narrow FP-margin bins, samples with higher first-order margin-contraction scores exhibit 4.19 to 27.04 percentage points higher misclassification rates. This supports the additional explanatory value of directional sensitivity among samples with comparable margins.
The combined-error baseline combines activation and weight errors rather than margins and error magnitude. We will make the baseline definitions and the scope of these findings explicit.
Robustness Across Corruption Severities
We extended the evaluation to severities 1, 3, and 4 for motion blur, fog, Gaussian noise, and contrast, using AdaLog W4A4 on ViT-Base, DeiT-Base, and Swin-Base. Each evaluation uses 50,000 images. The table reports the full-precision minus quantized top-1 accuracy gap in percentage points.
We observe substantial gaps below severity 5. At severity 3, the gaps range from 6.79 to 17.10 percentage points. They generally increase with severity, although not strictly for every configuration. For example, the DeiT-Base motion-blur gap decreases slightly from 9.97 points at severity 3 to 9.81 at severity 4.
These results show that the degradation is not confined to the most severe corruptions. Its magnitude depends on corruption type, severity, and architecture. We will include these evaluations in the revision.
Table Discrepancies and Activation-Precision Anomalies
We identified two separate issues: