BitRot: Understanding Quantization Fragility in Vision Transformers Under Distribution Shift

Post-training quantization (PTQ) enables efficient deployment of Vision Transformers, but preserving clean ImageNet accuracy do not necessarily preserve performance under distribution shift. This is important in deployment settings where low-precision models may encounter corruptions, visual domain shifts, and naturally adversarial examples, yet the interaction between aggressive quantization and robustness under distribution shift remains underexplored. We systematically study this interaction across architectures, PTQ methods, pretraining strategies, model scales, activation precisions, calibration settings, and several forms of distribution shift. We find that W4A4 models can retain near-lossless in-distribution accuracy while suffering substantially larger degradation on shifted data, with the severity varying across distributions and model configurations. We further test whether this fragility is primarily explained by activation-statistics and calibration mismatch. Mixed calibration using samples from multiple shifted distributions provides only modest and inconsistent recovery, leaving a substantial gap to full precision. Motivated by these observations, we develop a post-hoc prediction-level model that treats quantization as a perturbation to full-precision classification margins and characterizes susceptibility through both the available margin and its sensitivity to accumulated quantization residuals. The resulting predictor tracks quantization-induced degradation across distributions, architectures, model scales, and activation bit-widths. At the sample level, examples with comparable full-precision margins but higher directional quantization sensitivity are consistently more likely to be misclassified after quantization. These results show that clean ImageNet accuracy alone can substantially underestimate the deployment cost of low-precision ViTs, and that robustness under distribution shift should be evaluated explicitly when assessing PTQ methods.