Amortized-Precision Quantization for Early-Exit Vision Transformers

Vision Transformers (ViTs) achieve strong performance across vision tasks, yet their deployment with low-precision early exiting remains fragile. Existing quantization methods assume static full-depth execution, making them unstable when exit decisions are perturbed by quantization noise, which can amplify errors along dynamic inference paths. In this paper, we introduce **Amortized-Precision Quantization (APQ)**, a utilization-aware formulation that accounts for layer-wise stochastic exposure to quantization noise and reveals *depth–precision* trade-offs. Building on APQ, we propose **Mutual Adaptive Quantization with Early Exiting (MAQEE)**, a bi-level framework that jointly optimizes exit thresholds and bit-widths under explicit risk control to improve inference stability. MAQEE establishes a superior *Pareto frontier* in the accuracy–efficiency trade-off, reducing BOPs by up to 95% while maintaining accuracy and outperforming strong baselines by up to 20% across classification, detection, and segmentation tasks. Source code is available at [https://anonymous.4open.science/r/MAQEE-5F2F](https://anonymous.4open.science/r/MAQEE-5F2F).