Beyond a Single Deterministic Teacher: Invariant Synthesis for Data-Free Quantization of Vision Transformers

Data-Free Quantization (DFQ) of Vision Transformers (ViTs) enables post-training quantization without real calibration data by synthesizing surrogate calibration samples from a pre-trained full-precision teacher model. However, existing DFQ methods typically optimize synthetic samples against a single deterministic teacher, which overlooks their sensitivity to internal structural variations, such as attention-head responses and spatial token dependencies. As a result, synthesized samples may exhibit structural brittleness, leading to unreliable calibration statistics and degraded quantization performance. To address these issues, we revisit data-free synthesis for ViTs from the perspective of invariant synthesis over structurally perturbed teachers. We theoretically show that: 1) structural sensitivity of synthetic samples contributes to the discrepancy between real and synthetic calibration statistics; and 2) structurally perturbed teachers can expose response variations hidden by deterministic supervision. Motivated by these insights, we propose DropIS, a novel and effective Dropout-Induced Invariant Synthesis approach for DFQ of ViTs. Unlike conventional dropout regularization, DropIS employs structured dropout as a teacher probing mechanism during synthesis: attention head dropout suppresses excessive reliance on specific head responses, while spatial token dropout mitigates dependence on narrow local token regions. Furthermore, DropIS incorporates auxiliary attention-space regularizers to stabilize perturbed responses and synthesize semantically consistent calibration samples. Extensive experiments across diverse backbones and downstream tasks verify the advantages of DropIS over the state-of-the-art methods, achieving up to 8.46% Top-1 accuracy improvements on ImageNet under challenging low-bit regimes. Our code is available in supplementary material package.