acceptodds
Under review as a conference paper at ICLR 2027

Beyond a Single Deterministic Teacher: Invariant Synthesis for Data-Free Quantization of Vision Transformers

Abstract

Data-Free Quantization (DFQ) of Vision Transformers (ViTs) enables post-training quantization without real calibration data by synthesizing surrogate calibration samples from a pre-trained full-precision teacher model. However, existing DFQ methods typically optimize synthetic samples against a single deterministic teacher, which overlooks their sensitivity to internal structural variations, such as attention-head responses and spatial token dependencies. As a result, synthesized samples may exhibit structural brittleness, leading to unreliable calibration statistics and degraded quantization performance. To address these issues, we revisit data-free synthesis for ViTs from the perspective of invariant synthesis over structurally perturbed teachers. We theoretically show that: 1) structural sensitivity of synthetic samples contributes to the discrepancy between real and synthetic calibration statistics; and 2) structurally perturbed teachers can expose response variations hidden by deterministic supervision. Motivated by these insights, we propose DropIS, a novel and effective Dropout-Induced Invariant Synthesis approach for DFQ of ViTs. Unlike conventional dropout regularization, DropIS employs structured dropout as a teacher probing mechanism during synthesis: attention head dropout suppresses excessive reliance on specific head responses, while spatial token dropout mitigates dependence on narrow local token regions. Furthermore, DropIS incorporates auxiliary attention-space regularizers to stabilize perturbed responses and synthesize semantically consistent calibration samples. Extensive experiments across diverse backbones and downstream tasks verify the advantages of DropIS over the state-of-the-art methods, achieving up to 8.46% Top-1 accuracy improvements on ImageNet under challenging low-bit regimes. Our code is available in supplementary material package.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Data-Free Quantization, Vision Transformers, Invariant Synthesis, Structural Probing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4ruuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DropIS, a data-free quantization DFQ method for ViTs. Instead of synthesizing calibration images against a single deterministic full-precision teacher, it optimizes them over a family of structurally perturbed teachers (attention-head dropout and block-wise spatial token dropout), with SSIM/entropy attention regularizers borrowed from MimiQ.

Critical strengths

I think it is a plug-in idea with consistent gains across backbones and bit-widths. Even K=1 gives about 4 points at 1.009 times total cost, which is the strongest evidence. The broad empirical coverage, ie. component, probe-sampling, K, r, lamda, sample-count, 2-bit and dense-task studies.

Critical weaknesses

It seems the main multi-probe gain is confounded with compute. K=4 costs about 5 times synthesis time (3.34 times total), yet no compute-matched baseline is given. In Fig 1b, the motivating margin-variance evidence is measured under the same dropout probes DropIS optimizes, so it is circular. Moreover, it looks like prop 2.1 is a triangle inequality that never shows DropIS reduces the sensitivity term? Also the prop 2.2's lower bound vanishes at large margins?

Questions for authors

how were the hyper parameters chosen in a data-free setting, expecially p_h, p_s, r, lamda? Why does the Deterministic row in Table 2c (31.32) differ from reproduced MimiQ (27.39) on DeiT-S 3w3a? How do r=3 and p_s=0.2 relate, and how are zeroed heads handled in head-wise SSIM? It would be good if the author can report the seeds and variances for each experiment results.

Reviewer_jyQmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a new Data-Free Quantization (DFQ) method named DropIS for ViT. DropIS introduces the structural dropout mechanism for the full precision ViT teacher to synthesize surrogate calibration samples. Specifically, DropIS uses attention head dropout and spatial token dropout to suppresses excessive reliance on specific head responses and mitigates dependence on narrow local token regions respectively. DropIS is verified across various backbones~(ViT, DeiT and Swin series) and tasks~(image classification, object detection and sematic segmentation) compared with other sota methods.

Critical strengths
  1. The observation between the distribution of synthesizing data and model artitecture is valuable, which guide the motivation of DropIS. Thus, introducing the dropout mechanism to the teacher artitecture to synthesize data is straight and reasonable.
  2. Comprehensive experiments demonstrate the effectiveness of DropIS. DropIS reports ImageNet classifications results across ViT, DeiT and Swin backbones at 3W3A, 4W4A, 5W5A and 4W8A and DropIS achieves superior improvements compared with other SOTA methods. In supplementary, DropIS also get improvements on object detection and semantic segmentation tasks.
  3. The structure of this paper is well formulated and easy to follow.
Critical weaknesses
  1. The citation section of the paper uses a large number of \cite, and \citep should be used.
  2. There is a lack of a baseline that uses feature-dimension dropout; this experiment needs to be added in order to demonstrate the importance of structured dropout.
  3. The ablation of dropout rate across head and spatial needs more grids to demonstrate the optimization of the resulted hyperparameters.
Questions for authors

My main concern is the citation format and feature dropout baselines.

Reviewer_8Hq2OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method of data-free quantization (DFQ) for ViT, called DropIS. It discovers two things; (1) change of teacher network incurs large calibration activation distribution difference by real and synthetic data, (2) teacher with randomly dropped head or tokens shows very different activation responses. When synthesizing data, the DropIS applies attention-head dropout and spatial token dropout and attention-space regularizer for generating correct data for multiple different teachers. In ImageNet dataset, they report 8.46% empirical gain by the proposed DropIS.

Critical strengths
  • Simple but empirical effective method. Two types of dropout (head, token) for ViT brings meaningful empirical gains.
  • Provide theoretical justification (Sec. 2.2.1) (though I didn't check the correctness thoroughly)
  • Noticeable empirical gains, especially in low bit regimes. (Table 1)
Critical weaknesses
  • A bit of limited novelty: It is well explored that the idea of making input data (images) that teacher model can predict the class label correctly on randomly perturbed synthetic input data (e.g., DeepInversion, CVPR 2020)
  • Lack of hyperparameter sensitivity studies. L346-348 mentions hyperparameter details but the sensitivity of those values should be investigated for robustness of the algorithm to those hyperparams.
Questions for authors
  • Are calibration and reconstruction done with full precision model without dropout while images are synthesized by teachers with dropout?

  • Minor comments

    • L124: "Notation." should not be a link.
    • L336: "Qunatization." should not be a link.
2 comments
Reviewer_8Hq2OpenReview ↗

Thank you for the clarification.

The reviewer appreciates the authors' efforts to clarify my concerns. The reviewer maintains "3: weak accept" score for the value of simplicity of the method and theoretical justification as well as empirical gains.

We sincerely thank the reviewer for the positive assessment and for acknowledging our efforts to address the concerns. We are particularly grateful for the reviewer’s recognition of DropIS’s methodological simplicity, theoretical justification, and empirical effectiveness. The reviewer’s constructive feedback has helped us improve the clarity and rigor of the manuscript, and we will incorporate the corresponding revisions into the final version. We thank the reviewer again for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.