acceptodds
Under review as a conference paper at ICLR 2027

Vista: Accurate CLIP Model Inversion Framework with Semantic Targets and Appearance Coverage for Data-Free Quantization

Abstract

How can we quantize a CLIP model without access to real data while preserving its generalization capability? Quantization reduces the memory and computational cost of large models, but most methods require real data for calibration. Data-free quantization (DFQ) eliminates the need for real data, yet most existing methods synthesize images conditioned on predefined class labels as inversion targets. In contrast, CLIP models support open-vocabulary recognition without a fixed label space. Therefore, DFQ for CLIP models requires semantically diverse text prompts to extract broad semantic knowledge encoded in the model. A recent method manually constructs these targets, covering a narrow semantic range and degrading the generalization of the quantized model. In this paper, we propose Vista, an accurate CLIP model inversion framework for DFQ. Vista broadens the semantic coverage of inversion targets by systematically sampling visual prompts from the lexical hierarchy of WordNet. Vista captures semantic relationships among prompts by replacing one-hot supervision with identity-preserving soft targets derived from pairwise prompt similarities. Vista expands the visual coverage of synthetic images by decorrelating the appearance components that the target prompts do not explain, while enforcing view consistency to preserve semantic alignment. Experimental results show that Vista improves the average accuracy of quantized CLIP models by up to 9.45%p over existing DFQ methods and better preserves the generalization capability of CLIP models across various CLIP backbones, bit-widths, and downstream datasets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Data-Free Quantization, Zero-Shot Quantization, Model Inversion

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_xXqeOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a DFQ method that tackles problems previous works on CLIP had. The authors propose: 1) mining better prompts from WordNet, 2) using soft targets to account for prompts semantically related to each other and correlation of them rather than treating them as equally negative as in the non-related prompts , 3) inducing visual diversity via decorrelating the orthogonal prompts.

Critical strengths
  • I must say, this is probably the best structured, clearly written and organized paper Ive read. It is very straight-forward to follow and understand. Big kudos for that!
  • I like that the authors bring ideas from the (quite old) model inversion literature such as those in DeepDream (e.g., data augmentation), to the DFQ literature.
  • Avoiding one-hot supervision with soft targets is very interesting for DFQ. This is a variant of label smoothing but with the soft labels coming from WordNet. (potentially, you can make the soft targets come from CLIP text-to-text similarity also by simply measuring the similarity in the text space between your different prompts -- nice extension to consider)
  • I also like the idea of visual coverage of synthetic images while preserving their alignment with the target prompts. This idea of finding orthogonal vectors from representations and decorelating them has been used in other fields (e.g., to remove hallucinations in VLMs) and the authors manage to bring it beautifully to DFQ field!
  • Impressive improvements in DFQ compared to baseline methods across many bit rates and different CLIP models and 11 datasets! The experiments are very wide.
Critical weaknesses
  • In Table 3 the authors report the baseline result as 46.89. This is without using their method. However what exactly is this baseline? It does not seem to be D4C because according to Table 2 its is 41.77. It seems there is a missing link between the baseline and the method. Perhaps this missing link is causing the large improvements in Table 2 (not really the proposed method)
Questions for authors

Minor Weaknesses/Questions:

  • In Figure 1a, VISTA and D4C uses which prompts? I understand that the grey points refer to reference prompts derived from the downstream task labels, but then the orange/blue points of D4C/VISTA uses the WordNet prompts used to synthesize the images?
  • It is wrong to say that previous methods cannot work on CLIP because they are limited to closed-set classifiers. You can make CLIP a closed-set classifier. Essentially the classification weights of CLIP zero-shot classifier are simply the CLIP text encoder outputs. You can perfectly encode the text prompts and save them as a nn.Linear classifier and completely discard the CLIP text encoder. So previous methods can indeed be applied to CLIP.
  • Please cite Label Smoothing for Eq. 5 as this is exactly what Label Smoothing does, with the only difference being that the values are defined according to WordNet

Overall this paper is very strong fundamentally and experimentally. A big congrats for the authors on this work! Please clarify the major and minor weaknesses in the rebutal/discussion period.

Note for my confidence score of 2: I have never worked on DFQ in my research, but my research involves around model inversion and CLIP. Therefore, a basic understanding of quantization and DFQ are enough to review this paper. I have spent a few hours in understanding quantization methods and DFQ, before reviewing this paper.

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1)

We sincerely appreciate the reviewer for the careful and highly encouraging assessment of our work. We particularly appreciate the positive evaluation of VISTA’s problem formulation, methodological design, clear presentation, and broad experimental coverage. The review also identifies an important missing link between Tables 2 and 3, along with several places where the prompt sources, applicability claim, and relationship to label smoothing required more precise explanations. In response, we examined the missing link between the Table 3 baseline and D4C, specified the prompt sources in Figure 1(a), corrected our statement about applying existing DFQ methods to CLIP, and clarified Equation 5 as a structured label-smoothing variant. We address each point in detail below.

[W1] Missing link between the Table 3 baseline and Table 2 D4C.

[A1] CLIP quantization calibrates the image and text encoders separately under the protocol established by D4C [1]. PSE governs only the synthesis of calibration images, so a PSE-only image baseline still depends on the prompts used to calibrate the text encoder. Specifically, the baseline in Table 3 (46.89%) combines PSE-only synthetic images, generated without prompt-based inversion, with text-calibration prompts sampled from our WordNet prompt pool. In contrast, the D4C result in Table 2 (41.77%) combines D4C-generated images with text-calibration prompts from D4C’s fixed vocabulary.

We recognize that the original Table 5 does not isolate this text-calibration effect and therefore cannot fully account for the improvement over D4C. In particular, the 5.12%p gap between the D4C results of Table 2 and the baseline of Table 5 cannot be attributed solely to VISTA’s prompt-guided image synthesis, semantic relaxation, or diverse objectives. It also reflects differences in the calibration image source and the text-calibration prompt source.

We therefore treat PSE-only image synthesis with D4C’s fixed prompt pool for text-encoder calibration as the starting point of the controlled component ablation. This baseline retains D4C’s text-calibration source while replacing D4C-generated images with PSE-only images (i.e., images synthesized using Larch\mathcal{L}_{\mathrm{arch}} without any prompt-conditioned inversion objective). From this starting point, we change one factor at a time: the text-calibration prompt set, WordNet-guided image synthesis, semantic relaxation, view consistency, and appearance diversity. This design separates the effects of text calibration, image synthesis, and the remaining components.

I1-TI1-SI2I3-VI3-AAvg. [%]Gain [%p]
Baseline (PSE with D4C’s prompt)39.270.00
✔︎47.17+7.90
✔︎✔︎47.82+8.55
✔︎✔︎✔︎47.83+8.56
✔︎✔︎✔︎✔︎47.88+8.61
✔︎✔︎✔︎✔︎48.01+8.74
✔︎✔︎✔︎✔︎✔︎48.13+8.86
  • I1-T: WordNet-derived text calibration prompt set
  • I1-S: WordNet-guided image synthesis
  • I2: Semantic relaxation
  • I3-V: View consistency
  • I3-A: Appearance diversity

The revised results show that replacing only D4C’s text-calibration prompt pool with our WordNet-derived pool changes the average accuracy from 39.27 to 47.17 (+7.90%p). Adding WordNet-guided image synthesis further changes it to 47.82 (+8.55%p). The remaining semantic-relaxation and diversity components yield the final accuracy of 48.13 (+8.86%p). These results isolate the previously hidden text-calibration effect from the gains produced by prompt-guided image synthesis and the remaining VISTA component.

We clarify the separate calibration of the image and text encoders in Section 4.1, revise the component ablation in Section 4.4 and Table 5 to isolate the text-calibration prompt source, and detail the method-specific prompt pools in Appendix D.2.

AuthorsOpenReview ↗

Rebuttal by Authors (2)

[Q1] Prompt sets visualized in Figure 1(a).

[A2] We appreciate the reviewer for requesting clarification on the prompt sets visualized in Figure 1(a). As the reviewer pointed out, the gray points represent reference prompts derived from downstream dataset labels. However, the orange and blue sets require two clarifications: the orange D4C prompts are not derived from WordNet, and both sets are sampled for visualization using the same method-specific prompt sampling procedures employed for image synthesis.

Figure 1(a) visualizes CLIP ViT-B/32 text embeddings from three sets, each containing 110 prompts formatted as “a photo of a {concept}.” The gray reference set contains 10 randomly sampled class names from each of the 11 downstream datasets and serves only as a post-hoc reference for semantic coverage analysis. These downstream labels are not used for image synthesis or quantization calibration.

The orange D4C set contains prompts sampled from D4C’s manually curated vocabulary of 180 concepts. In contrast, the blue VISTA set contains prompts sampled from our WordNet-derived pool using the same sampling procedure employed for image synthesis.

The orange and blue points are independently sampled for this post-hoc analysis from their respective prompt pools using the same method-specific sampling procedures employed for image synthesis. Thus, they characterize the inversion target distribution induced by each method’s sampling process, rather than being selected to reproduce the prompt set from one particular synthesis run.

[Q2] Applicability of existing closed-set DFQ methods to CLIP.

[A3] We agree with the reviewer that CLIP can be made structurally equivalent to a closed-set classifier once a fixed set of text-derived class weights is specified. Given a fixed set of KK class prompts, their normalized CLIP test embedding can be computed once and stored as the weight matrix W∈RK×dW \in \mathbb{R}^{K \times d} of a bias-free linear classifier. For a normalized image embedding z∈Rdz \in \mathbb{R}^{d}, this classifier computes the same logits as the original CLIP zero-shot classifier. The text encoder can therefore be discarded after constructing WW, allowing existing closed-set DFQ methods to be applied to the resulting classifier.

The data-free setting, however, requires an externally supplied class prompt-set to determine both KK output classes and the corresponding classifier weights. Native open-vocabulary CLIP does not provide such a fixed prompt set. Conventional closed-set models trained on datasets such as ImageNet or CIFAR do not encounter this ambiguity because their pretrained classifier heads already fix the output classes and weights.

D4C addresses this requirement with a manually curated vocabulary of 180 concepts. This construction enables closed-set inversion, but restricts the synthetic data to the semantics represented by those predefined concepts. Our contribution is therefore not to make DFQ technically applicable to CLIP, but to systematically construct a broad inversion-target pool from WordNet without manual vocabulary curation.

Accordingly, our original statement concerns the direct application of existing closed-set DFQ methods to CLIP in its native open-vocabulary form, rather than their applicability after an external prompt set is used to construct a closed-set classifier.

[Q3] Relationship between Equation 5 and label smoothing.

[A4] We appreciate the reviewer for pointing out this connection. We agree that Equation 5 follows the label smoothing framework and that the original work should be cited [2]. Standard label smoothing reserves most of the target probability for the corresponding class and distributes the remaining mass uniformly across all classes. Equation 5 follows the same principle by reserving probability δ\delta for the corresponding prompt, but distributes the remaining mass 1−δ1-\delta among non-target prompts according to their normalized WordNet affinities.

We cite the original label smoothing work and clarify this relationship in Section 3.2.

References.

[1] Zhang et al., "D4C: Data-Free Quantization for Contrastive Language-Image Pre-Training Models.", CVPR Findings 2026

[2] Szegedy et al., “Rethinking the Inception Architecture for Computer Vision”, CVPR 2016

Reviewer_m5pVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a data-free quantization framework for CLIP (VISTA). It targets the generation of synthetic calibration images to avoid accessing real data. Its main contributions are to introduce WordNet prompt sampling to improve coverage of the distribution, to use an identity-preserving approach to increase visual diversity while maintaining semantic alignment with the target prompts, and visual diversity regularization. Altogether, those components allow for improvements over existing methods across several CLIP architectures and quantization settings.

Critical strengths

The paper identifies well some limitations of previous CLIP inversion method and solves it well.

The use of WordNet to increase semantic coverage is interesting, particularly compared with manually selected prompts from other methods.

The experimental evaluation is extensive and shows large improvements over previous methods.

Critical weaknesses

The methodological contributions seem a bit limited, particularly when compared to D4C which already introduced CLIP model inversion to generate synthetic calibration images from text prompts for data-free quantization.

The evaluation focuses only on zero shot classification, while CLIP is being used for a lot of other applications (image-text retrieval, question answering through LLMs...). It would have been interesting to explore other setups.

Additionally, the method is only evaluated on CLIP, it would have been interesting to see if it generalizes across other backbones (like SigLIP).

Finally, an analysis of the computational cost of the method would have been interesting to show.

Questions for authors

Can the authors provide results on another task (like retrieval or question answering), or at least an intuition of whether the method would fail or succeed and why?

How sensitive is performance to the selection of WordNet categories and the size of the prompt pool?

Finally, can the authors run a single experiment but on SigLIP to see if results are consistent?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1)

We sincerely appreciate the reviewer for the careful and constructive assessment of our work. We particularly appreciate the recognition that VISTA addresses important limitations of prior CLIP inversion methods, expands semantic coverage through WordNet, and provides an extensive evaluation across CLIP architectures and quantization settings. The review raises important questions about the methodological distinction from D4C, generalization beyond zero-shot classification and CLIP, computational cost, and sensitivity to the WordNet categories and prompt-pool size. In the response, we clarify VISTA’s contribution beyond D4C, report image-text retrieval results, discuss the scope and limitations for VQA, evaluate VISTA on SigLIP, provide a detailed computational cost analysis, and examine sensitivity to both category selection and prompt pool size. We address each point below.

[W1] Methodological contribution beyond D4C.

[A1] We agree that D4C introduced text-guided CLIP inversion for data-free quantization, and we do not claim CLIP inversion itself as our contribution. Our primary distinction lies in how the inversion targets are constructed and how their generalization is evaluated.

D4C constructs its inversion targets from a manually curated vocabulary of 180 concepts and evaluates the resulting quantized models primarily on CIFAR and ImageNet benchmarks. VISTA instead systematically constructs a broader inversion target pool from the lexical hierarchy of WordNet, without manually enumerating the target concepts. We evaluate whether the resulting calibration data preserve CLIP’s open-vocabulary generalization across 11 downstream classification datasets spanning diverse visual domains.

We further examine two additional forms of generalization during the rebuttal period. First, we evaluate VISTA on SigLIP to test whether the approach transfers beyond the CLIP architecture and training objective. Second, we evaluate the quantized models on image-text retrieval using Flickr30k and MS-COCO to test whether the generated calibration data preserve cross-modal representations beyond zero-shot classification. These additional results are detailed in our responses to Q1 and Q3.

Accordingly, VISTA’s contribution is not the first application of text-guided inversion to CLIP. It is the systematic construction of WordNet-derived inversion targets and the validation of their generalization across diverse datasets, model families, and downstream tasks. Identity-preserving semantic relaxation and visual-diversity regularization complement this target construction; we report their isolated effects in the revised ablation study rather than attributing the full improvement over D4C to these objectives.

[W4] Computational cost analysis.

[A2] We profile synthetic-data generation using CLIP ViT-B/32 under the same hardware, generation-step count, image resolution, and 128 images.

MethodTime per 1 stepTime for 128 imagesRelative step time
D4C20.6ms8.2 minx 1.00
VISTA28.3ms11.3 minx 1.37
VISTA (w/o PSE)21.7ms8.7 minx 1.05

VISTA requires 28.3 ms per generation step, compared with 20.6 ms for D4C, corresponding to a 1.38× generation-time overhead. Generating the complete 128-image calibration set takes approximately 11.3 minutes for VISTA and 8.2 minutes for D4C under the same experimental setting.

VISTA requires 1.37× the generation time of D4C, while removing PSE reduces this ratio to 1.05×, indicating that most of the additional cost originates from PSE. This overhead arises because PSE collects intermediate token features and computes layer-wise patch-similarity statistics at each synthesis step.

The additional cost is incurred only during offline calibration-image generation. VISTA has the same parameter count and inference-time computation as D4C, under the same quantization setting.

AuthorsOpenReview ↗

Rebuttal by Authors (2)

[Q1 & W2] Generalization beyond zero-shot classification.

[A3] We appreciate the reviewer for suggesting an evaluation beyond zero-shot classification. We agree that classification accuracy alone does not establish whether the quantized model preserves the cross-modal representations required by other CLIP applications. We therefore evaluate image-text retrieval on Flickr30k [1] and the MS-COCO 5K [2] test set without task-specific fine-tuning.

We use CLIP ViT-B/32 and the same quantized checkpoints as in our unified classification comparison. We report the Recall@1 and Recall@5 for both text-to-image and image-to-text retrieval.

MethodW/AFlicker 30k
Image-to-textText-to-image
R@1R@5R@1R@5
Full Precision32/3278.994.958.883.5
D4C8/870.192.149.076.2
VISTA (proposed)8/871.391.951.678.6
D4C4/865.187.946.574.4
VISTA (proposed)4/867.090.649.877.0
MethodW/AMS-COCO 5K
Image-to-textText-to-image
R@1R@5R@1R@5
Full Precision32/3250.175.030.456.0
D4C8/843.169.025.449.6
VISTA (proposed)8/844.069.626.751.0
D4C4/839.365.824.648.5
VISTA (proposed)4/840.466.325.849.2

We have not evaluated VISTA in a full VQA pipeline. Unlike retrieval, which directly uses the quantized CLIP image and text embeddings, VQA typically connects the vision encoder to a learned projector and a language model. Its performance therefore depends not only on preservation of the CLIP embedding space, but also on the projector, multimodal alignment, language model, and the quantization strategy applied to these additional modules. The retrieval results provide evidence for applications that directly depend on CLIP’s cross-modal embedding space, but they do not by themselves establish performance on VQA. We state this limitation explicitly and consider end-to-end VQA evaluation as future work.

We add the retrieval results to Section 4.3 and expand the limitations section to distinguish direct embedding-based applications from downstream multimodal systems such as VQA.

[Q2] Sensitivity to WordNet category selection and prompt-pool size.

[A4] We separately examine the sensitivity of (1) the selection of WordNet categories and (2) the size of the resulting prompt pool, because category composition and pool size can affect the calibration data in different ways.

Sensitivity to WordNet category selection

We first compare the four categories selected by VISTA with the full set of 26 candidate categories and additionally evaluate 30 randomly sampled four-category combinations. All results are reported as the average zero-shot classification accuracy across the 11 downstream datasets using the W8/A8 quantized ViT-B/32. For the unfiltered and filtered settings, we report the mean ±\pm standard deviation over three random seeds, whereas for random sampling, we report the mean ±\pm standard deviation across the 30 sampled category combinations. We update Table 14 and Appendix F.2 to clarify these evaluation details. As shown in the following table, random sampling achieves a slightly higher average accuracy than the VISTA-selected category set (53.60% vs. 53.40%).

128180200400800
W8/A852.0453.4151.6753.6153.38
W4/A846.0948.1347.2248.3548.04

Performance varies with the prompt-pool size, but we observe no monotonic relationship between the number of candidate concepts and quantization accuracy. This non-monotonic behavior may arise from two factors. First, increasing the cardinality of the candidate pool does not guarantee a proportional increase in effective semantic coverage because the additional concepts may be semantically redundant. Second, a larger candidate pool does not necessarily increase the number of prompts actually used for image synthesis or text-encoder calibration in a given run. These sampling budgets are controlled separately. To examine the latter factor, we additionally vary the number of synthesized images and text prompts actually consumed during calibration.

AuthorsOpenReview ↗

Rebuttal by Authors (3)

Effective calibration and observer budgets

To distinguish the pool size from the amount of data actually consumed during quantization, we additionally vary three effective budgets: the number of synthesized images, the number of text-calibration prompts, and the number of text samples used for activation observation. We utilize ViT-B/32 under W4/A8 quantization and report average accuracy across the 11 downstream datasets. Each experiment changes only the indicated budget while holding the remaining budgets fixed. The reference configuration uses 128 synthetic images, 512 text-calibration prompts, and 32 text samples for activation observation.

[# of synthetic images]

3264128256
W4/A845.5948.02 (+2.43)48.13 (+0.11)48.39 (+0.26)

[# of text-calibration prompts]

1282565121,024
W4/A846.4347.74 (+1.31)48.13 (+0.39)47.97 (-0.16)

[# of activation-observer texts]

32512
W4/A848.1349.86 (+1.73)

Overall, VISTA is robust to the choice of WordNet categories and does not exhibit a systematic dependence on the size of the prompt pool. The results instead show that performance depends more directly on the composition of the sampled prompts and the effective image, text-calibration, and observer budgets.

[Q3 & W3] Generalization to SigLIP

[A5] We evaluate D4C and VISTA on SigLIP B/16 (google/siglip-base-patch16-224) [3]. We apply the image-generation and quantization setting used for CLIP ViT-B/32 without SigLIP-specific hyperparameter tuning.

We adapt D4C to the same SigLIP encoder and evaluation pipeline because D4C does not provide an official SigLIP implementation. D4C and VISTA use the same calibration image budget, quantization procedure, bit-width settings, and 11 zero-shot classification datasets. The two methods retain their respective prompt source and image synthesis objective. We report the average top-1 accuracy across the 11 downstream datasets. We add these results in Section 4.2 and Table 3.

MethodW/AAvg. [%]Diff [%p]
Full Precision32/3275.25
D4C8/866.50
VISTA (proposed)72.49+5.99
D4C6/656.98
VISTA (proposed)64.41+7.43
D4C4/859.680.00
VISTA (proposed)68.48+8.80
D4C4/43.020.00
VISTA (proposed)3.00-0.02

VISTA outperforms the adapted D4C baseline at W8A8, W6A6, and W4A8. This result provides initial evidence that the benefit of WordNet-derived inversion targets transfers beyond CLIP to a vision–language model trained with a different contrastive objective. The improvement is therefore not restricted to CLIP’s softmax-based image–text training formulation.

At W4A4, however, both methods collapse to near-random performance. VISTA does not resolve this extreme low-precision failure, suggesting that the information loss from jointly quantizing SigLIP’s weights and activations to 4 bits cannot be addressed by calibration-data generation alone under the current quantization procedure.

References.

[1] Young et al., “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions”, TACL 2014

[2] Lin et al., "Microsoft COCO: Common Objects in Context", ECCV 2014

[3] Zhai et al., “Sigmoid loss for language image pre-training”, ICCV 2023

Reviewer_Hw9wOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VISTA, a data-free model inversion framework for quantizing CLIP models while preserving their zero-shot classification performance. The proposed method consists of three components: (1) WordNet-guided prompt sampling to broaden semantic coverage, (2) identity-preserving semantic relaxation to incorporate semantic relationships into contrastive supervision, and (3) view consistency and prompt-orthogonal residual decorrelation to improve the diversity of synthetic calibration data. The experiments cover seven CLIP backbones, four weight/activation bit-width settings, and eleven downstream classification datasets. The reported results demonstrate substantial average accuracy improvements over D4C and conventional data-free quantization baselines in several settings.

Critical strengths

1.Strong empirical improvements across diverse evaluation settings. The evaluation includes multiple architectures, bit-width configurations, and downstream datasets. These results suggest that improving synthetic calibration data can be particularly valuable for preserving CLIP's zero-shot performance.

Critical weaknesses

1.Unexplained discrepancies in experimental results and insufficient reproducibility information. The reported results are inconsistent across sections. Table 2 reports 56.26% for ViT-B/32 under W8/A8, while Table 12 reports 53.91%. Figure 4(a) peaks near 48.5%, compared to approximately 54% in Figures 4(b–c), despite the stated identical setting. The paper does not clarify these discrepancies, report multi-seed statistics, or explain whether downstream evaluation data were used for hyperparameter selection. These issues weaken confidence in the reported results. 2.Insufficient validation of the claimed methodological contributions. The ablation study does not adequately isolate the proposed mechanisms. Semantic relaxation provides only a 0.17 percentage-point gain without comparisons to standard label smoothing or alternative semantic affinities. Moreover, the appearance loss decorrelates samples potentially associated with different prompts, which does not directly demonstrate within-prompt diversity. View consistency and appearance diversity are also ablated jointly rather than separately. Stronger evidence is needed to establish the individual contributions and novelty of these components.

Questions for authors

1.Could the authors reconcile the numerical differences among Tables 2, 3, and 12 and Figure 4? Were these results obtained with identical models, seeds, and configurations? Please provide the exact settings. 2.How were the hyperparameters selected? Were any labeled downstream evaluation examples or test accuracies used during hyperparameter selection? If not, what independent criterion determined the reported settings? 3.How does WordNet-based semantic relaxation compare with uniform label smoothing, shuffled affinities, and CLIP text-embedding affinities? Could the authors also separately ablate the view consistency and appearance diversity losses and report multi-seed results? 4.How does the proposed appearance loss improve diversity within the same semantic target, rather than merely decorrelating samples associated with different targets? Can the authors provide direct measurements of within-prompt diversity and layer-wise quantization errors?

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1)

We sincerely appreciate the reviewer for the careful and constructive feedback. The reviewer identifies important concerns regarding inconsistent experimental settings, downstream data use in hyperparameter selection, limited multi-seed evidence, and insufficient isolation of individual components. We address these concerns by standardizing the evaluation protocol, disclosing the complete selection procedure, reporting seed variability, conducting controlled component-wise ablation study, and revising claims not supported by these analyses. We provide the detailed results and corresponding manuscript revisions below.

[Q1] Inconsistent results across tables and figures

[A1] We notice that the reported numbers are not obtained under identical settings. We list the exact settings behind each number.

Table 2Table 2Table 2Table 2Table 3Table 12
PrecisionW8A8W6A6W4A8W4A4W4A8W8A8
# of images32323232128128
# of text prompts for observer5125125125123232

The largest factors are the number of calibration images and text prompts used to initialize the activation observers. We rerun the VISTA rows of Table 2 and the whole of Table 3 under one setting, the D4C recipe used for the baseline rows: 128 synthetic images 512 calibration text sampled from 180 concepts ×\times 18 templates of each method’s own prompt set 32 observer images and texts

Across the seven backbones, VISTA remains 5.9-11.4%p above D4C at W8A8, W6A6, and W4A8 (28.6%p for ViT-L/14 W6A6, where D4C collapses).

We further clarify that the three panels of Figure 4 are originally evaluated under different experimental configurations. Specifically, Figure 4(a) uses W4/A8 quantization, 32 images, and 512 observer text prompts, whereas Figures 4(b) and 4(c) use W8/A8 quantization, 128 images, and 32 observer text prompts. Consequently, the results across the three panels are obtained under inconsistent conditions and are not directly comparable.

To correct this inconsistency, we reran experiments under a unified configuration using W8/A8 quantization, 128 images, and 32 observer text prompts. We accordingly update Figure 4(a) with the newly obtained results. Under this consistent setting, the best performance reaches 53.93% in Figures 4(a).

[Q2] Hyperparameter selection and use of downstream labels

[A2] The hyperparameter sensitivity analysis in Section 4.5 evaluates the sensitivity of VISTA’s newly introduced hyperparameter using downstream test accuracy. We search only these new hyperparameters over the space in Table 7 and select the final hyperparameters for each model based on the average test accuracy across the 11 datasets; all hyperparameters inherited from D4C [1] and DeepInversion [2] are retained without additional tuning.

This hyperparameter search protocol is typical in the data-free quantization (DFQ) work, where data-free generally denotes that the original training data are not used during synthetic-data generation, calibration, or quantization, rather than that downstream labels are never consulted during method development. Prior DFQ studies [3-6] commonly report objective weights selected through grid search; for example, GDFQ selects its generation loss weights β\beta and γ\gamma through simple grid search [3], and PSAQ-ViT similarly selects α1\alpha_1 and α2\alpha_2 [4]. Our protocol follows this convention, but we agree that the selection criterion and the resulting scope of the data-free claim should have been stated explicitly.

Despite this concern, VISTA consistently outperforms D4C across the evaluated hyperparameter configurations. For ViT-B/32 under W8A8 quantization, the four-seed average accuracies of the 13 configurations in Figure 4(b) range from 53.15% to 53.53%, corresponding to a variation of only 0.38%p. These results indicate that downstream-informed hyperparameter selection may affect the exact reported accuracy, but does not account for the consistent advantage over D4C.

AuthorsOpenReview ↗

Rebuttal by Authors (2)

[Q3] Multi-seed reliability and component-wise contributions

[A3] We address this question in three parts: (1) quantifying seed variability, (2) comparing WordNet-based semantic relaxation with one-hot targets, uniform label smoothing, shuffled WordNet affinities, and CLIP text affinities, and (3) separately ablating the view-consistency and appearance-diversity losses. The detailed results are provided below.

Multi-seed experiments

Multi-seed evaluation is important for quantifying experimental variability and assessing the reliability of the main performance comparison. We are therefore conducting additional multi-seed experiments for the ViT-B/16 results in Table 1 across the reported quantization settings. Unfortunately, due to limited GPU availability during the rebuttal period, these experiments are still in progress and cannot be included in our initial response. We will update the response with the mean ±\pm standard deviation as soon as the experiments are completed.

Semantic relaxation and WordNet affinity

To directly address the reviewer’s question, we evaluate W4A4 quantization performance by comparing one-hot supervision, uniform label smoothing, shuffled WordNet affinities, CLIP text affinities, and WordNet affinities under the same ViT-B/32 generation, δ=0.9\delta = 0.9, and quantization protocol.

MethodAverage Accuracy [%]Difference [%p]
One-hot47.940.00
Uniform smoothing47.92-0.02
Shuffled WordNet affinities47.60-0.34
CLIP text affinity47.78-0.16
WordNet affinity47.97+0.03

Although WordNet affinity produces the highest average accuracy, its gain over one-hot supervision is small (+0.03 percentage points). Therefore, we do not attribute a substantial standalone improvement to semantic relaxation. Instead, the results indicate that WordNet provides a suitable semantic structure that preserves performance.

Component-wise ablation of VISTA

We agree that the ablation study in Section 4.4 does not sufficiently isolate the contribution of each component. We therefore revise Table 5 to separately evaluate the effects of the text-calibration prompt set, WordNet-guided image synthesis, semantic relaxation, view consistency, and appearance diversity under a unified generation and quantization protocol. Each comparison changes only the component being evaluated while retaining the same calibration budget, prompt pool, and quantization settings.

I1-TI1-SI2I3-VI3-AAvg. [%]Gain [%p]
Baseline (PSE with D4C’s prompt)39.270.00
✔︎47.17+7.90
✔︎✔︎47.82+8.55
✔︎✔︎✔︎47.83+8.56
✔︎✔︎✔︎✔︎47.88+8.61
✔︎✔︎✔︎✔︎48.01+8.74
✔︎✔︎✔︎✔︎✔︎48.13+8.86
  • I1-T: WordNet-derived text calibration prompt set
  • I1-S: WordNet-guided image synthesis
  • I2: Semantic relaxation
  • I3-V: View consistency
  • I3-A: Appearance diversity

The revised component-wise ablation study shows that the largest gain comes from replacing D4C’s text-calibration prompts with the WordNet-derived prompt set. When the text calibration prompt set, WordNet-guided synthesis, and semantic relaxation are fixed, adding view consistency alone yields a +0.05%p gain, adding appearance diversity alone yields a +0.18%p gain, and adding both yields a +0.30%p gain, reaching the highest accuracy of 48.13%. Thus, the two losses provide only modest individual accuracy changes, although their joint use performs better than either loss alone.

AuthorsOpenReview ↗

Rebuttal by Authors (3)

[Q4] Diversity interpretation and layer-wise quantization error

[A4] We agree that the batch-wise appearance loss alone does not demonstrate that diversity increases within the same semantic target, because a batch may contain images generated from different prompts. We therefore directly measured the diversity among images generated from the same prompt, and the layer-wise quantization error with and without the appearance loss.

Direct measurement of within-prompt diversity

To directly measure diversity within the same semantic target, we conducted a prompt-controlled experiment using CLIP ViT-B/32 for both image synthesis and feature-distance measurement. We generate 16 images for each of 32 fixed prompts, with and without the appearance loss, and compute the pairwise CLIP-feature distance separately within each prompt.

Within-prompt distanceDifference
Without appearance loss0.197
With appearance loss0.401+0.204

The appearance loss increases within-prompt distance by 104%, with increases observed for 30 of the 32 prompts. The same trend is obtained using independent CLIP models: the distance increases by 54% with ViT-B/16 and by 39% with ViT-L/14. These results directly demonstrate that the appearance loss increases CLIP feature-space dispersion among images generated from the same prompt, rather than merely decorrelating images associated with different prompts.

Layer-wise quantization error

We measure the normalized MSE (NMSE) between the block outputs of the quantized and full-precision image encoders (CLIP ViT-B/32, W4A8), with and without the appearance loss. For this analysis only, we use 1,100 real test images (100 from each of the 11 downstream datasets); these images are not used for synthetic image generation or quantization calibration.

Without appearance lossWith appearance loss
Block 328.227.6
Block 639.838.5
Block 939.838.2
Block 1226.225.3
Average accuracy [%]47.8248.12
  • NMSE(x10^2) of block outputs. Accuracy is the average of the 11 downstream datasets on the full test sets.

The appearance loss reduces the quantization error at all four measured blocks. This consistent error reduction accompanies a +0.30%p gain in average accuracy.

Together with the within-prompt diversity results above, these findings suggest the following mechanism: dispersing the prompt-orthogonal components of synthetic images exposes the quantizer to a broader range of activations, so that calibration data better reflect real-image activations. The magnitudes of both the error reduction and the accuracy gain are modest, consistent with the component-wise ablation, where appearance diversity contributes 0.18%p on its own.

References

[1] Zhang et al., "D4C: Data-Free Quantization for Contrastive Language-Image Pre-Training Models.", CVPR Findings 2026

[2] Yin et al., “Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion”, CVPR 2020

[3] Xu et al., “Generative Low-bitwidth Data Free Quantization,” ECCV 2020

[4] Li et al., “Patch Similarity Aware Data-Free Quantization for Vision Transformers,” ECCV 2022

[5] Li et al., “PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformers”, TNNLS

[6] Kim et al., “SynQ: Accurate Zero-shot Quantization by Synthesis-aware Fine-tuning,” ICLR 2025

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.