- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a DFQ method that tackles problems previous works on CLIP had. The authors propose: 1) mining better prompts from WordNet, 2) using soft targets to account for prompts semantically related to each other and correlation of them rather than treating them as equally negative as in the non-related prompts , 3) inducing visual diversity via decorrelating the orthogonal prompts.
Critical strengths
- I must say, this is probably the best structured, clearly written and organized paper Ive read. It is very straight-forward to follow and understand. Big kudos for that!
- I like that the authors bring ideas from the (quite old) model inversion literature such as those in DeepDream (e.g., data augmentation), to the DFQ literature.
- Avoiding one-hot supervision with soft targets is very interesting for DFQ. This is a variant of label smoothing but with the soft labels coming from WordNet. (potentially, you can make the soft targets come from CLIP text-to-text similarity also by simply measuring the similarity in the text space between your different prompts -- nice extension to consider)
- I also like the idea of visual coverage of synthetic images while preserving their alignment with the target prompts. This idea of finding orthogonal vectors from representations and decorelating them has been used in other fields (e.g., to remove hallucinations in VLMs) and the authors manage to bring it beautifully to DFQ field!
- Impressive improvements in DFQ compared to baseline methods across many bit rates and different CLIP models and 11 datasets! The experiments are very wide.
Critical weaknesses
- In Table 3 the authors report the baseline result as 46.89. This is without using their method. However what exactly is this baseline? It does not seem to be D4C because according to Table 2 its is 41.77. It seems there is a missing link between the baseline and the method. Perhaps this missing link is causing the large improvements in Table 2 (not really the proposed method)
Questions for authors
Minor Weaknesses/Questions:
- In Figure 1a, VISTA and D4C uses which prompts? I understand that the grey points refer to reference prompts derived from the downstream task labels, but then the orange/blue points of D4C/VISTA uses the WordNet prompts used to synthesize the images?
- It is wrong to say that previous methods cannot work on CLIP because they are limited to closed-set classifiers. You can make CLIP a closed-set classifier. Essentially the classification weights of CLIP zero-shot classifier are simply the CLIP text encoder outputs. You can perfectly encode the text prompts and save them as a nn.Linear classifier and completely discard the CLIP text encoder. So previous methods can indeed be applied to CLIP.
- Please cite Label Smoothing for Eq. 5 as this is exactly what Label Smoothing does, with the only difference being that the values are defined according to WordNet
Overall this paper is very strong fundamentally and experimentally. A big congrats for the authors on this work! Please clarify the major and minor weaknesses in the rebutal/discussion period.
Note for my confidence score of 2: I have never worked on DFQ in my research, but my research involves around model inversion and CLIP. Therefore, a basic understanding of quantization and DFQ are enough to review this paper. I have spent a few hours in understanding quantization methods and DFQ, before reviewing this paper.
2 comments
Rebuttal by Authors (2)
[Q1] Prompt sets visualized in Figure 1(a).
[A2] We appreciate the reviewer for requesting clarification on the prompt sets visualized in Figure 1(a). As the reviewer pointed out, the gray points represent reference prompts derived from downstream dataset labels. However, the orange and blue sets require two clarifications: the orange D4C prompts are not derived from WordNet, and both sets are sampled for visualization using the same method-specific prompt sampling procedures employed for image synthesis.
Figure 1(a) visualizes CLIP ViT-B/32 text embeddings from three sets, each containing 110 prompts formatted as “a photo of a {concept}.” The gray reference set contains 10 randomly sampled class names from each of the 11 downstream datasets and serves only as a post-hoc reference for semantic coverage analysis. These downstream labels are not used for image synthesis or quantization calibration.
The orange D4C set contains prompts sampled from D4C’s manually curated vocabulary of 180 concepts. In contrast, the blue VISTA set contains prompts sampled from our WordNet-derived pool using the same sampling procedure employed for image synthesis.
The orange and blue points are independently sampled for this post-hoc analysis from their respective prompt pools using the same method-specific sampling procedures employed for image synthesis. Thus, they characterize the inversion target distribution induced by each method’s sampling process, rather than being selected to reproduce the prompt set from one particular synthesis run.
[Q2] Applicability of existing closed-set DFQ methods to CLIP.
[A3] We agree with the reviewer that CLIP can be made structurally equivalent to a closed-set classifier once a fixed set of text-derived class weights is specified. Given a fixed set of class prompts, their normalized CLIP test embedding can be computed once and stored as the weight matrix of a bias-free linear classifier. For a normalized image embedding , this classifier computes the same logits as the original CLIP zero-shot classifier. The text encoder can therefore be discarded after constructing , allowing existing closed-set DFQ methods to be applied to the resulting classifier.
The data-free setting, however, requires an externally supplied class prompt-set to determine both output classes and the corresponding classifier weights. Native open-vocabulary CLIP does not provide such a fixed prompt set. Conventional closed-set models trained on datasets such as ImageNet or CIFAR do not encounter this ambiguity because their pretrained classifier heads already fix the output classes and weights.
D4C addresses this requirement with a manually curated vocabulary of 180 concepts. This construction enables closed-set inversion, but restricts the synthetic data to the semantics represented by those predefined concepts. Our contribution is therefore not to make DFQ technically applicable to CLIP, but to systematically construct a broad inversion-target pool from WordNet without manual vocabulary curation.
Accordingly, our original statement concerns the direct application of existing closed-set DFQ methods to CLIP in its native open-vocabulary form, rather than their applicability after an external prompt set is used to construct a closed-set classifier.
[Q3] Relationship between Equation 5 and label smoothing.
[A4] We appreciate the reviewer for pointing out this connection. We agree that Equation 5 follows the label smoothing framework and that the original work should be cited [2]. Standard label smoothing reserves most of the target probability for the corresponding class and distributes the remaining mass uniformly across all classes. Equation 5 follows the same principle by reserving probability for the corresponding prompt, but distributes the remaining mass among non-target prompts according to their normalized WordNet affinities.
We cite the original label smoothing work and clarify this relationship in Section 3.2.
References.
[1] Zhang et al., "D4C: Data-Free Quantization for Contrastive Language-Image Pre-Training Models.", CVPR Findings 2026
[2] Szegedy et al., “Rethinking the Inception Architecture for Computer Vision”, CVPR 2016
Rebuttal by Authors (1)
We sincerely appreciate the reviewer for the careful and highly encouraging assessment of our work. We particularly appreciate the positive evaluation of VISTA’s problem formulation, methodological design, clear presentation, and broad experimental coverage. The review also identifies an important missing link between Tables 2 and 3, along with several places where the prompt sources, applicability claim, and relationship to label smoothing required more precise explanations. In response, we examined the missing link between the Table 3 baseline and D4C, specified the prompt sources in Figure 1(a), corrected our statement about applying existing DFQ methods to CLIP, and clarified Equation 5 as a structured label-smoothing variant. We address each point in detail below.
[W1] Missing link between the Table 3 baseline and Table 2 D4C.
[A1] CLIP quantization calibrates the image and text encoders separately under the protocol established by D4C [1]. PSE governs only the synthesis of calibration images, so a PSE-only image baseline still depends on the prompts used to calibrate the text encoder. Specifically, the baseline in Table 3 (46.89%) combines PSE-only synthetic images, generated without prompt-based inversion, with text-calibration prompts sampled from our WordNet prompt pool. In contrast, the D4C result in Table 2 (41.77%) combines D4C-generated images with text-calibration prompts from D4C’s fixed vocabulary.
We recognize that the original Table 5 does not isolate this text-calibration effect and therefore cannot fully account for the improvement over D4C. In particular, the 5.12%p gap between the D4C results of Table 2 and the baseline of Table 5 cannot be attributed solely to VISTA’s prompt-guided image synthesis, semantic relaxation, or diverse objectives. It also reflects differences in the calibration image source and the text-calibration prompt source.
We therefore treat PSE-only image synthesis with D4C’s fixed prompt pool for text-encoder calibration as the starting point of the controlled component ablation. This baseline retains D4C’s text-calibration source while replacing D4C-generated images with PSE-only images (i.e., images synthesized using Larch without any prompt-conditioned inversion objective). From this starting point, we change one factor at a time: the text-calibration prompt set, WordNet-guided image synthesis, semantic relaxation, view consistency, and appearance diversity. This design separates the effects of text calibration, image synthesis, and the remaining components.
The revised results show that replacing only D4C’s text-calibration prompt pool with our WordNet-derived pool changes the average accuracy from 39.27 to 47.17 (+7.90%p). Adding WordNet-guided image synthesis further changes it to 47.82 (+8.55%p). The remaining semantic-relaxation and diversity components yield the final accuracy of 48.13 (+8.86%p). These results isolate the previously hidden text-calibration effect from the gains produced by prompt-guided image synthesis and the remaining VISTA component.
We clarify the separate calibration of the image and text encoders in Section 4.1, revise the component ablation in Section 4.4 and Table 5 to isolate the text-calibration prompt source, and detail the method-specific prompt pools in Appendix D.2.