- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The model proposed in this paper reweights alignment-sensitive targets while leaving teacher targets unaffected. This improves the student model's performance on downstream tasks, including zero-shot classification, retrieval, handling distribution shifts, and fine-grained recognition.
Critical strengths
- The dataset used in this paper is large-scale, and the method is validated across three corpora that vary in size and noise level.
- The alignment-sensitive objectives are reweighted, while the teacher-provided objectives remain unchanged.
- The ablation studies and sensitivity analyses are highly detailed, covering aspects such as online versus offline calibration.
Critical weaknesses
- AWD provides no theoretical guarantees regarding the convergence of the training objective, optimality, or generalization error bounds when sample weights are varied.
- All primary results (e.g., in Tables 1 and 2) are based on single runs, with no standard deviations reported; ideally, experiments should be conducted using multiple random seeds.
- When employing "recaptioning," AWD uses Qwen3-VL-4B-Instruct to generate synthetic descriptions—a model that may have been pre-trained on the training data itself.
- Although the paper discusses ALIP and NoC in the "Related Work" section, it fails to include them as baselines for comparative experiments.
- The main experiments are limited to ViT-B/16 and ViT-B/14 architectures, with ablation studies restricted to ViT-B/16; the scalability of the method to larger models (ViT-L/ViT-H) or smaller models (ViT-S) remains unproven.
Questions for authors
Has the impact of AWD on low-resource languages, less-represented languages, or non-Western image-text pairs been considered? For further concerns, please refer to the "Critical Weaknesses" section.
4 comments
Response to Reviewer rPtF (2/2)
W5. Other student architectures.
We trained two additional students on CC12M (20k steps), a ViT-S/16 and a ResNet-50. AWD improves every metric of Table 1 but one for ViT-S/16, and all of them for ResNet-50.
| CC12M, 20k steps | ZS | LP | k-NN | Dist. shift | I2T | T2I | Fine-grained | Avg. Δ |
|---|---|---|---|---|---|---|---|---|
| ViT-S/16, CLIP-KD | 42.6 | 64.4 | 59.6 | 38.5 | 53.0 | 40.0 | 40.8 | |
| ViT-S/16, + AWD | 43.1 | 64.9 | 60.1 | 39.4 | 55.4 | 41.0 | 42.3 | +1.1 |
| ResNet-50, CLIP-KD | 47.1 | 66.7 | 62.4 | 44.6 | 59.2 | 43.6 | 45.3 | |
| ResNet-50, + AWD | 48.2 | 66.9 | 62.6 | 45.6 | 62.9 | 46.0 | 47.3 | +1.7 |
| ViT-B/16, CLIP-KD | 48.3 | 68.2 | 62.6 | 45.4 | 59.4 | 45.0 | 47.1 | |
| ViT-B/16, + AWD | 51.0 | 69.6 | 64.3 | 47.7 | 63.4 | 47.7 | 49.8 | +2.6 |
Larger students (ViT-L or ViT-H) are beyond our compute budget, and less relevant to distillation, whose purpose is a compact model. These results will be added to the appendix.
Q1. Low-resource languages and non-Western pairs.
We did not study this specifically, and we will mention it in the limitations. AWD itself has no language-specific component: any effect would come from the teacher's scores. On DataComp-M, where about half of the captions are not in English, non-English captions are not concentrated among the low scores (49% of the lowest-scoring third, against 56% of the rest), although this statistic does not distinguish low-resource languages from widely spoken ones. Moreover, AWD does not remove any pair: a low-scoring pair keeps its distillation supervision, whereas filtering discards it entirely. We therefore believe that AWD may be preferable to filtering for such pairs, and that this deserves a dedicated study.
Thank you for the detailed response; the additional experiments have addressed most of the issues. However, the performance when using larger student models or dealing with low-resource languages remains to be explored. Therefore, I will maintain my rating of 3 (Weak Acceptance).
We thank the reviewer for the follow-up and for maintaining a positive rating. We would like to briefly comment on the two remaining points:
-
Regarding the student size, the CLIP distillation literature (cf. Sec. 2) typically distills into students of size ViT-B (CLIP-KD, ACID, MobileCLIP) or even smaller, e.g., ViT-M (TinyCLIP), and our ViT-B/16 student matches the largest of them. Reducing the size of the model remains the main goal of these distillations.
-
Regarding low-resource languages, the same CLIP distillation literature does not address this question at all, which remains essentially orthogonal to the task as it depends primarily on the teacher training, that the student only tries to mimic. As we focus on improving existing distillation techniques, it would not be exactly fair to penalize us (vs the others) on this ground.
Response to Reviewer rPtF (1/2)
We thank the reviewer for their constructive feedback, and for acknowledging the scale of the study, its validation on three corpora and the detailed ablations. Please find below our responses to the reviewer's concerns. We hope they adequately address the points raised and will encourage the reviewer to reconsider their rating.
W1. Theoretical guarantees.
Our contribution is empirical, and we do not provide new guarantees. We note that AWD stays close to the objective of CLIP-KD. Its weights lie between 0 and 2 and depend only on the frozen teacher, not on the parameters of the student, and all the losses are non-negative. At any point of training, the two objectives of Section 3 therefore satisfy LCLIP-KDAWD≤2LCLIP-KD, so the AWD loss is small whenever the CLIP-KD loss is. A theoretical analysis of this weighting is left to future work.
W2. Single runs.
We follow the protocol of CLIP-KD, which reports single runs, given the cost of each training. To measure the variation, we trained CLIP-KD and CLIP-KD + AWD on DataComp-M with two additional seeds (20k steps), each changing the initialization and the order of the data. AWD improves every metric of Table 1 on every seed, and its average gain is +1.21 ± 0.12 points. These results will be added to the paper.
W3. Recaptioning model.
Recaptioning is not part of AWD. It is a complementary experiment, which tests whether AWD still helps once the captions are rewritten. Both sides of this comparison, with and without AWD, are trained on the same synthetic captions, so what Qwen3-VL-4B-Instruct saw during its own pre-training affects them equally and cannot explain the gain of AWD. We will state this in the paper.
W4. ALIP and NoC as baselines.
We discuss ALIP and NoC in the related work because they explore the same idea as ours, using alignment as a graded training signal rather than as a filtering criterion, but in other settings. NoC is a captioning method, which conditions a captioner on the alignment level of each pair. ALIP pre-trains a model from scratch, without a teacher, on two captions of each image, its raw text and a synthetic one; it weights its contrastive losses by an alignment score computed by the model being trained, to balance the two sources of captions. Neither is a distillation method, so neither is a baseline for our setting. We compare instead with methods designed for distillation: Table 1 includes I-ACID and H-ACID, two recent data-selection methods, and adding AWD improves both.