acceptodds
Under review as a conference paper at ICLR 2027

Don't Drop the Weak Pairs: Alignement-Aware Distillation of Vision-Language Models

Abstract

Large-scale vision-language pretraining strongly benefits from data filtering, as poorly aligned image-text pairs provide noisy contrastive supervision and are often better discarded. We show that this intuition does not fully carry over to vision-language distillation. During distillation, weakly aligned pairs may indeed be harmful to the image-text contrastive objective but still provide valuable teacher supervision through feature, relation, or representation distillation. Rather than removing such samples, we propose to adapt how they are used. Specifically, we introduce an alignment-aware soft-gating mechanism that uses the teacher's image-text similarity to modulate, on a per-sample basis, alignment-sensitive objectives while preserving knowledge distillation. Well-aligned pairs receive stronger contrastive supervision, while weakly aligned pairs rely more heavily on the teacher signal. The gate is computed online from teacher embeddings already available for distillation, adding negligible overhead. Across CC12M and DataComp-M datasets, our approach consistently improves CLIP-KD and remains complementary to adaptive data selection with ACID. We further show that alignment weighting improves another distillation recipe, in TIPSv2, demonstrating that its benefits extend across different distillation settings. Our results suggest that, for vision-language distillation, noisy pairs are better reweighted than discarded. Code and pretrained weights will be open-sourced.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Knowledge Distillation, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rPtFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The model proposed in this paper reweights alignment-sensitive targets while leaving teacher targets unaffected. This improves the student model's performance on downstream tasks, including zero-shot classification, retrieval, handling distribution shifts, and fine-grained recognition.

Critical strengths
  1. The dataset used in this paper is large-scale, and the method is validated across three corpora that vary in size and noise level.
  2. The alignment-sensitive objectives are reweighted, while the teacher-provided objectives remain unchanged.
  3. The ablation studies and sensitivity analyses are highly detailed, covering aspects such as online versus offline calibration.
Critical weaknesses
  1. AWD provides no theoretical guarantees regarding the convergence of the training objective, optimality, or generalization error bounds when sample weights are varied.
  2. All primary results (e.g., in Tables 1 and 2) are based on single runs, with no standard deviations reported; ideally, experiments should be conducted using multiple random seeds.
  3. When employing "recaptioning," AWD uses Qwen3-VL-4B-Instruct to generate synthetic descriptions—a model that may have been pre-trained on the training data itself.
  4. Although the paper discusses ALIP and NoC in the "Related Work" section, it fails to include them as baselines for comparative experiments.
  5. The main experiments are limited to ViT-B/16 and ViT-B/14 architectures, with ablation studies restricted to ViT-B/16; the scalability of the method to larger models (ViT-L/ViT-H) or smaller models (ViT-S) remains unproven.
Questions for authors

Has the impact of AWD on low-resource languages, less-represented languages, or non-Western image-text pairs been considered? For further concerns, please refer to the "Critical Weaknesses" section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer rPtF (1/2)

We thank the reviewer for their constructive feedback, and for acknowledging the scale of the study, its validation on three corpora and the detailed ablations. Please find below our responses to the reviewer's concerns. We hope they adequately address the points raised and will encourage the reviewer to reconsider their rating.

W1. Theoretical guarantees.

Our contribution is empirical, and we do not provide new guarantees. We note that AWD stays close to the objective of CLIP-KD. Its weights lie between 0 and 2 and depend only on the frozen teacher, not on the parameters of the student, and all the losses are non-negative. At any point of training, the two objectives of Section 3 therefore satisfy LCLIP-KDAWD≤2 LCLIP-KD,\mathcal{L}^{\mathrm{AWD}}_{\mathrm{CLIP\text{-}KD}} \le 2\, \mathcal{L}_{\mathrm{CLIP\text{-}KD}}, so the AWD loss is small whenever the CLIP-KD loss is. A theoretical analysis of this weighting is left to future work.

W2. Single runs.

We follow the protocol of CLIP-KD, which reports single runs, given the cost of each training. To measure the variation, we trained CLIP-KD and CLIP-KD + AWD on DataComp-M with two additional seeds (20k steps), each changing the initialization and the order of the data. AWD improves every metric of Table 1 on every seed, and its average gain is +1.21 ± 0.12 points. These results will be added to the paper.

DataComp-M, 20k stepsZSLPk-NNDist. shiftI2TT2IFine-grainedAverage
CLIP-KD44.89 ± 0.2269.73 ± 0.1462.65 ± 0.2641.02 ± 0.2050.15 ± 0.8935.64 ± 0.3448.99 ± 0.1547.35 ± 0.14
+ AWD45.99 ± 0.3570.18 ± 0.2063.40 ± 0.2842.52 ± 0.2551.78 ± 0.6036.70 ± 0.5450.09 ± 0.1648.56 ± 0.03
Gain+1.10 ± 0.21+0.46 ± 0.07+0.75 ± 0.29+1.50 ± 0.36+1.63 ± 0.37+1.06 ± 0.77+1.10 ± 0.24+1.21 ± 0.12

W3. Recaptioning model.

Recaptioning is not part of AWD. It is a complementary experiment, which tests whether AWD still helps once the captions are rewritten. Both sides of this comparison, with and without AWD, are trained on the same synthetic captions, so what Qwen3-VL-4B-Instruct saw during its own pre-training affects them equally and cannot explain the gain of AWD. We will state this in the paper.

W4. ALIP and NoC as baselines.

We discuss ALIP and NoC in the related work because they explore the same idea as ours, using alignment as a graded training signal rather than as a filtering criterion, but in other settings. NoC is a captioning method, which conditions a captioner on the alignment level of each pair. ALIP pre-trains a model from scratch, without a teacher, on two captions of each image, its raw text and a synthetic one; it weights its contrastive losses by an alignment score computed by the model being trained, to balance the two sources of captions. Neither is a distillation method, so neither is a baseline for our setting. We compare instead with methods designed for distillation: Table 1 includes I-ACID and H-ACID, two recent data-selection methods, and adding AWD improves both.

AuthorsOpenReview ↗

Response to Reviewer rPtF (2/2)

W5. Other student architectures.

We trained two additional students on CC12M (20k steps), a ViT-S/16 and a ResNet-50. AWD improves every metric of Table 1 but one for ViT-S/16, and all of them for ResNet-50.

CC12M, 20k stepsZSLPk-NNDist. shiftI2TT2IFine-grainedAvg. Δ
ViT-S/16, CLIP-KD42.664.459.638.553.040.040.8
ViT-S/16, + AWD43.164.960.139.455.441.042.3+1.1
ResNet-50, CLIP-KD47.166.762.444.659.243.645.3
ResNet-50, + AWD48.266.962.645.662.946.047.3+1.7
ViT-B/16, CLIP-KD48.368.262.645.459.445.047.1
ViT-B/16, + AWD51.069.664.347.763.447.749.8+2.6

Larger students (ViT-L or ViT-H) are beyond our compute budget, and less relevant to distillation, whose purpose is a compact model. These results will be added to the appendix.

Q1. Low-resource languages and non-Western pairs.

We did not study this specifically, and we will mention it in the limitations. AWD itself has no language-specific component: any effect would come from the teacher's scores. On DataComp-M, where about half of the captions are not in English, non-English captions are not concentrated among the low scores (49% of the lowest-scoring third, against 56% of the rest), although this statistic does not distinguish low-resource languages from widely spoken ones. Moreover, AWD does not remove any pair: a low-scoring pair keeps its distillation supervision, whereas filtering discards it entirely. We therefore believe that AWD may be preferable to filtering for such pairs, and that this deserves a dedicated study.

Thank you for the detailed response; the additional experiments have addressed most of the issues. However, the performance when using larger student models or dealing with low-resource languages ​​remains to be explored. Therefore, I will maintain my rating of 3 (Weak Acceptance).

We thank the reviewer for the follow-up and for maintaining a positive rating. We would like to briefly comment on the two remaining points:

  • Regarding the student size, the CLIP distillation literature (cf. Sec. 2) typically distills into students of size ViT-B (CLIP-KD, ACID, MobileCLIP) or even smaller, e.g., ViT-M (TinyCLIP), and our ViT-B/16 student matches the largest of them. Reducing the size of the model remains the main goal of these distillations.

  • Regarding low-resource languages, the same CLIP distillation literature does not address this question at all, which remains essentially orthogonal to the task as it depends primarily on the teacher training, that the student only tries to mimic. As we focus on improving existing distillation techniques, it would not be exactly fair to penalize us (vs the others) on this ground.

Reviewer_pe7HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a small CLIP from scratch with a large frozen teacher and argues that badly captioned image-text pairs should not be thrown away, because the losses that copy the teacher do not depend on the caption being right. It scores each pair by the teacher's image-text cosine similarity and multiplies only the caption-dependent contrastive losses by a weight between 0 and 2, centered at the batch median, leaving the teacher-copying losses unchanged. It is tested on CLIP-KD and TIPSv2 distillation, on CC12M and DataComp-M, alone and combined with filtering and ACID data selection.

Critical strengths
  1. Consistent gains at no cost: on CC12M it adds 3.5 points on average to CLIP-KD and 3.2 to TIPSv2, it stacks with ACID data selection (from +1.6 to +4.1), and it slows training by only 1% because the teacher embeddings are already computed.
  2. Broad evaluation across two distillation recipes, two corpora, data-selection baselines and many downstream tasks.
Critical weaknesses
  1. The method is a small step from existing work. Weighting the contrastive loss by an image-text alignment score is already done in ALIP, which the paper cites, and scoring pairs with a teacher is the standard filtering signal. The new part is applying the weight to only the caption-dependent losses in distillation, and the evidence that this split matters is one short 20k-step ablation on ImageNet only, where weighting all losses gives 49.3 versus 51.0. The paper also describes AWD as reallocating supervision between the two sources, yet it only scales the contrastive losses by up to 2× while the distillation losses stay at 1, so strong pairs simply get a larger total loss; a true trade-off, such as w times the contrastive loss plus (1 − w) times the distillation loss, is neither used nor compared.

  2. The weighting rule is arbitrary and only relative. Mapping the batch median to weight 1 means half of every batch is always down-weighted and half up-weighted up to 2×, whatever the corpus quality, and the stated reason, that this "preserves the overall balance" between the two groups of losses, does not hold when scores are skewed: scores of −1, −1, 0, 1 and 10 give an average weight of 1.11. The paper's own sweep shows that a hard rule, weight 0 below the median and 2 above, also beats the baseline, so the sigmoid form is not shown to matter, and no simpler rule such as a fixed threshold or weights rescaled to average 1 is compared.

  3. On the noisier corpus, simply dropping weak pairs works better, and no result has error bars. On DataComp-M, removing the lowest-scoring third of pairs improves CLIP-KD by 1.4 points, more than AWD's 1.1, which cuts against the paper's title. With no repeated runs, the gains of about one point on DataComp-M cannot be separated from training noise.

Questions for authors
  1. What is the actual average weight during training on each corpus, and does AWD keep its gains if the weights are rescaled so they average exactly 1?
  2. How does AWD compare with simpler rules, such as a fixed similarity threshold or weights proportional to the score, which would show whether the median-centered sigmoid is needed?
  3. Why does AWD alone underperform plain filtering on DataComp-M, and does it still help on cleaner or noisier corpora than the two tested?
  4. Does a low teacher similarity actually mean a wrong caption, for example checked against human judgments on a sample of pairs?
6 comments
AuthorsOpenReview ↗

Response to Reviewer pe7H (1/5): novelty and evidence for the loss split

We thank the reviewer for their detailed and constructive feedback, and for acknowledging the consistent gains of AWD at a negligible cost and the breadth of the evaluation. Please find below our responses to the reviewer's concerns. We hope they adequately address the points raised and will encourage the reviewer to reconsider their rating.

W1. Novelty, evidence for the loss split, and the symmetric variant of AWD.

Novelty. AWD addresses a question specific to distillation: what a weakly aligned pair should contribute when its caption is not its only source of supervision. Its principle is that such a pair should rely on the teacher rather than on its caption.

  • On AWD vs ALIP. ALIP pre-trains a model from scratch, without a teacher, on two captions of each image, its raw text and a synthetic one. It does weight contrastive losses by an alignment score, but to balance two sources of captions, original and synthetic, and with a score computed by the model being trained. In contrast, AWD balances two types of loss terms, contrastive and distillation, which is specific to distillation, and uses the score of a frozen teacher, which does not depend on the student.
  • On the teacher score as a filtering signal. The teacher score is indeed the standard filtering signal, and the key of our contribution is to repurpose it. Filtering uses it for a binary decision on the whole pair: the weakly aligned pairs are removed, and with them the supervision the teacher could provide on them. AWD uses it as a continuous weight that sets how much each pair relies on its caption rather than on the teacher: every pair is kept, and its caption counts according to its alignment.

Evidence for the loss split. We address the two points raised by the reviewer about our ablation setup: its length and its evaluation breadth.

  • On the "short" ablation. Table 4 follows our ablation setting of 20k steps at a global batch size of 16,384, i.e. 328M training samples. This remains a substantial training, 40% of our full recipe (50k steps, 819M samples), and ablations at the full recipe are beyond our compute budget.
  • On the "ImageNet only" evaluation. We agree that reporting this ablation on tasks other than ImageNet strengthens its conclusion. Table 4 reports ImageNet-1k zero-shot and linear probing; we evaluated its four checkpoints on all the metrics of Table 1, with the naming of Table 4. The AWD weight, applied to the contrastive losses (CLIP and ICL), gains +2.6 points on average over CLIP-KD, against +1.4 for the uniform weight applied to all the losses, and it is better on each metric. This gap is well above the seed-to-seed variation measured in our seed study, where the average gain of AWD has a standard deviation of 0.12 points over three seeds (see W3). Table 4 will be extended with these additional evaluations.
Table 4 (CC12M, 20k steps)ZSLPk-NNDist. shiftI2TT2IFine-grainedAvg. Δ
CLIP-KD48.368.262.645.459.445.047.1
+ CLIP weight (CLIP)50.169.564.147.162.947.049.2+2.1
+ AWD weight (CLIP + ICL)51.069.664.347.763.447.749.8+2.6
+ uniform weight (all losses)49.368.763.346.761.546.848.2+1.4
AuthorsOpenReview ↗

Response to Reviewer pe7H (2/5): the symmetric variant of AWD

The symmetric variant of AWD. We thank the reviewer for this suggestion, which we find an interesting variant of AWD. We implemented it as AWD$_s$, with the per-pair loss wiLalign,i+(2−wi)Ldistill,iw_i\mathcal{L}_{\mathrm{align},i} + (2 - w_i)\mathcal{L}_{\mathrm{distill},i}; since wi∈(0,2)w_i \in (0, 2), the two weights are symmetric around 1. We trained it on both corpora at 20k steps.

CC12M, 20kZSLPDist. shiftI2TT2IFine-grainedAvg. ΔDataComp-M, 20kZSLPDist. shiftI2TT2IFine-grainedAvg. Δ
CLIP-KD48.368.245.459.445.047.1CLIP-KD44.969.741.050.135.649.0
AWD51.069.647.763.447.749.8+2.6AWD46.070.242.551.836.750.1+1.2
AWD$_s$51.770.148.163.448.350.0+3.0AWD$_s$47.170.743.451.037.250.7+1.7

AWD$_s$ slightly outperforms AWD on both corpora: the average gain over CLIP-KD goes from +2.6 to +3.0 points on CC12M, and from +1.2 to +1.7 on DataComp-M. It follows the same principle as AWD, a weakly aligned pair relying more on the teacher than on its caption, and extends it by also raising the distillation weight of such pairs. We will add AWD$_s$ to the paper as a variant of AWD.

Update: results at the full recipe (50k steps). Given the promising results of AWD$_s$, we have trained it with our full recipe (50k steps) on both corpora, and compare it with the CLIP-KD and AWD rows of Table 1.

CC12M, 50kZSLPDist. shiftI2TT2IFine-grainedAvg. ΔDataComp-M, 50kZSLPDist. shiftI2TT2IFine-grainedAvg. Δ
CLIP-KD53.470.950.764.149.251.6CLIP-KD52.874.049.859.142.856.0
AWD57.373.154.767.753.255.3+3.5AWD54.474.251.559.544.257.2+1.1
AWD$_s$57.172.953.867.852.754.1+3.1AWD$_s$55.175.251.760.445.057.9+1.8

At the full recipe, AWD$_s$ remains ahead of AWD on DataComp-M (+1.8 against +1.1), but falls slightly behind it on CC12M (+3.1 against +3.5). AWD and AWD$_s$ are therefore on par overall, and both improve consistently over CLIP-KD.

On the word "reallocating". We agree that the term "reallocating" used in the related work (Section 2) is misleading, as AWD does not move any weight onto the distillation losses (though this becomes true with the symmetric variant suggested by the reviewer). We will clarify this in the paper.

AuthorsOpenReview ↗

Response to Reviewer pe7H (3/5): form of the weighting rule

W2, Q1 and Q2. On the form of the weighting rule.

The reviewer describes our weighting rule as "only relative" and "arbitrary". It is indeed relative by design, but to the corpus and not to the batch, and we agree that its exact form matters little as long as it separates the two halves of the corpus.

"Only relative": to the corpus, not to the batch. The median μ\mu and the quartile deviation QQ are exponential moving averages over training (Section 3), i.e. statistics of the corpus, not of the batch: if all the pairs of a batch are well aligned, they all receive a weight above 1, and if they are all poorly aligned, they all receive a weight below 1. This makes the rule independent of the scale of the teacher's similarities, which changes with the corpus and with the teacher, so that no threshold has to be tuned.

"Arbitrary": why not the hard rule? The hard rule is a special case of our weight, its limit for a large sharpness α\alpha, and the sweep of Fig. 3 covers the whole family: the results are stable from α=0.5\alpha = 0.5 to the hard rule (51.0 to 51.4 in ImageNet zero-shot, against 48.3 for CLIP-KD), and the hard rule gives the same result as our default α=1\alpha = 1 (51.0 for both). Our formulation is simply more general, and it only degrades when the sharpness is too low to separate the two halves (49.7 at α=0.25\alpha = 0.25). Since α=1\alpha = 1 worked well on every corpus and recipe without tuning, we did not investigate the shape of the rule further, and leave possible improvements to future work.

Q1, average weight. The mean weight over training is 0.99 on CC12M and 1.03 on DataComp-M, stable from the first logged step (around step 50) to the end of training. The reviewer is right that the mean is exactly 1 only for a symmetric score distribution, and we will clarify this in the paper. As for the suggestion of rescaling the weights to an exact mean of 1, it would change them by only 1% to 3%, so we expect the gains of AWD to be similar.

Q2, simpler rules. Regarding the comparison of AWD with simpler rules:

  • Fixed threshold. A fixed similarity threshold is AWD with the hard rule and with the median computed offline, once, over the corpus. Both ingredients are covered by our ablations: the hard rule gives the same results as our soft weight (Fig. 3, see above), and computing μ\mu and QQ offline, in a first pass over the corpus, gives the same results as the online estimate (51.0 in ImageNet zero-shot for both, Section 4.2 and the appendix). We kept the online variant as our default as it does not require an additional pass over the training data.
  • Weights proportional to the score. Following the reviewer's suggestion, we implemented the rule w=max⁡(s,0)/μw = \max(s, 0) / \mu, which equals 1 at the median as in AWD. It improves over CLIP-KD but less than AWD (+1.5 against +2.6 on average, AWD being better on each metric).
CC12M, 20k stepsZSLPk-NNDist. shiftI2TT2IFine-grainedAvg. Δ
CLIP-KD48.368.262.645.459.445.047.1
Weights proportional to the score50.169.163.846.761.446.648.5+1.5
AWD51.069.664.347.763.447.749.8+2.6

We will clarify the corpus-level rule in Section 3, and report the measured mean weights and the proportional rule in the appendix.

Reviewer_vvNXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper suggests a simple fix to the distillation problem involving noisy image-text papers. The paper proposes to split the loss into two components: one that treat the dataset pair as a hard positive match (like CLIP, ICL) and one that relies on soft teacher distributions or unimodal features (like CRD, FD). By modulating the former loss components based on the teacher’s alignment confidence, they seek to reduce the effect of noisy pairs during distillation. This leads to consistent improvements across models and benchmarks.

Critical strengths

Strengths:

  1. The loss formulation is simple and is obtained for free, as the teacher’s embeddings are computed anyway.
  2. The conceptual insight is clear and reduces the impact of the noisy pairs during distillation of contrastively trained models.
  3. Well-thought-out ablations (Table 4) show that weighting only the alignment-sensitive losses (CLIP and ICL) losses yields better results than uniform weighting.
Critical weaknesses

Weaknesses:

  1. Why is the gain smaller on DataComp, a noisier dataset, compared to the cleaner CC-12M?
  2. Why does hard filtering+AWD on CC-12 lead to poorer results?
  3. The TIPSv2 distillation uses the text encoder for scoring similarity. This adds computation, so tempering expectation about zero-compute extension would be good.
Questions for authors

More like a suggestion: There is a lot of work on continuous weighting of similarity. It would be good to cite them for completeness. Like: CWCL, SoftCLIP, NLIP, etc. This would better contextualize your contributions.

3 comments
AuthorsOpenReview ↗

Response to Reviewer vvNX

We thank the reviewer for their constructive feedback, and for acknowledging the simplicity of the method, the clarity of its motivation and the ablations. Please find below our responses to the reviewer's concerns. We hope they adequately address the points raised and will encourage the reviewer to reconsider their rating.

W1. Smaller gain on DataComp-M than on CC12M.

We agree that one might intuitively expect AWD to yield a larger gain on DataComp-M, a noisier corpus, than on CC12M. Nevertheless, we found that, in addition to the level of noise, the number of repetitions of the pairs is a confounding factor: the pairs of DataComp-M are seen about nine times less often during training than those of CC12M, which we believe explains its smaller gain. Two experiments support this.

  • Level of noise. In the controlled experiment of Fig. 2, the DataComp-M subsets have the same size (10M pairs) and are trained for the same number of steps, and vary only by their fraction of low-alignment pairs. The results show that the gain of AWD is indeed greater on a noisier subset (+2.6 points with 75% of low-alignment pairs) than on a cleaner one (+1.7 points without any).
  • Repetition of the pairs. We compare two trainings of 20k steps on DataComp-M, on the full pool (80M pairs) and on the 50% subset of Fig. 2 (10M pairs), which has the same distribution of scores and therefore the same level of noise. They differ only by their size, so each pair of the subset is seen about 8 times more often. The gain of AWD is about twice as large on the subset, +2.1 points against +1.1.

W2. Filtering + AWD on CC12M.

We agree that on CC12M, adding AWD to filtering slightly lowers the results (0.4 points on average over the metrics of Table 1), and we do not have a clear explanation for it. It is, however, the only such case in the paper: in every other setting where we compare a method with and without AWD, adding AWD improves the average over the reported metrics (Avg. Δ in the tables). This holds on both corpora, for CLIP-KD and TIPSv2 distillation, combined with filtering on DataComp-M, with both ACID selection methods, and after recaptioning. On CC12M, the best results are obtained by keeping all the pairs: AWD alone gains 3.5 points, whereas filtering alone already loses 1.4.

W3. Cost of the score in TIPSv2.

We thank the reviewer for pointing this out. In TIPSv2 distillation, the text embeddings of the teacher are indeed not needed otherwise, so the score requires an additional forward pass of its text encoder. We measured its cost: AWD adds 5.5% (CC12M) and 6.2% (DataComp-M) to the training time of a TIPSv2 step, against about 1% for CLIP-KD, where these embeddings are computed anyway. We will report these numbers in the paper and clarify that the overhead of AWD is negligible (+1%) for CLIP-KD and small (+5 to 6%) for TIPSv2.

Q1. Related work on continuous weighting.

We thank the reviewer for these references. We will cite CWCL, SoftCLIP and NLIP in the related work, in the paragraph on alignment-aware learning from noisy pairs, and discuss how AWD, which applies such a weighting in the distillation setting, relates to them.

Thank you for the clarifications; please add them to the paper, especially the clarification on TIPSv2. I maintain my positive assessment of the work!

We thank the reviewer for the follow-up and the positive assessment. We have included these clarifications in the revised version, with the changes marked in blue.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their constructive feedback, which has helped us strengthen the paper. The reviewers recognized that AWD brings consistent gains at no or marginal additional cost (vvNX, pe7H), that its conceptual insight is clear (vvNX), that its ablations are well designed (vvNX, rPtF), and that its evaluation is broad (pe7H, rPtF). The main concerns have been addressed through additional experiments, clarifications, and planned updates for the revised version, which we summarize below.

  • Novelty and positioning (pe7H, rPtF). AWD addresses a question specific to distillation, what a weakly aligned pair should contribute when its caption is not its only source of supervision. We clarified how it differs from ALIP and from filtering, in setting and in mechanism, and will sharpen this positioning in the related work.
  • AWD versus filtering (pe7H, vvNX). We do not argue against data curation: filtering and AWD are complementary, and their combination is the best configuration on DataComp-M. We conducted a human study with five annotators, which shows that a low teacher score flags unusable pairs on DataComp-M (blank images, captions that describe nothing, such as "9580031_1.jpg") but mostly weakly aligned yet usable ones on CC12M, and explains why filtering helps on one corpus and hurts on the other. The scope of our claim will be made explicit in the introduction and the discussion.
  • Additional experiments (pe7H, rPtF). We ran three seeds on DataComp-M, where AWD gains +1.21 ± 0.12 points on average over CLIP-KD and improves every metric on every seed; we evaluated the loss-split ablation of Table 4 on all the metrics of Table 1, where weighting only the contrastive losses remains best on every metric; and we tested AWD on two more students (ViT-S/16, ResNet-50). The seed study and the additional backbones will be added to the appendix, and Table 4 will be extended.

We sincerely appreciate the reviewers' constructive feedback, which has greatly improved the clarity, scope and completeness of this work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.