acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Geometry Distillation for Vision–Language Dual Encoders

Abstract

We propose contrastive geometry distillation (CGD) for vision–language dual encoders. A decomposition of the teacher–student mutual information gap motivates two variants that combine feature distillation with KL matching of caption distributions for each image. CGD-N also matches distributions over unmatched images for each caption, while CGD-IW corrects selected unmatched pairs using teacher-to-student probability ratios. For CGD-N, exact matching recovers the teacher’s similarity matrix up to a global additive constant. Experiments show that CGD lowers similarities for negative pairs more than for positive pairs, widening retrieval margins while retaining alignment of image features with the teacher. The added losses supply small gradient corrections in directions distinct from feature distillation. The resulting students preserve the teacher’s candidate ordering more closely than CLIP-KD and CLIP-RD, with improved average retrieval and comparable classification accuracy.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
knowledge distillation, multimodal representation learning, vision-language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KKF3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Contrastive Geometry Distillation (CGD), a knowledge distillation framework designed to transfer the cross-modal similarity structure of a large vision-language teacher model to a smaller student model. The authors argue that conventional feature distillation does not sufficiently preserve the relative similarities and rankings of image-text pairs, particularly for challenging negative examples. To address this issue, authors propose two variants: 1) CGD-N: Matches teacher-student distributions over unmatched images using a negative-column KL divergence objective, and 2) CGD-IW: Uses importance-weighted corrections based on teacher-to-student probability ratios to refine selected unmatched image-text similarities. They then provide a theoretical analysis based on a decomposition of the teacher-student mutual information gap and show how the proposed objectives constrain the student’s similarity matrix. Experiments on various datasets such as MSCOCO, Flickr30K, CC3M, and ImageNet variants demonstrate that CGD achieves modest improvements in image-text retrieval over CLIP-KD and CLIP-RD while maintaining comparable zero-shot classification accuracy.

Critical strengths

The paper has the following strengths -

  1. I admire the fact that the Introduction and the whole paper in general are easy to follow.
  2. The problem they are trying to solve is timely.
Critical weaknesses

The paper has two critical weaknesses -

  1. Marginal performance improvements over existing distillation methods - I find that the empirical improvements over existing approaches are too marginal to convincingly establish the effectiveness of the proposed framework.
  • In Table 1 (Section 4.2), CGD-N and CGD-IW achieve average retrieval R@1 scores of 43.11 and 43.17, respectively, compared to 42.64 for CLIP-RD, corresponding to improvements of only 0.47–0.53 percentage points. The zero-shot classification improvements are similarly negligible, with CGD occasionally underperforming CLIP-RD.
  • Furthermore, Table 4 (Section 4.3) shows that the improvements are inconsistent across student architectures, particularly for ResNet-18 and MobileViT-S.

I would like to see stronger empirical improvements, statistical significance analysis, and comparisons of training time, computational overhead, and memory consumption to justify the additional distillation objectives.

  1. Limited methodological novelty and insufficient validation of the proposed theoretical motivation - I find the methodological novelty relatively limited, as the proposed approach primarily modifies existing feature and relational distillation objectives through alternative KL-divergence weighting and importance-weighted corrections.
  • In Section 2.3, the authors acknowledge that CLIP-KD and TinyCLIP already perform row- and column-wise distribution matching, while Section 3.2 introduces CGD-N by modifying the weighting of the negative-column KL term. Although the theoretical analysis in Section 3.1 motivates preserving contrastive geometry, I am not convinced that the experiments establish a substantial advantage from the proposed formulation.
  • In particular, Figure 7 (Section 4.5) shows that CGD improves global rank agreement but reduces top-64 candidate overlap, raising questions about whether it consistently preserves the most relevant retrieval relationships.

I believe more direct comparisons with alternative negative weighting strategies and stronger ablations are necessary to establish the novelty and practical significance of the proposed objectives.

Questions for authors

See Weaknesses section.

10 comments
AuthorsOpenReview ↗

Marginal improvements, significance, and student architectures [1/2]

Thank you for pointing out that the gains are small and that classification is on par with or below CLIP-RD, and for asking for significance tests.

We ran statistical tests, and we share the results of t-tests over independent training seeds on the performance gaps to CLIP-RD. In the tables of our responses, "from submission" marks the settings of the submitted paper, and "from rebuttal" marks the two settings that we selected during the rebuttal and describe below. The table below compares each of them with CLIP-RD, with 5 seeds for CLIP-RD and for the settings from the submission and 3 seeds for the settings from the rebuttal.

Comparison (vs. CLIP-RD)MetricMean difference (points)95% CI (points)p (Welch's t-test)
CGD-N (γ = 0.0382, from submission)Retrieval (R@1)+0.468[0.118, 0.818]0.0174
Classification (Top-1)+0.036[−0.286, 0.359]0.7891
CGD-IW (λ = 20.92 → γ = 0.10, from submission)Retrieval (R@1)+0.534[−0.070, 1.137]0.0743
Classification (Top-1)−0.045[−0.708, 0.618]0.8735
CGD-N (γ = 1.00, from rebuttal)Retrieval (R@1)+0.860[0.389, 1.332]0.00516
Classification (Top-1)+1.375[0.961, 1.789]0.000272
CGD-IW (γ = 0.50 → 0.20, from rebuttal)Retrieval (R@1)+0.724[−0.205, 1.654]0.0891
Classification (Top-1)+1.212[0.434, 1.991]0.0151

As you pointed out, the improvement in classification was not significant for either variant in the submission. We therefore ran additional experiments on the strength of each added loss.

The first was to decide the coefficient of the LN\mathcal L_N term, that is, the ratio γ\gamma between the feature-gradient norm of the LN\mathcal L_N term and that of the Lbase\mathcal L_{\mathrm{base}} term. The value in the submission was 0.0382, which had been matched to the value measured for the LIW\mathcal L_{\mathrm{IW}} term (Appendix D.1). To check whether this level was appropriate, we raised it to 0.20, 0.30 and 1.00 and trained again. We judged that the simplest value, 1, which gives the two terms equal gradient norms, is appropriate for the γ\gamma of CGD-N, and we selected it.

For CGD-IW, the submission used a fixed coefficient (λ\lambda = 20.92) and, in the last quarter of training, introduced a controller that adjusts the same ratio γ\gamma for the LIW\mathcal L_{\mathrm{IW}} term to 0.10 (Appendix D.1). We re-examined the training dynamics of CGD-IW and found that, when a large γ\gamma is kept until the end, the LIW\mathcal L_{\mathrm{IW}} term tends to conflict with the other loss terms late in training. To suppress this, we used a controller in which γ\gamma stays at 0.50 for the first three quarters of training and is lowered to 0.20 afterwards, and we confirmed that this controller shows comparatively stable training dynamics. We also changed the candidates that LIW\mathcal L_{\mathrm{IW}} corrects. The submission uses, for each image, the 64 unmatched captions that are most similar under the teacher (Top-64). The rebuttal setting adds the candidates that are most similar under the student, namely 32 unmatched captions for each image and 32 unmatched images for each caption (MIX32). With this controller the training dynamics improve stably, and both classification and retrieval improve over keeping γ\gamma = 0.50 until the end.

The table below extends Table 4 of the paper with the two rows marked "from rebuttal" (seed 0, 32 epochs, %). CGD-N (γ = 1.00) has the highest classification accuracy for all four student architectures, and its retrieval exceeds FD and CLIP-KD on all four and CLIP-RD on EfficientNet-B0 and Swin-Tiny. CGD-IW (γ = 0.50 → 0.20) exceeds FD, CLIP-KD and CLIP-RD in classification on all four, and has the highest retrieval R@1 on ResNet-18, EfficientNet-B0 and Swin-Tiny.

MethodResNet-18 Top-1ResNet-18 R@1EfficientNet-B0 Top-1EfficientNet-B0 R@1MobileViT-S Top-1MobileViT-S R@1Swin-Tiny Top-1Swin-Tiny R@1
FD34.9737.7142.6444.7242.3745.6453.4352.90
CLIP-KD35.7138.1243.0745.0243.0446.3853.4352.94
CLIP-RD35.8438.7543.4845.2443.4147.1454.2453.81
CGD-N (γ = 0.0382, from submission)35.9438.2543.6045.5943.2047.02‾\underline{47.02}54.2953.85
CGD-IW (λ = 20.92 → γ = 0.10, from submission)35.7438.76‾\underline{38.76}43.7045.3843.2246.8254.3853.97
CGD-N (γ = 1.00, from rebuttal)36.1138.6444.5545.97‾\underline{45.97}44.4046.7056.0954.09‾\underline{54.09}
CGD-IW (γ = 0.50 → 0.20, from rebuttal)35.95‾\underline{35.95}38.8744.29‾\underline{44.29}46.2144.12‾\underline{44.12}46.8155.19‾\underline{55.19}54.26

The per-dataset results for ViT-T/16 are in the next comment, which extends Table 1 of the paper.

AuthorsOpenReview ↗

Marginal improvements, significance, and student architectures [2/2]

The two tables below extend Table 1 of the paper with three rows, FD + KL (base) and the two rows marked "from rebuttal", and with the average over datasets. The row FD + KL (base), trained only with the loss Lbase\mathcal L_{\mathrm{base}} that we present in Section 3.2, shows the increment that the designs specific to CGD-N and CGD-IW actually contribute to the performance. Entries are mean ± sample standard deviation over seeds, and the parentheses in the two rows from the rebuttal give the difference in points from the highest mean among FD, CLIP-KD and CLIP-RD.

(a) Classification Top-1 (%)

MethodSeedsImageNetImageNet-V2ImageNet-RImageNet-SketchAvg.
FD542.22 ± 0.3636.48 ± 0.2746.54 ± 0.3725.72 ± 0.2237.74 ± 0.22
CLIP-KD543.08 ± 0.3037.11 ± 0.5347.73 ± 0.5126.93 ± 0.5138.71 ± 0.43
CLIP-RD543.05 ± 0.1937.06 ± 0.1748.53 ± 0.4827.66 ± 0.4639.07 ± 0.26
FD + KL (base)542.90 ± 0.1037.15 ± 0.3148.24 ± 0.3426.92 ± 0.3838.80 ± 0.14
CGD-N (γ = 0.0382, from submission)543.18 ± 0.3137.37 ± 0.4548.54 ± 0.1827.35 ± 0.2739.11 ± 0.12
CGD-IW (λ = 20.92 → γ = 0.10, from submission)543.08 ± 0.4237.10 ± 0.3148.63 ± 0.9527.31 ± 0.5739.03 ± 0.54
CGD-N (γ = 1.00, from rebuttal)344.27 ± 0.20 (+1.19)38.55 ± 0.04 (+1.44)50.24‾\underline{50.24} ± 0.38 (+1.71)28.72‾\underline{28.72} ± 0.23 (+1.06)40.45 ± 0.20 (+1.38)
CGD-IW (γ = 0.50 → 0.20, from rebuttal)344.05‾\underline{44.05} ± 0.25 (+0.97)37.92‾\underline{37.92} ± 0.53 (+0.81)50.30 ± 0.50 (+1.77)28.87 ± 0.46 (+1.21)40.29‾\underline{40.29} ± 0.39 (+1.22)

(b) Retrieval R@1 (%)

MethodSeedsCOCO I2TCOCO T2IFlickr30K I2TFlickr30K T2ICC3M-val I2TCC3M-val T2IAvg.
FD540.38 ± 0.4527.57 ± 0.1365.28 ± 1.1651.39 ± 0.8033.82 ± 0.4433.43 ± 0.2141.98 ± 0.24
CLIP-KD541.98 ± 0.7027.70 ± 0.3767.32 ± 1.0751.88 ± 0.7334.08 ± 0.2432.60 ± 0.0942.59 ± 0.43
CLIP-RD542.18 ± 0.3627.89 ± 0.2966.84 ± 1.0151.73 ± 0.5734.22 ± 0.1632.97 ± 0.3042.64 ± 0.29
FD + KL (base)541.63 ± 0.4127.97 ± 0.2267.44 ± 0.3551.98 ± 0.9234.03 ± 0.3533.67 ± 0.2242.79 ± 0.27
CGD-N (γ = 0.0382, from submission)542.61‾\underline{42.61} ± 0.2428.15 ± 0.1867.34 ± 0.3952.44 ± 0.2934.36 ± 0.2733.74‾\underline{33.74} ± 0.2643.11 ± 0.13
CGD-IW (λ = 20.92 → γ = 0.10, from submission)542.32 ± 0.5428.13 ± 0.4267.58 ± 1.6352.39 ± 0.6134.86 ± 0.4033.75 ± 0.2743.17 ± 0.48
CGD-N (γ = 1.00, from rebuttal)343.73 ± 0.18 (+1.55)28.46‾\underline{28.46} ± 0.21 (+0.57)69.93 ± 1.53 (+2.61)53.69 ± 0.10 (+1.81)33.63 ± 0.20 (−0.59)31.55 ± 0.64 (−1.88)43.50 ± 0.23 (+0.86)
CGD-IW (γ = 0.50 → 0.20, from rebuttal)342.61‾\underline{42.61} ± 0.22 (+0.43)28.55 ± 0.46 (+0.66)68.03‾\underline{68.03} ± 0.95 (+0.71)52.84‾\underline{52.84} ± 1.27 (+0.96)34.54‾\underline{34.54} ± 0.46 (+0.32)33.60 ± 0.42 (+0.17)43.36‾\underline{43.36} ± 0.45 (+0.72)

We also briefly present how we arrived at these two settings, and Appendices G.1 and G.2 of the revision give the full comparisons. For CGD-N, the table below compares the values of γ that we tried on seed 0.

γ of CGD-N (seed 0)Avg. retrieval R@1 (%)Avg. classification Top-1 (%)
0.0382 (from submission)43.0339.16
0.2043.6240.44
0.3043.3840.39
1.00 (from rebuttal)43.2940.60

For CGD-IW, the table below compares the setting of the submission with the settings of γ that we tried on seed 0. Its first row uses the Top-64 candidates and the other rows use MIX32 (both defined in the previous comment).

Setting of CGD-IW (seed 0)Avg. retrieval R@1 (%)Avg. classification Top-1 (%)
λ = 20.92 until epoch 24, then γ = 0.10 (from submission)43.19839.687
γ = 0.20 for all 32 epochs43.60039.932
γ = 0.50 for all 32 epochs42.87039.972
γ = 0.50 until epoch 24, then γ = 0.20 (from rebuttal)43.80140.479
AuthorsOpenReview ↗

Methodological novelty and its validation [1/2]

Thank you for raising the relation to existing distillation methods. We agree that CLIP-KD, CLIP-RD, and TinyCLIP already match row and column distributions, and that Section 3.2 introduces CGD-N through the decomposition of DKD. Our analysis in Section 3 shows, however, that column-KL matching limits the supervision of unmatched image candidates (i≠ji \ne j) in two ways, which CGD-N and CGD-IW address. The table below gives each method's column loss and its gradient on an unmatched image.

MethodColumn loss on caption jjGradient gij=−∂L/∂SijSg_{ij}=-\partial\mathcal L/\partial S^S_{ij} on unmatched image ii
CLIP-KD, CLIP-RD, TinyCLIPMj+(1−cjT(j)) NjM_j+(1-c_j^T(j))\,N_jcjS(i) (Wij−1)c_j^S(i)\,(W_{ij}-1)
DKDαMj+βNj\alpha M_j+\beta N_jcjS,−(i) [α (cjS(j)−cjT(j))+β (Wij−−1)]c_j^{S,-}(i)\,[\alpha\,(c_j^S(j)-c_j^T(j))+\beta\,(W^-_{ij}-1)]
CGD-NλNNj\lambda_N N_j (Eq. (17))cjS,−(i) (Wij−−1)c_j^{S,-}(i)\,(W^-_{ij}-1)
CGD-IWλIW∑i:(i,j)∈H(1−W~ij) efS(i,j)−1\lambda_{\mathrm{IW}}\sum_{i:(i,j)\in H}(1-\widetilde W_{ij})\,e^{f^S(i,j)-1} (Eq. (15))[cjS(i)]a (W~ij−1)[c_j^S(i)]^{a}\,(\widetilde W_{ij}-1) on selected pairs

Here MjM_j and Nj=KL(cjT,−∥cjS,−)N_j=\mathrm{KL}(c_j^{T,-}\Vert c_j^{S,-}) are the matched and unmatched parts of the column KL KL(cjT∥cjS)\mathrm{KL}(c_j^T\Vert c_j^S), Wij=cjT(i)/cjS(i)W_{ij}=c_j^T(i)/c_j^S(i) and Wij−=cjT,−(i)/cjS,−(i)W^-_{ij}=c_j^{T,-}(i)/c_j^{S,-}(i) are the teacher-to-student probability ratios, W~ij\widetilde W_{ij} is WijW_{ij} normalized over the selected pairs HH, aa is the slope of the critic fSf^S, and positive constants are omitted in the gradients (Section 3.2).

CGD-N weights the unmatched images independently of the teacher's confidence. In CLIP-KD, CLIP-RD, and TinyCLIP, the unmatched KL NjN_j is weighted by 1−cjT(j)1-c_j^T(j), which is close to zero whenever the teacher is confident in the matched pair, suppressing supervision on unmatched images. DKD uses constant weights (αMj+βNj\alpha M_j+\beta N_j), but in dual encoders, Proposition 2 (Eq. (18)) proves that once the row loss LKL\mathcal L_{\mathrm{KL}} matches each image's caption distribution, Nj=0N_j=0 for every caption gives P=QP=Q (SS=ST+c11⊤S^S=S^T+c\mathbf 1\mathbf 1^\top, B≥3B\ge3). Because MjM_j then adds no further constraint, CGD-N drops it, which removes the term α(cjS(j)−cjT(j))\alpha(c_j^S(j)-c_j^T(j)) of the matched image from DKD's gradient and supervises unmatched images independently of cjT(j)c_j^T(j).

CGD-IW addresses the vanishing gradients that KL-based losses assign to unmatched candidates with a small student probability. In the first three rows, and under any differentiable ff-divergence between column distributions (Appendix A), the gradient on an unmatched image scales linearly with the student probability cjS(i)c_j^S(i). Thus, candidates assigned a small probability by the student receive almost no gradient even when their probability ratio is far from 11. Derived from the Δinfo\Delta_{\mathrm{info}} term of the MI gap (Eq. (11)), LIW\mathcal L_{\mathrm{IW}} replaces cjS(i)c_j^S(i) with [cjS(i)]a[c_j^S(i)]^a, and we choose a=0.05a=0.05: when cjS(i)c_j^S(i) drops from 10−210^{-2} to 10−410^{-4} at the same ratio, KL-based gradients shrink by a factor of 100, whereas CGD-IW's gradient shrinks by a factor of only 1.26.

The following comment compares these methods in accuracy, retrieval margin, and agreement with the teacher's candidate ordering.

Reviewer_4x54OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper is about distillation of a cross-modal two encoders model (eg CLIP) from large to smaller architectures. The losses are motivated by a decomposition of a batch-level mutual information gap between teacher and student. The perform experiments of distillation with some standard ViT backbones and evaluate performance on standard cross-modal tasks.

Critical strengths
  • The paper includes a theoretically motivated formulation of CLIP distillation as matching the teacher’s relative cross-modal score structure, with a clean result for the proposed row and column objectives.
  • This is a plug-in objective that add little overhead and work on top of standard feature distillation.
  • The evaluation is rigorous, and clean with multi-seed statistics.
Critical weaknesses

W1 - The evaluation is extensive but despite the many different tasks and settings covered, the proposed method always shows insignificant or non consistent improvements, in most of the cases the improvement is within 1 std.

W2 - The paper is not easy to follow because it is very densely written and leaves several intemediate steps to the reader.

W3 - Disconnect between the MI framing and the objective. The method is framed as reducing a teacher–student MI gap, yet the authors note that equal MI does not preserve candidate relations and that the real target is P=QP = Q. So the MI seems like a detour that gives a principled-looking narrative while the losses actually come from matching PP and QQ plus some substitutions.

W4 - there is no comparison to TinyCLIP, MobileCLIP, CLIP-CID or DKD, while all these are quite relevant. The manuscipt acknowledges this by discussing them, but the dicrect comparison is missing.

post-rebuttal: I will maintain my original rating mainly due to the incremental and non-consistent improvements. The latter holds in the update, while the former is handled but "and we also looked at its benchmark results during this exploration" it seems to be through tuning on the test set performance, which is not a valid practice.

Questions for authors

The major reasons for rejecting the paper is the experimental validation and results (W1, W4) - the authors should tackle that in their response. W2 adds up into grading the paper as not ready for publication at ICLR.

7 comments
AuthorsOpenReview ↗

W1 [1/2]: significance and consistency of the improvements

Thank you for pointing out that the gains are within the seed standard deviation and are not consistent.

We ran statistical tests, and we share the results of t-tests over independent training seeds on the performance gaps to CLIP-RD. In the tables of our responses, "from submission" marks the settings of the submitted paper, and "from rebuttal" marks the two settings that we selected during the rebuttal and describe below. The table below compares each of them with CLIP-RD, with 5 seeds for CLIP-RD and for the settings from the submission and 3 seeds for the settings from the rebuttal.

Comparison (vs. CLIP-RD)MetricMean difference (points)95% CI (points)p (Welch's t-test)
CGD-N (γ = 0.0382, from submission)Retrieval (R@1)+0.468[0.118, 0.818]0.0174
Classification (Top-1)+0.036[−0.286, 0.359]0.7891
CGD-IW (λ = 20.92 → γ = 0.10, from submission)Retrieval (R@1)+0.534[−0.070, 1.137]0.0743
Classification (Top-1)−0.045[−0.708, 0.618]0.8735
CGD-N (γ = 1.00, from rebuttal)Retrieval (R@1)+0.860[0.389, 1.332]0.00516
Classification (Top-1)+1.375[0.961, 1.789]0.000272
CGD-IW (γ = 0.50 → 0.20, from rebuttal)Retrieval (R@1)+0.724[−0.205, 1.654]0.0891
Classification (Top-1)+1.212[0.434, 1.991]0.0151

As you pointed out, the improvement in classification was not significant for either variant in the submission. We therefore ran additional experiments on the strength of each added loss.

The first was to decide the coefficient of the LN\mathcal L_N term, that is, the ratio γ\gamma between the feature-gradient norm of the LN\mathcal L_N term and that of the Lbase\mathcal L_{\mathrm{base}} term. The value in the submission was 0.0382, which had been matched to the value measured for the LIW\mathcal L_{\mathrm{IW}} term (Appendix D.1). To check whether this level was appropriate, we raised it to 0.20, 0.30 and 1.00 and trained again. We judged that the simplest value, 1, which gives the two terms equal gradient norms, is appropriate for the γ\gamma of CGD-N, and we selected it.

For CGD-IW, the submission used a fixed coefficient (λ\lambda = 20.92) and, in the last quarter of training, introduced a controller that adjusts the same ratio γ\gamma for the LIW\mathcal L_{\mathrm{IW}} term to 0.10 (Appendix D.1). We re-examined the training dynamics of CGD-IW and found that, when a large γ\gamma is kept until the end, the LIW\mathcal L_{\mathrm{IW}} term tends to conflict with the other loss terms late in training. To suppress this, we used a controller in which γ\gamma stays at 0.50 for the first three quarters of training and is lowered to 0.20 afterwards, and we confirmed that this controller shows comparatively stable training dynamics. We also changed the candidates that LIW\mathcal L_{\mathrm{IW}} corrects. The submission uses, for each image, the 64 unmatched captions that are most similar under the teacher (Top-64). The rebuttal setting adds the candidates that are most similar under the student, namely 32 unmatched captions for each image and 32 unmatched images for each caption (MIX32). With this controller the training dynamics improve stably, and both classification and retrieval improve over keeping γ\gamma = 0.50 until the end.

Per dataset, CGD-N with γ=1.00\gamma=1.00 is also above CLIP-RD on all four classification datasets and in both retrieval directions of MSCOCO and Flickr30K, and each of these eight differences is larger than the seed standard deviation of either method.

With other student architectures (seed 0, epoch 32), CGD-N (γ=1.00\gamma=1.00) has the highest classification accuracy on all four, and its retrieval exceeds CLIP-RD on EfficientNet-B0 and Swin-Tiny. CGD-IW (γ=0.50→0.20\gamma=0.50\to0.20) exceeds CLIP-RD in both on ResNet-18 and EfficientNet-B0. The results for each architecture are in our response to Reviewer KKF3 ("Marginal improvements, significance, and student architectures [1/2]"), in a table that extends Table 4 of the paper.

The per-dataset results for ViT-T/16 are in the next comment, which extends Table 1 of the paper. The results of the TinyCLIP and DKD baselines you requested are in our response to W4.

AuthorsOpenReview ↗

W1 [2/2]: significance and consistency of the improvements

The two tables below extend Table 1 of the paper with three rows, FD + KL (base) and the two rows marked "from rebuttal", and with the average over datasets. The row FD + KL (base), trained only with the loss Lbase\mathcal L_{\mathrm{base}} that we present in Section 3.2, shows the increment that the designs specific to CGD-N and CGD-IW actually contribute to the performance. Entries are mean ± sample standard deviation over seeds, and the parentheses in the two rows from the rebuttal give the difference in points from the highest mean among FD, CLIP-KD and CLIP-RD.

(a) Classification Top-1 (%)

MethodSeedsImageNetImageNet-V2ImageNet-RImageNet-SketchAvg.
FD542.22 ± 0.3636.48 ± 0.2746.54 ± 0.3725.72 ± 0.2237.74 ± 0.22
CLIP-KD543.08 ± 0.3037.11 ± 0.5347.73 ± 0.5126.93 ± 0.5138.71 ± 0.43
CLIP-RD543.05 ± 0.1937.06 ± 0.1748.53 ± 0.4827.66 ± 0.4639.07 ± 0.26
FD + KL (base)542.90 ± 0.1037.15 ± 0.3148.24 ± 0.3426.92 ± 0.3838.80 ± 0.14
CGD-N (γ = 0.0382, from submission)543.18 ± 0.3137.37 ± 0.4548.54 ± 0.1827.35 ± 0.2739.11 ± 0.12
CGD-IW (λ = 20.92 → γ = 0.10, from submission)543.08 ± 0.4237.10 ± 0.3148.63 ± 0.9527.31 ± 0.5739.03 ± 0.54
CGD-N (γ = 1.00, from rebuttal)344.27 ± 0.20 (+1.19)38.55 ± 0.04 (+1.44)50.24‾\underline{50.24} ± 0.38 (+1.71)28.72‾\underline{28.72} ± 0.23 (+1.06)40.45 ± 0.20 (+1.38)
CGD-IW (γ = 0.50 → 0.20, from rebuttal)344.05‾\underline{44.05} ± 0.25 (+0.97)37.92‾\underline{37.92} ± 0.53 (+0.81)50.30 ± 0.50 (+1.77)28.87 ± 0.46 (+1.21)40.29‾\underline{40.29} ± 0.39 (+1.22)

(b) Retrieval R@1 (%)

MethodSeedsCOCO I2TCOCO T2IFlickr30K I2TFlickr30K T2ICC3M-val I2TCC3M-val T2IAvg.
FD540.38 ± 0.4527.57 ± 0.1365.28 ± 1.1651.39 ± 0.8033.82 ± 0.4433.43 ± 0.2141.98 ± 0.24
CLIP-KD541.98 ± 0.7027.70 ± 0.3767.32 ± 1.0751.88 ± 0.7334.08 ± 0.2432.60 ± 0.0942.59 ± 0.43
CLIP-RD542.18 ± 0.3627.89 ± 0.2966.84 ± 1.0151.73 ± 0.5734.22 ± 0.1632.97 ± 0.3042.64 ± 0.29
FD + KL (base)541.63 ± 0.4127.97 ± 0.2267.44 ± 0.3551.98 ± 0.9234.03 ± 0.3533.67 ± 0.2242.79 ± 0.27
CGD-N (γ = 0.0382, from submission)542.61‾\underline{42.61} ± 0.2428.15 ± 0.1867.34 ± 0.3952.44 ± 0.2934.36 ± 0.2733.74‾\underline{33.74} ± 0.2643.11 ± 0.13
CGD-IW (λ = 20.92 → γ = 0.10, from submission)542.32 ± 0.5428.13 ± 0.4267.58 ± 1.6352.39 ± 0.6134.86 ± 0.4033.75 ± 0.2743.17 ± 0.48
CGD-N (γ = 1.00, from rebuttal)343.73 ± 0.18 (+1.55)28.46‾\underline{28.46} ± 0.21 (+0.57)69.93 ± 1.53 (+2.61)53.69 ± 0.10 (+1.81)33.63 ± 0.20 (−0.59)31.55 ± 0.64 (−1.88)43.50 ± 0.23 (+0.86)
CGD-IW (γ = 0.50 → 0.20, from rebuttal)342.61‾\underline{42.61} ± 0.22 (+0.43)28.55 ± 0.46 (+0.66)68.03‾\underline{68.03} ± 0.95 (+0.71)52.84‾\underline{52.84} ± 1.27 (+0.96)34.54‾\underline{34.54} ± 0.46 (+0.32)33.60 ± 0.42 (+0.17)43.36‾\underline{43.36} ± 0.45 (+0.72)

We also briefly present how we arrived at these two settings, and Appendices G.1 and G.2 of the revision give the full comparisons. For CGD-N, the table below compares the values of γ that we tried on seed 0.

γ of CGD-N (seed 0)Avg. retrieval R@1 (%)Avg. classification Top-1 (%)
0.0382 (from submission)43.0339.16
0.2043.6240.44
0.3043.3840.39
1.00 (from rebuttal)43.2940.60

For CGD-IW, the table below compares the setting of the submission with the settings of γ that we tried on seed 0. Its first row uses the Top-64 candidates and the other rows use MIX32 (both defined in the previous comment).

Setting of CGD-IW (seed 0)Avg. retrieval R@1 (%)Avg. classification Top-1 (%)
λ = 20.92 until epoch 24, then γ = 0.10 (from submission)43.19839.687
γ = 0.20 for all 32 epochs43.60039.932
γ = 0.50 for all 32 epochs42.87039.972
γ = 0.50 until epoch 24, then γ = 0.20 (from rebuttal)43.80140.479
AuthorsOpenReview ↗

W2: dense writing and omitted steps

Thank you for pointing out that the paper is densely written and leaves intermediate steps to the reader. For this reason we rewrote Section 3, and we used the additional tenth page of the revision for it. The equations, the propositions and the claims are the same as in the submission. What changed is that each step is now written out, and the changes are marked in blue in the revised PDF. The table below lists the steps that the submission left to the reader and that the revision now states.

Place in the revisionStep that is now written out
Section 3.1, openingEqual mutual information does not fix the order of the candidates. For this reason the gap is decomposed into terms that compare PP and QQ directly, and we ask which loss controls each term.
Section 3.1, IQIWI_Q^{\mathrm{IW}} (Eq. (8))IPI_P and IQI_Q differ in two respects, the pointwise mutual information and the distribution over which it is averaged. The intermediate quantity changes one of the two at a time.
Section 3.1, before Proposition 1 (Eq. (10))The chain-rule step that turns the first difference into Δdist−KL(PV∥QV)\Delta_{\mathrm{dist}}-\mathrm{KL}(P_V\Vert Q_V) is displayed.
Section 3.1, after Proposition 1Matching the caption distribution of every image exactly leaves one constant per image undetermined, and these constants change the comparison among the images of a caption. LN\mathcal L_N and LIW\mathcal L_{\mathrm{IW}} address this comparison.
Section 3.1, Eq. (11)The role of each term is stated: J+\mathcal J_+ is bounded by feature distillation, J−\mathcal J_- contains the teacher-to-student probability ratio, and R\mathcal R is the slack.
Section 3.2, Lbase\mathcal L_{\mathrm{base}} (Eq. (13))The term that feature distillation controls and the term that LKL\mathcal L_{\mathrm{KL}} controls are named, followed by the two reasons why neither LFD\mathcal L_{\mathrm{FD}} nor LKL\mathcal L_{\mathrm{KL}} supervises the images of a caption directly.
Section 3.2, LIW\mathcal L_{\mathrm{IW}} (Eq. (15))The substitutions that lead from J−\mathcal J_- to the loss are introduced one at a time: the column ratio, the selected pairs HH, and the normalization.
Section 3.2, LN\mathcal L_N (Eq. (17))The column KL of a caption, KL(cjT∥cjS)\mathrm{KL}(c_j^T\Vert c_j^S), is displayed as the sum of a part for the matched image and a part for the unmatched images, and the part that CGD-N keeps is identified.
Figure 4 (pair differences)The figure showed results before the losses were defined. It now follows the margin identity (Eq. (19)) in Section 3.2.
Section 3.2, Eq. (20)The paragraph states what each gradient coefficient means and where each loss does not act.
Reviewer_o2xLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The central theorem is that during CLIP-like model distillation, we should preserve teacher's "contrastive geometry". In plain words the student must preserve how the teach ranks a batch of images given a caption. The proposed method contains a base objective = CLIP loss + feature distillation (FD) + row KL(treats each image as a query.) plus two correction terms: CGD-N is column KL (treats each caption as a query.) over unmatched images. CGD-IW is an importance-weighted correction on the teacher's hardest negatives.

The main setup is a large teacher (LAION ViT-B/16) and a small student (ViT-T/16), trained on CC3M+CC12M with 5 random seeds. Evaluation uses zero-shot ImageNet classification and image–text retrieval.

Critical strengths

The analysis is careful and the diagnostics are insightful. Experiments are well designed, e.g. the "continue training" experiments start every method from the same saved model and use the same data order, so differences come from the loss terms alone. The two add-on terms are intuitive.

Critical weaknesses
  • My main concern is the marginal gains over the strongest baseline. For example, against CLIP-RD, mean retrieval R@1 improves by roughly half a point, classification is on par.
Questions for authors
  • Can we add the TinyCLIP's affinity mimicking as a baseline?
  • Can we apply DKD as-is to rows and columns as a baseline?
  • How sensitive are the results to the controller that scales the added loss and to its schedule (e.g., CGD-IW is switched on only in the last quarter of training)?
  • I am giving a score of 2 but would increase the it if the gains are statistically significant or justified more clearly (for example add the two baselines)
10 comments
AuthorsOpenReview ↗

Gains over CLIP-RD and their statistical significance [1/2]

Thank you for asking us to consider the size of the gain together with the variation across seeds.

We ran statistical tests, and we share the results of t-tests over independent training seeds on the performance gaps to CLIP-RD. In the tables of our responses, "from submission" marks the settings of the submitted paper, and "from rebuttal" marks the two settings that we selected during the rebuttal and describe below. The table below compares each of them with CLIP-RD, with 5 seeds for CLIP-RD and for the settings from the submission and 3 seeds for the settings from the rebuttal.

Comparison (vs. CLIP-RD)MetricMean difference (points)95% CI (points)p (Welch's t-test)
CGD-N (γ = 0.0382, from submission)Retrieval (R@1)+0.468[0.118, 0.818]0.0174
Classification (Top-1)+0.036[−0.286, 0.359]0.7891
CGD-IW (λ = 20.92 → γ = 0.10, from submission)Retrieval (R@1)+0.534[−0.070, 1.137]0.0743
Classification (Top-1)−0.045[−0.708, 0.618]0.8735
CGD-N (γ = 1.00, from rebuttal)Retrieval (R@1)+0.860[0.389, 1.332]0.00516
Classification (Top-1)+1.375[0.961, 1.789]0.000272
CGD-IW (γ = 0.50 → 0.20, from rebuttal)Retrieval (R@1)+0.724[−0.205, 1.654]0.0891
Classification (Top-1)+1.212[0.434, 1.991]0.0151

As you pointed out, the improvement in classification was not significant for either variant in the submission. We therefore ran additional experiments on the strength of each added loss.

The first was to decide the coefficient of the LN\mathcal L_N term, that is, the ratio γ\gamma between the feature-gradient norm of the LN\mathcal L_N term and that of the Lbase\mathcal L_{\mathrm{base}} term. The value in the submission was 0.0382, which had been matched to the value measured for the LIW\mathcal L_{\mathrm{IW}} term (Appendix D.1). To check whether this level was appropriate, we raised it to 0.20, 0.30 and 1.00 and trained again. We judged that the simplest value, 1, which gives the two terms equal gradient norms, is appropriate for the γ\gamma of CGD-N, and we selected it.

For CGD-IW, the submission used a fixed coefficient (λ\lambda = 20.92) and, in the last quarter of training, introduced a controller that adjusts the same ratio γ\gamma for the LIW\mathcal L_{\mathrm{IW}} term to 0.10 (Appendix D.1). We re-examined the training dynamics of CGD-IW and found that, when a large γ\gamma is kept until the end, the LIW\mathcal L_{\mathrm{IW}} term tends to conflict with the other loss terms late in training. To suppress this, we used a controller in which γ\gamma stays at 0.50 for the first three quarters of training and is lowered to 0.20 afterwards, and we confirmed that this controller shows comparatively stable training dynamics. We also changed the candidates that LIW\mathcal L_{\mathrm{IW}} corrects. The submission uses, for each image, the 64 unmatched captions that are most similar under the teacher (Top-64). The rebuttal setting adds the candidates that are most similar under the student, namely 32 unmatched captions for each image and 32 unmatched images for each caption (MIX32). With this controller the training dynamics improve stably, and both classification and retrieval improve over keeping γ\gamma = 0.50 until the end.

The per-dataset results for ViT-T/16 with these two settings are in the next comment, which extends Table 1 of the paper. The results of DKD and the TinyCLIP loss follow in the comment after it, and the results for each controller setting of CGD-IW, with the measurement of the late conflict, are in the comment "Sensitivity to the controller and its schedule [1/2]".

AuthorsOpenReview ↗

Gains over CLIP-RD and their statistical significance [2/2]

The two tables below extend Table 1 of the paper with three rows, FD + KL (base) and the two rows marked "from rebuttal", and with the average over datasets. The row FD + KL (base), trained only with the loss Lbase\mathcal L_{\mathrm{base}} that we present in Section 3.2, shows the increment that the designs specific to CGD-N and CGD-IW actually contribute to the performance. Entries are mean ± sample standard deviation over seeds, and the parentheses in the two rows from the rebuttal give the difference in points from the highest mean among FD, CLIP-KD and CLIP-RD.

(a) Classification Top-1 (%)

MethodSeedsImageNetImageNet-V2ImageNet-RImageNet-SketchAvg.
FD542.22 ± 0.3636.48 ± 0.2746.54 ± 0.3725.72 ± 0.2237.74 ± 0.22
CLIP-KD543.08 ± 0.3037.11 ± 0.5347.73 ± 0.5126.93 ± 0.5138.71 ± 0.43
CLIP-RD543.05 ± 0.1937.06 ± 0.1748.53 ± 0.4827.66 ± 0.4639.07 ± 0.26
FD + KL (base)542.90 ± 0.1037.15 ± 0.3148.24 ± 0.3426.92 ± 0.3838.80 ± 0.14
CGD-N (γ = 0.0382, from submission)543.18 ± 0.3137.37 ± 0.4548.54 ± 0.1827.35 ± 0.2739.11 ± 0.12
CGD-IW (λ = 20.92 → γ = 0.10, from submission)543.08 ± 0.4237.10 ± 0.3148.63 ± 0.9527.31 ± 0.5739.03 ± 0.54
CGD-N (γ = 1.00, from rebuttal)344.27 ± 0.20 (+1.19)38.55 ± 0.04 (+1.44)50.24‾\underline{50.24} ± 0.38 (+1.71)28.72‾\underline{28.72} ± 0.23 (+1.06)40.45 ± 0.20 (+1.38)
CGD-IW (γ = 0.50 → 0.20, from rebuttal)344.05‾\underline{44.05} ± 0.25 (+0.97)37.92‾\underline{37.92} ± 0.53 (+0.81)50.30 ± 0.50 (+1.77)28.87 ± 0.46 (+1.21)40.29‾\underline{40.29} ± 0.39 (+1.22)

(b) Retrieval R@1 (%)

MethodSeedsCOCO I2TCOCO T2IFlickr30K I2TFlickr30K T2ICC3M-val I2TCC3M-val T2IAvg.
FD540.38 ± 0.4527.57 ± 0.1365.28 ± 1.1651.39 ± 0.8033.82 ± 0.4433.43 ± 0.2141.98 ± 0.24
CLIP-KD541.98 ± 0.7027.70 ± 0.3767.32 ± 1.0751.88 ± 0.7334.08 ± 0.2432.60 ± 0.0942.59 ± 0.43
CLIP-RD542.18 ± 0.3627.89 ± 0.2966.84 ± 1.0151.73 ± 0.5734.22 ± 0.1632.97 ± 0.3042.64 ± 0.29
FD + KL (base)541.63 ± 0.4127.97 ± 0.2267.44 ± 0.3551.98 ± 0.9234.03 ± 0.3533.67 ± 0.2242.79 ± 0.27
CGD-N (γ = 0.0382, from submission)542.61‾\underline{42.61} ± 0.2428.15 ± 0.1867.34 ± 0.3952.44 ± 0.2934.36 ± 0.2733.74‾\underline{33.74} ± 0.2643.11 ± 0.13
CGD-IW (λ = 20.92 → γ = 0.10, from submission)542.32 ± 0.5428.13 ± 0.4267.58 ± 1.6352.39 ± 0.6134.86 ± 0.4033.75 ± 0.2743.17 ± 0.48
CGD-N (γ = 1.00, from rebuttal)343.73 ± 0.18 (+1.55)28.46‾\underline{28.46} ± 0.21 (+0.57)69.93 ± 1.53 (+2.61)53.69 ± 0.10 (+1.81)33.63 ± 0.20 (−0.59)31.55 ± 0.64 (−1.88)43.50 ± 0.23 (+0.86)
CGD-IW (γ = 0.50 → 0.20, from rebuttal)342.61‾\underline{42.61} ± 0.22 (+0.43)28.55 ± 0.46 (+0.66)68.03‾\underline{68.03} ± 0.95 (+0.71)52.84‾\underline{52.84} ± 1.27 (+0.96)34.54‾\underline{34.54} ± 0.46 (+0.32)33.60 ± 0.42 (+0.17)43.36‾\underline{43.36} ± 0.45 (+0.72)

We also briefly present how we arrived at these two settings, and Appendices G.1 and G.2 of the revision give the full comparisons. For CGD-N, the table below compares the values of γ that we tried on seed 0.

γ of CGD-N (seed 0)Avg. retrieval R@1 (%)Avg. classification Top-1 (%)
0.0382 (from submission)43.0339.16
0.2043.6240.44
0.3043.3840.39
1.00 (from rebuttal)43.2940.60

For CGD-IW, the table below compares the setting of the submission with the settings of γ that we tried on seed 0. Its first row uses the Top-64 candidates and the other rows use MIX32 (both defined in the previous comment).

Setting of CGD-IW (seed 0)Avg. retrieval R@1 (%)Avg. classification Top-1 (%)
λ = 20.92 until epoch 24, then γ = 0.10 (from submission)43.19839.687
γ = 0.20 for all 32 epochs43.60039.932
γ = 0.50 for all 32 epochs42.87039.972
γ = 0.50 until epoch 24, then γ = 0.20 (from rebuttal)43.80140.479
AuthorsOpenReview ↗

Requested baselines: TinyCLIP and DKD

We measured the performance on the same task with the loss functions of DKD [1] and TinyCLIP [2]. DKD was not proposed for a vision–language dual encoder, so we applied it as in the dual-encoder structure: we computed PP and QQ on the B×BB\times B matrix of our paper, applied the loss to its rows and columns, and added it on top of the CLIP loss. For TinyCLIP, we removed the weight inheritance and trained with the affinity mimicking loss alone. The results are given below, in two tables that place the two baselines, with two seeds each, next to the CGD rows of Table 1 of the paper.

(a) Classification Top-1 (%)

MethodSeedsImageNetImageNet-V2ImageNet-RImageNet-SketchAvg.
CLIP + DKD242.04 ± 0.3536.06 ± 0.4550.15 ± 0.7428.61 ± 0.4839.21 ± 0.51
TinyCLIP affinity mimicking loss239.62 ± 0.2033.84 ± 0.3546.23 ± 0.3325.51 ± 0.4836.30 ± 0.34
CGD-N (γ = 0.0382, from submission)543.18 ± 0.3137.37 ± 0.4548.54 ± 0.1827.35 ± 0.2739.11 ± 0.12
CGD-IW (λ = 20.92 → γ = 0.10, from submission)543.08 ± 0.4237.10 ± 0.3148.63 ± 0.9527.31 ± 0.5739.03 ± 0.54
CGD-N (γ = 1.00, from rebuttal)344.27 ± 0.20 (+2.23)38.55 ± 0.04 (+2.49)50.24‾\underline{50.24} ± 0.38 (+0.09)28.72‾\underline{28.72} ± 0.23 (+0.11)40.45 ± 0.20 (+1.24)
CGD-IW (γ = 0.50 → 0.20, from rebuttal)344.05‾\underline{44.05} ± 0.25 (+2.01)37.92‾\underline{37.92} ± 0.53 (+1.86)50.30 ± 0.50 (+0.15)28.87 ± 0.46 (+0.26)40.29‾\underline{40.29} ± 0.39 (+1.08)

(b) Retrieval R@1 (%)

MethodSeedsCOCO I2TCOCO T2IFlickr30K I2TFlickr30K T2ICC3M-val I2TCC3M-val T2IAvg.
CLIP + DKD242.21 ± 0.0727.82 ± 0.0266.15 ± 1.2051.72 ± 1.3934.05 ± 0.2832.84 ± 0.6542.47 ± 0.57
TinyCLIP affinity mimicking loss241.14 ± 0.9326.25 ± 0.1665.00 ± 1.2749.48 ± 0.2031.15 ± 0.1529.26 ± 0.1040.38 ± 0.04
CGD-N (γ = 0.0382, from submission)542.61‾\underline{42.61} ± 0.2428.15 ± 0.1867.34 ± 0.3952.44 ± 0.2934.36 ± 0.2733.74‾\underline{33.74} ± 0.2643.11 ± 0.13
CGD-IW (λ = 20.92 → γ = 0.10, from submission)542.32 ± 0.5428.13 ± 0.4267.58 ± 1.6352.39 ± 0.6134.86 ± 0.4033.75 ± 0.2743.17 ± 0.48
CGD-N (γ = 1.00, from rebuttal)343.73 ± 0.18 (+1.52)28.46‾\underline{28.46} ± 0.21 (+0.64)69.93 ± 1.53 (+3.78)53.69 ± 0.10 (+1.97)33.63 ± 0.20 (−0.42)31.55 ± 0.64 (−1.29)43.50 ± 0.23 (+1.03)
CGD-IW (γ = 0.50 → 0.20, from rebuttal)342.61‾\underline{42.61} ± 0.22 (+0.40)28.55 ± 0.46 (+0.73)68.03‾\underline{68.03} ± 0.95 (+1.88)52.84‾\underline{52.84} ± 1.27 (+1.12)34.54‾\underline{34.54} ± 0.46 (+0.49)33.60 ± 0.42 (+0.76)43.36‾\underline{43.36} ± 0.45 (+0.89)

Entries are mean ± sample standard deviation over seeds. In the two rows marked "from rebuttal", the value in parentheses is the difference in points from the higher mean of DKD and TinyCLIP in the same column.

In the averages of both tables, CGD-N and CGD-IW with the settings from the rebuttal show a marked improvement over both DKD and TinyCLIP. This comparison is made under the same training protocol: for each of seeds 0 and 1, DKD and TinyCLIP were trained for 32 epochs from the same initial state and with the same data order as CGD. Within this protocol, DKD uses its released ImageNet setting (α = 1, β = 0.5, T = 1), and the average of its row loss and its column loss is added to the CLIP loss with weight 1, without feature distillation or LKL\mathcal L_{\mathrm{KL}}.

Of the two baselines, DKD is the more closely related to CGD-N, because both methods match the distribution over the unmatched candidates. They differ in what they match besides this distribution. DKD matches, in each row and in each column, the probability of the matched candidate against the total probability of the unmatched candidates. CGD-N instead matches the distribution over the captions for each image with LKL\mathcal L_{\mathrm{KL}} (Eq. (13)) and adds only LN\mathcal L_N (Eq. (17)), which matches the distribution over the unmatched images for each caption. Our response to Reviewer KKF3 ("Methodological novelty") explains what this difference changes in theory.

[1] B. Zhao, Q. Cui, R. Song, Y. Qiu, J. Liang. Decoupled Knowledge Distillation. CVPR 2022.

[2] K. Wu et al. TinyCLIP: CLIP Distillation via Affinity Mimicking and Weight Inheritance. ICCV 2023.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our response and revision

We thank the Area Chair and all reviewers for their time, effort, and constructive feedback. This comment summarizes the revision, in which all changes are highlighted in blue, focusing on the points raised by multiple reviewers.

1. Size and significance of the improvements (o2xL, 4x54, KKF3). We ran t-tests over independent training seeds, and we examined γ, the ratio between the gradient norms of the added loss and Lbase\mathcal L_{\mathrm{base}}, which was small in the submission. The losses, equations, and propositions are unchanged: CGD-N changes only γ, and CGD-IW changes the schedule of γ and its candidate set (MIX32).

MethodSeedsAvg. R@1 (%)Avg. Top-1 (%)
FD541.9837.74
TinyCLIP affinity mimicking240.3836.30
CLIP + DKD242.4739.21
CLIP-KD542.5938.71
CLIP-RD542.6439.07
CGD-N (γ = 1.00, from rebuttal)343.5040.45
CGD-IW (γ = 0.50 → 0.20, from rebuttal)343.36‾\underline{43.36}40.29‾\underline{40.29}

CGD-N improves over CLIP-RD by 0.86 points in R@1 (p = 0.005) and 1.38 in Top-1 (p = 0.0003), which is as large as the gain of CLIP-RD over FD (0.66 and 1.33; Tables 24 and 26). These gains remain on seeds 1 and 2 alone (+0.96 and +1.30), which were not used to select the settings. The results under the original settings are kept in the appendix (Tables 34, 39, and 41).

2. Difference from existing objectives (4x54, KKF3, o2xL). The row and column KL of CLIP-KD and CLIP-RD share the optimum P=QP=Q with CGD, but they supervise an unmatched image only weakly when the teacher is confident in the matched pair or when the student assigns the image a small probability. CGD-N addresses the first case and CGD-IW the second (Sec. 3.2). Unlike DKD, CGD-N also omits the matched term, which Proposition 2 shows is not needed to identify P=QP=Q. We rewrote Section 3 using the additional tenth page so that each step from the decomposition of the MI gap to the two losses is stated in the text.

3. Baselines and student architectures (o2xL, 4x54, KKF3). We added TinyCLIP's affinity mimicking and DKD applied to rows and columns, each trained with the loss of its own paper under the same protocol as CGD (Table 1, App. G.1, G.9). With ResNet-18, EfficientNet-B0, MobileViT-S, and Swin-Tiny as students, CGD-N achieves the highest classification accuracy across all four architectures, and CGD-IW outperforms FD, CLIP-KD, and CLIP-RD in classification on all four (Table 4, App. G.8).

4. Additional analyses. The revision also includes the following:

  • Sensitivity of CGD-IW to the controller, its schedule, and the candidate set (o2xL: App. G.2).
  • Ablations and training dynamics with the two settings above (4x54, KKF3: Table 3, Sec. 4.4, App. G.3).
  • Agreement with the teacher's candidate ordering, including the top-64 overlap (KKF3: Fig. 7, App. G.4, G.9).
  • Training time and memory, where CGD adds less than 1% to the time per step (KKF3: App. G.7).

We hope that the revision addresses your concerns, and we thank you again for the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.