- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Contrastive Geometry Distillation (CGD), a knowledge distillation framework designed to transfer the cross-modal similarity structure of a large vision-language teacher model to a smaller student model. The authors argue that conventional feature distillation does not sufficiently preserve the relative similarities and rankings of image-text pairs, particularly for challenging negative examples. To address this issue, authors propose two variants: 1) CGD-N: Matches teacher-student distributions over unmatched images using a negative-column KL divergence objective, and 2) CGD-IW: Uses importance-weighted corrections based on teacher-to-student probability ratios to refine selected unmatched image-text similarities. They then provide a theoretical analysis based on a decomposition of the teacher-student mutual information gap and show how the proposed objectives constrain the student’s similarity matrix. Experiments on various datasets such as MSCOCO, Flickr30K, CC3M, and ImageNet variants demonstrate that CGD achieves modest improvements in image-text retrieval over CLIP-KD and CLIP-RD while maintaining comparable zero-shot classification accuracy.
Critical strengths
The paper has the following strengths -
- I admire the fact that the Introduction and the whole paper in general are easy to follow.
- The problem they are trying to solve is timely.
Critical weaknesses
The paper has two critical weaknesses -
- Marginal performance improvements over existing distillation methods - I find that the empirical improvements over existing approaches are too marginal to convincingly establish the effectiveness of the proposed framework.
- In Table 1 (Section 4.2), CGD-N and CGD-IW achieve average retrieval R@1 scores of 43.11 and 43.17, respectively, compared to 42.64 for CLIP-RD, corresponding to improvements of only 0.47–0.53 percentage points. The zero-shot classification improvements are similarly negligible, with CGD occasionally underperforming CLIP-RD.
- Furthermore, Table 4 (Section 4.3) shows that the improvements are inconsistent across student architectures, particularly for ResNet-18 and MobileViT-S.
I would like to see stronger empirical improvements, statistical significance analysis, and comparisons of training time, computational overhead, and memory consumption to justify the additional distillation objectives.
- Limited methodological novelty and insufficient validation of the proposed theoretical motivation - I find the methodological novelty relatively limited, as the proposed approach primarily modifies existing feature and relational distillation objectives through alternative KL-divergence weighting and importance-weighted corrections.
- In Section 2.3, the authors acknowledge that CLIP-KD and TinyCLIP already perform row- and column-wise distribution matching, while Section 3.2 introduces CGD-N by modifying the weighting of the negative-column KL term. Although the theoretical analysis in Section 3.1 motivates preserving contrastive geometry, I am not convinced that the experiments establish a substantial advantage from the proposed formulation.
- In particular, Figure 7 (Section 4.5) shows that CGD improves global rank agreement but reduces top-64 candidate overlap, raising questions about whether it consistently preserves the most relevant retrieval relationships.
I believe more direct comparisons with alternative negative weighting strategies and stronger ablations are necessary to establish the novelty and practical significance of the proposed objectives.
Questions for authors
See Weaknesses section.
10 comments
Marginal improvements, significance, and student architectures [2/2]
The two tables below extend Table 1 of the paper with three rows, FD + KL (base) and the two rows marked "from rebuttal", and with the average over datasets. The row FD + KL (base), trained only with the loss that we present in Section 3.2, shows the increment that the designs specific to CGD-N and CGD-IW actually contribute to the performance. Entries are mean ± sample standard deviation over seeds, and the parentheses in the two rows from the rebuttal give the difference in points from the highest mean among FD, CLIP-KD and CLIP-RD.
(a) Classification Top-1 (%)
| Method | Seeds | ImageNet | ImageNet-V2 | ImageNet-R | ImageNet-Sketch | Avg. |
|---|---|---|---|---|---|---|
| FD | 5 | 42.22 ± 0.36 | 36.48 ± 0.27 | 46.54 ± 0.37 | 25.72 ± 0.22 | 37.74 ± 0.22 |
| CLIP-KD | 5 | 43.08 ± 0.30 | 37.11 ± 0.53 | 47.73 ± 0.51 | 26.93 ± 0.51 | 38.71 ± 0.43 |
| CLIP-RD | 5 | 43.05 ± 0.19 | 37.06 ± 0.17 | 48.53 ± 0.48 | 27.66 ± 0.46 | 39.07 ± 0.26 |
| FD + KL (base) | 5 | 42.90 ± 0.10 | 37.15 ± 0.31 | 48.24 ± 0.34 | 26.92 ± 0.38 | 38.80 ± 0.14 |
| CGD-N (γ = 0.0382, from submission) | 5 | 43.18 ± 0.31 | 37.37 ± 0.45 | 48.54 ± 0.18 | 27.35 ± 0.27 | 39.11 ± 0.12 |
| CGD-IW (λ = 20.92 → γ = 0.10, from submission) | 5 | 43.08 ± 0.42 | 37.10 ± 0.31 | 48.63 ± 0.95 | 27.31 ± 0.57 | 39.03 ± 0.54 |
| CGD-N (γ = 1.00, from rebuttal) | 3 | 44.27 ± 0.20 (+1.19) | 38.55 ± 0.04 (+1.44) | ± 0.38 (+1.71) | ± 0.23 (+1.06) | 40.45 ± 0.20 (+1.38) |
| CGD-IW (γ = 0.50 → 0.20, from rebuttal) | 3 | ± 0.25 (+0.97) | ± 0.53 (+0.81) | 50.30 ± 0.50 (+1.77) | 28.87 ± 0.46 (+1.21) | ± 0.39 (+1.22) |
(b) Retrieval R@1 (%)
| Method | Seeds | COCO I2T | COCO T2I | Flickr30K I2T | Flickr30K T2I | CC3M-val I2T | CC3M-val T2I | Avg. |
|---|---|---|---|---|---|---|---|---|
| FD | 5 | 40.38 ± 0.45 | 27.57 ± 0.13 | 65.28 ± 1.16 | 51.39 ± 0.80 | 33.82 ± 0.44 | 33.43 ± 0.21 | 41.98 ± 0.24 |
| CLIP-KD | 5 | 41.98 ± 0.70 | 27.70 ± 0.37 | 67.32 ± 1.07 | 51.88 ± 0.73 | 34.08 ± 0.24 | 32.60 ± 0.09 | 42.59 ± 0.43 |
| CLIP-RD | 5 | 42.18 ± 0.36 | 27.89 ± 0.29 | 66.84 ± 1.01 | 51.73 ± 0.57 | 34.22 ± 0.16 | 32.97 ± 0.30 | 42.64 ± 0.29 |
| FD + KL (base) | 5 | 41.63 ± 0.41 | 27.97 ± 0.22 | 67.44 ± 0.35 | 51.98 ± 0.92 | 34.03 ± 0.35 | 33.67 ± 0.22 | 42.79 ± 0.27 |
| CGD-N (γ = 0.0382, from submission) | 5 | ± 0.24 | 28.15 ± 0.18 | 67.34 ± 0.39 | 52.44 ± 0.29 | 34.36 ± 0.27 | ± 0.26 | 43.11 ± 0.13 |
| CGD-IW (λ = 20.92 → γ = 0.10, from submission) | 5 | 42.32 ± 0.54 | 28.13 ± 0.42 | 67.58 ± 1.63 | 52.39 ± 0.61 | 34.86 ± 0.40 | 33.75 ± 0.27 | 43.17 ± 0.48 |
| CGD-N (γ = 1.00, from rebuttal) | 3 | 43.73 ± 0.18 (+1.55) | ± 0.21 (+0.57) | 69.93 ± 1.53 (+2.61) | 53.69 ± 0.10 (+1.81) | 33.63 ± 0.20 (−0.59) | 31.55 ± 0.64 (−1.88) | 43.50 ± 0.23 (+0.86) |
| CGD-IW (γ = 0.50 → 0.20, from rebuttal) | 3 | ± 0.22 (+0.43) | 28.55 ± 0.46 (+0.66) | ± 0.95 (+0.71) | ± 1.27 (+0.96) | ± 0.46 (+0.32) | 33.60 ± 0.42 (+0.17) | ± 0.45 (+0.72) |
We also briefly present how we arrived at these two settings, and Appendices G.1 and G.2 of the revision give the full comparisons. For CGD-N, the table below compares the values of γ that we tried on seed 0.
| γ of CGD-N (seed 0) | Avg. retrieval R@1 (%) | Avg. classification Top-1 (%) |
|---|---|---|
| 0.0382 (from submission) | 43.03 | 39.16 |
| 0.20 | 43.62 | 40.44 |
| 0.30 | 43.38 | 40.39 |
| 1.00 (from rebuttal) | 43.29 | 40.60 |
For CGD-IW, the table below compares the setting of the submission with the settings of γ that we tried on seed 0. Its first row uses the Top-64 candidates and the other rows use MIX32 (both defined in the previous comment).
| Setting of CGD-IW (seed 0) | Avg. retrieval R@1 (%) | Avg. classification Top-1 (%) |
|---|---|---|
| λ = 20.92 until epoch 24, then γ = 0.10 (from submission) | 43.198 | 39.687 |
| γ = 0.20 for all 32 epochs | 43.600 | 39.932 |
| γ = 0.50 for all 32 epochs | 42.870 | 39.972 |
| γ = 0.50 until epoch 24, then γ = 0.20 (from rebuttal) | 43.801 | 40.479 |
Methodological novelty and its validation [1/2]
Thank you for raising the relation to existing distillation methods. We agree that CLIP-KD, CLIP-RD, and TinyCLIP already match row and column distributions, and that Section 3.2 introduces CGD-N through the decomposition of DKD. Our analysis in Section 3 shows, however, that column-KL matching limits the supervision of unmatched image candidates () in two ways, which CGD-N and CGD-IW address. The table below gives each method's column loss and its gradient on an unmatched image.
| Method | Column loss on caption | Gradient on unmatched image |
|---|---|---|
| CLIP-KD, CLIP-RD, TinyCLIP | ||
| DKD | ||
| CGD-N | (Eq. (17)) | |
| CGD-IW | (Eq. (15)) | on selected pairs |
Here and are the matched and unmatched parts of the column KL , and are the teacher-to-student probability ratios, is normalized over the selected pairs , is the slope of the critic , and positive constants are omitted in the gradients (Section 3.2).
CGD-N weights the unmatched images independently of the teacher's confidence. In CLIP-KD, CLIP-RD, and TinyCLIP, the unmatched KL is weighted by , which is close to zero whenever the teacher is confident in the matched pair, suppressing supervision on unmatched images. DKD uses constant weights (), but in dual encoders, Proposition 2 (Eq. (18)) proves that once the row loss matches each image's caption distribution, for every caption gives (, ). Because then adds no further constraint, CGD-N drops it, which removes the term of the matched image from DKD's gradient and supervises unmatched images independently of .
CGD-IW addresses the vanishing gradients that KL-based losses assign to unmatched candidates with a small student probability. In the first three rows, and under any differentiable -divergence between column distributions (Appendix A), the gradient on an unmatched image scales linearly with the student probability . Thus, candidates assigned a small probability by the student receive almost no gradient even when their probability ratio is far from . Derived from the term of the MI gap (Eq. (11)), replaces with , and we choose : when drops from to at the same ratio, KL-based gradients shrink by a factor of 100, whereas CGD-IW's gradient shrinks by a factor of only 1.26.
The following comment compares these methods in accuracy, retrieval margin, and agreement with the teacher's candidate ordering.
Marginal improvements, significance, and student architectures [1/2]
Thank you for pointing out that the gains are small and that classification is on par with or below CLIP-RD, and for asking for significance tests.
We ran statistical tests, and we share the results of t-tests over independent training seeds on the performance gaps to CLIP-RD. In the tables of our responses, "from submission" marks the settings of the submitted paper, and "from rebuttal" marks the two settings that we selected during the rebuttal and describe below. The table below compares each of them with CLIP-RD, with 5 seeds for CLIP-RD and for the settings from the submission and 3 seeds for the settings from the rebuttal.
As you pointed out, the improvement in classification was not significant for either variant in the submission. We therefore ran additional experiments on the strength of each added loss.
The first was to decide the coefficient of the LN term, that is, the ratio γ between the feature-gradient norm of the LN term and that of the Lbase term. The value in the submission was 0.0382, which had been matched to the value measured for the LIW term (Appendix D.1). To check whether this level was appropriate, we raised it to 0.20, 0.30 and 1.00 and trained again. We judged that the simplest value, 1, which gives the two terms equal gradient norms, is appropriate for the γ of CGD-N, and we selected it.
For CGD-IW, the submission used a fixed coefficient (λ = 20.92) and, in the last quarter of training, introduced a controller that adjusts the same ratio γ for the LIW term to 0.10 (Appendix D.1). We re-examined the training dynamics of CGD-IW and found that, when a large γ is kept until the end, the LIW term tends to conflict with the other loss terms late in training. To suppress this, we used a controller in which γ stays at 0.50 for the first three quarters of training and is lowered to 0.20 afterwards, and we confirmed that this controller shows comparatively stable training dynamics. We also changed the candidates that LIW corrects. The submission uses, for each image, the 64 unmatched captions that are most similar under the teacher (Top-64). The rebuttal setting adds the candidates that are most similar under the student, namely 32 unmatched captions for each image and 32 unmatched images for each caption (MIX32). With this controller the training dynamics improve stably, and both classification and retrieval improve over keeping γ = 0.50 until the end.
The table below extends Table 4 of the paper with the two rows marked "from rebuttal" (seed 0, 32 epochs, %). CGD-N (γ = 1.00) has the highest classification accuracy for all four student architectures, and its retrieval exceeds FD and CLIP-KD on all four and CLIP-RD on EfficientNet-B0 and Swin-Tiny. CGD-IW (γ = 0.50 → 0.20) exceeds FD, CLIP-KD and CLIP-RD in classification on all four, and has the highest retrieval R@1 on ResNet-18, EfficientNet-B0 and Swin-Tiny.
The per-dataset results for ViT-T/16 are in the next comment, which extends Table 1 of the paper.