- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies the trade-off between image–text alignment and instance-level visual discrimination when distilling an instance-oriented teacher (MOFI) into a vision–language model (CLIP). The authors introduce “constraint density”: the number of neighborhood relations actually transferred to the student during distillation. For distribution-based objectives, this is measured by the perplexity of the teacher’s distribution. Building on this view, they propose TNCD, which uses the teacher’s nearest neighbor within each batch as a hard pseudo-positive for contrastive learning over student image–image similarities. It represents the lowest-density setting and supports heterogeneous teachers directly. In fine-tuning experiments, low-density objectives achieve a clearly better COCO–GPR1200 trade-off than high-density ones.
Critical strengths
-
The problem is practically relevant and well motivated. Strong image–text alignment does not necessarily imply good instance retrieval, a gap that matters in applications such as retrieval, product recognition, and landmark recognition. The retrieval examples in Figure 1 and the similarity correlation analysis in Figure 2 show how different training objectives produce different neighborhood structures.
-
The paper unifies three common distillation objectives under the lens of neighborhood relation transfer. Using perplexity as a computable density measure provides an interpretable basis for choosing distillation objectives.
Critical weaknesses
-
The method has limited novelty. TNCD is essentially the one-hot limit of CRD as the teacher temperature τ̂ → 0. The paper’s own experiments show that their curves nearly coincide at τ̂ = 0.01.
-
The main finding is that hard-label distillation outperforms soft-label distillation in this setting. This runs counter to the conventional view that soft labels convey “dark knowledge,” yet the paper does not discuss prior work on hard- versus soft-label distillation. Its contribution relative to existing findings is therefore unclear.
-
The core analysis uses only one teacher–student pair (MOFI → OpenCLIP ViT-B/16) and one instance retrieval benchmark (GPR1200). It remains untested whether the principle holds for other instance-oriented or visual teachers, such as DINOv2, or other student architectures.
-
The OpenCLIP ViT-B/16 and ViT-L/14 baselines in Table 1 use no teachers, whereas TNCD uses two strong teachers, SigLIP2-Base and MOFI-B/16, both trained on large datasets. Much of the “B/16 approaches L/14” result may come from the teachers’ knowledge, rather than the TNCD objective itself.
-
Pseudo-label quality and batch dependence are insufficiently analyzed. TNCD selects pseudo-positives only within the current batch, and Table 6 shows strong sensitivity to batch size, with the best result requiring a global batch of 131k.
-
Computational overhead is not reported. Training from scratch requires two teacher forward passes for each of 13B samples, all-gathering teacher features across a global batch of 131k, and computing a B × B similarity matrix for nearest-neighbor search. This adds substantial cost.
Questions for authors
See Weaknesses.
3 comments
Official Comment by Authors
Response to Weakness 4 (Purpose of the OpenCLIP Comparisons)
The purpose of Section 3.4 is not to establish the superiority of TNCD over standard CLIP training, but to examine whether cross-modal semantic alignment and instance-level visual discrimination can be jointly achieved through large-scale scratch training with heterogeneous teachers. We include OpenCLIP ViT-B/16 and ViT-L/14 as reference points to contextualize the performance achieved on both retrieval tasks, rather than as baselines for isolating the contribution of the TNCD objective.
We apologize for the lack of clarity and will revise Section 3.4 to avoid implying methodological superiority over the OpenCLIP baselines.
Response to Weakness 5 (Pseudo-label Quality and Batch Dependence)
We acknowledge that robustness to incorrect nearest-neighbor selections has not been explicitly evaluated, and performance degradation with extremely small candidate pools remains possible. However, TNCD already achieves favorable trade-offs with a global batch size of 16,384 in Section 3.2, demonstrating that a batch size of 131,072 is not required for its effectiveness.
As shown in Appendix Table 6, increasing the batch size consistently improves performance, suggesting that the quality of the selected nearest neighbors is an important factor. Furthermore, a global batch size of 131,072 is within the range of large-scale contrastive learning settings.
Response to Weakness 6 (Computational Overhead of Distillation)
As clarified in our response to Weakness 4, the purpose of Section 3.4 is not to establish computational or methodological superiority over standard CLIP training.
We acknowledge that large-scale distillation with pretrained teachers incurs additional computational costs. However, teacher forward passes and pairwise similarity computations are not specific to TNCD, but are also required by other distillation objectives, such as CRD and similarity matching.
Thank you for the authors’ response. I remain skeptical about the paper’s contributions. Neither the hard-label distillation nor the distillation of nearest-neighbor information represents a substantial contribution to the field of knowledge distillation. Therefore, I have decided to maintain my score.
Official Comment by Authors
We thank the reviewer for the detailed feedback and for recognizing the practical relevance of our problem setting and the value of our unified perspective on distillation objectives.
Response to Weakness 1 (Novelty)
We agree that TNCD corresponds to the zero-temperature limit of CRD when the teacher's nearest neighbor is unique. However, TNCD is intended as a simple realization of the lowest-constraint-density regime, rather than the primary contribution of our work.
Our main contribution lies in introducing constraint density as a common design axis for characterizing representative distillation objectives and understanding their trade-off behaviors. Specifically, feature matching implicitly constrains all pairwise neighborhood relations, while similarity matching explicitly constrains them, both corresponding to a constraint density of B−1 per anchor, where B is the batch size. In contrast, distribution matching allows constraint density to vary continuously, which we quantify using the perplexity of the teacher distribution. TNCD corresponds to the lowest-density regime, with a constraint density of 1.
Importantly, our experiments show that constraint density captures a systematic relationship with trade-off strength across different distillation formulations, despite differences in how neighborhood relations are constrained. Although constraint density does not fully determine trade-off behavior, it provides a useful design axis for balancing cross-modal semantic alignment and instance-level visual discrimination. We will clarify this unified characterization in Section 2.3 of the revised manuscript.
Response to Weakness 2 (Constraint Density vs. Hard/Soft-Label Distillation)
Our work does not claim that hard-label distillation is generally superior to soft-label distillation. Rather, our central finding is that lower-constraint-density objectives exhibit more favorable trade-offs between cross-modal semantic alignment and instance-level visual discrimination.
Importantly, as shown in Figures 3 and 4, dense objectives, including feature matching, similarity matching, and CRD with a higher teacher temperature, achieve higher peak performance on instance-level image retrieval. This suggests that transferring richer neighborhood information can be beneficial for instance-level discrimination, consistent with the conventional motivation for transferring dark knowledge. However, such dense supervision also leads to a stronger trade-off with cross-modal semantic alignment.
Thus, our contribution is not to establish the superiority of hard labels over soft labels, but to characterize how the density of transferred neighborhood relations is associated with the balance between competing capabilities.
Response to Weakness 3 (Generalizability Across Teachers and Student Architectures)
Our analysis is motivated by the potential conflict between task-dependent neighborhood relations, rather than properties specific to MOFI or OpenCLIP ViT-B/16. Constraint density characterizes the effective number of neighborhood relations transferred during distillation and is therefore not inherently tied to a particular teacher or student architecture.
Regarding evaluation benchmarks, GPR1200 is a general-purpose image retrieval benchmark comprising six existing retrieval datasets spanning diverse visual domains, rather than a single-domain dataset. Furthermore, Appendix Table 8 demonstrates the applicability of TNCD to a different student backbone (EVA02-Base) and training dataset composition.
We acknowledge that the relationship between constraint density and trade-off strength has not been systematically evaluated with other instance-oriented teachers or student architectures. Nevertheless, our controlled experiments provide empirical evidence supporting the proposed perspective, while the additional experiments demonstrate the applicability of TNCD beyond the primary experimental setting.