acceptodds
Under review as a conference paper at ICLR 2027

Balancing Cross-Modal Semantic Alignment and Instance-Level Visual Discrimination by Controlling Constraint Density in Vision-Language Contrastive Distillation

Abstract

Vision-language models trained on large-scale image–text pairs achieve strong cross-modal semantic alignment and have become the dominant approach for cross-modal retrieval. Despite substantial progress in CLIP and its recent variants, they remain limited in the instance-level visual discrimination required for image-to-image retrieval. We first show that distilling visual discrimination knowledge from instance-oriented teachers introduces a trade-off between cross-modal semantic alignment and instance-level visual discrimination. We then identify the density of transferred neighborhood relations (constraint density) as a key factor governing the strength of this trade-off, providing a unified perspective on representative feature-, similarity-, and distribution-based distillation objectives. Based on this principle, we propose Teacher-guided Nearest-neighbor Contrastive Distillation (TNCD) as a simple realization of the lowest-constraint-density regime. Extensive experiments demonstrate that lower constraint density consistently leads to a more favorable trade-off, with TNCD serving as a practical realization that naturally supports heterogeneous teachers.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision-language models, knowledge distillation, contrastive learning, image retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_QjegOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the trade-off between image–text alignment and instance-level visual discrimination when distilling an instance-oriented teacher (MOFI) into a vision–language model (CLIP). The authors introduce “constraint density”: the number of neighborhood relations actually transferred to the student during distillation. For distribution-based objectives, this is measured by the perplexity of the teacher’s distribution. Building on this view, they propose TNCD, which uses the teacher’s nearest neighbor within each batch as a hard pseudo-positive for contrastive learning over student image–image similarities. It represents the lowest-density setting and supports heterogeneous teachers directly. In fine-tuning experiments, low-density objectives achieve a clearly better COCO–GPR1200 trade-off than high-density ones.

Critical strengths
  1. The problem is practically relevant and well motivated. Strong image–text alignment does not necessarily imply good instance retrieval, a gap that matters in applications such as retrieval, product recognition, and landmark recognition. The retrieval examples in Figure 1 and the similarity correlation analysis in Figure 2 show how different training objectives produce different neighborhood structures.

  2. The paper unifies three common distillation objectives under the lens of neighborhood relation transfer. Using perplexity as a computable density measure provides an interpretable basis for choosing distillation objectives.

Critical weaknesses
  1. The method has limited novelty. TNCD is essentially the one-hot limit of CRD as the teacher temperature τ̂ → 0. The paper’s own experiments show that their curves nearly coincide at τ̂ = 0.01.

  2. The main finding is that hard-label distillation outperforms soft-label distillation in this setting. This runs counter to the conventional view that soft labels convey “dark knowledge,” yet the paper does not discuss prior work on hard- versus soft-label distillation. Its contribution relative to existing findings is therefore unclear.

  3. The core analysis uses only one teacher–student pair (MOFI → OpenCLIP ViT-B/16) and one instance retrieval benchmark (GPR1200). It remains untested whether the principle holds for other instance-oriented or visual teachers, such as DINOv2, or other student architectures.

  4. The OpenCLIP ViT-B/16 and ViT-L/14 baselines in Table 1 use no teachers, whereas TNCD uses two strong teachers, SigLIP2-Base and MOFI-B/16, both trained on large datasets. Much of the “B/16 approaches L/14” result may come from the teachers’ knowledge, rather than the TNCD objective itself.

  5. Pseudo-label quality and batch dependence are insufficiently analyzed. TNCD selects pseudo-positives only within the current batch, and Table 6 shows strong sensitivity to batch size, with the best result requiring a global batch of 131k.

  6. Computational overhead is not reported. Training from scratch requires two teacher forward passes for each of 13B samples, all-gathering teacher features across a global batch of 131k, and computing a B × B similarity matrix for nearest-neighbor search. This adds substantial cost.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the detailed feedback and for recognizing the practical relevance of our problem setting and the value of our unified perspective on distillation objectives.

Response to Weakness 1 (Novelty)

We agree that TNCD corresponds to the zero-temperature limit of CRD when the teacher's nearest neighbor is unique. However, TNCD is intended as a simple realization of the lowest-constraint-density regime, rather than the primary contribution of our work.

Our main contribution lies in introducing constraint density as a common design axis for characterizing representative distillation objectives and understanding their trade-off behaviors. Specifically, feature matching implicitly constrains all pairwise neighborhood relations, while similarity matching explicitly constrains them, both corresponding to a constraint density of B−1B-1 per anchor, where BB is the batch size. In contrast, distribution matching allows constraint density to vary continuously, which we quantify using the perplexity of the teacher distribution. TNCD corresponds to the lowest-density regime, with a constraint density of 11.

Importantly, our experiments show that constraint density captures a systematic relationship with trade-off strength across different distillation formulations, despite differences in how neighborhood relations are constrained. Although constraint density does not fully determine trade-off behavior, it provides a useful design axis for balancing cross-modal semantic alignment and instance-level visual discrimination. We will clarify this unified characterization in Section 2.3 of the revised manuscript.

Response to Weakness 2 (Constraint Density vs. Hard/Soft-Label Distillation)

Our work does not claim that hard-label distillation is generally superior to soft-label distillation. Rather, our central finding is that lower-constraint-density objectives exhibit more favorable trade-offs between cross-modal semantic alignment and instance-level visual discrimination.

Importantly, as shown in Figures 3 and 4, dense objectives, including feature matching, similarity matching, and CRD with a higher teacher temperature, achieve higher peak performance on instance-level image retrieval. This suggests that transferring richer neighborhood information can be beneficial for instance-level discrimination, consistent with the conventional motivation for transferring dark knowledge. However, such dense supervision also leads to a stronger trade-off with cross-modal semantic alignment.

Thus, our contribution is not to establish the superiority of hard labels over soft labels, but to characterize how the density of transferred neighborhood relations is associated with the balance between competing capabilities.

Response to Weakness 3 (Generalizability Across Teachers and Student Architectures)

Our analysis is motivated by the potential conflict between task-dependent neighborhood relations, rather than properties specific to MOFI or OpenCLIP ViT-B/16. Constraint density characterizes the effective number of neighborhood relations transferred during distillation and is therefore not inherently tied to a particular teacher or student architecture.

Regarding evaluation benchmarks, GPR1200 is a general-purpose image retrieval benchmark comprising six existing retrieval datasets spanning diverse visual domains, rather than a single-domain dataset. Furthermore, Appendix Table 8 demonstrates the applicability of TNCD to a different student backbone (EVA02-Base) and training dataset composition.

We acknowledge that the relationship between constraint density and trade-off strength has not been systematically evaluated with other instance-oriented teachers or student architectures. Nevertheless, our controlled experiments provide empirical evidence supporting the proposed perspective, while the additional experiments demonstrate the applicability of TNCD beyond the primary experimental setting.

AuthorsOpenReview ↗

Official Comment by Authors

Response to Weakness 4 (Purpose of the OpenCLIP Comparisons)

The purpose of Section 3.4 is not to establish the superiority of TNCD over standard CLIP training, but to examine whether cross-modal semantic alignment and instance-level visual discrimination can be jointly achieved through large-scale scratch training with heterogeneous teachers. We include OpenCLIP ViT-B/16 and ViT-L/14 as reference points to contextualize the performance achieved on both retrieval tasks, rather than as baselines for isolating the contribution of the TNCD objective.

We apologize for the lack of clarity and will revise Section 3.4 to avoid implying methodological superiority over the OpenCLIP baselines.

Response to Weakness 5 (Pseudo-label Quality and Batch Dependence)

We acknowledge that robustness to incorrect nearest-neighbor selections has not been explicitly evaluated, and performance degradation with extremely small candidate pools remains possible. However, TNCD already achieves favorable trade-offs with a global batch size of 16,384 in Section 3.2, demonstrating that a batch size of 131,072 is not required for its effectiveness.

As shown in Appendix Table 6, increasing the batch size consistently improves performance, suggesting that the quality of the selected nearest neighbors is an important factor. Furthermore, a global batch size of 131,072 is within the range of large-scale contrastive learning settings.

Response to Weakness 6 (Computational Overhead of Distillation)

As clarified in our response to Weakness 4, the purpose of Section 3.4 is not to establish computational or methodological superiority over standard CLIP training.

We acknowledge that large-scale distillation with pretrained teachers incurs additional computational costs. However, teacher forward passes and pairwise similarity computations are not specific to TNCD, but are also required by other distillation objectives, such as CRD and similarity matching.

Thank you for the authors’ response. I remain skeptical about the paper’s contributions. Neither the hard-label distillation nor the distillation of nearest-neighbor information represents a substantial contribution to the field of knowledge distillation. Therefore, I have decided to maintain my score.

Reviewer_18jTOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates the trade-off between cross-modal semantic alignment and instance-level visual discrimination when distilling knowledge from an instance-oriented teacher into a vision-language model. The authors introduce constraint density, defined as the effective number of transferred neighborhood relations, as a perspective for understanding feature-, similarity-, and distribution-based distillation objectives. Based on this perspective, they propose Teacher-guided Nearest-neighbor Contrastive Distillation (TNCD), which uses the teacher's nearest neighbor as a positive target in a contrastive learning objective. The paper evaluates the relationship between constraint density and retrieval performance through controlled fine-tuning experiments with different distillation objectives and teacher temperatures. It also reports large-scale training experiments with heterogeneous teachers, evaluating zero-shot classification, cross-modal retrieval, and image-to-image retrieval.

Critical strengths

None.

Critical weaknesses

Limited methodological novelty. The proposed TNCD objective is essentially a standard contrastive learning loss that treats the teacher's nearest neighbor as a positive target. More importantly, TNCD is mathematically equivalent to the zero-teacher-temperature limit of the CRD objective already discussed in the paper, assuming a unique nearest neighbor. The authors' own experiments further show that TNCD and CRD with a sharp teacher distribution exhibit nearly identical trade-off curves. Consequently, it is unclear what substantive methodological contribution TNCD offers beyond a hard-target formulation of an existing distillation objective. This significantly weakens the novelty of the proposed method.

Questions for authors

Motivation for a shared representation. The paper does not sufficiently explain why cross-modal semantic alignment and instance-level visual discrimination must be jointly optimized within the same embedding space. These tasks naturally require different forms of supervision and potentially different similarity criteria. Why is a single shared similarity space necessary or preferable to alternatives such as task-specific projection heads or similarity functions? Clarifying the practical motivation and the intended constraints of the problem setting would help establish the significance of the proposed trade-off.

3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for taking the time to evaluate our work and for providing feedback on the novelty of TNCD and the motivation for a shared representation.

Response to Weakness 1 (Novelty and the Role of Constraint Density)

We agree that TNCD corresponds to the zero-temperature limit of CRD when the teacher's nearest neighbor is unique. TNCD is not intended to introduce a fundamentally new contrastive loss, but rather to serve as a simple realization of the lowest-constraint-density regime.

The primary contribution of our work lies in introducing constraint density as a unified design axis for understanding the trade-off between cross-modal semantic alignment and instance-level visual discrimination. Specifically, feature matching implicitly constrains all pairwise neighborhood relations, while similarity matching explicitly constrains them, both corresponding to a constraint density of B−1B-1 per anchor, where BB is the batch size. In contrast, distribution matching allows constraint density to vary continuously, which we quantify using the perplexity of the teacher distribution.

Despite differences in how these objectives constrain neighborhood relations, our experiments reveal a systematic relationship between constraint density and trade-off strength across different distillation formulations (Section 3.2) and within CRD (Section 3.3). Although constraint density does not fully determine trade-off behavior, it provides a common design axis that captures systematic trends across objectives with different geometric constraints.

Response to Question 1 (Motivation for a Shared Representation)

We agree that task-specific projection heads or similarity functions are valid alternatives for accommodating different similarity notions. Our work does not claim that a single shared embedding space is strictly necessary or universally preferable.

However, a shared embedding space offers practical advantages for retrieval systems, as it allows cross-modal and image-to-image retrieval to use the same image embeddings and retrieval index. In contrast, task-specific representations or similarity functions may require separate indices or additional task-specific retrieval procedures, increasing system complexity.

Therefore, although task-specific representations may alleviate the trade-off, achieving both cross-modal semantic alignment and instance-level visual discrimination within a single shared embedding space remains a practically meaningful objective, as it enables a simpler and more unified retrieval system.

Thank you for the clarification. However, my main concern regarding the conceptual novelty and rigor of constraint density remains unresolved. The controlled CRD experiments primarily demonstrate the effect of teacher-distribution sharpness on the empirical trade-off. I remain unconvinced that these results establish constraint density as a sufficiently distinct and general principle beyond existing knowledge about distillation temperature and target distributions. Therefore, my overall assessment remains unchanged.

AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the follow-up. Since the conceptual novelty and rigor of constraint density were not explicitly raised as critical weaknesses in the initial review, we would like to address this concern.

Constraint density is not simply a reformulation of teacher-distribution sharpness. It provides a common design axis across different distillation objectives: feature and similarity matching correspond to a dense regime with B−1B-1 constrained neighborhood relations per anchor, while distribution matching allows the effective density to vary continuously, as quantified by teacher-distribution perplexity. TNCD corresponds to the lowest-density regime with density 1.

Importantly, our empirical evidence is not limited to teacher-temperature sweeps within CRD. Section 3.2 compares feature matching, similarity matching, distribution matching, and TNCD, demonstrating systematic trade-off trends across different objective formulations. Section 3.3 further examines the relationship between constraint density and trade-off strength within a fixed CRD formulation. Although constraint density does not fully determine trade-off behavior, these results support its value as a common design axis across objectives that impose different geometric constraints.

Reviewer_JKefOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the trade-off between cross-modal semantic alignment and instance-level visual discrimination in vision-language models like CLIP and SigLIP2. Distilling from instance-oriented teachers improves discrimination but can hurt cross-modal retrieval. This paper introduces constraint density, the effective number of neighborhood relations transferred from teacher to student, and argues that transferring fewer relations reduces conflict between the two similarity structures. This view unifies feature-, similarity-, and distribution-based distillation. It proposes TNCD, which transfers only the teacher’s nearest neighbor. Experiments show lower constraint density gives better trade-offs, and large-scale training with heterogeneous teachers improves both cross-modal and image-to-image retrieval.

Critical strengths
  1. Improving instance discrimination without losing cross-modal ability is a practical challenge, well illustrated by qualitative examples.
  2. Viewing distillation as neighborhood-relation transfer is intuitive and unifying.
  3. Objective comparisons, CRD temperature sweeps, and pool-size and perplexity analyses support the hypothesis.
  4. TNCD is a straightforward contrastive loss that works with heterogeneous teachers.
Critical weaknesses
  1. TNCD uses only image-to-image similarities, and scratch training sets λCLIP=0\lambda_{CLIP}=0, so nothing appears to train the text encoder. How zero-shot and cross-modal performance arise is unexplained.
  2. The link between perplexity and actual optimization constraints is not established; a one-positive contrastive loss still involves all samples, and feature and similarity matching differ geometrically despite both being “dense.”
  3. Varying temperature also changes target sharpness, gradients, and neighbor reliability, so the results show correlation, not causation.
  4. TNCD is essentially zero-temperature CRD, and the paper reports nearly identical trade-offs for the two.
  5. Comparisons against public OpenCLIP models ignore the strength and cost of the pretrained teachers, and other dual-teacher objectives are not tested under identical settings.
Questions for authors
  1. How is the text encoder trained from scratch? Is there an extra objective or pretrained initialization? Please give the full loss.
  2. Can constraint density be defined more formally, e.g., via gradients, and why is perplexity the right measure?
  3. How is constraint density separated from effects like gradient magnitude or loss scaling?
  4. What does TNCD contribute beyond low-temperature CRD, and how do they compare in large-scale dual-teacher training?
  5. Can TNCD be compared with CRD and similarity matching using identical teachers, data, architecture, and budget?
  6. How sensitive is TNCD to incorrect neighbors and smaller pools, and what is needed to support very large batches?
3 comments
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the thoughtful feedback and for recognizing the practical relevance of our problem setting and the experimental evidence supporting our perspective.

Response to Weakness 1 / Question 1 (Cross-Modal TNCD Formulation)

We apologize for the lack of clarity. We apply TNCD to both cross-modal and image-to-image supervision. For cross-modal supervision, we construct nearest-neighbor targets using SigLIP2-Base and compute the TNCD loss in both image-to-text and text-to-image directions, thereby training both the image and text encoders. For image-to-image supervision, we construct nearest-neighbor targets using MOFI-B/16.

The full training objective is provided in Eq. (5) of Appendix A. For scratch training, we set λCLIP=0\lambda_{\mathrm{CLIP}}=0, λcross=1.0\lambda_{\mathrm{cross}}=1.0, and λinst=0.1\lambda_{\mathrm{inst}}=0.1. Both encoders are trained from random initialization without the original CLIP loss.

We will clarify the cross-modal TNCD formulation in the main text.

Response to Weakness 2 / Question 2 (Constraint Density and Optimization Constraints)

We thank the reviewer for raising this important point. Constraint density characterizes the effective number of teacher-imposed neighborhood relations that the student is required to reproduce, rather than the number of samples involved in optimization.

For CRD, the gradient of the cross-entropy loss with respect to the student similarity sijs_{ij} is given by ∂Li/∂sij=(pij−pij^)/τ\partial L_{i} / \partial s_{ij} = (p_{ij} - \hat{p_{ij}}) / \tau. For each anchor ii, neighbors jj assigned non-negligible teacher probabilities pij^\hat{p_{ij}} are constrained to reproduce the relative similarities and rankings specified by the teacher distribution. In contrast, for neighbors with negligible pij^\hat{p_{ij}}, the gradient becomes approximately pij/τp_{ij} / \tau, which can be made small by assigning sufficiently low student probabilities, without requiring their individual similarities or rankings to match those of the teacher. Thus, although all samples contribute to the contrastive denominator, the effective number of teacher-imposed neighborhood constraints depends on the concentration of the teacher distribution. We use perplexity to quantify this effective number, as it measures the effective support size of the teacher distribution.

We agree that feature matching and similarity matching impose different geometric constraints. However, both constrain all pairwise neighborhood relations and correspond to a dense regime. Despite their different formulations, both exhibit stronger trade-offs than low-density objectives in Section 3.2. We do not claim that constraint density fully determines trade-off behavior; rather, it captures a systematic trend across representative distillation objectives, providing a useful design axis for balancing competing capabilities.

Response to Weakness 3 / Question 3 (Causal Interpretation)

We agree that our experiments do not isolate the independent causal effect of constraint density. Nevertheless, our goal is to identify a common design axis that captures systematic trends in trade-off behavior across different distillation objectives, rather than to establish constraint density as an independent causal factor.

We will revise the wording in Section 3.3 and elsewhere to clarify the scope of our empirical findings and avoid implying complete causal isolation.

AuthorsOpenReview ↗

Official Comment by Authors

Response to Weakness 4 / Question 4 and 5 (TNCD vs. CRD)

We agree that TNCD corresponds to the zero-temperature limit of CRD when the teacher's nearest neighbor is unique. TNCD is not intended to outperform CRD with a low teacher temperature, but rather to provide a simple and practical realization of the lowest-constraint-density regime.

A practical advantage of TNCD is that it requires only the teacher's nearest-neighbor indices, without constructing teacher similarity distributions or tuning teacher temperatures. This allows TNCD to be applied to teachers trained with different objectives, such as softmax-based CLIP and sigmoid-based SigLIP, without objective-specific treatment. In Section 3.4, we demonstrate that TNCD can jointly achieve strong cross-modal semantic alignment and instance-level visual discrimination through large-scale scratch training with heterogeneous teachers. While we do not directly compare distillation objectives in large-scale dual-teacher training, comparisons among representative objectives are conducted under controlled finetuning settings in Section 3.2, as our large-scale experiments focus on demonstrating the joint acquisition of both capabilities rather than methodological superiority.

Response to Weakness 5 (Comparison with OpenCLIP Baselines)

The purpose of Section 3.4 is to investigate whether cross-modal semantic alignment and instance-level visual discrimination can be jointly achieved through large-scale scratch training with heterogeneous teachers, rather than to establish the superiority of TNCD over standard CLIP training. We include OpenCLIP ViT-B/16 and ViT-L/14 as reference points to contextualize the performance achieved on both cross-modal and image-to-image retrieval tasks, rather than as teacher-matched baselines for evaluating methodological superiority.

We apologize for the lack of clarity in our presentation. We will revise Table 1 and Section 3.4 to clarify the purpose and limitations of these comparisons.

Response to Question 6 (Nearest-neighbor Quality and Batch Size)

We acknowledge that robustness to incorrect nearest-neighbor selections has not been explicitly evaluated, and performance degradation with extremely small candidate pools remains possible. However, TNCD already achieves favorable trade-offs with a global batch size of 16,384 in Section 3.2. At the same time, as shown in Appendix Table 6, increasing the batch size consistently improves performance, suggesting that the quality of the selected nearest neighbors is an important factor.

We used the same distributed implementation across all batch sizes, without requiring any batch-size-specific modifications.

Thanks for the detailed rebuttal. My main concern on the interpretation of the large-scale dual-teacher results remains, so I will keep my rating.

Reviewer_6PznOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Distilling CLIP-family models with task-specific visual objectives enables the models to acquire domain-specific task capabilities while preserving their general cross-modal abilities. This paradigm has become a popular research direction in computer vision. This paper identifies a critical issue within such pipelines: distilling visual discrimination knowledge from instance-oriented teachers creates a trade-off between cross-modal semantic alignment and instance-level visual discrimination. The authors further conduct an in-depth analysis of the root cause, examining the geometric relationships of feature spaces across different models to explain the origin of this tension. From this perspective, they analyze three existing distillation paradigms: feature matching, similarity matching, and distribution matching. Building on distribution matching, they introduce the concept of constraint density, defined as the number of teacher relations being transferred. Through experiments, they demonstrate that lower constraint density yields a more favorable trade-off between the two objectives. Based on this finding, the authors proposed Teacher-guided Nearest-neighbor Contrastive Distillation (TNCD), a simple realization of the lowest-constraint-density regime . The hypotheses are extensively validated via experiments across multiple datasets.

Critical strengths

1.The problem studied by the authors is practically valuable, and attributing the trade-off to transferred neighbour information is a reasonable angle for investigation. 2.The experiments are comprehensive, featuring thorough ablation studies, parameter analysis and validation, and more. Besides, the illustrative figs (Fig. 1, Fig. 2) is highly intuitive, making it easy to grasp the motivation behind the research problem. 3.The isolation experiment(Sec. 3.3) is well designed. Fixing CRD and changing only the teacher temperature, with median perplexity reported, shows that two nearly equal densities (0.001 and 0.01) give nearly equal trade-off curves.

Critical weaknesses

1.Novelty. Analyzing the trade-off of models across different tasks and examining the distribution discrepancy between the teacher’s and student’s feature spaces is not a novel contribution. Relevant studies can be traced back at least to the era of statistical learning. The transferred neighbour relation (or neighbourhood information) is not a highly original finding; it is even a fundamental idea in manifold learning. This criticism also applies to Section 2.2 of the paper. This section delivers limited new insights, which makes the claim of a unified perspective an overclaim. 2.In experiments, the proposed TNCD does not outperform CRD (\tau = 0.01).

Questions for authors

For n_i in Equation 4, which denotes the teacher’s nearest neighbor of sample i: is this neighbor located within the current batch? If not, can the sum of similarities be guaranteed to equal one?

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We thank the reviewer for the constructive feedback and for recognizing the practical significance of our problem setting and the value of our experimental analyses.

Response to Weakness 1 (Novelty)

We agree that task-dependent trade-offs and neighborhood relation transfer have been studied in prior work, and we do not claim these concepts themselves as novel. However, our main contribution lies in introducing constraint density as a common axis for characterizing distillation objectives and understanding their trade-off behaviors.

Specifically, feature matching implicitly constrains all pairwise neighborhood relations, while similarity matching explicitly constrains them, both corresponding to a constraint density of B−1B-1 per anchor, where BB is the batch size. In contrast, distribution matching allows constraint density to vary continuously, which we quantify using the perplexity of the teacher distribution. TNCD corresponds to the lowest-density regime, with a constraint density of 11. Thus, constraint density provides a unified characterization of feature-, similarity-, and distribution-based distillation objectives, rather than being specific to distribution matching.

Importantly, this perspective provides insight into the trade-off between cross-modal semantic alignment and instance-level visual discrimination. Section 3.2 shows that high-density objectives exhibit stronger trade-offs despite differences in their formulations, whereas low-density objectives achieve more favorable trade-offs. Section 3.3 further demonstrates a systematic relationship between constraint density and trade-off strength within CRD. Although objective-specific factors also influence the trade-off, these results support constraint density as a common design axis for understanding and balancing competing capabilities in vision-language distillation.

We will revise Section 2.3 to explicitly characterize the constraint density of each representative distillation objective, including TNCD, and clarify its role as a common design axis across different formulations.

Response to Weakness 2 (TNCD vs. CRD)

We agree that TNCD does not outperform CRD with τ^=0.01\hat{\tau}=0.01. However, TNCD is not intended to outperform CRD with a low teacher temperature, but rather to serve as a simple and practical realization of the lowest-constraint-density regime derived from our proposed perspective.

In fact, TNCD corresponds to the zero-temperature limit of CRD when the teacher's nearest neighbor is unique. In practice, TNCD transfers only the teacher's nearest-neighbor relation without requiring the construction of teacher similarity distributions or tuning teacher temperatures. This allows TNCD to use teachers trained with different objectives, such as softmax-based CLIP and sigmoid-based SigLIP, without objective-specific treatment. We demonstrate this practical applicability through large-scale scratch training with heterogeneous teachers (SigLIP2 and MOFI) in Section 3.4.

Thus, the contribution of TNCD lies not in outperforming CRD, but in providing a simple realization of the proposed design principle with practical applicability to heterogeneous teacher distillation.

Response to Question 1 (Nearest-Neighbor Selection and Softmax Normalization)

Yes, nin_i is selected from all non-self samples within the current global batch in the teacher embedding space. The denominator in Eq. (4) is computed over the same candidate set, ensuring proper softmax normalization. We will clarify this in Section 2.4.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official Comment by Authors

We thank all reviewers for their constructive feedback and valuable suggestions. We have carefully revised the manuscript to address the concerns raised in the reviews. The main revisions are summarized below.

  1. Clarified the definition and interpretation of constraint density (Sections 2.3 and 2.4). We explicitly characterize constraint density across feature-, similarity-, and distribution-based distillation objectives, clarify its relationship to teacher-distribution perplexity, and acknowledge the equivalence between TNCD and the zero-teacher-temperature limit of CRD.
  2. Refined the strength and scope of our claims (Sections 3.2 and 3.3, and throughout the manuscript). We clarified that the experiments reveal a systematic relationship between constraint density and trade-off strength without isolating its independent causal effect, and revised the wording throughout the manuscript to ensure that our claims accurately reflect the scope of the empirical evidence.
  3. Clarified the heterogeneous-teacher training setup and the purpose of the OpenCLIP comparisons (Sections 3.1 and 3.4). We specified the training objective and loss weights for scratch training and clarified that the OpenCLIP models serve as reference points for evaluating the joint acquisition of cross-modal semantic alignment and instance-level visual discrimination, rather than as teacher-matched baselines.

The changes in the revised manuscript are highlighted in blue for ease of reference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.