acceptodds
Under review as a conference paper at ICLR 2027

TE-VLM: Transfer Entropy-Inspired Distillation for Vision-Language Models

Abstract

Transfer entropy (TE) is a principled measure of directed information flow, but it is intractable to estimate in high-dimensional multimodal representation spaces. We use optimization-step TE, the information a frozen teacher provides about a student's next-step representations beyond the student's current state, as a conceptual lens for vision–language model (VLM) distillation and derive from it two tractable proxy objectives, TE1 and TE2. Both reward the student for reproducing the teacher's local embedding geometry by matching within-batch finite-difference directions up to orientation through squared cosine similarity, per modality (TE1) or jointly across image and text (TE2), with no density estimation, no extra teacher passes, and negligible compute. In CLIP-style distillation with RN50, ViT-B/16, and RN5016 teachers and RN34/RN18 students, the proxies consistently improve image-to-text retrieval over contrastive, KL, MSE, interactive-contrastive, mutual-information, and relational Gram-matching baselines while remaining competitive on text-to-image retrieval. With the base objective fixed, adding TE lifts MSCOCO image-to-text R@1 from 6.1% to 10.3%, against 8.7% for the strongest competing regularizer. The gains persist across temperatures, batch sizes, datasets (MSCOCO, Flickr8k, Flickr30k), and five random seeds; transfer better under distribution shift (MSCOCOFlickr8k); and extend to classification, where the distilled student surpasses its teacher in label-free Food-101 evaluation and generalizes better to ImageNet-1k zero-shot classification. Representation-level analyses show that TE-distilled students preserve the teacher's neighborhood structure and joint image-text geometry more faithfully than mutual-information distillation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Transfer entropy, vision-language models, knowledge distillation, multimodal learning, representation alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_KpQTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TE-VLM, a transfer-entropy-inspired framework for distilling CLIP-style vision–language dual encoders using two geometric regularizers. TE1 aligns teacher–student pairwise embedding differences separately for images and text, while TE2 aligns their concatenation; both preserve secant directions up to sign through squared cosine similarity. Experiments across teacher–student configurations and retrieval datasets show consistent image-to-text improvements but mixed text-to-image results. Additional transfer experiments and ablations support the regularizers’ empirical usefulness, although their connection to transfer entropy is not directly established.

Critical strengths
  • The proposed finite-difference alignment is simple, inexpensive, and empirically useful. It operates on already-computed embeddings without additional teacher forward passes. Importantly, the controlled comparison in Table 2 isolates its contribution: adding TE2 increases image-to-text R@1 from 6.08% to 10.32%, compared with 8.65% for the MI regularizer. This provides evidence that matching teacher–student difference directions up to sign offers useful supervision beyond the existing distillation objectives.

  • The image-to-text improvements are supported by robustness and transfer experiments. The temperature and batch-size ablations (Table 3), five-seed results across multiple teacher–dataset configurations (Table 22), and cross-dataset retrieval evaluation (Table 4) strengthen confidence that the gains are not specific to a single training configuration. This broader empirical support makes the lightweight regularizer a potentially useful addition to CLIP-style distillation.

Critical weaknesses
  • The connection to transfer entropy could be explained more clearly. Eq. (5) describes how the teacher provides additional information about the student’s next-step representations, given its current state. In contrast, TE1/TE2 compare embedding differences between samples within a single training step. The authors acknowledge this distinction in Appendix H (Limitations, p. 43), noting that the experiments validate the geometric regularizer rather than the underlying TE quantity. The relationship between these two types of changes remains unclear, limiting the explanatory value of the TE motivation.

  • Comparisons with state-of-the-art CLIP distillation methods would strengthen the evaluation. Table 2 provides useful comparisons between loss terms within the authors’ training pipeline. However, comparisons against complete methods from recent CLIP distillation work, under matched architectures, training data, evaluation protocols, and comparable compute budgets, would better establish practical competitiveness beyond the loss-level comparisons.

  • The retrieval improvements vary by direction. In Table 2, TE2 improves I2T R@1 from 6.08% to 10.32%, exceeding MI’s 8.65%. However, its T2I R@1 is 7.33%, compared with 7.65% for MI. This does not diminish the I2T gains, but indicates that the advantage is task-direction dependent and would benefit from further explanation.

  • The contribution of the specific directional objective could be better isolated. Table 2 compares TE1/TE2 with Gram matching and finite-difference MSE, providing useful evidence for the proposed regularizer. However, these alternatives differ in several respects, including their treatment of direction and magnitude. A direct comparison between ordinary cosine and squared-cosine alignment on the same sample differences would help establish whether the proposed sign-invariant formulation provides a distinct benefit.

Questions for authors
  1. Comparison with existing distillation methods. Could the authors clarify how the implemented baselines relate to the complete methods in the cited CLIP distillation literature? Comparisons with strong existing methods under matched architectures, data, and training budgets would help assess practical competitiveness.

  2. Retrieval asymmetry. What explains the stronger improvements in I2T than T2I? For example, Table 2 reports TE2 versus MI R@1 of 10.32% versus 8.65% for I2T, but 7.33% versus 7.65% for T2I. Beyond the teacher’s existing asymmetry, is there evidence explaining why the regularizer favors one retrieval direction?

  3. Choice of directional objective. Have the authors compared squared cosine with ordinary cosine alignment using the same sample differences and comparable weight tuning? This would help determine whether allowing parallel and anti-parallel differences to receive the same reward provides a distinct benefit.

  4. Does directional alignment benefit downstream multimodal LLMs? My concern is that preserving teacher-specific directions may improve direct CLIP retrieval without necessarily improving downstream tasks when an LLM can adapt to the representations.

    • Rethinking Projector Training in Multimodal LLMs (PORTAL) shows that multiple initial projector mappings can support comparable downstream performance after joint adaptation. This suggests that a unique initial alignment may not be necessary, although it does not imply that arbitrary initializations work equally well.
    • This raises the hypothesis that teacher-specific directions may be a useful training reference rather than a necessary property of transferable representations. A trainable downstream LLM might learn to use alternative representations equally well, so improved directional agreement need not translate into better downstream performance.
    • Do the authors expect TE regularization to remain beneficial in this setting, and why? As a future extension, comparing otherwise matched multimodal LLMs using representations distilled with and without TE regularization on tasks such as VQA would help test this hypothesis. I view this as a question about broader applicability, rather than a required rebuttal experiment.
2 comments
AuthorsOpenReview ↗

Reply to Critical Weaknesses

We are grateful for the reviewer’s detailed comments and valuable suggestions. Below, we respond to the issues raised.

W1: We agree and have made this distinction explicit in Secs. 4.1–4.2. Eq. (5) concerns the teacher’s additional predictive information about student representations across optimization steps. TE1/TE2 instead align teacher and student differences across inputs at a single step. Their relationship is motivational, not a derivation.

For a fixed input, the frozen teacher has no optimization-step change. We therefore compare variation across inputs, where both teacher and student representations vary. This design choice does not make the proxies estimators of Eq. (5). Their functional form is adapted from Eq. (4), and their primary mathematical characterization is secant-line alignment (Eq. 16).

Appendix C provides a restricted linear–Gaussian correspondence with a correlation-based reference, not evidence of directional information flow. Our contribution is the regularizer and its empirical behavior; we do not claim that TE is measured or explains the gains.

W2: Appendix H.6 now evaluates the CLIP-KD objective (CL + CRD + FD + ICL), using its official loss weights with the same architectures, data, epochs, batch size, and evaluation protocol as our method. MSCOCO I2T / T2I R@1 (%) are:

  • CLIP-KD: 7.77 / 6.72
  • Ours, five seeds: 10.08 ± 0.17 / 7.12 ± 0.19
  • Base + RKD distance and angle: 6.94 / 6.42

Our base contains the same four loss components as CLIP-KD, with the proposed regularizer added. Its measured overhead is approximately one minute over a 10-epoch run (Table 21), giving comparable compute budgets.

This comparison evaluates CLIP-KD’s objective under our controlled training protocol, rather than reproducing its large-scale recipe. We therefore claim improvements within this setting, not state-of-the-art absolute performance or demonstrated competitiveness at web scale.

W3: We agree that the advantage over MI is primarily in I2T retrieval. Across five seeds, TE achieves 10.08 ± 0.17% versus MI’s 8.35 ± 0.22% I2T R@1, while MI remains higher on T2I (7.35 ± 0.28% versus 7.12 ± 0.19%; Table 23). The smaller T2I gap should not be dismissed solely because the standard deviations overlap.

The trade-off depends on the weight and temperature. The reference weight was selected for I2T; reducing γ\gamma to 2.5 gives 7.57% T2I R@1 while retaining 9.48% I2T R@1 (Table 6). At τ\tau = 0.03, TE exceeds MI in both directions, and the two perform similarly on Flickr30k.

The modality-specific ablations also suggest different contributions from image-side and text-side alignment. However, we do not have a complete mechanistic explanation. We now qualify the direction-dependent advantage and identify separate modality weights as future work.

W4: Appendix H.6 now directly compares ordinary and squared cosine on the same within-batch differences, with the same base objective and weight. Their retrieval performance is comparable, suggesting that sign invariance is not essential to the gains.

For individually unit-normalized differences, normalized MSE is equivalent to ordinary-cosine loss up to a constant factor absorbed into the loss weight. Thus, the same ablation covers this alternative. Together with the unnormalized MSE-Δ\Delta baseline, these results support scale-independent direction matching as the useful component. We retain squared cosine for its projector interpretation and restricted Gaussian correspondence, without claiming a distinct accuracy advantage.

AuthorsOpenReview ↗

Reply to "Questions For Authors"

Q1: The original rows in Tables 1–2 compare individual loss components within our pipeline, rather than complete methods. These include CLIP-KD’s feature and interactive-contrastive losses, KL-based similarity distillation, a CRD-style InfoNCE objective, and Gram matching. We have clarified this distinction in the captions and text.

Appendix H.6 now evaluates the CLIP-KD objective with its official loss weights and RKD added to our base, using matched architectures, data, epochs, batch size, and evaluation. MSCOCO I2T / T2I R@1 (%) are:

  • Base + RKD: 6.94 / 6.42
  • CLIP-KD: 7.77 / 6.72
  • Ours, five seeds: 10.08 ± 0.17 / 7.12 ± 0.19

The regularizer adds approximately one minute over 10 epochs, giving comparable training budgets. This evaluates CLIP-KD’s objective under our protocol, not its large-scale recipe. Our claim is therefore limited to controlled comparisons in this setting, rather than state-of-the-art absolute performance.

Q2: We agree that the improvement over MI is primarily in I2T. Across five seeds, TE achieves 10.08 ± 0.17% versus MI’s 8.35 ± 0.22% I2T R@1, while MI remains higher on T2I (7.35 ± 0.28% versus 7.12 ± 0.19%). Three observations help characterize this trade-off:

  • Modality-specific effects: Image-only alignment improves I2T from 6.08% to 7.46% but lowers T2I from 5.92% to 5.69%. Text-only alignment improves both, to 7.49% and 7.03%, respectively.
  • Different preferred weights: I2T peaks at γ\gamma = 7.5, whereas T2I peaks at γ\gamma = 2.5. Selecting the reference weight for I2T therefore accounts for part of the trade-off.
  • Unequal teacher agreement: Student–teacher cosine agreement is higher for text than images (0.840 versus 0.726). This is consistent with the interpretation that image queries benefit from a better-aligned text gallery, but does not establish that explanation.

These observations characterize the asymmetry but do not establish a complete mechanism. Separately weighting image-side and text-side alignment is a natural extension.

Q3: Appendix H.6 compares ordinary and squared cosine on the same differences and base objective. Ordinary cosine achieves 9.85 / 6.79% I2T / T2I R@1, compared with 10.08 ± 0.17 / 7.12 ± 0.19% for squared cosine.

The ordinary-cosine result is a single run at γ\gamma = 7.5, the weight selected for squared cosine; it was not independently tuned. Thus, this is a matched-weight ablation, not a comparison with equal tuning budgets. It shows that substantial gains persist without sign invariance, but does not establish whether either form is superior after tuning.

We retain squared cosine for its projector interpretation and its correspondence with the Gaussian reference under Appendix C’s assumptions, without claiming a distinct accuracy benefit.

Q4: Thank you for this suggestion. We have not evaluated downstream multimodal LLMs, so we distinguish our hypothesis from demonstrated results.

We agree that teacher-specific directions may serve as a useful training reference without being necessary for downstream performance. A trainable projector and LLM may compensate for changes in representation coordinates, so improved direct retrieval does not establish improved downstream performance.

A possible transferable benefit is preservation of useful distinctions between inputs. The TE student has higher teacher-neighborhood overlap than MI (kNN overlap@10: 0.605 versus 0.546), although this alone does not demonstrate downstream utility. A projector cannot recover information absent from a frozen encoder’s features; joint encoder adaptation could change this limitation.

Our regularizer acts on global embeddings, whereas multimodal LLMs commonly use patch-level features, so this mismatch also requires investigation. We have added a future-work comparison of otherwise matched multimodal LLMs using encoders distilled with and without the regularizer on VQA-style tasks, together with a citation to PORTAL.

Reviewer_kCX1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces new regularizers for distilling CLIP-style vision-language models that match the direction of the student's local embedding changes to the teacher's. These regularizers come in two variants, TE1 and TE2 — both in the form of squared cosine similarity between within-batch finite differences — serving as a cheap, tractable proxy for transfer entropy. It shows that this orientation-invariant geometry matching outperforms static similarity, MI, KL, and MSE baselines on image-to-text retrieval (while remaining competitive on text-to-image) at negligible cost.

Critical strengths

1. Well-motivated idea

The paper provide a new perspective to match the underlying geometry of the student's updates rather than its representations, framed through transfer entropy (the teacher guides the student's next-step refinement). The idea is interesting, non-obvious, and honestly separates the ideal TE from the tractable proxy actually optimized.

2. Broad validation

Gains hold across multiple teachers (RN50, ViT-B/16, RN50×16), students (RN34, RN18), datasets (MSCOCO, Flickr8k, Flickr30k), temperatures, batch sizes, and five seeds, plus cross-dataset transfer and out-of-domain classification. This extensive evidence strongly supports the effectiveness of the method.

Critical weaknesses

1. The TE framing is motivational only, TE is never estimated or measured.

TE1/TE2 are just squared-cosine finite-difference regularizers, and no experiment shows the TE-specific content (directional/asymmetric information flow) actually contributes. Strip the TE language and the method is "match local update directions" with an information-theoretic story attached.

2. Many notation definitions are unclear.

For example, the key variables of the ``TE surrogate'' (aa, bb, zz, ℓ\ell, h~\tilde{h}) are undefined. The definition of δx\delta x in Eq.~(6) is also unclear.

Questions for authors

I have two questions: Question 1: Why does the objective match one-dimensional subspaces (spanning each finite difference) rather than, say, full difference vectors or higher-dimensional response subspaces?

Question 2: Why is squared cosine similarity chosen over alternatives such as ordinary cosine, Euclidean distance, or a divergence. And specifically, what does squaring contribute beyond sign-invariance?

1 comment
AuthorsOpenReview ↗

TE Motivation, Geometric Interpretation, and Additional Ablations

We appreciate the reviewer’s careful examination of our work and helpful feedback. We address the comments point by point below.

W1: We agree that TE is not estimated or measured. TE1/TE2 are squared-cosine regularizers on within-batch finite differences, not optimization-step updates. The revised manuscript presents their geometric interpretation as the primary characterization; TE provides conceptual motivation only.

Appendix C provides a controlled sanity check under linear–Gaussian assumptions, where the proxies track a reference determined by teacher–student correlation. This does not establish directional information flow. The proxies are symmetric, and freezing the teacher introduces training asymmetry, not an asymmetric information measure. Moreover, signed cosine performs comparably to squared cosine (Sec. 4.2.2; Appendix H.6), suggesting that sign invariance is not essential to the gains.

Our contribution is therefore the regularizer and its empirical behavior. Under the same training protocol, it achieves 10.08 ± 0.17% MSCOCO I2T R@1, compared with RKD (6.94%), Gram matching (7.01%), CLIP-KD (7.77%), and MI (8.65%). These results support the usefulness of aligning teacher–student difference geometry, without attributing the gains to TE-specific directional content.

W2: Thank you for pointing this out. We have revised both passages so that every symbol is defined before it is used.

TE surrogate (Eqs. 3–4). ℓℓ is the layer index; aa and bb are the source and target components of the model; h~\tilde h is the hidden representation of a component at a given layer for one example of the mini-batch; and Δz\Delta z is the difference of h~\tilde h between adjacent examples.

Eq. (6) fSf_S and fTf_T are the student and teacher encoders of one modality; xx is an input (an image or a caption); JS(x)J_S(x) and JT(x)J_T(x) are their Jacobians with respect to the input; and δx is the difference between two inputs of a mini-batch.

Q1: Each nonzero finite difference defines a secant line. Matching these lines provides a simple, scale-invariant way to align teacher and student variations in their shared embedding space.

Versus full differences. Full-vector matching penalizes both direction and magnitude and overlaps with pointwise feature matching. With feature errors ei=si−tie_i = s_i - t_i, the difference error satisfies ∥ej−ei∥2≤2∥ei∥2+2∥ej∥2\|e_j-e_i\|^2 \leq 2\|e_i\|^2 + 2\|e_j\|^2. Thus, feature MSE already controls the absolute difference error, whereas direction matching constrains geometry independently of magnitude. Empirically, unnormalized MSE on differences gives modest gains at the tested weights (6.52–6.67% versus 6.08% base I2T R@1).

Versus higher-dimensional subspaces. Exact alignment of corresponding nonzero secant lines preserves their collective span. Matching only the span permits internal re-mixing without preserving individual correspondences. Random pairing samples multiple secants across training, and line matching requires only differences and cosines, with O(Bd)O(Bd) cost. We have not evaluated higher-dimensional subspace matching and leave it to future work.

Q2: Squared cosine provides an exact projector interpretation (Eq. 16) and a correspondence with the Gaussian reference under Appendix C’s assumptions. However, our ablations suggest that squaring is not essential to the gains.

  • Ordinary cosine: Signed cosine achieves 9.85 / 6.79% I2T / T2I R@1, comparable to squared cosine’s 10.08 ± 0.17 / 7.12 ± 0.19% (Appendix H.6).
  • Euclidean distance: Unnormalized MSE on differences gives smaller gains. For individually unit-normalized differences, squared Euclidean distance equals twice the signed-cosine loss, so the same ablation covers this alternative after adjusting the loss weight.
  • Divergences: Our base includes KL distillation, and we compare against an InfoNCE-style MI objective. These comparisons do not exhaust possible divergences on difference representations.

Beyond sign invariance, squaring changes gradient weighting: ∇(c2)=2c∇c\nabla(c^2)=2c\nabla c, where cc is cosine similarity. Near-orthogonal pairs receive a smaller gradient factor, while either positive or negative alignment is reinforced. Squared cosine also equals the fraction of squared vector norm captured by projection onto the other vector’s span.

We retain squared cosine for these geometric and motivational reasons, without claiming an empirical advantage over signed cosine or attributing the gains to TE-specific content.

Reviewer_behvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes two auxiliary objectives for CLIP-style distillation, motivated by transfer entropy. Within each batch, it forms difference vectors between randomly paired embeddings and rewards the squared cosine similarity between the teacher's and the student's difference vectors, either per modality (TE1) or on concatenated image-text differences (TE2). Experiments distill CLIP RN50, ViT-B/16 and RN50x16 teachers into small RN34/RN18 students trained on MSCOCO, Flickr8k, Flickr30k or Food-101.

Critical strengths
  1. The objective is simple, adds negligible compute and no extra teacher passes, and can be combined with any CLIP-style distillation loss.

  2. In the controlled comparison with a fixed base objective (Table 2), the proxy consistently improves image-to-text retrieval over the MI, Gram, and MSE-on-differences regularizers, and this holds across temperatures, batch sizes, teacher-student pairs, and datasets.

Critical weaknesses
  1. The link to transfer entropy is nominal, and the closest baselines are missing.

The TE in Eq. 5 is defined over optimization steps t and t+1, but the proxy compares difference vectors between random samples of the same batch at a single step, so it is not derived from Eq. 5. The Jacobian argument also does not apply to random pairs, and nearest-neighbor pairing performs about the same (Table 10). Without the framing, the method is a relational KD loss that matches normalized pairwise differences up to sign. This is close to the angle term of RKD, which is cited but not compared; the only relational baseline is a Gram-matrix loss. The paper also does not ablate the defining choices, such as squared vs. signed cosine or a scale-normalized MSE on differences, so the source of the gain and the reason for sign invariance remain unclear.

  1. The evaluation protocol makes the gains hard to trust and to compare.

(a) The loss weights are tuned on "MSCOCO validation Recall@K" (Sec. 5), Appendix G.1 lists only the training and 40,504-image validation splits, and Appendix G.12 states that all metrics are computed on the full validation split. Selection and reporting therefore appear to use the same data, and the nonstandard split prevents comparison with published results on the Karpathy 5K test set.

(b) The claim that the student surpasses its teacher on Food-101 is not attributable to the method: the student is trained on Food-101 images while the teacher is zero-shot, and the baseline without TE (80.23%) already exceeds the teacher (79.80%).

(c) Students are trained from scratch on about 82K images for 10 epochs and reach about 10% image-to-text R@1, so it is unclear whether the gains carry over to the web-scale settings where CLIP distillation is used. The representation diagnostics also compare only against MI, which is not designed to preserve teacher geometry.

Questions for authors
  1. Which split was used for selecting the loss weights, and which for the reported results? Could you report results with weights selected on held-out training data and on the Karpathy 5K test split?

  2. Why do the TE and MI rows in Table 22 use different α and β values, and were the MI settings tuned over the same range?

  3. Are the students initialized from pretrained weights, and how is the comparison dimension obtained for the 512-dimensional ViT-B/16 teacher?

  4. Could you add comparisons with RKD and with a complete CLIP distillation recipe (e.g., TinyCLIP or CLIP-KD), together with ablations of squared vs. signed cosine and a normalized MSE on differences?

2 comments
AuthorsOpenReview ↗

Reply to Critical Weaknesses

We thank the reviewer for the thorough assessment and constructive suggestions. We respond to each concern below.

W1: We agree that TE1/TE2 are neither derived from Eq. (5) nor estimators of it. Their functional form is adapted from Liang & Shi (2026) to teacher–student differences across within-batch pairs. This is an analogy, not a derivation from optimization-step TE. The abstract’s phrase “derive from it” overstated this connection and has been removed.

The objectives have an exact geometric interpretation: Eq. (16) shows that maximizing squared cosine is equivalent to minimizing the squared Frobenius distance between rank-one projectors onto nonzero teacher and student difference vectors. TE1 applies this separately per modality; TE2 uses the concatenated space. These symmetric objectives align secant-line geometry but do not measure directional information flow. Appendix C provides a Gaussian reference under restricted assumptions, not a general equivalence to TE.

The revised manuscript explicitly distinguishes optimization-step TE from within-batch differences and moves the projector interpretation from Appendix E.7 to Sec. 4.2.2 as the primary mathematical characterization.

Regarding Jacobian interpretation, we agree that the Jacobian interpretation applies only to nearby pairs. Table 10 suggests that locality is not essential. The revised text therefore describes secant-direction alignment and restricts the Jacobian interpretation to the local case.

RKD matches angles formed by triplets within each model; TE1/TE2 directly align teacher and student differences in a shared space. New comparisons in Appendix H.6 follow the Table 2 protocol (MSCOCO I2T / T2I R@1, %):

  • Base + RKD angle: 6.85 / 6.37
  • Base + RKD distance + angle: 6.94 / 6.42
  • Base + Gram matching: 7.01 / 5.92
  • Base + TE (five seeds): 10.08 ± 0.17 / 7.12 ± 0.19

Our objective achieves higher recall in both directions.

Regarding defining choices, as discussed in Sec. 4.2.2, signed cosine performs comparably to squared cosine, suggesting that sign invariance is not essential to the gains. MSE between individually unit-normalized differences is equivalent to signed-cosine loss up to a constant factor absorbed into the loss weight, so the same ablation covers both.

W2 (a) We acknowledge that the original reference weights were selected and evaluated on val2014. Appendix H.14 now separates selection on Karpathy 5K validation from evaluation on the disjoint 5K test set, with neither used for training. The selected TE model exceeds MI on I2T R@1 (29.70% vs. 28.34%), while MI remains stronger on T2I (22.47% vs. 20.56%). Table 26 reports the full results; training remains train2014-only.

(b) We agree and have removed the teacher-surpassing claim. The student receives in-domain images, and the base already exceeds the zero-shot teacher. TE improves the base from 80.23% to 82.46% top-1, comparable to MI’s 82.37%. On ImageNet-1k, without ImageNet images used in distillation, TE achieves 10.62% versus MI’s 8.61%.

(c) Web-scale effectiveness remains untested. However, the image encoder is pretrained; only the text encoder is trained from scratch. We test robustness across three teachers, two student architectures, three retrieval datasets, multiple temperatures and batch sizes, and five seeds. Appendix H.6 also adds neighborhood-overlap and cosine-agreement diagnostics for RKD, Gram matching, and CLIP-KD, addressing the limitation of comparing geometry only against MI.

AuthorsOpenReview ↗

Reply to "Questions For Authors"

Q1: Students are trained on train2014. In the submission, weights were selected and results reported on the full val2014 split. Appendix H.14 now separates selection on Karpathy 5K validation from evaluation on the disjoint 5K test set; neither is used for training. Selecting by validation I2T R@1 gives γ\gamma = 7.5 and η\eta = 5. On the test set, TE exceeds MI on I2T R@1 (29.70% vs. 28.34%) and R@5 (57.56% vs. 56.58%), while MI leads on T2I R@1 (22.47% vs. 20.56%). Table 26 reports all settings. Selection uses held-out validation data, not a subset of train2014.

Q2: Table 23 reports five-seed reruns of each method’s best I2T configuration from its single-seed sweep. The RN50 settings match. For ViT-B/16, MI was also evaluated at TE’s selected α/β\alpha/\beta settings but performed worse than at its own selected settings. We acknowledge that MI’s sweep was narrower: η∈\eta\in{1, 2.5, 5, 7.5}, versus γ∈\gamma\in{1, 2.5, 5, 7.5, 10}, with fewer α/β\alpha/\beta variants. With matched α\alpha = 1 and β\beta = 50, TE still exceeds MI on ViT-B/16 MSCOCO (7.87% vs. 7.28% I2T R@1) and Flickr30k (40.24% vs. 39.55%), although the margins are smaller.

Q3: The RN34/RN18 image encoder uses torchvision pretrained weights, with a randomly initialized replacement output layer. The two-layer text encoder, including token and positional embeddings, is trained from scratch. All objectives use the same initialization protocol. For the ViT-B/16 teacher, the student image output and text width are both 512-dimensional. All vector-space losses operate directly on dimension-matched embeddings, without a teacher projection or auxiliary projector.

Q4: Appendix H.6 adds comparisons under the Table 2 training protocol. MSCOCO I2T / T2I R@1 (%) are:

  • Base + RKD angle: 6.85 / 6.37
  • Base + RKD distance + angle: 6.94 / 6.42
  • CLIP-KD, using its official loss weights: 7.77 / 6.72
  • Signed cosine: 9.85 / 6.79
  • Squared cosine, five seeds: 10.08 ± 0.17 / 7.12 ± 0.19

For individually unit-normalized differences, normalized MSE equals twice the signed-cosine loss, so the same ablation covers both after adjusting the loss weight. The comparable signed- and squared-cosine results suggest that sign invariance is not essential to the gains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.