- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TE-VLM, a transfer-entropy-inspired framework for distilling CLIP-style vision–language dual encoders using two geometric regularizers. TE1 aligns teacher–student pairwise embedding differences separately for images and text, while TE2 aligns their concatenation; both preserve secant directions up to sign through squared cosine similarity. Experiments across teacher–student configurations and retrieval datasets show consistent image-to-text improvements but mixed text-to-image results. Additional transfer experiments and ablations support the regularizers’ empirical usefulness, although their connection to transfer entropy is not directly established.
Critical strengths
-
The proposed finite-difference alignment is simple, inexpensive, and empirically useful. It operates on already-computed embeddings without additional teacher forward passes. Importantly, the controlled comparison in Table 2 isolates its contribution: adding TE2 increases image-to-text R@1 from 6.08% to 10.32%, compared with 8.65% for the MI regularizer. This provides evidence that matching teacher–student difference directions up to sign offers useful supervision beyond the existing distillation objectives.
-
The image-to-text improvements are supported by robustness and transfer experiments. The temperature and batch-size ablations (Table 3), five-seed results across multiple teacher–dataset configurations (Table 22), and cross-dataset retrieval evaluation (Table 4) strengthen confidence that the gains are not specific to a single training configuration. This broader empirical support makes the lightweight regularizer a potentially useful addition to CLIP-style distillation.
Critical weaknesses
-
The connection to transfer entropy could be explained more clearly. Eq. (5) describes how the teacher provides additional information about the student’s next-step representations, given its current state. In contrast, TE1/TE2 compare embedding differences between samples within a single training step. The authors acknowledge this distinction in Appendix H (Limitations, p. 43), noting that the experiments validate the geometric regularizer rather than the underlying TE quantity. The relationship between these two types of changes remains unclear, limiting the explanatory value of the TE motivation.
-
Comparisons with state-of-the-art CLIP distillation methods would strengthen the evaluation. Table 2 provides useful comparisons between loss terms within the authors’ training pipeline. However, comparisons against complete methods from recent CLIP distillation work, under matched architectures, training data, evaluation protocols, and comparable compute budgets, would better establish practical competitiveness beyond the loss-level comparisons.
-
The retrieval improvements vary by direction. In Table 2, TE2 improves I2T R@1 from 6.08% to 10.32%, exceeding MI’s 8.65%. However, its T2I R@1 is 7.33%, compared with 7.65% for MI. This does not diminish the I2T gains, but indicates that the advantage is task-direction dependent and would benefit from further explanation.
-
The contribution of the specific directional objective could be better isolated. Table 2 compares TE1/TE2 with Gram matching and finite-difference MSE, providing useful evidence for the proposed regularizer. However, these alternatives differ in several respects, including their treatment of direction and magnitude. A direct comparison between ordinary cosine and squared-cosine alignment on the same sample differences would help establish whether the proposed sign-invariant formulation provides a distinct benefit.
Questions for authors
-
Comparison with existing distillation methods. Could the authors clarify how the implemented baselines relate to the complete methods in the cited CLIP distillation literature? Comparisons with strong existing methods under matched architectures, data, and training budgets would help assess practical competitiveness.
-
Retrieval asymmetry. What explains the stronger improvements in I2T than T2I? For example, Table 2 reports TE2 versus MI R@1 of 10.32% versus 8.65% for I2T, but 7.33% versus 7.65% for T2I. Beyond the teacher’s existing asymmetry, is there evidence explaining why the regularizer favors one retrieval direction?
-
Choice of directional objective. Have the authors compared squared cosine with ordinary cosine alignment using the same sample differences and comparable weight tuning? This would help determine whether allowing parallel and anti-parallel differences to receive the same reward provides a distinct benefit.
-
Does directional alignment benefit downstream multimodal LLMs? My concern is that preserving teacher-specific directions may improve direct CLIP retrieval without necessarily improving downstream tasks when an LLM can adapt to the representations.
- Rethinking Projector Training in Multimodal LLMs (PORTAL) shows that multiple initial projector mappings can support comparable downstream performance after joint adaptation. This suggests that a unique initial alignment may not be necessary, although it does not imply that arbitrary initializations work equally well.
- This raises the hypothesis that teacher-specific directions may be a useful training reference rather than a necessary property of transferable representations. A trainable downstream LLM might learn to use alternative representations equally well, so improved directional agreement need not translate into better downstream performance.
- Do the authors expect TE regularization to remain beneficial in this setting, and why? As a future extension, comparing otherwise matched multimodal LLMs using representations distilled with and without TE regularization on tasks such as VQA would help test this hypothesis. I view this as a question about broader applicability, rather than a required rebuttal experiment.
2 comments
Reply to "Questions For Authors"
Q1: The original rows in Tables 1–2 compare individual loss components within our pipeline, rather than complete methods. These include CLIP-KD’s feature and interactive-contrastive losses, KL-based similarity distillation, a CRD-style InfoNCE objective, and Gram matching. We have clarified this distinction in the captions and text.
Appendix H.6 now evaluates the CLIP-KD objective with its official loss weights and RKD added to our base, using matched architectures, data, epochs, batch size, and evaluation. MSCOCO I2T / T2I R@1 (%) are:
- Base + RKD: 6.94 / 6.42
- CLIP-KD: 7.77 / 6.72
- Ours, five seeds: 10.08 ± 0.17 / 7.12 ± 0.19
The regularizer adds approximately one minute over 10 epochs, giving comparable training budgets. This evaluates CLIP-KD’s objective under our protocol, not its large-scale recipe. Our claim is therefore limited to controlled comparisons in this setting, rather than state-of-the-art absolute performance.
Q2: We agree that the improvement over MI is primarily in I2T. Across five seeds, TE achieves 10.08 ± 0.17% versus MI’s 8.35 ± 0.22% I2T R@1, while MI remains higher on T2I (7.35 ± 0.28% versus 7.12 ± 0.19%). Three observations help characterize this trade-off:
- Modality-specific effects: Image-only alignment improves I2T from 6.08% to 7.46% but lowers T2I from 5.92% to 5.69%. Text-only alignment improves both, to 7.49% and 7.03%, respectively.
- Different preferred weights: I2T peaks at = 7.5, whereas T2I peaks at = 2.5. Selecting the reference weight for I2T therefore accounts for part of the trade-off.
- Unequal teacher agreement: Student–teacher cosine agreement is higher for text than images (0.840 versus 0.726). This is consistent with the interpretation that image queries benefit from a better-aligned text gallery, but does not establish that explanation.
These observations characterize the asymmetry but do not establish a complete mechanism. Separately weighting image-side and text-side alignment is a natural extension.
Q3: Appendix H.6 compares ordinary and squared cosine on the same differences and base objective. Ordinary cosine achieves 9.85 / 6.79% I2T / T2I R@1, compared with 10.08 ± 0.17 / 7.12 ± 0.19% for squared cosine.
The ordinary-cosine result is a single run at = 7.5, the weight selected for squared cosine; it was not independently tuned. Thus, this is a matched-weight ablation, not a comparison with equal tuning budgets. It shows that substantial gains persist without sign invariance, but does not establish whether either form is superior after tuning.
We retain squared cosine for its projector interpretation and its correspondence with the Gaussian reference under Appendix C’s assumptions, without claiming a distinct accuracy benefit.
Q4: Thank you for this suggestion. We have not evaluated downstream multimodal LLMs, so we distinguish our hypothesis from demonstrated results.
We agree that teacher-specific directions may serve as a useful training reference without being necessary for downstream performance. A trainable projector and LLM may compensate for changes in representation coordinates, so improved direct retrieval does not establish improved downstream performance.
A possible transferable benefit is preservation of useful distinctions between inputs. The TE student has higher teacher-neighborhood overlap than MI (kNN overlap@10: 0.605 versus 0.546), although this alone does not demonstrate downstream utility. A projector cannot recover information absent from a frozen encoder’s features; joint encoder adaptation could change this limitation.
Our regularizer acts on global embeddings, whereas multimodal LLMs commonly use patch-level features, so this mismatch also requires investigation. We have added a future-work comparison of otherwise matched multimodal LLMs using encoders distilled with and without the regularizer on VQA-style tasks, together with a citation to PORTAL.
Reply to Critical Weaknesses
We are grateful for the reviewer’s detailed comments and valuable suggestions. Below, we respond to the issues raised.
W1: We agree and have made this distinction explicit in Secs. 4.1–4.2. Eq. (5) concerns the teacher’s additional predictive information about student representations across optimization steps. TE1/TE2 instead align teacher and student differences across inputs at a single step. Their relationship is motivational, not a derivation.
For a fixed input, the frozen teacher has no optimization-step change. We therefore compare variation across inputs, where both teacher and student representations vary. This design choice does not make the proxies estimators of Eq. (5). Their functional form is adapted from Eq. (4), and their primary mathematical characterization is secant-line alignment (Eq. 16).
Appendix C provides a restricted linear–Gaussian correspondence with a correlation-based reference, not evidence of directional information flow. Our contribution is the regularizer and its empirical behavior; we do not claim that TE is measured or explains the gains.
W2: Appendix H.6 now evaluates the CLIP-KD objective (CL + CRD + FD + ICL), using its official loss weights with the same architectures, data, epochs, batch size, and evaluation protocol as our method. MSCOCO I2T / T2I R@1 (%) are:
Our base contains the same four loss components as CLIP-KD, with the proposed regularizer added. Its measured overhead is approximately one minute over a 10-epoch run (Table 21), giving comparable compute budgets.
This comparison evaluates CLIP-KD’s objective under our controlled training protocol, rather than reproducing its large-scale recipe. We therefore claim improvements within this setting, not state-of-the-art absolute performance or demonstrated competitiveness at web scale.
W3: We agree that the advantage over MI is primarily in I2T retrieval. Across five seeds, TE achieves 10.08 ± 0.17% versus MI’s 8.35 ± 0.22% I2T R@1, while MI remains higher on T2I (7.35 ± 0.28% versus 7.12 ± 0.19%; Table 23). The smaller T2I gap should not be dismissed solely because the standard deviations overlap.
The trade-off depends on the weight and temperature. The reference weight was selected for I2T; reducing γ to 2.5 gives 7.57% T2I R@1 while retaining 9.48% I2T R@1 (Table 6). At τ = 0.03, TE exceeds MI in both directions, and the two perform similarly on Flickr30k.
The modality-specific ablations also suggest different contributions from image-side and text-side alignment. However, we do not have a complete mechanistic explanation. We now qualify the direction-dependent advantage and identify separate modality weights as future work.
W4: Appendix H.6 now directly compares ordinary and squared cosine on the same within-batch differences, with the same base objective and weight. Their retrieval performance is comparable, suggesting that sign invariance is not essential to the gains.
For individually unit-normalized differences, normalized MSE is equivalent to ordinary-cosine loss up to a constant factor absorbed into the loss weight. Thus, the same ablation covers this alternative. Together with the unnormalized MSE-Δ baseline, these results support scale-independent direction matching as the useful component. We retain squared cosine for its projector interpretation and restricted Gaussian correspondence, without claiming a distinct accuracy advantage.