acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Vision-Language Embedding Distillation Through Negative Composition

14 views

Abstract

Contrastive learning is a core objective for vision–language embedding models, where in-batch examples define the negative relations that shape the learned space. In heterogeneous training, globally mixed batches introduce negatives from datasets whose candidate pools are evaluated separately. We study this negative-source mismatch in vision–language embedding distillation. Across FastVLM-0.5B and LLaVA-OneVision-0.5B, classification and visual question answering, five objectives, and three training seeds, task-consistent batches improve overall Precision@1 by 1.54–3.72 points relative to the corresponding mixed-task objective. For contrastive-only training, the gains are 2.03–3.04 points. By comparison, under task-consistent batching, auxiliary distillation changes the overall score by only to points relative to contrastive-only training. Dataset-level effects are heterogeneous, with large gains on many datasets and regressions on a few. Because the samplers handle per-dataset remainders differently, the comparison does not isolate the causal effect of negative-source composition. Nevertheless, the aggregate association is consistent across settings, identifying negative-source composition as a first-order experimental axis and motivating matched-exposure and gradient-level tests of the proposed mechanism.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
contrastive learning, vision language embeddings, negative sampling, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_C5W7OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper argues that in vision–language embedding distillation the in-batch negative set matters more than the auxiliary KD loss, crossing five objectives with mixed-task and task-consistent samplers on two 0.5B students, two MMEB task families and three seeds: task-consistent batching lifts Precision@1 by 1.54–3.72 points, while under task-consistent batching adding the tested KD losses changes it by only −0.60 to +0.57 relative to contrastive-only training (Tables 1–2).

  • Three mechanism accounts are offered: a density-ratio argument at the population level, an exact gradient-gap decomposition at the finite-batch level, and a single-seed rebatching probe.

  • The design is systematic, every headline range checks against the tables, and the association is a useful corrective for how KD objectives are compared.

  • The paper has not yet done the controlled, documented and mechanism-tied work its own stronger wording calls for: the abstract concedes the comparison "does not isolate the causal effect", the recipe is unreported, the appendix's no-teacher baselines move by 9–46 points, and the probe is not tied to the per-dataset outcomes.

Critical strengths

A systematic factorial design whose numbers all reproduce from the tables.

  • The design is a full factorial reported with variance and per-dataset detail, and its arithmetic holds up. Specifically, two students × two task families × five objectives × three samplers × three seeds, every main-table cell with a standard deviation and a per-dataset breakdown; every range quoted in the abstract and Sec. 4.2 (1.54–3.72, 2.03–3.04, −0.60 to +0.57, 2.80–3.72, 1.46–2.76) re-derives from Tables 1–2, and the Figure 1 bar heights match.

A practically important message, stated with unusual care about what each result does not show.

  • The message matters for how KD objectives are compared, and the paper states its limits plainly. Specifically, if a sampler choice moves a student by 1.54–3.72 points while, under task-consistent batching, adding any of four KD losses moves it by at most 0.60, then sampler differences can confound the attribution of gains to a KD objective; and the paper says Prop. 1 "does not imply that homogeneous training is better" and that the probe is "descriptive rather than causal evidence".
Critical weaknesses

The comparison the paper rests on is, by its own account, not controlled, the recipe needed to judge it is unreported, and the appendix's no-teacher baselines are inconsistent with the setting it says was fixed.

  • The body claims a causal result the abstract concedes the comparison does not isolate, and the recipe needed to judge the KD comparison is absent. Specifically, the abstract says the samplers "handle per-dataset remainders differently" so the comparison "does not isolate the causal effect", yet Contribution 1 says negative composition "outweighs" distillation and Sec. 3.1 says the contrast "isolates task compatibility"; Sec. 4.1 gives no learning rate, step count, temperature or KD weight.

  • The appendix's no-teacher baselines contradict its own claim that only the teacher changed. Specifically, Appendix B.2 says only the teacher changed, yet the teacher-free contrastive-only row falls from 70.96 to 61.94 on CLS and from 53.79 to 7.50 on VQA between Tables 4–5 and Tables 8–9, with no explanation, so the second-teacher check is not clean.

The mechanism evidence is a single-seed embedding-space probe not yet tied to the outcomes it is meant to explain, and a promised measurement is missing.

  • The probe is narrow, and the submission does not establish a matched quantitative link between it and the reported per-dataset gains. Specifically, it covers one student, task family and seed and measures embedding-space descent rather than a model update; the single-setting Dfull values and the objective-averaged Table 3 gains are not the same measurement (N24News −3.18 against +0.19, Hateful Memes −0.08 against −1.31 on their face), and the largest gains are out of distribution, beyond the probe's reach.

  • A promised measurement is absent and a large collapse is undiagnosed. Specifically, Sec. 3.5 says "we report" the Eq. (21) statistics, yet none appear; and the 25.11–31.61-point FastVLM VQA hard-batch collapse is labelled, not explained.

Questions for authors

See weaknesses.

Reviewer_HzXDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the choice of negative-samples sampling in training vision-language models and how the choice impacts the training loss. They start by analyzing how the loss target changes with negative samples drawn from a homogeneous distribution as opposed to drawn from a mixed distribution (a mix of different datasets). They then break down the gradient and attribute the difference between the two to three terms for interpretation. In the experiment, they train two small models with variants of distillation objectives and three sampling choices, and evaluate them on both in-distribution and out-of-distribution sets. The key takeaway from the experiment results is that the task-consistent sampling improves the performance more than the mixed-task batches, and the gap is attributed to the lack of within-task coverage in the batch sampling.

Critical strengths
  • The core question on the choice of batch sampling is a crucial problem in the training recipe of VLMs that may be overlooked in the prior works.
  • The connection made to knowledge distillation is a practical consideration given how distillation is widely adopted for VLM training.
  • The experiment datasets are comprehensive and make the results more robust and convincing.
Critical weaknesses
  • The theoretical justification does not directly address why one would be better than the other (mixed negatives or task-consistent negatives). Proposition 1 only demonstrates whether negative sampling impacts the loss target. Proposition 2 doesn't provide more information than the intuition that whichever gradient aligns better with the reference has the lower reference loss. The takeaway can be communicated clearer to the reader on what the analysis aims to convey. Lastly, the gradient decomposition in Section 3.4. only provides information about how the gradient is changed in the three variants for probing. Similarly, the KD analysis does not cover the KD variants tested and the gradient angle attribution is also not measured to illustrate the contribution.
  • The details and goal for the ablation is not clearly stated. Figure 4 shows a mixed result where the gradient alignment doesn't alway decrease or increase for all tasks. The loss prediction is on the right subplot is almost certain because the reference loss is supposed to follow the reference gradient. Figure 5 attributes the DfullD_{\text{full}} to DFD_F, and yet the batch size is also a difference between the restricted set and the homogeneous set. This can be a confounder where the homogeneous batch has more samples and can just be a larger batch size that helps with stability and more stable estimate of the reference gradient.
  • The related work section mentioned several prior works on the design choices of batches but none of them is compared. It'll be more convincing if the breakdown and attribution can be generalized to those settings.
  • The presentation can be improved where the establishment of purpose would be very helpful including what the takeaway of the analysis, the goal of experiments, and the purpose of the ablation are.
  • nit: Figure 3 legend seems too small; Figure 5 has the same issue
Questions for authors
  • Can you clarify how the analysis demonstrates the task-consistent sampling being better than the mixed-batches ones? For example, Dfull<0D_\text{full} < 0 is only one possibility and Dfull>0D_\text{full} > 0 is also possible. This gives no information on which one is more likely to happen, which would reflect what the empirical results show.
  • Can you clarify how the knowledge distillation analysis is related to the empirical results such as how the different KD objectives can be broken down and would behave differently under the analytical framework proposed in the paper?
Reviewer_dRxXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines negative-source composition in vision-language embedding training with auxiliary knowledge distillation. All evaluated objectives retain a contrastive loss, and four variants separately add MSE, RKD, HieRD, or Merge KD. Across two compact students, the CLS and VQA families of MMEB, and three seeds, dataset-homogeneous batching improves aggregate Precision@1 by 1.54–3.72 points over mixed batching, whereas adding distillation under homogeneous batching changes performance by only −0.60 to +0.57 points. The paper also develops a gradient decomposition and a paired rebatching probe suggesting that reduced same-dataset coverage contributes to the mixed-batch deficit.

Critical strengths
  1. Controlling the student's underlying contrastive training when evaluating auxiliary distillation is a meaningful concern. The paper appropriately treats homogeneous batching as an experimental intervention rather than a new method.

  2. Crossing sampling conditions with contrastive-only training and four auxiliary distillation objectives is well aligned with comparing their conditional performance gains. Evaluation across two students, two task families, and three seeds provides useful breadth within the chosen setting.

  3. The paired rebatching probe holds query–positive pairs fixed while changing companion candidates, providing a useful diagnostic beyond final performance. The paper also acknowledges the limits of its population analysis, embedding-space updates, and path-dependent decomposition.

Critical weaknesses
  1. The empirical novelty is insufficiently distinguished from existing batching studies. The paper acknowledges that homogeneous batching is established, but broadly characterizes prior evidence as recipe-level ablations entangled with other choices. However, Arctic-Embed already includes a targeted source-stratification ablation, VLM2Vec-V2 compares source-mixing granularities from fully mixed to fully homogeneous batches, and WeMM-Embedding reports a 3.4-point MMEB-v2 decrease when task-consistent batching is removed. The paper should establish more clearly what its distillation-specific comparison and mechanistic analysis add beyond these existing results.

  2. The evidence for the distillation-specific interpretation is incomplete. A larger performance change from batching than from adding KD establishes a comparison within the tested configurations, but does not explain whether teacher supervision becomes redundant once the contrastive objective receives suitable negatives. Small KD gains could also reflect teacher quality, loss scaling, or tuning. The work does not report teacher performance under the same evaluation protocol or sufficiently specify KD coefficients and their selection. Section 3.5 introduces gradient-alignment and relative-magnitude diagnostics, but their measured values are absent. Moreover, its illustrative KL result does not cover the four evaluated KD objectives. These measurements are needed to connect the proposed explanation to the observed marginal gains, rather than leaving the central result primarily as an improvement to the contrastive baseline.

  3. The sampler comparison leaves important controls unresolved. The abstract acknowledges different handling of per-dataset remainders, but actual sample exposure, dataset weighting, and update counts are not quantified. Consequently, the comparison does not isolate negative-source composition. A matched-exposure experiment is needed to strengthen the interpretation. The sampling baselines are also incomplete: VLM2Vec-V2 finds that intermediate same-source sub-batches outperform both extremes on image tasks. Including such a condition would help establish whether the reported benefit reflects complete dataset homogeneity or a more general trade-off in negative composition. Distinguishing same-dataset slots from unique, informative negatives is also important when labels or answers repeat.

  4. The gradient analysis provides suggestive rather than sufficiently general mechanistic evidence. The empirical probe covers one student, one task family, and one seed, and measures query-embedding gradients with candidate representations fixed. This does not establish the corresponding behavior of shared model-parameter updates. The decomposition also passes through a smaller restricted batch, and its component shares are path-dependent descriptive quantities rather than independent causal effects. The population result itself permits settings with no within-dataset ranking shift. Although these limitations are acknowledged, validation using actual parameter gradients or controlled model updates, together with evidence from VQA and additional seeds, is needed to substantiate the proposed explanation.

Questions for authors

Please refer to the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.