- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
-
The paper argues that in vision–language embedding distillation the in-batch negative set matters more than the auxiliary KD loss, crossing five objectives with mixed-task and task-consistent samplers on two 0.5B students, two MMEB task families and three seeds: task-consistent batching lifts Precision@1 by 1.54–3.72 points, while under task-consistent batching adding the tested KD losses changes it by only −0.60 to +0.57 relative to contrastive-only training (Tables 1–2).
-
Three mechanism accounts are offered: a density-ratio argument at the population level, an exact gradient-gap decomposition at the finite-batch level, and a single-seed rebatching probe.
-
The design is systematic, every headline range checks against the tables, and the association is a useful corrective for how KD objectives are compared.
-
The paper has not yet done the controlled, documented and mechanism-tied work its own stronger wording calls for: the abstract concedes the comparison "does not isolate the causal effect", the recipe is unreported, the appendix's no-teacher baselines move by 9–46 points, and the probe is not tied to the per-dataset outcomes.
Critical strengths
A systematic factorial design whose numbers all reproduce from the tables.
- The design is a full factorial reported with variance and per-dataset detail, and its arithmetic holds up. Specifically, two students × two task families × five objectives × three samplers × three seeds, every main-table cell with a standard deviation and a per-dataset breakdown; every range quoted in the abstract and Sec. 4.2 (1.54–3.72, 2.03–3.04, −0.60 to +0.57, 2.80–3.72, 1.46–2.76) re-derives from Tables 1–2, and the Figure 1 bar heights match.
A practically important message, stated with unusual care about what each result does not show.
- The message matters for how KD objectives are compared, and the paper states its limits plainly. Specifically, if a sampler choice moves a student by 1.54–3.72 points while, under task-consistent batching, adding any of four KD losses moves it by at most 0.60, then sampler differences can confound the attribution of gains to a KD objective; and the paper says Prop. 1 "does not imply that homogeneous training is better" and that the probe is "descriptive rather than causal evidence".
Critical weaknesses
The comparison the paper rests on is, by its own account, not controlled, the recipe needed to judge it is unreported, and the appendix's no-teacher baselines are inconsistent with the setting it says was fixed.
-
The body claims a causal result the abstract concedes the comparison does not isolate, and the recipe needed to judge the KD comparison is absent. Specifically, the abstract says the samplers "handle per-dataset remainders differently" so the comparison "does not isolate the causal effect", yet Contribution 1 says negative composition "outweighs" distillation and Sec. 3.1 says the contrast "isolates task compatibility"; Sec. 4.1 gives no learning rate, step count, temperature or KD weight.
-
The appendix's no-teacher baselines contradict its own claim that only the teacher changed. Specifically, Appendix B.2 says only the teacher changed, yet the teacher-free contrastive-only row falls from 70.96 to 61.94 on CLS and from 53.79 to 7.50 on VQA between Tables 4–5 and Tables 8–9, with no explanation, so the second-teacher check is not clean.
The mechanism evidence is a single-seed embedding-space probe not yet tied to the outcomes it is meant to explain, and a promised measurement is missing.
-
The probe is narrow, and the submission does not establish a matched quantitative link between it and the reported per-dataset gains. Specifically, it covers one student, task family and seed and measures embedding-space descent rather than a model update; the single-setting Dfull values and the objective-averaged Table 3 gains are not the same measurement (N24News −3.18 against +0.19, Hateful Memes −0.08 against −1.31 on their face), and the largest gains are out of distribution, beyond the probe's reach.
-
A promised measurement is absent and a large collapse is undiagnosed. Specifically, Sec. 3.5 says "we report" the Eq. (21) statistics, yet none appear; and the 25.11–31.61-point FastVLM VQA hard-batch collapse is labelled, not explained.
Questions for authors
See weaknesses.