acceptodds
Withdrawn from ICLR 2027

DARK: Diagonal-Anchored Repulsive Knowledge Distillation for Vision-Language Models under Extreme Compression

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models provide useful representations for fetal ultrasound, but their computational cost makes deployment on portable devices difficult. Knowledge distillation offers a route to smaller models, although continued imitation of a large teacher may constrain a much smaller student. We propose Diagonal-Anchored Repulsive Knowledge Distillation (DARK), which preserves supervision on matched image–caption pairs while gradually changing supervision on other pairs from imitation to repulsion. This lets the student first learn the teacher's relationships and then depart from them without abandoning matched-pair alignment. We use DARK to compress FetalCLIP into MobileFetalCLIP, whose visual encoder has approximately 26× fewer parameters and runs in 1.6 ms on an iPhone 16 Pro. Compared with static distillation, DARK improves brain sub-plane macro-F1 from 0.701 to 0.775. On an external cohort of 403 images from five African countries, anatomical macro-F1 rises from 0.770 to 0.895, with gains in all four classes. Ablations show that the sign change is essential: fixed decoupled weights, as in decoupled KD, and stopping imitation at zero both fall short. Linear probes on the compact model's frozen features retain 97–98% of the teacher's downstream performance. These results support selective departure from teacher imitation as a way to learn compact fetal-ultrasound representations for both zero-shot recognition and supervised downstream use.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
knowledge distillation, vision-language models, contrastive learning, model compression, fetal ultrasound, medical imaging, on-device inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bnseOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to compress the fetal ultrasound vision-language model FetalCLIP into a smaller model, MobileFetalCLIP. The main idea is to have the student mimic the teacher's image-text similarity judgments in the early stage of training, then deliberately move away from the teacher's predictions for mismatched image-text pairs in the later stage, while continuing to strengthen the alignment of matched pairs.

Critical strengths
  • The problem is well defined, and the proposed method is simple. The changes are limited to the training objective, without introducing additional modules at inference time, which makes the approach well suited to model compression.
  • The authors clearly distinguish HC18 agreement from actual gestational age error, as well as variation across training seeds from uncertainty due to patient sampling. They also acknowledge that the same gains were not observed with a larger student model.
Critical weaknesses
  • The paper is largely incremental, with relatively modest improvements over prior work.
  • The Composite Score is heavily influenced by the HC18 proxy metric, so it does not necessarily reflect the model's overall capabilities.
  • The visualizations are poorly presented and look rather ad hoc.
  • The key experiment supporting the necessity of the negative phase is based on a single run, which is not enough to draw a robust conclusion.
  • The authors should consider whether a "DKD-style" baseline can really serve as a substitute for the original DKD method. The current claims about the comparison with related methods seem somewhat too strong.
Questions for authors

see weakness

Reviewer_yg1fOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

DARK decomposes CLIP-style image–text distillation into matched-pair (diagonal) and non-target (off-diagonal) terms. It maintains positive supervision on matched pairs while linearly changing the non-target coefficient from attraction to repulsion. The method is used to distill FetalCLIP into MobileFetalCLIP, reducing the visual encoder by approximately 26×. The paper reports improvements over static KD on fetal-ultrasound benchmarks, including brain sub-plane recognition, HC18 validity, and an external cohort from five African countries, alongside reduced mobile inference latency.

Critical strengths
  1. Compressing domain-specific fetal-ultrasound models for resource-constrained devices is a meaningful application. The reductions in visual-encoder size, computation, and mobile inference latency are practically valuable.
  2. DARK improves brain sub-plane F1 and HC18 validity over static KD across four matched seeds. The external cohort evaluation also shows substantial gains across five countries, providing useful evidence beyond the benchmarks used for schedule selection.
Critical weaknesses
  1. Limited methodological novelty and insufficient evidence for the proposed mechanism. Separating target and non-target supervision is closely related to DKD (Zhao et al., 2022), although the loss decompositions differ. DARK's main methodological distinction is the sign-changing non-target coefficient, but the experiments do not establish that beneficial departure from teacher relationships explains its gains. From Equation (5), at β = −0.8, the gradient with respect to a non-target student logit is (0.2 + 1.8p_ii)q_ij + 0.8p_ij > 0. Thus, the objective directly suppresses non-target logits, with additional suppression proportional to teacher probabilities. Teacher-weighted hard-negative suppression is therefore a plausible alternative explanation that the existing ablations do not isolate. Furthermore, Table S12 compares fixed positive weights within DARK's own decomposition rather than a faithful adaptation of DKD's original objective. Without appropriate controls, the distinct contribution of the proposed mechanism over existing distillation and negative-pair weighting strategies remains unconvincing.
  2. Experimental validity concerns and insufficient support for generality. Appendix E.1 acknowledges that the default schedule was selected on Planes DB and HC18 without a separate validation split, yet results on these same benchmarks support the main claims. Repeating the selected schedule across seeds does not eliminate this selection bias. Although the external cohort provides independent evidence, it evaluates only coarse anatomical recognition and does not validate the brain sub-plane or HC18 improvements. Moreover, the confidence-penalty baseline attributed to Pereyra et al. (2017) is incorrectly implemented: training/loss.py smooths teacher targets toward uniform instead of penalizing student prediction entropy, while its configuration additionally enables KD-weight decay from 1 to 0.1. These differences are not disclosed, and the resulting comparison cannot support the claimed distinction from confidence regularization. Finally, Table S13 shows reduced Composite Scores for the larger S2 (−0.073) and ViT-B (−0.013) students. Together with evaluation using only one teacher and domain, these findings leave both the robustness and broader applicability of DARK insufficiently established.
Questions for authors
  1. How can the authors distinguish beneficial departure from teacher relationships from teacher-weighted hard-negative suppression? In particular, can DARK be compared with uniform non-target repulsion and alternative hard-negative weighting strategies under matched schedules, optimization budgets, and seeds? What evidence establishes that changing the teacher's relational geometry is responsible for the improvements?
  2. Can the authors report a faithful adaptation of the original DKD objective, including its target-versus-rest and normalized non-target KL terms, rather than relying solely on fixed positive weights within DARK's decomposition? How does DARK compare with other relevant approaches such as DIST or logit standardization under comparable conditions?
  3. Can the authors select the schedule using a separate validation set and report results on untouched test sets, particularly for brain sub-plane recognition and HC18? This would help establish whether the gains persist without benchmark-dependent schedule selection.
  4. Why does the implementation smooth teacher probabilities instead of penalizing student prediction entropy as in Pereyra et al. (2017)? Why is KD-weight decay additionally enabled for this baseline? Can the authors correct the implementation and provide a comparison under matched training conditions?
Reviewer_XcWcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DARK, a knowledge distillation strategy for VLM under extreme compression. DARK separates supervision on matched image–text pairs from supervision on non-matched pairs. It retains positive matched-pair supervision while gradually changing the coefficient of the non-target distillation term from positive to negative, which helps the student to first learn from and subsequently depart from the teacher’s relationships. The proposed method improves brain sub-plane recognition, head-circumference–age consistency, and external anatomical classification over static distillation.

Critical strengths
  • The paper studies a relevant and interesting problem.


  • The proposed method is reasonable. Early imitation can help the student establish useful image–text relationships before it is encouraged to depart from the teacher’s non-target structure. This is more compelling than applying repulsion from the beginning.

  • The method is simple and clearly formulated.
The central modification is a scheduled coefficient on the off-diagonal distillation term, without introducing additional complex modules.

Critical weaknesses
  • The rationale for repelling semantically meaningful non-matched relationships remains incomplete.
As the authors acknowledged, off-diagonal pairs are not necessarily incorrect: different images and captions may describe the same anatomical plane or related structures. High teacher probabilities for such pairs may therefore encode useful semantic information. Although the ablations support the aggregate benefit of scheduled repulsion, they do not establish which relationships should be discarded and which should be preserved. Analyses separating semantically compatible and incompatible pairs would help clarify why teacher-weighted repulsion is beneficial.

  • 
Captions are constructed from anatomical labels, templates, and image-specific metadata. This raises the possibility that a batch contains semantically similar or repeated descriptions. Applying the repulsive term to all off-diagonal entries could introduce additional bias against valid alternative image–text matches.

  • The robustness of the coefficient schedule needs to be further justified. The selection/effectiveness of \beta may vary across different settings (e.g., temperature, batch composition). Controlled sensitivity analyses would strengthen the method’s practical applicability. 


  • The CLIP objective already encourages matched-pair alignment. The independent contribution of teacher-confidence-weighted diagonal supervision is not fully isolated. Holding the off-diagonal schedule fixed while comparing no additional diagonal term/a matched constant positive-pair weight or teacher-confidence weighting might be useful. 


  • The proposed explanation for the student-capacity effect remains unclear. The authors explain that larger students may push non-target probabilities closer to their lower bound, leading to excessive repulsion, whereas limited capacity may help stabilize smaller students. This needs more experimental results to support.

Questions for authors

Please see the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.