acceptodds
Withdrawn from ICLR 2027

Generalizable Bilevel Knowledge Distillation from Vision Foundation Models for 3D Medical Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generalizable 3D medical image segmentation is increasingly vital for clinical deployment, given cross-site distribution shifts and restricted access to target-site data. However, cross-site distribution shifts and restricted access to target-site data during training make generalization to unseen sites challenging. Vision foundation models (VFMs) offer transferable segmentation priors, but their computational demands limit direct deployment. Existing distillation approaches transfer these priors to lightweight students through teacher adaptation (TA) and student distillation (SD). However, optimizing these tasks primarily for source-domain fitting does not explicitly align knowledge transfer with the student’s cross-site generalization. We propose generalizable bilevel knowledge distillation (BiLD), a framework that couples lower-level TA with upper-level SD. A dynamic response strategy incorporates the response of teacher adaptation into student updates using a Gauss–Newton approximation and conjugate-gradient iterations. To support this coupled optimization without target-site data, we introduce a pseudo-site generator that constructs diverse appearance variations from source-domain volumes through acquisition-motivated perturbations while preserving anatomical labels. These pseudo-sites expose the upper-level distillation objective to simulated distribution shifts, encouraging knowledge transfer beyond source-domain matching.Experiments on brain and cardiac segmentation across five external datasets demonstrate improved zero-shot cross-site generalization, with mean out-of-domain Dice exceeding the strongest baseline in each task by 3.23% and 5.99%, respectively. Two target-domain fine-tuning experiments further utperforms the strongest baseline in each task by 2.25% and 3.37%in Dice for brain and cardiac, respectively. Code will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Bi-level optimization, Generalization-oriented distillation, 3D Medical segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_GbMmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a method for cross-site generalization in medical image segmentation. It proposes to jointly do teacher adaptation to the task, where the teacher is a vision foundation model, and knowledge distillation to a smaller student model. The student model is exposed to augmented pseudo-site data allowing better cross-site generalization. The claim of the paper is to follow a novel bilevel optimization approach coupling the optimization of the teacher and students for better performance.

Critical strengths

Domain generalization is an important but much addressed problem in medical imaging. I have not identified a clear strength due to the main problems below.

Critical weaknesses

Firstly, the main weakness of the paper is its methodological soundness. The positioning of the paper is around the bilevel optimization and that is the main claim of methodological novelty. The bilevel optimization process assumes an outer level loss LSDL_{SD} optimized with respect to θ\theta with a dependence on the optimal value ω⋆(θ)\omega^\star(\theta) of the parameters of another model w.r.t. an inner-level loss LTA(θ,ω)L_{TA}(\theta,\omega). LTAL_{TA} is optimized w.r.t. ω\omega, giving ω⋆(θ)\omega^\star(\theta). The gradient of the outer-level introduces a dependence to ∂ω⋆/∂θ\partial\omega^\star/\partial\theta, and bilevel optimization addresses this problem.

The main issue is that LTAL_{TA} used in the present paper, as defined in Eq. 9, does not depend on θ\theta. As far as I can tell, this is not a typo or a notational shortcut. Therefore I could not understand the point of the whole bilevel optimization, since the two problems are decoupled. The paper even includes the following sentence: "This source-supervised term is independent of the student parameters θ; it does not, by itself, define the differentiable TA–SD coupling required by the response formulation in Sec. 2.3.". I could not assert from the rest of the paper which part of the formulation defines the coupling required by the response formulation.

The paper makes no anonymous github available at this stage, and therefore it is not possible to directly assert this point by checking the codebase.

Secondly, the novelty is lacking: the paper uses foundation models, adaptation, knowledge distillation, in ways that are fairly standard.

Third, given the large body of literature on domain generalization, the experimental section should include relevant baselines. For a segmentation paper at ICLR, I would also expect a straightforward nnU-Net baseline.

Questions for authors

Depending on the rating of other reviewers, clarifications with respect to the points mentioned above would be welcome.

Reviewer_5CbyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes BiLD, a framework for distilling vision foundation models into lightweight 3D UNet student for cross-domain 3D medical image segmentation without access to target domain. BiLD pairs a teacher-adaptation (TA) loop on source data with a student-distillation (SD) loop on simulated pseudo-site data. To couple the two loops, the authors propose a "dynamic response strategy" that approximates the implicit hypergradient ∇θF=∂θLSD−C⊤H−1∇ωLSD\nabla_\theta F = \partial_\theta \mathcal{L}_{SD} - C^\top H^{-1} \nabla_\omega \mathcal{L}_{SD} using Gauss-Newton and conjugate gradient iterations. Experiments on brain MRI and cardiac CT show improvement in Dice and HD95 over traditional 3D segmentation networks, 3D SAM adapters, etc.

Critical strengths
  1. The problem is relevant, particularly distilling heavy 2D slice-wise foundation models into fast 3D UNet that generalizes across clinical sites without target domain data addresses a genuine bottleneck in clinical medical deployment.
  2. The evaluation across Brain MRI and Cardiac CT under both zero-shot cross-domain transfer and target-domain fine-tuning under seven different VFMs is quite rigorous and extensive.
  3. I liked the simple nature of pseudo-site augmentation which provides a clean way to expose the student to domain shifts while feeding clean source views to the teacher.
Critical weaknesses
  1. In the bilevel formulation in Eq. 1 and Gauss-Newton/CG implicit hypergradient: ∇θF=∂θLSD+(∂ω∗∂θ)⊤∇ωLSD,where ∂ω∗∂θ=−H−1C=−(∂2LTA∂ω2)−1∂2LTA∂θ∂ω\nabla_\theta F = \partial_\theta \mathcal{L}_{SD} + \left(\frac{\partial \omega^*}{\partial \theta}\right)^\top \nabla_\omega \mathcal{L}_{SD}, \quad \text{where } \frac{\partial \omega^*}{\partial \theta} = -H^{-1} C = -\left(\frac{\partial^2 \mathcal{L}_{TA}}{\partial \omega^2}\right)^{-1} \frac{\partial^2 \mathcal{L}_{TA}}{\partial \theta \partial \omega} however, if we pick the terms from Section 2.5 (Eq. 8 and 9), we get: a. LTA(ω;Dsrc)=1∣Dsrc∣∑LDice(qω(x),y)\mathcal{L}_{TA}(\omega; \mathcal{D}_{src}) = \frac{1}{\vert{}\mathcal{D}_{src}\vert{}} \sum \mathcal{L}_{Dice}(q_\omega(x), y) depends only on the teacher decoder ω\omega and ground-truth labels yy. Because θ\theta does not appear in LTA\mathcal{L}_{TA}, the optimal teacher ω∗\omega^* is independent of the student θ\theta, so C=0C = \mathbf{0} and ∂ω∗∂θ=0\frac{\partial \omega^*}{\partial \theta} = \mathbf{0}. b. Lcomp=αLDice(p,y)+βLCE(p,y)+γLCon(ES,ET)\mathcal{L}_{comp} = \alpha \mathcal{L}_{Dice}(p, y) + \beta \mathcal{L}_{CE}(p, y) + \gamma \mathcal{L}_{Con}(E_S, E_T), where line 236 explicitly defines ETE_T as "frozen teacher encoder features from its paired source view". Because the SAM image encoder is frozen and the trainable 3D mask decoder ω\omega is not used in Lcomp\mathcal{L}_{comp}, ∇ωLSD=0\nabla_\omega \mathcal{L}_{SD} = \mathbf{0}. This implies the trainable teacher decoder ω\omega has zero influence on the student SθS_\theta!
  2. Because of this argument, I wonder how in ablations presented in Table 2, replacing alternate optimization (+ATA+PSG) with a second-order hypergradient correction (+BiLD - whose gradients come out to be zero), cause a jump of +3.2%+3.2\%.
  3. +DA and +AT is in Fig. 8 is undefined.
  4. LCon(ES,ET)\mathcal{L}_{Con}(E_S, E_T) is not defined, can you write the explicit form of the contrastive distillation loss?
  5. Is the CAT08 dataset used in Tables 1, 3, 5 same as Cardia32 in Fig 1b, 4, 13?
Questions for authors
  1. Please address the concerns mentioned in Weaknesses above.
  2. Please provide the explicit equation for the contrastive distillation loss LCon(ES,ET)\mathcal{L}_{Con}(E_S, E_T) in Eq. (8). How are positive and negative pairs sampled, and how are the 2D slice-wise SAM encoder features ETE_T aligned spatially and channel-wise with the 3D U-Net student features ESE_S?
  3. What is the performance of the baseline Supervised 3D U-Net when trained directly with your Pseudo-Site Generator (U-Net + PSG, without any SAM teacher)? How does BiLD compare against established single-source medical domain generalization methods such as Ouyang et al. (2023) (CSD) or BigAug?
Reviewer_wLqPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents BiLD to improve cross-site generalization in 3D medical image segmentation when target-site training data are unavailable. It combines vision-foundation-model teacher adaptation, student distillation, and pseudo-site augmentation, using a dynamic-response strategy to coordinate teacher and student optimization. The authors report promising results on external brain and cardiac datasets. However, the stated equations raise questions about whether the proposed dynamic response is actually implemented and whether it independently improves performance.

Critical strengths
  1. The problem is practically important: cross-site generalization remains a major challenge for deploying medical image segmentation models.
  2. The combination of pseudo-site augmentation and teacher-feature distillation is a potentially useful direction for domain generalization.
  3. The authors report strong results on external datasets, suggesting that the overall training setup may have practical value.
Critical weaknesses
  1. The core bilevel mechanism is inconsistent with the stated loss definitions. Under Eqs. (8)–(9), the required teacher–student parameter dependencies are absent, so the dynamic-response term may reduce to zero.
  2. The ablations do not sufficiently isolate the contribution of dynamic response. The reported gains cannot be clearly attributed to pseudo-site augmentation, feature distillation, or the bilevel response strategy.
  3. Important implementation details are missing, including pseudo-site perturbation parameters, CG settings, and the definition of the actual curvature operator, limiting reproducibility and verification.
Questions for authors
  1. How does the distillation loss in Eq. (8) depend on the trainable teacher decoder parameters ω? If the teacher features come only from the frozen SAM encoder, why is ∇ω L_SD nonzero?
  2. The teacher adaptation loss in Eq. (9) does not depend on the student parameters θ. How is a nonzero mixed derivative C obtained in the implementation? Please provide the exact coupled objective or computation graph.
  3. Could the authors directly compare ordinary student-gradient updates with updates using the response correction, and report the magnitude or norm of the response term to show that it affects training?
  4. For the Dice lower-level loss, which curvature approximation and regularization are used? What are the CG iteration count and stopping criterion?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.