acceptodds
Under review as a conference paper at ICLR 2027

Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation

Abstract

Despite significant advances in natural domain vision, biomedical semantic segmentation remains challenged by severe label scarcity, low signal-to-noise ratios, and fine anatomical structures. We introduce ViTC-UNet, a framework that bridges this gap by conditioning a UNet pixel decoder on frozen Vision Transformer (ViT) representations via a multi-stage, two-way attention decoder. Semantic identity is decoupled from output geometry using target-specific structure tokens, implemented either as learnable class vectors or embeddings from a language model. This design unifies the global representational capacity of large-scale ViTs with the spatial inductive biases and high-resolution decoding precision of UNets, bypassing computationally intensive and data-hungry Transformer fine-tuning. ViTC-UNet consistently outperforms baselines across CT and MRI modalities in both closed- and open-vocabulary settings, as well as out of distribution, demonstrating that structure-conditioned UNet decoding provides a sample-efficient, highly adaptable pathway for biomedical foundation models. Code will be released upon publication.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
semantic segmentation, domain adaptation, biomedical imaging, computer vision, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_8oLgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses a medical segmentation model architecture ViTC-UNet, that conditiona a CNN based UNEt with a frozen ViT backbone like DINO and SAM. The authors integrated a SAM-style 2 way attention blocks in the forzen ViT embeddings with prior tokens. The contributions are first the work combines ViT with UNet withour training or fine-tuning ViT. Second, a single channel, token conditioned output supporting open-vocabulary segmentation. Public datasets are evaluated with a diverse sources of baselines.

Critical strengths
  1. This work is easy to follow and proved effective on the architecture combinations. Introducing a SAM encoder or a DINO-pretrained ViT to support open-vocabulary segmentation is smart.
  2. The decoder changes and conditioning on a structure token that supports zero-shot transfer are good for generalizability.
Critical weaknesses
  1. This architectural combination in general are well studied in the domain, like combining SAM and UNet model architecture, or combine a text encoder and segmentation model, to support multi-modality input, prompt input are well popular in the community.
  2. Baseline comparison lacks many exisitng works, and comparing to raw baseline nnUNet shall not convincing for the innovation.
Questions for authors
  1. The baseline comparison with nnUNET setup is not clear, 2D versus 3D, Dice vs IoU, splits. Also other existing SAM-style medical segmentation methods.
  2. Some baselines are 2D, some are 3D, how is the final score evaluated? 3D metrics, case level?
  3. How to eval the effectiveness of the text encoder? or prompt encoder?
Reviewer_gi5zOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ViTC-UNet, a model for semantic segmentation of medical images that combines the global representation capability of large-scale visual transformers with the spatial induction bias of U-Net. Its core architecture includes frozen visual encoder, conditional decoder, and pixel decoder. The experiment covers CT and MRI modalities, involving closed vocabulary segmentation, cross dataset evaluation, class incremental learning, and attention mechanism ablation.

Critical strengths
  1. Transferring frozen visual basic models to medical image segmentation, balancing the representation ability of large-scale visual models with UNet spatial induction bias, has valuable direction.
  2. The structure of frozen backbone, conditional decoding, and pixel decoding is clear.
Critical weaknesses
  1. Table 1 shows the mean and standard deviation of five runs, but SAT involves direct testing with a standard deviation of 0.20, which requires further explanation; The result only retains two decimal places, and the difference in methods is not significant enough.
  2. Although the core architecture is technically reasonable, the substantial innovation compared to existing conditional segmentation methods has not been fully elucidated.
Questions for authors
  1. Please explain whether TransUNet, Swin UNETR, and other baselines use consistent training strategies and statistical methods, and provide convergence and validation performance; Can key baselines be re evaluated under a unified protocol and the results reported to four decimal places?
  2. Please clarify the substantial innovation of this article compared to existing conditional segmentation methods, and explain why DINOv2 was chosen instead of the BiomedCLIP image encoder pre trained on medical data.
Reviewer_C7PEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ViTC-UNet for 2D (slice-wise) CT/MRI segmentation. A frozen natural-image ViT (DINOv2-S or SAM 2-S) produces patch embeddings. These pass through a stack of SAM-style two-way attention blocks together with a "structure token" that names the target: either a learnable class vector (closed vocabulary) or concatenated BiomedCLIP image+text embeddings (open vocabulary). The output of the n-th attention block is concatenated into the n-th skip connection of a 2D nnU-Net (DynUNet), which also receives the raw image and predicts a single-channel binary mask for the queried structure. Multi-class outputs come from iterating over prompts (Sec. 3). Experiments on 15 datasets from the BiomedParse compilation report about 0.90 mean foreground mIoU in the closed-vocabulary setting (Fig. 3a, Tables S2–S3), large wins over TransUNet, Swin UNETR, SAM 3 and a ViT-UNet hybrid, and 0.88–0.94 mIoU zero-shot on 13 unseen datasets after training only on TotalSeg + AMOS (Table 1).

Critical strengths
  1. Sensible, practical design idea. Decoupling class identity from output channels with a token-conditioned single-channel UNet decoder, on top of a frozen general-purpose ViT, is a reasonable recipe. It supports label-space extension without changing the head.

  2. Broad dataset coverage. 15 CT/MRI datasets spanning organs, vessels, airways, tumours and lesions, with five runs per configuration.

  3. A reasonably systematic decoder ablation (Fig. S1, Tables S4–S5): conditioning type (addition, one-way, two-way) and where features are injected (last-deep, last-shallow, multi-scale), on two backbones.

Critical weaknesses
  1. The headline results are implausible
  • Zero-shot beats in-domain. Trained only on TotalSeg (CT) + AMOS (MRI), the model reaches 0.92–0.94 mIoU zero-shot on BraTS (brain tumour), ISLES (stroke lesion), QIN (prostate lesion), Spider (spine) and ProstateADC/T2 (Table 1). None of these anatomies or pathologies are in the training data. On ProstateADC/T2 (0.93 vs 0.73), QIN-Lesion (0.94 vs 0.80) and Spider (0.92 vs 0.84), the zero-shot results exceed the same architecture trained on the target dataset itself (Table S2). Zero-shot segmentation of a never-seen lesion type should not beat in-domain supervised training.
  • Likely explanations to rule out: (a) the metric counts empty-prediction/empty-ground-truth slices or volumes as IoU = 1. The class-sampling protocol queries every class on every slice (Sec. 4), so most queries have empty targets. (b) The "BiomedCLIP image embedding" in the open-vocabulary token (Sec. 4, "concatenating BiomedCLIP image and text embeddings") is computed from something derived from the ground truth, e.g. a masked or cropped target region, which would leak the answer. (c) Overlap between the BiomedParse "holdout" set and the training data. Please give the exact metric definition, including how empty targets are scored, and specify exactly what image the BiomedCLIP image embedding is computed from.
  1. Baselines appear to be broken, so the comparison is not informative
  • Many baselines score at failure level: Swin UNETR 0.01 on AirwayTree, ISLES and QIN and 0.04 on Abdomen1K; TransUNet 0.01 on HaN-Seg and ISLES; SAM 3 ≤ 0.17 on 14 of 15 datasets; the ViT-UNet hybrid 0.01 on ISLES and QIN (Table S3). VoxelPrompt, trained on the same TotalSeg + AMOS data, scores 0.01–0.17 (Table 1). SAT-Pro was trained on many of these public datasets (including BraTS and ACDC), so it is neither zero-shot nor plausibly at 0.02–0.32.
  1. The comparison with nnU-Net is not valid
  • nnU-Net numbers are copied from the literature (Table S1, Fig. 3 caption). Those come from different splits, mostly 3D configurations, and in many cases Dice rather than IoU.
  1. The "Unconditioned" ablation does not control for capacity
  • The pixel decoder outputs a single channel regardless of the number of classes (Sec. 3.3). Removing the structure token therefore makes multi-class segmentation ill-posed: the network has no way of knowing which class to output, so near-chance results (0.04 on Abdomen1K, mean 0.24; Table S4) are expected. The claim that "performance gains stem from dynamic target conditioning rather than added decoder parameters" (Sec. 5) does not follow.
  1. The theory in Appendix A.1 is incorrect and does not support the design
  • Lemma 18 is false as stated. If f = f_n ∘ … ∘ f_1 is a homeomorphism, the factors need not be (e.g. f_1 injective but not surjective into a larger space, followed by a left inverse). The proof assumes "without loss of generality" that every f_i maps X → X, which is not WLOG, especially for ViT layers and decoder blocks with different input/output spaces.
  • Corollary 17 rests on a vacuous assumption ("Assuming SAM2 is an optimal segmentation function"). Theorem 16 proves injectivity only on a quotient space Ω/∼ of images, but the corollary concludes the encoder is a homeomorphism on images. The proof of Theorem 16 also uses that Ψ is finite, in which case continuity and homeomorphism statements are trivial.
Questions for authors
  1. Exactly how is mIoU computed? Per volume or per slice? How are queries whose structure is absent (empty ground truth) scored, and are they included?
  2. In the open-vocabulary setting, what image is the BiomedCLIP image embedding computed from? Could it encode the ground-truth region?
  3. How can zero-shot performance on unseen lesions and anatomies (BraTS, ISLES, QIN, Prostate, Spider) match or exceed in-domain supervised training? Please provide per-structure results and failure cases.
  4. What training budgets, hyperparameters and preprocessing were used for TransUNet, Swin UNETR, SAM 3, SAT and VoxelPrompt? Can you show at least one baseline reproducing its published number on a standard benchmark in your pipeline?
  5. Can you retrain nnU-Net (2D and 3D) on your splits and report the same metric?
  6. Can you add a multi-channel "unconditioned" control with the same frozen ViT, attention stack and multi-scale injection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.