acceptodds
Withdrawn from ICLR 2027

Uncovering Modality Discrepancy: Rethinking the Validation of Foundation Models for General-Purpose 3D Medical Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Foundation models have emerged as a transformative paradigm in 3D medical imaging, with the promise of unified quantitative analysis across diverse targets and imaging modalities. However, these models are predominantly developed and evaluated on datasets largely concentrated around a limited set of imaging modalities and anatomical regions. To provide an evaluation of out-of-the-box robustness of these foundation models, we curate a large-scale dataset comprising 490 whole-body PET/CT and 464 whole-body PET/MRI scans (approximately 675k 2D images and 12k 3D annotations) and conduct a thorough and comprehensive evaluation of representative 3D segmentation foundation models. Our analysis reveals a substantial gap between benchmark-reported performance and real-world generalization, with marked degradation on previously unseen data distributions and particularly severe failures on functional imaging modalities. These findings suggest that current foundation models remain far from achieving true universality. We suggest a reconsideration of how universality is defined and validated, extending evaluation beyond regional structural benchmarks toward whole-body structural and functional imaging. Bridging this gap will be essential for translating foundation models from controlled evaluation settings to real-world medical practice.

Primary area:
datasets and benchmarks
Keywords:
Medical Image Segmentation, General-Purpose Segmentation, Vision Foundation Model

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_Bz1FOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper focus on benchmarking 3D segmentation foundation models. They curated 490 whole-body PET/CT and 464 PET/MRI scans and conducted a thorough benchmarking over representative 3D segmentation foundation models like SAT, nnInteractive, SAM-Med3D, VISTA3D, e.t.c. The results show that the existing foundation models are not performing well on this benchmark, and PET imaging are not well supported.

Critical strengths

This is a benchmarking paper and they are benchmarking over the popular topic of 3D medical segmentation foundation models, and revealed the limitations of current models. The topic is very important.

Critical weaknesses
  1. The benchmark is a valuable but relatively small (~1000 paired PET-CT, PET-MR scans) with only 13 common organs. The benchmark itself, although reveals limitation of current foundation models, is not impactful enough to be of broad interest to ICLR community.
  2. The details of the benchmark is not fully discussed. How the annotation is acquired? Did you use auto-segmentation first then human correction? or fully human annotated? The quality of the annotation is unknown. Second, those foundation models are trained on huge cohort of CT volumes, and CTs are usually much stable/ less domain variance than MRI, why do we see such a huge performance gap even in CT? Those foundation models have been trained/validated on almost all publically available CT datasets, what is so special about this dataset? The author didn't provide such explanation.
  3. I don't see the paper mentioning open sourcing the dataset.
Questions for authors
  1. What is the domain shift of your CT/MRI dataset compared with existing public datasets like totalsegmentator, amos e.t.c?
  2. in Table2, VISTA3D has a 0.9 dice score on Liver while others are low, but completely 0 in other organs. This is a very strange value for any evaluation, and seems like a bug to me.
Reviewer_YzA1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes UMD, a paired image benchmark for evaluating the cross-modal performance of 3D medical segmentation foundation models. The benchmark includes 490 sets of PET/CT examinations and 464 sets of PET/MRI examinations, providing segmentation annotations for 13 organs. The authors evaluated five published models: SAM-Med3D-turbo, SegVol, nnInteractive, VISTA3D, and SAT-Pro, using their respective point prompts, class IDs, or text prompts interfaces, and used nnU-Net with 10 training samples as a task-specific reference. The paper reports overall and organ-by-organ results, showing that several models have limited performance under this evaluation protocol.

Critical strengths
  1. Paired structural and functional images provide valuable evaluation resources. UMD paired PET/CT and PET/MRI datasets, allowing models to be tested on the same subject and target organ across different modalities. This design reduces interference from population and target organ differences compared to directly comparing CT and PET datasets from different sources and with varying patient compositions. Both cohorts contained hundreds of examinations, and the evaluation covered 13 organs, providing a substantial scale and anatomical scope for the research. Manual quality control was also included in the annotation process. The appendix explains that the authors first generated initial annotations using existing segmentation tools, which were then reviewed, corrected, and refined by radiologists using LIFEx.

  2. Organ-by-organ results reveal valuable model discrepancies and failure modes. The paper reports overall performance, organ-by-organ results, segmentation visualizations, and reference performance of task-specific models. These results help identify "which model underperformed, in which organ, and in which modality," and also show that cross-modal variations are not consistent across all organs. For example, in the PET/CT cohort, nnInteractive's pancreas Dice decreased from 0.4007 to 0.0844, but the brain result increased from 0.7940 to 0.8796. Using a small number of labeled nnU-Net references is also meaningful. Provided its training and testing splits are well explained, these results can help determine whether these images contain segmentation information that can be learned by task-specific models.

Critical weaknesses
  1. The implementation verification of key anomalous results, as well as the disclosure of labels and statistical procedures, are insufficient to support a reliable explanation. The most crucial result to verify is the VISTA3D result: it achieves a Dice of 0.9186 on CT of the liver, but almost all of the other twelve organs are zero, with an overall CT mean of only 0.0706. This pattern should prompt authors to examine organ class IDs, output channels, preprocessing, and empty prediction processing, and demonstrate the verification results of the same code on the model's official support data.

The authors have stated that they used the officially released model and default process, and the appendix explains why VISTA3D uses automatic class-ID branches. This information is commendable, but insufficient to allow readers to rule out the influence of actual class mapping, output interpretation, or input processing. Figure 3 provides an in-domain example of SegVol, but it doesn't fully address key anomalies in models like VISTA3D and SAT. This doesn't necessarily mean the authors' code is flawed; the problem is that current evidence cannot reliably distinguish between true generalization failure and evaluation implementation factors.

There are also gaps in data and statistics. Although the annotations were revised by doctors, how the PET ground truth is formed, whether it's mapped from structural images, and how registration quality is checked are not fully explained. The selection of the 10 training samples for nnU-Net, whether they are separated from the test by independent patients, and the specific definitions of the values ​​in table brackets and significance tests also need to be supplemented. Since this information directly affects the credibility of the core results, revisions may involve substantive verification and re-evaluation, and cannot be solved simply by adding a few sentences of explanation.

  1. Some conclusions regarding modality differences and failure causes exceed the scope supported by the comparative design and actual data. Table 1 clearly states that the literature scores differ from UMD in category, cohort, annotation, prompts, and averaging methods, making direct numerical comparisons impossible. However, the main text uses these score differences to claim that existing validation exhibits "systematic overestimation." Differences in scores across different evaluation tasks can indicate out-of-distribution performance issues, but cannot be directly used as evidence that the original evaluation overestimated the performance of the same target.

Modality changes do not show a uniform downward trend across all models. SAM-Med3D-turbo's CT and PET scores in the PET/CT cohort are 0.404 and 0.410, respectively; SegVol's MRI and PET scores in the PET/MRI cohort are 0.134 and 0.148, respectively. Therefore, the paper should describe the differences model-by-model, avoiding generalizing the significant decrease in nnInteractive to a common result for all models. For models already close to zero scores, a small decrease cannot be interpreted as cross-modal stability.

Furthermore, while the paired design reduces patient and organ variability, it does not eliminate the effects of imaging resolution, organ boundary visibility, prompt information, and preprocessing. Comparing the nnU-Net trained with 10 supervised cases to the zero-shot foundation model also alters the task information acquired by the model. Therefore, the current results are insufficient to definitively attribute the failure to training distribution bias, or to conclude that the model only memorizes intensity patterns and lacks anatomical representation. A more reasonable conclusion that can be drawn from the paper is that these specific models have significant limitations under UMD and current inference protocols, requiring more targeted cross-modal validation.

Questions for authors
  1. Regarding implementation and anomalous results: Could you provide the class-IDs and output mappings for the thirteen organs in VISTA3D, key preprocessing steps, and verification results of VISTA3D and SAT using the same evaluation code on officially supported data? How do you explain the pattern where VISTA3D performs well on the liver but almost entirely zero on other organs?

  2. Regarding labeling and registration: Are the PET ground-truth annotations ​​independently labeled, or revised after mapping from structural images? How do the authors check for registration errors, handle organs with unclear boundaries in PET, and assess annotation consistency among physicians? Can a small number of paired images, annotations, and prediction results be provided for independent verification?

  3. Regarding the reference model and statistics: How were the 10 training samples for nnU-Net selected? Were training and testing divided by independent patients? How were duplicate examinations handled?

  4. Regarding the scope of conclusions: After avoiding direct comparison of heterogeneous literature scores and acknowledging the inconsistent modality differences between different models, which main conclusions still hold? Can the authors limit the conclusions to the system performance under the current data and inference protocol? If explanations for training bias or insufficient anatomical understanding are retained, what control results are needed to support this?

Reviewer_A8QvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces UMD, a benchmark containing 490 paired PET/CT examinations and 464 paired PET/MRI examinations with segmentation annotations for 13 organs. It evaluates five released 3D medical segmentation foundation models without fine-tuning, using their native point-, class-, or text-prompt interfaces. The paired design permits comparison of the same patient’s organs across structural and functional images. The authors report substantial failures for several evaluated systems and argue that validation of general-purpose medical segmentation should extend to functional modalities and broader anatomical coverage.

Critical strengths

The intra-subject paired dataset is a meaningful contribution. Evaluating identical organs in co-registered PET and structural images reduces anatomical and cohort differences that complicate comparisons between unrelated datasets. Its scale and coverage could make it a useful resource for studying modality-dependent segmentation behavior.

The organ-level evaluation across five released systems provides more useful diagnostic information than aggregate scores alone. It identifies substantial deficits for some model–modality combinations while also exposing exceptions. This could support a valuable characterization of current systems’ operating limits, provided that the inference and annotation protocols are sufficiently validated.

Critical weaknesses
  1. The evaluation pipeline is insufficiently documented and validated for a benchmark whose main evidence consists of severe model failures. Referring to official pipelines does not specify the exact checkpoints, preprocessing, semantic mappings, or prompt-generation procedure. The near-zero structural CT/MRI results for VISTA3D and SAT particularly require positive controls demonstrating that the same implementation works on an appropriate in-domain dataset. Annotation and registration quality are also insufficiently quantified, and no concrete dataset-access plan is stated. These omissions make it difficult to assess whether the reported failures are reproducible properties of the evaluated releases under the stated conditions.

  2. Several central interpretations exceed the evidence. Table 1 explicitly acknowledges that published scores and UMD scores are not directly comparable, yet the text interprets their numerical difference as systematic performance overestimation. Within-patient pairing supports a more controlled description of modality-dependent performance, but does not establish that failure results from memorization or lack of anatomical understanding. PET visibility and resolution, annotation construction, acquisition conditions, and inference behavior remain relevant. The PET disadvantage is also not uniform—for example, SAM-Med3D-turbo scores 0.410 on PET versus 0.404 on paired CT, and SegVol scores 0.148 on PET versus 0.134 on paired MRI. Regional generalization is not independently tested. The paper should therefore narrow its conclusions to the observed model-, organ-, and protocol-specific findings.

Questions for authors
  1. Can the authors provide exact checkpoint identifiers, preprocessing, prompt-generation rules, semantic mappings, and inference settings, together with in-domain positive-control results from the same implementation?

  2. How were the PET annotations obtained and reviewed? Were structural-image masks transferred through registration, and which modalities were available to the radiologists during correction?

  3. Will the images, annotations, evaluation code, and a fixed evaluation protocol be available to other researchers?Please provide an access plan.

Reviewer_BCcLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces UMD, a paired benchmark comprising 490 PET/CT studies and 464 PET/MRI studies with annotations for 13 organs. It evaluates five representative 3D segmentation foundation models using their native prompting interfaces, with nnU-Net as an additional reference. The paper reports limitations in out-of-the-box generalization and argues for broader validation across imaging modalities and anatomical regions.

Critical strengths
  1. The within-subject paired design enables comparisons of corresponding organs across structural and functional images, reducing confounding from differences in patient anatomy and target selection.
  2. The dataset scale and paired multimodal annotations offer potential resource value. Organ-wise results and the nnU-Net reference also provide useful information for understanding differences beyond overall model performance.
  3. The manuscript appropriately acknowledges that the experiments do not isolate training-data bias from model and implementation factors, or independently manipulate modality and regional coverage.
Critical weaknesses
  1. The extreme failure patterns require stronger validation. Table 2 reports a CT liver Dice of 0.9186 for VISTA3D but near-zero scores for most other organs; SAT achieves a mean CT Dice of only 0.0181. These may be genuine failures, but the current implementation description does not sufficiently exclude class-mapping, preprocessing, or spatial-alignment errors. Given the centrality of these results, please provide reproducible configurations, checks on known reference examples, and representative UMD predictions that validate the complete inference-to-scoring pipeline.
  2. Several central conclusions are inconsistent with the results or exceed the evidence. Table 1 does not generally support the claim that nnU-Net has a smaller structural-functional gap than the foundation models: its CT-PET and MRI-PET gaps are 0.081 and 0.092, and only nnInteractive has larger gaps in both cohorts. Some models also have slightly higher PET means than their structural counterparts, requiring qualification of the claim of PET degradation across all models. The Table 1 caption states that literature and UMD scores are not numerically comparable, yet the text uses their differences to infer systematic overestimation. Likewise, poor segmentation alone does not establish that the model has memorized dataset-specific patterns or lacks anatomical understanding. The paper should distinguish observed performance limitations from hypotheses about their causes.
  3. Key information underlying the paired comparison is missing. Appendix B does not clarify whether PET reference labels were independently annotated or transferred from CT/MRI, or how registration and cross-modal label quality were verified. These details are central to the validity of the matched-organ comparison. Essential acquisition information, particularly MRI sequences, CT scan type, and spatial resolution, is also missing. Please clarify the annotation and registration procedures and report the acquisition characteristics needed to interpret the tested distributions.
  4. Statistical reporting and reproducibility remain insufficient. Tables 2 and 3 do not define the parenthesized statistics, significance tests, or statistical units. Please explain the aggregation procedure and report paired modality differences with confidence intervals that account for within-patient dependence. Independent patient counts and training/test separation should also be clear. Finally, the paper should specify how the benchmark and evaluation can be accessed or independently verified, for example through controlled access and reproducible code.
Questions for authors
  1. What checks substantiate the VISTA3D and SAT results, especially the contrast between strong CT liver performance and near-zero scores on other organs for VISTA3D?
  2. How were PET reference labels generated, registered, and reviewed, and what acquisition information characterizes the paired datasets?
  3. What patient-level paired analyses support the modality comparisons, and how will the authors reconcile the claims contradicted by Table 1 and enable independent verification?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.