- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper focus on benchmarking 3D segmentation foundation models. They curated 490 whole-body PET/CT and 464 PET/MRI scans and conducted a thorough benchmarking over representative 3D segmentation foundation models like SAT, nnInteractive, SAM-Med3D, VISTA3D, e.t.c. The results show that the existing foundation models are not performing well on this benchmark, and PET imaging are not well supported.
Critical strengths
This is a benchmarking paper and they are benchmarking over the popular topic of 3D medical segmentation foundation models, and revealed the limitations of current models. The topic is very important.
Critical weaknesses
- The benchmark is a valuable but relatively small (~1000 paired PET-CT, PET-MR scans) with only 13 common organs. The benchmark itself, although reveals limitation of current foundation models, is not impactful enough to be of broad interest to ICLR community.
- The details of the benchmark is not fully discussed. How the annotation is acquired? Did you use auto-segmentation first then human correction? or fully human annotated? The quality of the annotation is unknown. Second, those foundation models are trained on huge cohort of CT volumes, and CTs are usually much stable/ less domain variance than MRI, why do we see such a huge performance gap even in CT? Those foundation models have been trained/validated on almost all publically available CT datasets, what is so special about this dataset? The author didn't provide such explanation.
- I don't see the paper mentioning open sourcing the dataset.
Questions for authors
- What is the domain shift of your CT/MRI dataset compared with existing public datasets like totalsegmentator, amos e.t.c?
- in Table2, VISTA3D has a 0.9 dice score on Liver while others are low, but completely 0 in other organs. This is a very strange value for any evaluation, and seems like a bug to me.