- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces LesionBench, a benchmark for lesion segmentation in 3D CT. The benchmark contains 12,824 training and 3,035 test CT scans from multiple sites, curated from 24 datasets that span various cancer types, patient demographics, lesion locations, and fields of view. To standardize clinician labels, the authors consider a binary lesion segmentation problem and utilize the nnU-Net framework. The experiments reveal that generalizing 3D CT lesion segmentation across different data distributions is still a challenging problem.
Critical strengths
- The benchmark is constructed from 24 datasets, providing a unified large-scale 3D CT segmentation dataset. The proposed rigorous de-duplication pipeline also reduces potential train-test data leakage that could be caused by derived source datasets. The authors also promise to make the benchmark code, preprocessing pipeline, and data split publicly available.
- The analysis is quite thorough. The authors analyze convolution-, transformer-, and hybrid-based models within the nnU-Net framework for fair comparison. The authors also analyze the performance on different fov, population, and lesion location, and compare the models trained on the full dataset vs. each dataset as a specialist model.
- The finding reveals an interesting, well-supported observation: despite rapid advancements in CT lesion segmentation, a reliable generalist lesion segmentation model is still hard to achieve.
Critical weaknesses
The benchmark only considers a binary segmentation problem. While this binary segmentation problem can reveal the challenging nature of 3D CT segmentation across various distributions, performance on different cancer types could provide further insight into each cancer type segmentation difficulty.
Questions for authors
Did the authors try to pre-train the models on the full LesionBench dataset and then fine-tune them on a specific dataset? It would be interesting to know if pre-training on a large 3D CT lesion segmentation dataset improves model initialization.
4 comments
Fine-Tuning
We thank the reviewer for their interesting question on fine-tuning on an unseen dataset. We fine-tuned three ResEnc-XL models (MSD-Pa and PANORAMA specialists and the generalist) on PanTS [1], a pancreatic CT dataset that is not part of LesionBench. Each was fine-tuned for 500 epochs on the 875 lesion-bearing PanTS training scans in two settings, frozen encoder (decoder only) and full fine-tuning. All were tested, alongside the zero-shot models (no fine-tuning), on the 151 lesion-bearing PanTS test scans (161 lesions). Lesion recall and precision use the same lesion matching methodology as the lesion-level analysis in our previous response (IoU ≥ 0.1).
| Setting | Model | DSC | NSD | Lesion recall | Lesion precision |
|---|---|---|---|---|---|
| Zero-shot | Generalist | 0.36 ± 0.33 | 0.24 ± 0.24 | 0.62 | 0.14 |
| MSD-Pa specialist | 0.37 ± 0.36 | 0.32 ± 0.29 | 0.52 | 0.55 | |
| PANORAMA specialist | 0.34 ± 0.34 | 0.25 ± 0.24 | 0.50 | 0.48 | |
| Frozen encoder | Generalist | 0.46 ± 0.34 | 0.36 ± 0.28 | 0.66 | 0.50 |
| MSD-Pa specialist | 0.38 ± 0.35 | 0.29 ± 0.27 | 0.54 | 0.30 | |
| PANORAMA specialist | 0.41 ± 0.34 | 0.33 ± 0.27 | 0.58 | 0.49 | |
| Full fine-tuning | Generalist | 0.49 ± 0.34 | 0.37 ± 0.28 | 0.68 | 0.58 |
| MSD-Pa specialist | 0.46 ± 0.34 | 0.34 ± 0.28 | 0.63 | 0.43 | |
| PANORAMA specialist | 0.48 ± 0.33 | 0.35 ± 0.28 | 0.68 | 0.47 |
Zero-shot, the generalist detects more lesions than the specialists but with low precision, the same pattern we observe across datasets within LesionBench. Training only its decoder raises its precision from 0.14 to 0.50 while keeping its recall, and brings its DSC to 0.46, close to the 0.49 of full fine-tuning. The specialists gain less with a frozen encoder and need full fine-tuning to reach the generalist's recall, which suggests that the generalist's encoder transfers better to unseen data. The differences in DSC between the models are small compared with the spread across scans. These results will be added to the appendix.
[1] Wenxuan Li, Xinze Zhou, Qi Chen, Tianyu Lin, Pedro R. A. S. Bassi, Szymon Płotka, Jarosław B. Ćwikła, Xiaoxi Chen, Chen Ye, Zheren Zhu, Kai Ding, Heng Li, Kang Wang, Yang Yang, Yucheng Tang, Daguang Xu, Alan L. Yuille, and Zongwei Zhou. PanTS: The pancreatic tumor segmentation dataset. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.
Dear authors. Thank you for addressing my concerns. Please find my notes below:
- the generalist's lower DSC is driven mainly by additional predictions, not missed lesions. --> I believe that in practice, precision is also important to prevent false positives which could cause unnecessary medical costs and could affect patients. Thus I suggest the authors to acknowledge this limitation in the paper.
- We then test the specialist performance out of domain compared to the generalist model. --> When comparing the specialist against the generalist, was the target/test domain also excluded from the generalist's training? If not, the authors should acknowledge this and be careful not to overclaim, as the gain could potentially be a result of having the same domain in the generalist's training data.
- The fine-tuning on an unseen dataset demonstrates the strong generalizability of the generalist's frozen features.
Considering the authors' clarifications, I keep my positive rating.
Thank you for the renewed support in our work.
We recognise that the false positives are an issue and we have clearly highlighted this in the results discussion of the updated manuscript. We agree on the strong capability of the generalist to generalise on an unseen dataset. Thanks to the further experiments the reviewer suggested, we further believe that future attempts should focus on maintaining the generalist capabilities while working on reducing false postive predictions.
Lesion Level Analysis
We would like to thank the reviewer for their thoughtful comments on the value of LesionBench. Please find below our analysis at the lesion-level performance which support a more detailed analysis across different cancer types:
To evaluate lesion-level performance ground-truth and prediction masks are split into connected components and matched one to one by IoU; a lesion counts as detected at IoU ≥ 0.1. The analysis covers the full test set (3,034 scans, 9,346 lesions) using the best specialist per dataset and the ResEnc-XL generalist.
Across all lesions the specialists detect 43% and the generalist 39%, but they differ sharply in precision (0.49 against 0.09): the generalist's lower DSC is driven mainly by additional predictions, not missed lesions. AutoPET contributes 52% of all lesions; excluding it, recall is 0.73 and 0.66 and precision 0.55 and 0.19.
We then test the specialist performance out of domain compared to the generalist model.
Out of domain the specialists detect 30% of lesions, down from 67% on their own test sets, while the generalist detects 55% on the same scans and has the higher recall. Although, when aggregated by anatomical region the specialists and the pooled generalist are level in precision (0.12 and 0.13). We interpret this as a trade-off. A specialist is accurate on its own data and on average loses more than half of its detections on another dataset of the same location, whereas the generalist detects lesions at a stable rate across sources and pays for it in false positives.
The lesions the generalist misses are not a random subset. Excluding AutoPET, it misses 90% of lesions that are both under 0.1 cm³ and under 20 HU of contrast with the surrounding tissue, against 6% of those over 10 cm³ and over 200 HU. Over all datasets, of its 41,210 predicted components, 91% match no lesion, against 51% for the specialists.
The full analysis, including miss rates by lesion volume and contrast and the false-positive breakdown, will be added to the revised manuscript and supplementary.