acceptodds
Under review as a conference paper at ICLR 2027

LesionBench: A 3D Computed Tomography Cancer Agnostic Segmentation Benchmark

Abstract

Medical oncology relies heavily on imaging for multiple tasks, from screening to computer-aided surgery. However, lesion segmentation is a significantly more complex task compared to multi-organ one, as lesions can appear in different body locations, exhibit highly variable sizes, shapes, and visual appearances when imaged using different scanners or protocols. To develop the next generation of AI models that can deliver acceptable performances compared to radiologists, there is a clear need to create broad, diverse, and structured 3D CT datasets with lesion annotations. Despite remarkable efforts, most available datasets are limited to a single cancer type or have scattered annotations. To address these shortcomings, we present LesionBench, a large-scale CT binary segmentation benchmark that spans more than 7 types of cancers (including primary and metastatic cancers) across the entire human body. LesionBench comprises 12,824 training CT scans and 3,035 testing CT scans from multiple centers worldwide. By collating 24 existing cancer datasets, while de-duplicating cross-dataset samples, we provide a unified, large-scale, and cancer-agnostic medical AI benchmark. We benchmark well-established segmentation models within the gold-standard nnU-Net auto-configuring framework, which includes CNNs, Transformers, and hybrid architectures on both cancer-agnostic training/evaluation settings, as well as on single-cancer settings (in- and out-of-distribution), enabling the assessment of classical segmentation metrics (DSC, NSD) and the clinically motivated SRVE. Our results show that lesion segmentation remains far from solved: the best model achieves a case-weighted DSC of 0.53 and a macro-averaged per-dataset DSC of 0.45. We further find that single-dataset specialists transfer poorly across datasets of the same organ, whereas cancer-agnostic pooling trades per-dataset precision for substantially improved robustness. To support future research in cancer-agnostic segmentation, we release the benchmark code, de-duplication pipeline, and data splits. LesionBench: https://tinyurl.com/LesionBenchmark

Primary area:
datasets and benchmarks
Keywords:
Medical Imaging, Deep Learning, Segmentation, Cancer Agnostic, Computed Tomography

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UVe5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LesionBench, a benchmark for lesion segmentation in 3D CT. The benchmark contains 12,824 training and 3,035 test CT scans from multiple sites, curated from 24 datasets that span various cancer types, patient demographics, lesion locations, and fields of view. To standardize clinician labels, the authors consider a binary lesion segmentation problem and utilize the nnU-Net framework. The experiments reveal that generalizing 3D CT lesion segmentation across different data distributions is still a challenging problem.

Critical strengths
  • The benchmark is constructed from 24 datasets, providing a unified large-scale 3D CT segmentation dataset. The proposed rigorous de-duplication pipeline also reduces potential train-test data leakage that could be caused by derived source datasets. The authors also promise to make the benchmark code, preprocessing pipeline, and data split publicly available.
  • The analysis is quite thorough. The authors analyze convolution-, transformer-, and hybrid-based models within the nnU-Net framework for fair comparison. The authors also analyze the performance on different fov, population, and lesion location, and compare the models trained on the full dataset vs. each dataset as a specialist model.
  • The finding reveals an interesting, well-supported observation: despite rapid advancements in CT lesion segmentation, a reliable generalist lesion segmentation model is still hard to achieve.
Critical weaknesses

The benchmark only considers a binary segmentation problem. While this binary segmentation problem can reveal the challenging nature of 3D CT segmentation across various distributions, performance on different cancer types could provide further insight into each cancer type segmentation difficulty.

Questions for authors

Did the authors try to pre-train the models on the full LesionBench dataset and then fine-tune them on a specific dataset? It would be interesting to know if pre-training on a large 3D CT lesion segmentation dataset improves model initialization.

4 comments
AuthorsOpenReview ↗

Lesion Level Analysis

We would like to thank the reviewer for their thoughtful comments on the value of LesionBench. Please find below our analysis at the lesion-level performance which support a more detailed analysis across different cancer types:

To evaluate lesion-level performance ground-truth and prediction masks are split into connected components and matched one to one by IoU; a lesion counts as detected at IoU ≥ 0.1. The analysis covers the full test set (3,034 scans, 9,346 lesions) using the best specialist per dataset and the ResEnc-XL generalist.

GroupSpecialist lesion recallGeneralist lesion recallSpecialist lesion precisionGeneralist lesion precisionSpecialist DSCGeneralist DSCSpecialist NSDGeneralist NSDSpecialist SRVE %Generalist SRVE %
Head and neck0.690.560.530.110.560.470.400.3125.933.8
Lung0.600.580.540.240.580.440.610.4428.440.3
Liver0.640.360.660.100.670.390.520.2318.844.0
Pancreas0.830.810.600.080.610.330.490.2026.553.7
Kidney0.730.530.670.130.870.630.760.457.226.3
Adrenal1.001.000.850.100.910.800.670.385.016.5
Colon0.590.520.340.030.470.260.520.1839.950.1
ULS230.870.940.520.740.600.670.650.7127.021.8
AutoPET0.160.140.340.030.360.170.190.0835.658.3
All0.430.390.490.090.590.530.540.4726.332.2
All excluding AutoPET0.730.660.550.190.600.550.560.5025.730.5

Across all lesions the specialists detect 43% and the generalist 39%, but they differ sharply in precision (0.49 against 0.09): the generalist's lower DSC is driven mainly by additional predictions, not missed lesions. AutoPET contributes 52% of all lesions; excluding it, recall is 0.73 and 0.66 and precision 0.55 and 0.19.

We then test the specialist performance out of domain compared to the generalist model.

Lesion locationIn-domain specialist recallIn-domain specialist precisionOut-of-domain specialist recallOut-of-domain specialist precisionGeneralist recall (same scans)Generalist precision (same scans)
Kidney0.730.670.000.000.530.13
Lung0.600.540.220.320.580.24
Liver0.640.660.380.280.360.10
Pancreas0.830.600.600.430.810.08
Head and neck0.690.530.320.110.560.11
All0.670.550.300.120.550.13

Out of domain the specialists detect 30% of lesions, down from 67% on their own test sets, while the generalist detects 55% on the same scans and has the higher recall. Although, when aggregated by anatomical region the specialists and the pooled generalist are level in precision (0.12 and 0.13). We interpret this as a trade-off. A specialist is accurate on its own data and on average loses more than half of its detections on another dataset of the same location, whereas the generalist detects lesions at a stable rate across sources and pays for it in false positives.

The lesions the generalist misses are not a random subset. Excluding AutoPET, it misses 90% of lesions that are both under 0.1 cm³ and under 20 HU of contrast with the surrounding tissue, against 6% of those over 10 cm³ and over 200 HU. Over all datasets, of its 41,210 predicted components, 91% match no lesion, against 51% for the specialists.

The full analysis, including miss rates by lesion volume and contrast and the false-positive breakdown, will be added to the revised manuscript and supplementary.

AuthorsOpenReview ↗

Fine-Tuning

We thank the reviewer for their interesting question on fine-tuning on an unseen dataset. We fine-tuned three ResEnc-XL models (MSD-Pa and PANORAMA specialists and the generalist) on PanTS [1], a pancreatic CT dataset that is not part of LesionBench. Each was fine-tuned for 500 epochs on the 875 lesion-bearing PanTS training scans in two settings, frozen encoder (decoder only) and full fine-tuning. All were tested, alongside the zero-shot models (no fine-tuning), on the 151 lesion-bearing PanTS test scans (161 lesions). Lesion recall and precision use the same lesion matching methodology as the lesion-level analysis in our previous response (IoU ≥ 0.1).

SettingModelDSCNSDLesion recallLesion precision
Zero-shotGeneralist0.36 ± 0.330.24 ± 0.240.620.14
MSD-Pa specialist0.37 ± 0.360.32 ± 0.290.520.55
PANORAMA specialist0.34 ± 0.340.25 ± 0.240.500.48
Frozen encoderGeneralist0.46 ± 0.340.36 ± 0.280.660.50
MSD-Pa specialist0.38 ± 0.350.29 ± 0.270.540.30
PANORAMA specialist0.41 ± 0.340.33 ± 0.270.580.49
Full fine-tuningGeneralist0.49 ± 0.340.37 ± 0.280.680.58
MSD-Pa specialist0.46 ± 0.340.34 ± 0.280.630.43
PANORAMA specialist0.48 ± 0.330.35 ± 0.280.680.47

Zero-shot, the generalist detects more lesions than the specialists but with low precision, the same pattern we observe across datasets within LesionBench. Training only its decoder raises its precision from 0.14 to 0.50 while keeping its recall, and brings its DSC to 0.46, close to the 0.49 of full fine-tuning. The specialists gain less with a frozen encoder and need full fine-tuning to reach the generalist's recall, which suggests that the generalist's encoder transfers better to unseen data. The differences in DSC between the models are small compared with the spread across scans. These results will be added to the appendix.

[1] Wenxuan Li, Xinze Zhou, Qi Chen, Tianyu Lin, Pedro R. A. S. Bassi, Szymon Płotka, Jarosław B. Ćwikła, Xiaoxi Chen, Chen Ye, Zheren Zhu, Kai Ding, Heng Li, Kang Wang, Yang Yang, Yucheng Tang, Daguang Xu, Alan L. Yuille, and Zongwei Zhou. PanTS: The pancreatic tumor segmentation dataset. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.

Dear authors. Thank you for addressing my concerns. Please find my notes below:

  • the generalist's lower DSC is driven mainly by additional predictions, not missed lesions. --> I believe that in practice, precision is also important to prevent false positives which could cause unnecessary medical costs and could affect patients. Thus I suggest the authors to acknowledge this limitation in the paper.
  • We then test the specialist performance out of domain compared to the generalist model. --> When comparing the specialist against the generalist, was the target/test domain also excluded from the generalist's training? If not, the authors should acknowledge this and be careful not to overclaim, as the gain could potentially be a result of having the same domain in the generalist's training data.
  • The fine-tuning on an unseen dataset demonstrates the strong generalizability of the generalist's frozen features.

Considering the authors' clarifications, I keep my positive rating.

Thank you for the renewed support in our work.

We recognise that the false positives are an issue and we have clearly highlighted this in the results discussion of the updated manuscript. We agree on the strong capability of the generalist to generalise on an unseen dataset. Thanks to the further experiments the reviewer suggested, we further believe that future attempts should focus on maintaining the generalist capabilities while working on reducing false postive predictions.

Reviewer_Pj14OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a unified benchmark for pathological lesion segmentation by integrating datasets from multiple cancer sites. It aims to provide standardized training and evaluation protocols for assessing segmentation methods across diverse clinical settings. The authors evaluate existing segmentation approaches and report comparative results to highlight their performance and generalization capabilities.

Critical strengths

The paper addresses an important problem in pathological lesion segmentation by integrating diverse datasets into a unified benchmark. The standardized evaluation protocol may facilitate systematic comparisons across existing methods. However, the limited analysis of cross-domain generalization and disease-specific performance reduces its contribution beyond dataset integration.

Critical weaknesses

W1. Limited evaluation on unseen cancer sites. Although the paper provides cross-dataset evaluations, the cancer-agnostic models are primarily evaluated on datasets represented during training. Leave-one-cancer-site-out experiments would provide stronger evidence of generalization to unseen pathological settings. W2. Limited analysis of dataset composition. The benchmark reveals a clear trade-off between specialist accuracy and generalist robustness. However, the effects of dataset imbalance, cancer-type diversity, and training data composition on this trade-off remain insufficiently explored. W3. Heterogeneous annotation protocols. Combining datasets with different lesion definitions, annotation protocols, and fields of view into a unified binary segmentation task may introduce inconsistencies. The influence of these differences on model evaluation requires further analysis.

Questions for authors

refer to the weakness

3 comments
AuthorsOpenReview ↗

Unseen Sites, Dataset Composition and Heterogeneous Protocols

We thank the reviewer for highlighting LesionBench’s positive impact in the important problem of pathological lesion segmentation evaluation, while highlighting relevant improvement points. We address them point by point below:

Unseen cancer sites: We agree that the cross-dataset analysis test transfer between sources that the generalist had seen in training, and that an evaluation on unseen sites would be stronger evidence. To account for this, we select PanTS [1] a pancreatic CT dataset, with scans from a broad range of institutions as the unseen site. We took its official test split and kept the 151 cases that contain a pancreatic lesion, consistent with LesionBench settings. We ran three ResEnc-XL models on PanTS in inference: MSD Pancreas specialist, PANORAMA specialist, and the generalist trained on all of LesionBench's training set. Because PanTS was assembled from public sources, we checked it against both pancreatic cancer training sets included in LesionBench with a duplicate search and found no duplicates. Results are reported below:

ModelDiceNSDSRVE (%)VolumePrecisionRecallEmpty
MSD-Pa specialist0.37 ± 0.360.32 ± 0.2949.75 ± 37.59under0.590.3628
PANORAMA specialist0.34 ± 0.340.25 ± 0.2452.16 ± 34.76under0.510.3419
Generalist (2000 ep.)0.36 ± 0.330.24 ± 0.2448.17 ± 33.27over0.370.501

All models reach similar DSC scores with the specialists losing the in-domain advantage the paper reports. Interesting to be noted is that the models arrive at that DSC by opposite routes. The specialists under-segment and return no prediction at all on 28 and 19 of 151 scans and predict about two thirds of the reference volume, with precision 0.59 and 0.51 against recall 0.36 and 0.34. In contrast, the generalist over-segments, but only leaves one scan empty, predicting about three times the reference volume, and finds half the reference volume on average, with recall 0.50 against precision 0.37. We believe that for a model used to infer on unseen data, a complete miss rate of around 19% (MSD-PA specialist) where we know lesions are present within the scan is the most serious failure mode.

We will add this experiment to the supplementary material of the paper as an example of external evaluation that can be achieved with the LesionBench trained models.

Dataset composition analysis: We examined whether the trade-off between specialists and the generalist follows the composition of the training set. For each of the 24 datasets we compared its share of the pooled training set, which ranges from 0.2% to 40%, with the difference in DSC between the generalist and the best specialist on that dataset's test set. The size of a dataset's share does not track how much it loses: PANORAMA contributes 4.3% of training scans and loses 0.32 DSC under pooling, RADCURE contributes 19% and loses 0.05, and HN-Cetuximab contributes 0.2% and loses 0.12. Dataset imbalance alone does not seem to explain the trade-off. We agree that a controlled study that retrains the generalist under different compositions would be needed to separate the effects of imbalance and cancer-type diversity, although due to the generalist training time it is not achievable within the rebuttal period. We will add the per-dataset analysis to the supplementary and better discuss this point in the main paper limitation section.

Heterogeneous annotation protocols: The heterogeneity of lesion definitions, annotation protocols and fields of view across the 24 sources is a deliberate design choice. It is the condition any cancer-agnostic model meets in practice, and a benchmark that oversees it would not test what LesionBench is designed to test. We therefore do not harmonise the annotations. Instead, we quantify their influence in two ways. First, every result is reported per dataset and per organ, so no conclusion rests on comparing scores across protocols. Second, the cross-dataset evaluation measures the cost of a change in protocol directly: a specialist applied to another dataset of the same organ retains about two thirds of its in-domain DSC in liver, pancreas and head and neck. We agree that differences in annotation convention between sources contribute to the in-domain cost of pooling, and we will state this explicitly in the discussion. However, we also believe that a general model should see and learn how to robustly perform across variable segmentation protocols which may reflect the real-world variability between clinicians' annotation standards.

[1] Wenxuan Li, Xinze Zhou, Qi Chen, Tianyu Lin, Pedro R. A. S. Bassi, Szymon Płotka, Jarosław B. Ćwikła, Xiaoxi Chen, ..., & Zongwei Zhou. PanTS: The pancreatic tumor segmentation dataset. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.

Reviewer_Pj14OpenReview ↗

Reply

Thank you for the additional experiments and clarifications. The rebuttal partially addresses my concerns, although I am not fully satisfied with the response to W1. The new PanTS evaluation provides useful evidence of cross-dataset generalization, which I appreciate. However, despite training on more diverse datasets, the generalist achieves comparable Dice but lower NSD and precision than the MSD-Pa specialist. These results leave the generalization benefits of multi-cancer training somewhat unclear.

AuthorsOpenReview ↗

Generalisation Benifits

Thanks for taking the time to review our further experiments. We agree the evaluation metrics are comparable. Previous analysis has shown that the generalist does produce more false positives, which reduce the patient level metrics. Important to notice on the generalisation benefits of multi-cancer training is that the generalist is able to detect lesions at a much higher rate when compared to the specialists on their target task out of distribution. The specialists in our setup predicted 28/19 of the 151 scans (MSD-Pa/PANORAMA) to be healthy, whereas the generalist did so for only 1, which for a cancer segmentation model is a relevant achievement. When looking at individual scans, we found that on the scans where the specialists did not locate a lesion, the generalist was able to detect one in 37/34% of them.

Restricting the evaluation to the pancreas (ground-truth mask plus a 10 mm margin) explains that our generalist scan level results are driven by false positives away from the target organ. 44% of the generalist's predicted volume falls outside this region (11/6% for the specialists). Inside the region, the zero-shot generalist outperforms both specialists, with a DSC of 0.46, up from 0.36 (specialists 0.37/0.34), higher on 78/80 scans and lower on only 38/31, while matching their lesion precision (0.61, up from 0.14; specialists 0.60/0.54) and detecting more lesions (recall 0.62 against 0.52/0.50).

Although we agree that such numbers are not absolute, we believe that it is an interesting point for future work to explore false positive reduction in the generalist's model development. We found that decoder-only fine-tuning was effective enough to increase the precision of the generalist to specialist levels.

Reviewer_zekbOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the fragmentation of public CT lesion data, where most datasets are small, restricted to one cancer site, or annotated only partially or in 2D, and argues that single-site benchmarks encourage methods tailored to one pathology. LesionBench gathers 24 public collections covering kidney, lung, liver, colon, pancreas, head and neck, adrenal and multi-location lesions into 12,824 training and 3,035 test volumes, removes non-lesion structures and maps all lesion labels to a single foreground class, fuses multi-rater masks by majority presence and STAPLE, and detects train/test duplicates across datasets with a statistical fingerprint, a ball-tree search, a shape filter and a cosine similarity on 128³ downsampled volumes, followed by manual review of 963 candidates that led to 154 test cases being moved to training. Eight architectures (nnU-Net, ResEnc M/L/XL, MedNeXt M/L, Primus M/L) are trained in the nnU-Net framework on the pooled data for 2k epochs and as single-dataset specialists for 1k epochs, and evaluated with DSC, NSD at 1 mm, SRVE and complete-miss rates. The best pooled model, ResEnc-XL, reaches 0.53 case-weighted and 0.45 macro DSC, specialists transfer poorly to other datasets of the same organ (KiTS23 and KiPA22 fall to at most 0.004 DSC in both directions), and pooling lowers the ResEnc-XL complete-miss rate from 34.2% to 7.1% while reducing in-domain DSC on most sources. The authors conclude that cancer-agnostic lesion segmentation is far from solved and that neither specialist nor pooled training is sufficient.

Critical strengths
  1. This work assembles a multi-site, multi-cancer CT lesion benchmark that is reproducible without redistributing images. The release of conversion code, label mappings, STAPLE consensus code, fixed case-identifier split lists and digest-pinned Docker images is the right design for data under heterogeneous licenses, five of which require a data-use agreement, and it makes subsets of the benchmark usable by groups that focus on a single site.

  2. This work proposes a cross-dataset de-duplication step and shows that leakage is a real problem for benchmarks built from overlapping challenges. Because many challenges are derived from earlier collections (MSD Liver from LiTS is the example given), patient-level splits made independently per dataset can place the same scan in training and test; the paper documents the similarity distribution (Figure 2 and Table 6), describes where false positives begin (below about 0.997 similarity), and reassigns 154 confirmed leaks.

  3. This work reports a specialist versus generalist analysis with complementary metrics that exposes concrete failure modes. The KiTS23 and KiPA22 collapse is traced to field-of-view differences, lung transfer shows a size-dependent volume bias (under-prediction on NSCLC sets, over-prediction on LIDC and LNDb), NSD and DSC diverge in head and neck transfer (65% of in-domain DSC retained versus 53% of NSD), and precision varies about three times more than recall across models (0.23 to 0.68 versus 0.59 to 0.74), which indicates that false-positive volume is the main source of disagreement.

Critical weaknesses
  1. Lack of any treatment of partial annotation when the 24 sources are pooled. Each source labels lesions only within its own target: KiTS23 labels renal masses, LIDC and LNDb label lung nodules, the head and neck sets keep the primary GTV under different conventions (Table 7), and AutoPET labels FDG-avid lesions over the whole body. In the pooled binary task, a liver metastasis in a KiTS scan or a nodal lesion in a lung CT is background during training and a false positive at test time. The general over-segmentation in Table 3 and the wide precision spread in Figure 8 may therefore partly measure unannotated lesions rather than errors, which directly affects the conclusion that pooling costs in-domain accuracy. A supplementary experiment is needed in which the loss is masked outside the annotated organ or field of view, as is done for partially labeled multi-organ data (for example the CLIP-Driven Universal Model, Liu et al., ICCV 2023), and in which DSC, precision and SRVE are also reported inside the annotated organ region.

  2. Lack of a planning protocol that is appropriate for pooled heterogeneous data, which confounds the architecture and pooling conclusions. nnU-Net derives one target spacing, patch size and network topology from the dataset fingerprint (as described in the appendix on model configuration), so the pooled model is planned once for a mixture of tight kidney crops (KiPA22), lesion-centred crops (ULS23), head and neck scans and whole-body PET/CT acquisitions, while every specialist is planned for its own data. Part of the pooling penalty can therefore come from a spacing and patch size that suit no subset well. The same issue affects the architecture comparison: on the pooled data ResEnc-L and ResEnc-XL are the same 527.4M network and differ only in patch size (80×256×256 versus 112×320×256), so the gap between them, and the jump from default nnU-Net (0.23 global DSC) to ResEnc-L (0.48), says more about patch size and context than about model family or capacity. Analyses with a fixed spacing and patch across architectures, and with pooled training under per-source resampling, are needed before attributing the gap to architecture or to pooling.

  3. Lack of sufficiently strong and verified baselines. The ResEnc-XL specialists on LIDC and LNDb collapse to 0.00 DSC in and out of domain (Table 4), which points to a training failure rather than task difficulty, yet these runs are kept in the averages; generalists are trained for 2k epochs and specialists for 1k; no seeds or confidence intervals are given; and Adrenal is evaluated on 11 cases. The specialist results are not compared with published results on the constituent challenges (KiTS23, HECKTOR, PANORAMA, ULS23) under their official protocols, which is the most direct check that the training recipe is competitive. Universal and promptable 3D models are absent, for example the CLIP-Driven Universal Model (Liu et al., ICCV 2023), SAT (Zhao et al., npj Digital Medicine 2025), VISTA3D (He et al., CVPR 2025) and SegVol (Du et al., NeurIPS 2024). A supplementary experiment with at least one such model and with an nnU-Net conditioned on source or organ (an input prompt or separate output heads) would show whether the trade-off reported in the Discussion is intrinsic or a consequence of naive pooling.

  4. Lack of an estimate of how many duplicates the de-duplication pipeline misses. Each volume is normalized to [0, 1] by its own percentiles, summarized by 13 intensity statistics, and compared after resampling to a 128³ grid, with pairs discarded if their resampled shapes differ by more than five voxels. This design detects near-identical volumes, but it cannot match a cropped derivative to its source, different contrast phases or time points of the same patient, or the same study exported with a different field of view. This matters for at least two sources in the benchmark. ULS23 releases lesion-centred volumes of interest, and besides its new annotations it is built from lesions processed out of KiTS21, LIDC-IDRI, LiTS, MSD tasks 6, 7 and 10 (lung, pancreas, colon), NIH-LN and CCC18 (de Grauw et al., 2025); KiTS21 cases are part of KiTS23, and LIDC, MSD-Lu, MSD-Pa and MSD-Co are separate sources in LesionBench, so the same lesion can appear as a full scan in one split and as a ULS23 crop in the other, which the five-voxel shape filter excludes by construction. Similarly, the four Québec centres of HECKTOR (CHUM, CHUS, HGJ, HMR) take their contours from the study of Vallières et al. (2017), whose 298 patients form the TCIA Head-Neck-PET-CT collection included here as HN-PET. Only pairs above 0.995 similarity were reviewed, and moving leaked cases from test to train changes the composition of the test set without being reported per dataset. Matching on TCIA patient identifiers and DICOM UIDs where they exist, a registration-based check for crops, and a recall estimate obtained by injecting known cropped or resampled duplicates would make the leakage claim much stronger.

  5. Lack of a lesion-level and clinically meaningful evaluation. Every test volume contains a lesion, so false-positive behaviour on lesion-free scans, which matters for any screening or triage use, is never measured. Each scan yields one overlap score regardless of the number of lesions, so a large lesion can hide missed metastases in AutoPET, LIDC or CLiMet, and there is no per-lesion detection rate or FROC analysis. The case-weighted DSC is dominated by the largest sources, NSD uses a 1 mm tolerance that is below the slice spacing of many scans in these collections and therefore close to a voxel-exact criterion, and no subgroup results are given despite the reported 2:1 sex imbalance and the concentration on two scanner vendors.

Questions for authors
  1. How are lesions outside the target organ of each source handled during pooled training and evaluation, and how do DSC, precision and SRVE change if evaluation is restricted to the annotated organ region?
  2. What caused the collapse of the ResEnc-XL specialists on LIDC and LNDb, and were these runs repeated or excluded from the averages?
  3. Which spacing and patch size did nnU-Net choose for the pooled dataset, and how does pooled performance on cropped and whole-body sources change under per-source planning?
  4. Why do generalists and specialists use different epoch budgets, and does the trade-off persist under equal training?
  5. How do the specialist results compare with published results on KiTS23, HECKTOR, PANORAMA and ULS23 under the official protocols?
  6. Have you estimated the recall of the de-duplication pipeline? In particular, have you checked whether ULS23 crops derived from KiTS21, LIDC-IDRI and MSD tasks 6, 7 and 10 overlap across splits with the corresponding full-scan sources, and whether HECKTOR cases from the Québec centres overlap with HN-PET at the patient level?
  7. Will the original lesion categories (cancer type, organ, primary versus nodal target) be released alongside the binary masks?
3 comments
AuthorsOpenReview ↗

Partial Annotations and Protocol/Planning

We thank the reviewer for the careful reading of our work and for pointing out LesionBench's value and strengths, while pointing to improvement points. We have addressed them below:

Partial Annotation: We agree that false positives may generate from unlabeled true lesions outside the target organ. We tested this on three single-organ test sets: KiTS23, MSD Liver and MSD Lung. For each test scan we segmented the target organ with TotalSegmentator (TS) [1], merging the organ and the lesion mask and added a 10 mm margin over the TS organ boundary, then recomputed DSC, precision and SRVE inside this region. We report the share of each model's predicted volume that falls outside it. All models are ResEnc-XL.

Dataset (n)ModelDSC whole → regionPrecision whole → regionSRVE % whole → regionPredicted volume outside region
KiTS23 (98)In-domain specialist0.85 → 0.860.89 → 0.909.8 → 9.31%
Generalist0.60 → 0.780.55 → 0.8428.4 → 14.437%
MSD Liver (25)In-domain specialist0.51 → 0.540.78 → 0.8342.0 → 39.37%
Cross-dataset specialists (CLiMet / TRIALS / HCC-TACE)0.46 / 0.35 / 0.25 → 0.48 / 0.37 / 0.270.74 / 0.70 / 0.44 → 0.77 / 0.73 / 0.635% / 7% / 40%
Generalist0.23 → 0.370.22 → 0.5358.4 → 43.171%
MSD Lung (11)In-domain specialist0.56 → 0.560.66 → 0.6631.2 → 31.20%
Cross-dataset specialist (NSCLC-RG)0.71 → 0.720.72 → 0.7315.2 → 14.04%
Generalist0.45 → 0.520.46 → 0.5928.8 → 26.418%

The in-domain specialists place almost all of their predictions inside the annotated region (0–7% outside), whereas a substantial share of the generalist's predicted volume lies outside it (18% in lung, 37% in kidney, 71% in liver). Restricting evaluation to the region raises the generalist's precision markedly (0.55 to 0.84 on KiTS23, 0.22 to 0.53 on MSD Liver). The gap between specialist and generalist narrows inside the region (DSC 0.86 vs 0.78, 0.54 vs 0.37, 0.56 vs 0.52).

To further test whether those outside predictions are unannotated lesions, we exploit AbdomenAtlas 2.0 [2] which provides clinically grounded annotations of lesions outside the considered target organs (58 additional lesions in 22 of 133 scans). The generalist detects 13 of them (FP rate after accounting for these is still 89%), against 1 for the specialists, so part of its over-segmentation is correct detection of unannotated lesions; with so few annotated scans we cannot confirm this is prevalent throughout LesionBench. We believe the conclusion that pooling costs in-domain accuracy holds. Masking the loss outside the annotated organ during training, as in the CLIP-Driven Universal Model, will be discussed as future work.

Protocol and Planning: For the pooled dataset, nnU-Net selected the following target spacing and patch size for each model:

Pooled modelParametersTarget spacing (mm)Patch size (voxels)Global DSC
nnU-Net88.4M2.0 × 0.82 × 0.8264 × 192 × 1600.23
ResEnc-M383.2M2.0 × 0.82 × 0.8264 × 160 × 1600.31
ResEnc-L527.4M2.0 × 0.82 × 0.8280 × 256 × 2560.48
ResEnc-XL527.4M2.0 × 0.82 × 0.82112 × 320 × 2560.53
MedNeXt-M17.55M1.0 × 1.0 × 1.0128 × 128 × 1280.29
MedNeXt-L61.78M1.0 × 1.0 × 1.0128 × 128 × 1280.31
Primus-M244.51M2.0 × 0.82 × 0.8264 × 192 × 1600.25
Primus-L428.29M2.0 × 0.82 × 0.8264 × 192 × 1600.26

MedNeXt uses one fixed configuration, so its specialists and generalists share the same spacing and patch size. With planning held fixed, the MedNeXt-M generalist still scores below its specialist on 21 of the 24 datasets, by 0.23 DSC on average, so the cost of pooling is not explained by planning alone. On the architecture comparison we agree: pooled ResEnc-L and ResEnc-XL are the same network and differ only in patch size, so the difference between them reflects spatial context and not capacity, and we have revised the text accordingly.

A fixed plan across all architectures, and pooled training under per-source resampling, both require retraining the pooled models, which is not feasible within the rebuttal period. Concurrent work [3] explores planning that adapts to each dataset, which could be explored for a pooled model. We will add the pooled and per-dataset plans to the appendix and state the planning confound as a limitation.

[1] J Wasserthal, et al. TotalSegmentator: Robust segmentation of 104 anatomic structures in CT images. Radiology: Artificial Intelligence, 5(5):e230024, 2023.

[2] Q Chen, et al. Scaling tumor segmentation: Best lessons from real and synthetic data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 24001–24013, 2025.

[3] C U Harsy, et al. nnFoundation: 3D foundation models for radiology. arXiv preprint arXiv:2609.26924, 2026.

AuthorsOpenReview ↗

Baselines and Deduplication

Baselines: The ResEnc-XL runs on LIDC-IDRI and LNDb completed and the loss decreased, but the models predict background only. We attribute this to large patches combined with nodules of a few millimetres, which leaves almost no foreground signal per patch; other architectures with the same recipe reach 0.59 (ResEnc-L, LIDC) and 0.32 (nnU-Net, LNDb). We kept them in the tables as real outcomes of the recipe, and we reported averages both with and without them. To avoid any misinterpretation, we will better stress when reporting averaged values if that two zeroed runs are included/excluded.

Specialists use the standard 1000-epoch nnU-Net schedule, by which point their training curves have plateaued. The generalist had not converged at 1000 epochs on the much larger pool, so we trained it to 2000 in order to compare converged models. The benchmark comprises 152 trained models of up to 527M parameters, several taking days on an A100, so repeating them across seeds is beyond our compute; we will state this, and the smaller test sets, as limitations. We report the standard deviation over cases and release all per-case results so that any interval can be computed.

Looking at published challenges, although the future end goal we envision is to help in the development of a generalist model that can be better or comparable to highly optimized specialists, this is currently outside of our main scope. Additionally, several of the constituent challenges evaluate models on hidden test sets, which are not included in LesionBench. The best performing models are generally ensembles and post-processing tuned for the leaderboard, so a single-fold model scored on our own split cannot be compared to a leaderboard number on equal terms, and presenting it as such would be misleading. Conditioned and promptable models are interesting comparisons although promptable models address a different task (prompted segmentation rather than unprompted detection), so a fair protocol needs design. We will add this comparison as possible future work.

Regarding the original lesion categories, we provide the code to convert the original downloaded annotations into the binary masks considered within LesionBench. So, an end user can either decide to follow the LesionBench protocol or keep more dataset related targets.

Deduplication Verification: To characterise what our pipeline does and does not detect, we built a probe from the benchmark itself: 50 LesionBench CTs, five each from ten datasets. From these images we created duplicates covering possible seen duplication instances. Every true pair is therefore known, and we ran the pipeline unchanged on this set.

TransformImitatesRecall
Identical copysame file in two sources50/50
Re-saved as float32re-export, other dtype or compression50/50
Reorienteddifferent storage orientation50/50
Gaussian noise, 10 HUreprocessed or re-reconstructed scan50/50
Slice spacing doubledresampled export50/50
In-plane spacing × 1.5resampled export42/50
Shifted 4 voxels against headerre-registered export30/50
In-plane crop to body outlinepartial field of view2/50
Axial crop, central halfpartial coverage0/50
Crop in both axesvolume of interest0/50
False Positive PairsN/A46 pairs out of 500

The pipeline detects whole-volume duplicates reliably, including re-exported, reoriented, noise-perturbed and resampled copies, and detects shifted copies partially. As the reviewer anticipates, it does not match a cropped derivative to its source scan: the shape filter rejects pairs whose resampled shapes differ, and the whole-volume hash assumes a shared field of view. It also reports pairs between anatomically similar patients, 46 in this probe, all in HECKTOR and TRIALS which is why pairs were reviewed manually at submission. We will add this as a stated limitation of the de-duplication methodology.

AuthorsOpenReview ↗

Lesion Level Analysis

We thank the reviewer for their comment and agree that a per-scan overlap score can hide missed lesions, so we have performed a lesion-level analysis. One clarification first: LesionBench contains no lesion-free scans by construction, so false-positive behaviour on healthy scans is outside the benchmark's scope.

To evaluate lesion-level performance ground-truth and prediction masks are split into connected components and matched one to one by IoU; a lesion counts as detected at IoU ≥ 0.1. The analysis covers the full test set (3,034 scans, 9,346 lesions) using the best specialist per dataset and the ResEnc-XL generalist.

GroupSpecialist lesion recallGeneralist lesion recallSpecialist lesion precisionGeneralist lesion precisionSpecialist DSCGeneralist DSCSpecialist NSDGeneralist NSDSpecialist SRVE %Generalist SRVE %
Head and neck0.690.560.530.110.560.470.400.3125.933.8
Lung0.600.580.540.240.580.440.610.4428.440.3
Liver0.640.360.660.100.670.390.520.2318.844.0
Pancreas0.830.810.600.080.610.330.490.2026.553.7
Kidney0.730.530.670.130.870.630.760.457.226.3
Adrenal1.001.000.850.100.910.800.670.385.016.5
Colon0.590.520.340.030.470.260.520.1839.950.1
ULS230.870.940.520.740.600.670.650.7127.021.8
AutoPET0.160.140.340.030.360.170.190.0835.658.3
All0.430.390.490.090.590.530.540.4726.332.2
All excluding AutoPET0.730.660.550.190.600.550.560.5025.730.5

Across all lesions the specialists detect 43% and the generalist 39%, but they differ sharply in precision (0.49 against 0.09): the generalist's lower DSC is driven mainly by additional predictions, not missed lesions. AutoPET contributes 52% of all lesions; excluding it, recall is 0.73 and 0.66 and precision 0.55 and 0.19.

We then test the specialist performance out of domain compared to the generalist model.

Lesion locationIn-domain specialist recallIn-domain specialist precisionOut-of-domain specialist recallOut-of-domain specialist precisionGeneralist recall (same scans)Generalist precision (same scans)
Kidney0.730.670.000.000.530.13
Lung0.600.540.220.320.580.24
Liver0.640.660.380.280.360.10
Pancreas0.830.600.600.430.810.08
Head and neck0.690.530.320.110.560.11
All0.670.550.300.120.550.13

Out of domain the specialists detect 30% of lesions, down from 67% on their own test sets, while the generalist detects 55% on the same scans and has the higher recall. Although, when aggregated by anatomical region the specialists and the pooled generalist are level in precision (0.12 and 0.13). We interpret this as a trade-off. A specialist is accurate on its own data and on average loses more than half of its detections on another dataset of the same location, whereas the generalist detects lesions at a stable rate across sources and pays for it in false positives.

The lesions the generalist misses are not a random subset. Excluding AutoPET, it misses 90% of lesions that are both under 0.1 cm³ and under 20 HU of contrast with the surrounding tissue, against 6% of those over 10 cm³ and over 200 HU. Over all datasets, of its 41,210 predicted components, 91% match no lesion (marginally over-estimated given non annotated lesions outsider target organs), against 51% for the specialists.

We did not compute FROC curves because probability maps were not retained, and recomputing them is not feasible within the rebuttal period. The full analysis, including miss rates by lesion volume and contrast and the false-positive breakdown, will be added to the revised manuscript and supplementary.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Manuscript Revision

We greatly appreciate the reviewers for their time and constructive comments which allow us to improve the quality of our work. We have uploaded a revised version of the paper that integrates the additional experiments and better reflects the discussion from the rebuttal period. Below is a short summary of the changes made:

Main text

  • Lesion-level analysis (Section 3.3, Figure 2; zekb, UVe5): We added an evaluation at the level of individual lesions, comparing the generalist with the best specialist of each dataset, in-domain and out of domain. It reports lesion recall and precision, the lesions the generalist misses by volume and contrast, and where its false positives fall.

  • Discussion and limitations (Section 4; zekb, Pj14): The limitations are rewritten and now cover the evaluation protocol (connected components as lesion instances, no lesion-free scans, target-only annotation), single-seed training and small test sets, the effect of nnU-Net planning, the single training-set composition, and label noise. The discussion now notes that differences in annotation convention between sources may contribute to the cost of pooling.

Supplementary material

  • De-duplication probe (Appendix A.2; zekb): The recall of the de-duplication pipeline on synthetic duplicates, and its limitation on cropped scans.

  • Planned configurations (Appendix A.6, Table 9; zekb): The nnU-Net plan of every specialist and of the pooled models, including that the pooled ResEnc-L and ResEnc-XL share one network and differ only in patch size, and the MedNeXt comparison in which specialists and generalist share one fixed plan.

  • Lesion-level analysis (Appendix A.8; zekb, UVe5): The method, the choice of IoU threshold, and the full in-domain and out-of-domain tables.

  • Predictions outside the annotated organ (Appendix A.9; zekb): An organ-region analysis and a check against the additional lesion annotations of AbdomenAtlas 2.0.

  • External evaluation on PanTS (Appendix A.10; Pj14, UVe5): The generalist and the two pancreas specialists on a dataset outside LesionBench, zero-shot (without adaptation, including an analysis restricted to the pancreas region), with decoder-only fine-tuning and with full fine-tuning.

  • Training-set composition (Appendix A.11; Pj14): The difference between the generalist and the best specialist against each dataset's share of the pooled training set.

Other changes

  • We changed the colour palette to improve readability for readers with colour-blindness and improved the sentence structure for readability and corrected values throughout the paper. This includes the parameter counts of the nnU-Net, ResEnc and Primus models, which had counted shared weights more than once. A test case was removed (being a flagged duplicate that got missed at submission time) and therefore some averages were corrected accordingly. None of these corrections changes the conclusions of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.