LesionBench: A 3D Computed Tomography Cancer Agnostic Segmentation Benchmark

Medical oncology relies heavily on imaging for multiple tasks, from screening to computer-aided surgery. However, lesion segmentation is a significantly more complex task compared to multi-organ one, as lesions can appear in different body locations, exhibit highly variable sizes, shapes, and visual appearances when imaged using different scanners or protocols. To develop the next generation of AI models that can deliver acceptable performances compared to radiologists, there is a clear need to create broad, diverse, and structured 3D CT datasets with lesion annotations. Despite remarkable efforts, most available datasets are limited to a single cancer type or have scattered annotations. To address these shortcomings, we present LesionBench, a large-scale CT binary segmentation benchmark that spans more than 7 types of cancers (including primary and metastatic cancers) across the entire human body. LesionBench comprises 12,824 training CT scans and 3,035 testing CT scans from multiple centers worldwide. By collating 24 existing cancer datasets, while de-duplicating cross-dataset samples, we provide a unified, large-scale, and cancer-agnostic medical AI benchmark. We benchmark well-established segmentation models within the gold-standard nnU-Net auto-configuring framework, which includes CNNs, Transformers, and hybrid architectures on both cancer-agnostic training/evaluation settings, as well as on single-cancer settings (in- and out-of-distribution), enabling the assessment of classical segmentation metrics (DSC, NSD) and the clinically motivated SRVE. Our results show that lesion segmentation remains far from solved: the best model achieves a case-weighted DSC of 0.53 and a macro-averaged per-dataset DSC of 0.45. We further find that single-dataset specialists transfer poorly across datasets of the same organ, whereas cancer-agnostic pooling trades per-dataset precision for substantially improved robustness. To support future research in cancer-agnostic segmentation, we release the benchmark code, de-duplication pipeline, and data splits. LesionBench: https://tinyurl.com/LesionBenchmark