- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces BenchX, a benchmark of 101,822 CT scans from six cohorts (with 54,624 scans from four cohorts being promised to be accessible publicly). The authors evaluate 12 segmentation models (trained on AbdomenAtlas2.0) for tumor detection across different demographic groups and imaging protocols. The authors identify substantial performance disparities across age, race, and contrast phases based off of measured F1 Scores in tumour detection. The authors also explore synthetic data augmentation, which they refer to as SynthX, as a potential strategy to counter these disparities.
Critical strengths
I believe this is an important and clinically relevant problem. Evaluating model performance across demographic groups and imaging protocols is an important gap in medical imaging to understand biases in models and understand this dimension for any clinical deployment.
I view the dataset as a valuable contribution as well. Large scale radiology datasets are scarce and difficult to release, so I believe the promised release of this data, along with the preprocessing and radiologist verifications, would stand to benefit the community. I also appreciate the more heterogeneous nature of the data that's being released, as I feel this will allow for better generalizability for the models that eventually use this data and could allow for interesting experiments.
Critical weaknesses
My main concerns have to do with the evaluation methodology
W1. All models are trained on one dataset, so the benchmark may mainly be showing the biases of that dataset. All 12 models are trained on the same data, namely AbdomenAtlas 2.0. The paper does not analyse how this training set’s distribution of age, sex, race or contrast phase can affect performance, which makes the results hard to disentangle between genuine biases and simply a data distribution shift. For instance, AbdomenAtlas 2.0 is described as contrast-enhanced CT, so weaker performance on non-contrast scans would be expected. It does not measure the biases of tumor-detection models in general, as the abstract suggests. I am curious to know why other public datasets, such as MSD-Pancreas or PANORAMA (both discussed in Figure 2), were not used to train additional models, or why the trained versions of the model were not used on the new data collected to measure the biases of actual current models published, rather than one only trained on AbdomenAtlas2.0. Training on several datasets would better reflect the biases across currently available data, and would better support the paper’s general conclusions that this is an issue the current state of the art faces.
W2. The subgroup findings are difficult to assess with the information given. The subgroup comparisons rely mainly on F1 scores. As F1 depends on prevalence, the same model can get a higher F1 in a group with more tumor-positive scans or vice versa. The authors note that tumor prevalence differs across subgroups. While the authors do perform confounder-controlled analyses, these are limited to five models and two cohorts. The authors also provide some subgroup counts and confidence intervals in the appendix, but these are not consistently reported for all the highlighted findings, making it difficult to judge how reliable some of the reported results are. Additionally, I am not clear on how the validation and test splits are used. In Sec. 4.1, the authors define a 1:9 validation/test split, and the tables in Appendix N are labeled as validation results. Are the main benchmarking results also from the validation split, or are they from the test split? More clarity on the purpose of this split is needed.
W3. I am not sure if the localization aspect in the title and introduction is evaluated. The main results rely on scan-level tumor detection, which does not require the predicted segmentation mask to overlap with the actual tumor. It could be that models make a segmentation mask, but it is not localised correctly, and so I am not sure how the paper currently assesses biases with respect to localization. Dice scores are used for the SynthX results, but there is no supporting analysis to help me contextualize this aspect of the results.
W4. The authors claim that using synthetic data can help reduce subgroup gaps. However, the SynthX results mainly show improvements in performance across subgroups, rather than consistent reductions in the gaps between them. For instance, the F1 gap between the best and worst age groups seems to actually increase looking at the F1 column of detection in Figure 8. These improvements could simply reflect the general benefit of synthetic data augmentation rather than specific reductions in subgroup disparities. The implementation details are also missing. The paper says to look to Appendix K for this, but Appendix K doesn't mention SynthX anywhere (I could have missed it in the appendix, and I'm happy to go back and have a look if the authors can point it out).
Questions for authors
The main questions I have, based on my weaknesses, is why only one dataset was used to train the 12 models from scratch, and more clarity on the dataset composition and test/eval split. The localization aspect and synthx implementation is less a question and moreso either missing material or analysis/results that do not support the claims being made.
4 comments
W2. Subgroup findings rely on F1, controlled analyses are limited, and the validation/test split is unclear
W2. The subgroup findings are difficult to assess with the information given. The subgroup comparisons rely mainly on F1 scores. [...] While the authors do perform confounder-controlled analyses, these are limited to five models and two cohorts. [...] Are the main benchmarking results also from the validation split, or are they from the test split? More clarity on the purpose of this split is needed.
- We agree: subgroup comparisons now lead with sensitivity and specificity. Unlike F1, they do not depend on prevalence. F1 becomes a secondary metric. Counts and CIs for every highlighted finding are in our reply to Reviewer 7139 (W1 & Q1).
- Controlled analyses now cover four cohorts. Due to limited time during the rebuttal, we add N-California and Merlin here. The results agree with Appendix M: for nnU-Net, sensitivity is higher on venous scans in all four cohorts, and higher in patients over 60 in all three cohorts with age results. We will add the remaining analyses in the revised version.
| Cohort | Comparison | Positives | nnU-Net gap: first − second group |
|---|---|---|---|
| E-Coast | Age ≤ 60 vs. > 60 | 874 / 1,433 | −9.8 |
| E-Coast | Female vs. male | 1,187 / 1,120 | −0.2 |
| E-Coast | Arterial vs. venous | 1,155 / 1,152 | −4.3 |
| N-Europe | Age ≤ 60 vs. > 60 | 55 / 284 | −11.9 |
| N-Europe | Female vs. male | 202 / 174 | −3.1 |
| N-Europe | Arterial vs. venous | 89 / 287 | −15.6 |
| N-California | Non-contrast vs. venous | 16 / 15 | −13.8 |
| Merlin | Age ≤ 60 vs. > 60 | 18 / 25 | −21.1 |
| Merlin | Arterial vs. venous | 27 / 25 | −8.2 |
- The main results are on the test data; BenchX is not used for validation. All six BenchX cohorts are used only for testing. The validation set comes from AbdomenAtlas 2.0, our training data: we randomly split it 9:1 into training and validation sets with a fixed random seed, and the validation set covers all subgroups where possible. We use the validation set only to select checkpoints. For testing, E-Coast uses all 5,102 scans. The other five cohorts are too large to test in full, so we test on a random sample from each with a fixed seed: Merlin 976, PanTS 901, N-California 993, S-Europe 1,043 and N-Europe 1,086 scans.
W3. Is localization evaluated, given that scan-level detection does not require overlap with the tumor?
W3. I am not sure if the localization aspect in the title and introduction is evaluated. The main results rely on scan-level tumor detection, which does not require the predicted segmentation mask to overlap with the actual tumor. [...]
Scan-level detection checks location at the organ level, but not at the tumor level. After post-processing, a predicted tumor is kept only if it overlaps the pancreas (Appendix L.1). We do not require overlap with the reference tumor, because many BenchX scans do not yet have voxel-level tumor masks. So the main results use scan-level detection, as in prior work on tumor detection in CT [1–3]. To evaluate localization at the tumor level, we add results on PanTS and E-Coast, the two cohorts with tumor masks.
Experiment. On PanTS and E-Coast, we use the annotated tumor masks. A positive scan counts as detected only if the predicted tumor overlaps an annotated tumor by at least one voxel. The table below shows sensitivity (%) without → with this rule.
| Model | PanTS | E-Coast |
|---|---|---|
| nnU-Net | 68.9 → 62.9 | 69.6 → 58.2 |
| ResEnc | 60.9 → 51.0 | 32.3 → 27.8 |
| STU-Net-B | 66.9 → 60.9 | 68.5 → 56.6 |
| MedNeXt | 72.2 → 68.9 | 52.3 → 46.3 |
| VSmTrans | 21.2 → 18.5 | 77.5 → 68.0 |
Conclusion. When overlap is required, sensitivity drops by 2.7–11.9 pp. So most scan-level detections hit the annotated tumor, but some fall on a wrong spot in the pancreas.
References
[1] Liu, Jie, et al. "CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection." ICCV (2023), pp. 21152–21164.
[2] Chen, Qi, et al. "Scaling Tumor Segmentation: Best Lessons from Real and Synthetic Data." ICCV (2025), pp. 24001–24013.
[3] Chen, Jieneng, et al. "CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans." ICCV (2023), pp. 21327–21338.
W4. SynthX improves all subgroups rather than reducing the gaps, and its implementation details are missing
W4. The authors claim that using synthetic data can help reduce subgroup gaps. However, the SynthX results mainly show improvements in performance across subgroups, rather than consistent reductions in the gaps between them. [...] The implementation details are also missing. The paper says to look to Appendix K for this, but Appendix K doesn't mention SynthX anywhere [...]
We agree: SynthX does not consistently reduce the gaps. Figure 8 shows higher sensitivity, Dice and NSD in every E-Coast subgroup, but not smaller gaps. We revise Sec. 4.5 to claim only these gains, and remove the claim that SynthX reduces subgroup gaps.
The gains are not only from adding more synthetic data. SynthX differs from prior tumor synthesis in what it conditions on: patient age, sex and CT phase, so it can add tumors for under-represented subgroups. Prior methods generate tumors with hand-crafted rules [4], or with generative models conditioned on tumor masks [1, 2] or text [3]. On E-Coast, we compared SynthX with DiffTumor [1], FreeTumor [2] and SynTumor [4], using the same segmentation model, real training scans and number of synthetic tumors. SynthX has higher DSC, NSD and BAUC in all 12 subgroups. Removing the age, sex and CT-phase conditions from SynthX gives DiffTumor [1], so this comparison is also a condition ablation. Full results are in our reply to Reviewer 7139 (W3 & Q4).
You did not miss it: Appendix K does not describe SynthX. We apologize for the wrong pointer. We will add the implementation details in the revision, including the generator, the conditions (age, sex and CT phase), the number of synthetic tumors and the training setup.
Thank you again for your careful review. We hope our replies address your concerns, and we would be grateful if you would consider raising your score.
References
[1] Chen, Qi, et al. "Towards Generalizable Tumor Synthesis." CVPR (2024), pp. 11147–11158.
[2] Wu, Linshan, et al. "FreeTumor: Advance Tumor Segmentation via Large-Scale Tumor Synthesis." arXiv:2406.01264 (2024).
[3] Li, Xinran, et al. "Text-Driven Tumor Synthesis." IEEE Transactions on Medical Imaging (2026).
[4] Hu, Qixin, et al. "Label-Free Liver Tumor Segmentation." CVPR (2023), pp. 7422–7432.
W1. All models are trained on one dataset; why not train on MSD-Pancreas or PANORAMA, or test published models?
Thank you for the thoughtful review. We appreciate your view that this is "an important and clinically relevant problem," that you "view the dataset as a valuable contribution," and that its release "would stand to benefit the community."
In brief: (1) we add PanDx, a model trained on PANORAMA, explain that MSD-Pancreas is already in our training data, and show how the training data relate to the gaps (W1); (2) we lead with sensitivity and specificity, report counts and CIs, add two cohorts to the controlled analyses, and explain the splits (W2); (3) we add localization results (W3); (4) we narrow the SynthX claim, show a condition ablation, and will add the missing implementation details (W4).
We agree that our results do not measure "tumor-detection models in general." We narrow the abstract and add a model trained on other data. We used one training set on purpose: with the same training data, differences between the 12 models come from the models themselves, not from the data.
References
[1] Liu, Han, et al. "PanDx: AI-Assisted Early Detection of Pancreatic Ductal Adenocarcinoma on Contrast-Enhanced CT." International Workshop on Applications of Medical AI (2025), pp. 63–71.