acceptodds
Under review as a conference paper at ICLR 2027

BenchX: Benchmarking AI Models for Cancer Detection and Localization with Demographic and Protocol Biases

Abstract

Artificial intelligence (AI) has achieved remarkable success in medical imaging, but it is widely recognized that these models often perform inconsistently across real-world clinical settings. Such inconsistencies occur when patient demographics and imaging protocols vary, for example, in detecting small tumors, analyzing scans from different contrast phases, or evaluating patients of different ages or sexes. To quantify these inconsistencies, we develop a large-scale, open benchmark of \numofct CT scans that systematically evaluates \numofmodels tumor-detection AI models across tumor size, location, patient subgroup, and imaging protocol. We leverage large language models (LLMs) to extract and organize subgroup information from clinical data, which makes the analysis both scalable and reproducible. Our benchmark reveals that current state-of-the-art AI models, optimized for average accuracy, perform poorly in rare or underrepresented subgroups, such as young, female African Americans. However, collecting sufficient annotated data for these rare cases is often impractical. The benchmark provide a foundation for building more reliable and robust AI models for tumor detection and highlighting the need for rigorous, subgroup-level evaluation in medical imaging and computer vision. Code, models, and data will be released upon acceptance. In total, 54,624 CT scans will be publicly accessible, supporting direct evaluation of demographic and protocol biases.

Primary area:
datasets and benchmarks
Keywords:
Tumor Detection, Tumor Localization, Benchmark, Demographic and Protocol Biases

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_NdTSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces BenchX, a benchmark of 101,822 CT scans from six cohorts (with 54,624 scans from four cohorts being promised to be accessible publicly). The authors evaluate 12 segmentation models (trained on AbdomenAtlas2.0) for tumor detection across different demographic groups and imaging protocols. The authors identify substantial performance disparities across age, race, and contrast phases based off of measured F1 Scores in tumour detection. The authors also explore synthetic data augmentation, which they refer to as SynthX, as a potential strategy to counter these disparities.

Critical strengths

I believe this is an important and clinically relevant problem. Evaluating model performance across demographic groups and imaging protocols is an important gap in medical imaging to understand biases in models and understand this dimension for any clinical deployment.

I view the dataset as a valuable contribution as well. Large scale radiology datasets are scarce and difficult to release, so I believe the promised release of this data, along with the preprocessing and radiologist verifications, would stand to benefit the community. I also appreciate the more heterogeneous nature of the data that's being released, as I feel this will allow for better generalizability for the models that eventually use this data and could allow for interesting experiments.

Critical weaknesses

My main concerns have to do with the evaluation methodology

W1. All models are trained on one dataset, so the benchmark may mainly be showing the biases of that dataset. All 12 models are trained on the same data, namely AbdomenAtlas 2.0. The paper does not analyse how this training set’s distribution of age, sex, race or contrast phase can affect performance, which makes the results hard to disentangle between genuine biases and simply a data distribution shift. For instance, AbdomenAtlas 2.0 is described as contrast-enhanced CT, so weaker performance on non-contrast scans would be expected. It does not measure the biases of tumor-detection models in general, as the abstract suggests. I am curious to know why other public datasets, such as MSD-Pancreas or PANORAMA (both discussed in Figure 2), were not used to train additional models, or why the trained versions of the model were not used on the new data collected to measure the biases of actual current models published, rather than one only trained on AbdomenAtlas2.0. Training on several datasets would better reflect the biases across currently available data, and would better support the paper’s general conclusions that this is an issue the current state of the art faces.

W2. The subgroup findings are difficult to assess with the information given. The subgroup comparisons rely mainly on F1 scores. As F1 depends on prevalence, the same model can get a higher F1 in a group with more tumor-positive scans or vice versa. The authors note that tumor prevalence differs across subgroups. While the authors do perform confounder-controlled analyses, these are limited to five models and two cohorts. The authors also provide some subgroup counts and confidence intervals in the appendix, but these are not consistently reported for all the highlighted findings, making it difficult to judge how reliable some of the reported results are. Additionally, I am not clear on how the validation and test splits are used. In Sec. 4.1, the authors define a 1:9 validation/test split, and the tables in Appendix N are labeled as validation results. Are the main benchmarking results also from the validation split, or are they from the test split? More clarity on the purpose of this split is needed.

W3. I am not sure if the localization aspect in the title and introduction is evaluated. The main results rely on scan-level tumor detection, which does not require the predicted segmentation mask to overlap with the actual tumor. It could be that models make a segmentation mask, but it is not localised correctly, and so I am not sure how the paper currently assesses biases with respect to localization. Dice scores are used for the SynthX results, but there is no supporting analysis to help me contextualize this aspect of the results.

W4. The authors claim that using synthetic data can help reduce subgroup gaps. However, the SynthX results mainly show improvements in performance across subgroups, rather than consistent reductions in the gaps between them. For instance, the F1 gap between the best and worst age groups seems to actually increase looking at the F1 column of detection in Figure 8. These improvements could simply reflect the general benefit of synthetic data augmentation rather than specific reductions in subgroup disparities. The implementation details are also missing. The paper says to look to Appendix K for this, but Appendix K doesn't mention SynthX anywhere (I could have missed it in the appendix, and I'm happy to go back and have a look if the authors can point it out).

Questions for authors

The main questions I have, based on my weaknesses, is why only one dataset was used to train the 12 models from scratch, and more clarity on the dataset composition and test/eval split. The localization aspect and synthx implementation is less a question and moreso either missing material or analysis/results that do not support the claims being made.

4 comments
AuthorsOpenReview ↗

W1. All models are trained on one dataset; why not train on MSD-Pancreas or PANORAMA, or test published models?

Thank you for the thoughtful review. We appreciate your view that this is "an important and clinically relevant problem," that you "view the dataset as a valuable contribution," and that its release "would stand to benefit the community."

In brief: (1) we add PanDx, a model trained on PANORAMA, explain that MSD-Pancreas is already in our training data, and show how the training data relate to the gaps (W1); (2) we lead with sensitivity and specificity, report counts and CIs, add two cohorts to the controlled analyses, and explain the splits (W2); (3) we add localization results (W3); (4) we narrow the SynthX claim, show a condition ablation, and will add the missing implementation details (W4).

W1. All models are trained on one dataset, so the benchmark may mainly be showing the biases of that dataset. [...] I am curious to know why other public datasets, such as MSD-Pancreas or PANORAMA (both discussed in Figure 2), were not used to train additional models, or why the trained versions of the model were not used on the new data collected to measure the biases of actual current models published [...]

We agree that our results do not measure "tumor-detection models in general." We narrow the abstract and add a model trained on other data. We used one training set on purpose: with the same training data, differences between the 12 models come from the models themselves, not from the data.

  1. MSD-Pancreas is already in our training data. AbdomenAtlas 2.0 includes the 281 MSD-Pancreas training scans. A model trained on MSD-Pancreas alone would learn from a small part of our training data, not from new data. AbdomenAtlas 2.0 is also much larger: 10,134 CT scans with 13,223 tumor annotations (Appendix K.1), compared with 281 scans in MSD-Pancreas and 2,238 in PANORAMA.
  2. The training data explain part of the gaps. In the training data, 64.7% of scans are venous, 26.4% arterial and 8.4% non-contrast, and 65.2% are male (Appendix K.2, Figure 24). As you expected, the phases with the lowest sensitivity have the fewest training scans, and on N-Europe, males have higher sensitivity. How much sensitivity drops depends on the cohort: non-contrast sensitivity reaches 92.3% on PanTS but at most 30.6% on N-California.
  3. New: we test PanDx, a published model trained on PANORAMA dataset. Few published pancreatic tumor detectors release their trained weights. PanDx [1] does: it won the PANORAMA challenge and is trained on PANORAMA. We test it on BenchX with the same post-processing. PANORAMA has only one CT phase (Table 5), and PanDx's phase results follow its training data (table below). At the same specificity on E-Coast, PanDx has 6.6 points higher sensitivity than nnU-Net on venous scans but 8.2 points lower on arterial scans. On N-Europe non-contrast scans, it is 11.7 points lower. For age and sex, the two models differ by less than 3 points.
ModelVenousArterialNon-contrastFemaleMaleAge ≤ 60Age > 60
nnU-Net66.9 (64.6–69.1)72.3 (70.2–74.4)45.5 (34.8–56.5)70.7 (68.0–73.2)74.0 (71.3–76.5)64.9 (61.7–68.0)76.8 (74.5–78.9)
PanDx73.5 (71.4–75.6)64.1 (61.8–66.3)33.8 (24.2–44.9)71.1 (68.5–73.6)75.3 (72.7–77.7)65.1 (61.9–68.2)79.7 (77.5–81.7)

References

[1] Liu, Han, et al. "PanDx: AI-Assisted Early Detection of Pancreatic Ductal Adenocarcinoma on Contrast-Enhanced CT." International Workshop on Applications of Medical AI (2025), pp. 63–71.

AuthorsOpenReview ↗

W2. Subgroup findings rely on F1, controlled analyses are limited, and the validation/test split is unclear

W2. The subgroup findings are difficult to assess with the information given. The subgroup comparisons rely mainly on F1 scores. [...] While the authors do perform confounder-controlled analyses, these are limited to five models and two cohorts. [...] Are the main benchmarking results also from the validation split, or are they from the test split? More clarity on the purpose of this split is needed.

  1. We agree: subgroup comparisons now lead with sensitivity and specificity. Unlike F1, they do not depend on prevalence. F1 becomes a secondary metric. Counts and CIs for every highlighted finding are in our reply to Reviewer 7139 (W1 & Q1).
  2. Controlled analyses now cover four cohorts. Due to limited time during the rebuttal, we add N-California and Merlin here. The results agree with Appendix M: for nnU-Net, sensitivity is higher on venous scans in all four cohorts, and higher in patients over 60 in all three cohorts with age results. We will add the remaining analyses in the revised version.
CohortComparisonPositivesnnU-Net gap: first − second group
E-CoastAge ≤ 60 vs. > 60874 / 1,433−9.8
E-CoastFemale vs. male1,187 / 1,120−0.2
E-CoastArterial vs. venous1,155 / 1,152−4.3
N-EuropeAge ≤ 60 vs. > 6055 / 284−11.9
N-EuropeFemale vs. male202 / 174−3.1
N-EuropeArterial vs. venous89 / 287−15.6
N-CaliforniaNon-contrast vs. venous16 / 15−13.8
MerlinAge ≤ 60 vs. > 6018 / 25−21.1
MerlinArterial vs. venous27 / 25−8.2
  1. The main results are on the test data; BenchX is not used for validation. All six BenchX cohorts are used only for testing. The validation set comes from AbdomenAtlas 2.0, our training data: we randomly split it 9:1 into training and validation sets with a fixed random seed, and the validation set covers all subgroups where possible. We use the validation set only to select checkpoints. For testing, E-Coast uses all 5,102 scans. The other five cohorts are too large to test in full, so we test on a random sample from each with a fixed seed: Merlin 976, PanTS 901, N-California 993, S-Europe 1,043 and N-Europe 1,086 scans.
AuthorsOpenReview ↗

W3. Is localization evaluated, given that scan-level detection does not require overlap with the tumor?

W3. I am not sure if the localization aspect in the title and introduction is evaluated. The main results rely on scan-level tumor detection, which does not require the predicted segmentation mask to overlap with the actual tumor. [...]

Scan-level detection checks location at the organ level, but not at the tumor level. After post-processing, a predicted tumor is kept only if it overlaps the pancreas (Appendix L.1). We do not require overlap with the reference tumor, because many BenchX scans do not yet have voxel-level tumor masks. So the main results use scan-level detection, as in prior work on tumor detection in CT [1–3]. To evaluate localization at the tumor level, we add results on PanTS and E-Coast, the two cohorts with tumor masks.

Experiment. On PanTS and E-Coast, we use the annotated tumor masks. A positive scan counts as detected only if the predicted tumor overlaps an annotated tumor by at least one voxel. The table below shows sensitivity (%) without → with this rule.

ModelPanTSE-Coast
nnU-Net68.9 → 62.969.6 → 58.2
ResEnc60.9 → 51.032.3 → 27.8
STU-Net-B66.9 → 60.968.5 → 56.6
MedNeXt72.2 → 68.952.3 → 46.3
VSmTrans21.2 → 18.577.5 → 68.0

Conclusion. When overlap is required, sensitivity drops by 2.7–11.9 pp. So most scan-level detections hit the annotated tumor, but some fall on a wrong spot in the pancreas.

References

[1] Liu, Jie, et al. "CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection." ICCV (2023), pp. 21152–21164.

[2] Chen, Qi, et al. "Scaling Tumor Segmentation: Best Lessons from Real and Synthetic Data." ICCV (2025), pp. 24001–24013.

[3] Chen, Jieneng, et al. "CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans." ICCV (2023), pp. 21327–21338.

AuthorsOpenReview ↗

W4. SynthX improves all subgroups rather than reducing the gaps, and its implementation details are missing

W4. The authors claim that using synthetic data can help reduce subgroup gaps. However, the SynthX results mainly show improvements in performance across subgroups, rather than consistent reductions in the gaps between them. [...] The implementation details are also missing. The paper says to look to Appendix K for this, but Appendix K doesn't mention SynthX anywhere [...]

We agree: SynthX does not consistently reduce the gaps. Figure 8 shows higher sensitivity, Dice and NSD in every E-Coast subgroup, but not smaller gaps. We revise Sec. 4.5 to claim only these gains, and remove the claim that SynthX reduces subgroup gaps.

The gains are not only from adding more synthetic data. SynthX differs from prior tumor synthesis in what it conditions on: patient age, sex and CT phase, so it can add tumors for under-represented subgroups. Prior methods generate tumors with hand-crafted rules [4], or with generative models conditioned on tumor masks [1, 2] or text [3]. On E-Coast, we compared SynthX with DiffTumor [1], FreeTumor [2] and SynTumor [4], using the same segmentation model, real training scans and number of synthetic tumors. SynthX has higher DSC, NSD and BAUC in all 12 subgroups. Removing the age, sex and CT-phase conditions from SynthX gives DiffTumor [1], so this comparison is also a condition ablation. Full results are in our reply to Reviewer 7139 (W3 & Q4).

You did not miss it: Appendix K does not describe SynthX. We apologize for the wrong pointer. We will add the implementation details in the revision, including the generator, the conditions (age, sex and CT phase), the number of synthetic tumors and the training setup.

Thank you again for your careful review. We hope our replies address your concerns, and we would be grateful if you would consider raising your score.

References

[1] Chen, Qi, et al. "Towards Generalizable Tumor Synthesis." CVPR (2024), pp. 11147–11158.

[2] Wu, Linshan, et al. "FreeTumor: Advance Tumor Segmentation via Large-Scale Tumor Synthesis." arXiv:2406.01264 (2024).

[3] Li, Xinran, et al. "Text-Driven Tumor Synthesis." IEEE Transactions on Medical Imaging (2026).

[4] Hu, Qixin, et al. "Label-Free Liver Tumor Segmentation." CVPR (2023), pp. 7422–7432.

Reviewer_nQ47OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces BenchX, which brings together 101,822 abdominal CT scans from six cohorts and defines subgroups based on age, gender, recorded ethnicity, contrast enhancement phase, and lesion characteristics. The authors use an LLM to extract demographic and scan metadata, employ automated models to assist physicians in establishing lesion labels, and evaluate 12 models. The appendix reports the involvement of eight radiologists, manual spot checks of approximately 5% of automatically accepted cases, and manual verification of metadata for approximately 1,000 scans.

Critical strengths
  • Value of the resource and control procedures: The subgroup evaluation of 101,822 scans across six cohorts has practical potential. The added appendix clarifies that manual review covers not only cases with model disagreement but also spot checks of approximately 5% of automatically accepted cases, and that 10% of reviewed cases undergo independent review by a second reader. Metadata verification reports extraction accuracies of 99.6%, 96.8%, and 95.9% for age, ethnicity, and enhancement phase, respectively. The appendix also provides cohort-specific release permissions and access procedures.
  • Concrete and verifiable findings: The appendix provides extensive details that facilitate understanding and verification of the data construction process.
Critical weaknesses
  • Definition of AUC: Appendix L.2 describes it as the area under the ROC curve under balanced sampling, complementing single-threshold metrics. However, all 756 sets of values in Appendix Tabs 10–15 equal (Sensitivity + Specificity)/2 within rounding precision. For example, in Tab 10, the ≤20-year subgroup yields (100.0 + 90.9)/2 ≈ 95.5 for nnU-Net and (0 + 100)/2 = 50.0 for MedNeXt. This is precisely balanced accuracy. Using binary predictions as ROC inputs would also produce this value, but it would no longer provide evidence of ranking ability across thresholds.
  • Overclaim: Appendix A.5 shows that the E-Coast ≤20-year subgroup contains only 2 positive cases among 28 scans, whereas the >80-year subgroup contains 119 positive cases among 121 scans. The extreme F1 differences are therefore affected by both small sample sizes and positive-class prevalence. In Tab. 10, U-Net, nnU-Net, and other models achieve 100% sensitivity in the ≤20-year subgroup; even where some have low F1 scores, these results cannot all be interpreted as missed diagnoses. Regarding enhancement phase, Tab. 11 shows that MedNeXt on PanTS achieves sensitivity/F1 of 92.3/92.3 in the non-contrast phase, higher than the 69.4/80.7 in the venous phase. The controlled analyses in Appendix M do support some gaps in N-Europe, but they do not cover the strong claims concerning the ≤20-year subgroup or ethnicity in Merlin. Per-ethnicity positive and negative counts and corresponding uncertainty intervals remain insufficient. The author should replace generalizations such as “widespread failure in young patients or non-venous phases” with findings qualified by cohort, model, and error type.
Questions for authors
  • Clarification of AUC. Is AUC computed from continuous scan-level confidence scores, post-processed binary (0/1) predictions, or directly as the mean of sensitivity and specificity? Could the authors provide the scoring function and explain how continuous scores are constructed, and why all 756 sets of results match balanced accuracy?
  • Clarification of evaluation denominators and confidence intervals. Please explicitly map the total dataset size to the number of scans with available metadata, the number actually evaluated by each model, and the number retained for the controlled analyses. Please also provide patient- and scan-level positive and negative counts, confusion matrices, and confidence intervals for the pediatric and Black/Pacific Islander subgroups. Does the source-case bootstrap treat multiple phases or repeated scans from the same patient as a single cluster?
  • Clarification of audit results and labeling targets. Given that the authors did audit approximately 5% of automatically accepted cases, could they further report the number of cases with unanimous negative predictions in this audit, the rate of missed lesion labels detected, and any subgroup differences, and clarify whether readers were blinded to model outputs?
4 comments
AuthorsOpenReview ↗

W1 & Q1. How is AUC computed, and why do all 756 values equal balanced accuracy?

Thank you for this careful review and for checking all 756 entries. We appreciate your recognizing that "the subgroup evaluation of 101,822 scans across six cohorts has practical potential" and that "the appendix provides extensive details that facilitate understanding and verification of the data construction process."

In brief: (1) our "AUC" was balanced accuracy; we rename it and add ROC-AUC (W1 & Q1); (2) we agree about the overclaims and list every revised statement (W2); (3) we map all denominators, and re-sample the smallest subgroups for re-evaluation (Q2); (4) we report the audit results (Q3).

W1 & Q1. Definition of AUC: Appendix L.2 describes it as the area under the ROC curve under balanced sampling, complementing single-threshold metrics. However, all 756 sets of values in Appendix Tabs 10–15 equal (Sensitivity + Specificity)/2 within rounding precision. [...] Is AUC computed from continuous scan-level confidence scores, post-processed binary (0/1) predictions, or directly as the mean of sensitivity and specificity? Could the authors provide the scoring function and explain how continuous scores are constructed, and why all 756 sets of results match balanced accuracy?

You are right. Our "AUC" was computed from the binary scan-level decision of Appendix L.1. With a single operating point, the ROC curve has one vertex, and its area equals (Sensitivity + Specificity)/2. This single-point AUC is also known as balanced accuracy [1]. We re-checked: all 756 values equal (Sensitivity + Specificity)/2 within ±0.05. Calling it "AUC" in Appendix L.2 was misleading, and we apologize for the confusion.

  1. The correct name is balanced accuracy. It is a standard metric for class-imbalanced evaluation [2]. In medical imaging, it was adopted by the ISIC 2018 skin lesion challenge at MICCAI [3] and is used in MICCAI and Medical Image Analysis papers [4, 5]. "AUC" is now "balanced accuracy" in all tables, figures and text, and Appendix L.2 defines it.
  2. ROC-AUC will be added. Probability maps were not stored at inference. Computing ROC-AUC requires re-running inference for all 12 models, which was not possible within the rebuttal period. We will add ROC-AUC with CIs to Tables 10–15 in the revised paper.

References

[1] Sokolova, Marina, et al. "Beyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation." AI 2006: Advances in Artificial Intelligence, LNAI 4304 (2006), pp. 1015–1021.

[2] Brodersen, Kay H., et al. "The Balanced Accuracy and Its Posterior Distribution." ICPR (2010), pp. 3121–3124.

[3] Codella, Noel, et al. "Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)." arXiv:1902.03368 (2019).

[4] Galdran, Adrian, et al. "Balanced-MixUp for Highly Imbalanced Medical Image Classification." MICCAI (2021), pp. 323–333.

[5] Wen, Junhao, et al. "Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluation." Medical Image Analysis 63 (2020), 101694.

AuthorsOpenReview ↗

W2. Overclaim: generalizations should be qualified by cohort, model and error type

W2. Overclaim. [...] The author should replace generalizations such as "widespread failure in young patients or non-venous phases" with findings qualified by cohort, model, and error type.

We agree, and thank you for the concrete examples. We went back to the counts behind each strong claim. Below, we show what each claim said, what the counts show, and how we will rewrite it.

  • Age ≤ 20: the low F1 comes mostly from false positives. Sec. 5 said "Top-performing models completely fail on patients under 20 years old." But most low F1 scores come from false positives, not from missed diagnoses. On E-Coast, this group has only 2 positive scans. Eight of 12 models detect both positive scans. We will change it to: "Most models detect both positive scans in the E-Coast ≤ 20 group. Their low F1 comes from false positives. This group has only 2 positive scans."

  • Non-contrast phase: results depend on the cohort and the model. Sec. 5 said "Non-contrast scans produce near-zero sensitivity." On N-Europe, non-contrast sensitivity ranges from 2.6% to 54.5% across models. On PanTS, it ranges from 23.1% to 100%. For example, MedNeXt on PanTS reaches 92.3% on non-contrast scans, higher than 69.4% on venous scans. We will change it to: "Non-contrast sensitivity depends on the cohort and the model. On PanTS, it is high for most models."

  • Race: we remove or rewrite the strong claims. Sec. 5 said "Racial disparities are severe and systemic." The Abstract mentioned "young, female African Americans." On Merlin, very few positive scans are from Black or Pacific Islander patients. No analysis in the paper tested young, female African Americans as one group. We will remove "severe and systemic" and rewrite the example in the Abstract. We will report race results with counts and confidence intervals.

  • Age ≤ 40: too few positive scans to estimate sensitivity. Sec. 5 said "Younger patients (Age ≤ 40) show dramatic performance drops with near-zero sensitivity." For ages 20–40, Merlin, N-California and PanTS have very few positive scans. We will change it to: "In most cohorts, the youngest groups have too few positive scans to estimate sensitivity. We report them with confidence intervals."

The controlled analyses in Appendix M support three gaps. On N-Europe, the venous vs. arterial gap holds for all five models, and the sex gap holds for three (Table 8). On E-Coast, the age gap (> 60 vs. ≤ 60 years) holds for four of five models (Table 9). As you note, these analyses do not cover the ≤ 20 group or race on Merlin. We have rewritten those claims above.

AuthorsOpenReview ↗

Q2. Evaluation denominators, confusion matrices and CIs for small subgroups, and bootstrap clusters

Q2. Clarification of evaluation denominators and confidence intervals. Please explicitly map the total dataset size to the number of scans with available metadata, the number actually evaluated by each model, and the number retained for the controlled analyses. Please also provide patient- and scan-level positive and negative counts, confusion matrices, and confidence intervals for the pediatric and Black/Pacific Islander subgroups. Does the source-case bootstrap treat multiple phases or repeated scans from the same patient as a single cluster?

Denominators. The table below maps the 101,822 scans to each stage.

CohortScansWith ageWith sexWith raceWith phaseEvaluation subsetControlled analysis
E-Coast5,1023,8753,8651,6445,1025,1023,863
Merlin25,39325,39225,39025,12825,386976–
PanTS9,9015,0925,28209,900901–
N-California42,09642,09642,07441,47012,854993–
S-Europe2,5101,1261,1621,0562,0941,043–
N-Europe16,82016,58016,580016,5801,086401 for age; 518 for sex and phase

Small subgroups: too few positive scans, so we re-sampled them. These subgroups have only 1–2 positive scans, which is too few for reliable confusion matrices. Even for a model that misses every positive scan, the 95% CI of sensitivity is 0–65.8% with 2 positives and 0–79.3% with 1 positive. We therefore re-sampled the test data so that each subgroup has 30 positive scans (table below). We will re-evaluate all 12 models on these sets and report patient- and scan-level counts, confusion matrices and 95% CIs in the revised paper.

CohortSubgroupCurrent: positive / negative scansAfter re-sampling: positive / negative scans
E-CoastAge ≤ 202 / 2230 / 122
MerlinBlack2 / 6930 / 289
MerlinPacific Islander1 / 1630 / 289

Bootstrap. Multiple phases of the same exam: yes (on E-Coast). Repeated scans of the same patient: no. When we treat all scans of a patient as one cluster, the 95% CIs barely change and no conclusion changes.

AuthorsOpenReview ↗

Q3. Audit of automatically accepted cases: unanimous negatives, missed-lesion rate, subgroups, and blinding

Q3. Clarification of audit results and labeling targets. Given that the authors did audit approximately 5% of automatically accepted cases, could they further report the number of cases with unanimous negative predictions in this audit, the rate of missed lesion labels detected, and any subgroup differences, and clarify whether readers were blinded to model outputs?

Thank you for this question. For the submission, radiologists audited the automatically accepted scans as follows (Appendix A.2):

  • About 5% of the automatically accepted scans were audited by hand. Of the concordant scans that were accepted automatically, 5,137 scans were audited, and 101 of them (1.97%) needed a label change. Among the unanimous-negative scans in the audit, 1.48% had a missed lesion. Among all audited scans, 1.19% had a missed lesion.

  • The readers were not blinded to model outputs. They saw the original CT, the model proposals and the available reports.

To validate the labels further, independent radiologists will re-check BenchX before its final release:

  • Every label will be re-checked, including all concordant scans. 20 radiologists who were not involved in the original annotation will review every scan together with its current label.

  • A blinded read will catch errors that the models share. 32 different radiologists will read a stratified random sample of scans from each cohort without seeing any existing label, model prediction or candidate segmentation. A second reader will independently re-read 10% of these scans, and senior readers will adjudicate disagreements.

  • We will report the missed-lesion rate by subgroup. We will report the number of scans checked, the label error rate and the missed-lesion rate with 95% CIs, overall and by subgroup, and the agreement between readers (κ).

Thank you again for your careful review. We hope our replies address your concerns, and we would be grateful if you would consider raising your score.

Reviewer_7139OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BenchX evaluates 12 pancreatic tumor models on 101,822 CT scans from six cohorts, with subgroup analyses by demographics and CT phase. It uses LLM-assisted metadata extraction, reports substantial performance disparities, and explores conditional tumor synthesis with SynthX.

Critical strengths
  1. The large, diverse benchmark and demographic/protocol metadata could provide a valuable resource for evaluating medical AI beyond average accuracy.
  2. Evaluation across multiple architectures and cohorts, together with covariate standardization, provides useful evidence of model robustness limitations.
Critical weaknesses
  1. The zero-F1 findings in Sec. 4.3 need subgroup positive counts and confidence intervals to distinguish systematic failures from small-sample effects. F1 also depends on prevalence and threshold selection. The adjusted comparisons in Sec. 4.4 do not directly validate the race gaps or the abstract's intersectional claim.
  2. Benchmark reliability needs clearer independent validation. Reviewing disagreements between two screening models (Sec. 3.1) may miss shared errors, particularly in difficult subgroups. Expert audits of concordant predictions and quantitative human validation of the LLM metadata are important to support the benchmark's central conclusions.
  3. SynthX adapts existing conditional diffusion methods, but its added value is unclear. The baseline-versus-SynthX comparison in Sec. 4.5 does not isolate demographic/phase conditioning from simply adding synthetic data. Matched prior-method comparisons and condition ablations would clarify this.
Questions for authors
  1. Please report the subgroup patient and positive-case counts underlying Fig. 6, and provide uncertainty estimates for the key detection results, particularly sensitivity in the zero-F1 groups.
  2. What independent expert audits validate concordant tumor labels and the LLM-extracted metadata? Please report sample sizes and error rates.
  3. How were detection thresholds selected, and what analyses support the race and intersectional claims? Please identify relevant supplementary tables.
  4. What distinguishes SynthX from prior conditional synthesis? Do matched comparisons or condition ablations show benefits beyond simply adding synthetic data?
5 comments
AuthorsOpenReview ↗

W1 & Q1. Are the zero-F1 groups systematic failures or small-sample effects? Counts and CIs behind Fig. 6

Thank you for the detailed review. We appreciate your view that "the large, diverse benchmark and demographic/protocol metadata could provide a valuable resource for evaluating medical AI beyond average accuracy" and that "evaluation across multiple architectures and cohorts, together with covariate standardization, provides useful evidence of model robustness limitations."

In brief: (1) the zero-F1 groups had few positive scans; we add test scans for them and report counts and CIs (W1 & Q1); (2) we report audit sample sizes and error rates (W2 & Q2); (3) thresholds were fixed before evaluation, and we list the tables behind each race result (Q3); (4) we add the SynthX ablation you suggested (W3 & Q4).

W1 & Q1. The zero-F1 findings in Sec. 4.3 need subgroup positive counts and confidence intervals to distinguish systematic failures from small-sample effects. F1 also depends on prevalence and threshold selection. The adjusted comparisons in Sec. 4.4 do not directly validate the race gaps or the abstract's intersectional claim. Please report the subgroup patient and positive-case counts underlying Fig. 6, and provide uncertainty estimates for the key detection results, particularly sensitivity in the zero-F1 groups.

Thank you for this important point. You are right that our description in Sec. 4.3 did not distinguish systematic failures from small-sample effects. Zero F1 occurs because these subgroups have few positive scans in our evaluation sets (table below), so a model that misses them gets an F1 of zero. For such a model, the 95% CI of sensitivity still extends up to 65.8–79.3%, so these results do not establish systematic failure.

CohortSubgroupPositive / negative scansSensitivity of zero-F1 models (95% CI)
E-CoastAge ≤ 202 / 220% (0–65.8%)
MerlinBlack2 / 690% (0–65.8%)
MerlinPacific Islander1 / 160% (0–79.3%)
  • Updated evaluation on expanded subgroups. Motivated by your comment, we re-sampled the test data so that each of these subgroups has 30 positive scans, as shown below. Due to time constraints during the rebuttal period, we have not yet re-evaluated all 12 models on the expanded test sets. We will complete this evaluation and include the updated detection results, subgroup patient and scan counts, and 95% confidence intervals in the revised paper.
CohortSubgroupPositive / negative scans (positive rate), before → after
E-CoastAge ≤ 202 / 22 (8.3%) → 30 / 122 (19.7%)
MerlinBlack2 / 69 (2.8%) → 30 / 289 (9.4%)
MerlinPacific Islander1 / 16 (5.9%) → 30 / 289 (9.4%)
  • Counts and prevalence. The next comment lists the positive and negative counts behind Fig. 6, at scan level and, where patient IDs exist, at patient level. Sensitivity and specificity do not depend on prevalence, so we use them as the main metrics for comparing subgroups and keep F1 as secondary.

  • Race in the adjusted analysis. You are correct that Sec. 4.4 does not include race. We have now added it on E-Coast. With the controls used in Sec. 4.4 (age, sex, CT phase and lesion category), sensitivity is lower for Black patients than for White patients (table below).

ModelBlack sensitivity (%)White sensitivity (%)
nnU-Net67.885.4
STU-Net-B62.279.7
VSmTrans79.484.6
  • The abstract's intersectional claim was not rigorous. You are right that the adjusted comparisons in Sec. 4.4 do not validate it. We will revise this claim in the abstract.

The revised paper will distinguish findings supported by the current data from those requiring additional evaluation, and replace the original categorical claims with conclusions supported by subgroup counts and uncertainty estimates.

AuthorsOpenReview ↗

W1 & Q1 (continued). Subgroup counts behind Fig. 6

Positive / negative scans for the subgroups in Fig. 6, with positive / negative patients in parentheses where patient IDs exist (E-Coast, N-California).

CohortAttributePositive / negative scans (patients)
E-CoastAge≤20: 2/22 (1/11); 20–40: 96/391 (49/199); 40–60: 708/929 (357/470); 60–80: 1,341/224 (679/114); >80: 162/2 (82/1)
E-CoastSexFemale: 1,187/958 (603/486); Male: 1,122/598 (566/303)
E-CoastPhaseArterial: 1,727/828 (1,726/828); Venous: 1,719/828 (1,718/828)
N-CaliforniaAge20–40: 3/56 (2/48); 40–60: 15/282 (13/237); 60–80: 49/467 (38/396); >80: 22/98 (17/89)
N-CaliforniaSexFemale: 45/409 (37/353); Male: 44/495 (33/412)
N-CaliforniaRaceAsian: 21/154 (16/128); Black: 6/82 (4/67); White: 51/525 (40/443)
N-CaliforniaPhaseNon-contrast: 36/467 (31/426); Arterial: 25/184 (21/172); Venous: 28/253 (21/189)
MerlinAge≤20: 1/19; 20–40: 4/194; 40–60: 18/300; 60–80: 27/296; >80: 26/91
MerlinSexFemale: 34/492; Male: 42/408
MerlinRaceAsian: 11/109; Black: 2/69; Pacific Islander: 1/16; White: 48/457
MerlinPhaseNon-contrast: 1/34; Arterial: 32/230; Venous: 43/636
PanTSAge≤20: 1/5; 20–40: 2/35; 40–60: 18/174; 60–80: 20/202; >80: 1/22
PanTSSexFemale: 29/193; Male: 13/256
PanTSPhaseNon-contrast: 13/121; Arterial: 27/135; Venous: 108/485; Delayed: 3/9
N-EuropeAge20–40: 5/1; 40–60: 116/22; 60–80: 429/99; >80: 359/55
N-EuropeSexFemale: 477/81; Male: 432/96
N-EuropePhaseNon-contrast: 77/12; Arterial: 179/36; Venous: 653/129
S-EuropeAge40–60: 20/387; 60–80: 21/442; >80: 2/57
S-EuropeSexFemale: 17/503; Male: 26/497
S-EuropeRaceTurkish: 35/894; Russian: 4/0
S-EuropePhaseNon-contrast: 1/26; Arterial: 4/120; Venous: 35/770
AuthorsOpenReview ↗

W2 & Q2. What independent expert audits validate the concordant tumor labels and the LLM-extracted metadata?

W2 & Q2. Benchmark reliability needs clearer independent validation. Reviewing disagreements between two screening models (Sec. 3.1) may miss shared errors, particularly in difficult subgroups. Expert audits of concordant predictions and quantitative human validation of the LLM metadata are important to support the benchmark's central conclusions. What independent expert audits validate concordant tumor labels and the LLM-extracted metadata? Please report sample sizes and error rates.

Thank you for this important point. We agree that reviewing only the disagreements between two screening models can miss errors that both models share. Appendix A.2 and A.3 report the expert checks done for the submission:

  • Concordant tumor labels were audited by radiologists. Eight radiologists reviewed every scan on which the segmentation and detection models disagreed. Of the concordant scans that were accepted automatically, about 5% (5,137 scans) were audited by hand, and 101 of them (1.97%) needed a label change. A second reader independently re-checked 10% of the reviewed scans (10,237 scans, 97.71% agreement), and senior readers resolved disagreements.

  • LLM-extracted metadata was checked against the reports. About 1,000 randomly sampled scans across cohorts were verified by hand (Appendix A.3, Table 2). The error rate (100% minus the accuracy in Table 2) was 0.4% for age, 0.7% for sex, 3.2% for race, 4.1% for CT phase and 5.2% for tumor attributes, and inter-rater agreement ranged from 94.2% to 99.1%. In addition, the two LLMs gave identical outputs for about 98% of scans, and a radiologist checked the other 2% against the report.

To validate the labels further, independent radiologists will re-check BenchX before its final release:

  • Every label will be re-checked, including all concordant scans. 20 radiologists who were not involved in the original annotation will review every scan together with its current label.

  • A blinded read will catch errors that the models share. 32 different radiologists will read a stratified random sample of scans from each cohort without seeing any existing label, model prediction or candidate segmentation. A second reader will independently re-read 10% of these scans, and senior readers will adjudicate disagreements.

  • We will report sample sizes and error rates. For tumor labels, we will report the number of scans checked, the label error rate and the missed-lesion rate with 95% CIs, overall and by subgroup, and the agreement between readers (κ). For metadata, we will report the accuracy of each field, including each race category.

We will complete this re-check before the final data release to ensure the quality and accuracy of the released labels and metadata.

Reviewer_ARADOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Collect datasets of abdominal CT scans for pancreatic cancer and extract diverse subgroup annotations (demographics, pathology characteristics, acquisition specifics) using LLMs and produce labels where missing using medical imaging models to produce a large and diverse evaluation dataset.

Use this dataset to benchmark 12 of the best-performing pancreatic cancer detection models and reveal severe performance problems in subgroups, which are not revealed in the typically presented summary performance metrics.

Critical strengths
  • The work highlights the importance of detailed analysis to reveal critical model failures on specific subgroups.
Critical weaknesses

Although the paper presents relevant methods and practices, ultimately the submission should be rejected for the following reasons:

  1. Lack of novelty:

    • Although the work itself is clearly solid, this paper does not introduce any novel benchmarking methods.
    • While acknowledging that there is substantial effort put into the work, the approach and methods used to produce the dataset are well-known.
  2. Lack of detailed analysis of observed disparity:

    • It is unclear whether the performance drop observed in some subgroups is due to inherent model flaws or whether it is simply a matter of lack of availability of these subgroups during model training. Thus one could ask: Are we testing the models for out-of-distribution performance or something else?
    • Clinical explanations are superficial.
  3. Usability: Half of the dataset described in the paper is not generally available to the community at this time, limiting the general relevance of the work.

The absence of either of the above weaknesses would have led to a much more favorable recommendation.

The following are points I would have liked to be more clearly and thoroughly explained in the paper, but the central reasons for rejections remain those listed above.

  • How should one expect CT images to differ by demographics, such as age, race and sex? Is it expected that there are significant differences between the subgroups in how the anatomy and pathologies are expressed that would impact model performance?
  • What are the differences between the various CT phases? Is it expected that performance will vary with CT phase? How?

Other questions I lack the expertise to assess:

  • Are the 12 models benchmarked in this work representative for state-of-the-art models in this domain (pancreatic cancer tumor detection)?

Other suggestions:

  • Figure 6: It is hard to read the difference in scores when they are somewhat similar in magnitude. Maybe use numbers or more easily discernible symbols instead?
Questions for authors

None

7 comments
AuthorsOpenReview ↗

W1. Lack of novelty: no new benchmarking method, and the dataset is built with well-known methods

Thank you for your review and for noting that "the work itself is clearly solid" and that it "highlights the importance of detailed analysis to reveal critical model failures on specific subgroups." You gave three reasons for rejection: novelty, the analysis of the disparities, and data availability. Our replies to W1–W3 answer them; the later comments answer your other questions.

W1. Lack of novelty. Although the work itself is clearly solid, this paper does not introduce any novel benchmarking methods. While acknowledging that there is substantial effort put into the work, the approach and methods used to produce the dataset are well-known.

We agree that BenchX does not introduce a new algorithm or metric. However, its contribution is a measurement that was not possible before, and the findings it yields. In a scientific paper, findings are often the most valuable contribution: they show the community where current models fall short and guide what future methods and benchmarks should address. BenchX provides such findings for pancreatic tumor detection. We describe both parts below.

  1. A measurement that did not exist. Public pancreatic tumor benchmarks report one average on venous-phase scans without demographic labels: MSD-Pancreas has 281 scans and PANORAMA 2,238 (Table 5). BenchX evaluates 12 models on 101,822 scans from six cohorts with age, sex, race and four CT phases. A subgroup gap can now be measured within a cohort, with counts and CIs, and compared across cohorts.
  2. Findings that existing benchmarks cannot reveal. Unlike existing benchmarks, which report only overall accuracy (Table 5), BenchX reveals where models perform inconsistently across patient demographics and imaging protocols, and its annotations enable single-factor comparisons that control for measured confounders (Sec. 1). For example, imaging protocol is a dominant factor, rivaling architecture (Sec. 5): on N-California, the best sensitivity on non-contrast scans (30.6%) is lower than the venous sensitivity of 9 of the 12 models, and 8 of 12 models are significantly less sensitive on non-contrast than on venous scans (Fisher exact test, p < 0.01). A single-protocol benchmark cannot show this, and an overall average hides it.

Many influential computer vision benchmarks and studies are measurements built from well-known methods. ImageNet [1] was built by collecting web images and verifying their labels on Amazon Mechanical Turk, and MMMU [2] collected 11.5K questions from college exams, quizzes and textbooks to evaluate existing multimodal models. CLEVR [3] is a diagnostic dataset used to analyze the abilities and limitations of existing visual reasoning systems, and VSI-Bench [4] built over 5,000 video question-answer pairs and found that multimodal LLMs show competitive but subhuman visual-spatial intelligence. ImageNet-C [5] applied common corruptions to ImageNet images to compare the robustness of existing classifiers. Torralba & Efros [6] showed that a standard classifier can tell which popular dataset an image comes from, and Liu & He [7] repeated this experiment with modern networks and found that it still holds. He et al. [8] trained standard Mask R-CNN models from random initialization and found them no worse than their ImageNet-pretrained counterparts on COCO. Tong et al. [9] built the MMVP benchmark from image pairs that CLIP perceives as similar and found that state-of-the-art multimodal LLMs, including GPT-4V, struggle with straightforward visual questions. Their value lies in what they measured. Two other reviewers describe BenchX the same way: "a valuable resource for evaluating medical AI beyond average accuracy" (7139) and "I view the dataset as a valuable contribution" (NdTS).

References

[1] Deng, Jia, et al. "ImageNet: A Large-Scale Hierarchical Image Database." CVPR (2009), pp. 248–255.

[2] Yue, Xiang, et al. "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI." CVPR (2024), pp. 9556–9567.

[3] Johnson, Justin, et al. "CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning." CVPR (2017), pp. 2901–2910.

[4] Yang, Jihan, et al. "Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces." CVPR (2025), pp. 10632–10643.

[5] Hendrycks, Dan, and Thomas Dietterich. "Benchmarking Neural Network Robustness to Common Corruptions and Perturbations." ICLR (2019).

[6] Torralba, Antonio, and Alexei A. Efros. "Unbiased Look at Dataset Bias." CVPR (2011), pp. 1521–1528.

[7] Liu, Zhuang, and Kaiming He. "A Decade's Battle on Dataset Bias: Are We There Yet?" ICLR (2025).

[8] He, Kaiming, Ross Girshick, and Piotr Dollár. "Rethinking ImageNet Pre-training." ICCV (2019), pp. 4918–4927.

[9] Tong, Shengbang, et al. "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs." CVPR (2024), pp. 9568–9578.

AuthorsOpenReview ↗

W2. Are the subgroup drops due to model flaws or to a lack of these subgroups in training?

W2. Analysis of the disparity. It is unclear whether the performance drop observed in some subgroups is due to inherent model flaws or whether it is simply a matter of lack of availability of these subgroups during model training. Thus one could ask: Are we testing the models for out-of-distribution performance or something else?

Thank you for this insightful question. Yes, BenchX tests out-of-distribution performance at the subgroup level: the models are trained on AbdomenAtlas 2.0, evaluated on six unseen cohorts, and subgroups are compared within each cohort. Training availability explains part of the drop, but not all of it. We explain below.

  1. Groups with fewer training scans do worse. In the training data, 64.7% of scans are venous, 26.4% arterial and only 8.4% non-contrast, and 65.2% are from male patients (Appendix K.2, Figure 24). The models are weakest on non-contrast and arterial scans, and on N-Europe they perform better for men than for women.
  2. But training data is not the only cause. The same models can do well or poorly on non-contrast scans, depending on where the scans come from. On PanTS, MedNeXt finds 12 of 13 tumors on non-contrast scans (92.3%); on N-California, no model finds more than 11 of 36 (30.6%). So the hospital, the scanner and the patients also matter.
  3. More training scans of a phase improve sensitivity on that phase. We fine-tuned the nnU-Net with the same settings (25K iterations): model A with 2,000 extra non-contrast scans (143 with a tumor), and model B with 2,000 extra contrast-enhanced scans. Both sets come from PanTS cases that are not used in any BenchX evaluation. We tested both on the five cohorts that have non-contrast CT (tables below, in %). Model A raises non-contrast sensitivity by 12.5 points (95% CI 3.9–21.2) and keeps specificity the same. Model B raises venous sensitivity by 6.0 points and arterial sensitivity by 12.4 points. So adding scans of a phase raises sensitivity on that phase.
OriginalA: + 2,000 non-contrast
Non-contrast46.959.4
Specificity81.881.8
OriginalB: + 2,000 contrast-enhanced
Venous56.462.4
Arterial46.458.8

In short, the drops come mainly from what the training data lacks, in CT phases and in hospitals, rather than from flaws of a specific model. BenchX tests out-of-distribution performance, at the subgroup level. We will add this analysis and the new experiment to the revised version.

AuthorsOpenReview ↗

W3. Usability: half of the dataset is not publicly available

W3. Usability. Half of the dataset described in the paper is not generally available to the community at this time, limiting the general relevance of the work.

Upon acceptance, 54,624 of 101,822 scans (53.6%) from four cohorts become publicly accessible, 24 times PANORAMA (2,238 scans) and 194 times MSD-Pancreas (281 scans). The public part alone has CT phase labels for 53,960 scans, age and sex labels for over 48,000 and race labels for 26,184, so subgroup analyses can be run on it directly.

The public cohorts include all four CT phases. For Merlin and PanTS we release our new scan-level tumor labels and the extracted metadata. N-California (42,096 scans) cannot be redistributed; it will serve third-party evaluation through a challenge (Appendix A.4), as ILSVRC [1] and MMMU [2] did by withholding their test-set annotations and scoring submissions on a server. Keeping an unseen cohort private also protects out-of-distribution evaluation from test-set leakage [3].

References

[1] Russakovsky, Olga, et al. "ImageNet Large Scale Visual Recognition Challenge." IJCV 115 (2015), pp. 211–252.

[2] Yue, Xiang, et al. "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI." CVPR (2024), pp. 9556–9567.

[3] Recht, Benjamin, et al. "Do ImageNet Classifiers Generalize to ImageNet?" ICML (2019), pp. 5389–5400.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.