- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents DREAM (Diagnostic Robustness by Error Analysis and Modeling), a framework designed to improve the robustness of CT-based cancer detection by addressing two systematic failure modes: false negatives on small early-stage tumors and false positives caused by non-cancerous abnormalities. DREAM consists of three components. First, an LLM-based pipeline mines challenging positive and negative cases from approximately 440,000 radiology reports. Second, report-refined masks incorporate tumor size and anatomical location information from clinical reports to improve automatically generated segmentation pseudo-labels. Third, a cascade verifier combines segmentation-derived evidence with radiomics-based abnormality classification to refine tumor detection scores. The framework is evaluated on pancreatic, bladder, and gallbladder tumor detection using internal and external cohorts. The authors report substantial improvements over representative segmentation baselines, particularly on challenging cases, and provide component ablations, cross-backbone evaluations, and a radiologist reader study.
Critical strengths
- Clinically meaningful problem formulation and scalable hard-case acquisition.
The paper addresses two important failure modes of medical AI systems that are often insufficiently represented in conventional benchmark evaluations: missed early tumors and false alarms caused by non-cancer abnormalities.
The report-based mining pipeline provides a practical approach to identifying rare and diagnostically challenging cases without exhaustively processing large hospital imaging archives. Integrating report-derived information into training data construction is particularly valuable in settings where dense expert annotations are expensive.
The proposed framework therefore offers a meaningful systems-level contribution to improving medical AI robustness.
- Strong empirical validation across multiple organs and external cohorts.
The evaluation is a major strength of this submission. DREAM demonstrates improvements across pancreatic, bladder, and gallbladder tumor detection, including external validation on independently collected datasets.
The authors provide controlled ablations showing the contributions of hard-case mining, report-refined masks, and cascade verification. Additional experiments indicate that the training strategy can also benefit alternative segmentation backbones.
The hard-negative specificity analysis and small-tumor detection results are especially relevant because they directly evaluate the failure modes motivating the proposed approach.
Critical weaknesses
- Insufficient independent validation of report-refined masks and limitations of the cascade verification mechanism.
Although report-refined masks are an important component of DREAM, their spatial accuracy is not sufficiently validated against expert annotations.
The proposed refinement procedure uses reported tumor diameters and anatomical locations to constrain existing model predictions. However, matching the approximate tumor size and organ subregion does not guarantee accurate three-dimensional localization or boundary delineation, particularly when the initial segmentation model produces unreliable predictions.
The quality-filtering strategy relies on agreement between the original prediction and the refined pseudo-mask. Such agreement does not necessarily establish anatomical correctness, since both representations depend on the same underlying model. The manuscript mentions radiologist review of only 20 masks for selecting the filtering threshold. A larger independent expert-annotated evaluation would provide stronger evidence for the reliability of the generated supervision.
The cascade verifier also has a conceptual limitation. According to Eq. (2), when the segmentation score and cascade score differ by at least 0.5, the correction term becomes zero, and the final prediction remains identical to the original segmentation score. Therefore, strong disagreements cannot be corrected by the verifier, even when they arise from a potentially severe segmentation error.
Furthermore, the Trust Gate relies on rank correlation with the original segmentation score, which does not necessarily measure whether a cascade branch can correct segmentation mistakes. Additional comparisons with alternative calibration and fusion strategies would help establish the necessity of the proposed design.
- Some evaluation claims require more careful qualification and clarification.
The external validation is encouraging, but several aspects limit the interpretation of the results.
First, the PanTS benchmark uses binary pancreatic-lesion annotations without distinguishing malignant from non-malignant lesion subtypes, as acknowledged in Appendix G. Consequently, performance on this dataset supports pancreatic lesion detection rather than specifically establishing pancreatic cancer detection performance.
Second, the radiologist reader study includes only 74 enriched cases and three readers, two of whom are relatively junior. Although the reported improvement is promising, the small study and absence of prior imaging or clinical context limit its generalizability to routine radiological practice.
Third, the reported E-Coast training cohort size differs between the main text (3,151) and Appendix G (3,440). The authors should clarify the origin of this discrepancy and specify the exact cohort used for each training stage.
Finally, the detailed component ablations primarily focus on pancreatic tumor detection. Although the full pipeline is evaluated across three organs, organ-specific ablations would strengthen the claim that each component contributes consistently across different cancer types.
These concerns do not invalidate the reported improvements, but resolving them would improve the rigor and interpretability of the clinical and methodological claims.
Questions for authors
Q1. Reliability of report-refined pseudo-labels.
Could the authors provide a quantitative comparison between report-refined masks and radiologist-drawn tumor annotations on an independent subset of cases? In particular, how frequently does matching the reported tumor diameter result in incorrect tumor localization or boundary delineation? Such an analysis would help establish whether the proposed quality-filtering criterion reliably identifies inaccurate pseudo-labels.
Q2. Design and limitations of the cascade verifier.
Equation (2) suppresses the correction term when the segmentation and cascade scores strongly disagree. How does the verifier handle high-confidence false-positive predictions in which the radiomics classifier correctly indicates a non-cancer abnormality? Could the authors compare the proposed trust-gated fusion against standard weighted averaging, learned calibration, and alternative confidence-based fusion strategies? A component-level analysis of hard-negative cases would be particularly informative.
Q3. Generalizability of the three components.
The detailed ablation study focuses primarily on pancreatic cancer detection. Could the authors provide corresponding component-level results for bladder and gallbladder tumor detection? This would help determine whether the improvements from report mining, pseudo-mask refinement, and cascade verification are consistent across organs or primarily driven by the larger pancreatic cohort.
Q4. Cohort definitions and clinical interpretation.
Could the authors clarify the discrepancy between the E-Coast sample counts reported in the main text and Appendix G? Additionally, given that PanTS contains pancreatic-lesion annotations without subtype labels, could the authors more explicitly distinguish lesion-detection results from malignancy-specific detection claims?
5 comments
W1 & Q2. Eq. (2) makes no correction when the segmentation and cascade scores differ by 0.5 or more, so the verifier cannot fix a confident segmentation error. How are such false positives handled, how does the rule compare with weighted averaging, learned calibration and confidence-based fusion, and what does each component do on hard negatives?
(1) High-confidence false positives that the radiomics arm flags as non-cancer.
DREAM targets these before the verifier, through hard-negative mining: we fine-tune the segmentation model on 673 report-mined hard negatives. On 656 pancreatic non-cancer abnormalities, the full DREAM pipeline raises specificity over the nnU-Net baseline from 57.0% to 74.4%, with gains up to +40.7 percentage points (pp) on cancer look-alikes such as fluid collection (Table 2; see (3) for the components).
When the segmentation and cascade scores strongly disagree, the verifier leaves the segmentation score unchanged by design. When the segmentation score D and the cascade score C differ by 0.5 or more, the agreement factor in Eq. (2) is zero, so the correction is zero. On PanTS and N-California, all four non-cancer scans with D ≥ 0.9 (three on PanTS, one on N-California) had C < 0.5 and received zero correction. These four cases therefore remain uncorrected by the verifier.
We retain segmentation as the primary score because spatial supervision can improve robustness to spurious correlations [1], while background bias can impair classifier generalization [2]. For example, suppose all cancer scans came from hospital A and all normal scans from hospital B. A classifier could then detect cancer from a hospital mark in the image. Segmentation supervision additionally requires the model to localize the tumor, which the mark alone does not identify. This provides an additional constraint against such shortcuts, although it does not eliminate them. DREAM therefore allows the classifiers to make bounded adjustments to the segmentation score.
Our fusion experiments show the generalization trade-off. A weighted average whose weight we learned on the calibration split put all of the weight on the arms. It raised AUC on N-California, the hospital system the classifiers were trained on, from 92.77 (D alone) to 98.25, but lowered AUC on the external PanTS cohort from 90.69 to 83.45. Our rule scored above D alone on both (97.29 and 91.47; see (2)).
(2) Comparison with weighted averaging, learned calibration and confidence-based fusion.
The table compares our rule with four alternatives on the three test cohorts. We fitted every weight, cutoff and gate decision on the 543-scan calibration split and froze them before scoring any test cohort. Values are AUC (%), with the change versus our rule in percentage points shown in parentheses.
| Rule | PanTS | N-California | Euro |
|---|---|---|---|
| D alone (segmentation only) | 90.7 (−0.8) | 92.8 (−4.5) | 96.2 (−0.9) |
| Ours: Eq. (2) | 91.5 | 97.3 | 97.1 |
| Weighted average, weight learned on the calibration split | 83.5 (−8.0) | 98.3 (+1.0) | 92.5 (−4.6) |
| Logistic fusion of D and all three arms | 49.9 (−41.6) | 95.1 (−2.2) | 68.7 (−28.4) |
| Confidence-based selector | 87.8 (−3.6) | 93.9 (−3.4) | 92.9 (−4.2) |
Values are AUC (%), with differences versus our rule in percentage points
The learned weighted average, logistic fusion and confidence selector fall below D alone on PanTS and Euro, where the classifier arms transfer poorly. The learned calibration we tested preserved the ranking of D, so its AUC matched D alone.
(3) What each component does on hard negatives.
Hard-negative mining directly introduces cancer look-alikes into segmentation training, and how the cases are mined matters. Mining hard cases from images instead of reports leaves more false alarms on cancer look-alikes: specificity on duct dilatation falls from 69.6% to 59.4%, and on atrophy from 86.4% to 77.2% (Appendix E).
The verifier can move a score by at most 2.5 percentile points, and only when D and C differ by less than 0.5. It therefore makes small adjustments to the ranking but cannot correct the high-scoring false positives for which segmentation and cascade scores differ by 0.5 or more.
References
[1] Hooper, Sarah, Mayee Chen, Khaled Saab, Kush Bhatia, Curtis Langlotz, and Christopher Ré. “A case for reframing automated medical image classification as segmentation.” NeurIPS (2023).
[2] Bassi, Pedro R. A. S., Sergio S. J. Dertkigil, and Andrea Cavalli. “Improving deep neural network generalization and robustness to background bias via layer-wise relevance propagation optimization.” Nature Communications 15, 291 (2024).
W2 & Q3. The component ablations cover only the pancreas. Do report mining, mask refinement and cascade verification also help bladder and gallbladder detection, or are the gains driven by the larger pancreatic cohort?
Thank you for this suggestion. We extended the ablation study to the external Euro cohorts for bladder (171 tumors, 227 non-cancer scans) and gallbladder (28 tumors, 243 non-cancer scans). The full DREAM pipeline achieves the highest AUC and sensitivity among the evaluated configurations for both organs, supporting its generalizability beyond pancreatic cancer.
We report AUC and sensitivity at a 95% specificity target, selecting the lowest threshold achieving at least 95% specificity on each test cohort.
| Model | Bladder AUC | Bladder sensitivity (%) | Gallbladder AUC | Gallbladder sensitivity (%) |
|---|---|---|---|---|
| nnU-Net, no fine-tuning | 0.807 | 25.7 | 0.843 | 46.4 |
| Random cases instead of report-mined | 0.809 | 24.8 | 0.818 | 39.3 |
| Report-mined cases, no refinement | 0.810 | 26.4 | 0.834 | 39.3 |
| DREAM, full pipeline | 0.834 | 29.0 | 0.895 | 53.6 |
Compared with the base nnU-Net model, DREAM improves AUC by 2.7 percentage points for bladder and 5.2 percentage points for gallbladder, with sensitivity gains of 3.3 and 7.2 percentage points, respectively. It also outperforms both random-case fine-tuning and report-mined training without refinement on both metrics in both organs. Thus, although the magnitude of improvement varies across organs, the benefit of the complete DREAM pipeline is observed in both additional external cohorts and is not confined to pancreatic cancer.
W1 & Q1. Report-refined masks are not validated against expert annotations, and the quality filter compares each mask with the same model's output. On an independent expert-annotated set, how often are refined masks wrongly placed or delineated, and does the filter reliably catch inaccurate pseudo-labels?
Thank you; we ran this comparison on an independent expert-annotated set. Report-refined masks are inexpensive silver-standard labels: they need no manual drawing, but they are approximate masks, and the quality filter aims to remove inaccurate ones. Even so, training with them performed better than training with the model’s raw masks.
On an independent public cohort (PANORAMA: 365 PDAC CTs, where lesions were manually delineated by trained investigators supervised by an expert pancreatic radiologist), we ran the refinement exactly as shipped, giving it only the initial model’s output, the organ (pancreas), and a tumor diameter measured from the expert mask as a radiologist would report it. We quantitatively compared the raw and retained refined masks against the expert annotations:
DREAM’s label pipeline keeps 99 of 365 masks (27%). The retained refined masks have better aggregate agreement with the expert annotations than the raw masks evaluated across the full cohort: median Dice 0.57 vs 0.47, NSD 0.41 vs 0.36, and HD95 14.4 vs 28.5 mm. No retained mask had zero overlap with the expert annotation, compared with 6.3% of raw masks.
These results quantify the quality of the pseudo-masks retained by the combined refinement-and-filtering pipeline and its trade-off between mask quality and training-set coverage. The retained masks remain approximate labels rather than expert-quality delineations.