- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces PULSE, a controlled evaluation framework for examining model behavior that aggregate performance metrics may obscure, demonstrated on chest X-ray nodule detection.
- Method. Analytic nodule-like probes with controlled size, contrast, intensity profile, morphology, and position are inserted into 800 nodule-negative NODE21 radiographs, divided into 500 discovery and 300 held-out hosts. Fourteen detectors spanning seven families and two training seeds are evaluated using a localized detection criterion that requires both an operating-threshold crossing and a minimum score increase over the original image. The resulting behavioral atlas summarizes detection coverage through RSFF and differences in standardized response-profile shape through BPD.
- Main findings. Models with similar conventional pFROC scores exhibit several-fold differences in probe coverage. At the largest tested probe size, increasing contrast reduces detection in both DINO seeds while driving Faster R-CNN toward saturation. Replacing the smooth-disk probe with a peak-matched Gaussian reduces detection across all models, and all detectors exhibit lower detection rates in the central-medial region than in the lower-visible lung region.
Critical strengths
1. A carefully controlled evaluation design with substantial evidence of reproducibility.
- The study compares detectors using matched interventions on common host images, controls unchanged probe parameters in paired comparisons, and estimates uncertainty through host-level bootstrap resampling. It also examines whether response patterns recur across discovery and held-out host banks and across two training seeds per detector family, alongside a prespecified confirmation study involving the original six models.
- These checks provide convincing evidence that key findings—particularly the response-profile geometry and regional detection deficits—are reproducible within the evaluated setting.
2. A convincing demonstration that similar aggregate performance can conceal substantially different conditional behavior.
- The paper provides concrete examples of detectors with similar conventional pFROC scores but several-fold differences in probe detection coverage, as well as contrasting responses to increasing probe contrast. These results show that aggregate performance alone does not adequately characterize how detectors respond to specific combinations of lesion-like appearance and anatomical background.
- The aligned response atlas makes these differences directly observable and comparable across models. Although the implications for natural lesions remain incompletely validated, the central argument for complementing aggregate scores with controlled conditional evaluation is convincingly supported within the analytic-probe setting.
Critical weaknesses
1. The connection between probe-derived response patterns and behavior on real lesions remains insufficiently validated.
- The central findings—including model-specific contrast responses, sensitivity to intensity profiles, and regional detection deficits—are established using analytic probes that the authors explicitly distinguish from the natural lesion distribution. The natural-lesion assay provides useful partial evidence: matched replacements recover detector responses to individual real lesions more closely than lesion erasure does. However, pointwise response matching does not establish that trends observed when varying analytic probe properties also hold for natural lesions. Appendix G explicitly states that the available natural-lesion support was insufficient to test this transfer.
- Consequently, it remains unclear whether the atlas identifies clinically relevant failure patterns or primarily characterizes sensitivity to the chosen synthetic stimuli. Validation of selected atlas-derived predictions on independent real nodules—for example, through analyses stratified by size, contrast, or location—would substantially strengthen the practical significance of the contribution.
2. The framework’s broader applicability remains untested, and key reporting details are difficult to verify from the manuscript alone.
-
PULSE is introduced as a general approach to testing learned systems, but its empirical demonstration is confined to chest X-ray nodule detection, with the main controlled atlas constructed on NODE21 hosts. The inclusion of fourteen detectors and multiple seeds strengthens the evidence within this setting, and the VinDr-CXR natural-lesion assay provides additional support, but neither establishes that the controlled-response findings extend to another host dataset, detection target, or modality. This limits the demonstrated scope of the framework, even if broader application is plausible.
-
The presentation also makes some results unnecessarily difficult to assess. pFROC and requested CNR appear before their operational explanations, while the placement rules underlying “geometry-valid support” are deferred to the appendix despite determining which observations enter the reported rates. The C5 “projected-field sign pattern” is named and reported as passing, but its precise statistic and decision rule are not sufficiently explained in the provided manuscript, which refers readers to a companion document. Bringing these definitions and essential decision rules into the relevant sections would make the findings more interpretable and independently assessable.
Questions for authors
-
Can the authors conduct experiments on different datasets, including non-CXR modalities such as mammography, to support the claim that PULSE is a general framework for testing learned systems?
-
Can the authors repeat the main atlas using nodule-negative hosts from another CXR dataset (e.g., VinDr-CXR, ChestX-ray14 or CheXpert)?
- This would show whether the findings reflect the detectors themselves or the NODE21 host population.
[Suggestions for future manuscript]
- Make the manuscript self-contained.
- Define pFROC, requested CNR, RSFF, geometry-valid support and the C1–C6 statistics at first use, or add a notation table. Move the prespecified decision rules, especially C5, from the companion into the appendix.
3 comments
I thank the authors for their response. My two major concerns, particularly the one about the diversity of datasets and modalities, have been reasonably addressed. A gap between natural lesions and synthesized probes is unavoidable, and some of the generated results still look somewhat unnatural from a clinical standpoint. However, given the systematic, well-reasoned design and the results supporting it, I find this acceptable.
Accordingly, I am raising my score to 3 (weak accept).
For the final assessment of this paper, I believe the following points raised by the other reviewers should be taken into account:
Hypotheses explaining the differences in model behavior (Reviewer yC9a). The authors have offered some explanation, but the paper would be stronger if these hypotheses were backed by more substantial evidence.
Methodological novelty (Reviewer mqWL). I agree that "controlled perturbations, synthetic lesion insertion, and spatial bias evaluation" have been explored to some extent in prior work. For medical AI papers, however, I believe the emphasis should be less on the novelty of the method itself and more on how existing techniques are adapted and how convincingly their effectiveness is demonstrated. From this perspective, I think the authors have presented a sufficiently strong framework. That said, if another reviewer or the area chair regards novelty as the foremost criterion, I would respect a final decision that reflects this.
In addition, I have two minor concerns of my own. They did not meaningfully affect my recommendation, but I think they are worth the AC's consideration in the final decision.
Extension to other lesion types. Although the authors have shown solid results on additional datasets and modalities, the method remains limited to nodules. Clinically important findings include many other lesion types (for example, datasets such as VinDr-CXR and PadChest-GR provide bounding boxes for a variety of lesions). Extending the method to non-nodule lesions would be a valuable direction for future work.
Paper presentation. The presentation has improved over the previous version, but many parts, from the abstract through the conclusion, could still be made clearer and more concise. I recognize that this is not feasible to request during the rebuttal period, and it did not significantly affect my score, but I encourage the authors to revisit the overall organization and writing in a later revision.
We sincerely thank the reviewer for the thoughtful reassessment and constructive suggestions.
-
We appreciate the reviewer's recognition of the value of our systematic measurement framework. Regarding model behavior, our additional analyses provide evidence about where the observed differences emerge and how consistently they persist, while further mechanistic investigation remains a valuable direction.
-
We intentionally focused this study on cancer-related lesions—pulmonary nodules and mammographic masses—to establish and validate the framework in a focused setting. We agree that extending to other clinically important lesion types is a natural and important next step that we plan to pursue.
-
We also appreciate the suggestions on presentation. We will further refine the writing for clarity and conciseness, taking into account the reviewer's comments and feedback from the other reviewers.
Thank you again for the constructive evaluation and for recognizing the contribution of our work.
Response to JaEH
We thank the reviewer for recognizing the controlled design and the evidence that aggregate scores obscure conditional behavior. We have substantially improved the presentation and added external CXR and mammography experiments addressing your questions.
Questions for Authors
Q1. Can PULSE be evaluated on other datasets and non-CXR modalities?
Yes. We extended the paired-intervention framework to four mammography image-level classifiers (GMIC, GLAM, DMV-CNN, Mammo-CLIP), measuring changes in malignancy scores rather than bounding-box detections.
In the primary study, all eight prespecified size/contrast effects were positive (152 patients for size, 151 for contrast; Holm p=0.003 each). In a separate spatial study, all four response maps recurred in 150 new patients (r=0.848–0.958; one-sided 95% lower bounds 0.717–0.935), and every classifier showed a smaller response to 15 mm than 10 mm probes at CNR 3. The new-patient cohort comes from another acquisition site within the same RSNA dataset, not an entirely independent mammography dataset (Sec. 5.5, p. 8; Fig. 5, p. 9; Tables 20–21, pp. 38–39).
Q2. Can the main atlas be repeated on another CXR dataset using nodule-negative hosts?
We added 300 VinDr-CXR and 300 Shenzhen nodule-negative hosts, using the original probes, frozen checkpoints, and NODE21 calibration thresholds. All 14 detectors were evaluated on the external global bank; the prespecified spatial evaluation covers FR1/RD1 rather than a full 14-detector spatial atlas.
Both principal effects replicated in both domains: the FR1−DN2 detection-rate gap was 0.250/0.252, and DN1's localized-score change from CNR 2 to 3 was −0.239/−0.314 (all four Holm p<0.001). The regional reversal H3 transferred more narrowly: supported on VinDr-CXR at the registered operating point and partial on Shenzhen (Sec. 5.4, pp. 7–8; Fig. 4, p. 8; Tables 17–19, pp. 35–37).
Critical Weaknesses
W1. Connection to natural lesions.
We added a natural-lesion comparison on annotated radiographs. On 278 images, fitted procedural replacements reproduced the mean real-lesion presence effect within a prespecified ±0.05 equivalence margin for all six original detectors (informatively for five). On 463 overlapping images, replacements approximated real-lesion scores more closely than erasure or a constant reference (Sec. 5.6, p. 10; Tables 23–24, pp. 41–42).
This supports mean-level response agreement, not transfer of synthetic size-, contrast-, or location-dependent trends to natural nodules. We state this remaining validation boundary explicitly (Appendix H.3).
W2. Broader applicability and reporting clarity.
The mammography extension and external CXR evaluation broaden the empirical evidence beyond the original NODE21 atlas. We also made the manuscript and Supplement self-contained:
In particular, C5 specifies the projected-field minus lower-visible detection-rate contrast, model-specific expected signs, five passing requirements, and minimum support of 240 hosts. All requirements passed on 292 held-out hosts, with a sign-aligned mean of 0.139 [0.132, 0.147] (Tables 15–16, pp. 33–34).
Suggestions for the Manuscript
We integrated the formerly separate methodological and confirmation details into Supplement Appendices A–H. Table 2 (p. 16) now links each Main result to its evidence level and supporting appendix, allowing readers to verify the definitions and decision rules directly.
The original CXR findings and C1–C6 outcomes remain unchanged; the revised presentation and targeted validation strengthen their interpretation and independently assessable scope.