acceptodds
Under review as a conference paper at ICLR 2027

Beyond Aggregate Scores: Revealing Conditional Model Behavior in Chest X-Ray Nodule Detection

Abstract

Aggregate model performance can rank systems without revealing the conditions under which their behavior diverges. We introduce PULSE, a controlled-response evaluation framework, and instantiate it in chest X-ray nodule detection. Analytic nodule-like probes vary in size, contrast, morphology and position on fixed chest radiographs, making fourteen detectors (seven families, two seeds each) directly comparable under matched conditions. The resulting behavioral atlas reveals sharply different behaviors behind similar conventional performance. At the largest tested probe size, increasing requested contrast drives Faster R-CNN toward saturation but reduces detection in both DINO seeds. Changing the intensity profile from a smooth disk to a peak-matched Gaussian also produces markedly different losses across models. These are structured differences: response-profile distances recur across discovery and held-out host banks (Spearman ), and every model is closest to its other seed. An earlier prespecified six-model study confirms the coarse geometry and regional patterns. Measurement-design analyses show that recovering an ordering, finding disagreements and reconstructing a response field require different sampling strategies. A matched natural-lesion assay provides partial pointwise behavioral support. Controlled evaluation makes conditional model behavior visible, reproducible and testable, complementing aggregate performance.

Primary area:
datasets and benchmarks
Keywords:
model evaluation, behavioral evaluation, conditional evaluation, datasets and benchmarks, object detection, medical imaging, chest X-ray, pulmonary nodule detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JaEHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces PULSE, a controlled evaluation framework for examining model behavior that aggregate performance metrics may obscure, demonstrated on chest X-ray nodule detection.

  • Method. Analytic nodule-like probes with controlled size, contrast, intensity profile, morphology, and position are inserted into 800 nodule-negative NODE21 radiographs, divided into 500 discovery and 300 held-out hosts. Fourteen detectors spanning seven families and two training seeds are evaluated using a localized detection criterion that requires both an operating-threshold crossing and a minimum score increase over the original image. The resulting behavioral atlas summarizes detection coverage through RSFF and differences in standardized response-profile shape through BPD.
  • Main findings. Models with similar conventional pFROC scores exhibit several-fold differences in probe coverage. At the largest tested probe size, increasing contrast reduces detection in both DINO seeds while driving Faster R-CNN toward saturation. Replacing the smooth-disk probe with a peak-matched Gaussian reduces detection across all models, and all detectors exhibit lower detection rates in the central-medial region than in the lower-visible lung region.
Critical strengths

1. A carefully controlled evaluation design with substantial evidence of reproducibility.

  • The study compares detectors using matched interventions on common host images, controls unchanged probe parameters in paired comparisons, and estimates uncertainty through host-level bootstrap resampling. It also examines whether response patterns recur across discovery and held-out host banks and across two training seeds per detector family, alongside a prespecified confirmation study involving the original six models.
  • These checks provide convincing evidence that key findings—particularly the response-profile geometry and regional detection deficits—are reproducible within the evaluated setting.

2. A convincing demonstration that similar aggregate performance can conceal substantially different conditional behavior.

  • The paper provides concrete examples of detectors with similar conventional pFROC scores but several-fold differences in probe detection coverage, as well as contrasting responses to increasing probe contrast. These results show that aggregate performance alone does not adequately characterize how detectors respond to specific combinations of lesion-like appearance and anatomical background.
  • The aligned response atlas makes these differences directly observable and comparable across models. Although the implications for natural lesions remain incompletely validated, the central argument for complementing aggregate scores with controlled conditional evaluation is convincingly supported within the analytic-probe setting.
Critical weaknesses

1. The connection between probe-derived response patterns and behavior on real lesions remains insufficiently validated.

  • The central findings—including model-specific contrast responses, sensitivity to intensity profiles, and regional detection deficits—are established using analytic probes that the authors explicitly distinguish from the natural lesion distribution. The natural-lesion assay provides useful partial evidence: matched replacements recover detector responses to individual real lesions more closely than lesion erasure does. However, pointwise response matching does not establish that trends observed when varying analytic probe properties also hold for natural lesions. Appendix G explicitly states that the available natural-lesion support was insufficient to test this transfer.
  • Consequently, it remains unclear whether the atlas identifies clinically relevant failure patterns or primarily characterizes sensitivity to the chosen synthetic stimuli. Validation of selected atlas-derived predictions on independent real nodules—for example, through analyses stratified by size, contrast, or location—would substantially strengthen the practical significance of the contribution.

2. The framework’s broader applicability remains untested, and key reporting details are difficult to verify from the manuscript alone.

  • PULSE is introduced as a general approach to testing learned systems, but its empirical demonstration is confined to chest X-ray nodule detection, with the main controlled atlas constructed on NODE21 hosts. The inclusion of fourteen detectors and multiple seeds strengthens the evidence within this setting, and the VinDr-CXR natural-lesion assay provides additional support, but neither establishes that the controlled-response findings extend to another host dataset, detection target, or modality. This limits the demonstrated scope of the framework, even if broader application is plausible.

  • The presentation also makes some results unnecessarily difficult to assess. pFROC and requested CNR appear before their operational explanations, while the placement rules underlying “geometry-valid support” are deferred to the appendix despite determining which observations enter the reported rates. The C5 “projected-field sign pattern” is named and reported as passing, but its precise statistic and decision rule are not sufficiently explained in the provided manuscript, which refers readers to a companion document. Bringing these definitions and essential decision rules into the relevant sections would make the findings more interpretable and independently assessable.

Questions for authors
  1. Can the authors conduct experiments on different datasets, including non-CXR modalities such as mammography, to support the claim that PULSE is a general framework for testing learned systems?

  2. Can the authors repeat the main atlas using nodule-negative hosts from another CXR dataset (e.g., VinDr-CXR, ChestX-ray14 or CheXpert)?

  • This would show whether the findings reflect the detectors themselves or the NODE21 host population.

[Suggestions for future manuscript]

  • Make the manuscript self-contained.
    • Define pFROC, requested CNR, RSFF, geometry-valid support and the C1–C6 statistics at first use, or add a notation table. Move the prespecified decision rules, especially C5, from the companion into the appendix.
3 comments
AuthorsOpenReview ↗

Response to JaEH

We thank the reviewer for recognizing the controlled design and the evidence that aggregate scores obscure conditional behavior. We have substantially improved the presentation and added external CXR and mammography experiments addressing your questions.

Questions for Authors

Q1. Can PULSE be evaluated on other datasets and non-CXR modalities?

Yes. We extended the paired-intervention framework to four mammography image-level classifiers (GMIC, GLAM, DMV-CNN, Mammo-CLIP), measuring changes in malignancy scores rather than bounding-box detections.

In the primary study, all eight prespecified size/contrast effects were positive (152 patients for size, 151 for contrast; Holm p=0.003 each). In a separate spatial study, all four response maps recurred in 150 new patients (r=0.848–0.958; one-sided 95% lower bounds 0.717–0.935), and every classifier showed a smaller response to 15 mm than 10 mm probes at CNR 3. The new-patient cohort comes from another acquisition site within the same RSNA dataset, not an entirely independent mammography dataset (Sec. 5.5, p. 8; Fig. 5, p. 9; Tables 20–21, pp. 38–39).

Q2. Can the main atlas be repeated on another CXR dataset using nodule-negative hosts?

We added 300 VinDr-CXR and 300 Shenzhen nodule-negative hosts, using the original probes, frozen checkpoints, and NODE21 calibration thresholds. All 14 detectors were evaluated on the external global bank; the prespecified spatial evaluation covers FR1/RD1 rather than a full 14-detector spatial atlas.

Both principal effects replicated in both domains: the FR1−DN2 detection-rate gap was 0.250/0.252, and DN1's localized-score change from CNR 2 to 3 was −0.239/−0.314 (all four Holm p<0.001). The regional reversal H3 transferred more narrowly: supported on VinDr-CXR at the registered operating point and partial on Shenzhen (Sec. 5.4, pp. 7–8; Fig. 4, p. 8; Tables 17–19, pp. 35–37).

Critical Weaknesses

W1. Connection to natural lesions.

We added a natural-lesion comparison on annotated radiographs. On 278 images, fitted procedural replacements reproduced the mean real-lesion presence effect within a prespecified ±0.05 equivalence margin for all six original detectors (informatively for five). On 463 overlapping images, replacements approximated real-lesion scores more closely than erasure or a constant reference (Sec. 5.6, p. 10; Tables 23–24, pp. 41–42).

This supports mean-level response agreement, not transfer of synthetic size-, contrast-, or location-dependent trends to natural nodules. We state this remaining validation boundary explicitly (Appendix H.3).

W2. Broader applicability and reporting clarity.

The mammography extension and external CXR evaluation broaden the empirical evidence beyond the original NODE21 atlas. We also made the manuscript and Supplement self-contained:

  • Requested CNR and validity: Sec. 3.1 (p. 3), Appendix A.1.
  • E0 and RSFF: Secs. 3.2–3.3 (pp. 3–4), Appendix A.2–A.3.
  • pFROC and calibration: Sec. 4 (p. 4), Appendix B.3.
  • C1–C6 decision rules: Appendix F.1 (pp. 33–34), with the complete passing, partial, and non-estimable criteria.

In particular, C5 specifies the projected-field minus lower-visible detection-rate contrast, model-specific expected signs, five passing requirements, and minimum support of 240 hosts. All requirements passed on 292 held-out hosts, with a sign-aligned mean of 0.139 [0.132, 0.147] (Tables 15–16, pp. 33–34).

Suggestions for the Manuscript

We integrated the formerly separate methodological and confirmation details into Supplement Appendices A–H. Table 2 (p. 16) now links each Main result to its evidence level and supporting appendix, allowing readers to verify the definitions and decision rules directly.

The original CXR findings and C1–C6 outcomes remain unchanged; the revised presentation and targeted validation strengthen their interpretation and independently assessable scope.

I thank the authors for their response. My two major concerns, particularly the one about the diversity of datasets and modalities, have been reasonably addressed. A gap between natural lesions and synthesized probes is unavoidable, and some of the generated results still look somewhat unnatural from a clinical standpoint. However, given the systematic, well-reasoned design and the results supporting it, I find this acceptable.

Accordingly, I am raising my score to 3 (weak accept).

 

For the final assessment of this paper, I believe the following points raised by the other reviewers should be taken into account:

Hypotheses explaining the differences in model behavior (Reviewer yC9a). The authors have offered some explanation, but the paper would be stronger if these hypotheses were backed by more substantial evidence.

Methodological novelty (Reviewer mqWL). I agree that "controlled perturbations, synthetic lesion insertion, and spatial bias evaluation" have been explored to some extent in prior work. For medical AI papers, however, I believe the emphasis should be less on the novelty of the method itself and more on how existing techniques are adapted and how convincingly their effectiveness is demonstrated. From this perspective, I think the authors have presented a sufficiently strong framework. That said, if another reviewer or the area chair regards novelty as the foremost criterion, I would respect a final decision that reflects this.

 

In addition, I have two minor concerns of my own. They did not meaningfully affect my recommendation, but I think they are worth the AC's consideration in the final decision.

Extension to other lesion types. Although the authors have shown solid results on additional datasets and modalities, the method remains limited to nodules. Clinically important findings include many other lesion types (for example, datasets such as VinDr-CXR and PadChest-GR provide bounding boxes for a variety of lesions). Extending the method to non-nodule lesions would be a valuable direction for future work.

Paper presentation. The presentation has improved over the previous version, but many parts, from the abstract through the conclusion, could still be made clearer and more concise. I recognize that this is not feasible to request during the rebuttal period, and it did not significantly affect my score, but I encourage the authors to revisit the overall organization and writing in a later revision.

We sincerely thank the reviewer for the thoughtful reassessment and constructive suggestions.

  1. We appreciate the reviewer's recognition of the value of our systematic measurement framework. Regarding model behavior, our additional analyses provide evidence about where the observed differences emerge and how consistently they persist, while further mechanistic investigation remains a valuable direction.

  2. We intentionally focused this study on cancer-related lesions—pulmonary nodules and mammographic masses—to establish and validate the framework in a focused setting. We agree that extending to other clinically important lesion types is a natural and important next step that we plan to pursue.

  3. We also appreciate the suggestions on presentation. We will further refine the writing for clarity and conciseness, taking into account the reviewer's comments and feedback from the other reviewers.

Thank you again for the constructive evaluation and for recognizing the contribution of our work.

Reviewer_mqWLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PULSE, a controlled evaluation framework for studying model behavior beyond aggregate detection scores. Synthetic nodule-like probes are inserted into chest radiographs while varying size, contrast, morphology, and location.

Experiments on 14 detectors from seven families reveal substantial differences in conditional behavior despite similar pFROC scores. The authors also examine reproducibility across host datasets and training seeds, efficient measurement strategies, and correspondence with natural lesions.

Critical strengths
  1. Important evaluation problem. The FR1/DN2 comparison is particularly compelling and there is clear evidence that aggregate scores can hide meaningful behavioral differences.

  2. Well-controlled experimental design that helps isolate conditional responses.

  3. The measurement-budget analysis adds practical value. Different acquisition strategies work best for recovering model rankings, finding disagreements, and reconstructing response regions. Useful guidance for designing more efficient behavioral evaluations.

Critical weaknesses
  1. Limited clinical validity of synthetic probes. Smooth disks and Gaussian profiles do not fully capture real pulmonary nodules, including irregular boundaries, heterogeneous texture, and anatomical interactions. The natural-lesion experiment only establishes partial pointwise response matching, not transfer of the observed contrast or morphology trends.

  2. The framework's methodological novelty is moderate. Controlled perturbations, synthetic lesion insertion, and spatial bias evaluation have been studied previously. PULSE mainly combines these approaches into a systematic behavioral atlas. The distinction from Zone Evaluation and iTRIALSPACE could be clearer.

  3. The primary detection metric depends on somewhat arbitrary choices: 16-pixel localization radius, 0.02 score-gain threshold, and 0.25 FP/image operating point. Appendix E shows substantial sensitivity for some models...

Questions for authors
  1. Can the authors test whether the counterintuitive contrast-response patterns also occur for natural nodules? The current natural-lesion assay measures pointwise response similarity but does not establish whether conditional trends transfer.

  2. How much of the observed behavioral variation is explained by preprocessing and input resolution? For example, does the DINO contrast decline persist under matched input resolution and intensity normalization?

  3. Could the authors more explicitly distinguish PULSE from Zone Evaluation and iTRIALSPACE? Which methodological capabilities or insights would not be obtainable through these existing controlled-evaluation approaches?

  4. How stable are the main findings across different detection operating points? Appendix E shows notable sensitivity to the gain threshold and localization radius for some models. Would the same model rankings and contrast-response patterns hold across a broader FP/image range?

  5. Can the authors evaluate the atlas on an entirely independent dataset or acquisition domain, ideally including real nodules? The current held-out hosts were not fully independent of earlier detector development, making external validity difficult to assess.

6 comments

We sincerely thank the reviewer for the thoughtful evaluation and for highlighting the importance of the FR1–DN2 comparison and measurement-budget analysis. We address the clinical, methodological, robustness, and external-validation questions in separate comments below.

[W1 / Q1 · Natural lesions and synthetic-probe validity]

Limited clinical validity of synthetic probes. Smooth disks and Gaussian profiles do not fully capture real pulmonary nodules, including irregular boundaries, heterogeneous texture, and anatomical interactions. The natural-lesion experiment only establishes partial pointwise response matching, not transfer of the observed contrast or morphology trends.

Can the authors test whether the counterintuitive contrast-response patterns also occur for natural nodules? The current natural-lesion assay measures pointwise response similarity but does not establish whether conditional trends transfer.

We added a natural-nodule assay rather than claiming that measured synthetic contrast trends were already established on natural lesions. For 278 annotated radiographs, fitted procedural replacements agree with the mean real-lesion presence effect within the prespecified ±0.05 margin for all six detectors; five comparisons are informative because erasure alone would not meet that margin. On 463 overlapping radiographs, replacement yields a lower six-model mean response error than erasure (−0.191 [−0.209, −0.172]) or a per-model constant (−0.091 [−0.103, −0.079]).

The fit uses measured image properties of the natural nodule and does not optimize against detector outputs. This evidence connects the assay to natural-lesion mean effects, but we have not experimentally established that the synthetic size-, contrast-, or location-response trends transport to independent natural-nodule cohorts. We therefore distinguish mean-level correspondence from clinical sensitivity or conditional-trend transport.

The results are in §5.6 (p. 10), Appendix H (pp. 40–41), and Tables 23–24 (p. 41). Figure 14 (p. 42) and Figure 15 (p. 43) show illustrative interventions on rights-cleared public images, not patient-cohort validation samples.

[Q2 · Preprocessing, resolution, and high-contrast decline]

How much of the observed behavioral variation is explained by preprocessing and input resolution? For example, does the DINO contrast decline persist under matched input resolution and intensity normalization?

We performed a separate eight-setting inference-input experiment: the original input path plus alternatives varying 1024/768-pixel resolution, intensity normalization (original, robust-lung window, or normalization refitted after insertion), and input clipping. Each probe was rendered once; its image was then passed to the compared input settings on matched host–site observations.

Table R1. DN1 contrast-response checks; outcome is localized native-score change from requested CNR 2 to 3 at the largest probe size.

CheckResult
Original input setting−0.288 [−0.341, −0.234] (95% interval)
Seven alternative settingsNegative in all seven, with each 95% interval below zero
Source-range-restricted probes (main experiment, E0 detection rate)−0.291 [−0.332, −0.254]
Earliest recorded DINO stageDecrease already present in encoder proposal scores, before top-900 selection

The decline therefore survives changes to input representation and clipping, but the magnitude changes. In particular the 768-pixel variants reduce pFROC (DN1 0.835 → 0.361 under the reported modification), so these are robustness interventions rather than a performance-matched architectural ablation. We do not attribute a causal percentage of the effect to preprocessing.

See §5.3 (p. 7), Appendix D.3–D.5 (pp. 28–29), Table 9 and Table 10 (p. 28), and Figure 10 (p. 29).

[W2 / Q3 · Methodological contribution relative to prior work]

The framework's methodological novelty is moderate. Controlled perturbations, synthetic lesion insertion, and spatial bias evaluation have been studied previously. PULSE mainly combines these approaches into a systematic behavioral atlas. The distinction from Zone Evaluation and iTRIALSPACE could be clearer.

Could the authors more explicitly distinguish PULSE from Zone Evaluation and iTRIALSPACE? Which methodological capabilities or insights would not be obtainable through these existing controlled-evaluation approaches?

We agree that synthetic lesion insertion and spatial-bias evaluation predate PULSE. The contribution is not the first controlled perturbation, but a repeatable experimental assay in which the same host and probe position are evaluated before and after insertion, the readout is paired and task-specific, and model responses are compared at fixed probe conditions over dense anatomical coordinates. The assay explicitly measures how much is lost by reporting only an overall mean or a coarse zone summary.

Table R2. What the additional paired spatial measurement reveals on the same intervention bank.

ComparisonCoarse/aggregate viewInformation visible with PULSE
FR1 vs RD2, medium probe, NODE21 discoveryLung-wide mean detection gap only 0.19 percentage pointsDifferent detection outcomes in 27.1% of matched host–cell tests, shown by FR1-only/RD2-only cartograms and the G / (D−G) / (Q−D) decomposition
FR1 vs RD1, same probeMean absolute aggregate gap G = 0.028Mean absolute cell difference D = 0.154 (82% cancels in the lung-wide mean)
FR1 vs RD1, zone pooling4-zone repeated difference 0.077Dense 310-cell repeated difference 0.150, gain 0.073 [0.0665, 0.0804]
Measurement-budget simulationAt 1 evaluation/cell: dense 0.04316-zone stratification 0.078; at 10 evaluations/cell dense 0.119 versus 16-zone 0.104

Figure 2 (p. 6) is now the direct visual example; Appendix J (pp. 44–49) adds all fourteen detectors, pairwise comparisons, changing probe conditions, and external FR1–RD1 maps. We clarify the relationship to Zone Evaluation and iTRIALSPACE in §2 (pp. 2–3): the zone-versus-dense result is an internal pooling ablation, not a claimed head-to-head reimplementation of either earlier method. Formal measurements are in §3.3 (p. 4), with supporting Tables 11–13 (pp. 30–31).

Reviewer_yC9aOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces PULSE, a benchmark for studying conditional behavior of models, demonstrated in an X-ray classification setting. Specifically, the authors test 14 chest X-ray nodule detectors by (digitally) adding fake nodules to normal X-rays, changing one thing at a time (such as size, brightness, shape, location), and recording which models detect them. Models that have nearly the same standard accuracy score behave very differently under these artificial shifts: for example, one of the models detects about five times as many fake nodules as another. The authors argue that their techniques offer a systematic way to evaluate machine learning models beyond accuracy.

Critical strengths
  • The study is well-controlled and the authors are able to fully attribute differences in behavior to model changes---model changes also seem consistent across distributions which provides further evidence of the validity of the results.

  • In general, looking past average accuracy and to more conditional notions of performance is crucial, especially in high-stakes domains such as medical.

Critical weaknesses
  • The paper is extremely difficult to read, it is full of acronyms and jargon that is inadequately explained, and I had to read several of the sentences over 5 times just to understand a specific result. Some sentences still do not make sense to me despite several readings (see Questions below). The authors should consider substantially revising the manuscript to improve clarity.

  • The clinical significance of the findings is unclear given the obvious artificial nature of the introduced lesions - the authors do not show (e.g., via observational data or otherwise) that the results have any possibility of transferring to real nodules, and it's possible that many of the effects are driven by models performing consistently incorrectly out-of-distribution (as observed by several prior works in computer vision) than by genuine sensitivity to properties of the nodules.

  • The models all differ in many ways at once, and so it is hard to pin down exactly what factors drive the difference between models. The paper successfully argues that model behavior differs but does not really give a predictive account of the underlying reasons.

  • The claim seems to be that PULSE is general with this being a specific application to X-rays, but much of the methodology and measurement seems quite specific to X-ray data.

Questions for authors

There were several sentences throughout the paper that I did not understand at all. A small subset of these questions:

  • Limitations: "The held-out host bank was withheld from PULSE analysis but overlaps earlier detector selection or qualification roles" - what does this mean? What does "qualification roles" mean in this context?

  • What is a "Shared Regional Deficit" (S4.5)

  • What is a "response geometry?" Is this a standard term? It is used several times throughout the paper

5 comments

We sincerely thank the reviewer for the careful and constructive feedback. We have revised the manuscript to make the measurement and its evidence easier to assess, and we respond to the related concerns below.

[W1 / Q1–Q3 · Clarity, terminology, and the evaluation set]

The paper is extremely difficult to read, it is full of acronyms and jargon that is inadequately explained, and I had to read several of the sentences over 5 times just to understand a specific result. Some sentences still do not make sense to me despite several readings (see Questions below). The authors should consider substantially revising the manuscript to improve clarity.

Limitations: "The held-out host bank was withheld from PULSE analysis but overlaps earlier detector selection or qualification roles" - what does this mean? What does "qualification roles" mean in this context?

What is a "Shared Regional Deficit" (S4.5)

What is a "response geometry?" Is this a standard term? It is used several times throughout the paper

We rewrote the assay around an explicit sequence—fixed real radiograph, valid probe placement, clean/probed inference, paired detection readout, and host-averaged response—and made the figures and Results follow that sequence. This is a chest X-ray nodule-detection task, not image classification. Figure 1 shows the actual intervention; Figure 2 shows that a 0.19-percentage-point gap in mean synthetic-probe detection can coexist with 27.1% disagreement over matched image–location tests.

The three phrases in the review are addressed directly:

Earlier wording or questionExplanation in the revised paperCurrent location
“qualification roles” / held-out hostsThe second 300-image NODE21 set was excluded from PULSE response discovery but had earlier roles: 89 checkpoint-selection negatives, 147 calibration negatives, 64 pilot backgrounds; 0 used to train detectors. It is not wholly independent of model development.§4, p. 5; Appendix B.1 and Table 3, pp. 20–21
“Shared Regional Deficit”We removed this shorthand. It referred to lower detection at medial mid-height positions than in the lower visible lung, quantified for the tested detectors.§5.2, p. 6; Table 14, p. 32
“response geometry”We removed this ambiguous term and define response-profile distance (BPD) explicitly: each model's score-change vector is standardized within a host and pairwise distances are averaged across hosts.§3.3, p. 4; Appendix A.3, pp. 17–18

Requested contrast (CNR), validity, and the detection rule are explained in §3 (pp. 3–4), and the conventional pFROC score is defined in §4 (p. 4). The methods and main findings are mapped to evidence in Supplement Table 2 (p. 16); the full measurement protocol is in Appendix A (pp. 16–19). We use Set 2 as the concise label in Main Figures 2–4, while defining the actual evaluation role once in §4.

[W2 · Artificial probes and natural-lesion relevance]

The clinical significance of the findings is unclear given the obvious artificial nature of the introduced lesions - the authors do not show (e.g., via observational data or otherwise) that the results have any possibility of transferring to real nodules, and it's possible that many of the effects are driven by models performing consistently incorrectly out-of-distribution (as observed by several prior works in computer vision) than by genuine sensitivity to properties of the nodules.

We added a direct comparison with annotated natural nodules. For each radiograph, an inpainting-based estimate with the lesion removed is compared with the original image and with a procedural replacement fitted to observable lesion properties without using detector scores. The estimand is the mean change in localized detector score caused by lesion presence.

Table R1. Direct evidence involving natural lesions (the 463-image analysis includes the 278 images; these are not independent samples).

AnalysisMain quantitative result
278 annotated radiographs, six detectorsReal-lesion and fitted-replacement mean effects agree within a prespecified ±0.05 equivalence margin for 6/6 detectors; the test is informative for 5/6
463 radiographs, six-detector macro meanReplacement reduces absolute response error relative to erasure by 0.191 [0.172, 0.209] and relative to an image-excluded per-model constant by 0.091 [0.079, 0.103]
Synthetic-probe artifact checkDN1's CNR 3-minus-2 detection change remains −0.291 [−0.332, −0.254] on probes whose pixels remain within the stored source range

The synthetic-probe findings also persist under 36 detection-rule variants, and the principal coverage and contrast effects recur in both external CXR domains. These results establish that the measurement is not explained simply by one operating rule or source-range exceedance, and connect the intervention to mean natural-lesion responses. They do not establish transfer of synthetic size-, contrast-, or location-dependent trends to natural nodules or estimate clinical sensitivity.

The natural-lesion assay is in §5.6 (p. 10), Appendix H (pp. 40–41), and Tables 23–24 (p. 41); the in-range control is in Appendix D.3 / Table 9 (p. 28), and external replication in §5.4 (pp. 7–8) and Table 17 (p. 35).

[W3 · What explains cross-model behavior?]

The models all differ in many ways at once, and so it is hard to pin down exactly what factors drive the difference between models. The paper successfully argues that model behavior differs but does not really give a predictive account of the underlying reasons.

We agree that comparing complete trained systems does not separately identify causal contributions from architecture, pretraining, input processing, and optimization. Our empirical contribution is a repeatable measurement of which model behaviors differ, together with targeted tests of where one counterintuitive response first appears.

Table R2. Evidence about the structure and stability of model differences.

TestMeasured result
Across 14 detectorsBetween-family response-profile distance exceeds within-family seed-pair distance by 0.326 [0.322, 0.329]
Model seedsEach of the 14 models is closest to its other training seed across the evaluated NODE21 banks
DN1 high-contrast declineNegative localized score change persists in all eight inference-input settings (resolution, normalization, and clipping variants)
Earliest recorded stageThe decline is already present in DINO encoder proposal scores; the unhooked backbone/feature pathway is not assigned a cause

For example, the original DN1 pipeline's continuous-score change was −0.288 [−0.341, −0.234] in the input-arm experiment. This locates an observable score decrease but does not prove its architectural mechanism; the 768-pixel variants also degrade conventional pFROC and are not performance-matched.

The profile-distance evidence is in §5.4 (pp. 7–8) and Figure 4 (p. 8), with definitions in §3.3 (p. 4). Input-stage and input-setting results appear in Appendix D.4–D.5 (pp. 28–29), Table 10 (p. 28), and Figure 10 (p. 29).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Revised manuscript and Supplement

We thank the reviewers for their careful and constructive feedback. We have uploaded a revised manuscript with an integrated Supplement (Appendices A–J). We have substantially improved the organization, writing, and figures for clarity, while preserving the original research question, paired-intervention framework, principal chest X-ray findings, and prespecified C1–C6 outcomes.

In response to the reviews, we added targeted validation: 36 detection-rule variants and a separate eight-setting input-path analysis; external evaluation on 300 VinDr-CXR and 300 Shenzhen radiographs, where both principal effects replicated; a four-classifier mammography extension with 150-patient confirmation; and natural-nodule comparisons supporting agreement of mean lesion-presence responses. We also expanded the comparative spatial analysis to show how detectors with nearly identical mean probe-detection rates can disagree on 27.1% of matched evaluations, revealing differences concealed by aggregate summaries.

Reading guide:

  • Figure 1 (p. 2): Controlled intervention and paired detection readouts.
  • Figure 2 (p. 6): Spatial disagreement between detectors and its quantitative decomposition (G, D, Q).
  • Figure 3 (p. 7): Non-monotonic contrast responses and robustness analyses.
  • Figure 4 (p. 8): Response reproducibility and external CXR validation.
  • Figure 5 (p. 9): Mammography response maps and confirmation.
  • Section 5.1 (p. 5) and Figure 12a (p. 32): The 5.3-fold probe-coverage difference despite similar conventional detection scores.
  • Section 5.6 (p. 10): Agreement with natural-lesion responses.

Supplement Table 2 (p. 16) maps the main claims to their evidence levels and supporting Appendices A–J. The supplementary code recomputes selected estimates and checks stored summaries. Our updated, issue-focused reviewer responses provide the supporting experiments, key quantitative results, and precise manuscript references.

We sincerely appreciate the reviewers' time and consideration.

We sincerely thank the reviewers again for their thoughtful and constructive feedback.

In response to their comments, we have made further refinements to the revised manuscript and updated our earlier responses accordingly. To make the discussion easier to follow, we have reorganized our responses into separate, issue-focused comments. Each comment quotes the relevant reviewer concerns, directly presents the supporting experiments and key quantitative results, and provides precise references to the current manuscript and Supplement (Appendices A–J).

We would be very grateful if the reviewers could revisit our updated responses and revised manuscript before the discussion period closes. We welcome any remaining questions or feedback and would be happy to provide further clarification.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.