- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
The paper examines what pooled claim-level AUROC measures when evaluating auditors of radiology reports. It separates within-concept from between-concept comparisons and derives the maximum AUROC achievable by scores that are constant within specified groups. Across five generators, this oracle ceiling exceeds the tested auditors, while a predictor based on training base rates remains competitive. The paper proposes evaluation within groups and studies claim triage and regional dependence across several audit signals.
Critical strengths
-
The paper addresses an important evaluation problem: strong rewards for group-level priors can obscure an auditor's ability to distinguish errors within a finding and polarity.
-
The decomposition and ceiling provide clear, useful diagnostics for interpreting pooled scores.
-
Mathematical proofs help to improve the theoretical contribution of this work.
Critical weaknesses
-
Some claims about the source of predictive information go beyond the mathematical results. A 90 to 94% share of between-concept pairs does not mean that priors account for the same fraction of a method's achieved AUROC. Visual discrimination can also improve these comparisons.
-
Stratification changes the evaluated claims and their weighting, alongside removing cross-group comparisons.
-
The relationship between the proposed metric and practical triage needs clarification. The probe leads the stratified ranking on all five generators. Yet at 20% review, it catches fewer errors.
-
Correctness is defined through agreement with parsed reference reports, treating unmentioned concepts as absent and excluding claims outside the concept bank.
-
The citation of Self-Supervised Learning From Images With a Joint-Embedding Predictive Architecture is not correct. I-JEPA is accepted to CVPR 2023 rather than ICCV.
Questions for authors
Please refer to the Weakness Section.