- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
RadDiff adapts a proposer–ranker framework to describe differences between radiology cohorts. It generates candidate differences from medical captions and image grids, then ranks them using CheXzero scores across the full cohorts. Feedback and cropped views guide further rounds of refinement. On a benchmark of 57 radiologist-validated cohort pairs, RadDiff achieves 47.4% Acc@1, compared with 1.8% for general-domain VisDiff. Three exploratory applications examine pneumonia mortality, age differences in COVID-19, and cohorts selected by a race classifier.
Critical strengths
-
Open-ended cohort comparison is a useful application for medical vision–language models. Separating expensive hypothesis generation from cheaper cohort-wide scoring is a sensible way to approach it.
-
The benchmark covers specific clinical findings, devices, and acquisition properties. Radiologists validate the concepts, and a human evaluation checks the agreement of the automatic judge.
-
Prompts, qualitative examples, and failure analyses help explain the system's behavior. The paper also states that its application findings are hypotheses requiring further validation.
-
Presentation is clear and easy to follow.
Critical weaknesses
-
The contribution of the search components is difficult to separate from additional computation and the choice of iteration.
-
The experiments rely on GPT-4o/4o-mini for benchmark construction and GPT-4.1-nano for both hypothesis generation and evaluation, without testing stronger, more recent models.
-
Using the same model as proposer and judge also warrants an independent check beyond the 51-item human comparison.
-
The benchmark measures recovery of a predefined difference more directly than the quality of open-ended discovery.
-
Differences between high-confidence predicted groups reveal associations with classifier output, but do not necessarily show that devices drive predictions or that anatomy contributes nothing.
Questions for authors
Please refer to the Weakness Section.
3 comments
Response to Reviewer k2ko (2/2): Open-Ended Discovery and Race Audit
W4: Recovering a predefined difference vs. open-ended discovery. We thank the reviewer for raising this distinction. We view RadDiff, and RadDiffBench, as open-ended discovery:
- RadDiff receives no hint of the difference. It must find, in free text, what distinguishes two large cohorts of hundreds of studies each. The intended difference is used only afterwards, to grade.
- RadDiff describes differences in open-ended natural language, far beyond any predefined set of differences. Of RadDiff's 1,425 top-5 hypotheses (5 seeds), 90.9% state information that a classifier over the 14 CheXpert labels cannot output (F.7):
| Hypothesis | Share |
|---|---|
| Finding outside the 14 labels | 6.0% |
| Label with added detail | 84.9% |
| — laterality | 49.9% |
| — anatomic location | 44.9% |
| — severity or extent | 40.4% |
| — device type or position | 21.5% |
| — temporal change | 11.4% |
| Plain label only | 9.1% |
(A hypothesis can carry several details.)
- The case studies have no predefined answer. RadDiff surfaces insights on prognosis, disease presentation and model behavior, and scientists then draw the conclusions. Two radiologists confirmed that the top-ranked findings are meaningful and clinically consistent.
W5: Race classifier, association vs. causation. We agree and have removed the causal claims (e.g., "rather than anatomy"). The associations RadDiff surfaces mainly disclose the underdiagnosis of underserved groups, consistent with prior work (Seyyed-Kalantari et al., 2021), and still provide useful hypotheses: we present the signs as candidates for targeted study, not mechanisms.
Thank you again; your comments made the paper clearer and its evidence stronger.
Follow-up on our response
Dear Reviewer k2ko,
Thank you again for your positive and precise review. We have carefully addressed your concerns and provided concrete results on stronger and more recent models, the choice of judge, separating the search components from computation, and open-ended discovery.
We would greatly appreciate it if you could kindly read our response and check the revision, and let us know if you have any further comments. As the discussion period ends soon, we hope to have the chance to discuss with you in case you have any further comments.
Thank you for your time.
Response to Reviewer k2ko (1/2): Computation, Stronger Models, and Judge
We sincerely thank the reviewer for the positive assessment and for precise concerns. Each one gave us the opportunity to add evidence, and every new analysis confirms our claims. The key results are summarized in the new Section 5.3 (Table 4) and detailed in Appendix F (changes in blue).
W1: Separating the search components from extra computation. This is an excellent point. We re-ran these analyses with a stronger and more recent proposer, Claude Haiku 5.5, graded by a separate judge, GPT-5.6-Sol, from a different model family. Three results separate the search components from computation and from the choice of iteration (5 seeds; F.2, Tab. 12–13):
W2: Stronger and more recent models.
RadDiffBench-Ext, Acc@1 (%), 3 seeds:
W3: Proposer and judge from the same model. We thank the reviewer for raising this; the new analyses show that our evaluation is robust.