acceptodds
Under review as a conference paper at ICLR 2027

RadDiff: Automated Radiology Discovery from Cohort Differences

Abstract

Many clinical questions in radiology are questions about cohorts: Which radiographic findings distinguish pneumonia patients who die from those who survive? How does COVID-19 present in older versus younger patients? What separates the radiographs a race classifier assigns to different groups? Answering them requires discovering, in open-ended language, what distinguishes two cohorts of hundreds of studies, a task that is laborious for radiologists and on which general-domain methods fail. We introduce RadDiff, a multimodal agent for automated radiology discovery from cohort differences. RadDiff runs a hypothesize-verify-refine loop: a proposer reads domain-adapted captions and images of a few sampled studies and proposes candidate differences, a ranker verifies every candidate on the full cohorts with a medical vision-language model, and the best-verified hypotheses, with zoomed-in views of the regions they refer to, are fed back to refine the next round. On RadDiffBench, a new benchmark of 57 radiologist-validated cohort pairs from MIMIC-CXR, RadDiff achieves 47.4% top-1 accuracy versus 1.8% for the general-domain VisDiff, with every component contributing. Applied to the three questions above, it surfaces radiologist-reviewed discoveries, from the device burden of pneumonia non-survivors to the non-anatomical cues behind a race classifier's predictions.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
radiology image reasoning, set difference, vision and language

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_k2koOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

RadDiff adapts a proposer–ranker framework to describe differences between radiology cohorts. It generates candidate differences from medical captions and image grids, then ranks them using CheXzero scores across the full cohorts. Feedback and cropped views guide further rounds of refinement. On a benchmark of 57 radiologist-validated cohort pairs, RadDiff achieves 47.4% Acc@1, compared with 1.8% for general-domain VisDiff. Three exploratory applications examine pneumonia mortality, age differences in COVID-19, and cohorts selected by a race classifier.

Critical strengths
  1. Open-ended cohort comparison is a useful application for medical vision–language models. Separating expensive hypothesis generation from cheaper cohort-wide scoring is a sensible way to approach it.

  2. The benchmark covers specific clinical findings, devices, and acquisition properties. Radiologists validate the concepts, and a human evaluation checks the agreement of the automatic judge.

  3. Prompts, qualitative examples, and failure analyses help explain the system's behavior. The paper also states that its application findings are hypotheses requiring further validation.

  4. Presentation is clear and easy to follow.

Critical weaknesses
  1. The contribution of the search components is difficult to separate from additional computation and the choice of iteration.

  2. The experiments rely on GPT-4o/4o-mini for benchmark construction and GPT-4.1-nano for both hypothesis generation and evaluation, without testing stronger, more recent models.

  3. Using the same model as proposer and judge also warrants an independent check beyond the 51-item human comparison.

  4. The benchmark measures recovery of a predefined difference more directly than the quality of open-ended discovery.

  5. Differences between high-confidence predicted groups reveal associations with classifier output, but do not necessarily show that devices drive predictions or that anatomy contributes nothing.

Questions for authors

Please refer to the Weakness Section.

3 comments
AuthorsOpenReview ↗

Response to Reviewer k2ko (1/2): Computation, Stronger Models, and Judge

We sincerely thank the reviewer for the positive assessment and for precise concerns. Each one gave us the opportunity to add evidence, and every new analysis confirms our claims. The key results are summarized in the new Section 5.3 (Table 4) and detailed in Appendix F (changes in blue).

W1: Separating the search components from extra computation. This is an excellent point. We re-ran these analyses with a stronger and more recent proposer, Claude Haiku 5.5, graded by a separate judge, GPT-5.6-Sol, from a different model family. Three results separate the search components from computation and from the choice of iteration (5 seeds; F.2, Tab. 12–13):

MethodAcc@1Acc@5
VisDiff + knowledge injection, budget-matched (same two iterations)38.1 ± 3.861.8 ± 2.1
RadDiff without visual search (iterative refinement, T = 2)40.7 ± 3.465.1 ± 2.1
RadDiff42.6 ± 2.969.3 ± 3.3
  • More refinement does not match visual search. Iterative refinement without visual search is already in the main text (Table 2, Fig. 4). Even with as many iterations as RadDiff or more, i.e. as much computation or more, its Acc@1 stays below RadDiff's (at most 43.9 vs. 47.4); with current models it reaches 40.7 vs. 42.6.
  • More computation for RadDiff's first stage does not match RadDiff. VisDiff + knowledge injection run for the same two iterations, with an independent second pass merged into the first, stays well below RadDiff (38.1 vs. 42.6 Acc@1).
  • The iteration count was not tuned. RadDiff uses T = 2, and accuracy plateaus after two to three iterations (Fig. 4, Fig. 21).

W2: Stronger and more recent models.

  • Proposers. We re-ran RadDiffBench with Claude Haiku 5.5: every component of RadDiff still raises both Acc@1 and Acc@5, and the full method reaches 42.6 ± 2.9 Acc@1 and 69.3 ± 3.3 Acc@5 (full component table: Tab. 12). On the external benchmark we tested 6 proposers from two providers, from small to large; RadDiff improves Acc@1 over its knowledge-injection stage for every one (pooled +9.9 Acc@1 (p < 0.001); F.3, Fig. 22). The framework benefits from, rather than depends on, the choice of LLM.
  • Stronger models do not saturate the task. The best configuration reaches 42.6 Acc@1 on RadDiffBench and 24.6 on hard pairs; even an oracle with the ground-truth reports reaches only 42.1. On RadDiffBench-Ext, the new external benchmark we built, even the strongest proposer (Claude Sonnet 5.5) reaches only 63.3. RadDiffBench therefore remains a challenging testbed for future research.
  • Benchmark construction. GPT-4o only brainstormed candidate concepts, which three radiologists then filtered. A radiologist audited GPT-4o-mini's report-to-cohort assignment at >95% precision.

RadDiffBench-Ext, Acc@1 (%), 3 seeds:

ProposerKnowledge injection onlyRadDiffΔ (p)
Claude Haiku 5.543.359.2+15.8 (p = 0.008)
Claude Haiku 5.5, no reasoning49.260.0+10.8 (p = 0.03)
Claude Sonnet 5.551.763.3+11.7 (p = 0.04)
GPT-6-Luna50.060.8+10.8 (p = 0.04)
GPT-6-Sol48.357.5+9.2 (p = 0.07)
GPT-5.6-Luna50.050.8+0.8 (p = 0.83)

W3: Proposer and judge from the same model. We thank the reviewer for raising this; the new analyses show that our evaluation is robust.

  • Separated. In all new experiments, Claude proposes and GPT-5.6-Sol judges.
  • The judge was selected against radiologists. We compared 19 candidate LLM judges with radiologists on 129 radiologist-graded hypotheses and, using criteria fixed before seeing the results, picked the judge that agrees with radiologists as well as the best candidate without inflating accuracy (F.1). GPT-5.6-Sol agrees with radiologists at Spearman ρ = 0.76.
AuthorsOpenReview ↗

Response to Reviewer k2ko (2/2): Open-Ended Discovery and Race Audit

W4: Recovering a predefined difference vs. open-ended discovery. We thank the reviewer for raising this distinction. We view RadDiff, and RadDiffBench, as open-ended discovery:

  • RadDiff receives no hint of the difference. It must find, in free text, what distinguishes two large cohorts of hundreds of studies each. The intended difference is used only afterwards, to grade.
  • RadDiff describes differences in open-ended natural language, far beyond any predefined set of differences. Of RadDiff's 1,425 top-5 hypotheses (5 seeds), 90.9% state information that a classifier over the 14 CheXpert labels cannot output (F.7):
HypothesisShare
Finding outside the 14 labels6.0%
Label with added detail84.9%
— laterality49.9%
— anatomic location44.9%
— severity or extent40.4%
— device type or position21.5%
— temporal change11.4%
Plain label only9.1%

(A hypothesis can carry several details.)

  • The case studies have no predefined answer. RadDiff surfaces insights on prognosis, disease presentation and model behavior, and scientists then draw the conclusions. Two radiologists confirmed that the top-ranked findings are meaningful and clinically consistent.

W5: Race classifier, association vs. causation. We agree and have removed the causal claims (e.g., "rather than anatomy"). The associations RadDiff surfaces mainly disclose the underdiagnosis of underserved groups, consistent with prior work (Seyyed-Kalantari et al., 2021), and still provide useful hypotheses: we present the signs as candidates for targeted study, not mechanisms.

Thank you again; your comments made the paper clearer and its evidence stronger.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer k2ko,

Thank you again for your positive and precise review. We have carefully addressed your concerns and provided concrete results on stronger and more recent models, the choice of judge, separating the search components from computation, and open-ended discovery.

We would greatly appreciate it if you could kindly read our response and check the revision, and let us know if you have any further comments. As the discussion period ends soon, we hope to have the chance to discuss with you in case you have any further comments.

Thank you for your time.

Reviewer_1MWtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies open-ended cohort difference discovery in chest X-rays: given two sets of images, the system outputs, in natural language, the imaging features that are more prevalent in one of the groups. The authors propose RadDiff, which extends the VisDiff proposer–ranker framework with medical domain models, multimodal inputs, iterative feedback, and localized visual search.

Specifically, the system samples a small number of chest X-rays from each group, uses CheXagent to generate descriptions, and then uses GPT-4.1-nano to propose candidate differences by combining the descriptions and the images. CheXzero computes an AUROC distinguishing the two groups based on the similarity between candidate descriptions and all images. High-scoring hypotheses, together with zoomed-in crops of the corresponding regions, are fed back to the proposer to further refine the candidates.

The authors also construct RadDiffBench, comprising 57 cohort pairs defined by reports, with target differences curated by radiologists. The full system achieves an Acc@1 of 47.4%, substantially higher than the general-purpose VisDiff at 1.8%. The paper further demonstrates three application cases: pneumonia mortality, age-related differences in COVID-19, and auditing of a race classifier.

Critical strengths

The problem definition is meaningful: it enables rapidly proposing differences from existing data to form hypotheses, which facilitates fast idea generation. In terms of system design, discovering differences on a small sample first and then iterating with embedding scores over the full sample improves computational efficiency. Multiple example analyses are provided, which helps readers understand the workflow.

Critical weaknesses

Section 6 states that all three case studies use MIMIC-CXR images linked to MIMIC-IV. Section 6.2 then reports COVID-19 patient cohorts with age > 60 and age ≤ 40, containing 901 and 807 images respectively. However, the official MIMIC-CXR documentation indicates the data were collected between 2011 and 2016 — before the COVID-19 pandemic. This significantly undermines the credibility of the results.

RadDiff uses the image–text similarity AUROC from CheXzero to evaluate candidates, and this score is repeatedly fed back to the proposer. This evaluation depends heavily on CheXzero's similarity measure, which does not necessarily correspond to genuinely valuable disease/abnormality findings. The authors need to validate whether this method can truly discover valuable cohort groupings, and whether substituting other models would work as well — i.e., to verify the effectiveness of the framework itself, rather than the discriminative power of this particular model.

In essence, if one has a reasonably good embedding model or a report-generation model, one could directly group images via embedding similarity, or generate reports and store them as structured text, then use a large language model to extract differences from the text. Such baselines should be established for comparison.

The showcased cases are all clear-cut examples that could be distinguished by conventional classification models. In my view, the greater value of this framework lies in discovering more non-obvious, previously unknown disease differences (presence/absence) between groups; more meaningful findings of this kind should be presented.

Questions for authors

see above weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1MWt (1/2): COVID-19 Cohorts and Ranker Dependence

We sincerely thank the reviewer for a thoughtful and careful review. We are especially grateful for the observation on the COVID-19 cohorts: correcting them made the case study both more rigorous and more informative. The key new results are summarized in the new Section 5.3 (Table 4) and detailed in Appendix F (changes in blue).

W1: COVID-19 cohorts. Thank you for catching this; you are right that the MIMIC-IV diagnosis codes do not place these radiographs in the COVID-19 encounter. We therefore redid the case study on COVID-19-NY-SBU, a dedicated COVID-19 dataset (Stony Brook University Hospital; RT-PCR-confirmed patients; radiographs from the COVID-19 encounter; matched cohorts; 3 seeds; §6, F.8), and added outcome comparisons:

ComparisonLeading findings (seeds in top 3)
Age 75–90 vs. 18–59older: bilateral pleural effusions (3/3), vascular congestion (#1 in 2/3); younger: low lung volumes, normal heart size (3/3)
Deceased vs. dischargeddeceased: extensive bilateral opacities (#1 in 3/3)
ICU vs. non-ICUICU: support devices (#1 in 2/3), bilateral opacities (3/3)

Your observation led to a stronger result. On true COVID-19 radiographs, RadDiff surfaces a key finding that the earlier cohorts did not reveal: bilateral pleural effusions are more common in older patients, among the top three differences in all three seeds. Prior work confirms it: among COVID-19 patients over 60, pleural effusions become more prevalent with age (Okoye et al., 2022). More broadly, RadDiff finds more disease in older patients and near-normal films in younger ones, in line with studies reporting more extensive disease and more frequent radiographic opacities in older COVID-19 patients (Cao et al., 2020; Ippolito et al., 2020; Igi et al., 2021). Other findings, such as vascular congestion alongside signs of prior cardiac surgery in older patients, plausibly reflect age-related comorbidity and are hypotheses for radiologists to verify, which is exactly the use RadDiff is designed for. We are grateful for this correction, which improved the paper.

W2: Does it work only because of CheXzero? RadDiff is a framework in which the ranker is a pluggable component; we did not tune CheXzero.

  • Swapping the ranker gives comparable accuracy (F.6; submission's setup, seed 0, graded by GPT-5.6-Sol; Acc@1 / Acc@5): CheXzero 42.1 / 62.3, BioViL 44.7 / 64.0, ConVIRT 43.9 / 65.8; neither differs significantly from CheXzero (paired Wilcoxon over pairs, p ≥ 0.37).
  • CheXzero tracks real differences. When a hypothesis names a CheXpert finding whose prevalence in the radiology reports differs by at least 20 percentage points between the two cohorts (n = 873), CheXzero scores the hypothesis higher for the cohort with more of the finding 91.5% of the time (F.6). This check covers only a subset (31% of hypotheses, those whose label clearly differs) and verifies only the label a hypothesis names, not its finer detail.
  • Other proposers, other data. On a new external benchmark from another institution (NIH ChestX-ray14, radiologist ground truth only), RadDiff improves Acc@1 over its knowledge-injection stage for all 6 proposer LLMs (3 seeds; pooled +9.9 Acc@1 (p < 0.001); F.3):
ProposerKnowledge injection onlyRadDiffΔ (p)
Claude Haiku 5.543.359.2+15.8 (p = 0.008)
Claude Haiku 5.5, no reasoning49.260.0+10.8 (p = 0.03)
Claude Sonnet 5.551.763.3+11.7 (p = 0.04)
GPT-6-Luna50.060.8+10.8 (p = 0.04)
GPT-6-Sol48.357.5+9.2 (p = 0.07)
GPT-5.6-Luna50.050.8+0.8 (p = 0.83)
AuthorsOpenReview ↗

Response to Reviewer 1MWt (2/2): Baselines and Non-Obvious Findings

W3: Embedding and report-generation baselines. We thank the reviewer for these suggestions, and the results show clearly where RadDiff's gains come from. We implemented both with RadDiff's own medical models (5 seeds; F.2, Tab. 13):

MethodAcc@1Acc@5
Embedding grouping → LLM (k-means on CheXzero)28.8 ± 1.458.2 ± 2.2
Report generation → LLM34.7 ± 4.658.4 ± 2.6
Report generation → LLM + CheXzero ranking38.4 ± 1.658.1 ± 4.0
RadDiff42.6 ± 2.969.3 ± 3.3

RadDiff outperforms all of them. Even when the report baseline uses RadDiff's own ranker, RadDiff still leads by +4.2 Acc@1 and +11.2 Acc@5. Summarized reports lose the laterality, location and severity that separate many cohorts, and embedding clusters follow dominant factors (devices, view) rather than the cohort difference.

W4: Are the findings non-obvious? Our contribution is a new task, open-ended discovery of natural-language differences between radiology cohorts, together with a framework and a benchmark for it. RadDiff generates hypotheses, and scientists then decide which to pursue and draw the conclusions, as the reviewer suggests. The three case studies form a progression, from findings any reader can check to uses that go deeper:

  • Prognosis (pneumonia). Greater device burden in non-survivors is intuitive even to general ML readers, which shows that the method works. RadDiff also surfaces clinical signs alongside it, such as extensive bilateral opacities and pleural effusions.
  • Disease understanding (COVID-19, old vs. young). Going deeper, RadDiff characterizes how the same infection presents with age: more disease, especially pleural effusions, in older patients, and near-normal films apart from low lung volumes in younger ones.
  • Model auditing (race classifier). Deepest, RadDiff audits a model that predicts what clinicians cannot see. The classifier's confident White predictions carry more devices and abnormalities, while its confident Asian predictions are dominated by normal-appearing studies. A radiologist who reviewed these signs judged that they are not biologically based; they likely reflect differences in case mix (which patients in each group are imaged, and for which conditions) that the classifier picks up. The contrast mirrors the underdiagnosis of underserved groups reported in prior work (Seyyed-Kalantari et al., 2021). The audit thus discloses the model's limitations rather than biological differences, which supports trustworthy AI; any sign that survives such review is a candidate for targeted study.
  • Beyond labels. 90.9% of top-5 hypotheses state information that a classifier over the 14 CheXpert labels cannot output (F.7).

Thank you again. The revised case studies and new baselines confirm our claims, and your review made our work better.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 1MWt,

Thank you again for your thoughtful review. We have carefully addressed your concerns and provided concrete results on the revised COVID-19 cohorts, the dependence on CheXzero, the embedding and report-generation baselines, and the significance of the findings.

We would greatly appreciate it if you could kindly read our response and check the revision, and let us know if you have any further comments. As the discussion period ends soon, we hope to have the chance to discuss with you in case you have any further comments.

Thank you for your time.

Reviewer_cSR7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to describe differences between two chest X-ray cohorts in natural language. RadDiff builds on the proposer–ranker framework of VisDiff. CheXagent generates captions for individual images, GPT-4.1-nano proposes cohort differences from sampled captions and image grids, and CheXzero ranks the proposals using the AUROC of image–text similarity scores across the cohorts. The highest-scoring proposals are fed back to the proposer, and a second iteration uses cropped, enlarged image regions to examine more specific findings. Each iteration includes three sampling rounds, with 20 studies drawn from each cohort per round.

The authors introduce RadDiffBench, which contains 57 pairs of MIMIC-CXR cohorts. RadDiff achieves 47.4% Acc@1, compared with 1.8% for general-domain VisDiff. The paper also presents three case studies on clinical outcomes, age-related differences, and classifier behavior.

Critical strengths
  1. This paper addresses a practically significant radiological imaging task and provides a corresponding benchmark. Describing cohort differences in natural language helps researchers examine datasets and identify patterns that would otherwise require extensive manual review to detect. The 57 pairs of cohorts, constructed based on difference concepts selected by radiologists, provide concrete evaluation scenarios.

  2. Component experiments demonstrate the importance of medical models for this task. Replacing general-domain description models and ranking models yielded significant improvements, while feedback and local image inspection further enhanced the overall results. Natural-language hypotheses and selected image regions also facilitate the inspection of model outputs.

Critical weaknesses
  1. More convincing comparisons are needed to demonstrate the method’s progress relative to VisDiff. The paper explicitly adopts VisDiff’s framework of separating candidate generation from validation. Small-scale subset sampling, merging of multiple batches of candidates, and ranking based on AUROC for image-text similarity have all appeared in VisDiff. RadDiff introduces medical components, joint image-description input, feedback, and local visual search. While these modifications are useful, the only independent method used for comparison is VisDiff—a general-domain approach not tailored to radiology. A comparison with stronger medical baselines would more clearly demonstrate the benefits that the refinement and search mechanisms provide beyond domain adaptation.

  2. Generalization ability beyond the source dataset has not yet been verified. Both RadDiffBench and the case studies use MIMIC-CXR, and CheXagent was also trained on this dataset. While this does not necessarily mean that the evaluation images themselves appeared in the training data, the paper does not clarify the overlap between the evaluation data and the training sets for each component, nor does it evaluate the method on external hospital data. Therefore, the current results are insufficient to demonstrate how the method would perform in new clinical cohorts with different imaging conditions and patient populations.

Questions for authors
  1. What temperature was used for CheXagent? How consistent are its image descriptions across repeated runs on the same image? For CheXzero, how much do the AUROC scores and final hypothesis rankings change when the cohorts are resampled? Quantifying this variation would help assess the stability of the reported findings.

  2. CheXagent was trained on MIMIC-CXR, which is also the source of RadDiffBench. Could you clarify whether the benchmark studies overlap with its training data, and provide results on an external dataset not used to train the medical components? Such an evaluation would better support the use of RadDiff on new clinical cohorts.

  3. VisDiff is the only separate method included in the comparison, and its general-domain components perform poorly on radiographs. Could you include stronger domain-specific baselines, or adapt existing cohort-difference methods using the same medical models and a comparable generation budget? This would help establish the advantage of RadDiff beyond replacing general-domain components with medical ones.

  4. In Table 2, adding image grids to the caption-only configuration improves overall Acc@1 from 29.8% to 33.3%, but reduces hard-pair Acc@1 from 7.7% to 0.0% and Acc@5 from 30.8% to 23.1%. Why does the visual input hurt performance on these harder cases? Section 5.2 attributes this to subtle findings being difficult to see in the grids; could you provide a case analysis or additional evidence for this explanation?

3 comments
AuthorsOpenReview ↗

Response to Reviewer cSR7 (1/2): Baselines, Generalization, and Training-Data Overlap

We sincerely thank the reviewer for a careful and constructive review. Your suggestions led to two of the most important additions in the revision: domain-specific baselines and an external benchmark with radiologist ground truth. We ran every requested analysis; the key results are summarized in the new Section 5.3 (Table 4) and detailed in Appendix F (changes in blue).

W1 / Q3: Baselines beyond VisDiff, and gains beyond domain adaptation. Thank you for this suggestion; the results confirm RadDiff's advantage clearly. We re-ran all experiments with a stronger and more recent proposer, Claude Haiku 5.5 (gpt-4.1-nano has been retired), graded by GPT-5.6-Sol, a judge from a different model family. Following your suggestion, we compared RadDiff with medical pipeline baselines that use RadDiff's own medical models, and, as you suggested, with VisDiff adapted to the same medical models at a comparable generation budget (budget-matched VisDiff + knowledge injection: CheXagent captioner, CheXzero ranker and radiology prompt, i.e. RadDiff's first component in Table 2, run for the same two iterations as RadDiff) (F.2, Tab. 12–13; 5 seeds):

MethodAcc@1Acc@5
Baselines
Report generation → LLM34.7 ± 4.658.4 ± 2.6
Report generation → LLM + CheXzero ranking38.4 ± 1.658.1 ± 4.0
One-shot multimodal LLM28.1 ± 4.153.3 ± 3.8
Compute control (RadDiff's first stage)
VisDiff + knowledge injection, budget-matched38.1 ± 3.861.8 ± 2.1
RadDiff42.6 ± 2.969.3 ± 3.3
  • RadDiff outperforms every baseline. RadDiff leads even the strongest, report generation → LLM with CheXzero ranking, by +4.2 Acc@1 and +11.2 Acc@5.
  • The gains go beyond knowledge injection (domain adaptation). RadDiff beats budget-matched VisDiff + knowledge injection by +4.6 Acc@1 and +7.5 Acc@5, so the gain is not extra computation. Every later component adds accuracy (Tab. 12): image grids +2.8, refinement +1.2, visual search +1.9 Acc@1.

W2 / Q2: Generalization and training-data overlap. This question led to some of the paper's strongest evidence. We now state the overlap explicitly in the paper (Limitations): about 98% of RadDiffBench images belong to the MIMIC-CXR training split on which CheXagent and CheXzero were trained. In short, the overlap concerns individual images, whereas RadDiff's task is a cohort-level comparison that no component was trained for and that remains hard. Four results support this:

  1. The components were never trained on this task. CheXagent and CheXzero learned to describe or match individual radiographs with their reports. RadDiff's task, finding what distinguishes two cohorts of hundreds of studies, never appears in their training. Having seen an image does not reveal how two cohorts differ.
  2. The overlap does not make the task easy. If seeing the training images made the task easy, accuracy would be near ceiling. Instead, RadDiff reaches 42.6% Acc@1 (24.6% on hard pairs), and even an oracle given the ground-truth reports reaches only 42.1%. The difficulty lies in comparing cohorts, not in recognizing images.
  3. RadDiff generalizes to another institution and to unseen images.
    • Another institution. We built RadDiffBench-Ext (F.3): 20 cohort pairs from NIH ChestX-ray14 with radiologist ground truth only (RSNA adjudicated labels; NIH radiologist boxes). CheXzero has never seen these images. On this easier benchmark, the gains of RadDiff's search components beyond domain adaptation are clear: RadDiff improves Acc@1 over its knowledge-injection stage for all 6 proposer LLMs (3 seeds; pooled +9.9 Acc@1 (p < 0.001), +3.9 Acc@5 (p = 0.04)):
ProposerKnowledge injection onlyRadDiffΔ (p)
Claude Haiku 5.543.359.2+15.8 (p = 0.008)
Claude Haiku 5.5, no reasoning49.260.0+10.8 (p = 0.03)
Claude Sonnet 5.551.763.3+11.7 (p = 0.04)
GPT-6-Luna50.060.8+10.8 (p = 0.04)
GPT-6-Sol48.357.5+9.2 (p = 0.07)
GPT-5.6-Luna50.050.8+0.8 (p = 0.83)
  • Images no component has seen (7 pairs; NIH official test split, not in RSNA, so outside the training data of both CheXagent and CheXzero). Re-running the whole pipeline on these images only, RadDiff still reaches 53.6 Acc@1.
  1. RadDiff does not depend on CheXagent. With a general-purpose VLM (Claude Haiku 5.5) writing the captions instead of CheXagent, RadDiff still reaches 36.8 Acc@1 on RadDiffBench and 57.5 on RadDiffBench-Ext (Tab. 15).
AuthorsOpenReview ↗

Response to Reviewer cSR7 (2/2): Component Stability and Hard Pairs

Q1: CheXagent temperature and CheXzero stability (F.4).

  • CheXagent decodes deterministically (beam search, no sampling), so temperature does not apply. Re-captioning gave identical captions for 51/51 images, and for 30/30 on another GPU.
  • CheXzero is stable under resampling of the cohorts (submission's runs, same 5 seeds): the top-1 hypothesis is kept in 69–74% of bootstrap resamples (Kendall τ ≈ 0.65 on the top 5). Resampling moves Acc@1 by ±3, less than the ±6 across proposer seeds; all new results are therefore means over 5 seeds.

Q4: Why grids hurt hard pairs (F.5, Tab. 16). We appreciate this question; the answer shows exactly why visual search matters.

  • Grids hide subtle findings. For each hard pair we showed the same 40 studies as the caption, the grid tile, or the full image (1,560 judgments). The full image beats the grid tile by +10.6 points (p < 0.001). Subtle findings are at or below chance in grid tiles (rib fracture 42%, nodule 30%), while larger ones are not (cardiomegaly 65%). Three hard pairs are temporal and cannot be decided from any single image.
  • Visual search zooms where the finding is. With frontal-view visual search, the top crop contains the radiologist-marked finding for 75.2% of studies, against 63.1% for a random crop of the same size (p < 0.001; higher in 17 of 19 pairs).
  • Hard-pair accuracy. The submission's drop was a single pair of 13. Across 5 seeds, every component added to VisDiff + knowledge injection raises hard-pair Acc@1: 19.2 → 20.8 (grids) → 21.5 (refinement) → 24.6 (RadDiff).

Thank you again. These analyses confirm our claims, and your review made the paper considerably stronger.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer cSR7,

We have carefully addressed your concerns and provided concrete results on stronger domain-specific baselines, generalization to an external benchmark and unseen images, component stability, and the hard-pair analysis.

We would greatly appreciate it if you could kindly read our response and check the revision, and let us know if you have any further comments. As the discussion period ends soon, we hope to have the chance to discuss with you in case you have any further comments.

Thank you for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions and New Evidence

R1 = Reviewer cSR7, R2 = Reviewer 1MWt, R3 = Reviewer k2ko

We sincerely thank all three reviewers for their careful and constructive reviews. We are encouraged that they found the problem "meaningful" (R2), "practically significant" (R1), and "a useful application for medical vision–language models" (R3), and the presentation "clear and easy to follow" (R3). Their questions led us to run a substantial set of new analyses, and every one of them strengthens the paper's three claims:

  1. The task is meaningful. As R1 notes, describing cohort differences in natural language "helps researchers examine datasets and identify patterns that would otherwise require extensive manual review to detect." Our case studies show that open-ended comparison of two large radiology cohorts generates clinically interpretable insights.
  2. The framework works, even without extensive tuning. The reviewers found its design "a sensible way to approach" the problem (R3), noting that "feedback and local image inspection further enhanced the overall results" (R1). Our goal is to demonstrate the framework, so we used standard components without extensive tuning; even so, RadDiff beats strong baselines and works across proposers, rankers, captioners, and datasets. Practitioners can tune its components further for higher accuracy.
  3. The benchmark is valuable. Its radiologist-selected cohort pairs "provide concrete evaluation scenarios" (R1) covering "specific clinical findings, devices, and acquisition properties" (R3). Furthermore, our results with more advanced models show that the task remains hard even for current frontier models, so the benchmark leaves ample room for future work.

We uploaded a revised PDF. A new Section 5.3 (Table 4) summarizes the key new evidence, and a new Appendix F gives the details, opening with a table that maps each reviewer question to its answer; all changed text is shown in blue. The main-text benchmark results are unchanged; in Section 6, the COVID-19 case study now uses COVID-19-NY-SBU, a dedicated COVID-19 dataset (thanks to Reviewer 2).

Re-evaluation with current models. As suggested by R3, and as additional evidence (gpt-4.1-nano has been retired), we re-ran the analyses with Claude Haiku 5.5 as the proposer and GPT-5.6-Sol as the judge, over 5 seeds with paired tests. The judge comes from a different model family than the proposer; it agrees well with radiologists (Spearman ρ = 0.76) and does not inflate scores.

Summary of new evidence

QuestionNew analysisResultWhere
Suggested baselines (R1, R2)Embedding grouping, report generation → LLM, one-shot multimodal LLMall below RadDiff (28.1–38.4 vs. 42.6 Acc@1)F.2
Compute control (R1, R3)Budget-matched VisDiff + knowledge injection (RadDiff's first stage)RadDiff 42.6 vs. 38.1 Acc@1; 69.3 ± 3.3 vs. 61.8 ± 2.1 Acc@5F.2
Generalization (R1, R2)New external benchmark (NIH; radiologist ground truth only)RadDiff improves Acc@1 over its knowledge-injection stage for all 6 proposer LLMs, pooled +9.9 Acc@1 (p < 0.001)F.3
Training-data overlap (R1)Pipeline re-run on images no component has seenRadDiff still reaches 53.6 Acc@1F.3
Dependence on CheXzero (R2)Ranker swapped for BioViL / ConVIRTcomparable accuracy (no significant difference)F.6
Grids on hard pairs (R1)5-seed re-evaluation with current models; perception test on the 13 hard pairsgrids no longer hurt: hard-pair Acc@1 rises from 19.2 to 20.8 with grids and to 24.6 with RadDiff; grid tiles hide subtle findings (full image +10.6 points, p < 0.001)F.2, F.5
Beyond labels (R2, R3)Specificity of top hypotheses90.9% go beyond the 14 CheXpert labelsF.7
COVID-19 cohorts (R2)Redone on COVID-19-NY-SBUmore disease and pleural effusions in older patients, consistent with prior imaging studies§6, F.8
Race classifier (R3)Associational wordingassociations consistent with underdiagnosis; candidates for targeted study, not mechanisms§6.3

The task remains genuinely hard, and the benchmark remains valuable. Even with current models, RadDiff reaches 42.6 Acc@1 on RadDiffBench and 24.6 on hard pairs. An oracle reading the ground-truth radiology reports reaches only 42.1. On RadDiffBench-Ext, the new external benchmark we built, even the strongest proposer (Claude Sonnet 5.5) reaches only 63.3. Stronger models help, but open-ended cohort comparison is far from solved. RadDiffBench is a lasting testbed for future work.

We are grateful for the reviewers' time and believe the revision is considerably stronger as a result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.