- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces RAD-Gym, an interactive environment for volumetric medical image reporting with a shared four-action interface: locate_organ, view_slice, write, and submit. It proposes ECI-RL, a GRPO-based RL method that combines terminal report rewards with process credit assigned to newly verified diagnostic progress, tracked through a coverage ledger and LLM-based expert judges. The paper also introduces trajectory-level metrics such as EHR, EER, NPHR, SE, and F-N, and evaluates on CT-RATE, AMOS-MM, Gastric-X, and MR-RATE. The claimed contributions are a modality-agnostic interactive environment, evidence-credit RL for long-horizon evidence seeking, and new state-of-the-art results on multiple medical report benchmarks.
Critical strengths
The paper targets interactive evidence acquisition in volumetric medical imaging, where fixed-input report generation cannot capture how evidence is selected. RAD-Gym’s shared four-action interface is a reasonable abstraction for CT and MRI, and the trajectory-level metrics are a useful direction for auditable medical agents.
Process-level evaluation is a valuable motivation. Extending report evaluation to evidence coverage and exploration efficiency is clinically meaningful. If validated properly, such metrics could help distinguish agents that produce plausible reports from agents that actually acquire supporting evidence.
Critical weaknesses
My main concern is that the empirical results do not support the paper’s main claim. ECI-RL is framed as a consistent improvement, but it is worse than SFT on several reported metrics: CT-RATE CIDEr drops from 27.81 to 26.82; AMOS-MM CIDEr drops from 22.58 to 22.20 and GREEN drops from 23.81 to 21.85; Gastric-X CIDEr drops from 32.65 to 30.59 and BioBERT drops from 70.54 to 70.28. F-N is also not best on AMOS-MM or MR-RATE. The paper reports point estimates only, with no seeds, variance, or significance tests. Given these mixed results, the “consistent” and SOTA claims are too strong.
The other major issue is the evidence-credit mechanism. The process reward relies on LLM judges for G, I, and X; only T is checked programmatically. There is no human validation, no agreement study, and no judge/prompt ablation. Since the method’s novelty is to reward “verified” diagnostic progress, this is a serious gap. Reproducibility is also limited: the pipeline uses several proprietary models, and the paper provides no code, compute budget, or sufficient training details.
Questions for authors
How were the LLM judges validated against radiologists? Please report agreement, failure cases, and sensitivity to the judge model and prompt.
Why does ECI-RL improve some clinical metrics but degrade CIDEr, GREEN, and BioBERT on several datasets? Which tradeoff should readers trust?
Response to Reviewer LQcQ
We have submitted a revised PDF version with the blue-highlighted revisions for reference.
[Weakness 1 (Concern 1) & Question 2]: Finding-level and process evaluation expose clinically important omissions that high report-metric scores can obscure, and our experiments support their complementary value. The SFT comparison shows both directions explicitly (Tables 1–2):
Recent work likewise extracts and matches structured findings (RadMatch [1]), aligns atomic finding sentences (RadSEM [2]), and scores abnormal findings with attribute-level errors (CRIMSON [3]). These support evaluating clinical units and their details. Our fixed-acquisition test additionally supports observation-aware scoring (Table 4):
Here is a representative example on Gastric-X (Gastric-X 1240) that has high overlap scores despite missing the primary gastric lesion. CIDEr and RadGraph-XL reach the 97th and 95th cohort percentiles, yet only 1/6 reference findings are matched, and EER is 0.
[Weakness 2 (Concern 2)]: ECI-RL rewards intermediate evidence acquisition with supporting ablations, and the revised appendices specify training and evaluation details. The ablations show (Table 3):
The revised appendices provide training settings, prompt templates and evaluator contracts (Appendices B.4, C and D).
[Question 1]: The reviewed radiologist evaluation achieves 73.5% agreement with the judge, and our failure case highlights the need to reserve budget for targeted verification. Several radiologists assessed whether the acquired images support each finding across 20 independent cases. After review, 147/200 valid judgments agree with the judge, with one skipped item (EER: 73.6%, NPHR: 73.1%).
References
[1] Corbière et al. RadMatch: Auditable Radiology Report Evaluation via Finding-Level Matching. arXiv:2609.01470, 2026.
[2] Yang et al. RadSEM: A Finding-by-Finding Metric for Clinical Consistency in Radiology Reports. arXiv:2606.17062, 2026.
[3] Baharoon et al. CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation. MICCAI, 2026.