- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ReportQA, a question-answering framework for evaluating whether generated radiology reports preserve the clinical information in reference reports. Using knowledge trees developed with radiologist involvement, the framework decomposes reports into clinical entities and attributes and constructs multiple-choice questions covering fine-grained information, ancestor-level concepts, and negative entities. The resulting resource contains approximately 660,000 QA pairs derived from 6,857 reports across four datasets. The authors further propose QAScore, which measures the preservation of positive clinical entities and the introduction of spurious entities. Experiments demonstrate strong rank correlations with expert judgments and reveal limitations in current models' handling of fine-grained clinical attributes. The paper also compares report-based and question-driven inference to investigate textual priors and the use of visual information.
Critical strengths
-
The framework provides fine-grained, interpretable error analysis with clear practical value. ReportQA evaluates not only whether clinical entities are present but also whether their attributes, such as location, shape, and severity, are correctly described. Hierarchical questions additionally capture different levels of clinical specificity. This design helps identify which clinical information a model omits or misrepresents, rather than reducing report quality to an overall similarity score. Entity-level aggregation also prevents an entity from receiving greater weight solely because it has more associated questions. These properties make the framework useful for diagnosing weaknesses in report generation systems and constitute an important contribution.
-
The resource has broad coverage and is supported by human verification and expert-correlation experiments. ReportQA covers chest X-rays and brain, chest, and abdominal CT, with approximately 660,000 QA pairs. Two radiologists independently audited 531 sampled questions, each achieving over 97% agreement with the predefined answers. QAScore also achieves higher Spearman and Kendall correlations with expert judgments than the compared metrics on both evaluated expert-annotated sets. These results support the quality of the QA resource and the practical value of the metric, extending the contribution beyond the proposal of an evaluation framework.
Critical weaknesses
-
The robustness experiments do not fully establish scoring reliability on erroneous generated reports. Appendix A.8 compares two judge models primarily by testing whether they can recover reference answers when given ground-truth reports. This differs from assessing generated reports containing omissions, incorrect attributes, or contradictory information: accurately reading a reference report does not necessarily imply reliably identifying errors in a generated report. Moreover, QA filtering itself depends on a judge model, potentially favoring questions that this evaluator can handle. Although the expert-correlation experiments provide meaningful evidence of validity, the paper does not show whether scores, system rankings, and correlations with expert judgments remain stable when the judge is changed on actual generated reports. Because a reliable evaluation tool is the paper's central contribution, this gap limits confidence in the robustness and reusability of the proposed protocol.
-
Differences in information and evaluation procedures between inference settings limit the conclusions about fine-grained clinical representations. In question-driven inference, the model receives questions derived from reference reports and a set of answer choices; some questions also specify the clinical entity and its anatomical location. Report-based inference instead requires the model to identify abnormalities, select relevant information, and organize a report, which is subsequently read by a separate judge to recover answers. The two settings therefore differ in their informational cues, output constraints, and evaluation procedures. The image-free experiments acknowledge textual priors but do not fully isolate these factors. The results support the observation that targeted questions can elicit clinical information more effectively, but they do not by themselves establish that report-based inference struggles to learn fine-grained clinical representations or that question-driven inference produces better complete reports. This weakens the broader conclusions about the relative merits of the two inference paradigms.
Questions for authors
-
Scoring robustness on erroneous reports. The cross-judge experiment in Appendix A.8 uses ground-truth reports as context. When actual generated reports are evaluated, does changing the judge affect report rankings, system rankings, or correlations with expert judgments? If available, please provide these results to support the extension from reference-report QA accuracy to reliable scoring of generated reports.
-
Controls for the inference-paradigm comparison. Question-driven inference receives reference-derived questions and answer choices, while report-based inference includes an additional report-reading step. What controlled experiments distinguish the effects of question cues, output constraints, and judge errors on the observed performance gap? After accounting for these factors, what evidence supports the conclusion that report-based inference struggles to learn fine-grained clinical representations?
W1 & Q1
Thanks for raising this question. We extended our robustness analysis to generated reports, which may contain errors. As shown in the table, we evaluated the generated reports with DeepSeek-V4-Flash and Qwen3.5-27B. For 97.40% of questions, the two judges selected the same answer. This result provides further evidence of robustness to a change of judge.
Additionally, we used Qwen3.5-27B during QA construction to screen out labels that could not be reliably recovered from the reference reports. The goal was to improve label quality. The choice of evaluation judge is independent of the choice of model used for QA filtering.
We will include this robustness analysis and clarify the purpose of LLM-based filtering during QA construction in the revised version.
W2 & Q2
Thanks for pointing this out. We agree that the comparison between the two inference paradigms may not be entirely fair. To make the comparison fairer, we included detailed guidance in the prompt for zero-shot report-based inference, as shown below.
We intended to emphasize that performance on base and negative questions points to a stronger negative bias in report-based inference, both in the zero-shot setting (Figure 5) and after SFT (Figure 4). With report-level training data, it is difficult to control the balance of examples for specific findings and attributes. As discussed in Section 4.6, question-driven inference offers a promising direction: balancing training examples at the question level may allow finer control over how models recognize positive and negative findings and their attributes.
We will clarify the two evaluation settings in the revised version.