acceptodds
Under review as a conference paper at ICLR 2027

ReportQA: QA-Based Radiology Report Evaluation

Abstract

Radiology report evaluation is essential for advancing automated radiology report generation. Common natural language generation metrics have limited clinical relevance. Clinical efficacy (CE) metrics evaluate important medical findings, but focus mainly on presence and cover only a limited set of entities. Due to heavy reliance on manual annotations, it is difficult for CE metrics to extend clinical entities or attributes. In clinical practice, radiology reports serve as a medium for information transfer. Clinicians use them to perform downstream diagnostic tasks without directly inspecting images. Based on this insight, we propose **ReportQA**, a clinical-related and flexible radiology report evaluation framework, supporting **detailed quantitative analysis** of radiology report generation systems. We first collect datasets covering multiple imaging modalities and anatomical regions. We then construct knowledge trees of clinical entities and attributes with radiologist guidance, and use large language models (LLMs) to extract structured information from free-form reports. Next, we generate QA pairs from predefined templates and apply quality control through self-filtering and report-based filtering. During evaluation, the report is treated as context, and an LLM acts as a judge model to answer the QA pairs. Based on the resulting QA accuracy, we introduce **QAScore** metric. Compared with existing metrics, QAScore shows better alignment with radiologist judgments. Experiments on multiple state-of-the-art vision-language models reveal that current report-based inference paradigms struggle to learn fine-grained clinical representations and exhibit strong negative prior biases. In contrast, **question-driven inference** provides a more effective alternative. For reproducibility and extensibility, we release knowledge trees, structured reports, and QA pairs, along with the pipeline code for QA construction and evaluation.

Primary area:
datasets and benchmarks
Keywords:
Radiology Reports Evaluation, Question Answering, Visual Question Answering, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_Vb5ZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ReportQA, a question-answering framework for evaluating whether generated radiology reports preserve the clinical information in reference reports. Using knowledge trees developed with radiologist involvement, the framework decomposes reports into clinical entities and attributes and constructs multiple-choice questions covering fine-grained information, ancestor-level concepts, and negative entities. The resulting resource contains approximately 660,000 QA pairs derived from 6,857 reports across four datasets. The authors further propose QAScore, which measures the preservation of positive clinical entities and the introduction of spurious entities. Experiments demonstrate strong rank correlations with expert judgments and reveal limitations in current models' handling of fine-grained clinical attributes. The paper also compares report-based and question-driven inference to investigate textual priors and the use of visual information.

Critical strengths
  1. The framework provides fine-grained, interpretable error analysis with clear practical value. ReportQA evaluates not only whether clinical entities are present but also whether their attributes, such as location, shape, and severity, are correctly described. Hierarchical questions additionally capture different levels of clinical specificity. This design helps identify which clinical information a model omits or misrepresents, rather than reducing report quality to an overall similarity score. Entity-level aggregation also prevents an entity from receiving greater weight solely because it has more associated questions. These properties make the framework useful for diagnosing weaknesses in report generation systems and constitute an important contribution.

  2. The resource has broad coverage and is supported by human verification and expert-correlation experiments. ReportQA covers chest X-rays and brain, chest, and abdominal CT, with approximately 660,000 QA pairs. Two radiologists independently audited 531 sampled questions, each achieving over 97% agreement with the predefined answers. QAScore also achieves higher Spearman and Kendall correlations with expert judgments than the compared metrics on both evaluated expert-annotated sets. These results support the quality of the QA resource and the practical value of the metric, extending the contribution beyond the proposal of an evaluation framework.

Critical weaknesses
  1. The robustness experiments do not fully establish scoring reliability on erroneous generated reports. Appendix A.8 compares two judge models primarily by testing whether they can recover reference answers when given ground-truth reports. This differs from assessing generated reports containing omissions, incorrect attributes, or contradictory information: accurately reading a reference report does not necessarily imply reliably identifying errors in a generated report. Moreover, QA filtering itself depends on a judge model, potentially favoring questions that this evaluator can handle. Although the expert-correlation experiments provide meaningful evidence of validity, the paper does not show whether scores, system rankings, and correlations with expert judgments remain stable when the judge is changed on actual generated reports. Because a reliable evaluation tool is the paper's central contribution, this gap limits confidence in the robustness and reusability of the proposed protocol.

  2. Differences in information and evaluation procedures between inference settings limit the conclusions about fine-grained clinical representations. In question-driven inference, the model receives questions derived from reference reports and a set of answer choices; some questions also specify the clinical entity and its anatomical location. Report-based inference instead requires the model to identify abnormalities, select relevant information, and organize a report, which is subsequently read by a separate judge to recover answers. The two settings therefore differ in their informational cues, output constraints, and evaluation procedures. The image-free experiments acknowledge textual priors but do not fully isolate these factors. The results support the observation that targeted questions can elicit clinical information more effectively, but they do not by themselves establish that report-based inference struggles to learn fine-grained clinical representations or that question-driven inference produces better complete reports. This weakens the broader conclusions about the relative merits of the two inference paradigms.

Questions for authors
  1. Scoring robustness on erroneous reports. The cross-judge experiment in Appendix A.8 uses ground-truth reports as context. When actual generated reports are evaluated, does changing the judge affect report rankings, system rankings, or correlations with expert judgments? If available, please provide these results to support the extension from reference-report QA accuracy to reliable scoring of generated reports.

  2. Controls for the inference-paradigm comparison. Question-driven inference receives reference-derived questions and answer choices, while report-based inference includes an additional report-reading step. What controlled experiments distinguish the effects of question cues, output constraints, and judge errors on the observed performance gap? After accounting for these factors, what evidence supports the conclusion that report-based inference struggles to learn fine-grained clinical representations?

1 comment

W1 & Q1

Thanks for raising this question. We extended our robustness analysis to generated reports, which may contain errors. As shown in the table, we evaluated the generated reports with DeepSeek-V4-Flash and Qwen3.5-27B. For 97.40% of questions, the two judges selected the same answer. This result provides further evidence of robustness to a change of judge.

Additionally, we used Qwen3.5-27B during QA construction to screen out labels that could not be reliably recovered from the reference reports. The goal was to improve label quality. The choice of evaluation judge is independent of the choice of model used for QA filtering.

We will include this robustness analysis and clarify the purpose of LLM-based filtering during QA construction in the revised version.

ModelBaseHierarchicalNegativeAverage
GPT-5.491.92%96.21%98.97%97.31%
Gemini 3.1 Pro94.65%97.16%99.12%98.06%
Claude Opus 4.694.10%96.53%97.90%97.02%
Qwen3.5-27B90.50%98.42%99.41%97.49%
InternVL3.5-38B93.67%95.90%97.25%96.41%
Hulu-Med-32B92.69%96.21%99.81%98.08%
Average92.92%96.74%98.74%97.40%

W2 & Q2

Thanks for pointing this out. We agree that the comparison between the two inference paradigms may not be entirely fair. To make the comparison fairer, we included detailed guidance in the prompt for zero-shot report-based inference, as shown below.

# for CTRG-Brain

You are a radiologist specializing in the interpretation of noncontrast brain CT.

Use the following report structure:

Findings: [A single paragraph describing the anatomy in the order listed below]
Impression: [Concise diagnostic conclusions, each in a separate sentence]

Cover each of the following in order:
1. Cerebral hemispheres: Assess symmetry.
2. Brain parenchyma: Look for focal hypoattenuating or hyperattenuating lesions.
3. Midline structures: Assess whether they are centered.
4. Cerebral sulci, fissures, cisterns, and subarachnoid spaces: Describe whether they are normal, widened, or narrowed.
5. Ventricles: Assess size and shape, noting any enlargement or compression.
6. Brainstem and cerebellum: Note any abnormalities.
7. Incidental extracranial findings: Describe findings such as paranasal sinus mucosal thickening only when present.

Reporting rules:
- Write the Findings as a single paragraph without line breaks.
- Begin with “The cerebral hemispheres are symmetric,”.
- For normal brain parenchyma, write “No obvious focal attenuation abnormality is seen in the brain parenchyma.”
- For each lesion, describe its morphology (patchy, punctate and patchy, or roughly round), attenuation (low or high), location, margins (well-defined or ill-defined), and whether mass effect is present.
- For normal sulci, write “The cerebral sulci and fissures are unremarkable in size and morphology.”
- For age-related sulcal widening, write “The cerebral sulci, cisterns, and subarachnoid spaces are mildly widened, with reduced gyral volume.”
- For a normal brainstem and cerebellum, write “No obvious abnormality is seen in the brainstem or cerebellum.”
- If the entire scan is normal, write the following Impression: “No obvious abnormality on noncontrast head CT. Consider MRI for further evaluation if clinically indicated.”
- If abnormalities are present, state each diagnosis in a separate sentence without joining them with conjunctions.
- Common Impression phrases include “Age-related cerebral changes.”, “Multiple intracranial lacunar infarcts.”, “White matter demyelinating changes.”, “Encephalomalacia in the [location].”, and “Sinusitis.”
- Hounsfield unit values, volumes, and other quantitative measurements are not required.
- Keep the Findings to 2–4 sentences and the Impression to 1–3 items.

Generate a report based on the provided brain CT images.

We intended to emphasize that performance on base and negative questions points to a stronger negative bias in report-based inference, both in the zero-shot setting (Figure 5) and after SFT (Figure 4). With report-level training data, it is difficult to control the balance of examples for specific findings and attributes. As discussed in Section 4.6, question-driven inference offers a promising direction: balancing training examples at the question level may allow finer control over how models recognize positive and negative findings and their attributes.

We will clarify the two evaluation settings in the revised version.

Reviewer_86MPOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReportQA, a framework that evaluates generated radiology reports by having an LLM judge answer questions using the report as context. Radiologist-guided knowledge trees of clinical entities and attributes are used to turn reference reports into structured records with DeepSeek-V3.2. Template questions are then generated at three levels (leaf, ancestor, and negative presence) and filtered by the Qwen3.5-27B judge, leaving about 660K QAs for 6857 reports from MIMIC-CXR, CTRG-Brain, CT-RATE, and AMOS-MM. The proposed QAScore =1/(1+Eentity)= 1/(1 + E_{\text{entity}}) counts missed or wrong positive entities and false-positive absent entities. The paper reports better alignment with radiologist judgments than existing metrics on RadEvalX and a CT-RATE subset, and uses the benchmark to study current VLMs and two inference paradigms (report-based vs. question-driven).

Critical strengths
  • The fixed, precomputed QA bank covers four imaging settings (chest X-ray and brain, chest, and abdominal CT), so all systems are scored on the same questions. The answer keys were also checked in a blinded radiologist audit, with 97.36% and 98.12% agreement for the two radiologists (A.2, Table 4).
  • The alignment study is broad and candidly reported. Table 2 compares QAScore with 11 metrics, including GREEN and three ICARE variants, and QAScore is best on Spearman and Kendall on both datasets (e.g., CT-RATE Spearman 0.8997 vs. 0.6518 for ICARE-AVG). The paper also openly notes that it is not the best on CT-RATE Pearson.
Critical weaknesses
  • I am not convinced that QAScore penalizes omissions enough, because correctly rejected negative entities contribute zero error. Using the released QA bank and scoring code, an empty report would score 0.4560 on CTRG-Brain and 0.0810 on AMOS-MM (if the judge rejects all negatives), above every model in Table 3 on both datasets (best 0.1471 and 0.0653). A trivial empty or "normal" report baseline is not reported.
  • I could not find evidence that the conclusions hold beyond one judge. Qwen3.5-27B is used both to filter the QA bank (removing items it answers poorly given the reference report, Sec. 3.3) and to score reports, so its 0.9597 accuracy (A.8) is high partly by construction. Tables 2 and 3 are not re-scored with a second judge, such as DeepSeek-V4-Flash.
  • I am not convinced by the claim that question-driven inference "provides a more effective alternative" (in abstract). The evidence comes from one model (Qwen3.5-2B) on one dataset (CTRG-Brain, Fig. 4). In the released CTRG-Brain bank, every positive question has the answer "Yes" and every negative one "No", so per-type accuracy mixes a yes/no tendency with actual recognition.
Questions for authors

Please see the weaknesses.

5 comments

W1

We thank the reviewer for highlighting the all-negative baseline. RRG can omit positive findings or introduce findings absent. Both types of error can pose clinical risks. QAScore therefore assigns them the same unit cost at the entity level. Completely omitting a positive entity and falsely reporting a negative entity each count as one error unit. Correctly rejecting a negative entity contributes zero error, while omissions still count in full toward the total error. A report that omits all N+N_+ positive entities and introduces no false positives scores 1/(1+N+)1/(1+N_+).

The main reason for the high all-negative score on CTRG-Brain is that the ground-truth distribution is skewed toward negative findings. 801 of the 2,001 GT reports (40.03%) describe normal examinations. These reports contain no positive findings and therefore provide no positive entities from which to generate positive QAs. Under the all-negative strategy, these cases incur neither omissions nor false positives and each scores 1, raising the overall mean to 0.4560. On the remaining 1,200 abnormal cases alone, the mean drops to 0.0930.

By comparison, all 390 reports in AMOS-MM contain positive QAs, with an average of 16.92 positive entities per case. Although the all-negative strategy introduces no false positives, omitting all positive entities yields a mean QAScore of only 0.0810, reflecting the cumulative penalty for omissions. This value still exceeds the best model score of 0.0653 in Table 3, because a generated report scores below the all-negative strategy on the same case when the false-positive error it introduces exceeds the error reduction from recovering positive information.

If an application has different tolerances for omissions and false positives, a relative weight λ\lambda can be introduced into the original formula.

= \frac{1}{1 + \lambda \sum_{e \in E^+}(1 - q_e) + \sum_{a \in E^-} f_a}, \qquad \lambda > 0.$$ Here, $q_e$ is the quality score of positive entity $e$, and $f_a$ is the false-positive error contribution of negative entity $a$. Setting $\lambda=1$ recovers the default setting in the paper. Applications that prioritize avoiding omissions can use $\lambda>1$, while those that prioritize avoiding false positives can use $0<\lambda<1$. This weight can be adjusted to reflect the priorities of the evaluation. Additionally, ReportQA supports fine-grained analysis by question type (base, hierarchical, and negative) and by attribute (e.g., presence, location, and size). Interpreting QAScore alongside these breakdowns gives a fuller account of what clinical information a report preserves, what it omits, and what errors it introduces. In the revised version, we will report and analyze the results for the all-negative baseline and explain how to adjust the relative weight between omissions and false positives to reflect the priorities of different applications. ## W2 Thank you for highlighting the consistency with different judge models. We evaluated the generated reports with DeepSeek-V4-Flash and compared its results with those of Qwen3.5-27B, as shown in the table. The two judges achieved 97.40% agreement, indicating strong consistency in their evaluations. We will include these results in the revised version. | Model | Base | Hierarchical | Negative | Average | | :-------------- | :----: | :----------: | :------: | :----: | | GPT-5.4 | 91.92% | 96.21% | 98.97% | 97.31% | | Gemini 3.1 Pro | 94.65% | 97.16% | 99.12% | 98.06% | | Claude Opus 4.6 | 94.10% | 96.53% | 97.90% | 97.02% | | Qwen3.5-27B | 90.50% | 98.42% | 99.41% | 97.49% | | InternVL3.5-38B | 93.67% | 95.90% | 97.25% | 96.41% | | Hulu-Med-32B | 92.69% | 96.21% | 99.81% | 98.08% | | Average | 92.92% | 96.74% | 98.74% | 97.40% | ## W3 Thank you for raising this point. We agree that “provides a more effective alternative” is too strong. We intended to emphasize that the results on base and negative questions point to a stronger negative bias in report-based inference, both in the zero-shot setting (Figure 5) and after fine-tuning (Figure 4). As discussed in Section 4.6, question-driven inference offers a promising direction: adjusting the balance of positive and negative training examples may allow finer control over these biases. We will soften the claim in the abstract of the revised version.

Thank you for the detailed response. My concern on W1 is only partially addressed: on AMOS-MM, an empty report still outscores every model, which you confirmed. Two questions:

  1. Would radiologists also rank an empty or “normal” template report above the model outputs? If not, adding these trivial baselines to the alignment study (Table 2) would show that QAScore disagrees with experts here. How do you justify this?

  2. If λ\lambda is the intended fix, how would λ\lambda be chosen in advance (not tuned to beat the baseline)? With that λ\lambda, do trivial baselines fall below all models on every dataset while Table 2 correlations hold?

We thank the reviewer for the response! We will answer both questions below:

  1. We would like to clarify that, although the all-negative baseline is a trivial baseline, it may not be the worst baseline. For example, if the ground-truth report contains five positive findings, the all-negative baseline misses all five and incurs five errors. If a generated report recovers one finding but introduces two false positives, it incurs six errors. Under the evaluation protocol* used in Table 2, radiologists would rank the all-negative baseline above the generated report in this example. Additionally, AMOS-MM covers multiple anatomical regions, including the chest, abdomen, and pelvis, which contain complex information. The results further demonstrate the poor performance of current VLMs in complex settings.

  2. We would like to clarify that the introduction of λ\lambda was intended to provide different evaluation settings, but not for a fix. The radiologist evaluation protocol in Table 2 treats omissions and false positives as equally severe for scoring purposes, which corresponds to λ\lambda = 1. For a specific application, λ\lambda should be determined before evaluation based on the relative costs of the two types of errors, without choosing it to make the all-negative baseline rank worst. Specifically, λ\lambda > 1 increases the relative cost of omitting or incorrectly describing positive entities, while 0 < λ\lambda < 1 increases the relative weight of false positives. Additionally, changing λ\lambda from 1 breaks the alignment in how errors are weighted. QAScore then assigns different weights to omissions and false positives, while the radiologists’ total error counts in Table 2 weight them equally.

*The radiologist evaluation protocol counts eight types of errors, including false-positive findings, omitted findings, incorrect location, incorrect severity, unnecessary or missing comparisons with prior examinations, and unnecessary or missing expressions of uncertainty. Report quality is measured by the sum of these error counts, with fewer errors indicating higher quality.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.