acceptodds
Under review as a conference paper at ICLR 2027

RAV: Retrieval-augmented Training of LLM Judges for Chest X-ray Report Verification

Abstract

The use of LLM-as-a-judge model is popular for evaluating language generation in radiology report generation. However, reference-free evaluation is prone to self-preference bias preferring hallucinated plausibility to clinical factuality when no objective ground truth is present. In this paper, we take a new approach to fact-checking by training such judge models to exploit the consensus present in prior radiology reports of similar patient images. Specifically, a new loss function is designed to capture the consensus of findings present in similar patient reports while allowing for patient-specific individuality and sparsity in the reporting of findings. We show through extensive studies across datasets, judge models and report generation models that such retrieval-based consensus-trained judge models improve the overall quality of automated reports by 8-17% as measured through comparison with ground truth reports on the reported findings in evaluation settings.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Retrieval-augmented verification, verification AI, radiology report generation, COT Reasoning-based verification

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Z9rDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes retrieval-augmented verification (RAV) for automatically generated chest X-ray reports. Given an image and a candidate report, RAV retrieves visually similar studies using RAD-DINO and uses their reports as contextual evidence. The verifier is trained on synthetic report perturbations with a consensus objective combining finding-frequency supervision, sparsity, and patient-specific supervision. It produces structured finding-level judgments and omissions. Experiments cover four datasets, five report generators, three verifier architectures, and ablations of retrieval and loss components.

Critical strengths
  1. A clinically relevant approach to verification without a reference report at inference time. Using reports from visually similar studies is a practical way to supply additional evidence when the current patient’s reference report is unavailable. The objective explicitly recognizes that retrieved cases are imperfect matches and attempts to balance cohort-level consensus with patient-specific findings.

  2. Evaluation across multiple generators, verifiers, and datasets. The experiments investigate whether the approach transfers beyond a single model configuration. The retrieval, loss-component, and shuffled-consensus analyses provide useful evidence about the behavior of the proposed system, particularly the risks of accepting retrieved findings indiscriminately.

Critical weaknesses
  1. The evaluation does not sufficiently establish the claimed report-verification capability. The primary endpoint is FFL-F1 computed from post-processed finding-level predictions. Although this representation captures attributes such as location, laterality, and severity, improved agreement between finding sets does not establish that the verifier correctly identifies errors in the original report. A model could improve this endpoint by independently predicting better findings without accurately identifying which statements are erroneous or why. Report verification also involves descriptive distinctions, uncertainty, temporal comparisons, and relationships between statements that may not be fully captured by this representation. The paper produces error categories and omission judgments but does not provide sufficient expert-annotated evaluation of these outputs. Appendix Table 16 reports RadGraph-F1 and CheXbert-F1, but without corresponding baseline scores, the claimed improvements under these additional metrics cannot be assessed directly.

  2. The baseline comparisons leave uncertainty about both the source and generality of the gains. It is unclear whether the CoT-only and retrieval-prompting baselines receive the same supervised fine-tuning data and training budget as the complete RAV model. Consequently, the comparisons do not clearly isolate retrieval-based consensus learning from the benefits of additional supervision or a second image-interpretation model. Furthermore, although five generators are evaluated, several recent high-performing systems represented on ReXrank are not considered. Testing RAV on stronger generators is important: their reports may contain fewer straightforward omissions and more subtle descriptive errors, leaving different opportunities for improvement. Where checkpoints or generated reports are available, such evaluations would strengthen the claim of generator-agnostic verification.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer Comment-1

Thank you for distinguishing agreement with the reference finding set from accuracy in verifying the original report. Our primary endpoint, FFL-F1, measures agreement between the final finding set and the reference. As the reviewer notes, a verifier could improve this score by independently predicting a better set of findings, even if it misidentified which candidate statements were erroneous. We will narrow the claims in the manuscript accordingly. The FFL lexicon supports up to 17 modifiers. Our evaluation used a subset that includes location, laterality, and severity. This captures clinically relevant distinctions despite differences in wording. Future work will handle uncertainty, temporal comparisons, and relationships between statements as mentioned in the manuscript. We agree that Table 16 reports only the absolute RadGraph-F1 and CheXbert-F1 scores for RAV. We will revise Appendix F to add the results from other baseline methods.

AuthorsOpenReview ↗

Response to Reviewer comment-2

Thank you for identifying two gaps in the comparison. The reported CoT-only and retrieval-prompting comparisons do not isolate the effect of the consensus objective from task-specific supervised fine-tuning or from a second model interpreting the image. We will clarify the training setup for each baseline. A matched comparison would use the same judge checkpoint, synthetic candidate reports, structured verification targets, and optimization budget as full RAV. The supervised-only controls would use the token-level objective, with and without retrieval. Comparing full RAV with the retrieval-enabled control would isolate the contribution of the consensus objective. This experiment will be added in the revised version.

When we ran the reported experiments, we did not have access to CARE-X or to per-study CARE-X reports for our evaluation cases. We agree that evaluating RAV on reports from newer, high-performing ReXrank generators would test whether the gains hold when errors are subtler but these remain moving targets due to the evolving nature of both reasoning models and verifier models evaluating them. Where checkpoints or matched reports are accessible, we will compare the original and RAV-verified findings on the same held-out cases with the verifier held fixed. We will scope the current claim to the five generators evaluated in the manuscript in the revised version.

Thank you for the detailed responses and clarifications. I appreciate the authors' acknowledgment of the limitations and their plans to address them in the revised version. My main concerns regarding the direct evaluation of report verification and the lack of matched supervised baselines remain unresolved by the current results. I will maintain my original rating.

Reviewer_ah31OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors observe that AI-generated chest X-ray reports often contain hallucinations and omissions, and that existing LLM-as-a-judge approaches tend to favor plausible-sounding but incorrect content when no ground-truth report is available. Although there is no ground-truth report for the current patient, hospital archives store a large number of historical images and reports from other patients. The authors therefore use image similarity to retrieve 10 visually similar prior studies, aggregate the findings mentioned in their reports into a frequency distribution, and use it as "noisy consensus evidence" to train a vision-language judge model. The core design is a three-part consensus loss: a frequency term (which favors findings that recur in the retrieved reports), a sparsity term (which prevents the model from simply copying all retrieved findings), and an individuality term (which uses the current patient's annotations to preserve patient-specific findings that are absent from the retrieved cohort). Candidate reports for training are synthesized from the RADCHECK perturbation space and are not tied to any specific report generator. Across 4 datasets, 5 report generators, and 3 judge architectures, verification improves finding-level F1 by 8–17% relatively, with an improvement of about 32% on rare findings.

Critical strengths
  1. The paper's perspective is somewhat novel. Although retrieving similar cases is not new in report generation, combining retrieval augmentation with report verification appears to be a fairly new attempt. The paper offers a clear insight: RAG needs diversity, whereas verification needs similarity. Building on this, the authors design a consensus loss to handle the noise in the retrieved evidence.

  2. The ablation studies are fairly complete and logically rigorous. The authors first use a control experiment (a prevalence-matched null) to show that image similarity indeed provides finding overlap beyond the corpus prior, and only then verify that downstream performance improves. They also conduct a shuffle control experiment: replacing the retrieved distribution with one from a random image drops performance by 73.2%, which rules out the possibility that the model has simply learned generic finding frequencies.

Critical weaknesses
  1. Lack of direct comparison with existing evaluation methods. The medical report domain already has many dedicated radiology report evaluation tools, such as GREEN[1], FineRadScore[2], and VERT[3], but the experiments only compare against "no verification" and the authors' own ablation variants. As a result, the paper cannot quantitatively show how much RAV improves over existing tools. Since the CoT-only baseline achieves an F1 of only 0.29, even lower than the 0.47 of the unverified reports, I believe the reference frame in these experiments is too weak.

  2. Potential data contamination. The training set, the retrieval index, and the evaluation sets CGold/MS-CXR all come from the MIMIC-CXR ecosystem, and CGold is itself a manually annotated subset of the Silver set used for training. The paper does not state whether evaluation samples are excluded from the index during inference-time retrieval. The only clear cross-domain evidence comes from a single dataset, VinDr-CXR.

  3. The evaluation subsets are small, with only 325 cases each, which weakens the reliability of the results. I suggest that the authors further increase the scale of the evaluation.

References:

[1] Ostmeier, Sophie, et al. "Green: Generative radiology report evaluation and error notation." Findings of the association for computational linguistics: EMNLP 2024. 2024.

[2] Huang, Alyssa, et al. "Fineradscore: A radiology report line-by-line evaluation technique generating corrections with severity scores." arXiv preprint arXiv:2405.20613 (2024).

[3] Bologna, Federica, et al. "VERT: Reliable LLM Judges for Radiology Report Evaluation." arXiv preprint arXiv:2604.03376 (2026).

Questions for authors

Please see the weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Comment-1

Thank you for suggesting GREEN, FineRadScore, and VERT. These methods evaluate a candidate radiology report given a reference report for the same study. RAV addresses the setting in which that reference is unavailable to the verifier at inference. It uses the query image, the candidate report, and evidence retrieved from other studies to produce structured finding-level verdicts. We use the query reference only for offline scoring of the reported test results. R We agree that the CoT-only result is weak. It achieves 0.29 F1 on CGold, below the 0.47 F1 of the unverified MedGemma reports. Full RAV achieves 0.51 F1 on the same cases. We will make the 0.47-to-0.51 comparison explicit and describe CoT-only as a prompting ablation. The current results do not establish a quantitative advantage over GREEN, FineRadScore, or VERT as those operate under different assumptions of ground truth availability.

AuthorsOpenReview ↗

Response to Reviewer comment-2 & 3

Thank you for raising this concern. Chest ImaGenome Silver and Gold both derive from MIMIC-CXR, but the Silver subset used to fine-tune RAV and the Gold subset used for evaluation did not overlap. We will specify the split and the identifiers used to check it. Since MS-CXR also draws from MIMIC-CXR, we will report its overlap check against the Silver fine-tuning subset separately.

As discussed in our response to Reviewer 1, the main verification experiments used a separate patient cohort for retrieval. The index excluded the query image, other views from the same study, and images and reports from other studies of the same evaluation patient. We will name the retrieval corpus and document the exclusions. We will also clarify that our external evaluations include both NIH ChestX-ray8 and VinDr-CXR

  1. The evaluation subsets are small, with only 325 cases each, which weakens the reliability of the results. I suggest that the authors further increase the scale of the evaluation.

Thank you for raising the concern about evaluation size. We agree that 325 cases per dataset limits the precision of the estimates, particularly for infrequent findings. However, Chest Imagenome Gold remains currently the only English language sources with both validated reports and ground truth findings. If ground truth reports-based evaluation restriction could be removed, we could use other datasets such as VinDrCXR, PadChest, etc.

Reviewer_rJzcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RAV, a vision-language judge that verifies generated chest X-ray reports using reports retrieved from visually similar cases. The model is trained on synthetically perturbed reports with a loss combining retrieval-frequency matching, a finding-count penalty, and patient-specific supervision. Experiments evaluate finding-level performance across four datasets, five report generators, and three judge architectures.

Critical strengths

The use of retrieved cases to support report verification is well motivated. The comparison against a prevalence-matched null is useful: it tests whether retrieval provides information beyond common findings in the dataset. The evaluation covers several generators and judge architectures rather than a single model combination. The reported improvements are consistent across the tested settings.

Critical weaknesses
  1. The consensus loss operates on finding probabilities y^i\hat y_i, but the model produces a structured text response. I could not determine how these probabilities are obtained from the language model or how the finding-level losses are differentiated. The definition of y^i\hat y_i as the probability that a finding occurs in the candidate report is also confusing: report content and image-supported findings are different targets. Furthermore, Equation 1 states that the loss coefficients are learned, without specifying their constraints or optimization. Simply minimizing unconstrained nonnegative weights would allow the model to reduce the auxiliary loss by driving those weights toward zero. These details are necessary to understand and reproduce the central contribution.

  2. Section 4.3 compares direct prompting, retrieval-augmented prompting, and the complete fine-tuned model. It is unclear whether the first two conditions receive the same synthetic training examples, ground-truth supervision, and optimization budget. Without this information, the improvement from 0.45 to 0.51 in Table 6 cannot be attributed specifically to the consensus loss. The key comparison is against the same judge fine-tuned on the same data with the ordinary supervised objective, both with and without retrieval. The component ablations in Table 7 do not include this baseline.

  3. The paper states that the image–report database is reused during training and inference, and that the query’s ground-truth report is unavailable at inference. However, the exclusion procedure is not described. The authors should specify the retrieval corpus for each experiment and establish that query images, other views from the same study, and reports from evaluation patients cannot enter training or retrieval. This is particularly important for a method whose predictions directly depend on retrieved reports. The current description does not establish that leakage occurred, but it does not adequately rule it out.

  4. The individuality term weights ground-truth supervision by 1−pi1-p_i. Consider a finding present in every retrieved report but absent from the query: pi=1p_i=1 and yigt=0y_i^{gt}=0. The individuality term then provides no corrective supervision, while the frequency loss encourages the incorrect finding. The language-generation objective may counteract this, but the proposed safeguard itself disappears in this case. An analysis of cases where retrieval confidently disagrees with the query would directly test the claimed selective use of evidence. The shuffle experiment shows sensitivity to retrieval—it does not establish that the judge can reject misleading retrieval.

  5. The main results compare the final finding set with reference findings. This is useful, but it does not show how accurately the judge identifies hallucinations, negation errors, location errors, or omissions—the outputs emphasized in the method. Error-type precision and recall, together with the rate at which correct candidate findings are incorrectly rejected, would make the verification claim more convincing. In addition, Table 16 reports RadGraph and CheXbert scores only for RAV, without the corresponding generator baselines, so it does not substantiate the stated improvement across those metrics.

Questions for authors
  1. Are the p_i values per-report occurrence frequencies or probabilities normalized across finding types? Figure 2 is labeled “Normalized Probability Distribution,” whereas Equation 3 interprets ∑ipi\sum_i p_i as the expected number of findings. Please clarify whether the figure uses a separate normalization.

  2. How are reference finding sets constructed and harmonized across datasets with different annotation inventories? In particular, how are unannotated findings distinguished from explicitly absent findings?

3 comments
AuthorsOpenReview ↗

Response to the reviewer Comments 1 & 2

Our response is split across comments due to character limit:

  1. The consensus loss operates on finding probabilities, but the model produces a structured text response. I could not determine how these probabilities are obtained from the language model or how the finding-level losses are differentiated. ....

We thank the reviewer for identifying an important ambiguity in our original description. We agree that extracting findings from an autoregressively generated report, by itself, does not establish a differentiable pathway through which the consensus loss can update the verifier. We have therefore clarified and extended the architecture to explicitly incorporate a differentiable finding-level prediction pathway within the verifier. Specifically, the revised architecture employs two complementary pathways built upon a shared Vision-Language Model (VLM) backbone. (1) COT-based language verification pathway: This pathway performs autoregressive reasoning over the query chest X-ray, candidate report, and retrieved reports, producing the final structured JSONL output. It is supervised using the standard next-token prediction objective, with the target representing the desired verification analysis derived from the candidate report and query-specific ground-truth report. (Please see sample COT template and output in appendix). (2) Finding-level verification pathway: The verifier has a separate, differentiable finding-prediction head attached to the shared model. In Equations 2–4, (\hat y_i) is this head’s estimated probability that finding (i) is present in the query study. We will revise the definition in Section 3.2 to distinguish this probability from whether the candidate report mentions the finding. Equation 2 uses retrieved finding frequencies as soft targets. For Equation 4, the ground-truth radiology report paired with the query image is used during training to extract finding labels for the auxiliary head. We compute the consensus loss directly from the auxiliary head’s probabilities, before decoding the JSON analysis. Its gradients update the head and the trainable parameters of the shared verifier. A separate token-level loss trains the verifier to generate the structured reasoning and verdicts. We will document where the head attaches, how its outputs are converted to finding probabilities, and how the two objectives are combined. We agree that directly minimizing Equation 1 over freely adjustable nonnegative coefficients allows all three weights to shrink toward zero. For this reason, we use one scalar (s_j) for each of the frequency, sparsity, and individuality losses and setting (\lambda_j=\exp(-s_j)). The revised consensus objective is actually [ L_{\mathrm{consensus}} =\sum_{j=1}^{3}\left[\exp(-s_j)L_j+s_j\right]. ] The scalars are optimized jointly with the verifier parameters. Each effective weight is positive, and driving it toward zero incurs an increasing penalty because (\exp(-s_j)\to 0) requires (s_j\to+\infty).

  1. Section 4.3 compares direct prompting, retrieval-augmented prompting, and the complete fine-tuned model. It is unclear whether the first two conditions receive the same synthetic training examples, ....

Thank you for pointing out the need for a matched supervised control. Prior training on MIMIC image–report pairs differs from task-specific fine-tuning to verify an error-containing candidate report. In the proposed control, the judge would receive the same query image and synthetic candidate report as full RAV and would be trained with the same ground-truth-derived structured verification targets. The errors are in the candidate input. The target identifies which candidate findings are supported or unsupported and which findings were omitted. We agree that the relevant comparison is ordinary token-level supervised fine-tuning of the same judge, with and without retrieval. Both controls would use the same starting checkpoint, synthetic examples, verification targets, and optimization budget as full RAV. The retrieval-enabled control would also receive the same retrieved evidence as full RAV. Comparing that control with full RAV would isolate the incremental effect of adding the consensus objective. We will clarify the training setup for each row in Table 6 and revise Section 4.3 to describe the increase from 0.45 to 0.51 F1 as a comparison between the reported systems. Table 7 addresses the contribution of individual terms within the consensus objective.

AuthorsOpenReview ↗

Response to Reviewer Comments 3-5

The paper states that the image–report database is reused during training and inference, and that the query’s ground-truth report is unavailable at inference. However, the exclusion procedure is not describe....

Thank you for asking us to clarify the retrieval splits. For the main verification experiments, the retrieval index was built from a separate patient cohort that was disjoint from the cohorts used for RAV fine-tuning, validation, and evaluation. The same reference index was queried during fine-tuning and inference. Because the split was at the patient level, the index excluded the evaluation query image, other views from the same study, and images and reports from any other study of the same evaluation patient. We will name the retrieval corpus used in each experiment and describe how patient and study overlap was checked. We will also document separately how evaluation patients were excluded from the fine-tuning data. Finally, we will distinguish the main verification experiments from the prevalence analysis in Appendix C, which used CGold as the retrieval corpus for VinDr-CXR queries.

  1. The individuality term weights ground-truth supervision ....

The individuality term is designed to preserve query-specific findings when they are uncommon in the retrieved reports. When all retrieved reports affirm finding (i) ((p_i=1)) and the query reference labels it absent ((y_i^{\mathrm{gt}}=0)), the factor (1-p_i) is zero, while the frequency term pushes the auxiliary prediction upward. The token-level verification objective can still supervise the correct verdict, but the individuality term provides no corrective signal in this case. We will clarify this limitation in the manuscript. The shuffle experiment establishes sensitivity to which studies are retrieved. Whether the judge can reject confidently misleading retrieval remains an open question. A follow-up analysis could measure how often most or all retrieved reports affirm a finding labeled absent in the query and how often the judge incorrectly flags that finding as an omission, with and without the retrieved evidence.

  1. The main results compare the final finding set with reference findings...

Thank you for distinguishing the quality of the final finding set from the accuracy of the judge’s verification decisions. Our main results evaluate the final findings against the reference. The aggregate false-negative and spurious-finding rates in Table 6 do not show whether the judge correctly identifies specific errors. We will clarify the scope of those results in the final version. We will add the breakdown for the hallucinations, etc. from direct evaluation of the structured JSON verdicts which can report precision, recall, and the number of reference cases for hallucinations, negation errors, location errors, and omissions. We will also report the fraction of correct candidate findings that the judge incorrectly flags as errors. This will show whether the judge identifies errors while preserving correct findings. We also agree that Table 16, as presented, does not establish improvement on RadGraph-F1 or CheXbert-F1. That comparison requires scoring the original generator reports and the corresponding RAV-verified outputs against the same reference reports on the same cases. We will revise Appendix F to describe the reported RadGraph and CheXbert values as absolute scores and reserve the claim of improvement for metrics with a matched generator baseline.

AuthorsOpenReview ↗

Response to reviewer for additional questions posted for authors

. Questions for authors: Are the p_i values per-report occurrence frequencies or probabilities normalized across finding types? Figure 2 is labeled “Normalized Probability Distribution,” ....

Thank you for pointing out the ambiguity. Figure 2(a) divides each finding’s occurrence count by the total number of finding occurrences across the ten reports retrieved for the example query. These values sum to one. Figure 2(b) shows raw corpus-wide counts for a subset of frequent findings. The purpose of the figure is to illustrate how the mix of findings in a patient-specific retrieved set can differ from the corpus-wide mix. We will revise the caption to state the denominator and the set of findings shown in each panel. The expected-count interpretation in Equation 3 requires a different denominator. The per-report frequency of finding (i) is the number of retrieved reports containing that finding divided by (K), counting each finding at most once per report. Summing those frequencies gives the mean number of distinct findings per retrieved report. Summing the values plotted in Figure 2(a) gives one. We will distinguish the two quantities in the notation and specify which was used in the reported training runs.

Question 2. How are reference finding sets constructed and harmonized across datasets with different annotation inventories? In particular, how are unannotated findings distinguished from explicitly absent findings?

Clinicians reconciled each dataset’s annotation inventory with the shared chest X-ray finding lexicon. We applied these mappings to construct a reference finding set for each study. We will add the clinician-derived mapping table to the appendix and show which findings each dataset covers. An explicitly absent finding has a negative annotation in the source dataset. An unannotated finding has no recorded label. For scoring, we treated both as absent, so a positive prediction for an unannotated finding could count as a false positive. This assumes that the curated annotations capture the findings relevant to the evaluation. Differences in annotation coverage across datasets can therefore affect the measured precision. We will make this scoring rule and its limitation explicit in the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.