acceptodds
Under review as a conference paper at ICLR 2027

Grounding Radiology Reports through Test-Time Latent Adaptation

Abstract

Generating factual radiology reports requires aligning clinical statements with image evidence. A common approach allocates additional test-time computation to sampling and reranking reports from a frozen vision-language model. We identify a coverage–quality gap in this approach: larger candidate pools recover more correct findings, yet these gains do not reliably translate into more factual selected re- ports. This observation motivates using evidence to guide the candidate-generating distribution itself. We introduce Evidence-grounded Latent Adaptation (ELA), a test-time framework that optimizes a small, study-specific latent state within a frozen model. Feedback from frozen visual evidence models guides the update by penalizing unsupported assertions and omitted findings. The adapted state then conditions report generation, after which a fixed evidence-based selector chooses the final report. Adaptation requires neither reference reports nor backbone param- eter updates, and the latent state is discarded after each study. Experiments across multiple backbones and chest X-ray benchmarks show consistent improvements in report factuality. Controlled analyses isolate gains from latent adaptation beyond evidence-based selection and show that adapted candidate pools can outperform larger pools from the fixed policy. These findings support evidence-guided adap- tation of the generation distribution as a complementary direction for test-time scaling in radiology reporting.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
latent reasoning, test-time, vision-language models, radiology report generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xvGkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Evidence-grounded Latent Adaptation (ELA). At test time, ELA optimizes a small, study-specific continuous latent state inside a frozen VLM using feedback from external frozen visual evidence models. The adapted latent state adjusts candidate report generation, and a fixed evidence-based selector chooses the final report.

Core Claim: External visual evidence can be used not only to filter reports post-hoc, but also to adjust the generation distribution itself via test-time latent optimization.

Critical strengths
  1. Distinguishing evidence-based selection from distribution adaptation: Prior work mainly relies on sampling more reports and reranking, but a selector cannot surface findings absent from the candidate pool. ELA feeds visual evidence into the generation step to actively alter the report distribution. This is a sound and valuable direction.

  2. Controlled comparison isolating the latent update: Under a matched no-update control (z=0z=0) sharing the same selector and generation protocol, ELA yields a +2.88 GREEN and +5.87 RadFact-F1 gain. This demonstrates that the performance gain does not come solely from stronger candidate selection.

Critical weaknesses
  1. Limited methodological novelty over prior test-time latent optimization: Methods like LTPO have already explored optimizing instance-level latents in frozen models via policy gradients. ELA’s main difference is swapping the confidence reward for external visual evidence in radiology reporting. While practical, the paper does not show that ELA introduces a new mechanism beyond existing test-time optimization frameworks. In particular, if the same visual evidence is used in existing latent optimization methods, or supplied directly as a conditioning input during generation, would it achieve comparable results? There is currently insufficient evidence showing that ELA's specific mechanism is irreplaceable.

  2. The coverage-quality gap lacks deeper analytical insight: That a pool covers a correct finding while the selected full report remains inaccurate is not particularly counterintuitive, as coverage and single-report factuality evaluate different things. Furthermore, Figure 2(c) shows substantial headroom between the selected candidate and the oracle best-in-pool report, indicating the selection mechanism itself remains a major bottleneck. The paper shows adapting the distribution helps, but does not clearly characterize when adapting the generator is more critical than improving the selector. Finally, comparing asymmetric and symmetric rewards simultaneously changes the underlying evidence model source, so this ablation does not cleanly isolate the reward formulation itself.

Questions for authors
  1. What is the fundamental mechanism in ELA that cannot be realized by existing instance-level latent optimization methods if they are given the same external visual feedback?

  2. Under what specific conditions is adapting the generation distribution strictly better than improving candidate selection? Can you offer a more generalizable takeaway from these results?

4 comments
AuthorsOpenReview ↗

Response to Reviewer xvGk: Methodological Contribution and Coverage–Quality Analysis

We thank the reviewer for the careful assessment and the specific comparison suggestions.

W1 & Q1. Novelty relative to LTPO and direct conditioning

A1. The same visual feedback can guide an existing latent optimizer, as the requested LTPO comparison confirms. We therefore position ELA within instance-level latent optimization. Its contribution is a report-level clinical objective that distinguishes unsupported assertions from omissions, used to adapt generation before evidence-based selection. ELA updates complete-report log-probabilities with group-relative rewards and a KL anchor; LTPO estimates updates through Gaussian probes.

On the same 296 IU-XRAY studies, LTPO with image-evidence feedback reaches 46.02 GREEN / 60.77 RadFact-F1, compared with 38.28 / 49.60 for confidence-based LTPO and 40.61 / 57.05 for ELA without final selection. The evidence-guided LTPO variant also scales the learning rate by 30 and assigns degenerate reports reward zero. This comparison demonstrates that the feedback transfers to LTPO; it does not establish that ELA is the only optimizer that can use it.

PromptNo updateELA updateLTPO-style update
ELA42.38 / 59.2345.26 / 65.1041.84 / 59.47
LTPO51.69 / 71.2449.50 / 69.4351.30 / 73.20

ELA updates improve the default-prompt control, while the stronger LTPO prompt produces a no-update baseline that ELA updates do not improve. These are comparisons of the implemented procedures: ELA uses two early-layer latent vectors, whereas the LTPO-style variant uses eight input latent tokens. The LTPO+evidence transfer above also scales the learning rate by 30 and assigns degenerate reports reward zero (Appendix D.5).

For direct conditioning, the two tested greedy prompts list either the 13 condition probabilities or the supported findings. They give 3.28 / 11.65 and 14.45 / 28.82, respectively, versus 24.54 / 36.45 for the original greedy prompt. Neither encoding improves generation. We have clarified the contribution relative to prior latent optimization in Section 2 and reported the feedback, prompt, and update comparisons in Section 5.3 and Appendix D.5 (Tables A11-A12).

W2 & Q2. Coverage, adaptation versus selection, and reward controls

A2. In the reported analysis, adaptation helps most when the initial report contains multiple assertions absent from the reference. This establishes a benefit over the fixed selector, not strict superiority over a stronger selector; the oracle gap still leaves substantial room to improve selection. The candidate-content audit explains why improved finding coverage need not yield a clean complete report. At N=32, only 12 of the 64 recovered findings have a candidate free of other unsupported assertions. A whole-report selector cannot combine correct findings scattered across different erroneous reports.

The study-level results below group the 296 IU-XRAY studies by assertions in the initial greedy report that are absent from the reference. Each entry reports GREEN / RadFact-F1:

Initial extra findingsStudiesSelection onlyELA
08256.68 / 78.0152.12 / 74.74
14039.59 / 51.3738.15 / 59.87
≥217436.29 / 52.1943.65 / 61.76

The group with at least two such assertions benefits most, while reports with none can regress. These are retrospective groups defined using reference reports. The corresponding image-evidence-based analysis across nine settings is reported in Appendix D.6.

The reviewer is also correct that the original reward ablation changed the evidence source. Holding CheXagent-only evidence fixed gives the following comparison, with the direction regularizer switched off or on. Entries are GREEN / RadFact-F1:

RewardNo regularizerWith regularizer
Symmetric38.26 / 53.2338.65 / 54.15
Asymmetric42.77 / 59.3143.09 / 60.35

The asymmetric reward improves both metrics with either regularizer setting. We have expanded the candidate-content and study-level analyses in Sections 3 and 5.3, and replaced Fig. 4(b) with this fixed-evidence reward comparison. Details are in Appendix D.6 (Tables A13-A14).

Dear Reviewer xvGk,

The authors have responded to your concerns regarding novelty and coverage-quality gap. Could you please take a look and indicate whether the response resolves the concerns, and whether your score changes? Thank you very much.

Best,
Area Chair

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. The new reward ablation and expanded evaluation are useful. However, my rating of 2 reflects my assessment of the paper's core mechanism and scientific contribution. The LTPO comparison shows that existing latent optimization methods can also benefit from the same visual feedback. The nine-setting results further show that the gains from ELA's latent updates are modest and vary across settings. I agree that evidence-guided adaptation has practical value, but I still do not see a sufficiently clear new mechanism or deeper insight into why this approach works better than existing alternatives. My concern is about the significance of the methodological contribution. I therefore maintain my rating of 2.

AuthorsOpenReview ↗

Response to Reviewer xvGk

We thank the reviewer for clarifying the concern. ELA builds on established latent optimization. Our intended contribution combines an analysis of a limitation in fixed-policy report selection with a method for using clinical evidence to improve candidate generation.

The analysis goes beyond observing that coverage and factuality measure different things. At N=32N=32, only 12 of the 64 recovered findings appear in a candidate free of other unsupported assertions. Correct findings are therefore often recovered within reports that still contain other errors. This identifies a candidate-content limitation alongside the substantial ranking gap measured by the selector–oracle comparison: both the reports available for selection and the selection decision leave room for improvement.

ELA addresses the generation side by converting image-derived condition evidence into a clinical objective that penalizes unsupported assertions and omissions, then using this feedback to update a temporary latent state within the frozen VLM. On IU-XRAY with Qwen2.5-VL-3B, the candidate-pool audit shows:

Candidate-pool GREENWithout adaptationELA
Mean candidate22.4627.57
Best-in-pool oracle59.4263.28

These measurements assess candidate quality independently of the final ranking rule. The improvement is observable in the available reports themselves, rather than solely in which report is selected.

The evidence-guided LTPO result shows that this feedback can also be used within another optimizer. The fixed-evidence reward comparison further examines how the clinical objective is formulated. Meanwhile, the nine-setting controls show that the incremental benefit of latent updates is modest and varies across settings. Taken together, the contribution is an empirical diagnosis of the candidate-content bottleneck and an evaluated generation-stage intervention. The results support using clinical evidence to improve generation alongside selection, while also identifying the limits of the current implementation.

Reviewer_LYPjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes ELA, a test-time method for improving RRG without updating the VLM backbone. It optimizes study-specific latent tokens using feedback from frozen visual evidence models, then generates and reranks candidate reports.

Experiments on IU-XRAY, CheXpert Plus, and MIMIC-CXR using three VLM backbones show consistent improvements in GREEN and RadFact-F1. Ablations suggest that latent adaptation provides additional gains beyond evidence-based reranking.

Critical strengths
  1. Coverage-quality gap is an interesting observation. On IU-XRAY, increasing the candidate pool from 8 to 32 recovers substantially more missing findings (24% to 65%), while GREEN only improves from 42.38 to 43.62.

  2. The proposed latent adaptation is lightweight in terms of trainable parameters. No reference reports or supervised parameter updates are needed at inference time.

  3. The method improves both primary factuality metrics across all nine backbone-dataset combinations.

  4. The component analysis is useful.

  5. The paper explicitly studies remaining limitations of the approach. The within-pool oracle analysis reveals substantial selection headroom while the condition-level analysis identifies regressions on certain findings.

Critical weaknesses
  1. The improvements attributable specifically to latent adaptation are relatively modest. These controlled experiments are restricted to Qwen2.5-VL-3B on IU-XRAY, so cannot establish whether the adaptation component consistently helps across other backbones and datasets.

  2. The computational overhead is not adequately evaluated. ELA requires 48 rollout reports, additional checkpoint evaluations, backward passes, and 8 final candidates per study. The comparison against a fixed-policy pool of 32 candidates is therefore not compute-matched.

  3. The reward is restricted to 13 CheXbert clinical conditions and does not explicitly evaluate laterality, severity, anatomical location, or temporal changes.

  4. The method depends heavily on the reliability of the frozen evidence models. CheXagent is used in both adaptation and selection, potentially introducing correlated errors. The paper does not sufficiently examine how inaccurate or poorly calibrated evidence scores affect the latent updates.

Questions for authors
  1. Can the authors report the selection-only versus full ELA comparison across all nine backbone-dataset combinations?
  2. What is the actual inference cost of ELA in GPU-seconds, memory usage, and total generated tokens per study?
  3. How sensitive is ELA to the choice and calibration of visual evidence models? In particular, what happens when CheXagent is removed from either the adaptation reward or the final selector?
  4. Could the authors include radiologist evaluation or a more comprehensive metric analysis using something like RadEval?
2 comments
AuthorsOpenReview ↗

Response to Reviewer LYPj: Adaptation Gains and Additional Evaluation. (Part I)

We thank the reviewer for the detailed and constructive review.

W1 & Q1. Adaptation gains across all nine settings

A1. Per reviewer request, we ran the selection-only control in all nine settings, with the final candidate count and selector unchanged and z=0.

SettingNo updateELAΔGREENΔRadFact-F1
IU-XRAY / Qwen2.5-VL-3B42.38 / 59.2345.26 / 65.10+2.88+5.87
IU-XRAY / Qwen3-VL-4B63.82 / 77.9563.47 / 76.79−0.35−1.16
IU-XRAY / HuatuoGPT-7B26.02 / 9.2622.88 / 10.77−3.14+1.51
CheXpert Plus / Qwen2.5-VL-3B19.28 / 52.2917.48 / 55.97−1.80+3.68
CheXpert Plus / Qwen3-VL-4B19.90 / 64.2422.20 / 64.44+2.30+0.20
CheXpert Plus / HuatuoGPT-7B14.98 / 41.5317.61 / 45.28+2.63+3.75
MIMIC-CXR / Qwen2.5-VL-3B13.04 / 47.8912.79 / 47.90−0.25+0.01
MIMIC-CXR / Qwen3-VL-4B18.06 / 56.5818.10 / 57.96+0.04+1.38
MIMIC-CXR / HuatuoGPT-7B11.72 / 34.8811.82 / 35.69+0.10+0.81
Mean25.47 / 49.3225.73 / 51.10+0.27+1.78

ELA improves RadFact-F1 in eight settings and GREEN in five, averaging +0.27 GREEN and +1.78 RadFact-F1. The contribution of latent updates varies with the backbone and dataset and is more consistent in RadFact-F1.

The condition-level breakdown counts assertions absent from the reference (Extra) and reference findings omitted from the report (Missed). Values show selection-only → ELA differences per study:

SettingExtraMissedTotal
IU-XRAY / Qwen2.5-VL-3B0.82 → 0.740.54 → 0.551.36 → 1.29
IU-XRAY / Qwen3-VL-4B0.61 → 0.640.47 → 0.471.08 → 1.11
IU-XRAY / HuatuoGPT-7B1.59 → 1.670.48 → 0.452.08 → 2.12
CheXpert Plus / Qwen2.5-VL-3B0.14 → 0.422.34 → 2.102.48 → 2.52
CheXpert Plus / Qwen3-VL-4B1.02 → 1.021.62 → 1.442.64 → 2.46
CheXpert Plus / HuatuoGPT-7B0.69 → 0.811.99 → 1.742.68 → 2.55
MIMIC-CXR / Qwen2.5-VL-3B0.79 → 0.882.24 → 2.063.03 → 2.94
MIMIC-CXR / Qwen3-VL-4B1.61 → 1.601.51 → 1.423.12 → 3.02
MIMIC-CXR / HuatuoGPT-7B1.47 → 1.451.87 → 1.773.34 → 3.22

ELA reduces missed findings in all six CP and MC settings and total condition-level errors in six of nine settings. Across settings, the gain is +0.77 G / +2.21 RF for studies with at least two evidence-unsupported assertions, versus −0.53 / +0.21 for those with none. We have added the nine-setting control comparison and the study-level analysis in Section 5.3, with detailed results in Appendices D.3 (Tables A6–A7) and D.6 (Table A13).

W2 & Q2. Inference cost and compute matching

A2. The fixed-policy N=32N=32 comparison is not compute-matched to ELA: ELA also performs adaptation rollouts, checkpoint evaluations, and backward passes. The measured costs below average 20 IU-XRAY studies with Qwen2.5-VL-3B on one A800-80G with FlashAttention-2. Time and generated tokens are per study; memory is peak usage.

MethodGPU-secondsMemory (GiB)Tokens
Greedy5.67.4159
Selection (N=8)44.57.61,248
Updates disabled122.510.310,242
ELA141.210.39,828

ELA takes 3.17 times as long as eight-candidate selection. The updates-disabled control retains the 48 rollouts and checkpoint evaluations with z=0z=0. Enabling updates adds 18.7 seconds and changes GREEN / RadFact-F1 from 42.38 / 59.23 to 45.26 / 65.10. It matches the schedule, rather than runtime.

We have added the measured costs in Section 5.3 (Table 4) and corrected the comparison-budget descriptions in Section 5.3 and Appendix D.4 (Tables A8-A9).

W3. Clinical attributes omitted by the reward

A3. The current reward does not directly supervise laterality, severity, anatomical location, or temporal changes. Reports can share the same 13 condition labels while differing in these attributes. The GREEN breakdown on 296 IU-XRAY studies characterizes the resulting errors:

ErrorGreedySelection onlyELA
False finding448300237
Missing finding516519524
Location / position383322
Severity8105
Extra comparison462130
Missing comparison91213

Relative to selection alone, false findings decrease from 300 to 237 and location errors from 33 to 22. Severity errors decrease from 10 to 5, although the counts are small. Missing findings increase from 519 to 524, and extra comparisons from 21 to 30. These results show that the current feedback does not provide complete fine-grained error correction. Direct supervision of these attributes would require finer-grained visual evidence. We have clarified the scope of the implemented reward in Section 4.2 and added the error breakdown in Section 5.3 and Appendix D.2 (Table A4).

AuthorsOpenReview ↗

Response to Reviewer LYPj: Adaptation Gains and Additional Evaluation. (Part II)

W4 & Q3. Evidence dependence and calibration

A4. Performance drops when CheXagent is removed from either the adaptation reward or the selector. The default reward uses CheXagent and TorchXRayVision; the default selector uses BiomedCLIP and CheXagent. These ablations use IU-XRAY with Qwen2.5-VL-3B:

ConfigurationGREENRadFact-F1
Default45.2665.10
Reward without CheXagent39.6056.66
Default, selector rerun*43.4060.58
Selector without CheXagent*38.1654.33

The selector comparison uses the same rerun candidate pool, whose default score is 43.40 / 60.58. It should not be paired with the original run's 45.26 / 65.10. Calibration was not varied independently, so these source-removal ablations do not isolate the effect of miscalibration or correlated errors between the two stages. We have added the removal ablations and evidence-model measurements in Section 5.3 and Appendix D.4 (Table A10), and stated the remaining calibration limitation.

Q4. Radiologist assessment and broader metrics

A-Q4. We added broader automatic evaluation on IU-XRAY with Qwen2.5-VL-3B. The table compares the complete ELA procedure with greedy decoding. A blinded radiologist study is not included.

MetricGreedyELA
RadGraph-F17.7811.01
BERTScore (rescaled)8.3210.64
BLEU-40.520.71
METEOR15.2317.06

We have added these metrics in Section 5.2 and Appendix D.1 (Table A3), and stated the evaluation scope in Section 5.1.

Reviewer_byJCOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Observing that enlarging candidate pools improves finding-level coverage but fails to reliably boost the factuality of the final selected report, the authors propose Evidence-grounded Latent Adaptation (ELA), which optimizes a study-specific latent state within a frozen vision-language model guided by visual evidence feedback, paired with a fixed evidence-based report selector.

Critical strengths
  1. The work finds the under-recognized coverage–quality gap in existing sample-and-rerank paradigms, with empirical evidence showing that larger candidate pools do not translate to better selected report quality, offering a valuable perspective for test-time medical VLM research.
  2. ELA performs instance-level latent optimization at test time without updating backbone parameters or requiring reference reports.
  3. The authors try to disentangle the contributions of latent adaptation and evidence-based selection, with improvements verified across multiple backbones and datasets.
Critical weaknesses
  1. The visual reward operates only on 13 coarse condition categories and cannot supervise fine-grained clinical attributes such as laterality, severity, and precise anatomical location, limiting the precision of factual error correction.
  2. ELA requires multiple rounds of sampling and gradient updates per study, incurring substantially higher inference cost than greedy decoding, while the paper lacks explicit compute–performance trade-off analysis.
  3. A gap exists between the evidence-based selector and the oracle best-in-pool performance, indicating that the current selection module still leaves considerable quality unexploited in the candidate pool.
Questions for authors

Please refer to Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer byJC: Clinical Feedback, Inference Cost, and Selection

We thank the reviewer for the positive assessment and the precise comments.

W1. Fine-grained clinical supervision

A1. We agree that the 13-condition reward does not directly supervise laterality, severity, or precise anatomical location. Two reports can share the same condition labels while differing in these attributes. We evaluated the resulting errors with GREEN on 296 IU-XRAY studies.

Relative to selection alone, the main improvements are fewer false findings and location errors. Severity errors also decrease, although the counts are small. Omissions and comparison errors increase. These changes do not provide the missing attribute-level supervision.

ErrorGreedySelection onlyELA
False finding448300237
Missing finding516519524
Location / position383322
Severity8105
Extra comparison462130
Missing comparison91213

We have stated this limitation in Section 4.2 and added the error breakdown in Section 5.3 and Appendix D.2 (Table A4).

W2. Inference cost and factuality

A2. ELA has substantially higher inference cost than greedy decoding. The table shows the associated factuality gains on IU-XRAY with Qwen2.5-VL-3B. Quality is evaluated on the 296 studies; runtime is averaged over 20 studies on one A800-80G with FlashAttention-2.

MethodTime (s)GREENRadFact-F1
Greedy5.624.5436.45
Updates disabled122.542.3859.23
ELA141.245.2665.10

ELA takes about 25 times as long as greedy decoding. The updates-disabled control retains the rollout and checkpoint-evaluation schedule with z=0; enabling updates adds 18.7 seconds and improves GREEN / RadFact-F1 by 2.88 / 5.87 points. This control isolates the update within the same schedule, but is not wall-time matched. Using four update steps with four rollouts per step gives 45.12 GREEN / 62.17 RadFact-F1 with 16 adaptation rollouts, versus 48 for the default. We have added the measured quality-cost comparison in Section 5.3 (Table 4) and the budget results in Appendix D.4 (Tables A8-A9).

W3. Remaining selector-oracle gap

A3. We agree that the current selector leaves substantial quality unexploited. In the adapted-pool audit in Fig. 2(c), the selected reports score 62.6 RadFact-F1, compared with 83.0 for the reference-based best-in-pool oracle. ELA does not close this gap.

On shared candidate pools, our selector outperforms CCS on both metrics in all nine settings. We have expanded Section 3 to distinguish improved candidate quality from the remaining selection headroom. The nine-setting selector comparison is reported in Section 5.3 and Appendix D.3 (Table A5).

SettingCCSOurs
IU-XRAY / Qwen2.5-VL-3B35.17 / 43.1844.84 / 57.92
IU-XRAY / Qwen3-VL-4B59.06 / 74.3160.87 / 75.46
IU-XRAY / HuatuoGPT-7B14.40 / 4.0218.33 / 6.86
CheXpert Plus / Qwen2.5-VL-3B16.31 / 48.0917.25 / 54.86
CheXpert Plus / Qwen3-VL-4B15.54 / 57.4416.43 / 61.78
CheXpert Plus / HuatuoGPT-7B10.68 / 29.0111.26 / 34.45
MIMIC-CXR / Qwen2.5-VL-3B9.32 / 38.3112.48 / 46.66
MIMIC-CXR / Qwen3-VL-4B14.07 / 51.8315.23 / 54.83
MIMIC-CXR / HuatuoGPT-7B8.90 / 27.1910.01 / 29.61

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.