Grounding Radiology Reports through Test-Time Latent Adaptation

Generating factual radiology reports requires aligning clinical statements with image evidence. A common approach allocates additional test-time computation to sampling and reranking reports from a frozen vision-language model. We identify a coverage–quality gap in this approach: larger candidate pools recover more correct findings, yet these gains do not reliably translate into more factual selected re- ports. This observation motivates using evidence to guide the candidate-generating distribution itself. We introduce Evidence-grounded Latent Adaptation (ELA), a test-time framework that optimizes a small, study-specific latent state within a frozen model. Feedback from frozen visual evidence models guides the update by penalizing unsupported assertions and omitted findings. The adapted state then conditions report generation, after which a fixed evidence-based selector chooses the final report. Adaptation requires neither reference reports nor backbone param- eter updates, and the latent state is discarded after each study. Experiments across multiple backbones and chest X-ray benchmarks show consistent improvements in report factuality. Controlled analyses isolate gains from latent adaptation beyond evidence-based selection and show that adapted candidate pools can outperform larger pools from the fixed policy. These findings support evidence-guided adap- tation of the generation distribution as a complementary direction for test-time scaling in radiology reporting.