- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Evidence-grounded Latent Adaptation (ELA). At test time, ELA optimizes a small, study-specific continuous latent state inside a frozen VLM using feedback from external frozen visual evidence models. The adapted latent state adjusts candidate report generation, and a fixed evidence-based selector chooses the final report.
Core Claim: External visual evidence can be used not only to filter reports post-hoc, but also to adjust the generation distribution itself via test-time latent optimization.
Critical strengths
-
Distinguishing evidence-based selection from distribution adaptation: Prior work mainly relies on sampling more reports and reranking, but a selector cannot surface findings absent from the candidate pool. ELA feeds visual evidence into the generation step to actively alter the report distribution. This is a sound and valuable direction.
-
Controlled comparison isolating the latent update: Under a matched no-update control () sharing the same selector and generation protocol, ELA yields a +2.88 GREEN and +5.87 RadFact-F1 gain. This demonstrates that the performance gain does not come solely from stronger candidate selection.
Critical weaknesses
-
Limited methodological novelty over prior test-time latent optimization: Methods like LTPO have already explored optimizing instance-level latents in frozen models via policy gradients. ELA’s main difference is swapping the confidence reward for external visual evidence in radiology reporting. While practical, the paper does not show that ELA introduces a new mechanism beyond existing test-time optimization frameworks. In particular, if the same visual evidence is used in existing latent optimization methods, or supplied directly as a conditioning input during generation, would it achieve comparable results? There is currently insufficient evidence showing that ELA's specific mechanism is irreplaceable.
-
The coverage-quality gap lacks deeper analytical insight: That a pool covers a correct finding while the selected full report remains inaccurate is not particularly counterintuitive, as coverage and single-report factuality evaluate different things. Furthermore, Figure 2(c) shows substantial headroom between the selected candidate and the oracle best-in-pool report, indicating the selection mechanism itself remains a major bottleneck. The paper shows adapting the distribution helps, but does not clearly characterize when adapting the generator is more critical than improving the selector. Finally, comparing asymmetric and symmetric rewards simultaneously changes the underlying evidence model source, so this ablation does not cleanly isolate the reward formulation itself.
Questions for authors
-
What is the fundamental mechanism in ELA that cannot be realized by existing instance-level latent optimization methods if they are given the same external visual feedback?
-
Under what specific conditions is adapting the generation distribution strictly better than improving candidate selection? Can you offer a more generalizable takeaway from these results?
4 comments
Dear Reviewer xvGk,
The authors have responded to your concerns regarding novelty and coverage-quality gap. Could you please take a look and indicate whether the response resolves the concerns, and whether your score changes? Thank you very much.
Best,
Area Chair
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. The new reward ablation and expanded evaluation are useful. However, my rating of 2 reflects my assessment of the paper's core mechanism and scientific contribution. The LTPO comparison shows that existing latent optimization methods can also benefit from the same visual feedback. The nine-setting results further show that the gains from ELA's latent updates are modest and vary across settings. I agree that evidence-guided adaptation has practical value, but I still do not see a sufficiently clear new mechanism or deeper insight into why this approach works better than existing alternatives. My concern is about the significance of the methodological contribution. I therefore maintain my rating of 2.
Response to Reviewer xvGk
We thank the reviewer for clarifying the concern. ELA builds on established latent optimization. Our intended contribution combines an analysis of a limitation in fixed-policy report selection with a method for using clinical evidence to improve candidate generation.
The analysis goes beyond observing that coverage and factuality measure different things. At , only 12 of the 64 recovered findings appear in a candidate free of other unsupported assertions. Correct findings are therefore often recovered within reports that still contain other errors. This identifies a candidate-content limitation alongside the substantial ranking gap measured by the selector–oracle comparison: both the reports available for selection and the selection decision leave room for improvement.
ELA addresses the generation side by converting image-derived condition evidence into a clinical objective that penalizes unsupported assertions and omissions, then using this feedback to update a temporary latent state within the frozen VLM. On IU-XRAY with Qwen2.5-VL-3B, the candidate-pool audit shows:
| Candidate-pool GREEN | Without adaptation | ELA |
|---|---|---|
| Mean candidate | 22.46 | 27.57 |
| Best-in-pool oracle | 59.42 | 63.28 |
These measurements assess candidate quality independently of the final ranking rule. The improvement is observable in the available reports themselves, rather than solely in which report is selected.
The evidence-guided LTPO result shows that this feedback can also be used within another optimizer. The fixed-evidence reward comparison further examines how the clinical objective is formulated. Meanwhile, the nine-setting controls show that the incremental benefit of latent updates is modest and varies across settings. Taken together, the contribution is an empirical diagnosis of the candidate-content bottleneck and an evaluated generation-stage intervention. The results support using clinical evidence to improve generation alongside selection, while also identifying the limits of the current implementation.
Response to Reviewer xvGk: Methodological Contribution and Coverage–Quality Analysis
We thank the reviewer for the careful assessment and the specific comparison suggestions.
W1 & Q1. Novelty relative to LTPO and direct conditioning
A1. The same visual feedback can guide an existing latent optimizer, as the requested LTPO comparison confirms. We therefore position ELA within instance-level latent optimization. Its contribution is a report-level clinical objective that distinguishes unsupported assertions from omissions, used to adapt generation before evidence-based selection. ELA updates complete-report log-probabilities with group-relative rewards and a KL anchor; LTPO estimates updates through Gaussian probes.
On the same 296 IU-XRAY studies, LTPO with image-evidence feedback reaches 46.02 GREEN / 60.77 RadFact-F1, compared with 38.28 / 49.60 for confidence-based LTPO and 40.61 / 57.05 for ELA without final selection. The evidence-guided LTPO variant also scales the learning rate by 30 and assigns degenerate reports reward zero. This comparison demonstrates that the feedback transfers to LTPO; it does not establish that ELA is the only optimizer that can use it.
ELA updates improve the default-prompt control, while the stronger LTPO prompt produces a no-update baseline that ELA updates do not improve. These are comparisons of the implemented procedures: ELA uses two early-layer latent vectors, whereas the LTPO-style variant uses eight input latent tokens. The LTPO+evidence transfer above also scales the learning rate by 30 and assigns degenerate reports reward zero (Appendix D.5).
For direct conditioning, the two tested greedy prompts list either the 13 condition probabilities or the supported findings. They give 3.28 / 11.65 and 14.45 / 28.82, respectively, versus 24.54 / 36.45 for the original greedy prompt. Neither encoding improves generation. We have clarified the contribution relative to prior latent optimization in Section 2 and reported the feedback, prompt, and update comparisons in Section 5.3 and Appendix D.5 (Tables A11-A12).
W2 & Q2. Coverage, adaptation versus selection, and reward controls
A2. In the reported analysis, adaptation helps most when the initial report contains multiple assertions absent from the reference. This establishes a benefit over the fixed selector, not strict superiority over a stronger selector; the oracle gap still leaves substantial room to improve selection. The candidate-content audit explains why improved finding coverage need not yield a clean complete report. At N=32, only 12 of the 64 recovered findings have a candidate free of other unsupported assertions. A whole-report selector cannot combine correct findings scattered across different erroneous reports.
The study-level results below group the 296 IU-XRAY studies by assertions in the initial greedy report that are absent from the reference. Each entry reports GREEN / RadFact-F1:
The group with at least two such assertions benefits most, while reports with none can regress. These are retrospective groups defined using reference reports. The corresponding image-evidence-based analysis across nine settings is reported in Appendix D.6.
The reviewer is also correct that the original reward ablation changed the evidence source. Holding CheXagent-only evidence fixed gives the following comparison, with the direction regularizer switched off or on. Entries are GREEN / RadFact-F1:
The asymmetric reward improves both metrics with either regularizer setting. We have expanded the candidate-content and study-level analyses in Sections 3 and 5.3, and replaced Fig. 4(b) with this fixed-evidence reward comparison. Details are in Appendix D.6 (Tables A13-A14).