acceptodds
Under review as a conference paper at ICLR 2027

Think Again and Look Closer: Dual-Exploration Test-Time Reinforcement Learning for Medical Multimodal Reasoning

Abstract

Medical Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities across healthcare tasks, yet their post-training strategies still heavily depend on substantial annotated data, overlooking the potential of large-scale unlabeled clinical data. An alternative approach, test-time reinforcement learning, operates by sampling multiple trajectories for each test input and exploiting the consensus among their answers as a self-supervised signal. However, for medical visual question answering (VQA), consensus among reasoning answers alone does not constitute reliable evidence. From a textual perspective, over-reliance on answer consensus can amplify confidence in unsubstantiated conclusions and may even encourage shorter, less informative responses that shortcut reasoning in favor of textual agreement. Visually, when multiple sampled trajectories converge on the same answer, their consensus should be grounded in consistent visual evidence rather than textual consistency alone. Clinically, physicians focus intently on subtle abnormalities, whereas even with attention mechanisms, general-purpose models are distracted by large irrelevant areas in the whole image. These limitations call for test-time learning improvements that promote consensus without textual shortcuts, visual grounding consistency across trajectories, and localized exploration beyond holistic features. We therefore introduce TALC, a dual-exploration test-time reinforcement learning framework that asks the model to Think Again and Look Closer. Think Again first anchors on answer support, then ranks reasoning paths by semantic novelty within each support group while also measuring visual consistency across trajectories. Look Closer, in turn, through explicit and executable region-selection actions, assigns each region proposal a tool-token uncertainty score, and branches from that decision point to resample alternative local inspections. Experiments across multiple medical VQA benchmarks and MLLM backbones demonstrate the effectiveness and generalizability of TALC for test-time medical multimodal reasoning. Our code is available at: https://anonymous.4open.science/r/TALC.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Medical Multimodal Large Language Models, Medical Visual Question Answering, Test-Time Reinforcement Learning, Clinical Visual Reasoning, Medical Image Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BXLTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper constructs TALC, a test-time RL framework that consists of two parts, "Look closer" and "Think again". The "Look closer" part generates a few rollouts, exploring some other regions if the model is uncertain. The "Think again" part then assigns reward using criteria like semantic support, trajectory novelty and visual consensus. These rewards are turned into advantages by GRPO. Overall, this paper adopts the idea of TTRL in medical VQA, with domain specific modification such as structured generation interface, clinician-like zooming, and evidence-level consensus. Experiments on four medical VQA benchmarks (SLAKE, VQA-RAD, VQA-Med 2019, MedXpertQA-MM) with Qwen2.5-VL-7B and MedVLM-R1 show consistent gains over TTRL, TTRV, and other test-time baselines.

Critical strengths
  1. The problem is well-motivated. The "answer-level consensus is a fragile pseudo-reward" part is plausible. The label-free use of visual evidence is an important problem in medicine.

  2. This paper is well-written and the experiments are well-executed. The authors included multiple VQA datasets, spanning multiple modalities, and uses two backbones, one general and one medical specific.

  3. The idea about using semantic support as reward in unlabelled setting has some novelty.

  4. I appreciate that the authors provided qualitative cases spanning all datasets in the appendix, and they also released the code for reproduction.

Critical weaknesses

W1. I doubt the motivation of adopting TTRL [1] in medicine. In general tasks like maths, an improvement from 60% to 70% is satisfying. However, in high-stakes tasks such as medicine, TTRL can only marginally improve the performance of the model; it has a low ceiling and will never bring a clinically useless model to the level of useful, and for a clinically useful model (with an accuracy close to human clinicians), we generally would not need TTRL here.

W2. The framework combines many interacting components and hyperparameters (γ\gamma, q0q_0, top-kk, λ\lambda, α\alpha, λv\lambda_v, τc\tau_c, τa\tau_a, βKL\beta_{\mathrm{KL}}, βent\beta_{\mathrm{ent}}), several of which (q0q_0, τa\tau_a, kk) are not reported, yet the ablation only toggles the two top-level modules (Think Again and Look Closer), and only on SLAKE. On closed-ended accuracy, Think Again adds little (+0.5 over TTRL and +1.9 on top of Look Closer with Qwen2.5-VL-7B; +0.5 with MedVLM-R1); its gains concentrate on open-ended token-level metrics, which are sensitive to surface form (see W3). The individual reward terms (semantic support vs. exact-match voting, novelty, visual credit) are never ablated separately. In particular, visual credit only exists when Look Closer is active, and the λv\lambda_v sweep in Fig. 4 does not include λv=0\lambda_v = 0, so the contribution of the "evidence-level consensus" is not established. Most of the accuracy gain comes from Look Closer, which closely follows prior entropy-guided branching at tool calls (MedVR's EVR [2]; ARPO [3]; see W4).

W3. Unclear source of the performance gains.

(a) Depressed base performance under the unified format. Under the grammar-constrained four-field format and CONCLUSION extractor, Qwen2.5-VL-7B obtains 49.8% closed-ended accuracy on VQA-RAD, close to chance for mostly yes/no questions, whereas the Lingshu report [4] gives 64.5% for the same model. Several baselines fall below chance (TTRV-official: 14.2% on five-option MedXpertQA-MM; Intuitor: 24.9% on SLAKE closed-ended), suggesting format or extraction failures. Since TALC assigns −1 to invalid trajectories while baselines discard invalid rollouts, part of the improvement may be learned format compliance. Please report format-valid rates per method and results under each model's native prompting.

(b) Unequal access to visual information. Region requests are enabled only when Look Closer is active, so TALC can zoom in during both adaptation and evaluation while no baseline can. The ablation does not separate tool access from uncertainty-driven branching; the base model with the crop tool, and TTRL with the crop tool and independent rollouts, are needed as baselines.

(c) Metric sensitivity. Open-ended Recall and ROUGE-1 rely on exact token overlap without stemming or semantic matching, so shifts in output style during test-set adaptation can move these metrics substantially. Semantic or LLM-judge evaluation, along with conclusion-length statistics, would help.

W4. Limited novelty and insufficient attribution of prior work. Several components are presented as new, but they closely follow prior work that is either not cited or cited only for other purposes.

(a) Novelty reward. The novelty score ni=1−[ακˉig+(1−α)κimax⁡]n_i = 1 - [\alpha \bar{\kappa}^g_i + (1 - \alpha)\kappa^{\max}_i] with α=0.5\alpha = 0.5, the within-group min-max normalization, the non-overlapping reward bands [0.5,1][0.5, 1] / [−1,−0.5][-1, -0.5] with −1-1 for invalid outputs, and the use of Qwen3-Embedding-4B on the reasoning text all match EVOL-RL [5]. EVOL-RL is cited only for the echo-chamber and response-shortening observations, not for these design choices.

(b) Uncertainty-driven branching. The branching rule qt=clip(q0+γ(Ht−H0),0,1)q_t = \mathrm{clip}(q_0 + \gamma (H_t - H_0), 0, 1) closely follows MedVR's Entropy-guided Visual Regrounding [2], which branches at tool-coordinate generation with P=Pbase+γΔHtoolP = P_{\mathrm{base}} + \gamma \Delta H_{\mathrm{tool}}, as well as the entropy-based adaptive rollout of ARPO [3], which is not cited.

(c) Visual credit. The consensus mask built by pixel-wise majority voting over selected trajectories, with IoU-tiered rewards of 1 / 0.5, closely follows MedVR's Consensus-based Credit Assignment [2]. The main difference is that voters are selected by semantic support rather than answer correctness. MedVR is cited only as an example of region inspection. Label-free spatial-consensus rewards for test-time RL also appear in GUI-RCPO [6], which is not cited. Taken together, the main new element appears to be selecting voters by semantic support instead of correctness, which makes MedVR-style consensus credit usable without labels. The authors should state this contribution clearly, attribute the remaining components explicitly in Sec. 3 (e.g., with a comparison table), and include EVOL-RL as a baseline, since it is the most direct label-free counterpart of Think Again's textual reward.

References

[1] Y. Zuo, K. Zhang, L. Sheng, et al. TTRL: Test-Time Reinforcement Learning. arXiv:2504.16084, 2025.

[2] Z. Jiang, H. Guo, C. Fang, et al. MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning. ICLR 2026. arXiv:2604.08203.

[3] G. Dong, H. Mao, K. Ma, et al. Agentic Reinforced Policy Optimization. arXiv:2507.19849, 2025.

[4] W. Xu, H. P. Chan, L. Li, et al. Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning. arXiv:2506.07044, 2025.

[5] Y. Zhou, Z. Liang, H. Liu, et al. Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation. arXiv:2509.15194, 2025.

[6] Y. Du, Y. Yan, F. Tang, et al. Test-Time Reinforcement Learning for GUI Grounding via Region Consistency. AAAI 2026. arXiv:2508.05615.

[7] G. Xu, P. Jin, Z. Wu, et al. LLaVA-CoT: Let Vision Language Models Reason Step-by-Step. ICCV 2025. arXiv:2411.10440.

Questions for authors

See the weakness part. I will reconsider my rating based on the authors' response.

14 comments

We sincerely appreciate the reviewer’s careful and rigorous examination of our work. We are encouraged by your recognition of our “well-motivated” research problem, “well-written” presentation, “well-executed” experiments, and the novelty of semantic support in unlabeled settings. We also greatly value your appreciation of our qualitative analyses and code release. Importantly, your meticulous examination of methodological novelty, prior-work attribution, and reproducibility has provided valuable guidance for more precisely articulating our contributions and strengthening the scientific rigor of our paper. We have carefully addressed your concerns as follows:

  • Regarding motivation (W1): We clarify TALC’s role in label-free adaptation under distribution shifts and provide cross-dataset transfer evidence.
  • Regarding experimental validation (W2/W3): We add fine-grained component ablations, matched-tool comparisons, and independent semantic correctness evaluations, while specifying previously omitted hyperparameters.
  • Regarding novelty and attribution (W4): We clarify semantic support-guided visual consensus as our key methodological distinction, explicitly acknowledge the foundations provided by prior work, and introduce EVOL-RL comparisons across four benchmarks.
AuthorsOpenReview ↗

Motivation for Test-Time Reinforcement Learning in Medicine

We appreciate the reviewer's concern regarding the clinical relevance of test-time reinforcement learning (W1). Medical MLLMs frequently encounter distribution shifts across imaging modalities, acquisition protocols, and clinical environments, where obtaining expert annotations for model adaptation can be costly. Label-free test-time adaptation offers a practical research direction for improving model robustness under such changing distributions.

Our experimental results support this motivation. On Qwen2.5-VL-7B-Instruct, TALC improves accuracy over the Base Model by 13.28 and 25.10 percentage points on SLAKE and VQA-RAD, respectively. We further conduct cross-dataset generalization experiments using only unlabeled SLAKE inputs for adaptation. The adapted models are then frozen and evaluated on PathVQA [1], OphthalVQA [2], and PMC-VQA [3] without any target-domain adaptation.

Cross-dataset OOD generalization on Qwen2.5-VL-7B-Instruct.

MethodPathVQA Acc. ↑Recall ↑ROUGE-1 ↑OphthalVQA Recall ↑ROUGE-1 ↑PMC-VQA Acc. ↑
Base Model65.116.255.475.417.8553.50
TTRL66.666.896.136.178.4954.55
TTRV (official)62.676.425.185.647.3649.15
TTRV (valid rollouts)64.346.685.795.938.1451.35
Intuitor61.846.075.965.288.3249.85
ECHO66.186.745.886.028.2753.85
TALC (Ours)68.447.636.927.149.4856.15

TALC achieves the best results across all six reported metrics. Compared with the Base Model, it improves PathVQA accuracy by 3.33 percentage points and PMC-VQA accuracy by 2.65 points, while increasing OphthalVQA Recall by 1.73 points. These improvements on unseen datasets provide evidence that the benefits of label-free adaptation extend beyond the distribution used for adaptation.

We recognize that the performance achieved by the general-purpose MLLMs in our experiments remains insufficient for clinical deployment. Nevertheless, we envision TALC as a complementary adaptation strategy for medically specialized models, helping them address domain shifts encountered in real-world deployment. By leveraging unlabeled inputs to refine visual exploration and evidence-grounded reasoning, our approach provides a potential direction for preserving and better utilizing the capabilities of pretrained medical models under changing clinical conditions.

References

[1] He X, Cai Z, Wei W, et al. Towards visual question answering on pathology images[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 2021: 708-718.

[2] Xu P, Chen X, Zhao Z, et al. Unveiling the clinical incapabilities: a benchmarking study of GPT-4V (ision) for ophthalmic multimodal image analysis[J]. British Journal of Ophthalmology, 2024, 108(10): 1384-1389.

[3] Zhang X, Wu C, Zhao Z, et al. Development of a large-scale medical visual question-answering dataset[J]. Communications Medicine, 2024, 4(1): 277.

AuthorsOpenReview ↗

Fine-grained Ablation and Hyperparameter Analysis

To clarify the contributions of TALC's individual components (W2), we extend our ablation experiments to SLAKE and VQA-RAD, independently examining semantic support, reasoning novelty, and visual credit. Specifically, we replace semantic support with exact-match voting, remove the novelty reward, and set λv=0\lambda_v=0 to isolate Visual Credit, while keeping the remaining components and experimental settings unchanged.

Fine-grained ablation of trajectory-level reward components.

MethodSupportNoveltyVisual CreditSLAKE Acc. ↑Recall ↑ROUGE-1 ↑VQA-RAD Acc. ↑Recall ↑ROUGE-1 ↑
TALC w/ Exact SupportExact Match✓✓70.2235.4234.8973.3150.1446.27
TALC w/o NoveltySemantic✗✓70.4531.8633.4173.7147.6344.82
TALC w/o Visual CreditSemantic✓✗69.9836.7435.6372.5151.2847.19
Full TALC (Ours)Semantic✓✓71.4139.0837.7074.9053.7049.38

The fine-grained results reveal complementary contributions from all three reward components. Compared with exact-match support, semantic support improves accuracy by 1.19 and 1.59 percentage points on SLAKE and VQA-RAD, respectively. Reasoning novelty further contributes 0.96 and 1.19 points, while Visual Credit provides gains of 1.43 and 2.39 points. Importantly, these improvements are observed in closed-ended accuracy as well as Recall and ROUGE-1, indicating that Think Again's reward design contributes beyond changes in response wording.

The interaction between Think Again and Look Closer further supports their complementary roles. On Qwen2.5-VL-7B-Instruct, adding Think Again to Look Closer improves SLAKE accuracy from 69.50% to 71.41%, Recall from 23.42% to 39.08%, and ROUGE-1 from 32.18% to 37.70%. Look Closer facilitates localized evidence acquisition, while Think Again improves how the resulting trajectories are evaluated and reinforced. The fine-grained ablations further show that semantic support, reasoning novelty, and visual credit each contribute to the combined framework's performance.

To examine whether improvements in open-ended metrics reflect semantic correctness, we additionally conduct independent LLM-based evaluations. Compared with TTRL, Full TALC improves LLM-judged accuracy by 13.59, 12.00, and 6.43 percentage points on SLAKE, VQA-RAD, and VQA-Med, respectively. These results provide additional evidence that the improvements of the combined framework extend beyond token-level overlap. The complete evaluation protocol and results are discussed in our response to W3.

Regarding hyperparameter reproducibility, we clarify the previously unspecified settings: q0=0.5q_0=0.5, top-k=500k=500 for entropy estimation, and τa=0.5\tau_a=0.5. We will include these values in the revised manuscript. Additionally, our λv=0\lambda_v=0 ablation complements the sensitivity analysis in Figure 4 by directly evaluating the contribution of Visual Credit.

Reviewer_SFHHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that consensus-based test-time RL methods such as TTRL and TTRV are unreliable for medical VQA, for three reasons: majority voting creates an echo chamber that favors short answers, matching answers say nothing about whether the trajectories looked at the same image evidence, and the model is distracted by irrelevant regions. TALC adds two components. Think Again groups trajectories by answer support, ranks them by semantic novelty within each group and scores visual consistency across trajectories. Look Closer uses an executable region-selection tool, scores each region proposal by tool-token uncertainty and branches from uncertain decision points. The method is evaluated on four medical VQA benchmarks with two backbones (Qwen2.5-VL-7B and MedVLM-R1 (Qwen2-VL-2B)).

Critical strengths
  • S1. Sharp diagnosis of consensus-based TTRL. The three limitations (textual, visual, clinical) are clearly separated. The observation that TTRV's shared entropy bonus is cancelled by GRPO group normalization, leaving relative advantages unchanged, is concrete and checkable.
  • S2. Ranking by novelty within support groups keeps diversity without breaking consensus. Branching at uncertain tool tokens concentrates resampling on region selection. Fig. 3(a) gives some evidence that tool-token entropy tracks localization quality.
  • S3. Large gains on SLAKE and VQA-RAD. With Qwen2.5-VL-7B, closed-question accuracy beats TTRL by +10.5 on SLAKE and +16.3 on VQA-RAD. Fig. 3(b) shows TALC avoids the entropy collapse seen in TTRL.
  • S4. Broad, current baseline set. The comparison includes EM-INF, TTRL, two TTRV variants, Intuitor and ECHO, and the ablation covers both backbones.
Critical weaknesses
  • W1. The gains are confounded by the visual tool and output format. Table 3 shows that Look Closer contributes most of the accuracy gain on SLAKE, yet competing methods do not use the same crop tool or structured response format. A compute-matched TTRL baseline with identical tools and formatting is needed to isolate TALC’s algorithmic contribution.

  • W2. The generalizability claim is not consistently supported. With MedVLM-R1, TALC underperforms Intuitor on all VQA-Med metrics and falls below both the base model and TTRL on MedXpertQA-MM. The improvements therefore appear concentrated on SLAKE and VQA-RAD rather than consistently transferring across datasets and backbones.

  • W3. Statistical evidence and baseline reliability are insufficient. The paper reports no multi-seed results or confidence intervals. Moreover, several baselines perform substantially worse than the base model, raising concerns about implementation, formatting or hyperparameter fairness.

  • W4. Evaluation on newer and stronger backbones is missing. The experiments are limited to Qwen2.5-VL-7B and MedVLM-R1, leaving it unclear whether TALC remains effective with Qwen3-level or newer MLLMs that may provide stronger visual grounding, tool use and reasoning. Such backbones could either reduce TALC’s gains by correcting existing weaknesses or amplify them through better region selection.

Questions for authors
  1. Could you report TTRL with the same crop tool and output format but no uncertainty branching, under a matched rollout budget?
  2. Could you explain why Intuitor and TTRV (official) fall far below the base model?
  3. Could you evaluate open questions with exact match or an LLM judge, to separate answer quality from length effects?
  4. Could you ablate the visual-consistency term on its own, and quantify the ''closed loop'', for example how the Think Again reward shifts the distribution of selected regions over training?
9 comments

We sincerely thank the reviewer for the thoughtful, professional, and insightful assessment of our work. We particularly appreciate your recognition of our in-depth analysis of the limitations of consensus-based TTRL, the substantial performance gains on SLAKE and VQA-RAD, and the broad and representative baseline comparisons. We are also grateful for your positive evaluation of our novelty-aware ranking within support groups and uncertainty-guided region exploration. We have carefully addressed each of your concerns as follows:

  • Regarding comparison fairness (W1/Q1): We introduce TTRL + Crop with identical visual tools, output formats, and rollout budgets to ensure a fair comparison with TALC.
  • Regarding generalization (W2/W4): We extend our evaluation to two additional backbones, Qwen3-VL-8B-Instruct and Hulu-Med-7B, across four benchmarks.
  • Regarding further validation (W3/Q2–Q4): We analyze and clarify the performance of baseline methods and provide additional semantic correctness evaluations, reward-component ablations, and quantitative ROI measurements.
AuthorsOpenReview ↗

Fair Comparison with TTRL Using the Same Tools and Output Format

We appreciate the reviewer’s concern that TALC’s improvements may be confounded by visual tool access and structured output formatting (W1 and Q1). Importantly, the TTRL baseline in our main paper already uses the same structured output format as TALC. To further ensure a fair comparison, we additionally equip TTRL with the same Crop Tool, resulting in TTRL + Crop. This baseline retains TTRL’s original majority-voting reward mechanism and performs region inspections without uncertainty-guided branching. Both methods use the same rollout budget.

Comparison with matched visual tools, output formats, and rollout budgets on Qwen2.5-VL-7B-Instruct.

MethodCrop ToolStructured OutputUncertainty BranchingRolloutsSLAKE Acc. (%) ↑VQA-RAD Acc. (%) ↑
TTRL + Crop✓✓✗466.1568.13
Full TALC (Ours)✓✓✓471.4174.90

Under these controlled settings, TALC outperforms TTRL + Crop by 5.26 and 6.77 percentage points on SLAKE and VQA-RAD, respectively. These results show that TALC’s improvements extend beyond shared visual tool access and output formatting, supporting the combined benefits of uncertainty-guided exploration and trajectory-level reward design.

AuthorsOpenReview ↗

Generalization Across Datasets and Backbones

We appreciate the reviewer’s concerns regarding TALC’s generalizability across datasets and backbone models (W2 and W4). To further examine the generalizability of TALC, we extend our evaluation to two additional backbones, including the newer Qwen3-VL-8B-Instruct and the medical model Hulu-Med-7B [1].

Evaluation on additional backbone models. We evaluate both backbones on four medical VQA benchmarks, including SLAKE, VQA-RAD, VQA-Med, and MedXpertQA-MM. The complete results are reported below.

Performance comparison on four medical VQA datasets using Qwen3-VL-8B-Instruct.

MethodSLAKE Acc. ↑Recall ↑ROUGE-1 ↑VQA-RAD Acc. ↑Recall ↑ROUGE-1 ↑VQA-Med Acc. ↑Recall ↑ROUGE-1 ↑MedXpertQA-MM Acc. ↑
Base Model75.3649.9846.8872.5132.8733.7779.6930.3830.6819.00
TTRL77.3951.8448.5374.9034.6235.8181.2531.4231.6919.45
TTRV76.4450.6147.2973.7133.1834.1976.5628.4928.9316.70
TALC (Ours)80.8657.1953.8679.6840.2741.5482.8134.1634.5220.85

Performance comparison on four medical VQA datasets using Hulu-Med-7B.

MethodSLAKE Acc. ↑Recall ↑ROUGE-1 ↑VQA-RAD Acc. ↑Recall ↑ROUGE-1 ↑VQA-Med Acc. ↑Recall ↑ROUGE-1 ↑MedXpertQA-MM Acc. ↑
Base Model74.7655.3250.5471.3139.3840.0767.1930.4330.4626.65
TTRL76.0856.3951.4772.9140.3141.0970.3131.6831.7427.45
TTRV75.2455.7450.8270.5238.9439.6262.5028.5728.6422.55
TALC (Ours)78.2359.4154.6875.7043.5944.3273.4433.2733.4827.70

TALC achieves the best performance across all ten reported metrics on both additional backbones, outperforming TTRL across all four datasets. Notably, TALC also achieves the best results on VQA-Med and MedXpertQA-MM, where its performance on MedVLM-R1 was less consistent. These findings indicate that the limitations observed on MedVLM-R1 do not consistently extend to other backbones, providing broader evidence of TALC’s effectiveness across medical VQA benchmarks and model architectures.

References

[1] Jiang S, Wang Y, Song S, et al. Hulu-med: A transparent generalist model towards holistic medical vision-language understanding[J]. arXiv preprint arXiv:2510.08668, 2025.

Reviewer_LA71OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TALC, a label-free test-time reinforcement learning framework for medical multimodal question answering. The method combines two components: (1) Look Closer, which uses uncertainty-guided branching to explore alternative image regions, and (2) Think Again, which constructs trajectory-level rewards based on semantic answer support, reasoning novelty, and cross-trajectory visual consistency. The resulting rewards are used to optimize the model through GRPO. Experiments on four medical VQA benchmarks and two backbone models demonstrate improvements over several test-time adaptation baselines.

Critical strengths
  1. The paper identifies meaningful limitations of consensus-based test-time RL, particularly its reliance on answer agreement without verifying visual evidence. Combining uncertainty-guided region exploration with trajectory-level reward shaping is a potentially useful direction.

  2. The method achieves substantial improvements over several baselines on medical VQA benchmarks. The ablation study also suggests that Look Closer and Think Again provide complementary benefits, with the former contributing more strongly to accuracy and the latter improving open-ended answer recall.

Critical weaknesses
  1. Insufficient evidence of generalization beyond the adapted test distribution. The method directly optimizes model parameters using the unlabeled test inputs and evaluates on the corresponding benchmark test split. While this is consistent with the transductive test-time RL setting, the reported improvements may partly reflect adaptation to benchmark-specific input distributions rather than generalizable improvements in medical reasoning. The paper does not provide held-out evaluation, cross-dataset transfer, or explicit OOD experiments. This is particularly concerning because adaptation to distribution shifts across clinical institutions and imaging devices is an important motivation of the work.

  2. The proposed rewards are not directly validated against trajectory correctness. Semantic support measures agreement rather than correctness, reasoning novelty measures diversity rather than reasoning quality, and visual consistency measures overlap between selected regions rather than whether those regions contain clinically relevant evidence. Consequently, multiple trajectories sharing the same incorrect conclusion or localization bias may still receive high rewards. The paper lacks direct evaluation of reasoning correctness, grounding accuracy, and the correlation between the proposed rewards and actual trajectory quality. Therefore, it remains unclear whether TALC improves grounded medical reasoning or merely reinforces more consistent model-generated behaviors.

Questions for authors
  1. Since TALC performs adaptation directly on the test set, can the authors evaluate its generalization on held-out samples or OOD datasets without further adaptation?

  2. How well do the proposed trajectory rewards correlate with actual answer correctness and reasoning quality?

  3. Can the authors evaluate whether the selected image regions contain clinically relevant evidence, rather than merely measuring spatial consistency across trajectories?

  4. Can the authors provide more fine-grained ablations of semantic support, reasoning novelty, and visual consistency to clarify their individual contributions?

  5. How does TALC handle cases where the majority of trajectories share an incorrect conclusion or localization bias? Can it effectively identify and reinforce correct minority trajectories?

11 comments

We sincerely thank the reviewer for the thoughtful and insightful assessment of our work, particularly for recognizing TALC’s “substantial improvements”, the “complementary benefits” of Look Closer and Think Again, and the significance of addressing limitations in consensus-based test-time RL. We greatly appreciate your perceptive and constructive suggestions, which have been invaluable in strengthening the experimental rigor and validation of our work.

  • Regarding generalization (W1/Q1): We add cross-dataset OOD evaluations without further target-domain adaptation.
  • For reward reliability and grounding (W2/Q2–Q4): We provide answer correctness evaluations, semantic support correlation analysis, clinical ROI measurements, and fine-grained reward ablations.
  • Regarding incorrect consensus (Q5): We clarify how exploration and novelty preserve trajectory diversity.
AuthorsOpenReview ↗

Cross-Dataset OOD Generalization

We thank the reviewer for raising this important concern. Regarding W1 and Q1, the reviewer questions whether TALC's improvements primarily reflect adaptation to benchmark-specific test distributions rather than transferable medical reasoning capabilities, and asks for evaluation on held-out or OOD datasets without further adaptation. To directly address this concern, we conduct an additional cross-dataset OOD experiment using Qwen2.5-VL-7B-Instruct.

(1) Experimental protocol. For each test-time adaptation method, we adapt Qwen2.5-VL-7B-Instruct using unlabeled SLAKE test inputs, then freeze the adapted checkpoint and evaluate it directly on PathVQA [1], OphthalVQA [2], and PMC-VQA [3]. All methods follow the same cross-dataset evaluation protocol, with the Base Model evaluated without adaptation.

(2) Cross-dataset OOD generalization result on Qwen2.5-VL-7B-Instruct.

MethodPathVQA Acc. ↑PathVQA Recall ↑PathVQA ROUGE-1 ↑OphthalVQA Recall ↑OphthalVQA ROUGE-1 ↑PMC-VQA Acc. ↑
Base Model65.116.255.475.417.8553.50
TTRL66.666.896.136.178.4954.55
TTRV (official)62.676.425.185.647.3649.15
TTRV (valid rollouts)64.346.685.795.938.1451.35
Intuitor61.846.075.965.288.3249.85
ECHO66.186.745.886.028.2753.85
TALC (Ours)68.447.636.927.149.4856.15

(3) Analysis. TALC achieves the best performance across all six reported metrics, consistently outperforming the Base Model on PathVQA, OphthalVQA, and PMC-VQA. Specifically, TALC improves PathVQA accuracy from 65.11% to 68.44%, OphthalVQA Recall from 5.41 to 7.14, and PMC-VQA accuracy from 53.50% to 56.15%. These results demonstrate that the improvements obtained through adaptation on SLAKE can transfer to diverse medical VQA datasets, including pathology, ophthalmology, and biomedical literature-based question answering.

References

[1] He X, Cai Z, Wei W, et al. Towards visual question answering on pathology images[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 2021: 708-718.

[2] Xu P, Chen X, Zhao Z, et al. Unveiling the clinical incapabilities: a benchmarking study of GPT-4V (ision) for ophthalmic multimodal image analysis[J]. British Journal of Ophthalmology, 2024, 108(10): 1384-1389.

[3] Zhang X, Wu C, Zhao Z, et al. Development of a large-scale medical visual question-answering dataset[J]. Communications Medicine, 2024, 4(1): 277.

AuthorsOpenReview ↗

Empirical Validation of Trajectory-level Rewards and Clinical Visual Grounding

We thank the reviewer for examining TALC’s reward design and visual grounding (W2 and Q2–3). We address three questions: (1) whether TALC improves final answer correctness, (2) whether semantic support correlates with trajectory correctness, and (3) whether selected regions align with clinically relevant ROIs. We investigate the first two through independent answer evaluation and trajectory-level correlation analysis on open-ended medical VQA, and the third through ROI localization evaluation. Reasoning novelty aims to promote trajectory diversity rather than measure correctness, with its contribution assessed separately through component ablation.

(1) Independent evaluation of final answer correctness. We evaluate model predictions on SLAKE, VQA-RAD, and VQA-Med using Exact Match (EM) and independent LLM-based semantic evaluation. EM measures exact agreement between normalized predictions and reference answers. For semantic evaluation, GPT-6.1 Sol (official OpenAI API) serves as an independent judge, assessing the medical correctness of each predicted CONCLUSION given the question and acceptable reference answers. We additionally report Inspect Eval results following the open-ended semantic evaluation protocol implemented in Inspect Evals for VQA-RAD (https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/vqa_rad).

Independent evaluation of medical VQA answer correctness.

DatasetMethodEM (%) ↑LLM Judge (%) ↑Inspect Eval ↑
SLAKEBase Model0.2449.6853.18
TTRL0.0053.1056.68
TALC (Ours)0.3266.6969.40
VQA-RADBase Model0.0038.5042.50
TTRL0.5041.5044.50
TALC (Ours)0.5053.5057.50
VQA-MedBase Model4.8252.7555.50
TTRL5.2855.7358.72
TALC (Ours)6.1962.1665.14

TALC achieves the best or tied-best performance across all reported metrics on all three datasets. These results demonstrate that TALC improves the medical semantic correctness of final answers rather than merely promoting agreement among trajectories.

(2) Correlation between semantic support and answer correctness. To examine whether semantic support reflects trajectory correctness, we use the same GPT-6.1 Sol evaluator to independently assess the CONCLUSION of each sampled trajectory against the reference answers. We then calculate Spearman correlation and AUROC to measure the association between semantic support and answer correctness. Additionally, we compare the accuracy of high-support and low-support trajectories, defined as those with semantic support scores at or above the 75th percentile and at or below the 25th percentile of each dataset, respectively.

Correlation between semantic support and trajectory correctness.

DatasetSpearman ↑AUROC ↑High-support Acc. (%) ↑Low-support Acc. (%) ↑
SLAKE0.46830.785485.1929.74
VQA-RAD0.41760.752881.4627.63
VQA-Med0.34900.701777.3829.05

Semantic support exhibits a positive correlation with answer correctness across all three datasets, with AUROC values consistently above 0.70. Moreover, high-support trajectories achieve substantially higher accuracy than low-support trajectories across all datasets. These findings suggest that semantic support is a useful indicator of answer correctness.

(3) Evaluation of clinical visual grounding. We use the official SLAKE bounding-box annotations to evaluate the clinical relevance of selected regions. We report ROI Precision, measuring the fraction of inspected area within annotated ROIs, ROI IoU, measuring spatial overlap, and ROI Hit Rate, defined as the percentage of questions with over 50% ROI coverage.

To evaluate the contribution of visual consistency, we compare Full TALC with TALC without the visual reward (λv=0\lambda_v=0), keeping the other components and rollout budget unchanged. TTRL + Crop serve as reference methods.

Clinical ROI localization results on SLAKE.

MethodROI Precision (%) ↑ROI IoU (%) ↑ROI Hit Rate (%) ↑
TTRL + Crop16.2013.9576.03
TALC w/o Visual Reward21.4718.6282.95
Full TALC (Ours)41.8338.6493.41

Full TALC achieves the best results across all three localization metrics. Compared with TALC without the visual reward, it improves ROI Precision from 21.47% to 41.83%, ROI IoU from 18.62% to 38.64%, and ROI Hit Rate from 82.95% to 93.41%. These results demonstrate better overlap between inspected regions and annotated clinical ROIs, providing evidence that visual consistency contributes to clinically relevant region selection beyond agreement among trajectories alone.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Revision Summary

We are grateful to each of the three reviewers for the care they took in reading our paper and for their thoughtful suggestions.

We especially appreciate that the reviewers recognized our contribution to addressing the limitations of test-time reinforcement learning based on textual consensus in Medical VQA, acknowledged the value of our two modules, Look Closer and Think Again, and affirmed the potential of the semantic support mechanism for unlabeled medical reasoning tasks. Their questions and comments further helped us clarify our methodology, strengthen the support for our findings, and improve the overall presentation of the manuscript.

In response to the reviewers' comments, we conducted additional experiments and incorporated further analyses. The revisions made to address the concerns raised by all three reviewers are summarized as follows:

  • Generalization Across Datasets and Backbone Models (reviewers LA71, SFHH, BXLT). We extended our evaluation to two additional backbones, Qwen3-VL-8B-Instruct and Hulu-Med-7B, across four medical VQA benchmarks. TALC achieves higher accuracy than TTRL in 15 out of 16 backbone–dataset comparisons. We also introduced cross-dataset OOD evaluations, where TALC achieves the best performance across all six metrics without target-domain adaptation.

  • Reward Validation and Clinical Visual Grounding (reviewers LA71, SFHH, BXLT). We added independent LLM-based semantic evaluations, Inspect Eval, and trajectory-level correlation analyses. Furthermore, additional ROI evaluations based on expert annotations in SLAKE demonstrate that Visual Credit substantially improves clinical region localization, increasing ROI Precision from 21.47% to 41.83%.

  • Experimental Robustness (reviewers SFHH, BXLT). We introduced a TTRL + Crop baseline with identical visual tools, output formats, and rollout budgets. TALC achieves accuracy improvements of 5.26 and 6.77 percentage points on SLAKE and VQA-RAD, respectively. We also reported format-valid rates for all methods and conducted additional semantic evaluations on closed-ended questions to further explain baseline performance and distinguish answer correctness from output-format effects.

  • Fine-grained Ablations and Methodological Contributions (reviewers LA71, SFHH, BXLT). We conducted independent ablations of Semantic Support, Reasoning Novelty, and Visual Credit, demonstrating their complementary contributions. We further clarified TALC's methodological distinctions from existing approaches.

  • Advantages and Scope of Test-Time RL (reviewer BXLT). We further highlight that test-time RL enables direct policy optimization at test time using unlabeled target-domain data and self-supervised rewards, without requiring costly and time-consuming physician annotations. Future work will focus on validating TALC in real-world clinical settings.

All the points mentioned above have been thoroughly addressed and incorporated into the revised manuscript.

We have addressed all the reviewers' concerns individually. We would also welcome further discussion if the reviewers have any points that need further clarification.

We sincerely thank all reviewers and the Area Chair for their constructive feedback, careful consideration, and time throughout the review process. We are truly grateful for their recognition and support of our work.

AuthorsOpenReview ↗

A Response Summary to the Area Chairs and Reviewers: Additional Results, Analyses, and Clarifications

Dear Area Chairs and Reviewers,

We sincerely thank the Area Chairs for handling our manuscript and all reviewers for the time and effort they have devoted to the review process, as well as for their insightful and constructive comments. We are very glad that the reviewers recognized the importance of the problem we study, the novelty of our proposed method, and the effectiveness of the TALC framework.

At present, two of the three reviewers have given a score of 3 (Weak Accept). In addition, Reviewer LA71 acknowledged that our additional experiments addressed the concerns raised earlier and explicitly expressed willingness to raise the score:

“If you can include a quick evaluation or verification on VinDr-CXR, I would be happy to raise my score.”

In response to this suggestion, we conducted additional experiments on the VinDr-CXR-VQA dataset and provided further explanations and analyses in response to the reviewer’s follow-up questions. We also hope to hear further from Reviewer LA71!

The main improvements we have made are summarized as follows:

  • Generalization ability: We extended the experiments to two additional backbones and conducted a cross-dataset OOD generalization evaluation. Across 16 backbone–dataset comparisons, TALC achieved higher accuracy than TTRL in 15 of them.

  • More thorough evaluation: We added comparisons with TTRL + Crop using the same visual tools, independent LLM-based semantic evaluation, format validity analysis, and comparisons with more baseline methods.

  • Clinical visual grounding: We added ROI localization evaluation on SLAKE and further conducted experiments on VinDr-CXR-VQA, using radiologist-annotated lesion bounding boxes to validate the model’s clinical visual grounding ability.

  • Methodological contributions: We conducted fine-grained ablations on Semantic Support, Reasoning Novelty, and Visual Credit, and further clarified the differences between our method and existing work.

By addressing the reviewers’ concerns through additional experiments and in-depth analysis, we believe that TALC has been significantly improved in terms of research quality, experimental rigor, and completeness.

Thank you again for your time, assistance, and consideration!

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.