- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
This paper constructs TALC, a test-time RL framework that consists of two parts, "Look closer" and "Think again". The "Look closer" part generates a few rollouts, exploring some other regions if the model is uncertain. The "Think again" part then assigns reward using criteria like semantic support, trajectory novelty and visual consensus. These rewards are turned into advantages by GRPO. Overall, this paper adopts the idea of TTRL in medical VQA, with domain specific modification such as structured generation interface, clinician-like zooming, and evidence-level consensus. Experiments on four medical VQA benchmarks (SLAKE, VQA-RAD, VQA-Med 2019, MedXpertQA-MM) with Qwen2.5-VL-7B and MedVLM-R1 show consistent gains over TTRL, TTRV, and other test-time baselines.
Critical strengths
-
The problem is well-motivated. The "answer-level consensus is a fragile pseudo-reward" part is plausible. The label-free use of visual evidence is an important problem in medicine.
-
This paper is well-written and the experiments are well-executed. The authors included multiple VQA datasets, spanning multiple modalities, and uses two backbones, one general and one medical specific.
-
The idea about using semantic support as reward in unlabelled setting has some novelty.
-
I appreciate that the authors provided qualitative cases spanning all datasets in the appendix, and they also released the code for reproduction.
Critical weaknesses
W1. I doubt the motivation of adopting TTRL [1] in medicine. In general tasks like maths, an improvement from 60% to 70% is satisfying. However, in high-stakes tasks such as medicine, TTRL can only marginally improve the performance of the model; it has a low ceiling and will never bring a clinically useless model to the level of useful, and for a clinically useful model (with an accuracy close to human clinicians), we generally would not need TTRL here.
W2. The framework combines many interacting components and hyperparameters (, , top-, , , , , , , ), several of which (, , ) are not reported, yet the ablation only toggles the two top-level modules (Think Again and Look Closer), and only on SLAKE. On closed-ended accuracy, Think Again adds little (+0.5 over TTRL and +1.9 on top of Look Closer with Qwen2.5-VL-7B; +0.5 with MedVLM-R1); its gains concentrate on open-ended token-level metrics, which are sensitive to surface form (see W3). The individual reward terms (semantic support vs. exact-match voting, novelty, visual credit) are never ablated separately. In particular, visual credit only exists when Look Closer is active, and the sweep in Fig. 4 does not include , so the contribution of the "evidence-level consensus" is not established. Most of the accuracy gain comes from Look Closer, which closely follows prior entropy-guided branching at tool calls (MedVR's EVR [2]; ARPO [3]; see W4).
W3. Unclear source of the performance gains.
(a) Depressed base performance under the unified format. Under the grammar-constrained four-field format and CONCLUSION extractor, Qwen2.5-VL-7B obtains 49.8% closed-ended accuracy on VQA-RAD, close to chance for mostly yes/no questions, whereas the Lingshu report [4] gives 64.5% for the same model. Several baselines fall below chance (TTRV-official: 14.2% on five-option MedXpertQA-MM; Intuitor: 24.9% on SLAKE closed-ended), suggesting format or extraction failures. Since TALC assigns −1 to invalid trajectories while baselines discard invalid rollouts, part of the improvement may be learned format compliance. Please report format-valid rates per method and results under each model's native prompting.
(b) Unequal access to visual information. Region requests are enabled only when Look Closer is active, so TALC can zoom in during both adaptation and evaluation while no baseline can. The ablation does not separate tool access from uncertainty-driven branching; the base model with the crop tool, and TTRL with the crop tool and independent rollouts, are needed as baselines.
(c) Metric sensitivity. Open-ended Recall and ROUGE-1 rely on exact token overlap without stemming or semantic matching, so shifts in output style during test-set adaptation can move these metrics substantially. Semantic or LLM-judge evaluation, along with conclusion-length statistics, would help.
W4. Limited novelty and insufficient attribution of prior work. Several components are presented as new, but they closely follow prior work that is either not cited or cited only for other purposes.
(a) Novelty reward. The novelty score with , the within-group min-max normalization, the non-overlapping reward bands / with for invalid outputs, and the use of Qwen3-Embedding-4B on the reasoning text all match EVOL-RL [5]. EVOL-RL is cited only for the echo-chamber and response-shortening observations, not for these design choices.
(b) Uncertainty-driven branching. The branching rule closely follows MedVR's Entropy-guided Visual Regrounding [2], which branches at tool-coordinate generation with , as well as the entropy-based adaptive rollout of ARPO [3], which is not cited.
(c) Visual credit. The consensus mask built by pixel-wise majority voting over selected trajectories, with IoU-tiered rewards of 1 / 0.5, closely follows MedVR's Consensus-based Credit Assignment [2]. The main difference is that voters are selected by semantic support rather than answer correctness. MedVR is cited only as an example of region inspection. Label-free spatial-consensus rewards for test-time RL also appear in GUI-RCPO [6], which is not cited. Taken together, the main new element appears to be selecting voters by semantic support instead of correctness, which makes MedVR-style consensus credit usable without labels. The authors should state this contribution clearly, attribute the remaining components explicitly in Sec. 3 (e.g., with a comparison table), and include EVOL-RL as a baseline, since it is the most direct label-free counterpart of Think Again's textual reward.
References
[1] Y. Zuo, K. Zhang, L. Sheng, et al. TTRL: Test-Time Reinforcement Learning. arXiv:2504.16084, 2025.
[2] Z. Jiang, H. Guo, C. Fang, et al. MedVR: Annotation-Free Medical Visual Reasoning via Agentic Reinforcement Learning. ICLR 2026. arXiv:2604.08203.
[3] G. Dong, H. Mao, K. Ma, et al. Agentic Reinforced Policy Optimization. arXiv:2507.19849, 2025.
[4] W. Xu, H. P. Chan, L. Li, et al. Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning. arXiv:2506.07044, 2025.
[5] Y. Zhou, Z. Liang, H. Liu, et al. Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variation. arXiv:2509.15194, 2025.
[6] Y. Du, Y. Yan, F. Tang, et al. Test-Time Reinforcement Learning for GUI Grounding via Region Consistency. AAAI 2026. arXiv:2508.05615.
[7] G. Xu, P. Jin, Z. Wu, et al. LLaVA-CoT: Let Vision Language Models Reason Step-by-Step. ICCV 2025. arXiv:2411.10440.
Questions for authors
See the weakness part. I will reconsider my rating based on the authors' response.
14 comments
Motivation for Test-Time Reinforcement Learning in Medicine
We appreciate the reviewer's concern regarding the clinical relevance of test-time reinforcement learning (W1). Medical MLLMs frequently encounter distribution shifts across imaging modalities, acquisition protocols, and clinical environments, where obtaining expert annotations for model adaptation can be costly. Label-free test-time adaptation offers a practical research direction for improving model robustness under such changing distributions.
Our experimental results support this motivation. On Qwen2.5-VL-7B-Instruct, TALC improves accuracy over the Base Model by 13.28 and 25.10 percentage points on SLAKE and VQA-RAD, respectively. We further conduct cross-dataset generalization experiments using only unlabeled SLAKE inputs for adaptation. The adapted models are then frozen and evaluated on PathVQA [1], OphthalVQA [2], and PMC-VQA [3] without any target-domain adaptation.
Cross-dataset OOD generalization on Qwen2.5-VL-7B-Instruct.
| Method | PathVQA Acc. ↑ | Recall ↑ | ROUGE-1 ↑ | OphthalVQA Recall ↑ | ROUGE-1 ↑ | PMC-VQA Acc. ↑ |
|---|---|---|---|---|---|---|
| Base Model | 65.11 | 6.25 | 5.47 | 5.41 | 7.85 | 53.50 |
| TTRL | 66.66 | 6.89 | 6.13 | 6.17 | 8.49 | 54.55 |
| TTRV (official) | 62.67 | 6.42 | 5.18 | 5.64 | 7.36 | 49.15 |
| TTRV (valid rollouts) | 64.34 | 6.68 | 5.79 | 5.93 | 8.14 | 51.35 |
| Intuitor | 61.84 | 6.07 | 5.96 | 5.28 | 8.32 | 49.85 |
| ECHO | 66.18 | 6.74 | 5.88 | 6.02 | 8.27 | 53.85 |
| TALC (Ours) | 68.44 | 7.63 | 6.92 | 7.14 | 9.48 | 56.15 |
TALC achieves the best results across all six reported metrics. Compared with the Base Model, it improves PathVQA accuracy by 3.33 percentage points and PMC-VQA accuracy by 2.65 points, while increasing OphthalVQA Recall by 1.73 points. These improvements on unseen datasets provide evidence that the benefits of label-free adaptation extend beyond the distribution used for adaptation.
We recognize that the performance achieved by the general-purpose MLLMs in our experiments remains insufficient for clinical deployment. Nevertheless, we envision TALC as a complementary adaptation strategy for medically specialized models, helping them address domain shifts encountered in real-world deployment. By leveraging unlabeled inputs to refine visual exploration and evidence-grounded reasoning, our approach provides a potential direction for preserving and better utilizing the capabilities of pretrained medical models under changing clinical conditions.
References
[1] He X, Cai Z, Wei W, et al. Towards visual question answering on pathology images[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 2021: 708-718.
[2] Xu P, Chen X, Zhao Z, et al. Unveiling the clinical incapabilities: a benchmarking study of GPT-4V (ision) for ophthalmic multimodal image analysis[J]. British Journal of Ophthalmology, 2024, 108(10): 1384-1389.
[3] Zhang X, Wu C, Zhao Z, et al. Development of a large-scale medical visual question-answering dataset[J]. Communications Medicine, 2024, 4(1): 277.
Fine-grained Ablation and Hyperparameter Analysis
To clarify the contributions of TALC's individual components (W2), we extend our ablation experiments to SLAKE and VQA-RAD, independently examining semantic support, reasoning novelty, and visual credit. Specifically, we replace semantic support with exact-match voting, remove the novelty reward, and set to isolate Visual Credit, while keeping the remaining components and experimental settings unchanged.
Fine-grained ablation of trajectory-level reward components.
| Method | Support | Novelty | Visual Credit | SLAKE Acc. ↑ | Recall ↑ | ROUGE-1 ↑ | VQA-RAD Acc. ↑ | Recall ↑ | ROUGE-1 ↑ |
|---|---|---|---|---|---|---|---|---|---|
| TALC w/ Exact Support | Exact Match | ✓ | ✓ | 70.22 | 35.42 | 34.89 | 73.31 | 50.14 | 46.27 |
| TALC w/o Novelty | Semantic | ✗ | ✓ | 70.45 | 31.86 | 33.41 | 73.71 | 47.63 | 44.82 |
| TALC w/o Visual Credit | Semantic | ✓ | ✗ | 69.98 | 36.74 | 35.63 | 72.51 | 51.28 | 47.19 |
| Full TALC (Ours) | Semantic | ✓ | ✓ | 71.41 | 39.08 | 37.70 | 74.90 | 53.70 | 49.38 |
The fine-grained results reveal complementary contributions from all three reward components. Compared with exact-match support, semantic support improves accuracy by 1.19 and 1.59 percentage points on SLAKE and VQA-RAD, respectively. Reasoning novelty further contributes 0.96 and 1.19 points, while Visual Credit provides gains of 1.43 and 2.39 points. Importantly, these improvements are observed in closed-ended accuracy as well as Recall and ROUGE-1, indicating that Think Again's reward design contributes beyond changes in response wording.
The interaction between Think Again and Look Closer further supports their complementary roles. On Qwen2.5-VL-7B-Instruct, adding Think Again to Look Closer improves SLAKE accuracy from 69.50% to 71.41%, Recall from 23.42% to 39.08%, and ROUGE-1 from 32.18% to 37.70%. Look Closer facilitates localized evidence acquisition, while Think Again improves how the resulting trajectories are evaluated and reinforced. The fine-grained ablations further show that semantic support, reasoning novelty, and visual credit each contribute to the combined framework's performance.
To examine whether improvements in open-ended metrics reflect semantic correctness, we additionally conduct independent LLM-based evaluations. Compared with TTRL, Full TALC improves LLM-judged accuracy by 13.59, 12.00, and 6.43 percentage points on SLAKE, VQA-RAD, and VQA-Med, respectively. These results provide additional evidence that the improvements of the combined framework extend beyond token-level overlap. The complete evaluation protocol and results are discussed in our response to W3.
Regarding hyperparameter reproducibility, we clarify the previously unspecified settings: , top- for entropy estimation, and . We will include these values in the revised manuscript. Additionally, our ablation complements the sensitivity analysis in Figure 4 by directly evaluating the contribution of Visual Credit.
Summarize
We sincerely appreciate the reviewer’s careful and rigorous examination of our work. We are encouraged by your recognition of our “well-motivated” research problem, “well-written” presentation, “well-executed” experiments, and the novelty of semantic support in unlabeled settings. We also greatly value your appreciation of our qualitative analyses and code release. Importantly, your meticulous examination of methodological novelty, prior-work attribution, and reproducibility has provided valuable guidance for more precisely articulating our contributions and strengthening the scientific rigor of our paper. We have carefully addressed your concerns as follows: