- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TTRSD, a test-time reinforcement learning framework designed to improve the visual reasoning capabilities of vision-language models (VLMs) without relying on ground-truth rewards or external teacher models. The proposed framework consists of two main components: (1) Multi-View Answer-Level Self-Distillation (MVSD), which aggregates predictions from multiple visual transformations to construct answer-level pseudo-rewards, and (2) Visual Contrastive Token Selection (VCTS), which identifies visually sensitive tokens by comparing their log-probabilities under original and image-ablated inputs. Experiments on three VLM backbones and seven visual reasoning benchmarks demonstrate consistent improvements over several test-time reinforcement learning baselines. Additional ablation studies and cross-dataset evaluations are provided to support the effectiveness of the proposed approach.
Critical strengths
-
Relevant and meaningful research problem. The paper addresses the challenge of test-time reinforcement learning for VLMs without ground-truth rewards or external verifiers. Constructing reliable supervision signals and improving credit assignment in multimodal reasoning are important research directions.
-
Intuitive and reasonably motivated framework. The combination of multi-view answer-level self-distillation and visually guided token selection provides an intuitive approach to improving pseudo-reward construction and restricting policy-gradient updates to visually relevant positions.
-
Promising empirical improvements. The proposed method demonstrates improvements across three VLM backbones and seven benchmarks, outperforming the reported TTRL and TTRV baselines. The paper also includes component ablations, cross-dataset transfer experiments, and multi-seed evaluations, which provide useful empirical evidence.
Critical weaknesses
- Limited methodological novelty and insufficient differentiation from existing approaches.
The proposed method primarily combines several existing ideas, including multi-view consistency learning, self-distillation, contrastive visual dependence estimation, and token-masked GRPO optimization. Although their integration is reasonable, the individual components are conceptually straightforward, and the paper does not sufficiently establish a substantial methodological contribution beyond combining these techniques.
- The entropy-based diversity penalty appears ineffective under group-normalized GRPO.
A significant concern lies in the reward formulation in Eq. (8). For all student trajectories associated with the same input, the teacher entropy term is identical. However, Eq. (9) computes the advantage through group-wise reward normalization. Since the entropy penalty is constant within each group, it cancels out during mean subtraction and does not affect the reward standard deviation. Consequently, the resulting advantage is mathematically identical to that obtained using only the teacher-support reward. Therefore, under the optimization formulation presented in the paper, the entropy penalty does not appear to influence the policy-gradient update, contrary to the claimed diversity-control effect.
- Visual sensitivity does not necessarily correspond to meaningful token-level credit assignment.
The proposed VCTS module measures token-level visual sensitivity through the absolute difference between log-probabilities under original and image-ablated inputs. However, high visual sensitivity does not necessarily indicate that a token represents correct visual grounding or that optimizing this token would improve reasoning performance. For example, an incorrectly recognized object or numerical value may still exhibit a large probability difference when the visual input is removed. Consequently, the selected tokens may include visually sensitive but incorrect or misleading information.
Furthermore, all selected tokens within the same trajectory receive the same sequence-level advantage. Thus, the method performs selective gradient allocation rather than explicitly estimating the contribution of individual tokens to task success.
- The reliability of multi-view pseudo-rewards remains insufficiently investigated.
The proposed MVSD module assumes that aggregating predictions across multiple visual transformations produces more reliable supervision than single-view majority voting. However, all teacher predictions originate from the same underlying model and may share systematic perceptual or reasoning errors. Moreover, image cropping and downsampling can remove task-relevant information, particularly in mathematical diagrams, dense charts, or fine-grained visual reasoning tasks. As a result, consistency across transformed views does not necessarily imply answer correctness. Although the paper reports improvements in pseudo-label accuracy, the relationship between teacher answer support and actual answer correctness is not sufficiently analyzed.
Questions for authors
-
Entropy penalty: Since the teacher entropy penalty is constant across trajectories within the same group, how does it affect the group-normalized advantage in Eq. (9)? Can the authors provide an ablation without this term and clarify its actual contribution?
-
Token selection validity: Can the authors provide quantitative evidence that tokens selected by VCTS correspond to correct visual grounding or genuinely beneficial optimization targets, rather than merely being sensitive to image perturbations?
-
Pseudo-reward reliability: How frequently does multi-view consensus produce incorrect pseudo-labels, and how does the proposed method behave when the teacher distribution assigns high support to an incorrect answer?