acceptodds
Under review as a conference paper at ICLR 2027

TTRSD: Test-Time Reinforcement Learning with Self-Distillation for Vision-Language Models

10 views

Abstract

Test-time reinforcement learning enables vision-language models (VLMs) to adapt using unlabeled inputs. However, repeated sampling under fixed visual condi- tions can reinforce shared perceptual errors, while sequence-level rewards fail to isolate visual perception—the foundational bottleneck that anchors multimodal reasoning—risking the degradation of pre-trained reasoning capabilities. We pro- pose TTRSD, a test-time reinforcement learning framework combining multi-view answer-level self-distillation with visual contrastive token selection. A shared policy aggregates teacher predictions across original, cropped, and downsampled views into an answer distribution. Student trajectories generated from the orig- inal image receive rewards based on the support for their final answers in this distribution. To allocate this feedback precisely toward perceptual bottlenecks, we compare the log-probabilities of the same sampled tokens under original and visually ablated inputs while holding their textual prefixes fixed, selecting visually sensitive positions for policy-gradient updates. TTRSD separates update direction, determined by group-relative advantages, from update position, determined by visual sensitivity, without requiring ground-truth labels, external verifiers, or a separate teacher. With only 20 unlabeled adaptation samples, TTRSD improves performance across seven benchmarks and three VLMs, raising InternVL3-2B’s MMMU accuracy from 35.79% to 49.32% (+13.53%), demonstrating cross-dataset generalization while preserving inherent reasoning integrity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
TTRL, MLLM, OPD

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_D1hfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TTRSD, a test-time reinforcement learning framework designed to improve the visual reasoning capabilities of vision-language models (VLMs) without relying on ground-truth rewards or external teacher models. The proposed framework consists of two main components: (1) Multi-View Answer-Level Self-Distillation (MVSD), which aggregates predictions from multiple visual transformations to construct answer-level pseudo-rewards, and (2) Visual Contrastive Token Selection (VCTS), which identifies visually sensitive tokens by comparing their log-probabilities under original and image-ablated inputs. Experiments on three VLM backbones and seven visual reasoning benchmarks demonstrate consistent improvements over several test-time reinforcement learning baselines. Additional ablation studies and cross-dataset evaluations are provided to support the effectiveness of the proposed approach.

Critical strengths
  1. Relevant and meaningful research problem. The paper addresses the challenge of test-time reinforcement learning for VLMs without ground-truth rewards or external verifiers. Constructing reliable supervision signals and improving credit assignment in multimodal reasoning are important research directions.

  2. Intuitive and reasonably motivated framework. The combination of multi-view answer-level self-distillation and visually guided token selection provides an intuitive approach to improving pseudo-reward construction and restricting policy-gradient updates to visually relevant positions.

  3. Promising empirical improvements. The proposed method demonstrates improvements across three VLM backbones and seven benchmarks, outperforming the reported TTRL and TTRV baselines. The paper also includes component ablations, cross-dataset transfer experiments, and multi-seed evaluations, which provide useful empirical evidence.

Critical weaknesses
  1. Limited methodological novelty and insufficient differentiation from existing approaches.

The proposed method primarily combines several existing ideas, including multi-view consistency learning, self-distillation, contrastive visual dependence estimation, and token-masked GRPO optimization. Although their integration is reasonable, the individual components are conceptually straightforward, and the paper does not sufficiently establish a substantial methodological contribution beyond combining these techniques.

  1. The entropy-based diversity penalty appears ineffective under group-normalized GRPO.

A significant concern lies in the reward formulation in Eq. (8). For all student trajectories associated with the same input, the teacher entropy term is identical. However, Eq. (9) computes the advantage through group-wise reward normalization. Since the entropy penalty is constant within each group, it cancels out during mean subtraction and does not affect the reward standard deviation. Consequently, the resulting advantage is mathematically identical to that obtained using only the teacher-support reward. Therefore, under the optimization formulation presented in the paper, the entropy penalty does not appear to influence the policy-gradient update, contrary to the claimed diversity-control effect.

  1. Visual sensitivity does not necessarily correspond to meaningful token-level credit assignment.

The proposed VCTS module measures token-level visual sensitivity through the absolute difference between log-probabilities under original and image-ablated inputs. However, high visual sensitivity does not necessarily indicate that a token represents correct visual grounding or that optimizing this token would improve reasoning performance. For example, an incorrectly recognized object or numerical value may still exhibit a large probability difference when the visual input is removed. Consequently, the selected tokens may include visually sensitive but incorrect or misleading information.

Furthermore, all selected tokens within the same trajectory receive the same sequence-level advantage. Thus, the method performs selective gradient allocation rather than explicitly estimating the contribution of individual tokens to task success.

  1. The reliability of multi-view pseudo-rewards remains insufficiently investigated.

The proposed MVSD module assumes that aggregating predictions across multiple visual transformations produces more reliable supervision than single-view majority voting. However, all teacher predictions originate from the same underlying model and may share systematic perceptual or reasoning errors. Moreover, image cropping and downsampling can remove task-relevant information, particularly in mathematical diagrams, dense charts, or fine-grained visual reasoning tasks. As a result, consistency across transformed views does not necessarily imply answer correctness. Although the paper reports improvements in pseudo-label accuracy, the relationship between teacher answer support and actual answer correctness is not sufficiently analyzed.

Questions for authors
  1. Entropy penalty: Since the teacher entropy penalty is constant across trajectories within the same group, how does it affect the group-normalized advantage in Eq. (9)? Can the authors provide an ablation without this term and clarify its actual contribution?

  2. Token selection validity: Can the authors provide quantitative evidence that tokens selected by VCTS correspond to correct visual grounding or genuinely beneficial optimization targets, rather than merely being sensitive to image perturbations?

  3. Pseudo-reward reliability: How frequently does multi-view consensus produce incorrect pseudo-labels, and how does the proposed method behave when the teacher distribution assigns high support to an incorrect answer?

Reviewer_LsgyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Problem

This paper studies test-time reinforcement learning (TTRL) for VLMs on a unlabeled adaptation set (#n=20). When applied to VLMs, naive TTRL fails in two ways: (1) biased direction: sampling repeatedly from a static image reinforces correlated visual misperceptions; (2) misplaced updates: broadcasting noisy sequence-level rewards across all tokens.

Method

The proposed method, TTRSD, fixes these issues by decoupling update direction from update location. It estimates the reward direction via soft voting across three image views (original, 0.85 center-crop, 0.70 resize), and restricts GRPO gradient updates to the top 20% of tokens whose log-probabilities change the most when the image is replaced with a blank frame.

Results

Across 3 VLMs and 7 benchmarks, adapting on 20 unlabeled questions yields solid gains (+4.1% to +10.0% on average) and transfers across datasets.

Critical strengths

Good empirical insight. 

The multi-view voting is not new. It's a standard test-time augmentation.

What makes the paper compelling is the empirical insight: in low-resource, label-free RL, updating text reasoning tokens with noisy pseudo-rewards is actively harmful (as seen in TTRL crashing InternVL3-8B by nearly 10 points). Freezing 80% of the trajectory and applying the test-time bias strictly to the ~20% visual interface tokens is a simple but effective combination.

Cheap token attribution. 

Computing post-rollout requires just one gradient-free, teacher-forced forward pass with a blank image, adding <1% wall-clock time (94ms vs. 13.6s for rollout generation). This make it very practical.

Critical weaknesses

4x rollout budget mismatch. 

Section 4.1 claims all methods share the same rollout budget (16 rollouts per input). But Appendix I shows TTRSD samples 16 student rollouts plus 3*16 teacher rollouts = 64 rollouts for each question. Since rollout generation is 96.5% of step time, TTRSD uses 4x the compute of a 16-rollout baseline.

Confounding view diversity with sample size (Fig. 4). 

Cropping 15% off borders in geometry/chart tasks clips axis labels and vertices. Consequently, both crop (32.1%) and resize (30.9%) yield worse pseudo-labels than original (33.9%) on LogicVista. Combining all three reaches 36.2%, but aggregates 48 rollouts vs. 16 for original.

Without an original (48 rollouts) baseline, we cannot tell if this gain comes from multi-view diversity or simply 3x more samples.

Unexplained baseline (Table 1). 

Zero-shot InternVL3-8B is worse than 2B on MathVista (56.40% vs. 58.47%), and TTRL drives InternVL3-8B from 32.42% down to 8.51% on LogicVista. This looks like format collapse or parser failure rather than reasoning degradation.

Questions for authors
  1. Compute-matched baselines: Did TTRL/TTRV use 16 or 64 rollouts in Table 1? Please report results under an identical total rollout budget (e.g., baselines at 64 rollouts, or TTRSD at 16 total rollouts).
  2. 48-rollout single-view baseline: In Figure 4, what is the pseudo-label accuracy if you sample 48 rollouts purely from the Original view?
  3. InternVL3-8B collapse: Why does TTRL drop to 8.51% on LogicVista for InternVL3-8B (e.g., did it stop following the The answer is X format)?
  4. Soft weights vs. hard cutoff: Why use a sensitive top-20% step function instead of normalizing Δt\Delta_t as continuous weights on token advantages?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.