acceptodds
Under review as a conference paper at ICLR 2027

PrimeFocus: Harnessing Intrinsic Regularities for Test-Time Policy Optimization in Vision-Language Models

Abstract

Test-time reinforcement learning (RL) adapts vision-language models (VLMs) on unlabeled test data by rewarding agreement with a majority-voted pseudo-label. However this method has two weaknesses. First, the reward depends only on the final textual answer, so a consensus that arises from language priors is reinforced as strongly as one supported by the image. Second, training runs for a fixed number of steps, although the best stopping point differs across test sets and no labeled validation set is available. We propose PrimeFocus, a test-time policy optimization method that adds intrinsic signals of the policy to this recipe. For the reward, consensus-anchored attention sharpening derives three annotation-free terms from the attention of answer tokens to image tokens: the share of attention on the image (visual enhancement), the concentration of that attention (visual focus), and the concentration of the attention map averaged over all responses that give the same answer (group visual consensus). For stopping, we track the cumulative deviation of the policy's mean token entropy from its exponential moving average, a statistic motivated by prequential estimates of epiplexity, and terminate training when it plateaus. On three visual recognition and four visual question answering benchmarks with InternVL3-2B and InternVL2.5-4B, PrimeFocus improves the mean accuracy of TTRV under the same backbone by 2.27and 2.42. Ablations and diagnostics examine each reward term, attention-level failure modes, and the behavior of the stopping rule.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Test-time Reinforcement Learning, Vision-language models, Visual grounding, Attention sharpening, Early stopping

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_XuiNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PrimeFocus, a test-time reinforcement learning framework for vision-language models that addresses two limitations of majority-voting-based adaptation: the lack of visual grounding in pseudo-rewards and the use of fixed training schedules. The method augments TTRV's frequency-based reward with three attention-derived signals—visual enhancement, visual focus, and group visual consensus—and introduces a label-free early stopping criterion based on the stabilization of token-entropy deviations from an exponential moving average. Experiments on seven visual recognition and VQA benchmarks with two InternVL backbones report average accuracy improvements of 2.27 and 2.42 percentage points over TTRV. The paper also provides component ablations, attention diagnostics, and early stopping analyses.

Critical strengths
  • Clear motivation targeting concrete limitations of existing test-time RL methods. The paper identifies two relevant issues in consensus-based test-time adaptation: majority agreement does not necessarily imply visual grounding, and a fixed training schedule may be suboptimal across different datasets. Both problems are practically meaningful, and the proposed components are clearly connected to these motivations.

  • Simple and computationally practical reward design. The attention-derived rewards are computed from the model's own attention statistics without additional annotations or a separately trained reward model. They can be incorporated into the existing GRPO framework, and the reported additional computational overhead is relatively modest (approximately 12% per training step).

  • Consistent performance improvements with useful diagnostic analyses. PrimeFocus improves average accuracy over TTRV on both evaluated backbones, with positive gains in 13 of 14 model–benchmark combinations. The paper goes beyond final accuracy by analyzing individual reward components, attention smearing and collapse, attention–correctness correlations, and early stopping trajectories. These analyses provide some empirical support for the proposed design.

Critical weaknesses
  • Insufficient validation of visual grounding. The proposed rewards rely on attention statistics as proxies for visual grounding, but higher or more concentrated visual attention does not necessarily indicate reliance on relevant visual evidence. Moreover, the reported correlations between visual attention and correctness vary across tasks. The current experiments mainly establish correlations and changes in attention patterns, without directly demonstrating that PrimeFocus reduces visually unsupported predictions or corrects erroneous majority consensus.

  • The group visual consensus reward does not explicitly isolate inter-response agreement. Equation (7) measures the entropy of the averaged attention map, which combines individual attention concentration with inter-response disagreement. Consequently, its effectiveness may partly arise from additional attention sharpening rather than genuine visual consensus. Furthermore, when all responses share the same answer, the group-level reward becomes constant and cancels during GRPO advantage normalization. These limitations weaken the claimed interpretation and effectiveness of the proposed consensus mechanism.

  • Incomplete experimental validation and insufficient comparisons. The experiments do not adequately isolate the contributions of different reward components and early stopping, and comparisons with closely related VLM test-time RL methods are limited. The early stopping analysis covers only a few single-seed trajectories without comparisons against simpler alternatives, while generalization beyond InternVL backbones remains unexplored. Additionally, the reported adaptation samples, epochs, and training steps require clarification, raising concerns about reproducibility and fair comparison.

Questions for authors
  • Can the authors directly verify that the proposed attention-based rewards improve visual grounding rather than merely altering attention distributions? For example, could image perturbation or visual ablation experiments demonstrate reduced reliance on language priors and improved correction of incorrect majority predictions?

  • How much of the group visual consensus improvement comes from genuine inter-response agreement rather than individual attention concentration? Could the authors disentangle these effects through separate reward ablations and clarify how the mechanism handles confidently incorrect or unanimous answer clusters?

  • How does the proposed early stopping criterion compare with simpler label-free alternatives, such as entropy stabilization, reward plateaus, or policy KL divergence?

  • Could the authors provide more comprehensive and controlled evaluations, including comparisons with closely related methods such as TTIQ, complete reward-component ablations, and experiments on additional VLM families? Additionally, could they clarify the relationship between the reported adaptation samples, epochs, and training steps to ensure reproducibility and fair comparisons?

Reviewer_JvcQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PrimeFocus extends TTRV-style test-time RL for VLMs, i.e., GRPO with a majority-vote reward on unlabeled test samples. It adds two components:

  • 3 annotation-free rewards computed from final-layer answer-to-image attention: (1) the share of attention on image tokens; (2) the attention entropy over image tokens; (3) the entropy of the averaged attention map within each answer cluster.
  • A label-free stopping rule that halts training when the cumulative deviation of mean token entropy from its EMA plateaus. Under TTRV's protocol, with InternVL3-2B and InternVL2.5-4B on seven benchmarks, it raises mean accuracy over TTRV's published numbers by 2.27 and 2.42 points. Ablations, attention diagnostics and early-stopping analyses support the results.
Critical strengths
  1. The signals reuse GRPO's log-prob pass (about 12% overhead). The group visual consensus term scores an answer cluster by whether its members attend to shared image regions. To my knowledge this is new, and it is complementary to text-side vote corrections. Gains are consistent (13 of 14 settings), and the ablation identifies group consensus as the main driver.
  2. The paper tracks explicit reward-hacking modes and reports cases where the attention signal is uninformative or negative (MathVista, MathVerse). It also clearly scopes the stopping statistic as an entropy-stabilization signal, not an epiplexity estimator.
Critical weaknesses
  1. The TTRV numbers are copied from the TTRV paper, not re-run. The early-stopping gain in Table 2 is therefore confounded with setup differences.
  2. Several per-benchmark gains are below the reported seed std (1.58).
  3. Early stopping is tested with a single seed on three datasets and is not compared with simple label-free rules.
  4. The stated budget (20 samples, 8 prompts per step, 3 epochs) gives about 6 steps, too few for the rule to fire (earliest step 10).But runs of 30–120 steps are reported.
Questions for authors
  1. Which implementation and hyperparameters produced Tables 1–6? Please release the actual launch script, preprocessing and prompts. If the released code was used, please re-run with the described formulation or revise Section 3.
  2. Please report TTRV re-run in your own setup (same hardware, batch size and seeds) for all 14 settings, with standard deviations.
  3. What is the actual training budget per dataset? What is the stopping step for each run, and the value of L_t when training stops? How does the rule compare with simple label-free stopping rules across several seeds?

Comments on the paper

3 comments

Thank you for the rebuttal and additional experiments. I appreciate the authors' efforts, particularly the new analyses of visual grounding and group consensus. However, concerns remain regarding the limited early stopping evaluation, insufficient comparisons with related methods, and unclear training settings. While the rebuttal addresses some of my concerns, I will maintain my original rating.

AuthorsOpenReview ↗

Overall response to the reviewers

Thank you for your constructive feedback. In this rebuttal, we address the main concerns raised across the reviews through additional controlled experiments and analyses. We also clarify the training and evaluation protocol and explain what the new results support.

  • Matched baselines and training budget. We reran TTRV and PrimeFocus across all 14 model–benchmark settings with three paired seeds and the same six-update budget. Early stopping was disabled in these runs. PrimeFocus improves mean accuracy by 2.19 pp on InternVL3-2B and 2.53 pp on InternVL2.5-4B, with positive point estimates in 13 of 14 settings.
  • Early stopping. We compared the EMA-residual rule with raw-entropy, mean-reward, and policy-KL plateau rules by replaying them offline on the same extended trajectories. On RealWorldQA, EMA selects checkpoints 52 updates before the cap on average and improves accuracy by 2.88 pp over the cap. The separately trained six-update schedule scores higher on AI2D and RealWorldQA than the EMA-selected checkpoints.
  • Visual grounding and group consensus. On the annotated AI2D cohort, the evidence-versus-control masking contrast is 8.96 pp larger for PrimeFocus than for TTRV. In the 50–80% initial vote-share group, PrimeFocus corrects 26/97 initial errors versus 9/97 for TTRV. Both methods correct 0/24 unanimous errors. We also separate individual-map concentration from inter-response disagreement in the group reward and report additional ablations, related-method comparisons, and results on Qwen2.5-VL.

We appreciate the time you have spent reviewing our work. Do these results and clarifications address your concerns? We are looking forward to your responses.

Dear reviewers, we would appreciate a brief follow-up on whether our rebuttal addresses your main concerns and whether there are any points we can clarify further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.