acceptodds
Under review as a conference paper at ICLR 2027

Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning

20 views

Abstract

Test-time reinforcement learning can adapt vision–language models (VLMs) to unlabeled target data, but relies on reliable self-generated supervision. We analyze consensus-based VLM test-time reinforcement learning across diverse VQA datasets and model sizes and identify two limitations. First, much of its improvement comes from answer normalization rather than content correction. Second, many erroneous responses fail to jointly use the image and the question, allowing consensus rewards to reinforce grounding errors. We propose TTIQ, a test-time reinforcement learning framework that derives supervision from image–question dependence. For each sampled response, TTIQ measures how token likelihoods change under controlled image and question ablations, aggregates the resulting dependence into overall and sustained response-level scores for reward construction, and uses the token-level signals for credit assignment. The reward favors responses that are jointly supported by both inputs and sufficiently confident, while token-level credit emphasizes grounded parts of the response. This shifts supervision from output popularity toward grounded and reliable responses. Across eight VQA datasets and multiple model scales, TTIQ achieves the highest average performance at every evaluated scale, generalizes across VLM families, and transfers to unseen datasets after adaptation on a single source dataset.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Test-Time Reinforcement Learning, Vision-Language Models, Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_w5KUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies consensus-based test-time RL for VLMs (TTRV) on free-form VQA and finds two problems. Most exact-match gains are answer-format fixes: 72.3% of corrections are form-only, and exact match rises 3.36 points while ground-truth containment falls 0.41. In addition, 69% of incorrect responses don't depend on the image, the question, or both. The proposed method TTIQ, measures how much a sampled response's token likelihoods drop when the image is zeroed or the question is masked. Its reward is the within-group rank of min(image dependence, question dependence, confidence cap). Token-level credit goes mostly to tokens supported by both inputs, and training uses GRPO on 20 unlabeled examples per task. On eight VQA tasks and three InternVL backbones, TTIQ has the best average exact match at each scale: 58.77 / 62.41 / 66.94, against 57.48 / 61.36 / 66.67 for the best of TTRL, SCOPE and TTRV. The paper also reports ablations, cross-dataset transfer and results on Qwen2.5-VL.

Critical strengths
  1. Splitting exact-match gains into form-only and content corrections exposes a measurement artifact that probably affects other TTRL-style work. It also gives the community a simple check that can be applied to any label-free method.
  2. TTIQ costs two extra teacher-forced passes (about 1.6× time per update, same memory) and needs only 20 unlabeled examples. It beats TTRV in all 24 backbone–task cells, and code is released.
Critical weaknesses
  1. The claim that TTIQ corrects content rather than answer form is not well supported. e.g. 68.4% of TTIQ's own corrections are form-only, versus 72.3% for TTRV (Fig. 4). Ground-truth containment, the metric used against TTRV, is never reported for TTIQ. Gains concentrate where Base exact match is limited by format. On InternVL2.5-4B, COCO-QA-N goes from 36.0 to 74.7, and training only on TextVQA already reaches 72.4 (Table 3). This points to a task-agnostic format change, not better grounding. In Table 2, an image-only reward scores 57.66. That is above the full reward with the same credit (56.40) and close to min(I, Q) with joint-token credit (57.79), so the "joint" design adds little.

  2. The evaluation protocol likely inflates small margins. \delta is chosen per task by test accuracy, while the baselines are not tuned. As a result, TTIQ beats the \delta = 0 and \delta = 1 rows of Table 2 by construction. With the best fixed \delta (read from Fig. 5a), confidence adds only about 0.3 points. No seeds or variance are reported. Margins over the strongest baseline are +1.29, +1.05 and +0.27 at 2B, 4B and 8B; the 4B margin is +0.54 without COCO-QA-N. On Qwen2.5-VL-3B, TTIQ is only +0.24 over Base.

Questions for authors
  1. Was \delta selected per task on the evaluation split? Which run produced the 4B TextVQA result of 79.52, given that Fig. 5a peaks at about 79.1?
  2. Were the tables produced with the released code, which uses a sustained weight of 0.1 and a token-credit floor of 0.2?
  3. In Fig. 2, what threshold defines "lacking dependence," and what is the same breakdown for correct responses?
4 comments
AuthorsOpenReview ↗

Response to Reviewer w5KU (1/3)

We appreciate the reviewer’s detailed and constructive comments, which have prompted further analysis and improvements to the paper. We address the concerns below with additional evidence and explanations, and would be happy to provide further clarification or discuss any remaining questions.

W1: Content Corrections, Cross-Dataset Transfer, and Joint Dependence.

The claim that TTIQ corrects content rather than answer form is not well supported. e.g. 68.4% of TTIQ's own corrections are form-only, versus 72.3% for TTRV (Fig. 4). Ground-truth containment, the metric used against TTRV, is never reported for TTIQ. Gains concentrate where Base exact match is limited by format. On InternVL2.5-4B, COCO-QA-N goes from 36.0 to 74.7, and training only on TextVQA already reaches 72.4 (Table 3). This points to a task-agnostic format change, not better grounding. In Table 2, an image-only reward scores 57.66. That is above the full reward with the same credit (56.40) and close to min(I, Q) with joint-token credit (57.79), so the "joint" design adds little.

Thank you for your careful review and constructive feedback. Below, we address your concerns regarding 1) content corrections, 2) cross-dataset transfer, and 3) the joint design below.

1. TTIQ corrects substantially more content errors. We would like to clarify that correcting answer form also contributes to accuracy, particularly on datasets where answer-form mismatches account for many evaluation errors. Both form-only and content-changing corrections contribute to improved accuracy. Our goal is to correct more errors overall, especially content errors that existing baselines leave unresolved, while preserving the benefits of answer-form refinement.

Figure 4 reports percentages relative to each method’s own total corrections, which can obscure the difference in absolute content corrections. The table below reports content-changing correction counts for each dataset, summed across the three model backbones, together with TTIQ’s relative increase over TTRV. In total, TTIQ produces 2,105 content-changing corrections, compared with 1,144 for TTRV, an increase of 84.0%, with gains on all eight datasets.

DatasetTTRV content correctionsTTIQ content correctionsRelative increase
InfoVQA101147+45.5%
TextVQA100148+48.0%
GQA-Relation173371+114.5%
GQA-Attribute124211+70.2%
ST-VQA368734+99.5%
MathVerse-VD1522+46.7%
COCO-QA-N139189+36.0%
CLEVR-H124283+128.2%
Overall1,1442,105+84.0%

These results show that, alongside answer-form improvements, TTIQ delivers substantially more content corrections than TTRV.

2. TTIQ reinforces input use that transfers across tasks. We want to emphasize that TTIQ encourages a general ability to answer using both the image and the question. This ability is useful across datasets with different tasks and answer requirements, providing a basis for the strong transfer results.

The comparison with TTRV supports this interpretation. On InternVL2.5-4B, direct COCO-QA-N adaptation with TTRV reaches 69.58, but TextVQA adaptation reaches only 33.53, below Base at 36.01. Thus, TTRV’s substantial answer-form improvements do not successfully transfer. TTIQ, however, reaches 72.39 after TextVQA adaptation, close to 74.70 with direct adaptation. Its transfer also extends to GQA-Relation and GQA-Attribute, reaching 30.16 and 55.67, compared with 16.91 and 48.91 for TTRV.

Our transferable improvements can include both content and form corrections because TTIQ encourages a general ability to answer from the inputs. This does not exclude improvements in answer form. Specifically, responses expressing the same correct answer in different forms can receive different rewards because their image–question dependence differs under the given prompt. These reward differences can favor a more appropriate answer form. Furthermore, sustained dependence and joint-token credit favor expressions supported by both inputs, which may also help responses align with the information and answer format requested in the prompt. Thus, TTIQ can transfer improvements in both answer content and form by reinforcing a general ability to answer using the image and question.

3. Joint dependence addresses different task requirements. (Continue on next block)

AuthorsOpenReview ↗

Response to Reviewer w5KU (2/3)

3. Joint dependence addresses different task requirements. Image-only dependence is competitive. However, on some dataset like GQA-Rel., adding question dependence improves the average from 47.96 to 56.40.

We interpret this difference in light of task requirements. Image-only rewards may be particularly effective when salient visual content strongly constrains the answer, as in some OCR questions (such as TextVQA, InfoVQA). However, strong image dependence alone does not ensure that a response addresses the specific question. Question dependence provides an additional signal when the question determines which object, attribute, or relation is relevant, or which reasoning operation is required, as in GQA. Under minimum fusion, adding this signal can downweight responses that use the image but weakly depend on the specific question, substantially changing their ranking and the resulting training preferences.

These findings support the complementary roles of image and question dependence across different task requirements. The complete TTIQ configuration combines these signals with confidence and joint-token credit assignment, achieving the highest average accuracy in Table 2.

W2 & Q1: Evaluation Protocol, Confidence Selection, and Statistical Reliability.

W2: The evaluation protocol likely inflates small margins. \delta is chosen per task by test accuracy, while the baselines are not tuned. As a result, TTIQ beats the \delta = 0 and \delta = 1 rows of Table 2 by construction. With the best fixed \delta (read from Fig. 5a), confidence adds only about 0.3 points. No seeds or variance are reported. Margins over the strongest baseline are +1.29, +1.05 and +0.27 at 2B, 4B and 8B; the 4B margin is +0.54 without COCO-QA-N. On Qwen2.5-VL-3B, TTIQ is only +0.24 over Base. Q1: Was \delta selected per task on the evaluation split? Which run produced the 4B TextVQA result of 79.52, given that Fig. 5a peaks at about 79.1?

Thanks for raising this question. On λ\lambda selection. We'd like to clarify that the gains are not dependent on selecting the best λ\lambda for each task. Across the 24 model–dataset combinations, TTIQ outperforms the most closely related baseline, TTRV, in all 24 cells when reporting the best λ\lambda. More importantly, using the same fixed λ=1\lambda=1 for every cell still yields improvements in 23/24 cells. Even if we take the worst result among all evaluated λ\lambda values for each cell, TTIQ remains better in 20/24 cells. Thus, λ\lambda selection affects the magnitude of the gain but does not determine the overall conclusion. All baselines use the default settings recommended by their papers or official implementations.

On the contribution of confidence. The average gain under a fixed λ\lambda does not fully capture the value of confidence across diverse tasks. Although confidence contributes little on some tasks, it can be useful when model likelihood helps distinguish reliable responses from unreliable ones. For example, Figure 5(a) shows that λ=0.5\lambda=0.5 improves TextVQA by approximately 1.2 points over λ=0\lambda=0. Our goal is to develop a general method that accommodates diverse tasks, where different signals may be useful to different degrees. Therefore, we therefore retain confidence as an adjustable constraint that provides additional guidance when informative.

On the TextVQA discrepancy. The 79.52 TextVQA result and Figure 5(a) used different randomly sampled sets of 20 training examples; we apologize for the unclear reporting. We will refine the figure to make it more consistent.

Here, we additionally show that our method is robust to different training sets. We report results across 3 randomly sampled training sets below. TTIQ achieves the highest mean accuracy, supporting its effectiveness beyond a particular training sample.

ModelMethodGQA-RelGQA-AttrST-VQACOCO-QA (Num)
InternVL3-2BTTRL32.24 ± 0.8452.84 ± 1.6676.25 ± 0.3772.93 ± 0.19
SCOPE32.77 ± 0.8952.79 ± 1.6276.24 ± 0.3572.98 ± 0.39
TTRV27.47 ± 3.5952.50 ± 0.7576.06 ± 0.1074.08 ± 0.42
TTIQ (ours)33.63 ± 1.5353.76 ± 0.6077.08 ± 0.6174.66 ± 0.56
InternVL2.5-4BTTRL35.24 ± 0.5756.74 ± 0.6382.98 ± 1.0572.13 ± 2.91
SCOPE33.73 ± 1.4256.61 ± 1.0882.89 ± 0.8872.82 ± 2.34
TTRV28.52 ± 4.9056.95 ± 1.3081.53 ± 0.3472.22 ± 2.31
TTIQ (ours)36.37 ± 0.7057.69 ± 0.5583.13 ± 0.6274.64 ± 0.21

On training seeds and variance. We have also added two random seeds and now report mean and standard deviation across three seeds for every method and have updated our Table 1 in the manuscript. We found that variability is comparable across methods, and the three-seed averages retain the improvements, with better performance compared to baselines in several settings.

AuthorsOpenReview ↗

Response to Reviewer w5KU (3/3)

Q2: Implementation Settings and Sensitivity.

Were the tables produced with the released code, which uses a sustained weight of 0.1 and a token-credit floor of 0.2?

Thank you for your careful checking. Yes, all main-table TTIQ results used the released settings: α=0.1\alpha=0.1 for sustained dependence and τ=0.2\tau=0.2 for the token-credit floor. These values were set initially and kept fixed across all datasets without tuning. We are sorry for missing this detail and we have clarified it in the implementation section in the updated manuscript.

Specifically, the sustained term supplements overall dependence by measuring whether support persists across response tokens. The floor τ=0.2\tau=0.2 preserves positive credit for low-support tokens and limits the maximum token-weight ratio to 5:15{:}1. Subsequent mean-one normalization preserves this ratio.

Additional sensitivity checks show little variation across nearby settings:

Floor τ\tauCLEVR-H, 4BTextVQA, 8B
0.178.6784.34
0.278.8184.48
0.378.7784.49

Varying α\alpha from 0.10.1 to 0.30.3 changes pairwise reward-ranking accuracy by at most 0.05 percentage points in our sensitivity analysis.

Q3: Dependence Threshold and Correct-Response Breakdown.

In Fig. 2, what threshold defines "lacking dependence," and what is the same breakdown for correct responses?

Thank you for asking for this clarification. The threshold is 0. We applied the same criterion to correct responses using InternVL2.5-4B on eight datasets, with 200 questions per dataset and four pre-update rollouts per question:

Response subsetQQ onlyII onlyNeitherI+QI+QLacking at least one
Incorrect (N=3,351N=3{,}351)1,212 (36.2%)363 (10.8%)737 (22.0%)1,039 (31.0%)2,312 (69.0%)
Correct (N=3,049N=3{,}049)494 (16.2%)260 (8.5%)97 (3.2%)2,198 (72.1%)851 (27.9%)

Joint dependence is present in 72.1% of correct responses, compared with 31.0% of incorrect responses. This contrast supports its usefulness as a proxy for correctness during label-free test-time training, while showing that it is not a guarantee of correctness.

AuthorsOpenReview ↗

A gentle follow-up on our rebuttal

Dear reviewer w5KU,

We sincerely thank you for your time and effort in reviewing our paper, as well as your thoughtful feedback and constructive suggestions throughout the discussion.

In our rebuttal, we (1) provided a dataset-level analysis of corrections, showing that TTIQ produces 84.0% more content-changing corrections than TTRV, and explained how TTIQ encourages better use of both the image and the question, a general VQA ability that supports cross-dataset transfer; (2) addressed concerns regarding confidence weight by providing additional results showing that the overall gains remain robust without selection, and reported results across randomly sampled training sets and additional training seeds; (3) clarified the implementation settings and provided additional sensitivity analyses; and (4) clarified the dependence threshold in Figure 2 and provided the same breakdown for correct responses, showing that 72.1% exhibit joint image–question dependence, compared with 31.0% of incorrect responses.

With less than 13 hours remaining in the discussion period, we would greatly appreciate hearing whether our responses have adequately addressed your concerns. If any questions or concerns remain, we would be happy to provide further clarifications and continue the discussion before the deadline. Thank you again for your valuable feedback, time, and consideration, which have been greatly appreciated and helpful in improving our work.

Reviewer_iHZKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that consensus-based test-time reinforcement learning for vision-language models improves answer form more than answer content, and that a reward based on agreement can reinforce responses which do not really use the image and the question together; it supports both claims with analyses of corrections and of sampled responses. The method re-scores each sampled response under an ablated image and an ablated question, turns the resulting likelihood changes into image- and question-dependence scores, combines them with a confidence term into a response reward, and uses the same signals to redistribute credit across the tokens of each response during the GRPO update. Experiments cover several VQA datasets and backbones, with transfer, ablation and efficiency studies.

Critical strengths
  • The opening observation is easy to appreciate even without following the method: the paper shows that the usual improvement in this line of work is largely a change in how the answer is written rather than in what the model understands.
  • The extra cost is small, because the additional computation reuses answers that were already generated instead of generating new ones.
  • The experimental scope is wide: many datasets, several model sizes and families, and further studies on transfer and on individual design choices.
Critical weaknesses
  • Section 3.3 cannot be reproduced. The text says the per-token scores are turned into weights and normalized, but gives no formula for that step, and it does not say over which tokens the percentile ranks are computed.
  • I cannot tell how much of the improvement is a change of answer form rather than of answer content. The paper raises exactly this worry about its baseline, but does not report the same breakdown for its own method, dataset by dataset.
  • Table 2 does not support the component claims. Several rows are worse than the base model, which looks like training instability rather than a clean ablation of one component at a time, and the confidence term helps in one row while costing a great deal in another, with no explanation.
Questions for authors

See the weaknesses above

3 comments
AuthorsOpenReview ↗

Response to Reviewer iHZK (1/2)

We thank the reviewer for the careful review and helpful suggestions. We have added experiments and clarified key details in response to your comments. We respond to each point below and would be happy to provide further clarification or discuss any remaining questions.

W1: Clarification of Token-Level Credit Assignment.

Section 3.3 cannot be reproduced. The text says the per-token scores are turned into weights and normalized, but gives no formula for that step, and it does not say over which tokens the percentile ranks are computed.

Thank you for pointing out this omission. We have added the following formulation to Appendix A and mentioned in Section 4.1. We also clarify it here.

Percentile ranks are computed within each response, separately for image and question dependence, over valid generated tokens only. Let Vi\mathcal V_i denote these tokens and Mi=∣Vi∣M_i=|\mathcal V_i|. For Mi=1M_i=1, we set the token weight to wi,t=1w_{i,t}=1; for Mi≥2M_i\geq2, ascending ranks ri,tS∈{0,…,Mi−1}r^S_{i,t}\in\{0,\ldots,M_i-1\} give

\qquad S\in\{I,Q\},$$ with ties broken by token order. The joint support and normalized weights are $$g_{i,t}=\min\{P_i^I(t),P_i^Q(t)\}, \qquad \bar w_{i,t}=\tau+(1-\tau)g_{i,t}, \qquad w_{i,t}= \frac{\bar w_{i,t}} {\frac{1}{M_i}\sum_{u\in\mathcal V_i}\bar w_{i,u}}.$$ Here, $\tau$ sets a positive floor on the unnormalized weights so that low-support tokens retain some positive credit. The normalized weights have mean of one and apply only to positive response advantages, as specified in Eq. (10). We fix $\tau=0.2$ throughout the main experiments. We also conducted additional sensitivity expereiment below show similar performance across $\tau\in\{0.1,0.2,0.3\}$. | Floor $\tau$ | CLEVR-H, 4B | TextVQA, 8B | |---|---:|---:| | 0.1 | 78.67 | 84.34 | | 0.2 | 78.81 | 84.48 | | 0.3 | 78.77 | 84.49 | ### W2: Dataset-Level Breakdown of Form and Content Corrections. >I cannot tell how much of the improvement is a change of answer form rather than of answer content. The paper raises exactly this worry about its baseline, but does not report the same breakdown for its own method, dataset by dataset. Thank you for this thoughtful question. We would like to clarify that correcting answer form also contributes to accuracy, particularly on datasets where answer-form mismatches account for many evaluation errors. Both form-only and content-changing corrections contribute to improved accuracy. Our goal is to correct more errors overall, especially content errors that existing baselines leave unresolved, while preserving the benefits of answer-form refinement. To complement the overall proportions in Figure 4, we provide per-dataset correction counts summed across the three model backbones. Among corrected responses, a correction is form-only if the Base response already contains an accepted answer; otherwise, it is content-changing. | Dataset | TTRV form-only | TTIQ form-only | TTRV content | TTIQ content | Content increase | |---|---:|---:|---:|---:|---:| | InfoVQA | 31 | 70 | 101 | **147** | **+45.5%** | | TextVQA | 242 | 469 | 100 | **148** | **+48.0%** | | GQA-Relation | 659 | 1,098 | 173 | **371** | **+114.5%** | | GQA-Attribute | 209 | 250 | 124 | **211** | **+70.2%** | | ST-VQA | 721 | 1,386 | 368 | **734** | **+99.5%** | | MathVerse-VD | 12 | 9 | 15 | **22** | **+46.7%** | | COCO-QA-N | 1,060 | 1,169 | 139 | **189** | **+36.0%** | | CLEVR-H | 53 | 96 | 124 | **283** | **+128.2%** | | **Overall** | **2,987** | **4,547** | **1,144** | **2,105** | **+84.0%** | TTIQ produces **84.0%** more content-changing corrections than TTRV (2,105 vs. 1,144), with increases on all eight datasets. Content corrections more than double on CLEVR-H (+128.2%) and GQA-Relation (+114.5%). These results show that, alongside form refinement, TTIQ delivers substantially more content corrections across diverse tasks. We will include this dataset-level breakdown alongside Figure 4 in the revision.
AuthorsOpenReview ↗

Response to Reviewer iHZK (2/2)

W3: Interpreting Component Interactions in Table 2.

Table 2 does not support the component claims. Several rows are worse than the base model, which looks like training instability rather than a clean ablation of one component at a time, and the confidence term helps in one row while costing a great deal in another, with no explanation.

We appreciate the reviewer’s close examination of Table 2. Here, we'd like to clarify four points regarding table 2.

1. The components act as interdependent constraints. Under minimum fusion, adding or removing a component can substantially change response rankings and which responses are reinforced. Results below Base can therefore reflect harmful updates driven by the altered proxy reward; they do not, by themselves, establish training instability.

2. Task requirements affect the value of each component. Image-only rewards may be effective when salient visual content strongly constrains the answer, as in some OCR questions. However, strong image dependence alone does not ensure that a response addresses the specific question. Question dependence becomes useful when the question determines which object, attribute, relation, or reasoning operation is relevant, as in GQA. Adding it to minimum fusion downweights responses that depend on the image but weakly on the specific question. Sustained dependence further prevents a few highly dependent tokens from dominating the response score.

These findings support the complementary roles of image and question dependence across different task requirements. The complete TTIQ configuration combines these signals with confidence and joint-token credit assignment, achieving the highest average accuracy in Table 2.

3. Confidence can change these preferences. A useful image-dependent response may have low model likelihood and lose its ranking advantage when confidence becomes the bottleneck. This may explain why adding confidence hurts image-only rewards on GQA. Adding question dependence changes the ranking again by penalizing responses insufficiently tied to the requested attribute or relation. This provides a plausible explanation for the differences among image-only, image + confidence, and joint image–question + confidence configurations.

4. Regarding training instability. Table 2 uses the same training split, common hyperparameters, and update budget across variants. The full configuration performs best on all four tasks. We have also revised Table 1 to include two additional random seeds to assess training variability. We will clarify these interactions in the revised ablation discussion.

AuthorsOpenReview ↗

A gentle follow-up on our rebuttal

Dear reviewer iHZK,

We sincerely thank you for your time and effort in reviewing our paper, as well as your thoughtful feedback and constructive suggestions throughout the discussion.

In our rebuttal, we (1) clarified the normalization formulation used in joint-token credit assignment and provided additional sensitivity results; (2) presented a dataset-level breakdown of form-only and content-changing corrections, showing that TTIQ produces 84.0% more content-changing corrections than TTRV across eight datasets; and (3) explained the interactions among the components in Table 2, including their complementary roles and why their effects need not be additive. We also updated Table 1 with results from additional random seeds to assess training stability.

With less than 13 hours remaining in the discussion period, we would greatly appreciate hearing whether our responses have adequately addressed your concerns. If any questions or concerns remain, we would be happy to provide further clarifications and continue the discussion before the deadline. Thank you again for your valuable feedback, time, and consideration, which have been greatly appreciated and helpful in improving our work.

Reviewer_1JCjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TTIQ, a test-time reinforcement learning framework for vision-language models (VLMs) that constructs self-supervised rewards from image–question dependence rather than answer consensus. The authors identify two limitations of existing consensus-based test-time reinforcement learning methods: a substantial proportion of their exact-match improvements arise from answer normalization rather than genuine content correction, and frequently generated incorrect responses may be reinforced despite insufficient dependence on the image or question. To address these limitations, TTIQ estimates token-level image and question dependence by comparing the likelihoods of fixed generated responses under the original input and two controlled input ablations. Additionally, the method introduces joint-token credit assignment, which allocates greater positive advantage to tokens that exhibit stronger dependence on both modalities. Experiments across eight VQA datasets and multiple VLM scales demonstrate that TTIQ consistently achieves the highest average accuracy among the evaluated test-time reinforcement learning methods. The authors further investigate cross-dataset generalization, transfer across VLM families, reward components, and computational overhead.

Critical strengths
  1. Important and well-motivated research problem.

Constructing reliable rewards without ground-truth annotations is a fundamental challenge for test-time reinforcement learning. The paper appropriately highlights the limitations of consensus-based rewards, particularly when vision-language models rely on language priors or repeatedly generate incorrect answers. Moving beyond output agreement toward input-dependent supervision is a meaningful research direction.

  1. Interesting empirical analysis of consensus-based supervision.

The paper provides a useful diagnostic analysis showing that 72.3% of the identified exact-match corrections under TTRV are form-only changes, while 69.0% of incorrect sampled responses lack sufficient dependence on at least one input according to the authors' diagnostic. These findings raise important questions about whether improvements in exact-match accuracy necessarily correspond to better visual understanding.

  1. Intuitive integration of image and question dependence into reinforcement learning.

The proposed framework provides a coherent approach to incorporating input dependence into both response-level reward construction and token-level credit assignment. In particular, the minimum-bottleneck formulation is intuitively motivated by the requirement that a reliable response should be supported by both the image and the question, rather than relying strongly on only one input.

Critical weaknesses
  1. The relationship between joint image–question dependence and answer correctness is insufficiently established.

My primary concern is that the proposed reward relies on the assumption that responses exhibiting stronger joint dependence on the image and question are more likely to be correct. However, input dependence is not equivalent to grounding correctness. A model may genuinely attend to both the image and the question while still producing an incorrect answer due to recognition errors, counting mistakes, or faulty reasoning. Conversely, a correct response may exhibit relatively weak dependence on the original inputs at later generation steps because the necessary information has already been incorporated into the preceding context.

The paper reports that 69.0% of incorrect responses lack dependence on at least one input. However, this observation does not establish that responses with high joint dependence are predominantly correct, nor does it directly demonstrate that the proposed reward reliably ranks correct responses above incorrect ones. Since reward quality is the central contribution of the paper, a direct evaluation of the relationship between reward scores and answer correctness is necessary.

  1. The controlled input ablations may introduce out-of-distribution artifacts that undermine the interpretation of dependence scores.

TTIQ estimates image dependence by replacing the original image with an all-zero pixel tensor and question dependence by replacing question-content tokens with padding tokens. Although these interventions preserve the input layout, they may produce inputs that differ substantially from the distributions encountered during model pretraining. Consequently, the observed changes in token likelihood may reflect not only the removal of relevant information but also artifacts introduced by the ablation procedure. This concern is particularly relevant to question ablation, where padding tokens may alter the model's behavior in ways unrelated to genuine question dependence.

  1. The necessity of the relatively complex reward construction is not fully demonstrated against simpler alternatives.

TTIQ combines several design choices, including overall dependence, sustained dependence, nested rank normalization, minimum fusion, and confidence capping. While the ablation studies indicate that these components contribute to performance, the gains over substantially simpler alternatives appear relatively modest.

  1. Group-relative rank normalization does not eliminate the risk of reinforcing incorrect responses.

One motivation for TTIQ is to avoid the confirmation bias associated with consensus-based rewards. However, the proposed reward is still constructed entirely from the model's own sampled responses and normalized within each rollout group. If all sampled responses for a particular question are incorrect, some responses will nevertheless receive relatively high reward rankings, even when their absolute image–question dependence is weak. Consequently, the method may still reinforce incorrect answers simply because they are relatively preferable within an unreliable rollout group.

  1. The improvements over the strongest baselines are relatively small, and statistical reliability is insufficiently evaluated.

Although TTIQ achieves the highest average accuracy at every evaluated InternVL scale, the improvements over the strongest competing test-time reinforcement learning baselines are approximately 1.29, 1.05, and 0.27 percentage points for the 2B, 4B, and 8B settings, respectively. The improvement on the 8B model is particularly small, making it difficult to assess its practical significance without uncertainty estimates. Furthermore, the main experiments use only 20 randomly sampled unlabeled training examples per task. The choice of these examples may substantially affect test-time adaptation performance. The paper does not provide sufficient multi-seed evaluations, standard deviations, or confidence intervals to establish that the observed improvements are robust to support-set sampling and reinforcement learning randomness.

Questions for authors
  1. Can the authors directly evaluate the correlation between the proposed reward and ground-truth correctness? In particular, how frequently does TTIQ assign a higher reward to a correct rollout than to an incorrect rollout for the same question?

  2. How does TTIQ behave when none of the sampled responses is correct? Does the relative ranking mechanism still reinforce incorrect answers, and how does this compare with TTRL, SCOPE, and TTRV?

  3. How sensitive are the image and question dependence scores to the choice of ablation? Would blurred or unrelated images and alternative question-masking strategies produce similar reward rankings?

  4. How does the full TTIQ reward compare with image-only dependence combined with confidence, simple likelihood-based rewards, or other visual-contrastive scoring functions under matched training conditions?

  5. Can the authors report multi-seed results with different 20-shot support sets and reinforcement learning random seeds? Are the relatively small improvements on the 8B backbone statistically significant?

8 comments
AuthorsOpenReview ↗

Response to Reviewer 1JCj (1/3)

We sincerely thank the reviewer for the thoughtful and constructive feedback. Your comments have helped us strengthen the evaluation and clarify the method. We address each concern below and would be happy to provide further clarification or discuss any remaining questions.

W1 & Q1: Relationship Between TTIQ Reward and Answer Correctness.

The relationship between joint image–question dependence and answer correctness is insufficiently established;Can the authors directly evaluate the correlation between the proposed reward and ground-truth correctness? In particular, how frequently does TTIQ assign a higher reward to a correct rollout than to an incorrect rollout for the same question?

Thank you for your thoughtful suggestion. We agree that joint input dependence does not guarantee answer correctness. In label-free test-time training, ground-truth correctness is unavailable, so our goal is to identify a useful proxy signal that favors responses more likely to be correct. To evaluate this directly, we use three backbones (InternVL3-2B, InternVL2.5-4B, and InternVL3-8B) across eight datasets, using the 20 questions per model–dataset setting and 32 rollouts per question from the Base model. Of the 480 question–model groups, 285 contain both correct and incorrect responses, yielding 39,706 correct–incorrect pairs for within-question reward comparisons. TTIQ assigns a strictly higher reward to the correct rollout in 72.98% of cases, with a pairwise AUC of 73.56% (ties counted as 0.5). These results support its effectiveness as a proxy for correctness. Per-dataset results are shown below.

MetricInfoVQATextVQAGQA-Rel.GQA-Attr.ST-VQAMathVerse-VDCOCO-QA-NCLEVR-HOverall
Pair count5,9105,4876,6036,6604,5012,9585,7391,84839,706
Correct higher (%)81.4072.0179.2787.4670.9449.9360.4555.0372.98
Pairwise AUC (%)81.4272.4679.3388.0071.3154.5560.6655.0373.56

W2 & Q3: Sensitivity to Input Ablation Choices.

The controlled input ablations may introduce out-of-distribution artifacts that undermine the interpretation of dependence scores; How sensitive are the image and question dependence scores to the choice of ablation? Would blurred or unrelated images and alternative question-masking strategies produce similar reward rankings?

Thank you for raising this concern. Our goal is to measure how response likelihood changes when information from either input is removed. Therefore, we use zero-image and PAD-question inputs as simple null references that preserve the input layout without introducing unrelated semantic content. To assess whether our results depend on these specific choices, we evaluated alternative ablations from two perspectives: main-task performance and reward ranking stability.

1. Main-task performance. We repeated training under matched settings using image blur (σ=12\sigma=12), mismatched images, and question ablation that replaces each question-content token with a period token while preserving the token count.

Input ablationInfoVQATextVQAGQA-Attr.GQA-Rel.Avg.
Base57.8973.5545.5917.2548.57
Blur59.4979.1857.2736.2858.06
Mismatched image58.2377.4356.4136.1157.05
Period-token question59.1577.3555.9035.8457.06
Default (zero image / PAD question)59.7379.5257.1037.1558.38

All alternatives retain substantial gains over Base. In particular, blur closely matches the default (58.06 vs. 58.38), while replacing PAD tokens with period tokens also preserves substantial gains.

2. Reward ranking stability. Using InternVL2.5-4B with 200 questions per dataset and 32 rollouts per question, we rescored identical responses under each alternative ablation and computed Spearman’s ρ\rho against the default reward rankings.

Alternative ablationInfoVQATextVQAGQA-Attr.GQA-Rel.
Blur0.9750.9340.8100.827
Mismatched image0.9370.8780.7870.829
Period-token question0.9200.8890.8910.883
Masked question attention0.9080.8700.8710.863

Rank correlations remain high across all alternatives (0.787–0.975). These results indicate that TTIQ’s reward rankings and main-task gains are broadly consistent across the tested ablation choices, and the observed benefits are not tied to the specific zero-image or PAD-question references.

AuthorsOpenReview ↗

Response to Reviewer 1JCj (2/3)

W3 & Q4: Comparison with Simpler Reward Alternatives.

The necessity of the relatively complex reward construction is not fully demonstrated against simpler alternatives; How does the full TTIQ reward compare with image-only dependence combined with confidence, simple likelihood-based rewards, or other visual-contrastive scoring functions under matched training conditions?

Thank you for suggesting these comparisons. We compare the full TTIQ reward with three simpler alternatives: mean response-token log-likelihood, visual-contrastive likelihood (2ℓoriginal−ℓimage-ablated2\ell_{\mathrm{original}}-\ell_{\mathrm{image\text{-}ablated}}) in [1], and image-only dependence combined with confidence. All variants use uniform full-response credit.

RewardInfoVQATextVQAGQA-Attr.GQA-Rel.Avg.
Mean token log-likelihood58.2070.5244.671.8643.81
Visual-contrastive likelihood59.2474.9355.9617.0149.29
Image dependence + confidence58.1473.4344.0416.2447.96
TTIQ reward (ours)59.4976.3756.1933.5756.40

TTIQ achieves the highest accuracy on all four tasks, exceeding visual-contrastive likelihood and image dependence + confidence by 7.11 and 8.44 points on average, respectively. For mean token log-likelihood, we observed a shift toward longer, more complete sentences on GQA-Relation, which are poorly matched to GQA-Relation's short-answer exact-match evaluation and lead to the low score.

We interpret these differences in light of task requirements. Image-only rewards may be particularly effective when salient visual content strongly constrains the answer, as in some OCR questions (e.g., TextVQA, InfoVQA). Question dependence provides an additional signal when the question determines which object, attribute, or relation is relevant, or which reasoning operation is required, as in GQA. Under minimum fusion, adding question dependence can lower the scores of responses that depend on the image but only weakly on the specific question, substantially changing their ranking and the resulting training preferences. This offers a plausible explanation for the larger gains over image dependence + confidence on GQA. TTIQ's full design accounts for both inputs to cover these diverse scenarios.

W4 & Q2: Behavior When All Sampled Responses Are Incorrect.

Group-relative rank normalization does not eliminate the risk of reinforcing incorrect responses; How does TTIQ behave when none of the sampled responses is correct? Does the relative ranking mechanism still reinforce incorrect answers, and how does this compare with TTRL, SCOPE, and TTRV?

We thank the reviewer for highlighting this case. When all sampled answers are incorrect, both TTIQ and consensus-based baselines can assign positive advantages to incorrect responses under GRPO if rewards vary within the group. The distinction lies in how these rewards are constructed. Consensus-based rewards favor agreement with the majority answer, whereas TTIQ uses image–question dependence as a proxy signal. Even when all final answers are incorrect, some responses may use relevant visual and question information more effectively than others. In this challenging setting, TTIQ aims to use these differences to guide training, providing a potential learning signal even when no sampled response is fully correct.

To evaluate this challenging setting, we selected questions that Base model initially and repeatedly answered incorrectly and sampled 32 responses per question. For each K∈{4,8,16,32}K\in\{4,8,16,32\}, Base-0/KK denotes questions where all of the first KK sampled responses were incorrect. We then evaluated each method on these same question subsets, with one answer per question.

DatasetBase cohortExamplesTTRLSCOPETTRVTTIQ
TextVQA0/482410.92%8.98%2.06%12.26%
TextVQA0/86906.67%5.36%1.16%7.97%
TextVQA0/165472.56%1.83%0.37%3.29%
TextVQA0/324310.93%0.93%0.00%0.93%
GQA-Relation0/42,55813.14%12.67%10.24%15.25%
GQA-Relation0/82,2608.67%8.19%6.11%10.75%
GQA-Relation0/161,9614.39%3.98%2.40%6.37%
GQA-Relation0/321,6851.54%1.19%0.59%2.97%

TTIQ achieves the highest accuracy on seven of eight subsets and ties for best on the remaining subset. On the more challenging GQA-Relation-0/32 subset, TTIQ reaches 2.97%, compared with 1.54%, 1.19%, and 0.59% for TTRL, SCOPE, and TTRV, respectively. These results show that TTIQ can still recover some correct answers even when none of the initially sampled responses is correct.

AuthorsOpenReview ↗

Response to Reviewer 1JCj (3/3)

W5 & Q5: Robustness to Training Randomness and Support-set Randomness.

The improvements over the strongest baselines are relatively small, and statistical reliability is insufficiently evaluated; Can the authors report multi-seed results with different 20-shot support sets and reinforcement learning random seeds? Are the relatively small improvements on the 8B backbone statistically significant?

Thank you for emphasizing the need to assess robustness to training randomness and support-set sampling. We have added the suggested experiments. Given the size of the main results table, we have reported the multi-seed results in the updated Table 1 of the manuscript and present the support-set sampling results below.

On training seeds and variance. Due to limited time, we have added two random seeds and now report mean and standard deviation across three seeds for every method and have updated our Table 1 in the manuscript. We found that variability is comparable across methods, and the three-seed averages retain the improvements, with better performance compared to baselines in several settings.

20-training set with random seeds. We report results across 3 randomly sampled training sets on InterVL3-2B and InternVL2.5-4B below. TTIQ achieves the highest mean accuracy, supporting its effectiveness beyond a particular training sample.

ModelMethodGQA-RelGQA-AttrST-VQACOCO-QA (Num)
InternVL3-2BTTRL32.24 ± 0.8452.84 ± 1.6676.25 ± 0.3772.93 ± 0.19
SCOPE32.77 ± 0.8952.79 ± 1.6276.24 ± 0.3572.98 ± 0.39
TTRV27.47 ± 3.5952.50 ± 0.7576.06 ± 0.1074.08 ± 0.42
TTIQ (ours)33.63 ± 1.5353.76 ± 0.6077.08 ± 0.6174.66 ± 0.56
InternVL2.5-4BTTRL35.24 ± 0.5756.74 ± 0.6382.98 ± 1.0572.13 ± 2.91
SCOPE33.73 ± 1.4256.61 ± 1.0882.89 ± 0.8872.82 ± 2.34
TTRV28.52 ± 4.9056.95 ± 1.3081.53 ± 0.3472.22 ± 2.31
TTIQ (ours)36.37 ± 0.7057.69 ± 0.5583.13 ± 0.6274.64 ± 0.21

[1] Leng S, Zhang H, Chen G, et al. Mitigating object hallucinations in large vision-language models through visual contrastive decoding.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chairs and Reviewers,

We sincerely thank the reviewers for their constructive feedback. We have uploaded a revised manuscript and summarize below our contributions and the additional evidence provided during the rebuttal.

Our work makes three contributions:

  1. Diagnostic insights. Our analysis of consensus-based, label-free VLM test-time reinforcement learning shows that many exact-match gains reflect answer-form changes. We also find that incorrect VLM rollouts often lack dependence on the image, the question, or both.
  2. An effective learning signal from image–question dependence. Based on our diagnostic analysis, TTIQ uses controlled input ablations to derive label-free response rewards and token-level credit, encouraging models to use both inputs more effectively for VQA during test-time training.
  3. Effectiveness and generalization across broad settings. Across eight VQA benchmarks and three InternVL scales, TTIQ achieves the highest average accuracy in our three-seed evaluation using only 20 unlabeled training examples. Additional Qwen2.5-VL and cross-dataset evaluations support its generalization.

We sincerely thank the reviewers for recognizing the diagnostic value of our analysis (all three reviewers), the motivation for using image–question dependence in both reward construction and token-level credit assignment (Reviewer 1JCj), and the broad evaluation and modest computational overhead (Reviewers iHZK and w5KU).

During the rebuttal, we strengthened the evidence through:

  • Direct validation of reward quality: TTIQ ranks correct responses above incorrect ones in 72.98% of 39,706 within-question comparisons, supporting joint dependence as a useful correctness proxy.
  • Content-correction analysis: The dataset-level breakdown shows 84.0% more content-changing corrections than TTRV overall, with increases on all eight datasets.
  • Component comparisons and alternative input ablations: We evaluated mean log-likelihood, visual-contrastive likelihood, and image-only reward variants, including variants with identical joint-token credit. Full TTIQ performs best on all four evaluated datasets, supporting its combined reward and credit design. Additional experiments on alternative input ablations, including image blur and period-token question replacement, also show substantial performance gains.
  • Statistical significance and reliability: We updated Table 1 with three-seed means and standard deviations. The updated averages retain the overall advantage and further improve on some benchmarks. At 8B, paired tests across three matched seeds show significant average gains over all three baselines, even after Holm correction. Separately, evaluation across three randomly sampled training sets shows that TTIQ achieves the highest mean accuracy in all model–dataset settings.
  • Performance when initial rollouts are all incorrect: TTIQ performs best on all eight evaluated subsets where the base model initially sampled only incorrect answers, including subsets with 32 incorrect rollouts.
  • Robustness and reproducibility: We clarified token-credit normalization and implementation settings, provided additional sensitivity analyses, and showed that overall gains persist without selection. These results further support joint image–question dependence as useful supervision for label-free test time training, with benefits in content correction, consistent performance across training runs, and generalization.

We would welcome any remaining questions or feedback on the additional experiments and revised manuscript. Thank you again to the AC and reviewers for your time and consideration.

Best,

Submission 46992 Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.