acceptodds
Under review as a conference paper at ICLR 2027

Aligning Thoughts with Answers: Probability Rewards to Tame Thinking Drift

Abstract

This paper studies thinking–answer consistency in vision-language models. We focus on Visual Intention Grounding, where a model infers a target object based on a human intention query and predicts a bounding box. We reveal that previous IoU-based reinforcement learning (RL) frameworks suffer from “thinking drift”, where the model produces a correct bounding box, despite having an incorrect reasoning process pointing to a different target object. Thus, we propose Rita (ReInforcing Thinking–Answer consistency) as a novel RL paradigm to tame the drift. Specifically, Rita introduces two reasoning-label-free RL rewards, constructed from the conditional probability of reference answers: a thinking reward and a consistency reward. It also adopts a difficulty-aware data filtering strategy that selects informative easy-to-medium samples for RL using rollout error rate and reward variance. Extensive experiments on EgoIntention and the new RefEgo-Int benchmarks show that Rita performs consistently superior to the supervised finetuning approaches and vanilla RL-finetuned frameworks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
egocentric assistant, visual grounding, reinforcement fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zKeoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the "thinking drift" phenomenon that emerges when vision-language models (VLMs) are trained with reinforcement learning for Visual Intent Grounding (VIG) tasks. VIG requires models to infer target objects from implicit human intent queries and predict their bounding boxes. The core finding is that RL training with IoU-only rewards (e.g., VLM-R1, UniVG-R1) leads to reward hacking where "bounding boxes are correct but the reasoning chain points to the wrong target." To address this, the authors propose Rita (ReInforcing Thinking-Answer consistency), which adds two annotation-free rewards to standard GRPO training: a thinking reward (Equation 3) that replaces the model's sampled answer with the reference answer and measures the average probability of reference answer tokens conditioned on the sampled reasoning chain; and a consistency reward (Equation 5) that multiplicatively couples the thinking score with the IoU score. Rita also employs a difficulty-aware data filtering strategy, using only 5% of the SFT data. Experiments are conducted on EgoIntention and the newly constructed RefEgo-Int dataset using Qwen2.5-VL (3B and 7B). Rita achieves +4.8/+2.1 [email protected] improvement over the SFT baseline on EgoIntention, and +10.2/+7.5 zero-shot improvement on RefEgo-Int (Table 1, Figure 5).

Critical strengths
  1. The thinking drift phenomenon is clearly defined, and validated through statistical significance testing (McNemar's test p=3.7e-7, Appendix C). The finding that 15.8% of newly corrected bounding boxes under IoU-only RL exhibit reasoning-answer inconsistency (Table 5) is a concrete, actionable insight that is in principle applicable to any structured prediction task where reasoning is unconstrained under outcome-reward RL.
  2. The paper shows that the core insight of using the likelihood of the reference answer conditioned on the sampled reasoning chain (Equation 3), rather than requiring annotated reasoning chains. The ablation in Table 7 validates the necessity of the reference answer override. The multiplicative consistency reward (Equation 5) provides a natural coupling mechanism.
Critical weaknesses
  1. All evaluations cover only a single task (VIG), one model family (Qwen2.5-VL 3B/7B), and two closely related egocentric datasets (EgoIntention and RefEgo-Int, both derived from Ego4D). Moreover, the core algorithmic contribution is limited to modifying the reward design, and no experiments are provided on other benchmarks or VLM architectures. For a top venue like ICLR2027, I consider both the novelty and generalizability to be seriously insufficient.
  2. The core claim is that Rita improves thinking-answer consistency, yet the only reasoning metric (Acc_think) merely checks whether the output label string matches the ground-truth. This does not measure whether the reasoning chain itself is coherent and relevant. Without direct reasoning evaluation, the paper cannot distinguish between Rita produces better reasoning and Rita produces better category labels.
Questions for authors
  1. Could the authors report Rita's own thinking drift rate (analogous to Table 5)? The paper quantifies drift under IoU-only RL but never presents the corresponding data for Rita, leaving the core claim that Rita reduces drift without direct verification.
  2. The thinking reward (Equation 3) averages token probabilities rather than log-probabilities. Was log-probability considered? Averaging raw probabilities is dominated by high-confidence tokens and may fail to distinguish reasoning chains that differ on a few semantically critical tokens.
6 comments

W1: Scope, generality and contribution

We thank the reviewer for recognizing thinking drift as an insight that applies beyond VIG. The concern is whether the evidence is too narrow (one task, one model family, two Ego4D-derived datasets) and whether a change confined to the reward is a sufficient contribution. We focus on VIG by design, because this is where thinking drift arises and matters most; within this scope, Rita's gains hold across two model lines with different architectures, from 3B dense vision-language models to a 122B natively multimodal mixture-of-experts model, and on two third-person grounding benchmarks it never saw, Rita stays within 0.8 points of naive GRPO and improves over the model before RL on RefCOCOg.

Why we focus on VIG. Thinking drift was first observed in VIG, and VIG is where it matters most: the target must be inferred from an implicit intention, so the reasoning determines the answer rather than restating the query. The paper therefore studies one task in depth, from the diagnosis of drift (Appendix C) to a reward that counters it, on EgoIntention and on RefEgo-Int, which adds camera motion, no-target queries and long-tail categories (Appendix A).

Across model lines, architectures and scales. Beyond Qwen2.5-VL-3B/7B, we trained Rita on Qwen3-VL-4B, the next model of the Qwen-VL line, and on Qwen3.5-122B-A10B, which differs from the Qwen-VL models in architecture; we re-implemented Rita for both in two other RL frameworks. The table shows that, from the same starting model and RL data, Rita outperforms naive GRPO on both backbones (Context / Uncommon):

BackboneRL reward[email protected]mIoUAcc_thinkAcc_align
Qwen3-VL-4Bnaive GRPO67.9 / 58.30.627 / 0.54575.5 / 62.459.8 / 49.8
Rita70.0 / 61.00.646 / 0.56877.8 / 65.262.4 / 52.7
Qwen3.5-122B-A10Bnaive GRPO77.4 / 72.90.720 / 0.68189.8 / 82.272.7 / 67.8
Rita77.8 / 73.60.723 / 0.68791.5 / 84.474.2 / 69.6

Rita improves Acc_think by 1.7–2.8 points and [email protected] by 0.4–2.7 points over naive GRPO. At 122B, naive GRPO raises the share of samples whose box is correct but whose label is wrong from 3.9% / 4.5% to 4.8% / 5.1%, and Rita removes 25.0% / 21.6% of these cases (3.6% / 4.0%).

Other grounding benchmarks. We further evaluated the EgoIntention-trained models zero-shot on two third-person benchmarks that differ from EgoIntention in viewpoint and query type: RefCOCOg (referring expressions) and LISA-Grounding (queries that require reasoning). The table shows that RL with Rita transfers to RefCOCOg and that Rita's reasoning rewards do not cost grounding on unseen benchmarks (zero-shot [email protected]; best trained model in bold):

Zero-shot [email protected] (trained on EgoIntention only)Before RLnaive GRPORita
RefCOCOg, Qwen3-VL-4B77.180.681.4
LISA-Grounding, Qwen3.5-122B-A10B82.682.682.9

Contribution. The algorithmic change is confined to the reward by design: thinking drift arises because the IoU reward never sees the reasoning, so the reward is where it should be addressed. Our contribution is the diagnosis of thinking drift under IoU-reward RL for visual intention grounding, and a reasoning-label-free reward that counters it. The rebuttal experiments also show that this reward trains grounding without the IoU reward (xkeY, Q2) and improves the reasoning text itself (SmsG, W1), at the cost of one extra forward pass (SmsG, W5).

We will add these results to the paper.

W2: Does Rita improve the reasoning itself, or only the labels?

We thank the reviewer for this comment. The concern is whether Acc_think, which checks only the predicted label, can tell better reasoning from better labels. We now evaluate the reasoning text directly: read without the label, Rita's reasoning names the correct object more often and is rated more relevant to the person's need, so its gain is not confined to the label; these judgements cover the reasoning's conclusion and its relevance, not each step.

Reading the reasoning without the label. Three LLM judges (Qwen3-VL-32B, Nemotron-3.5-30B and DeepSeek-V4.1-Flash; protocol in our reply to Reviewer xkeY, W1) read only the <think> text and name the object it settles on. The table shows that Rita's reasoning identifies the correct object more often, and contradicts the model's own label less often, than that of IoU-only RL (% of test outputs, averaged over the two splits):

ModelReasoning names the ground-truth object ↑Reasoning contradicts the model's own label ↓
Qwen2.5-VL-3B, IoU-only RL73.46.0
Qwen2.5-VL-3B, Rita75.05.1
Qwen3-VL-4B, IoU-only RL74.77.8
Qwen3-VL-4B, Rita77.54.5
  • The gain appears in the reasoning itself. Rita's reasoning names the correct object 1.6 / 2.8 points more often. This measure never reads the label, so the gain cannot come from better labels alone.
  • Rita tightens the link between reasoning and label. Contradictions fall from 6.0% to 5.1% and from 7.8% to 4.5%, i.e., Rita removes 15.0% and 42.3% of them. If Rita improved only the labels, they would rise instead.
  • Acc_think is a fair proxy for both methods. For every model, IoU-only RL included, the reasoning and the predicted label refer to the same object in 88–96% of outputs, so comparing Acc_think between methods does not favour Rita.

Relevance and coherence. In a second pass, the same three judges read only the person's stated need and the reasoning, with no image, label or ground truth, and rate the reasoning's relevance to the need and its internal coherence from 1 to 5; we use the median of their three ratings. The judges agree more on relevance than on coherence, so we treat relevance as the main measure. Relevance responds to content: pairing a reasoning with another sample's need lowers its relevance from 3.87 to 1.60, and reasoning that names the correct object is rated more relevant than reasoning that names another (3.95–4.16 vs. 2.98–3.52). The table shows that R_think makes the reasoning more relevant on the same RL data, and that Rita raises relevance further (Qwen2.5-VL-3B, where all models write reasoning of similar length, 31–33 words):

Qwen2.5-VL-3BIoU-only RLIoU + R_think (same RL data)Rita
Mean relevance (1–5)3.813.873.89
Mean coherence (1–5)4.174.174.22
Both ≥ 4, all outputs (%)68.170.070.5
Both ≥ 4, Uncommon (%)50.254.454.8

On Qwen3-VL-4B, Rita's reasoning is also rated higher (both ≥ 4: 67.2% → 73.8%), but it is much shorter (62 vs. 90 words) and the judges favour shorter text, so we do not count this as evidence.

Scope. These judgements check the object the reasoning settles on and its relevance and coherence as text, not each step against the image; step-level grading would require reference reasoning, which EgoIntention does not provide and which would not be unique.

We will add both evaluations, the judge prompts, the agreement between judges and the per-sample judgements to the appendix, and revise Section 4.1 to state that Acc_think measures the target of the reasoning.

Q1: Rita's own thinking-drift rate

We thank the reviewer for this request; this comparison should have been in the paper. The concern is whether Rita actually reduces thinking drift when measured as in Table 5. It does: on all three backbones we trained, Rita removes 7.6–24.2% of the drifting cases left by IoU-only RL without any extra annotation, and on the two newer backbones it brings drift below its level before RL. It reduces drift rather than eliminating it, and drift read directly from the reasoning text is rare for both methods and changes little.

Drift rate as in Table 5. As in Table 5, we measure drift as the percentage of samples with a wrong predicted category among all samples whose box is correct (IoU ≥ 0.5), over both splits. An ideal model has zero drift: every correct box should come with a category that names the boxed object. Table 5 analyzes UniVG-R1, whose RL starts from a chain-of-thought cold start; here, IoU-only RL and Rita start from the same fine-tuned model of each backbone, so we compare the two RL methods with each other rather than with the numbers of Table 5.

  • Drift is substantial under IoU-only RL. After IoU-only RL, 9.9% of correct boxes come with a wrong category on Qwen2.5-VL-3B (up from 5.8% before RL), 13.2% on Qwen3-VL-4B and 6.6% on Qwen3.5-122B-A10B (up from 5.6%). [email protected] counts all of them as successes.
  • Rita removes 7.6–24.2% of these drifting cases without extra annotation. It lowers drift to 9.0% on Qwen2.5-VL-3B, 12.2% on Qwen3-VL-4B and 5.0% on Qwen3.5-122B-A10B, a relative reduction of 9.1%, 7.6% and 24.2% from the IoU-only level, using only the box annotations that IoU-only RL already uses. On the two newer backbones, drift ends below its level before RL.
  • Drift is not eliminated. On Qwen2.5-VL-3B, Rita removes only part of the increase caused by RL: among the boxes that RL newly corrects, 11.5% come with a wrong category under Rita, against 12.5% under IoU-only RL.

Drift read from the reasoning text. We also applied the reasoning-text classification of our reply to Reviewer xkeY (W1) to Rita. The table shows that Rita produces more outputs whose box and reasoning are both correct, while the two inconsistent categories change little (% of outputs, Context / Uncommon):

ModelCorrect and consistent ↑Drift: box correct, reasoning names another object ↓Off-target: reasoning correct, box misses it ↓
Qwen2.5-VL-3B, IoU-only RL57.0 / 50.32.0 / 4.413.4 / 9.6
Qwen2.5-VL-3B, Rita57.8 / 52.31.8 / 4.412.4 / 10.6
Qwen3-VL-4B, IoU-only RL62.8 / 52.92.5 / 2.49.9 / 9.6
Qwen3-VL-4B, Rita65.0 / 55.52.4 / 3.010.5 / 9.7
  • More outputs are correct and consistent. Averaged over the two splits, they rise from 53.7% to 55.0% on Qwen2.5-VL-3B and from 57.8% to 60.3% on Qwen3-VL-4B.
Reviewer_SmsGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Rita, a framework that helps vision-language models better understand what a person intends to interact with by making their reasoning consistent with predicted bounding boxes. For example, when a person reaches for an object, the model should explain its intention and correctly locate the object. Rita uses reinforcement learning to reward correct answers, consistent reasoning, and accurate localization, while adjusting data sampling based on task difficulty. Experiments show that this approach improves performance over supervised fine-tuning baselines.

Critical strengths
  1. This paper identifies thinking drift, which is intuitively sound as a key issue of the thinking process.

  2. This paper proposed an intuitive approach of using reference-answer likelihood to encourage consistency without requiring human-annotation.

  3. I appreciate the experiments that shows improved grounding accuracy across 2 model sizes, and many ablation studies, e.g., to examine reward components, reference-answer overwrite, and difficulty-aware sampling.

Critical weaknesses
  1. The thinking reward does not directly measure reasoning correctness. The probability could be high due to the confidence on incorrect thinking. Perhaps this is why the evaluation mainly measures object-label accuracy and bounding-box alignment, rather than the correctness of the reasoning process itself (or perhaps I'm too pessimistic).

  2. Other reward design may works, but lack discussions. For example, I think average log-likelihood is compatible with this work's formulation as well.

  3. Since the reference answer includes both bounding-box coordinates and an object label, it is not clear which components contribute most to the improvement. Ablations using label-only and coordinate-only rewards would help clarify the mechanism.

  4. Experiments use 3B and 7B models which are quite small, so it is unclear whether it works on larger models which might be already good at the thinking process.

  5. Computational efficiency is not fully evaluated. Although the authors acknowledge this as the limitation, there is no latency numbers reported. I think the training could be much slower as it requires an additional teacher forward pass to calculate the thinking reward.

  6. Although [email protected] improves over UniVG-R1, the mean IoU is lower (0.558 versus 0.576 for the 3B model). This makes the improvement gains less convincing.

Questions for authors
  1. Why exactly is this reward design, not other simple functions like average log-likelihood?

  2. How does the improvement looks like when apply this work to a very large model (100B+)?

  3. How is the computational efficiency?

7 comments

W1: Does R_think reward confident but incorrect reasoning?

We thank the reviewer for this question. The concern is whether R_think rewards the model's confidence rather than correct reasoning, and whether our evaluation checks the reasoning itself or only the label and the box. R_think does not reward confidence in the model's own answer: it scores the ground-truth box after the model's reasoning, so confident reasoning toward the wrong object receives a low reward. Reading the reasoning text directly, we find that Rita's reasoning names the correct object more often than that of IoU-only RL, although this check covers the reasoning's conclusion rather than every step.

R_think scores support for the correct answer, not confidence.

  • R_think replaces the model's own answer with the ground-truth box and measures how likely the policy finds that box after the model's reasoning (Eq. 3). Reasoning that commits to the wrong object makes the ground-truth box unlikely, however confident the model is in its own answer.
  • Confidence in the model's own answer is what the variant without reference-answer overwrite measures. At step 500, that variant lowers Acc_think from 83.7 (IoU-only RL) to 83.1, while R_think raises it to 85.1 (Table 7, Appendix D.2).
  • During training, rollouts with a correct box receive a higher R_think than rollouts with a wrong box (0.49–0.61 vs. 0.40–0.44 across runs). Trained with R_think alone, without the IoU reward, the model improves on both [email protected] and Acc_think over the SFT model (reply to Reviewer xkeY, Q2), so optimizing R_think by itself moves the model toward correct answers.

Evaluating the reasoning itself. Free-form reasoning has no unique ground truth: EgoIntention provides no reference reasoning, and many different chains can justify the same object. Rita is designed for this setting, so the paper used the predicted category as a proxy for the target of the reasoning (Acc_think). We now also evaluate the reasoning text directly: three LLM judges (Qwen3-VL-32B, Nemotron-3.5-30B and DeepSeek-V4.1-Flash; see our reply to Reviewer xkeY, W1) read only the <think> text and name the object it settles on.

The table shows that Rita's reasoning identifies the correct object more often, and contradicts the model's own label less often, than that of IoU-only RL on both backbones (% of test outputs, averaged over the two splits):

ModelReasoning names the ground-truth object ↑Reasoning contradicts the model's own label ↓
Qwen2.5-VL-3B, IoU-only RL73.46.0
Qwen2.5-VL-3B, Rita75.05.1
Qwen3-VL-4B, IoU-only RL74.77.8
Qwen3-VL-4B, Rita77.54.5
  • The reasoning itself improves. Read from the text alone, Rita's reasoning names the correct object 1.6 / 2.8 points more often than that of IoU-only RL on Qwen2.5-VL-3B / Qwen3-VL-4B.
  • The reasoning contradicts the predicted label less often: Rita removes 15.0% and 42.3% of these contradictions (6.0% → 5.1%, 7.8% → 4.5%).
  • The label proxy reflects the reasoning. Across all models we evaluated, the reasoning and the predicted label refer to the same object in 88–96% of outputs, so Acc_think tracks the object the reasoning settles on.
  • Scope. R_think cannot detect reasoning that is flawed in its details but still leads to the correct box, and the judges check which object the reasoning settles on, not each step. We therefore claim neither as a full measure of reasoning correctness.

We will add this evaluation, the judge prompts and the per-sample judgements to the appendix, and revise Section 4.1 to state that Acc_think measures the target of the reasoning.

W2 & Q1: Other reward forms, such as the average log-likelihood

We thank the reviewer for this suggestion. The concern is whether simpler reward forms, such as the average log-likelihood, would work as well, and why we chose ours. Our R_think is in fact the exponentiated average log-likelihood of the reference-answer tokens; trained with three different forms, Rita reaches nearly the same grounding accuracy with each, and ours gives the highest reasoning accuracy while keeping the reward in [0, 1], which R_cons requires.

R_think already uses the average log-likelihood. Our implementation computes R_think as the exponential of the average log-likelihood of the reference-answer tokens, i.e., the geometric mean of their probabilities. We apologize for an error in the paper: Eq. 3 and the sentence that follows it describe the arithmetic mean of the probabilities instead. All reported results use the geometric mean. The question is therefore why we exponentiate rather than use the average log-likelihood directly.

We trained Rita on Qwen2.5-VL-3B with three forms of R_think, with everything else identical. The table shows that all three improve grounding over SFT by a similar margin, and that ours gives the highest reasoning accuracy (Context / Uncommon):

R_think form[email protected]mIoUAcc_thinkAcc_align
SFT (no RL)63.4 / 58.90.589 / 0.54681.5 / 71.258.5 / 52.6
Mean of token probabilities67.4 / 61.60.624 / 0.57284.5 / 72.562.4 / 55.1
Average log-likelihood66.9 / 61.90.619 / 0.57484.9 / 73.561.7 / 55.3
Exponentiated average log-likelihood (ours)67.2 / 62.00.622 / 0.57385.3 / 74.562.6 / 56.1
  • Grounding is insensitive to the form. All three improve [email protected] over SFT by 2.7–4.0 points and lie within 0.5 points of each other.
  • Ours gives the highest reasoning accuracy, most clearly on Uncommon: Acc_think 74.5, against 73.5 with the average log-likelihood and 72.5 with the mean of probabilities.
  • Why not the average log-likelihood directly. It is negative and unbounded below, which breaks the consistency reward R_cons (Eq. 5): used in place of R_think, it makes the total reward decrease with IoU whenever it falls below −1, so a better box receives a lower reward. During training, this happened in up to 19% of rollouts and reversed the advantage order of up to 19% of rollout pairs whose IoU differs by more than 0.1, against at most 0.5% with ours. Exponentiating keeps the ranking of rollouts by the average log-likelihood but bounds the reward in [0, 1], on the same scale as R_IoU and R_fmt, so the sign of R_cons is set by the box as intended.
  • Why not the mean of probabilities. It is dominated by high-probability tokens: for the 10% of rollouts with the most uneven token probabilities, it still gives 0.68, against 0.31 with ours. It also compresses rewards within a rollout group (coefficient of variation 0.009 vs. 0.028), leaving GRPO less signal to separate reasoning traces.

We will correct Eq. 3 and its description, and add this comparison and the reward analysis to the appendix.

W3: Box coordinates or object label: which part of the reference answer matters?

We thank the reviewer for this suggestion. The concern is whether the improvement comes from the box coordinates or from the object label in the reference answer. It comes from the coordinates alone: the reference answer A⋆ scored by R_think and R_cons contains only the ground-truth box, so Rita is already the coordinate-only variant. The paper did not state this, and the label in our answer format makes the reviewer's reading natural.

What is scored. During RL, we replace the box in the model's answer with the ground-truth box and score only its coordinate tokens (Eq. 3). The label that the model writes after the box is never scored, and because it comes after the box, it cannot affect the scored probability either. No reward uses the category annotation, so during RL Rita uses the same supervision as the IoU-only baselines: the ground-truth box.

Object identity is learned from the coordinates alone.

  • Acc_think rises from 81.5 / 71.2 (SFT) to 85.3 / 74.5 with Rita, and to 85.8 / 74.8 when Rita is trained without the IoU reward (reply to Reviewer xkeY, Q2), although no reward ever sees a category name.
  • Among rollouts whose predicted box is wrong, those whose reasoning names the correct object still receive a higher R_think in 69% of rollout groups. The likelihood of the correct box after a reasoning trace thus reflects whether the trace identifies the right object.

Why we do not add a label reward. A label-only or box-plus-label reward would use the category annotation as additional RL supervision, which the IoU-only baselines do not receive. Such a variant would add information rather than isolate a component of Rita.

We will state in Section 3.2 that A⋆ consists of the box coordinates only.

W4 & Q2: Larger models (100B+)

We thank the reviewer for this question. The concern is whether Rita still helps on a much larger model that may already reason well. On Qwen3.5-122B-A10B, Rita still improves over naive GRPO on every metric, mainly in reasoning, with a small grounding gain. Thinking drift also appears at this scale: naive GRPO makes correct boxes with a wrong object more frequent, and Rita removes 25.0% / 21.6% of these cases, ending below the level before RL.

Setup. As in our reply to Reviewer xkeY (Q1), we fine-tune Qwen3.5-122B-A10B on EgoIntention to produce our think-then-answer output, then train it with naive GRPO (R_IoU + R_fmt) and with Rita from this same model, using the same RL data.

The table shows that the 122B model already reasons well before RL, that naive GRPO improves its boxes without improving its reasoning, and that Rita improves both (Context / Uncommon):

Qwen3.5-122B-A10B[email protected]mIoUAcc_thinkAcc_alignBox correct, label wrong ↓
Fine-tuned (before RL)76.7 / 72.60.713 / 0.67790.1 / 82.572.8 / 68.13.9 / 4.5
naive GRPO77.4 / 72.90.720 / 0.68189.8 / 82.272.7 / 67.84.8 / 5.1
Rita77.8 / 73.60.723 / 0.68791.5 / 84.474.2 / 69.63.6 / 4.0

The last column is the drift measure of Table 5: the share of samples whose box is correct but whose predicted label is wrong.

  • Rita improves over naive GRPO. It raises Acc_think by 1.7 / 2.2 and Acc_align by 1.5 / 1.8 points, and [email protected] by 0.4 / 0.7 points.
  • Thinking drift appears at 122B, and Rita reverses it. Before RL, the model already names the correct object in 90.1% / 82.5% of cases. Naive GRPO raises [email protected] by 0.7 / 0.3 points but lowers Acc_think by 0.3 / 0.3, and the share of samples whose box is correct but whose label is wrong rises from 3.9% / 4.5% to 4.8% / 5.1%. Rita lowers it to 3.6% / 4.0%, removing 25.0% / 21.6% of these cases.
  • The grounding gain is small at this scale. Rita's [email protected] lead over naive GRPO is 0.4 / 0.7 points at 122B, against 2.1 / 2.7 on Qwen3-VL-4B, as naive GRPO already reaches 77.4 / 72.9. Each 122B run takes 16–21 hours on four H200 GPUs, so we report one training run per method.

We will add the 122B results to Section 4.

Reviewer_xkeYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper mainly study on thinking–answer consistency in vision-language models.

It proposes that there is a in-consisitency where previous RL-based frameworks suffer from “thinking drift”, where the model produces a correct bounding box, despite having an incorrect reasoning process pointing to a different target object.

To mitigate this bias/disadvantages, this paper proposes Rita (ReInforcing Thinking–Answer consistency) as a novel RL paradigm to tame the drift.

Extensive experiments on EgoIntention and the new RefEgo-Int benchmarks show that Rita performs consistently superior than other RL methods.

Critical strengths
  1. This paper proposes the key problem locating in the problem of Visual Intention Grounding where they over-rely on the IoU reward/result, ignoring the probem of inconsistency related to the thinking. It tests that IoU-only RL improves grounding precision, yet 15.8% of the boxes it newly corrects come with reasoning that targets a different object, 2.4× the drift rate before RL. This is a neat and interesting finding which reveals the failure in former RL-based method (kind of).

  2. The design of Rita is neat and reasonable. Instead of learning IoU, its thinking reward encourages reasoning traces which directly contribute to higher conditional likelihood to the reference answer. And it also try the data filtering for efficiency. The result show its remarkable effectivness.

Critical weaknesses
  1. This paper proposes the in-consistency of thinking process and VIG result, resulting in the failure of it. This should be a neat finding and exploration whil there is a lack of section in systematically evaluate this phenomenon. This process should be easy, you can evluate the failure case of benchmark test and explore the percentage of cases where thery fails due to the inconsistency.
Questions for authors
  1. The models are too old. Why this paper will choose Qwen2.5 instead of qwen3/qwen3.5.

  2. This paper proposes that R_{think} and R_{con} can serve as great auxiliary reward for VIG. I am wondering what would happen if we do not train on IoU and compared to non-RL/pre-trained VLM.

5 comments

W1: Systematic evaluation of thinking–answer inconsistency

We thank the reviewer for finding thinking drift interesting and for suggesting this analysis. The concern is whether thinking–answer inconsistency is a systematic cause of failure on the benchmark, beyond the single statistic in Appendix C. It is: under IoU-only RL, the box and the reasoning disagree on the target object in 12–15% of test outputs, more often than the reasoning misreads the intention on the Context split, though not on the harder Uncommon split.

Setup. We classify the test outputs of IoU-only RL on Qwen2.5-VL-3B and Qwen3-VL-4B (see Q1) by whether the box is correct (IoU ≥ 0.5) and which object the <think> text settles on. Unlike Appendix C, which uses the predicted label as a proxy, this object is read from the reasoning itself by three LLM judges from different model families (Qwen3-VL-32B, Nemotron-3.5-30B and DeepSeek-V4.1-Flash). The judges see only the reasoning text, and we take their majority vote. As a check, a reasoning paired with another sample's label is judged inconsistent in 98–99% of cases.

Two of the six categories below are thinking–answer inconsistent (†): drift (box correct, reasoning names another object) and off-target (reasoning names the ground-truth object, box does not overlap it). The breakdown shows that inconsistent outputs are comparable in frequency to intention errors, and that most of them are visible failures rather than hidden successes (% of outputs, Context / Uncommon).

IoU-only RLCorrect and consistentDrift†Off-target† (IoU = 0)Near miss (reasoning correct, 0 < IoU < 0.5)Intention error (reasoning and box wrong)No single object named
Qwen2.5-VL-3B57.0 / 50.32.0 / 4.413.4 / 9.69.4 / 7.05.4 / 17.112.8 / 11.6
Qwen3-VL-4B62.8 / 52.92.5 / 2.49.9 / 9.68.1 / 6.211.2 / 22.45.5 / 6.7
  • Inconsistency is a frequent failure. Drift and off-target together cover 12.0–15.4% of test outputs. On Context, they outnumber intention errors (15.4% vs. 5.4% on Qwen2.5-VL-3B; 12.4% vs. 11.2% on Qwen3-VL-4B). On Uncommon, where intentions are atypical, intention errors are the largest failure (17.1% and 22.4%).
  • Many wrong boxes follow correct reasoning. Among wrong boxes whose reasoning names an object, the reasoning had already named the correct object in 62–81% of cases on Context and 41–49% on Uncommon. In more than half of these, the box does not overlap that object at all.
  • [email protected] hides part of it. Drift outputs, 2.0–4.4% of the total (3–8% of correct boxes whose reasoning names an object), count as successes under [email protected].

How Rita changes these categories, together with the label-based drift rate of Table 5, is reported in our reply to Reviewer zKeo (Q1).

We will add this analysis as a new subsection of Section 4, with the judge prompts, the validation check and the per-sample judgements in the appendix.

Q1: Why Qwen2.5-VL instead of Qwen3-VL / Qwen3.5

We thank the reviewer for this question. The concern is why we use Qwen2.5-VL rather than Qwen3-VL or Qwen3.5, and whether Rita still helps on these newer models. We developed Rita on VLM-R1, an RL training codebase that does not support Qwen3-VL or Qwen3.5; re-implemented in two other frameworks, Rita again improves over naive GRPO on Qwen3-VL-4B and Qwen3.5-122B-A10B on every metric, although the grounding gain is small at 122B.

Why Qwen2.5-VL. VLM-R1, the codebase on which we developed Rita, supports training Qwen2.5-VL but not Qwen3-VL or Qwen3.5, so our original experiments used Qwen2.5-VL only. For this response, we re-implemented Rita in EasyR1 (Qwen3-VL-4B) and verl (Qwen3.5-122B-A10B), with the same reward definitions and RL training data.

Rita on Qwen3-VL and Qwen3.5. The table shows that Rita outperforms naive GRPO (IoU and format rewards only) on both newer backbones, with both methods starting from the same model and using the same RL data (Context / Uncommon):

BackboneRL reward[email protected]mIoUAcc_thinkAcc_align
Qwen3-VL-4Bnaive GRPO67.9 / 58.30.627 / 0.54575.5 / 62.459.8 / 49.8
Rita70.0 / 61.00.646 / 0.56877.8 / 65.262.4 / 52.7
Qwen3.5-122B-A10Bnaive GRPO77.4 / 72.90.720 / 0.68189.8 / 82.272.7 / 67.8
Rita77.8 / 73.60.723 / 0.68791.5 / 84.474.2 / 69.6
  • The gains carry over to Qwen3-VL. On Qwen3-VL-4B, Rita leads by 2.1 / 2.7 points of [email protected] and by 2.3–2.9 points on both reasoning metrics, close to its reasoning gains on Qwen2.5-VL-3B (2.1–3.5 points, Table 2).
  • At 122B, the gain is mainly in reasoning. Rita improves Acc_think by 1.7 / 2.2 and Acc_align by 1.5 / 1.8 points, but [email protected] by only 0.4 / 0.7, as naive GRPO already reaches 77.4 / 72.9.

We will add these results to Section 4.

Q2: Training without the IoU reward

We thank the reviewer for this question. The concern is whether R_think and R_cons work only as auxiliaries to the IoU reward, and how a model trained without the IoU reward compares with the non-RL and pre-trained models. R_think is a grounding signal on its own: trained without the IoU reward, the model improves over the SFT model by 3.4 / 2.8 [email protected] and is within 0.5 points of full Rita on every metric, although R_think still uses the ground-truth box as its reference answer.

Setup. We train GRPO with R_fmt + R_think only and re-run full Rita alongside it, both from the same SFT model with otherwise identical settings. All rows come from this consistent re-run and evaluation, so they differ from the corresponding rows of Table 1. Removing R_IoU also removes R_cons, which is defined through the IoU of the predicted box (Eq. 5). We also evaluate the pre-trained Qwen2.5-VL-3B-Instruct without any training; note that the "Qwen2.5-VL-3B-Instruct" rows in Table 1 are SFT models.

The table shows that RL without the IoU reward improves clearly over both the pre-trained and the SFT model, and nearly matches full Rita (Qwen2.5-VL-3B, Context / Uncommon; the pre-trained model is evaluated on grounding only):

ModelRL rewards[email protected]mIoUAcc_thinkAcc_align
Pre-trained (zero-shot)–30.9 / 20.10.288 / 0.193––
SFT (no RL)–63.4 / 58.90.589 / 0.54681.5 / 71.258.5 / 52.6
RL without IoUR_fmt + R_think66.8 / 61.70.617 / 0.57085.8 / 74.862.4 / 56.0
RL, full RitaR_IoU + R_fmt + R_think + R_cons67.2 / 62.00.622 / 0.57385.3 / 74.562.6 / 56.1
  • R_think alone improves grounding and reasoning. Without the IoU reward, RL improves the SFT model by 3.4 / 2.8 points of [email protected] and 4.3 / 3.6 points of Acc_think. It is within 0.5 points of full Rita on every metric, and both are far above the pre-trained model (30.9 / 20.1 [email protected]).
  • Why grounding improves without the IoU reward. R_think is the likelihood of the ground-truth box tokens given the model's own reasoning, so it rewards reasoning under which the correct box is likely, and the generated box follows that reasoning. Within the same prompt, rollouts whose reasoning names the correct object receive a higher R_think in 68% of groups, and the accuracy of sampled training boxes, which is logged but not rewarded, rises from 60.5% to 67.1% [email protected]. Acc_think improves as well, although no object label is rewarded: the reference answer scored by R_think contains only the box coordinates.
  • The reasoning stays natural language. Without the IoU reward, the model could in principle game the likelihood reward, for example by reducing its reasoning to box coordinates. No test output and at most 0.1% of training rollouts have a reasoning trace that consists only of box coordinates.
  • What the IoU reward adds. It makes sampled training boxes more accurate (81% vs. 67% [email protected] at the end of training), but greedy test accuracy is nearly the same with and without it.

We will add this experiment to the ablation section and the pre-trained model to Table 1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Discussion

We thank the reviewers and the AC for their time. As the discussion period approaches its end, we summarize the main concerns, the evidence added in our replies, and the changes we will make.

Strengths recognized by the reviewers. All three reviewers recognize thinking drift: "a neat and interesting finding" (xkeY), "intuitively sound as a key issue of the thinking process" (SmsG), and "a concrete, actionable insight" (zKeo). They describe the method as "neat and reasonable" (xkeY) and "an intuitive approach ... without requiring human-annotation" (SmsG), and the consistency reward as "a natural coupling mechanism" (zKeo).

Main concerns and new evidence

1. Generality across model lines and scale (xkeY Q1, SmsG W4, zKeo W1). We re-implemented Rita in two other RL frameworks and trained it on Qwen3-VL-4B, the next model of the Qwen-VL line, and on Qwen3.5-122B-A10B, a natively multimodal mixture-of-experts model with a different architecture. The table shows that Rita outperforms naive GRPO on both backbones, from the same starting model and RL data (gain over naive GRPO, Context / Uncommon):

Rita − naive GRPO[email protected]Acc_thinkAcc_align
Qwen3-VL-4B+2.1 / +2.7+2.3 / +2.8+2.6 / +2.9
Qwen3.5-122B-A10B+0.4 / +0.7+1.7 / +2.2+1.5 / +1.8

At 122B, where the model already reasons well, naive GRPO increases thinking drift, and Rita brings it below its level before RL. Trained on EgoIntention only, Rita is also ahead of naive GRPO zero-shot on two third-person benchmarks, RefCOCOg and LISA-Grounding.

2. Evaluating the reasoning itself (SmsG W1, zKeo W2). Three LLM judges from different model families read only the reasoning text. Rita's reasoning names the correct object more often than that of IoU-only RL (73.4% → 75.0% on Qwen2.5-VL-3B, 74.7% → 77.5% on Qwen3-VL-4B), contradicts the model's own label less often, and is rated more relevant to the person's need.

3. How often drift occurs, and whether Rita reduces it (xkeY W1, zKeo Q1). Under IoU-only RL, the box and the reasoning disagree on the target object in 12.0–15.4% of test outputs. Rita removes 7.6–24.2% of the drifting cases on three backbones, with no supervision beyond the ground-truth boxes.

4. Relation to VeriFree and RLPR (zKeo follow-up). Rita shares one ingredient with these works, the likelihood of the reference answer after the model's reasoning, and the submission cites and discusses both (Section 2). It differs in three ways:

  • A different setting. VeriFree and RLPR target tasks without a verifier and use the likelihood in place of one. Grounding has a verifier, IoU, but it checks only the answer.
  • A different role for the likelihood. Rita keeps IoU as the reward on the answer and uses the likelihood to supervise the reasoning, which IoU never sees. Judged from the text alone, the reasoning itself improves (point 2); neither work evaluates the reasoning.
  • Coupling with the model's own answer. Neither work scores the rollout's own answer. R_cons rewards reasoning that supports the reference only when the model's own box overlaps the target, and penalizes it when the box misses, so it acts directly on drifting rollouts. It raises Acc_think from 85.1 / 73.7 to 86.6 / 75.0 (Table 2), and it has no counterpart in their setting, since it requires a verifiable answer.

For the same reason, an evaluation on short-answer VQA would test the part Rita shares with these works, not what it adds: R_cons is defined through the IoU of the predicted box, and only the reference-likelihood reward would carry over.

5. Reward-design ablations (SmsG W2–W3, zKeo Q2, xkeY Q2). Of three forms of R_think we trained, ours, the exponentiated average log-likelihood, gives the highest reasoning accuracy and keeps the reward in [0, 1], which R_cons requires. The gain comes from the box coordinates alone, as the reference answer contains no label. Trained with R_think alone, without the IoU reward, the model improves [email protected] over the SFT model by 3.4 / 2.8 points and is within 0.5 points of full Rita.

Reviewer SmsG confirmed that W2–W5 are addressed, and Reviewer zKeo thanked us for "demonstrating the effectiveness of the method across different model lines and the correctness of the thinking stage".

Changes in the revision. We will add the Qwen3-VL-4B and Qwen3.5-122B-A10B results, the two third-person benchmarks, the reasoning-text evaluation (with judge prompts, validation checks and per-sample judgements), the drift breakdown and the reward-design ablations, and make the other changes committed in our individual replies.

We hope this summary helps the AC weigh the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.