- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the "thinking drift" phenomenon that emerges when vision-language models (VLMs) are trained with reinforcement learning for Visual Intent Grounding (VIG) tasks. VIG requires models to infer target objects from implicit human intent queries and predict their bounding boxes. The core finding is that RL training with IoU-only rewards (e.g., VLM-R1, UniVG-R1) leads to reward hacking where "bounding boxes are correct but the reasoning chain points to the wrong target." To address this, the authors propose Rita (ReInforcing Thinking-Answer consistency), which adds two annotation-free rewards to standard GRPO training: a thinking reward (Equation 3) that replaces the model's sampled answer with the reference answer and measures the average probability of reference answer tokens conditioned on the sampled reasoning chain; and a consistency reward (Equation 5) that multiplicatively couples the thinking score with the IoU score. Rita also employs a difficulty-aware data filtering strategy, using only 5% of the SFT data. Experiments are conducted on EgoIntention and the newly constructed RefEgo-Int dataset using Qwen2.5-VL (3B and 7B). Rita achieves +4.8/+2.1 [email protected] improvement over the SFT baseline on EgoIntention, and +10.2/+7.5 zero-shot improvement on RefEgo-Int (Table 1, Figure 5).
Critical strengths
- The thinking drift phenomenon is clearly defined, and validated through statistical significance testing (McNemar's test p=3.7e-7, Appendix C). The finding that 15.8% of newly corrected bounding boxes under IoU-only RL exhibit reasoning-answer inconsistency (Table 5) is a concrete, actionable insight that is in principle applicable to any structured prediction task where reasoning is unconstrained under outcome-reward RL.
- The paper shows that the core insight of using the likelihood of the reference answer conditioned on the sampled reasoning chain (Equation 3), rather than requiring annotated reasoning chains. The ablation in Table 7 validates the necessity of the reference answer override. The multiplicative consistency reward (Equation 5) provides a natural coupling mechanism.
Critical weaknesses
- All evaluations cover only a single task (VIG), one model family (Qwen2.5-VL 3B/7B), and two closely related egocentric datasets (EgoIntention and RefEgo-Int, both derived from Ego4D). Moreover, the core algorithmic contribution is limited to modifying the reward design, and no experiments are provided on other benchmarks or VLM architectures. For a top venue like ICLR2027, I consider both the novelty and generalizability to be seriously insufficient.
- The core claim is that Rita improves thinking-answer consistency, yet the only reasoning metric (Acc_think) merely checks whether the output label string matches the ground-truth. This does not measure whether the reasoning chain itself is coherent and relevant. Without direct reasoning evaluation, the paper cannot distinguish between Rita produces better reasoning and Rita produces better category labels.
Questions for authors
- Could the authors report Rita's own thinking drift rate (analogous to Table 5)? The paper quantifies drift under IoU-only RL but never presents the corresponding data for Rita, leaving the core claim that Rita reduces drift without direct verification.
- The thinking reward (Equation 3) averages token probabilities rather than log-probabilities. Was log-probability considered? Averaging raw probabilities is dominated by high-confidence tokens and may fail to distinguish reasoning chains that differ on a few semantically critical tokens.
6 comments
W2: Does Rita improve the reasoning itself, or only the labels?
We thank the reviewer for this comment. The concern is whether Acc_think, which checks only the predicted label, can tell better reasoning from better labels. We now evaluate the reasoning text directly: read without the label, Rita's reasoning names the correct object more often and is rated more relevant to the person's need, so its gain is not confined to the label; these judgements cover the reasoning's conclusion and its relevance, not each step.
Reading the reasoning without the label. Three LLM judges (Qwen3-VL-32B, Nemotron-3.5-30B and DeepSeek-V4.1-Flash; protocol in our reply to Reviewer xkeY, W1) read only the <think> text and name the object it settles on. The table shows that Rita's reasoning identifies the correct object more often, and contradicts the model's own label less often, than that of IoU-only RL (% of test outputs, averaged over the two splits):
| Model | Reasoning names the ground-truth object ↑ | Reasoning contradicts the model's own label ↓ |
|---|---|---|
| Qwen2.5-VL-3B, IoU-only RL | 73.4 | 6.0 |
| Qwen2.5-VL-3B, Rita | 75.0 | 5.1 |
| Qwen3-VL-4B, IoU-only RL | 74.7 | 7.8 |
| Qwen3-VL-4B, Rita | 77.5 | 4.5 |
- The gain appears in the reasoning itself. Rita's reasoning names the correct object 1.6 / 2.8 points more often. This measure never reads the label, so the gain cannot come from better labels alone.
- Rita tightens the link between reasoning and label. Contradictions fall from 6.0% to 5.1% and from 7.8% to 4.5%, i.e., Rita removes 15.0% and 42.3% of them. If Rita improved only the labels, they would rise instead.
- Acc_think is a fair proxy for both methods. For every model, IoU-only RL included, the reasoning and the predicted label refer to the same object in 88–96% of outputs, so comparing Acc_think between methods does not favour Rita.
Relevance and coherence. In a second pass, the same three judges read only the person's stated need and the reasoning, with no image, label or ground truth, and rate the reasoning's relevance to the need and its internal coherence from 1 to 5; we use the median of their three ratings. The judges agree more on relevance than on coherence, so we treat relevance as the main measure. Relevance responds to content: pairing a reasoning with another sample's need lowers its relevance from 3.87 to 1.60, and reasoning that names the correct object is rated more relevant than reasoning that names another (3.95–4.16 vs. 2.98–3.52). The table shows that R_think makes the reasoning more relevant on the same RL data, and that Rita raises relevance further (Qwen2.5-VL-3B, where all models write reasoning of similar length, 31–33 words):
| Qwen2.5-VL-3B | IoU-only RL | IoU + R_think (same RL data) | Rita |
|---|---|---|---|
| Mean relevance (1–5) | 3.81 | 3.87 | 3.89 |
| Mean coherence (1–5) | 4.17 | 4.17 | 4.22 |
| Both ≥ 4, all outputs (%) | 68.1 | 70.0 | 70.5 |
| Both ≥ 4, Uncommon (%) | 50.2 | 54.4 | 54.8 |
On Qwen3-VL-4B, Rita's reasoning is also rated higher (both ≥ 4: 67.2% → 73.8%), but it is much shorter (62 vs. 90 words) and the judges favour shorter text, so we do not count this as evidence.
Scope. These judgements check the object the reasoning settles on and its relevance and coherence as text, not each step against the image; step-level grading would require reference reasoning, which EgoIntention does not provide and which would not be unique.
We will add both evaluations, the judge prompts, the agreement between judges and the per-sample judgements to the appendix, and revise Section 4.1 to state that Acc_think measures the target of the reasoning.
Q1: Rita's own thinking-drift rate
We thank the reviewer for this request; this comparison should have been in the paper. The concern is whether Rita actually reduces thinking drift when measured as in Table 5. It does: on all three backbones we trained, Rita removes 7.6–24.2% of the drifting cases left by IoU-only RL without any extra annotation, and on the two newer backbones it brings drift below its level before RL. It reduces drift rather than eliminating it, and drift read directly from the reasoning text is rare for both methods and changes little.
Drift rate as in Table 5. As in Table 5, we measure drift as the percentage of samples with a wrong predicted category among all samples whose box is correct (IoU ≥ 0.5), over both splits. An ideal model has zero drift: every correct box should come with a category that names the boxed object. Table 5 analyzes UniVG-R1, whose RL starts from a chain-of-thought cold start; here, IoU-only RL and Rita start from the same fine-tuned model of each backbone, so we compare the two RL methods with each other rather than with the numbers of Table 5.
- Drift is substantial under IoU-only RL. After IoU-only RL, 9.9% of correct boxes come with a wrong category on Qwen2.5-VL-3B (up from 5.8% before RL), 13.2% on Qwen3-VL-4B and 6.6% on Qwen3.5-122B-A10B (up from 5.6%). [email protected] counts all of them as successes.
- Rita removes 7.6–24.2% of these drifting cases without extra annotation. It lowers drift to 9.0% on Qwen2.5-VL-3B, 12.2% on Qwen3-VL-4B and 5.0% on Qwen3.5-122B-A10B, a relative reduction of 9.1%, 7.6% and 24.2% from the IoU-only level, using only the box annotations that IoU-only RL already uses. On the two newer backbones, drift ends below its level before RL.
- Drift is not eliminated. On Qwen2.5-VL-3B, Rita removes only part of the increase caused by RL: among the boxes that RL newly corrects, 11.5% come with a wrong category under Rita, against 12.5% under IoU-only RL.
Drift read from the reasoning text. We also applied the reasoning-text classification of our reply to Reviewer xkeY (W1) to Rita. The table shows that Rita produces more outputs whose box and reasoning are both correct, while the two inconsistent categories change little (% of outputs, Context / Uncommon):
| Model | Correct and consistent ↑ | Drift: box correct, reasoning names another object ↓ | Off-target: reasoning correct, box misses it ↓ |
|---|---|---|---|
| Qwen2.5-VL-3B, IoU-only RL | 57.0 / 50.3 | 2.0 / 4.4 | 13.4 / 9.6 |
| Qwen2.5-VL-3B, Rita | 57.8 / 52.3 | 1.8 / 4.4 | 12.4 / 10.6 |
| Qwen3-VL-4B, IoU-only RL | 62.8 / 52.9 | 2.5 / 2.4 | 9.9 / 9.6 |
| Qwen3-VL-4B, Rita | 65.0 / 55.5 | 2.4 / 3.0 | 10.5 / 9.7 |
- More outputs are correct and consistent. Averaged over the two splits, they rise from 53.7% to 55.0% on Qwen2.5-VL-3B and from 57.8% to 60.3% on Qwen3-VL-4B.
W1: Scope, generality and contribution
We thank the reviewer for recognizing thinking drift as an insight that applies beyond VIG. The concern is whether the evidence is too narrow (one task, one model family, two Ego4D-derived datasets) and whether a change confined to the reward is a sufficient contribution. We focus on VIG by design, because this is where thinking drift arises and matters most; within this scope, Rita's gains hold across two model lines with different architectures, from 3B dense vision-language models to a 122B natively multimodal mixture-of-experts model, and on two third-person grounding benchmarks it never saw, Rita stays within 0.8 points of naive GRPO and improves over the model before RL on RefCOCOg.
Why we focus on VIG. Thinking drift was first observed in VIG, and VIG is where it matters most: the target must be inferred from an implicit intention, so the reasoning determines the answer rather than restating the query. The paper therefore studies one task in depth, from the diagnosis of drift (Appendix C) to a reward that counters it, on EgoIntention and on RefEgo-Int, which adds camera motion, no-target queries and long-tail categories (Appendix A).
Across model lines, architectures and scales. Beyond Qwen2.5-VL-3B/7B, we trained Rita on Qwen3-VL-4B, the next model of the Qwen-VL line, and on Qwen3.5-122B-A10B, which differs from the Qwen-VL models in architecture; we re-implemented Rita for both in two other RL frameworks. The table shows that, from the same starting model and RL data, Rita outperforms naive GRPO on both backbones (Context / Uncommon):
Rita improves Acc_think by 1.7–2.8 points and [email protected] by 0.4–2.7 points over naive GRPO. At 122B, naive GRPO raises the share of samples whose box is correct but whose label is wrong from 3.9% / 4.5% to 4.8% / 5.1%, and Rita removes 25.0% / 21.6% of these cases (3.6% / 4.0%).
Other grounding benchmarks. We further evaluated the EgoIntention-trained models zero-shot on two third-person benchmarks that differ from EgoIntention in viewpoint and query type: RefCOCOg (referring expressions) and LISA-Grounding (queries that require reasoning). The table shows that RL with Rita transfers to RefCOCOg and that Rita's reasoning rewards do not cost grounding on unseen benchmarks (zero-shot [email protected]; best trained model in bold):
Contribution. The algorithmic change is confined to the reward by design: thinking drift arises because the IoU reward never sees the reasoning, so the reward is where it should be addressed. Our contribution is the diagnosis of thinking drift under IoU-reward RL for visual intention grounding, and a reasoning-label-free reward that counters it. The rebuttal experiments also show that this reward trains grounding without the IoU reward (xkeY, Q2) and improves the reasoning text itself (SmsG, W1), at the cost of one extra forward pass (SmsG, W5).
We will add these results to the paper.