acceptodds
Withdrawn from ICLR 2027

Enhancing Multi-modal Reasoning via Self-Revisiting Image Evidence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robust visual reasoning in Vision–language models (VLMs) often requires re-grounding intermediate steps in the underlying visual evidence. Recent approaches typically rely on external image operations such as zooming or cropping to re-access fine-grained details during inference, which requires additional image re-encoding and can disrupt the reasoning trajectory. We argue that VLMs already provide strong internal signals for identifying and reusing visual evidence, and that these signals can be directly leveraged to support image-grounded reasoning. Motivated by this insight, we propose an end-to-end self-revisit framework, SIEVE, that teaches models to re-engage image evidence through internal representations. SIEVE extracts embeddings of salient image regions and injects them into the reasoning chain when additional grounding is needed, enabling later steps to condition on relevant visual cues without external tool calls or re-encoding. We use reinforcement learning to teach the model when to trigger visual revisiting and which region embeddings to retrieve and insert during the reasoning process. Experiments across multiple visual reasoning benchmarks and evaluations of perception, reasoning, and hallucination show that SIEVE consistently improves performance, achieving an average gain of 8% across settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM Reasoning, thinking with embeddings

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_eyZdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors propose SIEVE, a self-revisit framework that allows VLMs to re-engage with image evidence through their internal representations instead of external tools. Region embeddings are identified via gradient saliency with respect to the question and cross-modal similarity and are then cached. An RL pipeline teaches the model when to insert the cached embeddings into the reasoning trace. Experiments across multiple benchmarks demonstrate improvements with self-revisit-grounded reasoning.

Critical strengths
  1. The motivation is intuitive, and the proposed method is straightforward.

  2. Experiments across benchmarks show the effectiveness of the proposed pipeline compared with naive GRPO.

Critical weaknesses
  1. Comparisons with highly related works, e.g., Qwen Look-again [1], Look-back [2], VisRef [3], etc., are missing. Qwen Look-again and Look-back share the same spirit of using RL to teach the model to revisit image embeddings/ refocus on image during reasoning, while VisRef proposes a training-free method for injecting relevant image embeddings during inference. None of these works rely on calling external tools or performing operations such as zooming. Compared with these related works, the contribution of this paper seems to be overclaimed, and the novelty of the work is limited.

  2. The statements in the paper are inconsistent with the experimental results: i) In the abstract, the authors state that the method achieves “an average gain of 8% across settings,” while in Tables 1 and 2, the margin is much smaller, even when compared with zero-shot performance. ii) “Can disrupt the reasoning trajectory” is stated as a drawback of external tool calling or image re-encoding, but in Figure 12, SIEVE exhibits the same “issue,” with repeated reasoning appearing after the image embeddings are injected.

  3. The claims in the abstract are inconsistent with the method. The abstract states that RL teaches the model “when to trigger visual revisiting and which region embeddings to retrieve and insert.” However, according to Section 3.3, the policy only chooses between answering and “triggering insertion of the cached visual evidence,” while the region itself is predetermined by the non-learned Algorithm 1.

  4. In Table 1, all baselines are training-free methods. Baselines involving training should also be incorporated into Table 1 for comprehensiveness.

  5. The quality of the writing needs to be improved, as there are many minor errors in the manuscript, e.g., i) Figure 5 in L148 should be Figure 2. ii) “Extropy” in L440 should be “entropy.” iii) In Table 1, SIEVE-8B achieves 87.96, while in Table 3, the number is 87.86. iv) In Section 4.3.2, the thinking token is <think>, while <thought> is used in the figures in the appendix.

[1] Chu, X., Chen, X., Wang, G., Tan, Z., Huang, K., Lv, W., Mo, T. and Li, W., 2025. Qwen look again: Guiding vision-language reasoning models to re-attention visual information. arXiv preprint arXiv:2505.23558.

[2] Yang, S., Niu, Y., Liu, Y., Ye, Y., Lin, B. and Yuan, L., 2026, March. Look-back: Implicit visual re-focusing in mllm reasoning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 40, No. 14, pp. 11694-11702).

[3] Ghosal, S.S., Kim, Y., Li, Z., Chaudhry, R., Xu, L., Zhang, H., Zablocki, J., Xing, Y. and Zhang, Q., 2026. VisRef: Visual Refocusing while Thinking Improves Test-Time Scaling in Multi-Modal Large Reasoning Models. arXiv preprint arXiv:2603.00207.

Questions for authors
  1. What is the motivation for building on Qwen3-VL-Instruct rather than Qwen3-VL-Thinking, given that the method primarily aims to insert visual cues into the thinking trace?

  2. What is the main contribution compared with prior work on visual re-injection and re-focusing?

  3. How is the visual evidence obtained at inference time? Is Algorithm 1, including a backward pass, run for every test question? Is the evidence extracted once or recomputed at each insertion? If it is extracted once from the question and reused, what is the advantage over methods that dynamically select visual tokens conditioned on the current reasoning state, e.g., attention-based routing in Qwen-LookAgain [1] or reasoning-state-dependent selection in VisRef [3]?

Reviewer_4aaVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SIEVE, which uses gradient saliency to select textual anchors and cross-modal matching to extract representations of relevant image regions. GRPO trains the model to decide when to insert these cached representations during reasoning. The authors report improvements across multiple perception, reasoning, and hallucination benchmarks.

Critical strengths
  1. Empirical performance gains. The method improves over the base models and the GRPO baselines on most reported evaluations across perception, reasoning, and hallucination benchmarks, supporting the effectiveness of the complete approach.
  2. Practical potential. Reusing existing visual representations avoids re-encoding revisited regions. The reported gains with a relatively small training set suggest a practical way to make better use of visual information already available to the model.
Critical weaknesses
  1. Unclear distinction from prior work. v1 [1] already uses a pointing mechanism to copy visual embeddings into the reasoning sequence, while Argus [2] reuses cached region representations. The paper should clarify its advantages over these mechanisms.
  2. Unclear contribution of gradient saliency. The comparison with randomly sampled embeddings in Section 4.3.1 and Figure 3 supports the usefulness of the selected embeddings, but does not separately establish the contribution of the gradient saliency step.
  3. Insufficient validation of insertion timing. The paper does not directly compare learned insertion timing with inserting evidence once at a fixed position. Although skipping insertion is allowed in Section 3.1, its appropriate use when additional evidence is unnecessary is not experimentally evaluated.
  4. Limited evidence for reasoning continuity. The Introduction argues that external tool use can disrupt reasoning continuity, but the experiments do not directly test whether SIEVE avoids this problem. Higher final-answer accuracy alone does not establish this advantage.

[1] Chung, J., et al. v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning. arXiv:2505.18842, 2025.

[2] Man, Y., et al. Argus: Vision-Centric Reasoning with Grounded Chain-of-Thought. arXiv:2505.23766, 2025.

Questions for authors
  1. Please clarify the differences from v1 and Argus, and substantiate the advantages of SIEVE through experimental comparisons or a detailed analysis.
  2. Please add an ablation comparing gradient saliency with an alternative rule for selecting textual anchors, while keeping the subsequent cross-modal matching procedure unchanged.
  3. Please compare learned insertion timing with inserting evidence once at a fixed position. Also report how often the model skips insertion on questions it can answer correctly without additional evidence.
  4. Please clarify how reasoning continuity is assessed and add a direct comparison with tool-based methods.
Reviewer_BNZ4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SIEVE enables VLMs to revisit image evidence by inserting cached region representations into their reasoning context, without crop-and-re-encode tools. Regions are identified using gradient saliency and cross-modal similarity, and GRPO trains the insertion behavior. Experiments mainly use Qwen3-VL-4B/8B trained on approximately 1,500 COCO-derived examples.

Critical strengths
  1. The proposed intervention is conceptually easy to understand and avoids repeated vision encoding, by supplying the model with what it already had access to.
  2. The reported training setup is small, with less than 2k examples and 60 GRPO steps on two GPUs, which is small cost for training.
  3. The action-reward ablation documents useful failure modes, including empty reasoning tags and reward collapse, which is a good set of informatives.
Critical weaknesses
  1. The setup for the headline metrics that the authors have fails to isolate the contribution of the method that the authors are proposing. Table 1 compares RL-trained SIEVE with training-free inference methods but omits the GRPO-only control, so the V* and HR-Bench gains therefore cannot be separated from ordinary RL gains. In terms of the ablation, they do not fully explain the gains. Specifically, Figure 3 shows sensitivity to selected versus random embeddings, but does not isolate the improvement over vanilla since random insertion runs on the RL-trained model. Random insertion collapses to roughly 25–30%, including below chance on binary spatial questions, which makes me think that generation and formatting failures should be examined. Moreover, there are a number of missing controls, including the same training and multi-turn format with empty or dummy evidence, matched-size random or center regions, and crop-and-re-encode of the region SIEVE selects.
  2. Some results are being exceeded by the author's claims. For example, on 4B V*, ZoomEye is both more accurate than SIEVE and faster. This contradicts blanket claims of superiority and the best efficiency–effectiveness trade-off. Also Zoom methods outperform SIEVE on HR-Bench FSP.
  3. Importantly, the evaluation uncertainty and reproducibility are poorly described. V* has only 191 questions. No paired uncertainty estimates or variation across training runs are reported, and hyperparameters are compared on it without any described separate validation protocol. Also the training-question construction is unspecified.
  4. There are a few inconsistencies between metrics reported by the authors and those in their original publications, which needs some explanation. Specifically, vanilla MathVista and LogicVista scores are below those reported in Qwen3-VL report, and the reproduced DeepEyes performance is also considerably lower than that of the original paper on V*. Perhaps the protocol differences should be explained.
  5. Section 2.2 mentions the latent visual reasoning methods but they were not evaluated. The ViCrop (ICLR 2025) is a close method that should be compared against but was omitted.
  6. Overall I think the biggest weakness is that there's no sufficient evidence demonstrating the gain from proposed methods. Table 1 compares SIEVE only to training-free methods, with no GRPO control, while table 2 has GRPO control with no tool baselines. And table 4 shows results with RL-trained tool baselines, but it is on a different backbone model and only the V* dataset (plus the fact that SIEVE looses to DeepEyes there).
Questions for authors

Other comments:

  1. The embedding reward pays for insertion whenever the answer is correct, whether insertion was necessary or not. Please report insertion rates and paired results showing how often insertion corrects or introduces errors.
  2. The representation and inference procedure are not sufficiently described in the paper. Specifically, it is unclear whether Algorithm 1 runs for each test example, whether its forward computation is reused, and whether its gradient computation and extraction costs are included in Table 3. Also, the abstract claims RL learns which embeddings to retrieve, but the described policy appears to trigger insertion of cached evidence. Cache refresh during training does not clarify whether the policy chooses among regions.

Questions for the authors:

  1. Can you provide GRPO-only results on V*/HR-Bench and a matched empty-evidence control?
  2. Which representations are inserted, with what position IDs, and does the policy select regions or only trigger insertion? 3, What is the complete test-time extraction procedure and cost?
  3. What are the insertion rates and token counts?
  4. How does crop-and-re-encode of the same selected region compare?
  5. How were training questions and hyperparameters chosen, and what explains the reported result discrepancies and baseline reproduction gaps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.