acceptodds
Withdrawn from ICLR 2027

VISER: Learning to See What Matters for Efficient Visual Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent “Thinking with Images” methods have improved visual reasoning by explicitly inspecting image regions or reasoning with latent visual representations. However, many of these methods introduce additional visual interactions or modify the base model's inference procedure. How can models learn to better attend to and use visual evidence while maintaining efficient inference? We propose VISER, a two-stage post-training framework that combines visual-token evidence supervision with reinforcement learning to guide evidence use in answer generation. The first stage jointly optimizes evidence prediction at visual-token positions and answer generation. The second stage derives contrastive rewards from the model's responses to original, evidence-corrupted, and background-perturbed images, combining them with task-correctness rewards for reinforcement learning. These rewards encourage reliance on visual evidence and robustness to irrelevant background perturbations. All additional supervision is confined to training; VISER preserves the base model's inference procedure without additional region cropping, visual tool calls, latent visual reasoning tokens, or multi-turn visual interactions. Experiments demonstrate that VISER achieves the best overall performance among the compared methods across five image VQA benchmarks while maintaining inference latency close to that of the base model. Under the same evaluation setup on the full VStar benchmark, VISER incurs approximately 4% latency overhead over the base model and achieves approximately 2.4×, 58×, and 9.3× speedups over TreeVGR, DeepEyes, and DeepEyesV2, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Visual Reasoning, Evidence use, Efficient Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jFjWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a two-stage post-training framework to improve visual reasoning with efficient inference. In stage I, it uses supervised fine-tuning with region-level evidence supervision at visual-token positions. In stage II, it conducts reinforcement learning combining task rewards with contrastive rewards derived from original images and additional perturbed images. Experiments show that VISER outperforms the compared methods on five image VQA benchmarks, with inference latency close to the base model on VStar. Analyses show that after stage II training, it yields fewer errors under background perturbations but becomes less sensitive to visual evidence corruption.

Critical strengths
  1. The paper addresses an important practical problem, which is improving visual reasoning without substantially increasing inference cost.
  2. The empirical evaluation is comprehensive, as it includes comparisons with multiple baselines and ablations of the training stages and reward components. These experiments clarify the contribution of the main design choices.
Critical weaknesses
  1. The incremental benefit of the contrastive rewards is modest. The full reward improves the five-benchmark average by 0.63 points over task-reward-only RL, while individual reward components even regress on some benchmarks. Moreover, the intervention analysis doesn't show increased sensitivity to evidence corruption after stage II (the evidence-corruption CDR decreases from 27.11% to 25.56%).
Questions for authors
  1. Have the authors checked TreeVGR-RL-37K for overlap with the VStar test set? A public report has raised a possible overlap here. If it exists, could the authors provide decontaminated VStar results?
  2. The latency comparison is reported only on VStar. Since inference efficiency is vital for the paper, could the authors report corresponding latency results on the remaining image VQA and high-resolution benchmarks?
  3. Could the authors provide an analysis of representative failure cases? It may help understand when evidence-focused post-training is beneficial.
Reviewer_p1viOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose VISER, a two-stage post-training framework that guides VLMs to answer questions using visual evidence. In the first stage, an evidence prediction objective is applied on top of the standard SFT loss, where the model is asked to predict whether each visual token lies within the annotated evidence region. In the subsequent RL stage, an evidence-corruption reward and a triplet-ranking reward are introduced to encourage the model to rely on visual evidence and remain robust to irrelevant background perturbations. Experiments on five benchmarks demonstrate that VISER outperforms previous training-based methods while keeping the inference cost almost the same as that of the base model.

Critical strengths
  1. The paper is well-written and easy to follow.

  2. The motivation is intuitive, and the proposed method is straightforward.

  3. The experiments and ablation studies are comprehensive, making the conclusions more convincing. The paper clearly ablates the stage composition, the Stage-I objective, and the individual Stage-II reward terms. VISER improves over the base model on all five benchmarks at both the 7B and 3B scales.

  4. The discussion of limitations is honest. The authors acknowledge that VISER still lags behind methods with explicit region inspection on high-resolution benchmarks and that the framework depends on evidence-box annotations.

Critical weaknesses
  1. Since pcorp^{cor} and ppertp^{pert} are shared at the question level, the triplet-ranking reward is ultimately a rescaling of the correctness reward. This is inconsistent with the statement, “To account for response stability under background changes, we penalize deviations from three desired relations using hinge terms,” since it does not directly optimize the margins within the triplet, and no individual response is favored for satisfying the triplet relations.

  2. VISER requires additional annotation and training, although there is almost no overhead during inference. For comprehensiveness, training-free methods such as DyFO [1] and ZoomEye [2] should also be compared and discussed, given that they require no training or annotation overhead.

  3. The paper lacks discussion of and comparison with prior works that share the same spirit, such as Look-back [3], which also encourages the model to implicitly refocus on the image, and PAPO [4], which maximizes, for each rollout, the KL divergence between the policy conditioned on the original image and that conditioned on the masked image, without requiring evidence annotations.

  4. In Table 4, the margins between ablations are small on several benchmarks, e.g., Task reward only vs. Task + ranking. Without reporting the standard deviation across multiple runs, it is unclear whether the improvements are statistically significant.

[1] Li, G., Xu, J., Zhao, Y. and Peng, Y., 2025, June. Dyfo: A training-free dynamic focus visual search for enhancing lmms in fine-grained visual understanding. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 9098-9108). IEEE.

[2] Shen, H., Zhao, K., Zhao, T., Xu, R., Zhang, Z., Zhu, M. and Yin, J., 2025, November. Zoomeye: Enhancing multimodal llms with human-like zooming capabilities through tree-based image exploration. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (pp. 6613-6629).

[3] Yang, S., Niu, Y., Liu, Y., Ye, Y., Lin, B. and Yuan, L., 2026, March. Look-back: Implicit visual re-focusing in mllm reasoning. In Proceedings of the AAAI conference on artificial intelligence (Vol. 40, No. 14, pp. 11694-11702).

[4] Wang, Z., Guo, X., Stoica, S., Xu, H., Wang, H., Ha, H., Chen, X., Chen, Y., Yan, M., Huang, F. and Ji, H., 2026, April. Perception-aware policy optimization for multimodal reasoning. In International Conference on Learning Representations (Vol. 2026, pp. 80502-80530).

Questions for authors
  1. In Figure 3(c), the attention maps seem to be almost the same across the base model, Stage I, and VISER. The base model also appears to attend to the correct region. How should readers interpret this figure?

  2. Does the proposed method generalize to more recent models such as Qwen3-VL/Qwen3.5?

Reviewer_tNtEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VISER, a two-stage post-training framework for visual reasoning. The first stage adds a linear evidence prediction head on the decoder hidden state of visual tokens, converting region annotations into binary classification targets and jointly training with answer generation. The second stage uses the answer results from the original image, damaged evidence regions, and background perturbation images to construct additional rewards and perform GRPO optimization. Both auxiliary supervision and image intervention are limited to the training stage, while the standard generation process of the base model is retained during inference. The paper reports performance improvements on multiple visual question-answering benchmarks, and provides efficiency comparisons, component ablations, and visual evidence usage analyses.

Critical strengths

The problem has practical value. Improving the use of visual evidence without increasing the invocation of reasoning tools, pruning, or additional latent tokens is a meaningful goal. Table 2 reports the near-base inference latency, which demonstrates the deployment value of this solution.

It provides useful component comparisons. Table 4 compares answer-only and evidence-grounded SFT under the same training samples and answer targets, and compares different RL rewards from the same Stage-I checkpoint. The background views and annotation perturbation experiments in the appendix also help to understand the solution.

Critical weaknesses

1. The related work coverage is too thin for a paper positioning itself against this active research area.

Section 4 is only one short paragraph, which is unusual for a submission that explicitly situates itself at the center of the current "thinking with images" landscape. Several directly relevant lines of work are not discussed in the main text, including VGR, iVGR, Evidence-RL, and DVRP. Some of these are mentioned briefly in the appendix, but the main-text framing gives the impression that the authors have not fully mapped the design space they are claiming to improve. For a paper whose core claim is "we do things differently from prior methods," a more thorough positioning against the most closely related prior work is needed in the main body, not buried in the appendix.

2. Building on the above, the incremental novelty of VISER over prior work is not clearly established.

Stage I's evidence-classification head is an incremental extension of existing auxiliary grounding supervision that VGR, TreeVGR, and iVGR have already explored in various forms. Stage II also has substantial overlap with Evidence-RL, which integrates counterfactual evidence-dependence signals into GRPO without requiring question-specific evidence annotations, and with DVRP, which already uses original, masked, and noise-perturbed views to encourage both visual sensitivity and robustness. The methods differ in details of supervision targets and optimization, but the manuscript does not articulate precisely what VISER contributes that these precedents do not already provide. The combination of two well-studied components is practical, but the conceptual contribution over prior work appears modest given how close the existing alternatives already are.

3. The specific contribution of evidence-aware SFT is not fully isolated. Table 4 shows that Stage II alone reaches 88.48% on VStar, compared with 89.53% for the full system, suggesting that RL accounts for most of the improvement on this benchmark. Stage I clearly helps on other benchmarks, especially BLINK, so the question is not whether it contributes at all—it does. Rather, what remains unclear is whether its benefit comes specifically from the proposed evidence objective, or from additional supervised training plus a stronger RL initialization in general. The answer-only SFT comparison is reported only before RL, and the paper does not run answer-only SFT through the same Stage-II training pipeline. As a result, the final numbers do not cleanly separate the value of evidence supervision from the value of extra SFT plus a better starting point for RL.

4. The reward design could use more justification and mechanistic validation. The objective combines correctness, evidence-corruption feedback, likelihood-based weighting, and three hinge penalties with fixed weights and margins. The component ablations do show incremental benefits from each piece, but the rationale for this particular combination—and how sensitive the results are to the chosen hyperparameters—remains unclear. The average gain over task-only RL is about 0.62pp, and without reported run-to-run variability it is hard to judge how robust this improvement really is. There is also a subtle interpretation issue: errors after evidence corruption can reflect both genuine evidence dependence and sensitivity to the intervention itself as an artifact. The area-matched background controls are a good step, but they do not fully demonstrate that the reward specifically strengthens reliance on task-relevant evidence, as opposed to mainly making the model more robust to the particular perturbations being tested.

5. The comparison and reward-verification setup leaves several open questions. BLINK results are missing for Pixel-Reasoner, DeepEyes, and DeepEyesV2 because their pipelines require adaptation to multi-image inputs. This limits how complete the cross-benchmark comparison looks, though it does not by itself mean the evaluation is unfair. The manuscript also does not spell out differences across methods in training data, compute budget, and tuning effort, which makes it harder to attribute the reported gains specifically to the proposed algorithm. Separately, Qwen3-1.7B provides correctness labels for both the original and intervened branches, but its agreement with human judgments is not reported. Since these labels feed directly into the auxiliary rewards, their reliability matters for whether the training signal is doing what the authors claim.

Questions for authors

1 If evidence is destroyed and the answer is incorrect, it does not necessarily mean that the model originally relied on that evidence.

A decrease in accuracy can also result from abnormal image artifacts or damage to surrounding information during the destruction process. For example, when the tag is blurred, the outline of the dog's neck is also destroyed, and the model may answer incorrectly due to an abnormal local scene. Conversely, answering correctly after destruction does not necessarily mean ignoring the evidence: the shape of the tag may still be recognizable, or other areas provide redundant clues. Therefore, pcorp_{\mathrm{cor}} is not a single, unambiguous indicator of evidence dependence.

2 The analysis of evidence usage is not sufficient to prove that the model is more dependent on visual evidence.

In Table 5, the area matching background destruction CDR of VISER decreased from 6.49% to 0.68%, while the evidence destruction CDR also decreased from 32.86% to 25.56%. Therefore, the increase in SpecRatio from 5.05 to 39.21 mainly reflects the decrease in background sensitivity, and cannot be directly interpreted as being more sensitive to evidence destruction. Additionally, the CDR of each model is based on the set of samples that the model answered correctly based on its original image. Cross-model comparison is not strictly paired; the test perturbation and training perturbation types are highly overlapping, and blurring or pixelation may not fully remove valid evidence. The existing results support the model's stronger robustness to the measured background perturbations, but the mechanism of "more dependent on visual evidence" still requires supplementary verification. It is recommended to conduct paired analysis on samples with correct answers in common, introduce interventions unseen during training, and test whether the model can adjust its answer accordingly when replacing key evidence and causing the correct answer to change.

3 The evidence position and destruction intensity in the training intervention are confounded.

During training, the evidence area is strongly blurred or pixelated, while the background is only slightly blurred. For example, a strongly pixelated tag leads to an incorrect answer, while a slightly blurred grassland does not affect the answer. This could either indicate that the tag is more important, or it could be partially due to the stronger destruction. The paper has included area matching and comparison based on the same destruction settings in the evaluation, which helps alleviate this doubt; however, the training rewards still use asymmetric intervention intensities. It is recommended to introduce an ablation study of training interventions with intensity and area matching to test whether the benefits are indeed from the evidence position rather than the difference in destruction intensity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.