acceptodds
Withdrawn from ICLR 2027

V-ERR: Visual Evidence Re-checking Reinforcement in Multimodal Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Despite their strong perception and generation abilities, vision-language models (VLMs) still struggle with complex reasoning. While GRPO-style post-training reinforcement learning has improved multimodal reasoning performance, it primarily optimizes external outcomes and underutilizes the model’s internal modality signals, especially the visual signals that should be tracked and verified across reasoning steps. To address these challenges, we draw inspiration from cognitive science on multimodal signal processing, and uncover two key phenomena in VLMs: signal partitioning, where modality signals are processed through partitioned pathways, and signal attenuation, where modality-specific signals progressively decays as reasoning unfolds. Building on these insights, we propose V-ERR (**V**isual **E**vidence **R**e-checking **R**einforcement), a post-training framework that reinforces attenuated signals during reasoning. Concretely, we first introduce a partitioning score to identify modality-specialized attention pathways and then incorporate GRPO with modality-specialized reward shaping to reinforce evidence-checking to strengthen the attenuated visual-modality signals. Extensive experiments demonstrate that V-ERR significantly enhances the reasoning capabilities of VLMs, consistently outperforming leading RL optimization strategies by a substantial margin. The code is available at https://anonymous.4open.science/r/V-ERR-BB96.

Primary area:
reinforcement learning
Keywords:
Multimodal, RL, GRPO, Attention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8bKCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper reports two patterns in VLMs: (1) attention to visual tokens is concentrated in a small set of attention heads, and (2) visual attention tends to weaken as reasoning progresses. Based on these observations, the authors propose V-ERR, a RLVR algorithm designed to encourage repeated checks of visual evidence. It first selects visual attention heads and identifies image tokens relevant to the question. During GRPO training, it gives greater weight to response tokens that show locally high attention to the selected evidence. Experiments with LLaVA and Qwen models on 5 multimodal reasoning benchmarks report marginal accuracy gains over GRPO and other baselines.

Critical strengths

1. The proposed method is easy to implement. V-ERR uses the model’s own attention patterns to adjust token-level advantages in GRPO. Its evidence selection is conditioned on the question, and it requires no additional annotations of evidence regions or reasoning steps.

2. Positive results. The main experiments show some improvements on two model families and five benchmarks.

Critical weaknesses

1. Limited support for the main hypothesis. V-ERR assumes that models use less visual evidence as reasoning continues. This claim is partly based on the rise in TFR during generation. However, this is no evidence supporting a more frequent rechecking = better performance. In other worlds, models may still use visual evidence through text positions while paying less direct attention to image positions.

2. Limited analysis of attention changes. V-ERR uses training to help models "refocusing" visual evidence during reasoning. The paper compares attention patterns before and after training: the pattern change is not consistent as shown in Figure 6. At the same time, the paper does not track TFR or evidence-checking measures at different training steps. It is therefore unclear how the intended behavior develops during training and how these changes relate to gains in accuracy.

Questions for authors
  1. Could you provide evidence on more frequent rechecking = better performance?

  2. Could you demonstrate the training dynamics of your proposed metrics in your baselines (e.g. GRPO) and your proposed method? Could you explain why the pattern change is not consistent in Figure 6?

Reviewer_WQPjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors analyze visual-head specialization and declining visual attention during generation, and propose V-ERR to refine token-level GRPO credit. It selects heads with high visual propensity, identifies task-relevant image tokens from prompt-to-image attention, and increases advantage weights at response positions showing locally salient attention to that evidence. The weights are normalized to have a sequence mean of one. Experiments train a separate model on each of five benchmarks using LLaVA-1.5-7B and Qwen2.5-VL-3B. The authors report five-task averages of 36.8 and 50.9, compared with GRPO's 33.8 and 48.3, respectively. Main-text ablations examine head selection ratios, evidence selection ratios, and task-relevance conditioning.

Critical strengths
  1. The reinforcement signal is localized both spatially and temporally. V-ERR selects task-conditioned image evidence before emphasizing response positions with locally elevated attention to that evidence, making its guidance more specific than weighting positions by attention to the image as a whole. The Focus comparison and evidence/head-ratio ablations support the usefulness of this design.

  2. The main experiments report consistent gains across model families and tasks. Relative to GRPO, the five-task average increases from 33.8 to 36.8 for LLaVA and from 48.3 to 50.9 for Qwen2.5-VL. This provides evidence of practical benefit across the evaluated benchmark-specific training setups.

Critical weaknesses
  1. The interpretation of visual attention decline remains ambiguous. Section 3 interprets declining direct attention to the image as weakening of visual evidence. However, the model may already have extracted the needed information into intermediate representations or text. The reported analyses do not show whether greater attenuation is associated with incorrect answers under comparable conditions, leaving the claimed link to reasoning failure insufficiently supported.

  2. The token-weighting rule is not clearly aligned with useful evidence re-checking. In Section 4.2, selected positions can receive stronger negative updates in negative-advantage responses, even if evidence was used correctly before a later reasoning error. The local z-score threshold can also miss sustained high attention. The paper does not directly examine these cases, leaving unclear whether the weighting reliably selects and reinforces useful evidence use.

Questions for authors
  1. For rollouts of the same question, how do late-stage TFR or VSFR differ between correct and incorrect responses at comparable generation progress?

  2. What motivates amplifying negative advantages at selected positions when a response can fail after correctly using visual evidence? Do existing analyses or ablations assess this choice and the exclusion of sustained high-attention spans?

Reviewer_EVqDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

V-ERR is a GRPO-based post-training method that uses internal visual attention to guide token-level credit assignment. It identifies heads with high image attention, selects prompt-relevant image tokens, and amplifies advantages at locally salient visual-attention events. Experiments on five benchmarks report average gains over GRPO of 3.0 percentage points for LLaVA-1.5-7B and 2.6 for Qwen2.5-VL-3B. Ablations examine pathway and evidence selection; appendices add Qwen3-VL-2B, longer completions, attention analyses, and region-relevance comparisons. The approach is interesting, but its effectiveness on stronger contemporary backbones and its claimed verification mechanism remain insufficiently established.

Critical strengths
  • A concrete training idea without additional annotations. Connecting internal visual-attention signals to RL credit assignment provides a practical approach to improving multimodal reasoning.
  • Useful supporting ablations. The task-conditioning comparison, pathway-selection experiments, and longer-completion results provide initial empirical support for the proposed design.
Critical weaknesses
  • The empirical evaluation does not convincingly establish effectiveness on stronger contemporary VLMs. The main study uses LLaVA-1.5-7B and Qwen2.5-VL-3B without explaining the omission of Qwen2.5-VL-7B. Although Qwen3-VL models have been available since 2025, Appendix C.2 only evaluates the 2B model against Rhythm. It also changes the benchmark suite: MMMU is omitted, while AI2D and MathV360K are added. This unexplained change makes it difficult to assess whether the main findings reproduce across backbones. Moreover, gains over the strongest reported Qwen comparator are only 0.4 points on MathVista, 0.3 on MathVision, 0.1 on ScienceQA, and 0.2 on MMMU. These limited margins provide weak support for the claim of substantial superiority. The MMMU results also require reconciliation: the official report gives 53.1 on validation for Qwen2.5-VL-3B, whereas Table 1 reports 41.9 after GRPO and 44.0 after V-ERR. Different protocols may explain the discrepancy, but the missing untuned-checkpoint accuracy under the authors’ own protocol prevents establishing improvement over the starting model.

  • The evidence does not establish faithful visual verification. Declining direct image attention does not demonstrate loss of visual information already propagated into other representations. Region-relevance preferences likewise do not establish correct checking of intermediate claims. Appendix D’s Case 5 asks the model to remove brown matte and blue metallic objects, but the new rationale explicitly removes a gray matte cylinder, while returning the correct answer B (6). This violates the requested operation and undermines the case study’s use as verification evidence. The reported answer improvements therefore do not establish the proposed explanation of genuine evidence rechecking.

Questions for authors
  1. Why use Qwen2.5-VL-3B rather than 7B, and restrict the Qwen3-VL comparison to 2B? Do the benefits persist on stronger contemporary backbones?
  2. Why does Table 8 omit MMMU and add AI2D and MathV360K? What are the Qwen3-VL results on the original five-benchmark suite with the same baseline comparisons?
  3. What does the untuned Qwen2.5-VL-3B checkpoint score under exactly the Table 1 protocol? Please explain the difference from the official 53.1 validation score, including split, prompting, preprocessing, and scoring differences.
  4. The main text cites Beta Normalization Policy Optimization, while B.4 only describes GRPOTrainer loss and normalization variants. Was the cited beta-normalization estimator implemented? Please provide the actual advantage and loss formulas.
  5. What evidence distinguishes faithful repeated verification from attention redistribution or improved downstream computation, and how do you explain Case 5’s incorrect intermediate reasoning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.