acceptodds
Under review as a conference paper at ICLR 2027

Visual Evidence Prototype-Guided Policy Optimization for Visual Reasoning

Abstract

Reinforcement learning with verifiable rewards has improved visual reasoning in large vision-language models. Existing methods either rely on visual dependence, while strong visual dependence does not imply correct use of visual information, or require external supervision over task-relevant image regions. To address these issues, we introduce Visual Evidence Prototypes (VEPs), which derive fine-grained visual supervision from on-policy rollout outcomes without region annotations. Specifically, for each question, VEPs aggregate the attention distributions of correct and incorrect rollouts over visual tokens into Success and Failure prototypes, and contrast them to capture the key differences. Empirical analyses show that VEPs identify key visual and response tokens for visual reasoning. Intervening on only the top 10% visual tokens selected by VEPs reduces answer accuracy by 20.15–28.45 percentage points. Based on these findings, we propose VEP-RL, which uses VEP to reweight trajectory-level updates and redistribute token-level credit to selected tokens. Across fourteen visual reasoning and perception benchmarks, VEP-RL improves average accuracy over the strongest baseline for each backbone by 2.44, 1.93, and 3.24 absolute percentage points on three models, respectively, with its effectiveness consistently validated in out-of-distribution benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Vision-Language Model, Reinforcement Learning, Visual Reasoning, Visual Evidence Prototype

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_D5R1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a visual attribution for each rollout by weighting its attention to image tokens according to how much the final answer relies on each reasoning step. It averages these attributions over correct and incorrect rollouts to form success and failure prototypes, and takes their difference as a contrastive signal of which visual regions separate success from failure. Pilot studies claim that this signal tells held-out correct rollouts apart from incorrect ones, that perturbing the regions it selects hurts accuracy much more than random perturbation, and that it identifies the response tokens that help or hurt the answer. Building on this, VEP-RL slightly rescales GRPO advantages according to how closely a rollout matches the prototypes, and shifts token-level credit toward the response tokens the prototypes select. The method is evaluated on three open vision-language backbones across many in-domain and out-of-domain benchmarks. The paper also includes ablations, a multi-seed comparison against DAPO, and a local theoretical analysis.

Critical strengths
  • I think the main strength is that the annotation-free signal is supported by proper controls, not just by attention visualizations. The pilot study on held-out rollouts shows that the prototype contrast clearly separates correct from incorrect responses, while shuffling the outcome labels brings it down to chance. The intervention experiment shows that replacing the visual states selected by the contrast hurts accuracy much more than replacing random ones, and the paired confidence intervals rule out noise. The leave-one-out variant in the main ablation keeps most of the gain, so the effect does not come from a rollout being compared against a prototype it helped build. Taken together, these results convinced me that the signal captures something real about which visual regions the model relies on.
  • I also find the comparison with the self-trained baselines convincing, because they share the training data, hyperparameters and evaluation protocol with VEP-RL, so the gap is not confounded by setup differences. The multi-seed experiment on the newest backbone shows that VEP-RL beats DAPO in every seed with a small spread, which suggests the improvement is not a lucky run. The training-cost table also shows that the extra overhead over GRPO is modest, lower than that of a comparable visual-perception method, since the extra computation only runs in groups that contain both correct and incorrect rollouts. This makes the method easy to adopt in practice.
Critical weaknesses
  • My main concern is that the mechanism behind the gain is very mild and was never varied. Both reweighting steps change the advantages only slightly, and the training-cost table shows that the guidance is active in only a minority of rollout groups. Even so, the reported gains over GRPO and DAPO are large. The strength coefficients, selection ratios and attention layers are all fixed, and multiple seeds are reported for only one backbone. The shuffle and contrast ablations are also single runs with small margins. I am therefore not convinced that the VEP selection itself, rather than some other difference in the training recipe, is what causes the gains.
  • I also think the comparisons with public baselines are not directly comparable. The evaluation-protocol part shows that several external methods are scored with a single greedy response, some are trained on much less data, and one runs with thinking disabled. The authors' models, by contrast, are averaged over several sampled responses, so the claimed margins are hard to interpret. The setup is also reported inconsistently. One backbone is described as using greedy decoding with several responses per question. The main table lacks the base-model rows and one GRPO run that was trained. It is also unclear whether VEP-RL is built on GRPO or on a DAPO-style recipe.
Questions for authors
  • Can you report a lambda_traj/lambda_tok sweep (e.g., 0, 0.05, 0.1, 0.2) and at least three seeds on a second backbone, including the DAPO baseline, the Table 8 shuffle controls and the 'w/o prototype contrast' row? Please also state exactly how VEP-RL's base recipe differs from your GRPO and DAPO runs, given that all three share clip 0.20/0.28 in table 4. A flat lambda curve, or shuffle controls whose seed intervals overlap VEP-RL, would lower my assessment.
  • Beyond model-internal proxies, do D+-selected visual tokens overlap annotated evidence regions more than Saliency-R1 or TAM selections do? And does the VEP similarity AUROC (Table 9) survive after controlling for response length and total visual attention? Appendix D.1 itself notes that the method 'does not identify annotated or causal evidence'.
7 comments
AuthorsOpenReview ↗

Responses to Reviewer D5R1 (Part 1/5)

We’d like to thank the reviewer for acknowledging the novelty and effectiveness of our proposed approach. We address your raised insightful points as follows.

W1. My main concern is that the mechanism behind the gain is very mild and was never varied. Both reweighting steps change the advantages only slightly, and the training-cost table shows that the guidance is active in only a minority of rollout groups. Even so, the reported gains over GRPO and DAPO are large. The strength coefficients, selection ratios and attention layers are all fixed, and multiple seeds are reported for only one backbone. The shuffle and contrast ablations are also single runs with small margins. I am therefore not convinced that the VEP selection itself, rather than some other difference in the training recipe, is what causes the gains.

Q1. Can you report a lambda_traj/lambda_tok sweep (e.g., 0, 0.05, 0.1, 0.2) and at least three seeds on a second backbone, including the DAPO baseline, the Table 8 shuffle controls and the 'w/o prototype contrast' row? Please also state exactly how VEP-RL's base recipe differs from your GRPO and DAPO runs, given that all three share clip 0.20/0.28 in table 4. A flat lambda curve, or shuffle controls whose seed intervals overlap VEP-RL, would lower my assessment.

We address your concerns about the coverage and strength of VEP guidance through the following analyses and experiments.

W1 (1). the training-cost table shows that the guidance is active in only a minority of rollout groups.

We would like to clarify that VEP guidance covers the majority of effective rollout groups, namely groups with nonzero advantages. The percentages in Table 6 are calculated over all sampled groups, including those with zero advantages. We support this through the following analysis and training statistics.

GRPO Advantage Analysis: GRPO computes advantages by normalizing rewards within each group. When all responses receive identical rewards (typically all correct or all incorrect), their advantages become zero. VL-Rethinker [1] also reports this vanishing-advantage problem: for Qwen2.5-VL-72B, the proportion of effective rollout groups decreases from approximately 40% to below 20% after training.

Empirical Validation: We count effective groups and VEP-guided groups throughout training for all three backbones. The results are presented below.

BackboneEffective / All groupsVEP-guided / All groupsVEP-guided / Effective groups
Qwen2.5-VL-7B35.2%27.8%78.9%
Qwen3-VL-8B27.2%20.6%75.7%
Qwen3.5-9B24.8%17.9%72.2%

As shown in the table, only 24.8–35.2% of all sampled groups have nonzero advantages, while VEP guidance covers 72.2–78.9% of these effective groups. These results show that VEP guidance covers the majority of effective rollout groups across all three backbones.


W1 (2). Both reweighting steps change the advantages only slightly

Q1 (2). Can you report a lambda_traj/lambda_tok sweep (e.g., 0, 0.05, 0.1, 0.2)? A flat lambda curve would lower my assessment.

The two reweighting steps act multiplicatively on each token’s advantage. With λtraj=λtok=0.1\lambda_{\mathrm{traj}}=\lambda_{\mathrm{tok}}=0.1, trajectory-level multipliers range from 0.9 to 1.1. When 5% of response tokens are selected, their token-level multiplier is 1.095 (1+0.1×0.951+0.1\times0.95), compared with 0.995 for the remaining tokens. The combined multiplier can therefore reach 1.1×1.095=1.20451.1\times1.095=1.2045, corresponding to a 20.45% increase in advantage magnitude.

We then vary each coefficient separately on Qwen3-VL-8B, keeping the other fixed at 0.1, to examine how the reweighting strength affects accuracy. All results are Acc. (%).

λtraj\lambda_{\mathrm{traj}}λtok\lambda_{\mathrm{tok}}Acc. (%) ↑
0074.48
00.176.77
0.050.177.52
0.10.178.34
0.20.176.54
0.1077.15
0.10.0577.86
0.10.275.10

The GRPO baseline (λtraj=λtok=0\lambda_{\mathrm{traj}}=\lambda_{\mathrm{tok}}=0) achieves 74.48% accuracy. As shown in the table, removing trajectory or token reweighting reduces accuracy by 1.57 and 1.19 percentage points, respectively. Increasing either coefficient to 0.2 also lowers accuracy, to 76.54% and 75.10%, respectively. Our setting (0.1, 0.1) achieves the highest accuracy among the tested settings.


AuthorsOpenReview ↗

Responses to Reviewer D5R1 (Part 2/5)

W1 (3). The strength coefficients, selection ratios and attention layers are all fixed

Attention layers

Our layer selection is motivated by prior findings that later-middle layers contribute more strongly to correct object-token predictions [2], and that multi-layer attention aggregation improves performance, with limited gains from adding more layers [3]. We therefore select three evenly spaced depths spanning the middle-to-late decoder layers: 45%, 65%, and 85%. To examine whether performance depends on these specific depths, we compare alternative layer selections and all-layer aggregation on Qwen3-VL-8B. The results are presented below.

Decoder depthsAcc. (%) ↑
25%, 45%, 65%76.38
45%, 65%, 85%78.34
55%, 75%, 95%78.17
All layers76.12

As shown in the table, our selected configuration achieves the highest accuracy. Moving to 55%, 75%, and 95% reduces accuracy by only 0.17 points, whereas the earlier configuration and all-layer aggregation reduce accuracy by 1.96 and 2.22 points, respectively.


Visual-token intervention ratio

We choose 10% based on the intervention analysis in Figure 7: answer accuracy under intervention decreases as the ratio increases to 10% and plateaus thereafter. We train VEP-RL on Qwen3-VL-8B with intervention ratios of 0%, 5%, 10%, 15%, and 20%, keeping trajectory-level reweighting at λtraj=0.1\lambda_{\mathrm{traj}}=0.1 and token-level reweighting at λtok=0.1\lambda_{\mathrm{tok}}=0.1. At 0% intervention, no response tokens are selected for reweighting, so this setting corresponds to trajectory-only guidance. The results are presented below.

Intervention RatioAcc. (%) ↑
0%77.15
5%77.71
10%78.34
15%77.97
20%77.79

As shown in the table, 10% achieves the highest accuracy of 78.34%. Increasing the ratio to 15% and 20% reduces accuracy by 0.37 and 0.55 points, respectively.


Response-token selection ratio

Our token selection is motivated by prior studies showing that RLVR gains can depend on a sparse subset of token decisions [4, 5]. We rank response tokens by their intervention scores and redistribute credit toward up to the top 5% with positive scores. To evaluate this choice, we compare selection ratios of 1%, 3%, 5%, 10%, and 15% on Qwen3-VL-8B, keeping trajectory-level reweighting at λtraj=0.1\lambda_{\mathrm{traj}}=0.1 and token-level reweighting at λtok=0.1\lambda_{\mathrm{tok}}=0.1. The results are presented below.

Selection ratioAcc. (%) ↑
1%77.53
3%77.98
5%78.34
10%77.67
15%76.90

As shown in the table, 5% achieves the highest accuracy of 78.34% among the tested ratios. Reducing the ratio to 3% lowers accuracy by 0.36 points, while increasing it to 10% and 15% lowers accuracy by 0.67 and 1.44 points, respectively.


Q1 (4). Please also state exactly how VEP-RL's base recipe differs from your GRPO and DAPO runs, given that all three share clip 0.20/0.28 in table 4.

W2 (5). It is also unclear whether VEP-RL is built on GRPO or on a DAPO-style recipe.

We'd like to clarify that VEP-RL is built on standard GRPO, as stated in Section 4. It adds only VEP-guided advantage reweighting at the trajectory and token levels. The GRPO and DAPO baselines follow their respective official training recipes, using VeRL as stated in Appendix B.1.

The clip-high value of 0.28 reported for GRPO and VEP-RL is a typo, and both use 0.20/0.20. We apologize for this typo and will correct Table 4 and the corresponding text. All three methods use ViRL-39K, two training epochs, and AdamW with a learning rate of 10−610^{-6}. The training settings used in our GRPO, DAPO, and VEP-RL runs are summarized below.

SettingGRPODAPOVEP-RL
Clip-low / clip-high0.20 / 0.200.20 / 0.280.20 / 0.20
Dynamic samplingNoYesNo
Overlong reward shapingNoYesNo
KL regularizationβ=10−4\beta=10^{-4}Disabledβ=10−4\beta=10^{-4}
VEP-guided advantage reweightingNoNoTrajectory and token levels
AuthorsOpenReview ↗

Responses to Reviewer D5R1 (Part 3/5)

Q1 (5). at least three seeds on a second backbone, including the DAPO baseline, the Table 8 shuffle controls and the 'w/o prototype contrast' row? Shuffle controls whose seed intervals overlap VEP-RL would lower my assessment.

Appendix B.2.2 (Table 7) reports results across three training seeds on Qwen3.5-9B. Following your suggestion, we conducted experiments on a second backbone, Qwen2.5-VL-7B, using three training seeds (42, 43, and 44), and report the 14-benchmark macro accuracy. The 95% CIs are computed across training seeds; paired Δ\Delta denotes VEP-RL minus each comparison method under the same seed. The results are presented below.

MethodSeed 42 Acc. (%)Seed 43 Acc. (%)Seed 44 Acc. (%)Mean Acc. ± Std (%) ↑95% CI of Acc. (%)Paired Δ Acc. ± Std (pp) ↑95% CI of Δ Acc. (pp)
DAPO65.2965.5065.1165.30±0.20[64.82,65.78]+3.62±0.04[3.52,3.72]
Token Shuffle67.8367.7767.5467.71±0.15[67.33,68.09]+1.20±0.10[0.96,1.44]
Trajectory Shuffle67.3167.5567.1967.35±0.18[66.89,67.81]+1.57±0.06[1.41,1.72]
w/o Prototype Contrast68.2268.1168.0668.13±0.08[67.93,68.33]+0.79±0.16[0.38,1.19]
VEP-RL (Ours)68.9569.0868.7268.92±0.18[68.46,69.37]——

As shown in the table, VEP-RL outperforms all comparison methods in every seed, and the 95% CI of its mean accuracy does not overlap with those of either shuffle control. The mean paired gains over Token Shuffle and Trajectory Shuffle are 1.20 and 1.57 percentage points, with 95% CIs of [0.96, 1.44] and [1.41, 1.72], respectively, both above zero. Removing prototype contrast reduces mean accuracy by 0.79 percentage points, with a paired 95% CI of [0.38, 1.19]. These results support the contributions of VEP-guided RL and prototype contrast.


W2. I also think the comparisons with public baselines are not directly comparable. The evaluation-protocol part shows that several external methods are scored with a single greedy response, some are trained on much less data, and one runs with thinking disabled. The authors' models, by contrast, are averaged over several sampled responses, so the claimed margins are hard to interpret. The setup is also reported inconsistently. One backbone is described as using greedy decoding with several responses per question. The main table lacks the base-model rows and one GRPO run that was trained. It is also unclear whether VEP-RL is built on GRPO or on a DAPO-style recipe.

W2 (1). some are trained on much less data

We would like to clarify that the strongest baseline on each backbone is trained on the same ViRL-39K dataset as VEP-RL. Under the fair evaluation protocol, VEP-RL outperforms VGPO on Qwen2.5-VL-7B, VPPO on Qwen3-VL-8B, and DAPO on Qwen3.5-9B by 2.68, 1.93, and 3.24 percentage points, respectively. For public baselines trained on smaller datasets, we evaluate their released checkpoints and state the training-data differences in Appendix B.2.1.


W2 (2). one runs with thinking disabled.

We would like to clarify that thinking mode is disabled for all evaluated methods, including VEP-RL, on Qwen3-VL-8B and Qwen3.5-9B. When describing VAD in Appendix B.2.1, we retained this detail from its official evaluation documentation. We will clarify that the same setting applies to all methods.


W2 (3). One backbone is described as using greedy decoding with several responses per question.

The originally reported results for GRPO, DAPO, and VEP-RL on Qwen3.5-9B were obtained by sampling eight responses per question with temperature 1.0 and top-p=1.0. The same sampling settings were also used for the other backbones. We apologize for the typo in Appendix B.2.1 and will correct the temperature from 0 to 1.0.

Reviewer_DXaxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes VEPs for visual reasoning. For each image-question pair, it aggregates visual attributions from correct and incorrect rolllouts into Success and Failure prototypes, then contrasts them to identify visual evidence associated with successful and failed reasoning. Based on this signal, VEP-RL re-weights trajectory-level updates and redistributes token-level credit, and shows consistent gains across three backbones and fourteen benchmarks.

Critical strengths

I think this paper genuinely brings visual evidence into RL optimization, instead of regarding vision only as the task context. These findings are quite informative, especially the evidence that success/failure prototypes can distinguish rollout outcomes. Besides, they identify visual and response tokens that matter for the final answer.

Another point is that this method is simple and easy to understand, but it is also consistently effective across multiple backbones and benchmarks. This simplicity is not weakness. The design is quite clean and the empirical gains are convincing.

Critical weaknesses

I do not have a major concern with the method itself, but I have one question about its coverage during training. VEP guidance is used only when a group of eight rollouts contains at least two correct and two incorrect samples. Otherwise, the update falls back to standard GRPO. In practice, this condition is satisfied in only about 25% of training groups. This means that most groups never use VEP guidance. It would be useful to understand whether the gains mainly come from this relatively small subset of informative groups, and whether the method could be extended to cover a larger fraction of training groups.

Questions for authors

See above weaknesses.

3 comments
AuthorsOpenReview ↗

Responses to Reviewer DXax

We’d like to thank the reviewer for acknowledging the novelty and effectiveness of our proposed approach. We address your raised insightful points as follows.

W1. I do not have a major concern with the method itself, but I have one question about its coverage during training. VEP guidance is used only when a group of eight rollouts contains at least two correct and two incorrect samples. Otherwise, the update falls back to standard GRPO. In practice, this condition is satisfied in only about 25% of training groups. This means that most groups never use VEP guidance. It would be useful to understand whether the gains mainly come from this relatively small subset of informative groups, and whether the method could be extended to cover a larger fraction of training groups.

Q1. See above weaknesses.

We would like to clarify that VEP guidance covers the majority of effective rollout groups, namely groups with nonzero advantages. The percentages in Table 6 are calculated over all sampled groups, including those with zero advantages. We support this through the following analysis and training statistics.

GRPO Advantage Analysis: GRPO computes advantages by normalizing rewards within each group. When all responses receive identical rewards (typically all correct or all incorrect), their advantages become zero. VL-Rethinker [1] also reports this vanishing-advantage problem: for Qwen2.5-VL-72B, the proportion of effective rollout groups decreases from approximately 40% to below 20% after training.

Empirical Validation: We count effective groups and VEP-guided groups throughout training for all three backbones. The results are presented below.

BackboneEffective / All groupsVEP-guided / All groupsVEP-guided / Effective groups
Qwen2.5-VL-7B35.2%27.8%78.9%
Qwen3-VL-8B27.2%20.6%75.7%
Qwen3.5-9B24.8%17.9%72.2%

As shown in the table, only 24.8–35.2% of all sampled groups have nonzero advantages, while VEP guidance covers 72.2–78.9% of these effective groups. These results show that VEP guidance covers the majority of effective rollout groups across all three backbones.


To further extend VEP coverage, we increase the rollout group size (G) so that more groups contain at least two correct and two incorrect responses. We compare G=8, 16, and 32 on Qwen3-VL-8B, using the same training data, two training epochs, and evaluation protocol. The results are presented below.

Group sizeVEP / All groupsVEP / Effective groupsGRPO Acc. (%)DAPO Acc. (%)VEP-RL Acc. (%)
820.6%75.7%74.4875.0778.34
1627.9%95.1%74.8075.6679.12
3228.9%96.2%75.1275.6179.31

As shown in the table, increasing G from 8 to 32 extends VEP guidance from 20.6% to 28.9% of all groups, and from 75.7% to 96.2% of effective groups. VEP-RL accuracy improves from 78.34% to 79.31%, consistently exceeding DAPO by 3.27–3.70 percentage points. Since G=32 requires four times as many rollouts as G=8 for a 0.97-point accuracy gain, we use G=8 to balance accuracy and training cost.


[1] Wang et al. “VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learning.” NeurIPS. 2025.

Dear Reviewer DXax,

Thank you for your detailed feedback and constructive comments on our submission.

As the author-reviewer discussion period ends in about 1 day, we would like to confirm whether we have sufficiently addressed all your concerns.

Thanks again for your time and help in improving the quality of this paper.

Best regards,

The authors of Submission 14938

Reviewer_DXaxOpenReview ↗

Thanks for the Clarification

I thank the authors for their detailed statistics which address this issue. Considering both the original paper and the rebuttal, I think my current score appropriately reflects the contributions of this paper. Thus, I will maintain my original score.

Reviewer_4gQ1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Visual Evidence Prototypes (VEPs) and VEP-guided Reinforcement Learning (VEP-RL) to improve visual reasoning in large vision-language models. The key motivation is that existing visual reinforcement learning approaches either measure visual dependence without distinguishing whether visual information is correctly utilized or require external supervision to identify task-relevant visual regions.

To address these limitations, the authors construct success and failure prototypes by aggregating visual attention distributions from correct and incorrect on-policy rollouts. The contrast between these prototypes is used to identify visual evidence associated with successful and unsuccessful reasoning. Building upon these signals, VEP-RL modifies GRPO at two levels: trajectory-level advantage reweighting based on prototype similarity and token-level credit redistribution based on visual intervention.

Experiments across three model backbones and fourteen benchmarks demonstrate consistent improvements over existing baselines. Additional analyses, including visual-token interventions, response-token interventions, ablation studies, and out-of-distribution evaluations, further support the effectiveness of the proposed approach.

Critical strengths

S1. A meaningful approach to obtaining fine-grained visual supervision without external annotations.

The paper addresses an important limitation of existing visual reinforcement learning methods: visual dependence does not necessarily indicate correct visual reasoning. Constructing outcome-conditioned visual evidence prototypes from on-policy rollouts provides a reasonable and potentially scalable alternative to explicit region annotations or external teacher supervision.

S2. Comprehensive empirical analyses supporting the effectiveness of visual evidence prototypes.

The authors go beyond reporting downstream benchmark improvements and conduct three pilot studies to characterize the proposed VEPs. In particular, the independent-rollout evaluation, visual-token interventions, and response-token interventions provide complementary evidence that VEPs capture useful signals associated with successful and unsuccessful reasoning. The intervention experiments are especially valuable in supporting the proposed motivation.

S3. Consistent performance improvements across multiple backbones and benchmarks.

VEP-RL achieves improvements of 2.44, 1.93, and 3.24 percentage points over the strongest baselines on Qwen2.5-VL-7B, Qwen3-VL-8B, and Qwen3.5-9B, respectively. The evaluation covers fourteen benchmarks, includes out-of-distribution testing, and provides component ablations and multi-seed experiments. The reported computational overhead of approximately 3–4% is also relatively modest.

Critical weaknesses

W1. The reliability and causal interpretation of the constructed visual evidence remain insufficiently established.

The proposed VEPs are derived from attention distributions conditioned on rollout correctness. However, differences between successful and failed rollouts may reflect correlations with model behavior rather than genuinely causal visual evidence. Although the intervention experiments demonstrate that selected tokens affect model predictions, replacing visual hidden states can also introduce distribution shifts. Therefore, the results do not fully establish that the identified regions correspond to the actual visual evidence responsible for correct reasoning. Direct evaluations against annotated evidence regions or more controlled counterfactual interventions would provide stronger validation.

W2. The applicability of VEP guidance is constrained by its reliance on mixed-outcome rollout groups.

VEP-RL requires at least two correct and two incorrect rollouts within each group to construct reliable prototypes. According to Table 6, only 17.9–27.8% of training groups satisfy this condition, meaning that most training examples fall back to standard GRPO updates. This raises concerns about the method's scalability and effectiveness on tasks where model predictions are predominantly correct or incorrect. The paper lacks a systematic analysis of how this eligibility constraint affects performance across different task difficulties, rollout group sizes, and model capabilities.

Questions for authors

Q1. How can the authors further distinguish genuine visual evidence from outcome-correlated attention patterns?

The intervention experiments support the importance of VEP-selected tokens, but it remains unclear whether these tokens correspond to semantically correct evidence regions. Could the authors evaluate VEP localization against human-annotated bounding boxes or segmentation masks, using metrics such as IoU or pointing accuracy? Such experiments would help establish whether VEPs identify genuinely relevant visual evidence rather than merely influential visual representations.

Q2. How sensitive is VEP-RL to the number and composition of on-policy rollouts?

The current implementation requires at least two successful and two failed trajectories within a group of eight rollouts. How would performance change with different group sizes, such as 4, 16, or 32? Additionally, could the authors report the performance gains separately for questions with different initial success rates to clarify whether VEP-RL primarily benefits moderately difficult questions?

Q3. How robust is VEP construction to the choice of attention layers and token-selection hyperparameters?

The method aggregates attention from three manually selected decoder depths (45%, 65%, and 85%) and uses fixed ratios for visual-token intervention (10%) and response-token selection (5%). How sensitive are the final results to these design choices? In particular, would dynamically selecting attention layers or intervention ratios provide additional improvements, and do the current settings generalize to architectures with substantially different attention mechanisms?

5 comments
AuthorsOpenReview ↗

Responses to Reviewer 4gQ1 (Part 1/4)

We’d like to thank the reviewer for acknowledging the novelty and effectiveness of our proposed approach. We address your raised insightful points as follows.

W1. The reliability and causal interpretation of the constructed visual evidence remain insufficiently established. The proposed VEPs are derived from attention distributions conditioned on rollout correctness. However, differences between successful and failed rollouts may reflect correlations with model behavior rather than genuinely causal visual evidence. Although the intervention experiments demonstrate that selected tokens affect model predictions, replacing visual hidden states can also introduce distribution shifts. Therefore, the results do not fully establish that the identified regions correspond to the actual visual evidence responsible for correct reasoning. Direct evaluations against annotated evidence regions or more controlled counterfactual interventions would provide stronger validation.

Q1. How can the authors further distinguish genuine visual evidence from outcome-correlated attention patterns? The intervention experiments support the importance of VEP-selected tokens, but it remains unclear whether these tokens correspond to semantically correct evidence regions. Could the authors evaluate VEP localization against human-annotated bounding boxes or segmentation masks, using metrics such as IoU or pointing accuracy? Such experiments would help establish whether VEPs identify genuinely relevant visual evidence rather than merely influential visual representations.

Q1 (1). Could the authors evaluate VEP localization against human-annotated bounding boxes or segmentation masks, using metrics such as IoU or pointing accuracy?

To evaluate whether VEP-selected regions correspond to human-annotated evidence, we conduct an additional localization experiment on Qwen3.5-9B. We collect 500 questions from our 14 evaluation benchmarks and ask three graduate students to manually annotate the evidence regions in the corresponding images with bounding boxes. We compare VEP with Saliency-R1, TAM, Answer Attention, and Reasoning Attention on this annotated set, using IoU, Box-center coverage (BCC), and Pointing Game accuracy (PG).

For each method, we rank the visual tokens by their attribution scores and select the top 5%, 10%, 15%, and 20% to obtain the corresponding image regions. We then compare these regions with the annotated boxes: IoU measures their intersection-over-union, while BCC checks whether the annotated box centers fall within the selected regions. For PG, we take the point with the highest attribution score and check whether it falls inside an annotated box. All three metrics are evaluated on the same 500 questions. The results are presented below.

MetricTop-k%VEPSaliency-R1TAMAnswer AttentionReasoning Attention
IoU ↑5%10.106.157.434.587.11
10%12.247.869.316.318.05
15%12.718.7410.077.308.38
20%12.969.1810.468.058.54
BCC ↑5%31.618.821.212.621.8
10%45.228.231.222.631.6
15%53.835.837.031.833.8
20%59.441.442.638.636.8
PG ↑—25.612.420.69.013.2

As shown in the table, VEP achieves the highest PG, as well as the highest IoU and BCC across all selection ratios. At the 10% selection ratio used in our experiments, it outperforms the strongest baseline by 2.93 percentage points in IoU and 13.6 percentage points in BCC, showing closer agreement with human-annotated visual evidence.

AuthorsOpenReview ↗

Responses to Reviewer 4gQ1 (Part 2/4)

W2. The applicability of VEP guidance is constrained by its reliance on mixed-outcome rollout groups. VEP-RL requires at least two correct and two incorrect rollouts within each group to construct reliable prototypes. According to Table 6, only 17.9–27.8% of training groups satisfy this condition, meaning that most training examples fall back to standard GRPO updates. This raises concerns about the method's scalability and effectiveness on tasks where model predictions are predominantly correct or incorrect. The paper lacks a systematic analysis of how this eligibility constraint affects performance across different task difficulties, rollout group sizes, and model capabilities.

Q2. How sensitive is VEP-RL to the number and composition of on-policy rollouts? The current implementation requires at least two successful and two failed trajectories within a group of eight rollouts. How would performance change with different group sizes, such as 4, 16, or 32? Additionally, could the authors report the performance gains separately for questions with different initial success rates to clarify whether VEP-RL primarily benefits moderately difficult questions?

W2 (1). According to Table 6, only 17.9–27.8% of training groups satisfy this condition, meaning that most training examples fall back to standard GRPO updates.

We would like to clarify that VEP guidance covers the majority of effective rollout groups, namely groups with nonzero advantages. The percentages in Table 6 are calculated over all sampled groups, including those with zero advantages. We support this through the following analysis and training statistics.

GRPO Advantage Analysis: GRPO computes advantages by normalizing rewards within each group. When all responses receive identical rewards (typically all correct or all incorrect), their advantages become zero. VL-Rethinker [1] also reports this vanishing-advantage problem: for Qwen2.5-VL-72B, the proportion of effective rollout groups decreases from approximately 40% to below 20% after training.

Empirical Validation: We count effective groups and VEP-guided groups throughout training for all three backbones. The results are presented below.

BackboneEffective / All groupsVEP-guided / All groupsVEP-guided / Effective groups
Qwen2.5-VL-7B35.2%27.8%78.9%
Qwen3-VL-8B27.2%20.6%75.7%
Qwen3.5-9B24.8%17.9%72.2%

As shown in the table, only 24.8–35.2% of all sampled groups have nonzero advantages, while VEP guidance covers 72.2–78.9% of these effective groups. These results show that VEP guidance covers the majority of effective rollout groups across all three backbones.


Q2 (2). How would performance change with different group sizes, such as 4, 16, or 32?

Our results show that increasing the rollout group size (G) expands VEP coverage and improves accuracy. We compare GRPO, DAPO, and VEP-RL on Qwen3-VL-8B with G=4, 8, 16, and 32, using the same training data, two training epochs, and evaluation protocol. The results are presented below.

Group sizeVEP / All groupsVEP / Effective groupsGRPO Acc. (%)DAPO Acc. (%)VEP-RL Acc. (%)Gain over DAPO (pp)
47.2%33.8%73.6274.3175.46+1.15
820.6%75.7%74.4875.0778.34+3.27
1627.9%95.1%74.8075.6679.12+3.46
3228.9%96.2%75.1275.6179.31+3.70

As shown in the table, increasing G from 4 to 8 expands VEP guidance from 33.8% to 75.7% of effective groups, while VEP-RL accuracy improves by 2.88 points. Coverage reaches 95.1% at G=16, and increasing G further to 32 adds only 0.19 points in accuracy. VEP-RL consistently outperforms both GRPO and DAPO across all tested group sizes. Since G=32 requires four times as many rollouts as G=8 for a 0.97-point accuracy gain, we use G=8 to balance accuracy and training cost.

AuthorsOpenReview ↗

Responses to Reviewer 4gQ1 (Part 3/4)

Q2 (3). Additionally, could the authors report the performance gains separately for questions with different initial success rates to clarify whether VEP-RL primarily benefits moderately difficult questions?

Our results show that VEP-RL achieves the largest accuracy gains on moderately difficult questions, while also improving performance on hard and easy questions. We group questions from the 14 evaluation benchmarks by their initial success rate p0p_0, estimated from 32 responses per question using the frozen Qwen3-VL-8B base model. The results are presented below.

Initial success rateGRPO Acc. (%)DAPO Acc. (%)VEP-RL Acc. (%)Gain over DAPO (pp)
Hard: 0≤p0≤0.250 \le p_0 \le 0.2521.522.926.0+3.1
Medium: 0.25<p0<0.750.25 < p_0 < 0.7568.069.077.0+8.0
Easy: 0.75≤p0≤10.75 \le p_0 \le 196.796.898.0+1.2

As shown in the table, VEP-RL outperforms both baselines in all three bins. Compared with DAPO, it improves accuracy by 8.0 points on moderately difficult questions, and by 3.1 and 1.2 points on hard and easy questions, respectively.


Q3. How robust is VEP construction to the choice of attention layers and token-selection hyperparameters? The method aggregates attention from three manually selected decoder depths (45%, 65%, and 85%) and uses fixed ratios for visual-token intervention (10%) and response-token selection (5%). How sensitive are the final results to these design choices? In particular, would dynamically selecting attention layers or intervention ratios provide additional improvements, and do the current settings generalize to architectures with substantially different attention mechanisms?

Q3 (1). The method aggregates attention from three manually selected decoder depths (45%, 65%, and 85%)

Our layer selection is motivated by prior findings that later-middle layers contribute more strongly to correct object-token predictions [2], and that multi-layer attention aggregation improves performance, with limited gains from adding more layers [3]. We therefore select three evenly spaced depths spanning the middle-to-late decoder layers: 45%, 65%, and 85%. To examine whether performance depends on these specific depths, we compare alternative layer selections and all-layer aggregation on Qwen3-VL-8B. The results are presented below.

Decoder depthsAcc. (%) ↑
25%, 45%, 65%76.38
45%, 65%, 85%78.34
55%, 75%, 95%78.17
All layers76.12

As shown in the table, our selected configuration achieves the highest accuracy. Moving to 55%, 75%, and 95% reduces accuracy by only 0.17 points, whereas the earlier configuration and all-layer aggregation reduce accuracy by 1.96 and 2.22 points, respectively.


Q3 (2). uses fixed ratios for visual-token intervention (10%)

We choose 10% based on the intervention analysis in Figure 7: answer accuracy under intervention decreases as the ratio increases to 10% and plateaus thereafter. We train VEP-RL on Qwen3-VL-8B with intervention ratios of 0%, 5%, 10%, 15%, and 20%, keeping trajectory-level reweighting at λtraj=0.1\lambda_{\mathrm{traj}}=0.1 and token-level reweighting at λtok=0.1\lambda_{\mathrm{tok}}=0.1. At 0% intervention, no response tokens are selected for reweighting, so this setting corresponds to trajectory-only guidance. The results are presented below.

Intervention RatioAcc. (%) ↑
0%77.15
5%77.71
10%78.34
15%77.97
20%77.79

As shown in the table, 10% achieves the highest accuracy of 78.34%. Increasing the ratio to 15% and 20% reduces accuracy by 0.37 and 0.55 points, respectively.


Q3 (3). response-token selection (5%)

Our token selection is motivated by prior studies showing that RLVR gains can depend on a sparse subset of token decisions [4, 5]. We rank response tokens by their intervention scores and redistribute credit toward up to the top 5% with positive scores. To evaluate this choice, we compare selection ratios of 1%, 3%, 5%, 10%, and 15% on Qwen3-VL-8B, keeping trajectory-level reweighting at λtraj=0.1\lambda_{\mathrm{traj}}=0.1 and token-level reweighting at λtok=0.1\lambda_{\mathrm{tok}}=0.1. The results are presented below.

Selection ratioAcc. (%) ↑
1%77.53
3%77.98
5%78.34
10%77.67
15%76.90

As shown in the table, 5% achieves the highest accuracy of 78.34% among the tested ratios. Reducing the ratio to 3% lowers accuracy by 0.36 points, while increasing it to 10% and 15% lowers accuracy by 0.67 and 1.44 points, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.