acceptodds
Under review as a conference paper at ICLR 2027

VC-RL: Visual Contrast-Guided Reinforcement Learning for Native Visual Thinking

Abstract

Fine-grained visual understanding requires multimodal large language models to identify local details accurately and use them in visual reasoning. Native thinking can improve this capability, but the use of local visual feedback to improve long-form reasoning over full images remains underexplored. Local views improve a teacher's access to fine details while also changing image scale and context, so differences between teacher and student predictions can reflect both target evidence and other changes in visual conditions. We propose Visual Contrast-Guided Reinforcement Learning (VC-RL), which combines Group Relative Policy Optimization (GRPO) with visual on-policy self-distillation. The student generates thinking trajectories from full images, and a self-teacher scores the same trajectories under size-matched positive-evidence and counterfactual views constructed with shared preprocessing rules. The contrast between these predictions guides token selection and bounded modulation of group-relative advantages derived from sequence-level rewards, without reversing the signs of nonzero advantages. Regional visual feedback thereby contributes selectively to token-level optimization, while task-level rewards constrain the policy updates. Across six fine-grained and high-resolution visual evaluations, VC-RL achieves average accuracies of 79.21% and 81.33% with Qwen3.5-4B and Qwen3.5-9B, respectively, improving over the corresponding base models by 4.45 and 3.90 percentage points, with additional gains on multiple general multimodal benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, fine-grained visual understanding, native visual thinking, reinforcement learning, on-policy self-distillation, visual contrast

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VnS1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a visual contrast-guided reinforcement learning framework that uses a self-EMA teacher to identify tokens sensitive to visual input and then selectively modulates those tokens during RL training. Experimental results show that VC-RL outperforms various methods on high-resolution and fine-grained visual understanding tasks.

Critical strengths
  • The proposed VC-RL outperforms various methods across different benchmarks.

  • Selective token modulation based on different visual regions is interesting and appears effective.

Critical weaknesses
  • The contribution seems limited, given that prior work, RLCSD [1] , has already proposed token-level selective self-distillation.

  • There is a lack of empirical analysis of the selected tokens and how reliably they are associated with visual evidence.

Reference:

[1] RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

Questions for authors

Any analysis show selected token match the visual evidence?

Reviewer_MxazOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

VC-RL post-trains vision-language models to use small image details during step-by-step reasoning. The model answers from the full image and is trained with GRPO. A copy of the model acts as a teacher and scores each generated token twice: once with a crop of the region that holds the answer, and once with a same-size crop of a different region. Tokens where the two scores differ most get their GRPO advantage scaled up or down, without flipping its sign. On six high-resolution benchmarks, average accuracy rises from 74.76% to 79.21% for Qwen3.5-4B and from 77.43% to 81.33% for 9B.

Critical strengths
  1. The gains are consistent. VC-RL improves all six benchmarks at both model sizes, by 4.45 and 3.90 points on average, and it beats Vision-OPD and OPD-V on every benchmark. The trained 4B model even beats the 9B base model. This is the paper's main strength, but it does not outweigh the weaknesses below.
Critical weaknesses
  1. The pipeline is a little complicated, which may make it hard to use in practice. In each training step, a teacher scores every token under two specially built crops. The result then goes through centering, a token threshold, bounded scaling, sign clamping and a two-part loss. The paper does not fully explain why each piece is needed. Building the crops also requires a box around the answer region for every training example, plus a search for a same-size negative region and a custom crop-and-pixelation step. Such boxes can be costly to collect, and the paper does not discuss how to obtain them for new data or other tasks. In addition, the writing is hard to follow, even after careful reading.

This is the main reason for my weak reject: the method is hard to understand and to scale up, and it may not be simple enough to apply to similar problems.

  1. The paper does not show that the selected tokens actually depend on the image. Apart from one example in Figure 1, it never measures which tokens are selected, and its own analysis shows that more than half of all tokens are selected (57.73%). Since this is the paper's main claim, I think should include more details here.
Questions for authors
  1. Every training example needs a box around the answer region, and such boxes are costly to collect for new data. How does VC-RL perform if the boxes come from an off-the-shelf detection or grounding model instead of the existing annotations?

  2. The paper argues that the contrast selects tokens that depend on the answer region, but it only shows one example. Among the selected tokens, what fraction describe image content, compared with generic reasoning text such as "Let me think"? Can you include more behavior analysis?

  3. The 6,159 training images come from the Vision-OPD data, and the paper checks image overlap only for its validation set. Have you checked whether any training images appear in the six test benchmarks?

Reviewer_PMZ3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VC-RL, a post-training method for improving fine-grained visual reasoning in native-thinking multimodal models. An EMA self-teacher scores student-generated trajectories under size-matched positive-evidence and counterfactual regional views. The resulting contrast selects tokens and modulates their GRPO advantages while preserving the signs of nonzero sequence advantages. Experiments on Qwen3.5-4B and 9B show improvements across six fine-grained and high-resolution visual evaluations without additional inference-time tools. Ablations and repeated training runs examine the contributions of regional view construction, contrast processing, and token-level modulation.

Critical strengths

1.VC-RL achieves clear gains over competing methods. At 9B, it improves average accuracy over Vision-OPD and OPD-V by 4.22 and 2.97 percentage points, respectively (Table 1). Repeated training runs also confirm gains over GRPO and visually adapted RLCSD (Table 7).

2.The ablations provide useful support for the design. Comparisons examine regional view construction, token selection, advantage modulation, and feedback during thinking versus final-answer generation. These experiments help identify which components contribute to the improvements.

Critical weaknesses

The methodological advance over RLCSD is limited. Contrastive token selection, bounded sign-preserving advantage modulation, and two-branch loss normalization are already present in RLCSD. VC-RL mainly adds matched regional views, response-level centering, and a modified modulation function. Although these changes improve performance, the contribution appears to be an incremental adaptation of an existing framework.

The gains are not evaluated against the additional training cost. VC-RL requires an EMA teacher, two teacher-scoring passes per trajectory, and annotated evidence regions. The paper does not report training time, memory usage, or comparisons under matched computational budgets. It therefore remains unclear whether its advantage over GRPO justifies the added complexity and cost.

Questions for authors

How does VC-RL compare with GRPO under matched computational budgets? Could the authors report training time, GPU memory usage?

Could VC-RL’s emphasis on local regions hurt multi-image/region understanding and contribute to the BLINK decline? Evaluations on MuirBench or TreeBench would help clarify this potential trade-off.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.