acceptodds
Under review as a conference paper at ICLR 2027

CV-OPD: Contrastive Vision On-policy Distillation

27 views

Abstract

On-policy distillation (OPD)reveals versatility and convenience during VLM post training, providing dense supervision for student from teacher. However, we ob served an anomaly that the distilled VLM could still answer the questions accu rately even when the related visual regions were masked or replaced with noise . Building on this observation, our quantitative and qualitative experiments demon strate that OPD, which uses the answer as a hint, has failed to teach students to genuinely seek out and rely on visual evidence . Our theoretical analysis suggests that this failure arises from a shortcut enabled by the lack of visual constraints dur ing distillation . To address the issue, we introduce Contrastive Vision On-policy Distillation (CV-OPD), which compares teacher distributions induced by positive and negative crops while conditioning on the same student prefix and augments optimization with a margin-based contrastive constraint, which is modulated with discrepancy gating across token and sample level . Specifically, CV-OPD first introduces a negative teacher unrelated to the answer and designs a Contrastive Divergence Loss (CDL) that constrains the feature distribution of student by con trasting the positive and negative teachers, correcting how the student exploits visual regions . The CV-OPD introduces a Discrepancy Aware Gate (DAG) that uses token-level and sample-level discrepancies to select visually relevant tokens for optimization,directing the student’s attention toward visual content . Extensive experiments across a broad range of fine-grained visual benchmarks demonstrate that CV-OPD substantially alleviates shortcut and consistently improves the visual reasoning capabilities of existing VLMs . All the reproduction code will be re leased on https://anonymous.4open.science/r/CV-OPD-9D04/ .

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, On-policy Distillation, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_utzsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper identifies a shortcut problem in Vision-OPD, where models can predict correct answers without relying on relevant visual evidence. To address this, the authors propose CV-OPD, which uses contrastive supervision and discrepancy-based gating to encourage better visual grounding.

Critical strengths
  1. Clear motivation. The paper identifies an important limitation of existing Vision-OPD methods and supports it with experiments on masked visual regions.
  2. Intuitive and well-structured method. The proposed CDL and DAG provide a straightforward way to introduce contrastive visual supervision into OPD. The overall framework is easy to follow and builds naturally on the existing Vision-OPD approach.
Critical weaknesses
  1. The evidence for improved visual grounding is not fully convincing. In Sec. 2.2, the base model already achieves around 42% accuracy when the answer-relevant evidence is masked, suggesting that visual shortcuts may already exist before OPD. The paper does not sufficiently explain why models can still produce correct answers without the necessary visual evidence, or whether the proposed method truly addresses the underlying issue. More controlled experiments and a deeper analysis of this behavior would strengthen the central claim.

  2. Limited improvement over strong baselines and unclear sources of performance gains. Although CV-OPD improves over Vision-OPD, its performance is very close to VAD (78.22% vs. 78.11%). It remains unclear whether this small improvement is meaningful and whether the gains from CDL and DAG come from better visual grounding rather than additional regularization or sample filtering. More targeted ablations and controlled comparisons would help establish the effectiveness of the proposed components.

Questions for authors
  1. Could the authors provide a more direct evaluation, such as comparing the effects of masking answer-relevant and irrelevant regions?
  2. How is the attention in Fig. 2(b) calculated? Is it averaged across the entire response sequence, or computed using specific tokens, layers, or attention heads?
  3. How sensitive is the method to incorrect positive crops or negative crops that still contain relevant evidence?
  4. The quality of Fig. 1(b) is difficult to read due to its low resolution.
  5. Why does Vision-OPD still achieve relatively high, even higher accuracy when the answer-relevant visual evidence is masked? A deeper analysis of why this happens would strengthen the paper's motivation and central claim.
  6. Can the authors demonstrate that the improvements from CDL and DAG come from better visual grounding rather than additional regularization or sample filtering?
Reviewer_sG4vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies on-policy distillation (OPD) for vision-language models and reports a diagnostic anomaly. After distilling a VLM with a crop-conditioned (privileged) teacher, the model still answers fine-grained questions correctly once the answer-relevant image region is replaced by black pixels or Gaussian noise. The authors argue that this reveals a visual shortcut, and attribute it to the absence of any constraint on which visual evidence the student uses. They propose CV-OPD, which adds a negative crop-conditioned teacher view and a margin-based Contrastive Divergence Loss (CDL), plus a Discrepancy Aware Gate (DAG) that restricts the contrastive term to high-discrepancy tokens and drops samples with no correct rollout. Evaluation is on six fine-grained visual benchmarks against OPD and RL baselines and open and closed VLMs, with qualitative masked-image examples. The work is an anonymous ICLR 2027 submission (preprint, not peer-reviewed).

Critical strengths
  1. The masking diagnostic is a useful, falsifiable finding (Fig. 1a, Fig. 2a, §2.2). Showing that a distilled VLM can hold accuracy while the decisive region is corrupted turns an abstract grounding concern into a concrete, checkable failure mode that the community can test on its own checkpoints.
  2. The CDL gradient is principled and cheap (Eqs. 10-11, 15-18; Table 2). On re-derivation, the claim that the update is locally proportional to τ+−τ− checks out, which gives an interpretable contrastive signal. The ablation (Table 2) shows that both CDL and DAG contribute.
Critical weaknesses
  1. The methodological delta over concurrent work is not established, and the closest cited method matches or exceeds the reported result. V-Zero (arXiv 2606.25319, 24 June 2026), cited in §5.2, already combines a question-relevant regional crop with a negative visual view and gates dense token-level distillation on that contrast. VAD (arXiv 2607.28590, 30 July 2026), also cited, contrasts evidence-present and evidence-removed views. Both works predate the ICLR 2027 deadline of 25 September 2026, and the submission cites them, so a comparison was feasible. The public VAD 4B checkpoint released alongside arXiv 2607.28590 (global_step_80) reports an unweighted six-benchmark average of 78.32 under the Vision-OPD official pipeline and the GPT-OSS-120B judge, against CV-OPD's 78.22 (Table 1; the submission's own Table 1 lists VAD at 78.11, so under either source the gap is ≤0.11). The paper's headline "+2.3% over Vision-OPD" (§4.2) is real, but the closest cited method matches it on the identical six benchmarks, with no demonstrated margin. This novelty concern bears directly on the rating.
  2. The load-bearing grounding claim (contribution 3) is not quantitatively supported for CV-OPD. §2.2 gives Vision-OPD's masked V* (43.46 black / 48.69 noise) and the base model's (42.41 / 42.93), but I could not find any numeric masked-V* accuracy for CV-OPD anywhere in the text or tables. It appears only as a bubble in Fig. 1(b) and as unlabeled bars in Fig. 2(a), plus qualitative Figs. 4-7. Combined with the absence of seeds and error bars, and with V*'s 191 questions (115 attribute + 76 spatial; the V*Bench paper, CVPR 2024, arXiv 2312.14135), the Base→Vision-OPD black-mask gap (+1.05 pts ≈ 2 questions) is plausibly noise. The "+2.3 Avg6" and "substantially/consistently" claims (§1, §4.2, Abstract) are asserted rather than statistically established.
Questions for authors
  1. Contribution 3 states that masked-V* demonstrates stronger visual reliance, but CV-OPD's masked-V* accuracy is only plotted (Fig. 1b, Fig. 2a). Please report CV-OPD's black-mask and Gaussian-noise masked-V* accuracies under the exact §2.2 protocol, with a paired test against Vision-OPD. If the grounding advantage is within noise, the central claim changes materially.
  2. How is the positive region ρ+ obtained in Eq. 7? Table 5 lists "Positive teacher image field: bbox_images", which suggests answer-related bounding boxes, while the same data is described elsewhere as auto-synthesized. If ρ+ depends on region annotations rather than Vision-OPD-style automatic crops, does the method still apply without such bboxes, and is ρ+ shared with the baseline?
  3. What is the precise methodological delta over V-Zero's contrastive evidence gating? V-Zero (arXiv 2606.25319) already combines a question-relevant regional crop with a negative visual view and gates dense token-level distillation on that contrast. Please identify which component of CV-OPD is not already covered by that design, ideally with a matched-budget comparison.
  4. Tables 1-4 are single runs with no seeds, error bars or significance tests; some benchmarks are small (for example V* has 191 questions). Please report multi-seed variance or paired tests, and confirm that baselines (VAD, V-Zero, VA-OPD) were tuned with a matched budget and identical data. Also, DAG's batch-mean threshold (Eq. 12) makes the selected token set batch-dependent; how stable is training as a function of batch composition?
  5. Section 4.2 notes that all distillation methods underperform the base model on HR-Bench-4K, and the paper has no Limitations section. A characterization of when the shortcut/contrastive fix helps and when it hurts (for example 9B backbone, thinking mode, other corpora and the ~10% overhead claim) would materially strengthen the contribution.
1 comment
AuthorsOpenReview ↗

Reminder of Discussion

Dear sG4v,

Thank you for your time and effort. We were wondering whether our responses have addressed your concerns and whether you have any further questions. Please feel free to share any remaining concerns and we would be happy to discuss how we can further improve this work.

Best regards,

Authors

Reviewer_PNYWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies at on-policy distillation for VLMs where the teacher is given an answer-relevant crop. It observes that a Vision-OPD student keeps answering visual questions after the relevant region is masked, and calls this a shortcut. The fix adds a second teacher that sees a random low-overlap crop, a hinge loss that pulls the student toward the crop teacher and away from the random-crop teacher, and a token gate that applies the hinge only where the student and crop teacher disagree more than the batch average. With Qwen3.5-4B the method gains +2.3 points over Vision-OPD on six benchmarks and lowers masked accuracy.

Critical strengths

The paper's main strengths are as follows.

  • The masked-input probe is simple and cheap, and the paper runs it on the base model as well as the distilled one.

  • The added cost is small. The negative teacher shares weights with the positive one, and training time grows by about 10%.

  • The ablations cover the main choices, with three alternative negatives and two gate thresholds.

Critical weaknesses

The main weaknesses are as follows.

  • Tables 2 to 4 report only accuracy, so the ablations never show which part lowers masked accuracy, and the grounding claim rests on one number from one 191 set. In Fig. 2(a) the VAD bar is the highest under both masks while VAD ties CV-OPD on average. Either masked accuracy is a poor shortcut measure or a method built for visual attribution fails at that, and neither is discussed. Attention mass inside the GT box over all of unmasked V* would be a better test than the examples in Figs. 4 and 5.

  • Figs. 5 to 7 show CV-OPD writing that the region is not visible and then picking an option anyway. A model that is more cautious, or weaker on hard cases, also loses masked accuracy. Nothing in the paper separates “says it cannot see” from “uses the right region”.

  • Sec. 2.3 blames the shortcut on the student fitting the full image and text. CV-OPD leaves this unchanged. The student still sees only the full image and both losses act only on output distributions. The negative teacher changes the target, not what the student can use, so the paper’s own argument applies to CV-OPD as well.

  • VAD and V-Zero also contrast positive and negative views to shape the target. The difference here is a random crop instead of an evidence-removed view, plus a hinge. Table 3 tries black and neighbor negatives but not an evidence-removed one, which is the comparison that would show the choice matters. The gain over VAD is 0.1 points.

  • In Table 3, sharing one negative crop across rollouts versus one per rollout moves Avg6 by 1.0, about the same as adding CDL at all (1.4). The variance explanation is not measured. With μ = 0.1 the hinge is zero once d− exceeds d+ by 0.1, and the gate drops below-mean tokens, so CDL may touch few positions. That fraction is never reported, so it is hard to tell whether CDL or training noise produces the gain.

  • The gate calls tokens with high JSD to the positive teacher “visually relevant”, but high disagreement can come from wording, formatting, or early tokens, and the selected tokens are never inspected. The sample filter b is counted as part of DAG, yet Sec. 4.1 lists it as setup, so it is unclear whether the w/o DAG rows remove it or whether the Vision-OPD baseline already uses it.

Questions for authors

The questions are as follows.

  • In Vision-OPD-6K the student image carries a drawn red box and the prompt says to focus inside it. Which image field and which prompt does the CV-OPD student use? If the box is drawn, how does the student learn where to look without it at test time?
  • Both Vision-OPD-6K and V* use SA-1B images. Was image overlap checked?
  • Does the positive teacher see the full image plus the crop (Eq. 2) or the crop alone (Eq. 8)?
  • Why does VAD score higher than Vision-OPD under masking?
  • How many of the 6,241 samples are dropped by the all-wrong filter, and is the same filter used for the Vision-OPD baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.