acceptodds
Withdrawn from ICLR 2027

Reinforcement Learning with Visually Attributable On-Policy Self-Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal reinforcement learning with verifiable rewards (RLVR) provides reliable task-level supervision, yet offers limited guidance for localizing errors in visual reasoning. On-policy self-distillation (OPSD) complements this supervision with dense token-level corrections from a privileged teacher equipped with an evidence-centered visual view. However, the resulting corrections may conflate visual evidence with non-visual shifts induced by linguistic priors, response style, or the privileged view itself. We formalize this phenomenon as privileged-correction contamination: a token-level correction is contaminated when its update direction cannot be consistently attributed to the privileged visual evidence. Directly distilling contaminated corrections may therefore reinforce erroneous token-level updates. To address this issue, we propose reinforcement learning with visually attributable on-policy self-distillation, an evidence-consistent paradigm that augments task-level RLVR supervision with privileged token-level corrections grounded in visual evidence. We instantiate this paradigm with Visual-Attribution Gating (VAG), which uses the verifier-derived advantage as the optimization anchor and validates each privileged correction against the contrast between evidence-present and evidence-degraded teacher views. By admitting only corrections directionally supported by this visual contrast, VAG preserves visually grounded guidance while suppressing inconsistent residuals. Experiments across six visual reasoning benchmarks show that VAG consistently outperforms GRPO and OPSD methods, establishing new state-of-the-art average performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reinforcement Learning, On-Policy Self-Distillation, Visual-Attribution Gating

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UQCgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Visual-Attribution Gating (VAG), which combines outcome-level RLVR supervision with token-level on-policy self-distillation for visual reasoning. A privileged teacher evaluates student-generated trajectories using an evidence-centered crop and a randomly masked version of that crop. The method retains a correction only when its sign agrees with the visual contrast between the two teacher views. Experiments shown that the proposed method can obtain improvement gains compared to previous RL methods.

Critical strengths

-The paper addresses an interesting problem, that a correct final answer does not provide precise supervision for intermediate visual decisions.

-The method is simple and introduces no additional trainable modules or inference-time burden.

-The method achieves good results on several benchmarks.

Critical weaknesses

-I have some concerns regard the novelty compared to VAD (Zhang et al., 2026). VAD already identifies mixed-source teacher corrections and uses a one-sided projection onto a visual-evidence direction to retain aligned components. In related work, it mentioned VAD but only briefly categorizes it as target reconstruction. VAG differs by applying a sign-agreement gate to corrections and incorporating them into a verifier-anchored GRPO advantage. The paper does not discuss its significance or provide a direct comparison with VAD under the same settings.

-The paper aims to learn from visual evidence that both supports and contradicts a generated token. However, Eq. 9 gives smaller weights when the visual contrast is more negative. This means that corrections that argue against a token may have very little effect, even if they pass the gate.

-The robustness of the incremental gains remains unclear. The average improvement over OPD-V is 1.32 percentage points on the 4B model, but only 0.22 points on the 9B model. No multi-seed results or uncertainty estimates are reported. Table 3 also shows substantial sensitivity to the correction weight and temperature.

K. Zhang et al. VAD: Attributing Visual Evidence for Target Reconstruction in Multimodal On-Policy Distillation. arXiv:2607.28590.

Questions for authors

See the weaknesses.

Reviewer_HWYnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies privileged-correction contamination in multimodal on-policy self-distillation. The proposed Visual-Attribution Gating (VAG) method combines verifier-based advantages with token-level teacher corrections, admitting a correction only when its direction agrees with the contrast between evidence-preserving and evidence-degraded teacher views. Across six visual reasoning benchmarks, VAG achieves 81.33% average accuracy on Qwen3.5-4B, compared with 80.01% for the strongest baseline, OPD-V.

Critical strengths

The paper is clearly written and addresses a relevant limitation of privileged distillation. The reported 39.1% rate of conflicting token-level signals offers useful motivation. VAG is simple, adds no trainable parameters, and switches between verifier supervision and token-level teacher corrections, favoring verifier supervision when a correction is rejected. Comparisons show competitive baselines, holdout evaluations, and ablations provide encouraging empirical evidence.

Critical weaknesses
  1. Sign agreement between the teacher correction $\delta_t$ and visual contrast vtv_t establishes directional consistency, but not that the correction is primarily attributable to visual evidence. Since δt=vt+(log⁡pT−−log⁡pS)\delta_t=v_t+(\log p_{T^-}-\log p_S), an accepted correction may still be dominated by non-visual residuals. The 39.1% conflict rate alone does not show that rejected corrections are harmful or retained ones are beneficial. A direct validation of correction quality would strengthen the claim.
  2. Equation (9) uses v~t=σ(vt/τ)\widetilde v_t=\sigma(v_t/\tau), which strongly downweights negative contrasts even when δt<0\delta_t<0 and vt<0v_t<0 pass the gate. This seems inconsistent with the stated goal of preserving both supporting and refuting visual evidence. Please justify this design and compare it with a magnitude-based weighting function.
  3. Table 4 introduces both gtg_t and the bounded tanh⁡(δt/τ)\tanh(\delta_t/\tau) transformation in the final comparison. The resulting improvement cannot be attributed specifically to the sign gate. A no-gate variant with identical bounded scaling, and ideally random gating at a matched acceptance rate, would be more conclusive.
  4. Gains over OPD-V are modest (1.32 points on 4B and 0.22 on 9B). Additionally, Figures 8 and 9 appear to reuse an unrelated OPD-V helmet response, while Figure 5 describes a correct prediction although the displayed counting answer is incorrect.
Questions for authors

See weakness.

Reviewer_KaFjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VAG, a method that extends on-policy self-distillation for fine-grained visual perception. A positive teacher uses an annotated zoomed crop to provide token-level corrections, while a negative teacher uses a randomly masked crop to measure visual support. Corrections are applied only when the two signals agree in sign, then weighted by a sigmoid function and bounded by tanh before being added to the GRPO advantage. Experiments use Qwen3.5-4B and 9B models trained on 6.2K Vision-OPD examples, with evaluations on six high-resolution benchmarks.

Critical strengths
  1. Comprehensive comparisons under matched settings. The paper compares SFT, GRPO, DAPO, OPSD, Vision-OPD, VA-OPD, and OPD-V using the same backbone, training data, and evaluation protocol. It also includes four held-out benchmarks and ablations of negative views, making the improvements easier to assess.
Critical weaknesses
  1. The improvement over Vision-OPD is sensitive to hyperparameter choices and lacks statistical validation. VAG improves the average score from 77.10 to 81.33, but both τ\tau and η\eta are selected on the six test benchmarks. The chosen value of 0.1 is also the smallest tested for both. Increasing η\eta to 0.2 reduces performance to 73.89, below Vision-OPD and the ungated variant (77.12). Similarly, τ=0.5\tau=0.5 gives only 74.34. All results come from single runs, and V* contains only 191 questions. Moreover, Table 4 changes the sign gate and tanh bound together, leaving the gate's individual contribution unclear.

  2. The weighting mechanism does not fully support its claims. With σ(vt/τ)\sigma(v_t/\tau) and τ=0.1\tau=0.1, a negative visual signal of vt=−0.5v_t=-0.5 receives a weight of only 0.007. This effectively suppresses strong refutation corrections, despite the claim that both support and refutation are preserved. Meanwhile, tokens with little visual evidence vt≈0v_t\approx0 still receive a weight near 0.5. In addition, Fig. 1(b) shows most signals concentrated near zero, where sign disagreements may arise from noise. The reported 39.1% conflict rate therefore does not establish meaningful contamination.

Questions for authors

Clarifications That Could Change My Assessment:

  1. How does the conflict rate change with the magnitudes of vtv_t and δt\delta_t? For example, what happens after excluding tokens with ∣vt∣<0.05|v_t|<0.05?

  2. Given the sigmoid weighting, what proportion and total magnitude of applied corrections come from the support and refutation quadrants?

  3. In Fig. 6, the student's image contains a red box, and the prompt explicitly refers to it. Does the student receive this annotation during training? The test images do not contain such boxes.

  4. Could the authors compare VAG with Vision-OPD and, if feasible, OPD-V across three seeds, selecting τ\tau and η\eta on a held-out validation split rather than the test benchmarks?

Suggestions:

  1. The OPD-V outputs in Fig. 8 (bottom) and Fig. 9 appear identical to the helmet example in Fig. 8 (top), suggesting a possible copying error?

  2. Please specify the judge model and its size, report the training cost associated with three forward passes per rollout, and restrict claims of surpassing GPT-5.4 and Qwen3.5-397B to high-resolution fine-grained benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.