acceptodds
Under review as a conference paper at ICLR 2027

RGB-OPD: Representation-Guided Budgeting for On-Policy Distillation in Multimodal Reasoning

Abstract

On-policy distillation (OPD) transfers reasoning capabilities from a teacher to a student on student-generated trajectories, providing supervision at the states actually visited during training. However, we find that image-induced changes are distributed much more broadly in teacher representations than in sampled-token output probabilities, leaving many representation-sensitive positions under-emphasized by standard OPD in multimodal reasoning. Motivated by this representation–output mismatch, we propose RGB-OPD, a representation-guided budgeting framework for multimodal on-policy distillation. RGB-OPD uses teacher representation sensitivity to rank response positions, keeping distillation entirely in the output space. It applies coarse distributional supervision over a compact support constructed from both teacher and student distributions, and reallocates a fixed supervision budget toward the most representation-sensitive positions. A bounded termination correction further addresses truncated reasoning trajectories. Across five out-of-domain multimodal reasoning benchmarks, RGB-OPD improves Standard OPD by 0.80 percentage points in the Qwen3-VL 8B2B run and by 0.91 percentage points in the 32B8B run, with higher point estimates on all five benchmarks at both scales.

Primary area:
foundation or frontier models, including LLMs
Keywords:
On-Policy Distillation, Multimodal Reasoning, Knowledge Distillation, Vision-Language Models, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_o25VOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates where on-policy distillation of VLMs should concentrate token-level supervision. Given that image-induced changes are distributed more broadly, this paper proposes RGB-OPD for multimodal on-policy distillation. RGB-OPD uses teacher representation sensitivity to maintain distillation in the output space, outperforming standard OPD by 0.91% for 32B to 8B run.

Critical strengths
  1. Teacher representation sensitivity is used to only rank positions, and the distillation target stays in the output space. This avoids the width and layer matching problems of hidden-state alignment.

  2. This paper gives honest reporting of the scores, like outperforming standard OPD by 0.91% for 32B to 8B run, together with +2.54 on MathVerse, +0.96 on VisOnlyQA, and +0.92 on WeMath.

Critical weaknesses
  1. The effect of representation-guided budgeting is not separated from the other changes, and the gains are small.

Relative to Standard OPD, RGB-OPD changes three things at once: the objective (sampled-token surrogate to coarse top-K reverse KL), the H-guided budget, and the termination hinge. Table 1 contains neither "Standard OPD + termination correction" nor a uniformly weighted coarse-KL baseline, so the +0.80/+0.91 gains cannot be attributed to H.

The only direct test of H is in Figure 4: seed 42, two of five benchmarks, 8B→2B only. RGB-OPD outperforms "w/o H-Budget" by 0.63 on MathVerse (about 5 of 788 items) and by 0.40 on MMStar (about 6 of 1,500). It outperforms Random by 0.89 and 0.74 points (7 and 11 items). Table 5 gives seed-to-seed SDs of the paired differences of 1.30 (MathVerse) and 1.11 (MMStar), so every one of these gaps is below 0.8 SD. This 42 can be a favorable seed, and the evidence is still within noise.

  1. The representation-output mismatch is measured against a narrow output-side probe, so the case for representations over the full output distribution is not made.

O uses only the sampled token's log-probability, while H aggregates relative ℓ2 changes over all layers and hidden dimensions. Appendix A.1 ("multi-token observability") argues that a broader output support exposes more of the displacement. The like-for-like comparison is a distribution-level output signal, for example, the JPEG-corrected KL between the teacher's next-token distributions under I and I_shuf, possibly on the same top-K support. That signal is not analyzed in Figure 2, and is also not tried as a selector.

RGB-OPD up-weights an output-space target at high-H positions. If the teacher's full distribution is image-sensitive there, a distribution-level output selector should find those positions without hidden states. If it is not, the up-weighted target carries little visual information, and it is unclear how matching it transfers visual reasoning.

It is important because the mismatch is the paper's first stated contribution and the justification for the cost of the probe.

Questions for authors
  1. Could you show the effect of representation-guided budgeting with clear separation from other changes?
  2. How could you justify that these small gains show the effectiveness of this proposed method?
  3. How does a distribution-level output-sensitivity selector compare with H, such as both in the Figure 2 overlap analysis and as a training selector with the same α and β?
  4. Is H correlated with the coarse discrepancy D_t or with student entropy at the same positions?
  5. How is OPRD-Bridge configured, e.g., which layers are aligned, how are teacher and student widths mapped, and what loss weight is used, does it include the termination correction? Because it nearly matches RGB-OPD on MathVerse at 8B→2B (25.76 vs. 25.89).
Reviewer_mvj6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RGB-OPD, an on-policy distillation method for multimodal reasoning. Its key observation is that image-induced changes are more broadly distributed in the teacher's hidden representations than in sampled-token output probabilities. Based on this observation, the method uses representation sensitivity HH to identify positions requiring stronger supervision, applies budgeted reverse-KL distillation over a compact output support, and introduces a termination correction for trajectories that fail to terminate properly.

Critical strengths
  • The observation of a representation–output mismatch is interesting and provides a new perspective on how output-side supervision may overlook visual information in multimodal OPD.
  • The paper is generally well written and clearly organized. The figures are informative, and the method, theoretical scope, and implementation details are presented thoroughly.
Critical weaknesses
  1. The empirical basis for using representation sensitivity as the budgeting signal requires further validation.

    • Mismatch measurement: Figure 2 compares high-dimensional, multi-layer hidden-state L2 changes with the log-probability change of a single sampled token, rather than with the full output distribution. Moreover, HH uses a non-negative distance, whereas OO takes the positive part of a signed change. The main findings should be validated using absolute log-probability changes or full-distribution metrics such as KL or JS divergence.
    • Distillation value: A large representation change under image perturbation does not necessarily indicate task-relevant visual reasoning or imply that the corresponding output distribution is more valuable to learn. The authors should examine whether high-HH positions correspond to answer-relevant visual reasoning steps and compare top-HH, bottom-HH, random, and output-based selectors under the same supervision budget.
  2. The experimental analysis could more clearly characterize the sources and nature of the reported improvements.

    • Component attribution: In Figure 4(a), the full method improves over Standard OPD by +2.03 points on MathVerse, but the improvement becomes −0.25 without the termination correction, while removing H-budgeting still yields +1.40. A complete factorial ablation across all five benchmarks is needed to isolate the contribution of representation-guided budgeting.
    • Termination correction: The analysis in Figure 4(c) is strongly coupled with the strict-match protocol, under which responses reaching the maximum length are automatically marked incorrect. Part of the gain may therefore come from avoiding automatic failures rather than improving reasoning quality. The paper should report accuracy excluding truncated responses, the correctness of newly terminated responses, and the relationship between response length and accuracy.
Questions for authors

See weakness

Reviewer_46hLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RGB-OPD uses changes in the teacher’s hidden states under image perturbation to decide which response positions should receive more distillation weight. The loss stays in output space, with a compact distributional objective and a separate correction for truncated responses. The paper evaluates two Qwen3-VL teacher–student pairs on five multimodal benchmarks.

Critical strengths
  1. The motivation makes sense. Figure 2 shows that looking only at the sampled token’s probability misses many positions where the teacher’s hidden states are sensitive to the image. Using this signal to choose where to put more weight, while keeping the target in output space, is a reasonable design.

  2. I appreciate that the authors include matched seeds and a comparison at roughly equal training time. RGB-OPD still improves on the longer Standard OPD run (Table 2). These controls make the positive results more credible.

Critical weaknesses
  1. My main concern is that the improvement is quite small. The average gain over Standard OPD is below one percentage point at both scales, and the gains over OPRD-Bridge are only 0.60 and 0.49 points (Table 1). MMBench and MMStar barely change in the main runs. The five-seed experiment is helpful, but some individual gains vary substantially: on MathVerse, the main run’s +2.03 becomes +0.56 ± 1.30 across seeds (mean ± SD; Table 5). There is no corresponding multi-seed result at the larger scale. I see evidence of a positive effect, but I am not yet convinced that the improvement is large or consistent enough to make a strong case for the method.

  2. It is also hard to tell how much the representation selector itself contributes. The matched selector ablations cover only MathVerse and MMStar at the smaller scale. removing the termination correction takes the MathVerse gain from +2.03 to −0.25 (Figure 4), so that separate component clearly matters. Given the 1.54× training time per step, I would like to see the hidden-state probe compared with cheaper signals, such as student uncertainty or teacher–student disagreement, with the loss and termination correction held fixed. The existing equal-time comparison is useful, but does not answer this question.

Questions for authors

Could you extend the matched selector ablations to the other benchmarks, keeping the termination correction fixed? Do cheaper uncertainty or disagreement signals give similar gains under the same loss and time budget? Do you have matched-seed results for the larger model pair?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.