acceptodds
Under review as a conference paper at ICLR 2027

Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization

Abstract

Recent post-training methods, particularly Reinforcement Learning with Verifiable Rewards (RLVR), have significantly enhanced the reasoning ability of Large Vision-Language Models (LVLMs). However, the sparse nature of verifiable rewards provides little token-level supervision for failed rollouts, often leading to inefficient exploration in complex multimodal reasoning tasks. Although policy distillation can offer dense guidance, external teacher based methods introduce substantial computational overhead, while answer conditioned tuning methods potentially expose answer-level information and induce shortcut-like generation behavior. To overcome above limitations, we propose a novel Privileged Tutoring Distillation Policy Optimization (PTD-PO) method for RLVR, whose core idea is to supplement supervision with guidance on the reasoning path rather than the answer itself. Specifically, our PTD-PO first constructs Way-revealing Structured Privileged Hints by considering both spatial attention guidance and intermediate textual reasoning steps, which are further employed to guide the student model toward corrective reasoning paths through in-context-learning. particularly, the student model is still optimized under the original question-only context, while only failed rollouts are aligned with the hint-augmented reference model at the token-distribution level. To further stabilize distillation under the distribution shift between guided and unguided contexts, we introduce a Top-K Jensen-Shannon divergence with tail compensation, focusing on aligning informative tokens while reducing memory overhead. Experiments on LVLMs ranging from 2B to 8B parameters show that our PTD-PO can provide effective dense guidance without relying on answer-revealing supervision and consistently outperforms state-of-the-art counterparts by a clear margin.

Primary area:
causal reasoning
Keywords:
Multimodal Reasoning, Reinforcement Learning, Policy Distillation, Large Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_r2SLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PTD-PO, a multimodal RLVR framework that uses structured, answer-free privileged hints to guide a frozen reference model and distills its token-level distributions into the student on failed trajectories.

Critical strengths
  1. The paper addresses the genuine challenge of sparse outcome rewards in multimodal RLVR, particularly the lack of informative supervision for failed trajectories.
  2. PTD-PO achieves strong results across Qwen3-VL models from 2B to 8B parameters, outperforming several RLVR and self-distillation baselines on diverse multimodal reasoning benchmarks.
  3. The combination of structured privileged hints, failure-routed distillation, and Top-K JSD with tail compensation is intuitive and supported by several useful ablations.
Critical weaknesses
  1. The paper assumes that answer-revealing distillation induces shortcut behavior and harms reasoning exploration. However, the evidence in Figures 3 and 5 is not sufficient to establish this claim. Shorter responses, larger distribution shifts, and higher success rates under CoT+GT conditioning do not necessarily indicate undesirable shortcuts. Consequently, the central motivation for replacing answer-revealing supervision with way-revealing hints remains insufficiently validated.
  2. Although PTD-PO outperforms HDPO, these methods differ in more than just the privileged information provided to the teacher. A comparison within the same PTD framework, varying only the teacher context (e.g., GT answer, CoT+GT, regular hints, and WSPH), is necessary to establish whether the proposed hint design is genuinely responsible for the improvement.
  3. The paper claims that WSPH provides useful reasoning guidance without revealing answer-identifying information. However, it lacks quantitative analysis of hint quality, implicit answer leakage, and the reliability of the filtering process. Furthermore, the additional cost of generating and validating hints with large external models is not sufficiently quantified.
Questions for authors
  1. What evidence demonstrates that answer-revealing distillation actually harms downstream reasoning? In particular, could the shorter responses and higher success rates observed under CoT+GT conditioning simply reflect more informative teacher supervision rather than shortcut behavior? Can the authors provide direct evidence from the trained student policies?
  2. Does WSPH still outperform answer-revealing supervision when the distillation framework is held fixed? Specifically, what happens if the proposed failure-routed PTD objective is used with GT-answer or CoT+GT teacher contexts instead of WSPH?
1 comment
AuthorsOpenReview ↗

Controlled Validation of Structured Privileged Guidance under Fixed Distillation Settings

W2 & Q2: Controlled Validation of Structured Privileged Guidance

We thank the reviewer for this valuable comment. Following the same experimental settings as those described in our response to W1 & Q1, we conducted additional experiments to further investigate this issue. The results are presented in Table C, followed by our analysis.

Controlled comparison within the PTD framework. Under the identical failure-routed PTD framework (Appendix E.4 protocol), WSPH achieves the highest mean accuracy (61.27%), outperforming CoT+GT (59.66%) by 1.61 pp and GT-only (56.68%) by 4.59 pp. These results directly confirm that WSPH provides additional benefits beyond the PTD optimization framework itself.

Cross-framework validation with HDPO. The advantage of WSPH also holds under HDPO, improving mean accuracy from 57.64% to 58.99% (+1.35 pp) over CoT+GT. Moreover, PTD consistently outperforms HDPO under both teacher contexts (+2.02 pp with CoT+GT; +2.28 pp with WSPH), further demonstrating the complementary contributions of structured guidance and the PTD framework.

Table C. Student accuracy under fixed-framework comparison.

Framework/contextMMK12MathVerseMMMU-ProLogicVistaMean
HDPO/CoT+GT (w/o. failure rollout sampling)63.4773.4637.0756.5657.64
PTD/regular hint (w. failure rollout sampling)69.0377.1337.3854.9559.62
PTD/GT only (w. failure rollout sampling)65.8772.9834.6153.2556.68
PTD/CoT+GT (w. failure rollout sampling)69.2877.4236.3155.6459.66
PTD/WSPH (w. failure rollout sampling)71.3377.5638.5657.6361.27
Reviewer_1Nt7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the lack of fine-grained learning signals for failed rollouts in multimodal reinforcement learning with verifiable rewards (RLVR). It proposes PTD-PO, which combines GRPO with distillation from a frozen reference model conditioned on Way-revealing Structured Privileged Hints (WSPH). Stronger models construct these hints offline from the image, question, and ground-truth answer. The hints identify relevant visual evidence and partial reasoning directions, with a refinement process intended to remove answer leakage. The student receives only the original problem, while the reference model supplies token distributions on the student's sampled prefixes. Distillation is restricted to failed trajectories.

Critical strengths
  1. The overall improvements hold across three model sizes and multiple tasks. PTD-PO achieves the highest overall score in all three model groups in Table 1. The 2B improvement over GRPO, from 50.63 to 61.21, is particularly substantial and suggests that auxiliary guidance can meaningfully complement sparse rewards. Table 3 reports improvements from 63.91 to 65.83 with DAPO and from 69.27 to 70.60 with GSPO, supporting some compatibility across optimization objectives. These are advantages in aggregate performance, rather than universal task-wise superiority: for example, HDPO outperforms PTD-PO on MMMU-Pro at 8B.

  2. The design targets a concrete failure mode and includes relevant component analyses. The student requires no privileged hints at inference, and distillation focuses on failed trajectories. In the 4B results of Table 5, failure-routed WSPH achieves 69.12, compared with 67.46 for distillation on all trajectories and 67.82 for regular hints. The 8B results also favor structured hints. Appendix D.3 explains the tail bucket rigorously, Appendix E.2 evaluates K=50/100/200, and Appendix E.3 reports time per step. Together, these analyses support a contribution beyond a new prompting template.

Critical weaknesses
  1. The central mechanistic explanation is not fully established by the evidence. Figure 3 shows that CoT+GT produces larger distribution shifts and shorter outputs, while Figure 5 shows more difficult questions reaching 16/16 correct responses. These observations are also consistent with a teacher simply having access to a complete solution; they do not independently demonstrate that its supervision induces harmful shortcuts in the final student evaluated without hints. WSPH, GT-only, and CoT+GT differ simultaneously in information content, context length, visual structure, and supervision difficulty. A controlled comparison that changes only the context while holding routing, JSD, loss strength, and budget fixed is missing. The regular-hint ablation also removes visual grounding and trap filtering together. This limits the main scientific explanation: evidence that the method works should be distinguished from evidence that it works specifically by preventing answer shortcuts.

  2. Hint auditing and experimental documentation are insufficient to fully assess the source and cost of the gains. Appendix C.1 describes an independent verifier and provides the generation prompt, but does not identify the verifier, its checking prompt, rejection/regeneration rates, Gemini fallback frequency, or a human audit of residual leakage and incorrect guidance. The issue is the unquantified reliability of existing quality control. The total cost comparison also omits offline hint construction with the 235B model and Gemini. In addition, the actual λ values by configuration and variation across independent training runs are not reported, and the reward used to identify failures is unclear. These omissions make it difficult to separate the effects of structured guidance, additional teacher information, and tuning, and to assess the scope of the efficiency claim.

Questions for authors
  1. [Mechanism] In a common 4B setting, could the authors hold failure routing, JSD, λ, and training budget fixed and compare WSPH, GT-only, and CoT+GT, reporting the final student's accuracy and entropy/diversity without hints? Which objective or routing differences remain in the current HDPO comparison? A persistent advantage under these controls would strengthen the proposed explanation.

  2. [Hint quality and cost] Which independent verifier, rules, and prompts are used? How many candidates are rejected, regenerated, or sent to Gemini? Please provide a human audit of leakage and incorrect guidance with a stated sample size, together with offline construction costs. Absence of an explicit final answer should be distinguished from absence of answer-identifying cues.

  3. [Objective and reproducibility] Do ri and the group mean in Equation (5) use binary answer accuracy or the combined 0.1 format + 0.9 accuracy reward? If the latter, under what conditions does the all-fail, zero-advantage analysis in Appendix D.4 apply? Please specify λ for each model, how the KL penalty enters Equation (7), and the number of independent training runs and associated uncertainty.

3 comments
AuthorsOpenReview ↗

Comprehensive Hint Quality Audit, Leakage Assessment, and Cost Analysis

W2 & Q2: Hint Quality, Leakage Audit, and Construction Costs

We thank you for this valuable comment. To address these concerns, we conducted an additional fine-grained audit of all accepted hints and provide further details on the hint generation process and associated costs.

Specifically, Qwen3-VL-235B-A22B-Thinking generated 34,027 accepted hints (87.54%) with a 16,384-token generation budget and up to three retries, while Gemini-3.0-Pro handled the remaining 4,843 hints (12.46%). During manuscript preparation, Gemini was also employed for lightweight filtering according to the criteria described in Appendix C.1.

For the additional quality audit, we employed Qwen3.5-397B-A17B as an independent evaluator, providing it with the image, question, hint, and GT answer for each of the 38,870 accepted hints. As reported in Table D, the evaluation identified 0.00% explicit final answers, 5.70% decisive intermediate information, 13.30% answer-space-narrowing cues, 0.00% complete solution traces, and 0.60% incorrect guidance. These categories are non-exclusive and the results are subject to evaluator error. This finer-grained analysis distinguishes the absence of explicit answer disclosure from the absence of implicit answer-identifying cues. In addition, an earlier manual inspection of 100 hints found no direct answer disclosure or misleading guidance.

Regarding computational costs, the reported USD 22 accounts for Gemini fallback generation. Since the offline hints are reused throughout training, the comparison of 67.1 versus 3,167 context tokens demonstrates a substantial reduction in online context consumption, rather than representing the total end-to-end construction and training cost.

Table D. Automated labels.

LabelRate
Explicit final answer0.00%
Decisive intermediate5.70%
Complete solution trace0.00%
Answer-space-narrowing cue13.30%
Incorrect guidance0.60%
AuthorsOpenReview ↗

Clarification of Reward Definitions, KL Regularization, and Training Reproducibility

Q3: Reward Definitions, KL Regularization, and Reproducibility

We sincerely thank you for this valuable comment. To address these issues, we clarify the distinction between routing and optimization rewards, the conditions underlying the all-fail analysis, and the role of KL regularization.

Reward definition and all-fail analysis. Specifically, routing uses binary correctness (aia_i), whereas GRPO uses the combined reward (ri=0.1fi+0.9air_i = 0.1f_i + 0.9a_i), where fif_i denotes the format reward. The mean correctness in Eqs. (3)–(5) is based on binary correctness, with ai=0a_i = 0 for incorrect answers during routing. In Appendix D.4, all-incorrect groups have zero advantages only when their format rewards are identical; Otherwise, differences in format rewards can produce non-zero advantages. This analysis excludes the separate KL term.

KL regularization and PTD coefficients. We also clarify that Eq. (1) omits KL regularization for simplicity, while the actual minimized loss is −Jclip+0.01LKL+λLPTD-J_{\mathrm{clip}} + 0.01L_{\mathrm{KL}} + \lambda L_{\mathrm{PTD}}. Accordingly, Eq. (7) uses JGRPO=Jclip−0.01LKLJ_{\mathrm{GRPO}} = J_{\mathrm{clip}} - 0.01L_{\mathrm{KL}}. The PTD coefficients are λ=0.05/0.5/0.05\lambda = 0.05/0.5/0.05 for the 2B/4B/8B models, respectively, and 0.50.5 for DAPO and GSPO.

Thank you for the detailed response. I have carefully considered the authors' clarifications and will maintain my score.

Reviewer_nM2sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PTD-PO aims to address the lack of dense supervision for failed trajectories in multimodal GRPO. A strong external model and training answers are used offline to construct text-based structured way-revealing guidance. In the training, the paper combine GRPO and OPD for trajectory distillation with privileged tutoring based on the offine guidance. Experiments cover Qwen3-VL-Thinking models at 2B, 4B, and 8B across nine reasoning benchmarks.

Critical strengths
  1. The idea of guidance on the reasoning path is better than answer itself is well motivated. The specific design choice of PTD-PO is also reasonable.
  2. The main results cover multiple scales Qwen3-VL 2B to 8B and nine different multimodal reasoning tasks. The result outperform the displayed baselines. The paper also includes substantive component ablations, supporting its choices within the proposed framework.
Critical weaknesses
  1. The core framework closely overlaps with direct prior work. Vision-OPD already uses a crop-conditioned teacher to supervise a full-image student in the multimodal setting [1]. HDPO combines GRPO with privileged self-distillation on all-fail prompts and uses JSD [2]. SRPO routes failed samples with available teacher information to self-distillation and the remaining samples to GRPO[3]. DOPD study using multi-modal reasoning path rather than answer as privilegae in OPD[4]. In this case, the differences of PTD-PO between these previous work are more like specific algorithm details and the strong privilegaed guidance from a larger offline model.
  2. The experiment do not show the influence of the offline-generated strong privilegaed guidance (structured or normal guidence) compared to pure self-distillation. The performance gain may come from the relatively high quality CoT from larger model.
  3. Moreover, considering SRPO is also a close related work for this work, the comparison should include it.

[1] Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation. [2] HDPO: Hybrid Distillation Policy Optimization via Privileged Self-Distillation. [3] Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing [4] DOPD: Dual On-policy Distillation

Questions for authors
  1. What is the main differences between PTD-PO and the four work mentioned in weaknesses section.
  2. What is the result of SRPO?
  3. If HDPO and SRPO use the same strong privilegaed guidance in the OPD, is there still advantage for PTD-PO?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.