acceptodds
Under review as a conference paper at ICLR 2027

LAMP: Learnability- and Horizon-Aware On-Policy Distillation from Privilege Modality

Abstract

Multimodal large language models (MLLMs) often struggle with perceptual information that is difficult to recover from RGB alone. Cross-modal on-policy distillation (OPD) addresses this issue by allowing the teacher to access privilege modalities such as depth or video, but existing methods treat token-level supervision uniformly, conflating transferable reasoning with unlearnable sensor information. Therefore, we propose LAMP, a learnability- and horizon-aware privilege-modality on-policy distillation framework for MLLMs. LAMP makes a two-fold contribution: First, it performs counterfactual modality attribution to identify whether each token’s teacher advantage is grounded in the student or privilege modality, enabling a self-calibrating gate to suppress untransferable supervision. Second, it introduces horizon-conditioned divergence weighting to emphasize tokens that have greater downstream influence on the reasoning trajectory. Extensive experiments on depth-to-RGB and video-to-RGB transfer show that LAMP consistently outperforms standard OPD and representative cross-modal distillation methods, achieving up to 6.1% and 10.7% performance gain over the strongest baselines on depth and video benchmarks, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, MLLM, reinforcement learning, on-policy distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6ersOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies cross-modal on-policy distillation (OPD) for MLLMs. The teacher (Qwen3-VL-8B-Thinking) sees a privileged modality (depth or video) and the student (2B/4B) sees only RGB. LAMP reweights the per-token reverse KL with two terms. The first is a learnability gate g_t built from two teacher sensitivity signals, Δ_priv (with vs. without the privileged input) and Δ_stud (clean vs. perturbed RGB). The second is a horizon bonus b_t computed from the student-teacher gap over the next k tokens. LAMP has the best average across six benchmarks (2B: 51.4 vs. 49.9; 4B: 58.3 vs. 56.5), and the paper adds self-distillation, ablation and sensitivity results.

Critical strengths
  1. The problem is well posed. In privileged-modality OPD, some teacher tokens depend on information the student cannot recover, so imitating all tokens equally can hurt. Separating privilege-grounded from privilege-mediated tokens distinguishes this setting from X³-OPD and CORD.
  2. The method is simple and plugs into existing OPD, needing only forward passes through the frozen teacher. It beats OPD fairly consistently across two student sizes, two modalities and self-distillation (Tables 1 and 2), and it loses less on off-target benchmarks (Table 4).
Critical weaknesses
  1. The gains are small, each comes from a single run, and the hyperparameters appear to be tuned on test sets. Several margins are 0.4 or less (EmbSpatial 72.9 vs. 72.8, V* 80.2 vs. 79.8), and the 2B VisuLogic scores are close to chance (about 25%). No seeds or CIs are given. In Table 5, RoboSpatial moves by about 5.6 points across hyperparameter settings, and no validation split is mentioned.
  2. The gating mechanism is never checked directly, and an important control is missing. The only evidence that g_t separates the two token types is two examples (Figs. 5 and 6), and the RGB perturbation behind Δ_stud is not specified. The paper also lacks OPD with the same 8B teacher seeing only RGB, so the gain cannot be attributed to the privileged modality rather than to reweighting. The cost of three teacher forward passes is not reported, the horizon term is close to CORD's weighting, and ViCuR is cited but not compared.
Questions for authors
  1. What exactly is the RGB perturbation in the depth and video settings, and how sensitive are the results to it? In the video setting, which frames does the student see?
  2. Please add OPD with an RGB-only 8B teacher, and LAMP weighting computed with that RGB-only teacher.
  3. Please report mean ± std over at least 3 seeds for Tables 1 and 3, and explain how hyperparameters were selected.
  4. Can you validate g_t on labeled tokens (depth-only vs. inferable from RGB)? A gate-only ablation (λ=0) would also help.
  5. What are the extra training time and memory relative to OPD and CORD, and how does LAMP compare at matched compute?
Reviewer_MeUDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a token-weighted on-policy distillation framework for transferring supervision from a teacher with privileged modalities to an RGB-only student. The method combines a modality-attribution gate, computed through teacher input interventions, with a horizon-conditioned weight based on subsequent teacher–student divergence. Experiments cover depth-to-RGB and video-to-RGB transfer across six benchmarks and two student sizes, with additional ablations and self-distillation evaluations.

Critical strengths

The paper identifies a meaningful distinction between teacher supervision supported by student-accessible information and supervision relying on information unavailable to the student. This provides a clear motivation for selectively weighting privileged-modality supervision rather than treating all token positions equally.

The proposed framework is clearly formulated and achieves consistently positive results across the evaluated benchmarks and student sizes. The ablations provide evidence that both components contribute: removing modality attribution reduces average accuracy across the three depth benchmarks by approximately 1.77 percentage points, while removing horizon weighting reduces it by approximately 0.70 percentage points.

Critical weaknesses
  1. The practical significance and reliability of the improvements are not sufficiently established. Many gains over the strongest competing baseline are around one percentage point or less, although larger improvements occur on some benchmarks. Such gains can be meaningful, but the paper does not report variability across independent training runs, making it difficult to determine whether the smaller improvements reliably exceed training noise. The method also requires scoring under full, ablated, and perturbed teacher inputs, however, the additional training cost is not quantified.

  2. The attribution gate may conflate the teacher’s modality preference with the student’s inability to learn from RGB. For example, occlusion cues in RGB may already establish which object is in front, but the teacher may rely primarily on depth. Removing depth could then substantially change its prediction, while perturbing RGB has little effect. The gate could consequently downweight supervision that the student could learn from RGB. Thus, sensitivity to modality interventions alone does not establish that the corresponding information is unrecoverable from the student’s input.

Questions for authors
  1. Could the authors report variability across independent training runs and quantify the additional training time or teacher-scoring cost relative to standard OPD? These results would help assess whether the relatively small gains are reliable and practically worthwhile.

  2. Is there evidence that the attribution gate distinguishes information genuinely unrecoverable from RGB from information that the teacher simply prefers to obtain through the privileged modality? An independently labeled or controlled evaluation, including performance as the proportion of unrecoverable supervision varies, would strengthen the central interpretation.

Reviewer_TP8FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LAMP, an on-policy distillation method for privileged-modality transfer. During training, the teacher has access to additional depth images or videos, while the student receives only RGB inputs. The method constructs a token-level attribution gate through modality removal and perturbation, and further adjusts distillation weights using teacher–student disagreement over subsequent tokens. Experiments cover depth-to-RGB and video-to-RGB transfer, along with self-distillation and component ablations.

Critical strengths
  1. The problem is meaningful. A teacher’s advantage may come from information unavailable to the student, making direct imitation less effective. The paper explicitly discusses this issue and offers a useful perspective on cross-modal distillation.

  2. The reported results are consistent. LAMP achieves the best results across the six benchmarks and two student sizes. Self-distillation experiments also show improvements.

Critical weaknesses
  1. The motivation lacks quantitative support. The paper does not directly establish that unrecoverable supervision is a major bottleneck in standard OPD. Although it explains why such supervision may be harmful, it does not measure its prevalence in the training data, show whether OPD failures concentrate on these cases, or demonstrate that removing them improves training.

  2. The evidence for the learnability claim is insufficient. Modality removal and perturbation measure the teacher’s sensitivity to its inputs, but this does not establish whether the student can recover the target from RGB alone. For example, the teacher may use RGB to identify an object and depth to determine its distance. Sensitivity to both modalities does not mean an RGB-only student can learn that judgment. The ablations demonstrate the gate’s utility, but do not validate its ability to distinguish recoverable from unrecoverable supervision.

  3. The justification for weighting current tokens by future disagreement is unclear. The score measures probability differences over subsequent tokens without intervening on the current token or comparing alternative continuations. Large disagreement may arise from upcoming difficulty, earlier errors, or privileged information rather than the current token’s influence. The results support this score as a useful heuristic, but its interpretation as downstream influence remains unverified.

  4. Some experimental details are missing. The RGB perturbation procedure and its strength are insufficiently specified, although these choices may directly affect attribution. The paper also does not report variability across training runs or the actual computational overhead of scoring three teacher configurations, limiting the assessment of reliability and efficiency.

  5. The figures and captions need improvement. Figures 1–3 have a noticeably AI-generated visual style, particularly the teacher icons in Figures 1 and 2, although appearance alone does not establish how they were created. More importantly, the captions provide insufficient explanation, especially for Figure 1, making the figures difficult to understand independently.

Questions for authors

My main concern is that the experiments show that both weighting signals help training, but do not establish that they measure learnability and downstream token influence. Removing depth or perturbing RGB reveals what information the teacher relies on, not whether the student can learn the target from RGB alone. For example, the teacher may use RGB to identify an object and depth to determine its distance; sensitivity to both does not make that distance recoverable from RGB. Likewise, high disagreement after a token does not establish its influence: the token may simply precede a difficult segment. Without replacing or correcting the token and examining the continuation, this interpretation remains unverified. Stronger evidence is needed to support these mechanistic claims. Thus,

  1. Can you directly validate whether the gate distinguishes RGB-recoverable supervision from supervision requiring privileged information?

  2. Does the future-disagreement score predict the effect of replacing or correcting the current token on subsequent reasoning or final accuracy?

  3. Can you quantify how often unrecoverable supervision occurs and whether removing it improves standard OPD?

1 comment
Reviewer_TP8FOpenReview ↗

Rating Update

Thank you for your response. My concerns have been addressed, and I would like to raise my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.