- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies cross-modal on-policy distillation (OPD) for MLLMs. The teacher (Qwen3-VL-8B-Thinking) sees a privileged modality (depth or video) and the student (2B/4B) sees only RGB. LAMP reweights the per-token reverse KL with two terms. The first is a learnability gate g_t built from two teacher sensitivity signals, Δ_priv (with vs. without the privileged input) and Δ_stud (clean vs. perturbed RGB). The second is a horizon bonus b_t computed from the student-teacher gap over the next k tokens. LAMP has the best average across six benchmarks (2B: 51.4 vs. 49.9; 4B: 58.3 vs. 56.5), and the paper adds self-distillation, ablation and sensitivity results.
Critical strengths
- The problem is well posed. In privileged-modality OPD, some teacher tokens depend on information the student cannot recover, so imitating all tokens equally can hurt. Separating privilege-grounded from privilege-mediated tokens distinguishes this setting from X³-OPD and CORD.
- The method is simple and plugs into existing OPD, needing only forward passes through the frozen teacher. It beats OPD fairly consistently across two student sizes, two modalities and self-distillation (Tables 1 and 2), and it loses less on off-target benchmarks (Table 4).
Critical weaknesses
- The gains are small, each comes from a single run, and the hyperparameters appear to be tuned on test sets. Several margins are 0.4 or less (EmbSpatial 72.9 vs. 72.8, V* 80.2 vs. 79.8), and the 2B VisuLogic scores are close to chance (about 25%). No seeds or CIs are given. In Table 5, RoboSpatial moves by about 5.6 points across hyperparameter settings, and no validation split is mentioned.
- The gating mechanism is never checked directly, and an important control is missing. The only evidence that g_t separates the two token types is two examples (Figs. 5 and 6), and the RGB perturbation behind Δ_stud is not specified. The paper also lacks OPD with the same 8B teacher seeing only RGB, so the gain cannot be attributed to the privileged modality rather than to reweighting. The cost of three teacher forward passes is not reported, the horizon term is close to CORD's weighting, and ViCuR is cited but not compared.
Questions for authors
- What exactly is the RGB perturbation in the depth and video settings, and how sensitive are the results to it? In the video setting, which frames does the student see?
- Please add OPD with an RGB-only 8B teacher, and LAMP weighting computed with that RGB-only teacher.
- Please report mean ± std over at least 3 seeds for Tables 1 and 3, and explain how hyperparameters were selected.
- Can you validate g_t on labeled tokens (depth-only vs. inferable from RGB)? A gate-only ablation (λ=0) would also help.
- What are the extra training time and memory relative to OPD and CORD, and how does LAMP compare at matched compute?
Rating Update
Thank you for your response. My concerns have been addressed, and I would like to raise my rating.