acceptodds
Withdrawn from ICLR 2027

Beyond Output Alignment: Capability Decomposed On Policy Latent Distillation for Vision Language Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy distillation (OPD) has achieved strong results in LLMs by providing dense teacher supervision on student-visited states, reducing the train–inference mismatch. However, OPD supervision is mediated through next-token distributions: it teaches the student how the teacher would act, but does not explicitly transfer the intermediate representations supporting visual grounding and reasoning. Distilling the internal visual-reasoning states is promising, yet it faces two critical challenges: teacher and student representations may fall into different spaces, and their reasoning behaviors can differ substantially in wording, length, and granularity. Addressing these challenges, we introduce CD-OPLD, a capability-decomposed framework for latent distillation in VLMs. CD-OPLD starts from a structured cold-start checkpoint, then learns capability-specific visual and reasoning projectors to align student and teacher representations. During on-policy training, student-generated trajectories are paired with answer-correct teacher reference trajectories, and a monotonic semantic block matcher enables direct latent alignment between semantically corresponding visual and reasoning states. Distilling Qwen3.5-9B into Qwen3.5-2B, CD-OPLD consistently improves over a cold-start-matched OPD baseline across seven multimodal benchmarks. Code is available at https://anonymous.4open.science/r/CD-OPLD-889F/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VLM Post-Training, On-Policy Distillation, Multimodal Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_as6aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CD-OPLD adds latent supervision to on-policy distillation (OPD) for VLMs in three stages. Stage 1 is a structured SFT cold start with <visual>/<reason>/<final answer> tags. Stage 2 pretrains visual and reasoning projectors with InfoNCE (student layer 20 to teacher layer 28). Stage 3 adds a cosine latent loss to reverse-KL OPD, pairing blocks of the student rollout with blocks of an answer-correct teacher rollout by monotonic dynamic programming. With a Qwen3.5-9B teacher and a 2B student trained on 8K ViRL39K samples, mean best@5 over seven benchmarks rises from 84.35 to 85.86.

Critical strengths
  1. The alignment problem is real. Student and teacher trajectories differ in length and wording, so their latent states do not line up by position. Monotonic many-to-many block matching handles this reasonably, and Table 2 shows it matching or beating whole-span and same-position matching.
  2. The setup is controlled and well documented. All methods share data, models and training hyperparameters, and Apps. B to F give full details, so the comparisons are easy to reproduce.
Critical weaknesses
  1. The gains are too small to separate from noise, and the claim of consistent improvement does not hold. Over cold-start OPD, acc@5 gains are mostly between +0.01 and +1.1. CD-OPLD is worse on ViRL8K (53.35 vs. 53.75) and on CRPE best@5, and it is below CompoDistill on POPE and MME, both of which are near ceiling. Every stage uses one seed, and no variance is reported.
  2. The closest baselines are missing, and extra signal and compute confound the comparison. OPLD, OPRD, VA-OPD and Decomposed OPD are not compared, and it is unclear whether CompoDistill and LaViT start from the same cold-start checkpoint. CD-OPLD also gets GT-filtered teacher references, five teacher rollouts per sample and a contrastive pretraining stage, with no compute cost reported, so the gain may come from this extra information rather than from latent alignment. The ablations lack visual-only and reasoning-only latent losses and sensitivity to λ and the matching threshold, and the error analysis counts corrections in one direction only.
Questions for authors
  1. Please report mean ± std over at least 3 seeds, with paired significance tests, for CD-OPLD and cold-start OPD.
  2. Please compare against OPLD and OPRD, and start CompoDistill and LaViT from the same cold-start checkpoint.
  3. Could the gain come from the GT-filtered teacher references? How does OPD do with the same references, and what is the extra training cost?
  4. At the 0.92 threshold, what fraction of rollouts and chunks get a matched block? How sensitive are the results to the threshold and to λ_v and λ_r?
  5. In the error analysis, how many cases does OPD get right that CD-OPLD gets wrong? How were error categories labeled?
Reviewer_RE4cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes an on-policy RL framework to transfer reasoning capabilities from teacher to student vision-language models. The paper proposes to align internal representations using cross-trajectory semantic block matching to address differences in reasoning trajectories together with a projector pretraining. The rest of the framework follows the standard 2-step on-policy training setup. The evaluation is done based on a Qwen3.5-9B teacher and a Qwen3.5-2B student. The evaluation shows improved performance for the student and is better than or on par with compared approaches.

Critical strengths
  • On-policy distillation is an interesting and relevant topic.
  • I personally like the idea of cross-trajectory semantic block matching.
Critical weaknesses
  • Limited comparison to state-of-the-art: The paper does not compare to other latent methods (e.g. LOPD, https://github.com/bingreeky/LOPD, or OPLD), but I'm also not sure if the latent representation is a significant part of the contribution, but the method name and abstract imply it.
  • Incremental performance gains: The overall performance increase compared to OPD is marginal. Based on Tab. 2, the latent matching strategies do not show a significant contribution in general.
  • Missing references: It would be good to see some more comparisons, e.g., to general on-policy distillation methods such as KDRL (Kdrl: Post-training reasoning LLMs via unified knowledge distillation and reinforcement learning) or VOLD (VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation).
Questions for authors
  • Results are pretty close to OPD. Has this been used as the base framework?
  • Is there any previous literature on cross-trajectory semantic block matching in LLMs?
Reviewer_wC2HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets on-policy distillation (OPD) for vision-language models and argues that supervising only the next-token distribution leaves a "capability-state gap": the student learns what to emit, but not what to represent internally about the image or the evolving reasoning state. Motivated by an error analysis in which most residual failures of an OPD-trained student are attributed to visual perception, the authors propose CD-OPLD, a three-stage pipeline. Stage 1 performs structured cold-start SFT on answer-correct teacher rollouts formatted as <visual>/<reason>/<final answer> spans. Stage 2 trains two student-side MLP projectors (visual, reason) with InfoNCE so that mean-pooled span representations, using wrong rollouts of the same sample and other samples in the batch as negatives. Stage 3 combines token-level reverse-KL OPD on student rollouts with a cosine latent loss between semantically matched blocks of the student rollout and a selected answer-correct teacher reference.

Critical strengths

-Clear and well-motivated problem framing: the "capability-state gap" is a real limitation of output-space OPD, and the paper grounds it in an error analysis rather than asserting it.

  • Interesting choice of alignment granularity. Rather than pairing individual tokens (which tends to match unrelated states) or pooling whole spans (which averages away decisive evidence), the paper aligns at the level of a semantic chunk.
Critical weaknesses
  • Missing comparison to output-space methods that target the same problem. VA-OPD, Decomposed OPD/VGS, and VAD are discussed in Related Work as motivation but didn’t run as baselines. Since these methods are orthogonal to LD, the paper cannot show that latent supervision is needed rather than a better-targeted output loss.
  • Stage 2 is not ablated. The paper treats projector pretraining as "an integral part of LD," so there is no random-projector, MSE-projector, or jointly-trained-projector control. Without this, the interpretation that Stage 2 learns a meaningful cross-model alignment (vs. the latent loss acting as a generic regularizer) is unsupported.
  • No conceptual justification for aligning hidden states across heterogeneous models. Hidden-state coordinates are arbitrary up to invertible transforms; the paper never discusses why a learned map should exist (e.g., model stitching, representation similarity literature) or analyzes the learned projectors. "Matching dimensions and distilling" needs an argument.
Questions for authors

[question]

  • Choice of cosine is unmotivated. Cosine discards the norm of the hidden state, which has been reported to carry confidence/information content
  • Cold start appears to cost hallucination/coverage on general benchmarks. All structured-cold-start variants fall below the base student on POPE and MME, while the non-structured baselines are fine.

[minor]

  • Table 1 is hard to read: acc@5 and best@5 are stacked in one cell without labels, and the ∆rel column is in a different row convention; two tables or an explicit row label would help.
  • How the error cases are analyzed?
  • How much wall-clock time does Stage 3 take relative to OPD alone (rollout generation, double teacher-forcing, DP)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.