- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates on-policy self-distillation in vision-language models, aiming to help student models learn not only the teacher's predictions but also how the teacher uses visual information. Through preliminary experiments, the authors find that models trained with Vision-OPD can answer more questions without images. Vision-OPD mainly transfers knowledge through output distributions and lacks direct supervision of how visual information is used. To address this issue, the authors propose Visual Causal Transport Distillation (VCT), which combines counterfactual visual interventions with hidden representation alignment. Experiments are conducted on Qwen3.5-4B and Qwen3.5-9B across six benchmarks, including HR-Bench, MMStar, TreeBench, Zoom, and MathVista. The results show that VCT achieves improvements in average accuracy over Vision-OPD on both models. In addition, ablation experiments analyze the effects of components such as hidden representation alignment and dynamic weight allocation on model performance.
Critical strengths
- The proposed counterfactual alignment provides a meaningful way to incorporate visual evidence into hidden representation distillation.
- The method achieves consistent improvements over Vision-OPD across two model sizes and six benchmarks, supporting its effectiveness.
Critical weaknesses
- My main concern is that the paper does not directly verify whether VCT improves the model's reliance on visual evidence. The motivating image-removed experiment is only reported for Vision-OPD, while the main experiments focus on benchmark accuracy. Therefore, it remains unclear whether the improvements actually come from better use of visual information.
- The contribution of the proposed counterfactual relative alignment is not sufficiently demonstrated. Although the paper includes several ablation experiments, it does not compare the proposed objective with standard hidden representation alignment methods, such as MSE. Without this comparison, it is difficult to determine whether the performance gains come from the proposed relative alignment or simply from additional hidden representation supervision.
Questions for authors
- Figure 2 shows that image-removed accuracy increases after Vision-OPD training. Have the authors conducted the same evaluation for VCT? I am curious whether VCT actually improves the model's reliance on visual evidence.
- Have the authors compared the proposed counterfactual relative alignment with standard hidden representation alignment methods, such as MSE? This would help clarify whether the improvements come from the proposed objective or simply from additional hidden representation supervision.
- What is the contribution of aligning cross-depth representation updates? Have the authors tested the method using only final-layer hidden representation alignment?
- I am a little confused about the results in Table 2. The caption states that the no-hidden-loss run is still pending, but the table already reports accuracy for this setting. Are these results final?
- The negative teacher is constructed by setting visual embeddings to zero, which may introduce out-of-distribution effects. Have the authors tested whether the method is sensitive to this choice of intervention?