acceptodds
Withdrawn from ICLR 2027

Visual Causal Transport Distillation via Counterfactual Relative Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual on-policy self-distillation transfers privileged visual information mainly through a teacher's output predictions, leaving how visual evidence is used implicit. In a training-set diagnostic, image-removed accuracy rises from 35.8% to 41.6% after Vision-OPD, suggesting that part of the acquired knowledge is accessible through image-independent priors. We introduce Visual Causal Transport Distillation (\method), which augments output distillation with counterfactual relative alignment of hidden representations. For each student-generated prefix, an EMA teacher is evaluated with privileged visual input and with its visual embeddings zeroed. Their contrast provides a reference for aligning the student's final representation and cross-depth updates toward evidence-conditioned computation. A dimensionless distance ratio performs relative alignment, while intervention energy and bounded evidence-gap weights focus supervision on visually sensitive tokens and trajectories. Across six benchmarks, \method achieves average accuracies of 72.10% and 75.11% with Qwen3.5-4B and Qwen3.5-9B, outperforming Vision-OPD by 1.39 and 3.57 percentage points, respectively. \method adds only one intervened teacher pass during training and leaves student inference unchanged.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_quxhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates on-policy self-distillation in vision-language models, aiming to help student models learn not only the teacher's predictions but also how the teacher uses visual information. Through preliminary experiments, the authors find that models trained with Vision-OPD can answer more questions without images. Vision-OPD mainly transfers knowledge through output distributions and lacks direct supervision of how visual information is used. To address this issue, the authors propose Visual Causal Transport Distillation (VCT), which combines counterfactual visual interventions with hidden representation alignment. Experiments are conducted on Qwen3.5-4B and Qwen3.5-9B across six benchmarks, including HR-Bench, MMStar, TreeBench, Zoom, and MathVista. The results show that VCT achieves improvements in average accuracy over Vision-OPD on both models. In addition, ablation experiments analyze the effects of components such as hidden representation alignment and dynamic weight allocation on model performance.

Critical strengths
  1. The proposed counterfactual alignment provides a meaningful way to incorporate visual evidence into hidden representation distillation.
  2. The method achieves consistent improvements over Vision-OPD across two model sizes and six benchmarks, supporting its effectiveness.
Critical weaknesses
  1. My main concern is that the paper does not directly verify whether VCT improves the model's reliance on visual evidence. The motivating image-removed experiment is only reported for Vision-OPD, while the main experiments focus on benchmark accuracy. Therefore, it remains unclear whether the improvements actually come from better use of visual information.
  2. The contribution of the proposed counterfactual relative alignment is not sufficiently demonstrated. Although the paper includes several ablation experiments, it does not compare the proposed objective with standard hidden representation alignment methods, such as MSE. Without this comparison, it is difficult to determine whether the performance gains come from the proposed relative alignment or simply from additional hidden representation supervision.
Questions for authors
  1. Figure 2 shows that image-removed accuracy increases after Vision-OPD training. Have the authors conducted the same evaluation for VCT? I am curious whether VCT actually improves the model's reliance on visual evidence.
  2. Have the authors compared the proposed counterfactual relative alignment with standard hidden representation alignment methods, such as MSE? This would help clarify whether the improvements come from the proposed objective or simply from additional hidden representation supervision.
  3. What is the contribution of aligning cross-depth representation updates? Have the authors tested the method using only final-layer hidden representation alignment?
  4. I am a little confused about the results in Table 2. The caption states that the no-hidden-loss run is still pending, but the table already reports accuracy for this setting. Are these results final?
  5. The negative teacher is constructed by setting visual embeddings to zero, which may introduce out-of-distribution effects. Have the authors tested whether the method is sensitive to this choice of intervention?
Reviewer_4XhPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Visual Causal Transport Distillation (VCT), an on-policy self-distillation method designed to improve fine-grained visual reasoning in multimodal language models. Instead of distilling only the teacher’s final output, VCT also tries to transfer how the teacher uses visual evidence and performs intermediate reasoning. For each prefix generated by the student, the same exponential moving average (EMA) teacher is evaluated in two settings. In the positive branch, the teacher is given a privileged crop containing relevant visual information. In the negative branch, the projected visual embeddings are set to zero, effectively removing the visual input. The difference between the teacher’s responses in these two branches is then used as a counterfactual reference. This reference guides the student’s final hidden representation and the changes in its representations across different layers. VCT also uses intervention energy and differences in output distributions to decide how much supervision should be assigned to different representations, tokens, and rollouts. Experiments on six multimodal benchmarks show consistent performance improvements.

Critical strengths

1.The method is well motivated and clearly presented. VCT naturally extends output-only on-policy distillation by introducing controlled positive and negative teacher branches. The difference between these branches provides supervision at the representation level, allowing the student to learn more than just the teacher’s final predictions.

2.The technical description is detailed and generally clear. The paper carefully explains the hidden representations used for supervision, shared normalization, the relative-distance objective, intervention-energy weighting, evidence-gap allocation, and distributed reduction procedure.

3.The results are promising at two model scales. VCT achieves the highest reported average accuracy among the compared methods for both the 4B and 9B backbones. The improvements appear across several fine-grained visual benchmarks rather than being limited to a single task.

Critical weaknesses

1.The paper does not directly verify its main claim that VCT teaches the student to rely more on visual evidence. The image-removal experiment used to motivate the method is conducted on training questions. However, the paper does not report the same diagnostic results after training with VCT. It also does not evaluate the student under image removal, image swapping, controlled visual counterfactuals, or other interventions that could reveal changes in the student’s internal computation. Therefore, the current accuracy improvements do not clearly show that VCT makes the student use visual evidence more faithfully. It is also possible that the method mainly works as an effective additional regularizer for hidden representations.

2.The comparisons and ablation studies do not clearly isolate the contributions of the main technical components. Although Figure 1 presents uniform hidden-state alignment as an important alternative, the experiments do not compare the proposed relative-ratio objective with a standard hidden-state MSE objective using the positive teacher. The paper also does not separately evaluate the effects of final-state alignment, cross-depth alignment, intervention-energy weighting, and output-based gating. In addition, the evaluation does not include closely related counterfactual visual-distillation methods such as VAD. As a result, it is difficult to determine whether the improvements come specifically from the proposed relative alignment and cross-depth transport mechanisms, or more generally from adding hidden-state supervision and counterfactual weighting.

Questions for authors
  1. Could the authors directly evaluate whether VCT makes the student rely more on relevant visual evidence? For example, could they compare the initial model, Vision-OPD, and VCT under image removal, image swapping, irrelevant image crops, or controlled counterfactual images on a held-out test set?
  2. Could the authors measure the student’s own response to visual interventions? For example, they could compare the student’s hidden states or output distributions with and without visual input, and then examine whether the student’s response becomes more similar to the teacher’s intervention effect. This would provide more direct evidence that VCT transfers evidence-conditioned computation.
  3. Under the same teacher branches and computational budget, could the authors compare the proposed relative-ratio objective with standard hidden-state MSE, margin-based contrastive or triplet objectives, and other methods for aligning positive and negative representations?

Overall, I appreciate the motivation of this paper and the thought that went into the proposed method. If the authors provide a strong and convincing rebuttal that adequately addresses the concerns above, I would consider raising my score.

Reviewer_wDB9OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the question of how to transfer visual evidence-dependent computations during on-policy self-distillation beyond merely matching output predictions. This paper introduce Visual Causal Transport Distillation (VCT) that uses counterfactual visual interventions to guide relative alignment of hidden representations and cross-depth updates. Experimental results show that VCT consistently outperforms Vision-OPD across benchmarks and model scales.

Critical strengths

The problem is interesting. This paper identifies that matching predictions does not explicitly constrain how visual evidence influences intermediate computations in Vision-OPD. The combination of final-state alignment and cross-depth updates is an interesting attempt to transfer evidence-sensitive computations beyond output distributions.

Critical weaknesses
  1. The paper motivates VCT using increased image-removed training accuracy after Vision-OPD. However, this observation does not establish that Vision-OPD weakens visual grounding, nor does the paper demonstrate that VCT reduces reliance on image-independent shortcuts. The reported improvements primarily concern standard benchmark accuracy, without corresponding image-removal, image-swapping, or controlled counterfactual evaluations of the trained VCT models. Consequently, the proposed contrast measures conditional model sensitivity rather than necessarily identifying visually grounded causal computations.

  2. VCT introduces an additional counterfactual teacher forward pass and hidden-state supervision, resulting in increased training computation. However, the experiments control training epochs rather than actual computational budgets, without reporting training FLOPs, wall-clock time, or GPU-hours. Consequently, it remains unclear whether the improvements over Vision-OPD would persist under matched compute budgets.

Questions for authors
  1. Could the authors provide direct evidence that VCT improves visual grounding rather than only benchmark accuracy?

  2. Could the authors report the additional training FLOPs, wall-clock time, and memory overhead introduced by VCT, and compare against Vision-OPD under matched computational budgets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.