- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes Visual Contrastive Self-Distillation (VCSD), an on-policy self-distillation method for vision-language models that requires no external teacher, privileged answers, or additional visual-evidence annotations. At each student-generated response prefix, the same EMA teacher predicts next tokens under the original image and a same-size black-image control. Their log-probability difference reweights the original-image distribution within a plausibility support, and the student learns the resulting target through forward KL. Only the updated student is used at inference. Experiments on ViRL39K cover six Qwen3-VL/Qwen3.5 configurations and seven benchmarks, with additional training on Vision-OPD6K and ablations of the target construction and teacher updates. The seven-benchmark mean improves from 72.51 to 76.26 for Qwen3-VL-8B and from 74.97 to 79.24 for Qwen3.5-9B relative to the corresponding base models.
Critical strengths
Teacher conditioning on answers or extra visual evidence can introduce preferences that depend on information unavailable to the student. VCSD constructs its target from paired visual conditions of the same teacher, anchored by the original-image prediction. It reduces supervision requirements by avoiding reference-answer rewards and additional evidence annotations, while retaining a single student model at inference.
Across all six model configurations, Table 1 reports aggregate gains of 1.86–4.80 percentage points over the base models and 1.76–4.61 points over answer-hint OPSD. The no-contrast, sharpening-only, inference-only, frozen-teacher, and alternative-control-image experiments examine the roles of the main components. On Qwen3-VL-2B, forward KL reaches 67.49 versus 64.43 for reverse KL, while the frozen teacher reaches 65.03 versus 67.49 for the full method. Table 6 also reports three independent training seeds at this scale.
Critical weaknesses
The core idea of this paper is to extract learning signals related to instance-specific visual information by comparing the model's predictions under the original image and content-erased/control image conditions, and then use these signals for self-distillation. However, using changes in model predictions under different visual conditions to measure visual dependency is not first proposed in this paper. For example, prior work such as VPPO in the RL setting and the more recent VA-OPD adopt similar ideas, which weakens the novelty of this work. In comparison, the main distinction of VCSD appears to lie in using visual contrast to directly reconstruct a plausible full-distribution teacher target, rather than performing advantage shaping, loss reweighting, or token selection. Therefore, I believe the paper should more clearly articulate its methodological novelty and distinguish its contributions from existing visual-dependency and visual-contrast approaches.
The main training experiments in this paper are conducted on ViRL39K, a dataset containing a substantial amount of data involving mathematical, geometric, and spatial reasoning, as well as chart understanding, all of which are closely related to multimodal reasoning. However, the current evaluation benchmarks are primarily focused on visual perception, fine-grained visual understanding, and hallucination. Among the seven main benchmarks, MathVista is the only one that directly evaluates visual mathematical reasoning. Given the close connection between ViRL39K and multimodal reasoning capabilities, I suggest including additional multimodal reasoning benchmarks, such as MMMU and MMMU-Pro, to provide a more comprehensive evaluation of the proposed method.
The baseline comparisons are also insufficient. Regarding methods related to OPSD, the main results table only includes a comparison with standard OPSD. Considering that several multimodal OPD/OPSD methods have recently been proposed, I suggest including at least a few closely related and competitive baselines under the same backbone, training data, and optimization budget. This would provide a more convincing demonstration of the advantages of the proposed method over existing approaches.
Questions for authors
The paper presents thorough experiments and demonstrates substantial performance improvements. However, I would like the authors to more clearly articulate the methodological novelty of their approach relative to existing visual-dependency and visual-contrast methods, particularly explaining why this work goes beyond simply transferring existing visual-contrast approaches from other domains to OPSD. In addition, I encourage the authors to include more evaluations on benchmarks closely related to the training data, such as MMMU, and to provide additional comparisons with existing OPSD-related methods.
If all these concerns are adequately addressed, I would consider raising my score to 3/4.
4 comments
W2. Evaluation on MMMU and MMMU-Pro
We thank the reviewer for suggesting additional evaluation on MMMU and MMMU-Pro, which provide complementary assessments of multimodal reasoning performance.
Following this suggestion, we evaluated all five methods (Base, SFT, GRPO, OPSD, and VCSD) on MMMU (validation) and MMMU-Pro using Qwen3.5-4B and Qwen3.5-9B.
The evaluations follow the same decoding and evaluation settings used in the main paper, where applicable.
MMMU (validation)
| Backbone | Base | SFT | GRPO | OPSD | VCSD (ours) |
|---|---|---|---|---|---|
| Qwen3.5-4B | 68.78 | 70.22 | 69.00 | 70.56 | 70.89 |
| Qwen3.5-9B | 69.00 | 72.22 | 72.11 | 67.00 | 73.78 |
MMMU-Pro
| Backbone | Base | SFT | GRPO | OPSD | VCSD (ours) |
|---|---|---|---|---|---|
| Qwen3.5-4B | 58.55 | 59.19 | 57.69 | 55.03 | 60.46 |
| Qwen3.5-9B | 60.23 | 61.79 | 62.25 | 55.84 | 65.20 |
VCSD achieves the highest accuracy among the evaluated methods across all four backbone-benchmark combinations.
At the 9B scale, VCSD obtains 73.78 on MMMU and 65.20 on MMMU-Pro, compared with 72.11 and 62.25 for GRPO, corresponding to improvements of +1.67 and +2.95 points, respectively.
At the 4B scale, VCSD also achieves the best results on both benchmarks, with 70.89 on MMMU and 60.46 on MMMU-Pro.
The absolute improvements over the base model are generally smaller than those observed on the perception-oriented benchmarks in the main paper. This is consistent with the nature of the ViRL39K training data and VCSD's emphasis on visually grounded prediction rather than explicit reasoning supervision.
Nevertheless, the consistent ranking across both additional benchmarks provides evidence that VCSD's improvements are not restricted to the original evaluation suite and can extend to more reasoning-oriented multimodal tasks.
Both result tables have been included in Appendix B of the revised manuscript.
W3. Comparisons with additional OPD-family baselines
We agree with the reviewer that comparisons with closely related OPD-family approaches are important for establishing the empirical contribution of VCSD.
We would like to clarify the existing comparisons and report an additional baseline evaluated in response to this concern.
1. Answer-hint OPSD (Table 1)
The main experiments already include the standard answer-hint OPSD baseline across all six evaluated backbone configurations.
This baseline uses privileged reference answers to condition the teacher, whereas VCSD constructs its supervision using only image-question pairs.
All methods are trained on the same ViRL39K dataset with the same 90-step optimization budget, and method-specific hyperparameters are provided in Appendix A.
2. Vision-OPD (Table 2)
We also compare VCSD with Vision-OPD, a recent multimodal OPD approach that constructs visual-evidence supervision using bounding-box-annotated cropped teacher views.
To ensure a controlled comparison, we reproduced Vision-OPD using the same training data and the same number of optimization steps across three backbone configurations.
| Backbone | Vision-OPD (same-data reproduction) | VCSD (ours) | Improvement |
|---|---|---|---|
| Qwen3-VL-4B | 70.34 | 73.82 | +3.48 |
| Qwen3.5-4B | 74.23 | 76.96 | +2.73 |
| Qwen3.5-9B | 77.15 | 79.26 | +2.11 |
VCSD outperforms the reproduced Vision-OPD baseline across all three configurations, with improvements ranging from +2.11 to +3.48 points.
An important difference is that VCSD does not require the bounding-box annotations used by Vision-OPD.
For completeness, Table 2 also reports results from the officially released Vision-OPD checkpoints under our evaluation protocol, together with a joint variant combining the two supervision signals.
3. Additional VA-OPD-style baseline
Following the reviewer's suggestion, we additionally evaluated a VA-OPD-style baseline under matched backbone, data, and optimization budget.
Specifically, we adapted the grouped token-reweighting mechanism to the self-distillation setting using a degraded visual control view.
We emphasize that this is a VA-OPD-style adaptation, rather than an exact reproduction of the original method, which uses an external 8B teacher. This adaptation allows us to examine the effect of contrast-based loss reweighting under a controlled training setup.
| Method (Qwen3-VL-2B) | 7-benchmark average |
|---|---|
| Answer-hint OPSD (privileged reference answers) | 63.63 |
| VA-OPD-style grouped reweighting (self-distillation adaptation) | 64.40 |
| GRPO (reward-based reference) | 65.59 |
| VCSD (ours; image-question pairs only) | 67.49 |
Under these matched settings, VCSD achieves the highest accuracy, outperforming both the answer-hint OPSD baseline and the VA-OPD-style reweighting adaptation.
Together with the Vision-OPD comparisons in Table 2 and the contrast-utilization study in W1, these results further support the effectiveness of VCSD's full-distribution target construction over the evaluated alternatives.
We hope these additional comparisons and clarifications address the reviewer's concerns regarding methodological novelty and baseline coverage.
As the discussion period is coming to a close, we wanted to briefly follow up on our rebuttal. If any of your concerns remain, we’d be happy to clarify. If our response addresses your main concerns, we’d appreciate it if you would consider revisiting your rating. Thank you!
We thank the reviewer for the constructive feedback and suggestions regarding the methodological novelty of VCSD, evaluation on additional benchmarks, and comparisons with related OPD-family methods. We have conducted several new experiments to address these concerns and incorporated the results into Appendix B of the revised manuscript.
W1. Methodological novelty relative to visual-dependency methods (VPPO, VA-OPD)
We appreciate the opportunity to clarify the methodological distinction between VCSD and prior methods that exploit visual dependency signals.
The central distinction is how the visual contrast is used to construct the learning objective.
Prior visual-dependency methods, such as VPPO and VA-OPD, generally use visual contrast to modulate an existing learning signal, for example through advantage shaping, loss reweighting, or token selection based on an underlying reward or supervision target.
In contrast, VCSD uses the visual contrast to construct the supervision target itself. Specifically, the contrast determines a full-distribution self-distillation target, rather than serving as an auxiliary weight on an existing reward- or label-based objective.
Consequently, VCSD does not require reward labels, reference answers, or external evidence annotations to construct its training supervision.
We agree that sharing a visual contrast signal does not, by itself, establish a substantive methodological difference. To investigate whether the distinction in its utilization matters empirically, we conducted an additional controlled comparison.
Controlled study: Different usages of the same visual contrast signal
We evaluated the same visual contrast signal Δ under five different applications, using matched backbone, training data, and experimental settings (Qwen3-VL-2B, seven-benchmark average).
The results reveal a consistent difference between the evaluated ways of utilizing the same visual contrast.
Simply incorporating the contrast signal into an existing objective does not reproduce VCSD's improvement. The trajectory-gating and token-reweighting variants achieve 63.48, 64.40, and 64.69, respectively, all below the GRPO reference of 65.59.
Constructing a full-distribution distillation target yields the strongest result among the evaluated approaches. VCSD achieves 67.49, exceeding both GRPO and the alternative contrast-utilization variants.
These results suggest that the benefit of VCSD is not attributable solely to the availability of a visual contrast signal; the way this signal is incorporated into the learning objective is an important factor.
We emphasize that these experiments compare representative implementations of different contrast-utilization mechanisms under controlled settings, rather than claiming to reproduce every detail of the original methods.
We have added this study to Appendix B to better substantiate the methodological distinction.