acceptodds
Under review as a conference paper at ICLR 2027

Self-Supervised Visual On-Policy Distillation

Abstract

Visual on-policy distillation relies heavily on an informative teacher-student asymmetry, through either a larger, stronger teacher or privileged supervision, such as reference answers or ground-truth regions of interest. This raises a fundamental question: where can informative asymmetry come from when nothing privileged is available? We answer this by inverting where the asymmetry comes from. Rather than adding privileged information to the teacher, we subtract information from the student. This asymmetry creates the same effective learning signal for free as a teacher with access to information unavailable to the student, without ground-truth annotations, rewards, or a separate stronger teacher model. Building on this principle, we introduce Self-Supervised Visual On-Policy Distillation (), a simple yet effective method that constructs on-policy learning signals from asymmetric augmented views. distills the teacher's distribution conditioned on the original image on-policy into the student distribution conditioned on a strongly augmented view of the same image. We systematically explore a broad design space of visual augmentations and uncover that (1) asymmetry matters: all four augmentation families improve performance, while symmetric self-distillation degrades it; (2) strength matters: performance peaks at a moderate strength; and (3) the gap must remain task-consistent: augmentations that completely remove the question-relevant evidence can induce large but uninformative discrepancies. Across six fine-grained perception benchmarks, improves Qwen3.5-4B from 70.7% to 77.4%, above all open-source models compared, up to Qwen3-VL at 235B, and surpasses GPT-5.4. While holding training data the same, it recovers 96% of the improvement achieved by methods with privileged information.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, vision-language models, large language models, on-policy self-distillation, self-supervised learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_cWZfOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Self-Supervised Visual On-Policy Distillation (S2VOPDS^2VOPD), a new method for visual on-policy distillation. S2VOPDS^2VOPD uses an EMA model based on the student model as the teacher model. The teacher observes the original image, while the student receives an augmented view of the same image, which creates a teacher–student asymmetry in their predictions. The student is trained on its own trajectories generated from the augmented view by minimizing the generalized Jensen–Shannon divergence to the token distribution of the teacher conditioned on the original image (on-policy distillation). The authors conduct experiments exploring the augmentation space to find the best configuration, and derive a best recipe that selects one of four augmentations (including no augmentation). Experimental results show that the 9B model trained with S2VOPDS^2VOPD outperforms the 397B-scale model and proprietary models, achieves better performance than the compared methods, and shows strong out-of-distribution generalization.

Critical strengths
  • S1. Strong empirical performance. Across six fine-grained perception benchmarks, the 9B model trained with S2VOPDS^2VOPD outperforms the 397B-scale model (Qwen3.5-397B), GPT-5.4, and Gemini-3-Flash. S2VOPDS^2VOPD also outperforms methods trained with privileged information and self-rewarding RL methods. On generalization to three math reasoning benchmarks, the 4B model underperforms the compared methods, while the 9B model outperforms them.

  • S2. Simple yet effective method. This paper brings training with augmented views, which is widely used in self-supervised and semi-supervised learning, into visual on-policy distillation, enabling training without privileged information. Although the technical novelty is limited, the method is simple and effective, and thus of high practical value.

Critical weaknesses
  • W1. Inconsistencies in reported results. Several numbers in the experiments are inconsistent: Base model accuracy: 70.58% in L258–259, but 66.5 in Figure 2(a)(b). Composition B at 4B: 77.28 in the left panel of Figure 2(c), but 76.33 in the right panel. Composition A at 4B: 76.94 in the right panel of Figure 2(c), but 75.89 in Figure 2(d). If the latter two pairs are results of the same configuration with different random seeds, accuracy varies by about 1 point across runs. In Figure 2, many conclusions, such as the comparison between composition and the best single augmentation and the effect of stacking weak augmentations, rely on differences smaller than 1 point. Since the number of seeds and the variance are not reported, it is impossible to distinguish whether these differences come from the augmentation configuration or from random variation.

  • W2. Unclear validity of the best recipe. Compositions A and B are introduced without any explanation of how they were selected, and the composition analysis (Figure 2(c)–(e)) is built upon them. Moreover, downscaling with Gaussian noise, which the single-augmentation analysis identifies as the most robust default, is not included in A or B, while mixed resolution, a key component of A and B, appears neither in the single-augmentation analysis nor in Table 5. Therefore, the validity of the best recipe, the conclusion of the analysis, is questionable.

  • W3. Minor weaknesses (not affecting my assessment): The “EMA teacher.” paragraph is duplicated in L457–461. It is not stated whether the student model or the EMA model is used for evaluation. There is no analysis of why S2VOPDS^2VOPD improves performance on math reasoning.

Questions for authors
  • Q1. Regarding W1, is my understanding of the reported results correct?
9 comments

We thank the reviewer for the constructive review and helpful suggestions. We also would like to express our gratitude to the reviewer for finding our work simple and effective with strong empirical performance.

>>> W1 & Q1 Several numbers are inconsistent: base model accuracy is 70.58% in L258–259 but 66.5 in Figure 2(a)(b); Composition B at 4B is 77.28 in the left panel of Figure 2(c) but 76.33 in the right panel; Composition A at 4B is 76.94 in the right panel of Figure 2(c) but 75.89 in Figure 2(d). If the latter two pairs are different seeds, accuracy varies by about 1 point, and many Figure 2 conclusions rely on differences smaller than that. Is my understanding of the reported results correct?

>>> A1 Thank you for pointing out these inconsistencies. We are deeply sorry for the misunderstanding and confusion caused. These inconsistencies arose from incomplete updates when we changed the evaluation settings, as well as from results imported from Vision-OPD [1]. The differing values for the same configuration reflect the previous and current evaluation protocols, rather than different training seeds. The two protocols are summarized below.

EvaluationPrevious protocolCurrent protocol
PerceptionGreedy decoding (T = 0), 2,048-token budgetGreedy decoding (T = 0), 4,096-token budget
MathT = 0.6, 16,384-token budgetT = 0.3, 24,576-token budget, top-p = 0.95, top-k = 20, presence penalty = 1.5

We adopted the current protocol because the smaller inference token budget in the previous protocol truncated reasoning traces, resulting in incomplete responses and lower performance. The current protocol yielded better base-model performance and closely reproduced the perception accuracy reported by Vision-OPD [1] (70.58% in our evaluation versus 70.67% in [1]). We subsequently re-evaluated all experiments and models using the current protocol. However, we inadvertently retained results from the previous protocol in parts of the text and Figure 2.

QuantityReported valuesExplanationCorrected value
No augmentation65.21% in Sec. 4.2 vs. 70.52% in Table 3Sec. 4.2 retained the result under the previous protocol70.52%
Base, 4B70.58% in L258–259 / 70.67% in the main table vs. 66.5% in Fig. 2a,bFigure 2 retained the previous result. The text reports our re-evaluation, while the main table cites the result from [1]70.58% for our evaluation; 70.67% as reported in [1]
Composition B, 4B77.28% in Fig. 2c (left) vs. 76.33% in Fig. 2c (right)The right panel retained the result under the previous protocol77.28%
Composition A, 4B76.94% in Fig. 2c (right) vs. 75.89% in Fig. 2dFigure 2d retained the result under the previous protocol76.94%

We agree that variation across training seeds should also be assessed, particularly for the smaller composition gains. We therefore repeated the main experiments and the composition-versus-single-augmentation comparisons with three training seeds under the current evaluation protocol.

ConfigurationQwen3.5-4BQwen3.5-9B
Base model from [1]70.67-
Base model70.5873.37
w/o Aug.70.5271.14
S²VOPD, main results (Table 1)77.72 ± 0.1178.80 ± 0.17
Best single augmentation, Sec. 4.376.61 ± 0.1877.73 ± 0.28
Composition, Fig. 2(c) (left)77.31 ± 0.2478.35 ± 0.13
Δ composition − single+0.70+0.62

The main results show low variation across seeds, and the composition improvements persist in the three-seed averages, reaching 0.70% at 4B and 0.62% at 9B. These repeated experiments support the composition-versus-single-augmentation comparison. The effect of stacking weak augmentations has not been evaluated across multiple seeds, and we will qualify that observation accordingly.

We will correct the outdated values, clarify the evaluation protocols and the source of the base-model results, and include mean ± standard deviation for all repeated configurations in the revision.

[1] Yuan, Qianhao, et al. "Vision-OPD: Learning to see fine details for multimodal LLMs via on-policy self-distillation." arXiv preprint arXiv:2605.18740 (2026).

>>> W2 Compositions A and B are introduced without any explanation of how they were selected, and the composition analysis is built upon them. Downscaling with Gaussian noise, which the single-augmentation analysis identifies as the most robust default, is not included in A or B, while mixed resolution, a key component of A and B, appears neither in the single-augmentation analysis nor in Table 5.

>>> A2 Thank you for raising this concern.

Compositions A and B are introduced without any explanation of how they were selected, and the composition analysis is built upon them.

Our original presentation did not clearly explain the relationship between the augmentation search and Compositions A and B. We defined a search space over augmentation operators, their strengths, and composition strategies. The insights in Figure 2(c)–(e), including the benefit of exclusive composition and the effect of stacking weak augmentations, emerged during this search and guided its subsequent refinement. This process ultimately yielded Compositions A and B. Thus, the composition analysis documents the findings that led to these recipes, rather than experiments built around predefined recipes. We also note that the search process use the training set itself as a validation set to select augmentation recipe and report the performance on the evaluated benchmarks. We will clarify this search procedure and its progression in Section 4.3.

Downscaling with Gaussian noise, which the single-augmentation analysis identifies as the most robust default, is not included in A or B, while mixed resolution, a key component of A and B, appears neither in the single-augmentation analysis nor in Table 5.

When updating the original figure, we inadvertently omitted the additional single augmentations explored later in the research, including mixed resolution, which informed the final compositions. The updated figure (https://imgur.com/a/rXmy1PJ) includes these results under the final evaluation protocol. Mixed resolution achieves 76.53%, followed closely by downscale plus Gaussian noise (76.33%), zoom-out (76.32%), and downscale plus color jitter (76.30%). The three members of Composition A are marked in the figure and all belong to this leading group.

The Figure 2 cc primarily compares the effectiveness of different augmentation families. Both mixed resolution and zoom-out belong to the information-reduction family and achieve competitive results among the leading operators. Their inclusion therefore reinforces the original conclusion that information reduction is the most effective augmentation family and preserves the insights derived from this analysis. We will revise the paper to reflect the expanded analysis, update Figure 2, and add the omitted operators and their parameters to Table 5.

We also conducted repeated runs of key experiments in composition v.s. single augmentation. Please refer to A1 for details.

>>> W3 Minor: the "EMA teacher." paragraph is duplicated in L457–461. It is not stated whether the student model or the EMA model is used for evaluation. There is no analysis of why the method improves performance on math reasoning.

>>> A3 Thanks for catching the duplicated paragraph; we have removed the paragraph from 448-456.

It is not stated whether the student model or the EMA model is used for evaluation.

We use the student model for evaluation in all experiments. We will revise the paper and add this information to the experiment setup section.

There is no analysis of why the method improves performance on math reasoning.

Thanks for the question. We have two hypothesis: 1. The math benchmarks we use are visually grounded: solving them requires requires accurately extracting visual information, such as geometric relations, chart values, and textual annotations, before performing the subsequent reasoning. We hypothesize that S²VOPD improves this visual evidence extraction process through supervision from the clean-view teacher, helping the student use visual evidence more effectively on these tasks. This provides a plausible explanation for the transfer to visual math despite training exclusively on perception data. Another hypothesis is that S²VOPD training leads to increased response length that allow the model to perform more intermediate reasoning steps, contributing to improved visual math performance. This increase remains stable throughout training, whereas the self-rewarding baselines eventually exhibit degenerate responses and performance collapse.

Thank you again for your constructive review.

We have responded to all the questions and concerns in above comments. As the discussion closes in a few days, we would be grateful to know whether these results resolve your concerns, or whether any specific issue still requires clarification. If the new evidence changes your assessment, we would sincerely appreciate your reflecting that in your final score. Thank you again for the constructive discussion.

Best,

Authors

Reviewer_nnZBOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Self-Supervised Visual On-Policy Distillation (S²VOPD), which improves vision-language models without using answer labels, region annotations, external rewards, or a separate stronger teacher during post-training. An EMA teacher observes the original image, while the student generates trajectories from an augmented image. The student learns from the teacher’s token-level distributions along these trajectories. The paper investigates augmentation families, strengths, and composition strategies. Experiments on Qwen3.5-4B and 9B report substantial improvements across six perception benchmarks and transfer gains on three mathematical reasoning benchmarks.

Critical strengths

1.Simple method with substantial reported improvements. S²VOPD improves average perception accuracy from 70.67 to 77.76 at 4B and from 73.37 to 78.85 at 9B without using privileged training annotations. This makes the approach potentially useful for reducing supervision requirements in VLM post-training.

2.Informative augmentation analysis. The investigation of augmentation strength and composition provides useful practical insights. In particular, the results suggest that a larger teacher–student discrepancy does not necessarily provide better supervision when perturbations remove question-relevant evidence.

Critical weaknesses

1.The specific contribution of on-policy training is not sufficiently established. Clean-teacher/perturbed-student learning is closely related to existing consistency-learning and self-training approaches. Without comparisons using teacher-generated or offline trajectories under the same augmentation setup, it remains unclear how much benefit comes specifically from on-policy distillation rather than asymmetric augmentation.

2.The validation of the augmentation recipe is insufficiently documented. The paper explores many configurations but does not clearly specify the validation data used to select the final recipe or report variability across training seeds. This matters particularly for the relatively small improvements from augmentation composition, such as 77.28 versus 76.81 at 4B. These results are less conclusive than the substantial gains over the base models.

Questions for authors

1.Under the same teacher, augmentation setup, and comparable training budget, how does S²VOPD compare with distillation using teacher-generated or offline trajectories? This would help establish the specific value of the on-policy component.

2.What validation data and selection criterion were used to choose the augmentation recipe, and were the final evaluation benchmarks involved in this process? Are repeated-run results available to establish the stability of the smaller composition gains?

3 comments

We thank the reviewer for the constructive review and helpful suggestions. We also would like to express our gratitude to the reviewer for finding our work simple and effective, useful for reducing supervision, with informative analysis.

>>> W1 & Q1 The specific contribution of on-policy training is not sufficiently established. Without comparisons using teacher-generated or offline trajectories under the same augmentation setup, it remains unclear how much benefit comes from on-policy distillation rather than asymmetric augmentation. Under the same teacher, augmentation setup, and comparable training budget, how does S²VOPD compare with distillation using teacher-generated or offline trajectories?

>>> A1 Following your suggestion, we trained three additional SFT baselines. The first uses ground-truth answers as supervision, the second uses fixed teacher-generated response trajectories, and the third uses teacher-generated answer labels (extracted from generated trajectories). Each baseline is trained for one epoch, with and without the same student-side augmentation used in S²VOPD. For the two pseudo-supervised baselines, the teacher generates supervision from clean images.

Post-training methodLabelsPerception, w/o augPerception, w/ same augΔ from augMath avg (w/ aug)
Base (from [1])–70.67–––
Base–70.58––69.54
SFT on GT answersGT73.4571.47−1.9864.33
SFT on teacher-generated trajectoriesPseudo75.1473.17−1.9770.62
SFT on teacher-generated labelsPseudo73.1571.79−1.3666.97
S²VOPD (ours)None70.79 (symmetric)77.76+6.9772.79

Adding the same augmentation to GT-supervised SFT reduces perception accuracy from 73.45% to 71.47%. Its math accuracy with augmentation is 64.33%, below the base model’s 69.54%. Augmentation also reduces perception accuracy for trajectory-based and label-based SFT by 0.97% and 1.36%, respectively. However, when combined with OPD, S²VOPD with augmentation exceeds the three baselines with augmentation by 6.29%, 4.59%, and 5.97% on perception and by 8.46%, 2.17%, and 5.82% on math reasoning, respectively. It also outperforms the three baselines without augmentation by 4.31%, 2.62%, and 4.61% on perception, respectively.

These comparisons show that the benefit of augmentation depends on how it is used in the training objective. They also illustrate that our work is not simply a combination of established techniques, but rather a novel finding: augmentation can substantially improve MLLM post-training when it creates an information asymmetry between the teacher and student in on-policy distillation.

[1] Yuan, Qianhao, et al. "Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation." arXiv preprint arXiv:2605.18740 (2026).

>>> W2 & Q2 The validation of the augmentation recipe is insufficiently documented. The paper does not specify the validation data used to select the final recipe or report variability across training seeds. This matters particularly for the small composition improvements, such as 77.28 versus 76.81 at 4B. What validation data and selection criterion were used, and were the final evaluation benchmarks involved? Are repeated-run results available?

>>> A2 Thanks for raising the concern. We used the Vision-OPD-6K training set itself to select the augmentation configurations. We only report the last checkpoints, except for the self-rewarding models (which we report the best due to their training instability). The reported benchmarks were used only for evaluation after selection. We will revise the experimental setup in the paper to explicitly describe the data and criteria used for both selection decisions.

Given the limited time and compute of the rebuttal period, we repeated only key experiments three times with different training seeds. Specifically, we repeat the S²VOPD main result experiment with Qwen3.5-4B and Qwen3.5-9B and the two composition comparisons the reviewer refers to (composition vs. the best single augmentation) with only 0.5–0.6% improvements. Each run uses an identical configuration and changes only the random seed. All numbers are the six-benchmark perception average under the main protocol (greedy, 4096 tokens), reported as mean ± std over three runs:

ConfigurationQwen3.5-4BQwen3.5-9B
Base model from [1]70.67-
Base model70.5873.37
w/o Aug.70.5271.14
S²VOPD, main results (Table 1)77.72 ± 0.1178.80 ± 0.17
Best single augmentation, Sec. 4.376.61 ± 0.1877.73 ± 0.28
Composition, Fig. 2(c) (left)77.31 ± 0.2478.35 ± 0.13
Δ composition − single+0.70+0.62

Main results are stable across seeds. The standard deviation of S²VOPD is 0.11% at 4B and 0.17% at 9B, substantially smaller than its respective gains of 7.05% and 5.43% over the base models.

Composition improvements persist when averaged across seeds. Composition outperforms the best single augmentation by 0.70% at 4B and 0.62% at 9B. These improvements exceed the standard deviations of both configurations at each scale, supporting the benefit of composition observed in the original experiments.

Together, these repeated experiments support both the stability of the main results and the average improvement from composition over the best single augmentation. We will include these seed statistics in the revised paper and report mean ± standard deviation for all repeated configurations.

[1] Yuan, Qianhao, et al. "Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation." arXiv preprint arXiv:2605.18740 (2026).

Thank you again for your constructive review.

We have responded to all the questions and concerns in above comments. As the discussion closes in a few days, we would be grateful to know whether these results resolve your concerns, or whether any specific issue still requires clarification. If the new evidence changes your assessment, we would sincerely appreciate your reflecting that in your final score. Thank you again for the constructive discussion.

Best,

Authors

Reviewer_L2LLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Self-Supervised Visual On-Policy Distillation (S²VOPD), a method for improving vision-language models without external rewards, ground-truth annotations, or a stronger teacher. The key idea is to create teacher–student asymmetry by providing the teacher with the original image while training the student on an augmented, degraded version. The authors systematically investigate different visual augmentations and their combinations to construct effective distillation signals. Experiments on Qwen3.5-4B and 9B demonstrate improvements in fine-grained visual perception and mathematical reasoning benchmarks.

Critical strengths

The paper presents a simple and intuitive approach to constructing self-supervised signals for on-policy distillation without requiring additional annotations or stronger teacher models. The experimental results show substantial performance improvements across multiple benchmarks, outperforming several privileged-information distillation and self-rewarding baselines. The authors also conduct relatively comprehensive ablation studies on augmentation types, strengths, and compositions, providing useful empirical insights into their effectiveness.

Critical weaknesses

The main contribution appears to be largely an application of conventional data augmentation within an on-policy self-distillation framework, rather than a fundamentally new learning mechanism. While OPD enables training without ground-truth labels, the supervision still comes from the model's own predictions on clean images, potentially limiting its ability to acquire capabilities beyond those already possessed by the teacher. The method therefore seems more naturally suited to improving robustness and consistency under degraded visual inputs than addressing fundamental perception failures.

More importantly, the paper does not sufficiently establish the necessity of the proposed training framework. Similar augmentation strategies could potentially be incorporated into standard pre-training or post-training pipelines by foundation model developers. Without comparisons against conventional augmentation-based training, it remains unclear whether the proposed OPD formulation offers meaningful advantages that justify its additional complexity and computational overhead.

Questions for authors

How does S²VOPD compare with conventional augmentation-based fine-tuning under matched training data and compute budgets, and what unique benefits does the OPD objective provide?

4 comments

We thank the reviewer for the constructive review and helpful suggestions. We also would like to express our gratitude to the reviewer for finding our work simple and intuitive, effective with comprehensive experiments.

>>> W1 The main contribution appears to be largely an application of conventional data augmentation within an on-policy self-distillation framework, rather than a fundamentally new learning mechanism. The supervision still comes from the model's own predictions on clean images, potentially limiting its ability to acquire capabilities beyond those already possessed by the teacher. The method seems more naturally suited to improving robustness and consistency under degraded visual inputs than addressing fundamental perception failures.

>>> A1 Thanks for raising this question. While we acknowledge that visual augmentation and on-policy distillation are established techniques, our contribution is to identify teacher–student information asymmetry as the principle underlying the use of privileged information in prior OPD and OPSD methods. These methods create this asymmetry by giving the teacher access to costly privileged information unavailable to the student, such as a ground-truth solution or an annotated region of interest. Our insight is that the same asymmetry can be created by reducing the information available to the student. Visual augmentation serves as a means of creating this asymmetry. By keeping the original image for the teacher and reducing the visual information available to the student, S²VOPD enables on-policy self-distillation without privileged annotations.

Our results directly address the concern that supervision from the model’s own predictions may limit what the student can learn. We evaluate on the original, unmodified test images from widely adopted benchmarks and observe significant improvements in both fine-grained perception and visual mathematical reasoning. S²VOPD with Qwen3.5-9B achieves 78.85% across six perception benchmarks and 75.97% across three visual math benchmarks, outperforming all compared privileged-information OPD and self-rewarding RL baselines on both averages. It also surpasses most of the evaluated proprietary models on perception. These gains on unmodified images show that supervision from the model’s own predictions can improve both fine-grained perception and visual mathematical reasoning.

>>> W2 & Q1 The paper does not sufficiently establish the necessity of the proposed training framework. Similar augmentation strategies could be incorporated into standard pre-training or post-training pipelines. Without comparisons against conventional augmentation-based training, it remains unclear whether the OPD formulation offers advantages that justify its additional complexity and computational overhead. How does S²VOPD compare with conventional augmentation-based fine-tuning under matched training data and compute budgets, and what unique benefits does the OPD objective provide?

>>> A2 Following your suggestion, we trained two standard post-training pipelines, SFT and GRPO, both with ground-truth labels, under strictly matched conditions: same Qwen3.5-4B initialization, same 6K images, same augmentation recipe and probability, same number of steps, same evaluation protocol. Each is run with and without the augmentation, so the effect of the augmentation can be read off directly:

Post-training methodLabelsPerception, w/o augPerception, w/ same augΔ from augMath avg (w/ aug)
Base (from [3])–70.67––-
Base–70.58––69.54
SFT on GT answersGT73.4571.47−1.9864.33
GRPO with GT rewardGT73.1074.06+0.9670.89
S²VOPD (ours)none70.79 (symmetric)77.76+6.9772.79

(Perception: average over V*, ZoomBench, HR-Bench 4K/8K, MME-RealWorld, MME-RealWorld-CN; greedy, 4096 tokens. Math: MathVista, MathVerse, MathVision.)

Augmentation has modest and mixed effects on perception in SFT and GRPO. Adding the same augmentation to SFT hurts performance, reducing perception from 73.45% to 71.47% and lowers math accuracy to 64.33%, even below the base model’s 69.54%. This is consistent with prior work showing that MLLM fine-tuning can hurt performance and induce catastrophic forgetting [1]. In GRPO, augmentation yields a modest perception gain from 73.10% to 74.06%, consistent with NoisyRollout’s finding [2]. In S²VOPD, using the same augmentation to create asymmetric teacher and student views raises perception accuracy from 70.79% to 77.76%. The clean-view teacher provides next-token targets along the student’s own trajectories using only image–question pairs.

These comparisons show that the benefit of augmentation depends on how it is used in the training objective. They also illustrate that our work is not simply a combination of established techniques, but rather a novel finding: augmentation can substantially improve MLLM post-training when it creates an information asymmetry between the teacher and student in on-policy distillation.

[1] Zhai, Yuexiang, et al. "Investigating the catastrophic forgetting in multimodal large language model fine-tuning." Conference on Parsimony and Learning. PMLR, 2024.

[2] Liu, Xiangyan, et al. "NoisyRollout: Reinforcing visual reasoning with data augmentation." Advances in Neural Information Processing Systems 38 (2025): 2923--2957.

[3] Yuan, Qianhao, et al. "Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation." arXiv preprint arXiv:2605.18740 (2026).

Thanks for your prompt response and for your additional suggestion, we'll add more citations and dicussion for the following papers upon revision:

Chen, Howard, et al. “Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting.” Proceedings of the 43rd International Conference on Machine Learning, edited by Tong Zhang et al., vol. 306, PMLR, 6–11 July 2026, pp. 16479–94.

Chen, Yan, et al. “On-Policy Replay for Continual Supervised Fine-Tuning.” arXiv, 2026, arxiv.org/abs/2605.29495.

Ming, Rui, et al. “One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient.” arXiv, 2026, arxiv.org/abs/2509.26313.

Zhao, Anhao, et al. "On-policy supervised fine-tuning for efficient reasoning." arXiv preprint arXiv:2602.13407 (2026).

Feel free to add any additional related work that you would like us to discuss in our revised related work section.

Again, thank you for your timely response and suggestions. We welcome any further questions and would be happy to provide additional clarification.

Thank you again for your constructive review and the follow-up.

We have responded to all the questions and concerns in above comments. As the discussion closes in a few days, we would be grateful to know whether these results resolve your concerns, or whether any specific issue still requires clarification. If the new evidence changes your assessment, we would sincerely appreciate your reflecting that in your final score. Thank you again for the constructive discussion.

Best,

Authors

Reviewer_QVvnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Self-Supervised Visual On-Policy Distillation (S²VOPD), which adapts a vision-language model without using answer labels, region annotations, external rewards, or a stronger teacher. An EMA teacher observes the original image, while the student generates responses from an augmented image and matches the teacher’s token distributions along those responses. The main contributions are this training formulation and an empirical study of augmentation families, strengths, and composition strategies.

Training on Vision-OPD-6K improves average accuracy across six perception benchmarks from 70.67% to 77.76% for Qwen3.5-4B and from 73.37% to 78.85% for Qwen3.5-9B. Evaluation on three mathematical reasoning benchmarks also shows average improvements. Ablations examine augmentation, teacher updates, and divergence objectives.

Critical strengths
  1. Substantial gains without additional privileged supervision. The method improves perception accuracy at both model sizes and exceeds the compared privileged-supervision baselines on the six-benchmark average. Obtaining these gains without using the available answer or region annotations makes the approach practically consequential.

  2. Useful empirical investigation beyond headline performance. The study examines augmentation strength, information preservation, composition, EMA updates, and divergence choice. The fixed-teacher result retains approximately 84% of the full perception improvement, providing particularly useful evidence about where the gains originate. Improvements on mathematical reasoning benchmarks further suggest transfer beyond the training domain.

Critical weaknesses
  1. The experimental reporting does not sufficiently establish the reliability of the selected recipe and its smaller gains. The paper searches many augmentation configurations but does not identify a separate validation set or clearly explain how configurations and checkpoints were selected. It also reports no variation across training seeds. This matters particularly for the approximately 0.5–0.6-point composition improvements. Numerical inconsistencies further weaken the evidence: the no-augmentation result is 65.21% in Section 4.2 versus 70.52% in Table 3, under different decoding budgets whose effect is not isolated. Some conclusions also overstate the tables: at 4B, S²VOPD’s math average is below Intuitor and RENT, and Vision-OPD improves both perception and math at both scales.

  2. The contribution of on-policy training is not isolated from established augmentation-consistency learning. The paper acknowledges related teacher–student consistency methods, but does not compare against a matched alternative using teacher-generated or fixed response trajectories with the same clean/augmented views. Consequently, the experiments demonstrate that the complete recipe works, but do not establish whether student-generated trajectories provide a meaningful advantage over simpler consistency-distillation alternatives. This limits the evidence for the distinctive methodological contribution.

Questions for authors
  1. Experimental clarification: Which data were used to select augmentation configurations and checkpoints? Were the reported evaluation benchmarks consulted during this selection?
  2. Reporting clarification: Can you reconcile the base and no-augmentation accuracies across Figure 2, Section 4.2, and Table 3, and explain the different composition-A results across Figure 2 panels?
  3. Additional evidence: How variable are the main results and composition improvements across training seeds?
  4. Contribution assessment: How does the method compare with clean-teacher/augmented-student distillation using teacher-generated trajectories under a matched training budget?
4 comments

We thank the reviewer for the constructive review and helpful suggestions. We also thank the reviewer for recognizing our work as effective without additional privileged supervision, with useful empirical investigation beyond headline performance.

>>> W1(a) & Q1 The paper searches many augmentation configurations but does not identify a separate validation set or clearly explain how configurations and checkpoints were selected. Which data were used to select augmentation configurations and checkpoints? Were the reported evaluation benchmarks consulted during this selection?

>>> A1 Thanks for raising the concern. We used the Vision-OPD-6K training set itself to select the augmentation configurations. We only report the last checkpoints, except for the self-rewarding models (which we report the best due to their training instability). The reported benchmarks were used only for evaluation after selection. We will revise the experimental setup in the paper to explicitly describe the data and criteria used for both selection decisions.

>>> W1(b) & Q3 The paper reports no variation across training seeds. This matters particularly for the approximately 0.5–0.6-point composition improvements. How variable are the main results and composition improvements across training seeds?

>>> A2 Thank you for raising this concern. Given the limited time and compute of the rebuttal period, we repeated only key experiments three times with different training seeds. Specifically, we repeat the S²VOPD main result experiment with Qwen3.5-4B and Qwen3.5-9B and the two composition comparisons the reviewer refers to (composition vs. the best single augmentation) with only 0.5–0.6% improvements. Each run uses an identical configuration and changes only the random seed. All numbers are the six-benchmark perception average under the main protocol (greedy, 4096 tokens), reported as mean ± std over three runs:

ConfigurationQwen3.5-4BQwen3.5-9B
Base model from [1]70.67-
Base model70.5873.37
w/o Aug.70.5271.14
S²VOPD, main results (Table 1)77.72 ± 0.1178.80 ± 0.17
Best single augmentation, Sec. 4.376.61 ± 0.1877.73 ± 0.28
Composition, Fig. 2(c) (left)77.31 ± 0.2478.35 ± 0.13
Δ composition − single+0.70+0.62

Main results are stable across seeds. The standard deviation of S²VOPD is 0.11% at 4B and 0.17% at 9B, substantially smaller than its respective gains of 7.05% and 5.43% over the base models.

Composition improvements persist when averaged across seeds. Composition outperforms the best single augmentation by 0.70% at 4B and 0.62% at 9B. These improvements exceed the standard deviations of both configurations at each scale, supporting the benefit of composition observed in the original experiments.

Together, these repeated experiments support both the stability of the main results and the average improvement from composition over the best single augmentation. We will include these seed statistics in the revised paper and report mean ± standard deviation for all repeated configurations.

[1] Yuan, Qianhao, et al. "Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation." arXiv preprint arXiv:2605.18740 (2026).

>>> W1(c) & Q2 Numerical inconsistencies further weaken the evidence: the no-augmentation result is 65.21% in Section 4.2 versus 70.52% in Table 3, under different decoding budgets whose effect is not isolated. Can you reconcile the base and no-augmentation accuracies across Figure 2, Section 4.2, and Table 3, and explain the different composition-A results across Figure 2 panels?

>>> A3 Thank you for pointing out these inconsistencies. They arose from incomplete updates when we changed the evaluation settings. The previous and current protocols are summarized below.

EvaluationPrevious protocolCurrent protocol
PerceptionGreedy decoding (T = 0), 2,048-token budgetGreedy decoding (T = 0), 4,096-token budget
MathT = 0.6, 16,384-token budgetT = 0.3, 24,576-token budget, top-p = 0.95, top-k = 20, presence penalty = 1.5

We adopted the current protocol because the smaller inference token budget in the previous protocol truncated reasoning traces, resulting in incomplete responses and lower performance. The current protocol yielded better base-model performance and closely reproduced the perception accuracy reported by Vision-OPD [1] (70.58% in our evaluation versus 70.67% in [1]). We subsequently re-evaluated all experiments and models using the current protocol. However, we inadvertently retained results from the previous protocol in parts of the text and Figure 2.

The corrected values and their sources are summarized below.

QuantityReported valuesExplanationCorrected value
No augmentation65.21% in Sec. 4.2 vs. 70.52% in Table 3Sec. 4.2 retained the result under the previous protocol70.52%
Base, 4B70.58% in Sec. 4.2 / 70.67% in the main table vs. 66.5% in Fig. 2a,bFigure 2 retained the previous result. The text reports our re-evaluation, while the main table cites the result from [1]70.58% for our evaluation; 70.67% as reported in [1]
Composition A, 4B76.94% in Fig. 2c (right) vs. 75.89% in Fig. 2dFigure 2d retained the result under the previous protocol76.94%

We will update Section 4.2 and Figure 2 to consistently reflect the current evaluation protocol and explicitly distinguish our base-model evaluation from the result cited from Vision-OPD [1].

[1] Yuan, Qianhao, et al. "Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation." arXiv preprint arXiv:2605.18740 (2026).

>>> W1(d) Some conclusions also overstate the tables: at 4B, S²VOPD's math average is below Intuitor and RENT, and Vision-OPD improves both perception and math at both scales.

>>> A4 Thank you for pointing this out. We will revise both statements to accurately reflect the results. Specifically, we will change “both scales achieve the strongest overall results among the compared methods” to “the 9B model achieves the best overall results among the compared methods, while the 4B model delivers competitive results.” We will also revise “It substantially outperforms privileged OPD methods, which improve perception at the cost of reasoning” to “It substantially outperforms OPSD, which improves perception at the cost of reasoning,” making clear that this observation applies specifically to OPSD. Finally, we will explicitly acknowledge that Vision-OPD improves both perception and math at both scales, while lags behind S²VOPD.

We also note that the comparison in Table 2 favors the three self-rewarding methods, as their results are reported at their best checkpoints (because the self-rewarding method exhibit substantial performance degradation as training progresses), whereas S²VOPD is evaluated at its last checkpoint.

Table R1. Math average (%) on MathVista, MathVerse, and MathVision for Qwen3.5-4B over training. Baselines are evaluated at steps 10 / 20 / 35 / 50 / 65, and S²VOPD at steps 13 / 26 / 39 / 52 / 65. The base model scores 69.54%.

Method~13~26~39~5265 (last)
TTRL§72.6073.0470.3955.3551.50
RENT§73.2867.9048.0929.7920.43
Intuitor§73.0014.152.171.995.14
S²VOPD73.5272.9473.3973.5072.79

§ Results in Table 2 are reported at the best checkpoint; S²VOPD is reported at the last checkpoint.

All three baselines peak early in training and subsequently exhibit substantial degradation. By step 65, their math averages fall to 51.50% for TTRL, 20.43% for RENT, and 5.14% for Intuitor, largely due to degenerate or empty responses. S²VOPD maintains a math average of 72.79–73.52% across the evaluated checkpoints. Under a last-checkpoint comparison, S²VOPD achieves 72.79%, exceeding TTRL, RENT, and Intuitor by 21.29%, 52.36%, and 67.65%, respectively. Under a best-checkpoint comparison among the checkpoints shown, S²VOPD reaches 73.52%, exceeding TTRL (73.04%), RENT (73.28%), and Intuitor (73.00%).

>>> W2 & Q4 The contribution of on-policy training is not isolated from established augmentation-consistency learning. The paper does not compare against a matched alternative using teacher-generated or fixed response trajectories with the same clean/augmented views. How does the method compare with clean-teacher/augmented-student distillation using teacher-generated trajectories under a matched training budget?

>>> A5 Following your suggestion, we trained three additional SFT baselines. The first uses ground-truth answers as supervision, the second uses fixed teacher-generated response trajectories, and the third uses teacher-generated answer labels (extracted from generated trajectories). Each baseline is trained for one epoch, with and without the same student-side augmentation used in S²VOPD. For the two pseudo-supervised baselines, the teacher generates supervision from clean images.

Post-training methodLabelsPerception, w/o augPerception, w/ same augΔ from augMath avg (w/ aug)
Base (from [1])–70.67–––
Base–70.52––69.54
SFT on GT answersGT73.4571.47−1.9864.33
SFT on teacher-generated trajectoriesPseudo75.1473.17−1.9770.62
SFT on teacher-generated labelsPseudo73.1571.79−1.3666.97
S²VOPD (ours)None70.79 (symmetric)77.76+6.9772.79

Adding the same augmentation to GT-supervised SFT reduces perception accuracy from 73.45% to 71.47%. Its math accuracy with augmentation is 64.33%, below the base model’s 69.54%. Augmentation also reduces perception accuracy for trajectory-based and label-based SFT by 0.97% and 1.36%, respectively. However, when combined with OPD, S²VOPD with augmentation exceeds the three baselines with augmentation by 6.29%, 4.59%, and 5.97% on perception and by 8.46%, 2.17%, and 5.82% on math reasoning, respectively. It also outperforms the three baselines without augmentation by 4.31%, 2.62%, and 4.61% on perception, respectively.

These comparisons show that the benefit of augmentation depends on how it is used in the training objective. They also illustrate that our work is not simply a combination of established techniques, but rather a novel finding: augmentation can substantially improve MLLM post-training when it creates an information asymmetry between the teacher and student in on-policy distillation.

[1] Yuan, Qianhao, et al. "Vision-opd: Learning to see fine details for multimodal llms via on-policy self-distillation." arXiv preprint arXiv:2605.18740 (2026).

Thank you again for your constructive review.

We have responded to all the questions and concerns in above comments. As the discussion closes in a few days, we would be grateful to know whether these results resolve your concerns, or whether any specific issue still requires clarification. If the new evidence changes your assessment, we would sincerely appreciate your reflecting that in your final score. Thank you again for the constructive discussion.

Best,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of rebuttal and updated revision for AC

Dear Area Chair,

We thank the reviewers for the constructive reviews and helpful suggestions. We also would like to express our gratitude to the reviewers for finding our work simple and effective. We summarize our rebuttal and changes below:

1. Matched baselines (QVvn, L2LL, nnZB). Following the reviewers' suggestion, we trained additional baselines. Adding the same augmentation to GT-supervised SFT reduces perception accuracy from 73.45% to 71.47%. In GRPO under strictly matched conditions, augmentation yields a modest perception gain from 73.10% to 74.06%. In S²VOPD, using the same augmentation to create asymmetric teacher and student views raises perception accuracy from 70.79% to 77.76%. These comparisons show that the benefit of augmentation depends on how it is used in the training objective. Reviewer L2LL acknowledged the extra experiments, and we'll add more citations and discussion on on-policy SFT upon revision.

2. Selection and seeds (QVvn, nnZB, cWZf). We used the Vision-OPD-6K training set itself to select the augmentation configurations. The reported benchmarks were used only for evaluation after selection. We repeated the main experiments and the composition-versus-single-augmentation comparisons with three training seeds. The main results show low variation across seeds, and the composition improvements persist in the three-seed averages (0.70% at 4B and 0.62% at 9B).

3. Novelty (L2LL). While we acknowledge that visual augmentation and on-policy distillation are established techniques, our contribution is to identify teacher–student information asymmetry as the principle underlying the use of privileged information in prior OPD and OPSD methods. Our insight is that the same asymmetry can be created by reducing the information available to the student, which enables on-policy self-distillation without privileged annotations.

4. Evaluation numbers (QVvn, cWZf). Some numerical inconsistencies arose from incomplete updates following changes to the evaluation protocol and from differences in training setups between the augmentation analysis and main experiments.

Within Figure 2, differing values reported for the same configuration reflect the previous and current evaluation protocols, rather than different training seeds. The corrected values under the final evaluation protocol and the sources of the adopted baseline results are provided in our responses.

Between Figure 2 and Table 1 and subsequent ablations, the discrepancy arises from different training setups. Due to computational constraints, the augmentation analysis in Figure 2, involving 42 experimental runs, uses dynamic batching, larger batch sizes, and gradient accumulation to accelerate training. In contrast, Table 1 and subsequent ablations use a training recipe that reproduces Vision-OPD performance for fair comparison. Importantly, the experiments within Figure 2 remain internally consistent and fairly comparable, but their absolute values are not directly comparable with Table 1 or subsequent ablations. We have clarified this distinction in the updated revision Lines 255–257.

We will correct outdated values, clarify the evaluation and training protocols and the sources of baseline results, and report mean ± standard deviation for all repeated configurations. We will also explicitly describe the data and criteria used for both augmentation selection decisions, provide the complete set of operators and their parameters in Table 5, clarify the search procedure and its progression in Section 4.3, and update Figure 2 accordingly.

Summary of revision:

  • Added matched SFT and GRPO baselines with and without the same augmentation, plus citations and discussion on on-policy SFT.
  • Described the experimental setup more explicitly: selection data and criteria, checkpoint reporting, and student-model evaluation.
  • Reported three-seed mean ± s.d. for the main results and the composition comparison, and qualified the weak-augmentation stacking result.
  • Corrected outdated values and distinguished our base-model evaluation (70.58%) from the Vision-OPD result (70.67%).
  • Added detail to the brief mention in paper of how Figure 2 uses a different training setup and is not directly comparable with Table 1.
  • Expanded the augmentation analysis: how compositions A and B emerged, the later single augmentations in Figure 2, zoom-out re-categorized, and mixed resolution added to Table 5.
  • Revised claims on math results and OPSD
  • Removed the duplicated "EMA teacher" paragraph.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.