acceptodds
Under review as a conference paper at ICLR 2027

Weak-to-Strong On-Policy Distillation via Perception Gap Supervision

Abstract

On-policy distillation (OPD) supervises a student on its own rollouts using token-level targets from a teacher. Existing OPD methods typically rely on a teacher model that is stronger than the student. Training such a teacher can be expensive or even impractical. We study weak-to-strong OPD for vision-language models, where the teacher is weaker than the student, typically being smaller or less extensively trained. Directly distilling the weak teacher's predictions can degrade student performance. We introduce Weak-to-Strong Distillation via Perception Gap (W2S-PG), which constructs supervision from the discrepancy between the same weak teacher's predictions under two answer-preserving visual perturbations. While teacher-derived supervision is effective, the student becomes stronger during training, and its own predictions may eventually provide more reliable supervision than the teacher-derived signal. To select the more reliable source of supervision, we further propose Adaptive Supervision Signal Selection (AS3), a token-level mechanism that adaptively chooses between teacher-derived perception-gap supervision and self-training according to model confidence. Across six perception benchmarks and four teacher–student pairs, W2S-PG consistently improves the student and outperforms other OPD variants such as vanilla OPD and previous weak-to-strong distillation approaches. When combined with W2S-PG, AS3 provides further gains of 0.3–2.4 points across all benchmarks. Ablations of augmentation and discrepancy choices demonstrate the robustness of W2S-PG, and evaluations on seven additional benchmarks show that its benefits extend beyond perception to visual reasoning and other tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
on-policy distllation, weak-to-strong generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1G3POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies weak-to-strong on-policy distillation for vision-language models. W2S-PG constructs supervision by applying softmax to the squared probability differences of one weak teacher under two answer-preserving visual augmentations, then distills this target on student-generated trajectories. AS3 selects between teacher-derived supervision and self-training at each token using confidence scores. The main contributions are the perception-gap target and adaptive supervision selection. Evaluation covers four teacher–student pairs, six perception settings, component ablations, and seven additional benchmarks after training on a separate dataset.

Critical strengths

The substantial reported gains across models and tasks are the main strength. For example, the 4B student improves by 13.5 points on VStar and 19.1 points on ZoomBench, with improvements reported across all four teacher–student pairs. This breadth gives the empirical finding potential significance beyond an isolated model or benchmark and constitutes the main positive evidence for acceptance.

Critical weaknesses

Weakness 1: Core inconsistencies between the equations, experimental statistics, and claimed mechanism.

Equation (2) bounds teacher-derived confidence by e/(V-1+e), while student confidence is at least 1/V, where V is the vocabulary size. Equation (4) therefore selects student supervision with probability greater than 1/e (approximately 36.8%). Yet Fig. 5's stored PDF polyline contains at least eight batch means below 20% in the first 100 steps. Using Table 5's batch size of 128 and Algorithm 1's tokenwise sampling, for equally weighted averages over at least 128 valid gates, with gate-independent masking and fresh independent draws each step, Hoeffding bounds the single-step probability below 20% by 0.0007355. A union bound over eight-step subsets of all 300 training steps gives at most 1.27e-10, on the order of one in ten billion. Under these conditions, ordinary sampling fluctuations cannot plausibly explain the observed frequency. This raises a question about whether the experiments implement the stated algorithm, not merely how the results were plotted.

Section 2.5 does not establish transfer of visual dependence either. It analyzes the teacher's response to perturbations, but the objective does not match teacher and student responses. A teacher whose probabilities vary linearly with the input can produce an image-independent discrepancy under fixed small perturbations, allowing a student that completely ignores the image to achieve zero distillation loss. Equation (2) is also nearly uniform: its total-variation distance from uniform is at most 3.44e-5 for a vocabulary of 100,000.

Table 3 further undermines the perception-transfer interpretation. Under Algorithm 1's exchangeable view sampling, swapping the views reverses the teacher's log-ratio term without changing the student distribution at a fixed prefix. For the reverse-KL objective in Eq. (3), the teacher-dependent terms therefore cancel in the expected token-level distillation gradient, leaving entropy maximization. Nevertheless, this variant achieves approximately 92% of the squared-difference variant's improvement over the baseline student on ZoomBench. The ablation does not establish that these gains come from perceptual knowledge rather than entropy regularization. Additionally, Table 3 never defines how scalar KL/JSD values become the vocabulary-level supervision targets required by Eq. (2).

Weakness 2: The core designs of both claimed contributions have explicit precedents.

Visual-condition-based supervision combined with reliability selection is already an explicit distillation module in IC-VCO. Its VCDist module (ICML 2026, Sec. 3, Eq. (11)) distills a multi-image visual-contrastive preference distribution into a single-image branch, activating supervision only when the supervising branch supports the correct preference and is more confident. The central idea of obtaining supervision from different visual conditions and filtering it by relative reliability is already implemented there.

AS3's core selection principle also appears as an explicit component of an existing method. TSD-KD's "Direct Distillation based on Uncertainty Gap" (ICLR 2026, Sec. 3.2, Eq. (11)) uses the teacher-student entropy gap to regulate token-level teacher intervention on student-generated trajectories. CrossRectify (Pattern Recognition, 2023, Sec. 4.1, Eq. (8)) selects between self and peer pseudo-labels by comparing maximum class probabilities. SelecTKD's Non-Greedy Spec-k branch (CVPR 2026) already uses clipped teacher/student token-probability ratios and uniform random draws to gate distillation. These are concrete supervision-selection rules, not merely related motivations. I regard the weak-teacher setting, squared-probability target, and randomized loss switching as incremental adaptations of these established principles, rather than two substantial independent methodological innovations.

Questions for authors

Question 1: Equations (2) and (4) imply that the probability of selecting student supervision at each valid token is greater than 1/e (approximately 36.8%), yet Fig. 5 repeatedly shows batch means below 20%. Under the averaging and sampling conditions stated in Weakness 1, the probability bound for this frequency is only 1.27e-10. How do the authors reconcile this statistical inconsistency? Were the reported experiments actually conducted using AS3 as defined in the paper?

Question 2: Under the paper's exchangeable view sampling, the teacher-dependent terms of the log-ratio variant of Eq. (3) cancel in the expected token-level distillation gradient at a fixed student prefix, leaving entropy maximization. Nevertheless, this variant achieves approximately 92% of the squared-difference variant's improvement over the baseline student on ZoomBench. What evidence attributes these gains to the transfer of the weak teacher's perceptual knowledge rather than entropy regularization? How does Table 3 substantiate the claimed mechanism?

Reviewer_oxLfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates weak-to-strong on-policy distillation (OPD) for vision-language models, where a weaker teacher supervises a stronger student. The authors propose Weak-to-Strong Distillation via Perception Gap (W2S-PG), which constructs a token-level supervision distribution from the squared difference between the weak teacher's predictions under two answer-preserving visual augmentations.

The paper further introduces Adaptive Supervision Signal Selection (AS3), which switches between teacher-derived supervision and self-training according to token-level confidence. Experiments across four teacher–student pairs and six perception benchmarks demonstrate substantial improvements over the original students and several OPD baselines. The authors also evaluate augmentation choices, alternative discrepancy measures, and supervision-selection strategies, with additional experiments on seven benchmarks using a separate training dataset.

Critical strengths

S1. Substantial empirical improvements. W2S-PG achieves large gains across multiple perception benchmarks and four teacher–student pairs. Comparisons against weak-teacher OPD, strong-teacher OPD, and OPD-Δ provide useful evidence of practical effectiveness.

S2. Interesting supervision design and broad evaluation. Constructing supervision from different visual conditions of a single weak teacher is a potentially useful approach. The augmentation, discrepancy, and supervision-selection ablations, together with additional task evaluations, provide meaningful empirical support.

Critical weaknesses

W1. A fundamental ambiguity in the central supervision distribution and its compatibility with AS3. Eq. (2) applies softmax to squared differences between teacher probability distributions. Since each squared difference lies in ([0,1]), the maximum probability of the resulting distribution is bounded by (e/(V-1+e)), where (V) is the vocabulary size. For a large vocabulary, this target is nearly uniform. Consequently, whenever the student's maximum probability exceeds this bound, AS3 must select student-derived supervision. This makes the substantial teacher-selection frequency suggested by Figure 5 difficult to explain under the stated formulation. If the implementation uses logits, temperature scaling, or restricted vocabulary support, these essential details are missing.

W2. Insufficient justification of the perception-gap supervision mechanism. The squared prediction difference measures sensitivity to visual perturbations but discards the direction of probability changes. The Taylor analysis in Eqs. (8)–(9) characterizes local sensitivity without establishing that larger discrepancies correspond to more reliable or correct token targets. Although the paper evaluates several discrepancy functions and supervision-selection strategies, the relationship between the proposed targets, visual evidence, and prediction correctness remains insufficiently validated.

Questions for authors

Q1. Exact supervision construction. Does Eq. (2) operate on normalized teacher probabilities or pre-softmax logits? Are temperature scaling or vocabulary truncation used? Please provide the exact implementation and report the resulting target entropy and maximum token probabilities.

Q2. Consistency with AS3. How can the teacher-derived confidence scores in Eq. (4) produce the selection frequencies shown in Figure 5 under the stated Eq. (2)? Can the authors report the distributions of student and teacher confidence, together with actual teacher-selection rates during training?

Q3. Reliability of perception-gap supervision. What evidence shows that tokens with larger prediction discrepancies provide more reliable supervision? Can the authors analyze the relationship between discrepancy magnitude and token correctness, or compare against a computation-matched augmentation-consistency objective?

Q4. Alternative discrepancy definitions. Table 3 includes KL and Jensen–Shannon divergences, which are ordinarily scalar quantities. How are these implemented as vocabulary-level supervision distributions? Please specify whether per-token contributions or additional transformations are used.

Q5. Data and experimental reliability. How are answer-preserving augmentations verified, and what fraction of samples is filtered? How is separation from evaluation benchmarks established? Are the reported improvements consistent across independent training seeds, and what additional teacher-forward computation does W2S-PG require?

Reviewer_q9ZFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes W2S-PG, which trains a stronger VLM using a weak teacher's prediction differences across two image augmentations. AS3 switches between this signal and student self-training. Experiments report gains across perception benchmarks and additional multimodal tasks.

Critical strengths
  1. The paper addresses a useful problem: improving VLMs without a stronger teacher. Tables 1 and 2 report consistent gains across four teacher and student pairs.

  2. Ablations compare augmentation, discrepancy, and supervision choices, helping assess which components contribute.

Critical weaknesses
  1. Why the student should match this target distribution. Unlike comparisons before and after RL or between correct and wrong hints, neither random image view is defined as better. Their difference measures sensitivity, which does not imply correctness. In a two-token example, teacher distributions (0.8,0.2)(0.8,0.2) and (0.6,0.4)(0.6,0.4) both favor correct token A, but Equation (2) maps their squared differences (0.04,0.04)(0.04,0.04) to (0.5,0.5)(0.5,0.5), losing that preference. Section 2.5 explains local sensitivity but not why the student's output should match it.

  2. If Equation (2) uses full-vocabulary probabilities, its maximum target probability is bounded by e/(V−1+e)e/(V-1+e), where V is the vocabulary size. This makes entropy regularization a plausible alternative explanation for the gains. It also affects AS3, which compares the student's confidence with this small maximum probability.

Questions for authors

Why should greater sensitivity imply a higher output probability?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.