acceptodds
Under review as a conference paper at ICLR 2027

Do As I Do, Not As I Say - On-Policy Distillation Inherits the Teacher in Vision-Language Models

Abstract

Improving the reasoning capabilities of vision-language models (VLMs) often relies on post-training with supervision from stronger models. Two common strategies are supervised fine-tuning (SFT), which trains on teacher-generated demonstrations, and on-policy distillation (OPD), which provides teacher feedback on trajectories generated by the student itself. While OPD can be more data-efficient by supervising student-generated trajectories, it is unclear how the teacher's supervision at each state affects the student's learned behavior. To study this question, we conduct a controlled comparison of OPD and filtered SFT across data budgets and tasks where the teacher exhibits different levels of reliability, and analyze how these differences shape student accuracy and completion behavior. Across the two tasks, we find that OPD is effective only when the teacher remains reliable on student-generated trajectories. On ChartQA, where the teacher completes 99% of its solutions, OPD trained on only 10% of the data comes within 1.4 accuracy points of full-data SFT. In contrast, on Geometry3K, where the teacher fails to complete 22% of its solutions, OPD underperforms SFT by 7.9 points, with most of the gap explained by incomplete responses. More strikingly, starting OPD from an SFT student reduces its completion rate from 76% to 62%, effectively erasing the completion behavior learned through SFT. Filtering teacher demonstrations for completeness alone recovers much of SFT's advantage. These results show that on-policy supervision can transfer not only teacher knowledge but also persistent failure modes on student-visited states. Code is available at https://anonymous.4open.science/r/iclr2027-4DA2/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, knowledge distillation, vision-language models, multimodal reasoning, chart question answering, geometry problem solving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_t8bhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares on-policy distillation (OPD) with SFT on correct teacher solutions for distilling Qwen3-VL-8B into Qwen3-VL-2B, and finds OPD competitive on ChartQA. On Geometry3K, where the teacher leaves 22% of its solutions unfinished, OPD students finish less often and trail SFT by up to 7.9 points.

Critical strengths
  1. The design is careful, and the main effect is robust. The Geometry3K gap at 900 and 2,101 problems holds under every interval type and numeric tolerance tested.

  2. The completion analysis shows a failure that accuracy alone hides. Starting OPD from the SFT student drops completion from 76% to 62%, a useful warning now that OPD is a standard post-training stage.

Critical weaknesses
  1. The title's explanation is not tested directly, and this is the main reason for my rating. Teacher reliability only changes between the two tasks, which also differ in token budget, solution length, and difficulty. The paper itself notes that the causal question "awaits a teacher trained to conclude from such states".

  2. Some results seem to fit a simpler reading, that OPD teaches little about finishing rather than passing on a teacher failure. From the base student, plain OPD (0.63) and OPD-term (0.64), which drops truncated rollouts from the loss, stay near the zero-shot level (0.65). Also, more than half of the completion gap to SFT comes from the 391 problems the teacher itself finishes (Table 9).

  3. The evidence seems narrow for the abstract's rule that OPD is effective only with a reliable teacher. The rule rests on one task where the teacher is reliable and one where it is not, with one teacher-student pair. The 32B teacher was only tried on ChartQA, where completion is not an issue.

  4. The high-level lesson seems somewhat expected, so the paper's value rests on how general the completion failure is. DAgger [1] already assumes that the expert's action on the learner's states is worth imitating (Section 2.1). Qiu et al. [2] and Sun et al. [3], cited by the paper, also report weaker teacher feedback on student prefixes with errors.

  5. The positive half of the rule also seems weak. On ChartQA, OPD's lead over a length-tuned SFT is 1.1 to 1.5 points, which the paper itself calls not robust. At full data, SFT is 1.5 points ahead.


References:

[1] Stéphane Ross et al. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011.

[2] Chenhao Qiu et al. OPD-Aha: From linguistic momentum to visual reflection in multimodal on-policy distillation. arXiv preprint arXiv:2609.16459, 2026.

[3] Changhui Sun et al. Step-level on-policy distillation: Interpolating between on-policy distillation and supervised fine-tuning. arXiv preprint arXiv:2608.16333, 2026.

Questions for authors

Questions:

  1. Can teacher reliability be changed within Geometry3K, for example with a larger rollout budget, the 32B teacher, or OPD only on problems the teacher finishes? If completion then tracks teacher reliability, the title claim would be much better supported, and I could raise my score.
  2. Does OPD-term started from the SFT student also lose completion? If it keeps completion, that would point to the teacher's feedback on overlong prefixes and would ease my main concern.
  3. Does the 7.9-point accuracy gap at 900 problems hold if OPD's step count, now fixed at 150, grows with the data? If it holds, the accuracy claim would be more convincing to me.

Suggestions:

  1. Without a within-task test, the title and abstract could present the link to the teacher as an association.
  2. Showing how many truncated OPD outputs are repetition loops would help readers see what the student learned.
  3. With the vision encoder frozen, a short note on what is specific to vision-language models would help.
4 comments

Thank you for the detailed review and for stating which experiments would change your assessment. We ran the within-task test (Q1) and the two other experiments you asked for (Q2, Q3). Intervals are 95% and question-clustered over four seeds unless stated otherwise.

W1 and Q1. Changing the teacher within Geometry3K. Within Geometry3K the student's completion changes with the teacher. It is 0.62 under the paper's teacher and 0.97 under InternVL3-8B, with the student, the data and the recipe fixed. We ran two of the three routes you suggest and added three teachers.

Qwen3-VL-8B (paper)Qwen3-VL-32BInternVL3-8BInternVL3-2B: answer line (any answer form)
Student completion, OPD from the SFT student (0.760 before OPD)0.6230.6180.9740.362 (0.972)
Student completion, OPD from the base student (0.648 before OPD)0.6280.6150.9840.372 (0.978)
Teacher concludes from the truncated outputs of the OPD / SFT / zero-shot student0.15 / 0.15 / 0.210.29 / 0.25 / 0.430.79 / 0.83 / 0.660.46 / 0.46 / 0.56 (0.77 / 0.79 / 0.74)
  • InternVL3-8B. Completion is 35.1 points higher [31.5, 38.8] from the SFT student and 35.6 points higher [31.9, 39.2] from the base student than under the paper's teacher, in every seed.
  • The 32B teacher. It is more accurate than the 8B teacher (0.658 against 0.570 zero-shot) and completes more of its own solutions (0.82 against 0.78). From the base student, the student's accuracy rises from 0.402 to 0.448 (+4.5 [2.2, 6.9]). Its completion does not change: 0.615 against 0.628 from the base student (-1.4 [-3.6, 0.9]) and 0.618 against 0.623 from the SFT student (-0.5 [-2.8, 1.8]).
  • OPD only on problems the teacher finishes. Completion is 0.631 against 0.628 (+0.4 [-1.7, 2.4]). When OPD starts from an SFT student trained on the same subset, completion still falls by 10.7 points [8.1, 13.3].
  • A larger rollout budget. We have not trained OPD with a larger rollout budget. Evaluated alone, the teacher completes 422 of the 500 test problems at 4,096 tokens, against 391 at 1,536.
  • Qwen3-VL-8B fine-tuned to conclude from student prefixes, with a control teacher fine-tuned on unselected continuations. Its conclusion rate rose by 11 points; we had required 15 in advance. The student's completion did not change (0.650 against 0.652 from the SFT student, 0.635 against 0.627 from the base student).
  • InternVL3-2B. This teacher stops on its own in 97% of its solutions but writes the answer line in only 52% of them. Otherwise it gives a boxed value or a sentence. Its student does the same: 97% of its outputs stop on their own, 36 to 37% contain the answer line, and 97% contain an answer in some form.

Completion therefore tracks the teacher, but not the teacher's reliability on its own solutions: the 32B teacher and the restriction to finished problems both improve that measure and leave the student unchanged. The continuation probe agrees with all the results. On the student's truncated outputs the two Qwen teachers conclude in 15 to 43% of cases and the two InternVL3 teachers in 66 to 83% (any answer form). The student's completion is 0.62 under the former and 0.97 to 0.98 under the latter (any answer form). In practice, a stronger teacher of the same family does not fix the problem.

One limitation remains. The two teachers under which the student completes its solutions are both InternVL3 models, and the relation is a step between two groups of teachers, not a gradual one. The experiment therefore shows the effect of the teacher and does not identify the property of the teacher that produces it.

W2. "OPD teaches little about finishing." The teacher replacement separates this reading from ours. From the same zero-shot student, OPD reaches 0.984 completion under InternVL3-8B and 0.628 under the paper's teacher. OPD therefore does teach the student to finish when the teacher finishes. Under the paper's teacher the two readings predict the same numbers, which is why the submitted paper could not distinguish them.

On Table 9 (Table 10 in the revised PDF): the 391 problems are those that the teacher finishes when it solves them alone. On these same problems the teacher concludes only 24% of the student's truncated prefixes, against 84% of the student's concluded prefixes. A completion gap on these problems is therefore also expected if the student follows the teacher.

Two further results bear on this point. OPD from the SFT student removes a behaviour the student already had: the termination rate of its rollouts falls from 0.79 in the first five steps to 0.54 in steps 21 to 40. And an InternVL3-2B student, which zero-shot writes the answer line in only 52% of its outputs, writes it in 96% after OPD under InternVL3-8B.

Q2. OPD-term started from the SFT student. It also loses completion. It ends at 0.613, against 0.623 for standard OPD from the SFT student (-1.0 [-2.9, 0.9]) and 0.760 for the SFT model. The loss of completion therefore does not require scoring unterminated rollouts. In these runs only the rollouts that ended carry loss (63% of all rollouts), and the termination rate of the student's rollouts still falls from 0.77 in the first ten steps to 0.56 in steps 21 to 40. Under standard OPD it falls from 0.77 to 0.54. By the criterion you propose, this result does not point to the teacher's feedback on overlong prefixes. The token-level measurements agree. Removing the end-of-turn and answer tokens from the loss leaves completion at 0.607 (global comment, Section 3). Under InternVL3-8B, standard OPD from the same SFT student ends at 0.974. The choice of teacher therefore changes the outcome, and we have not identified which tokens of the solution carry the effect.

Q3. Does the 7.9-point gap hold when OPD's step count grows with the data? Only in part. With 450 steps, three times the protocol's 150, SFT leads by 3.5 accuracy points [1.0, 6.1] and 10.5 completion points [7.9, 13.2]. With 600 steps it leads by 2.4 accuracy points [-0.1, 5.0] and 8.5 completion points [6.0, 11.1]. SFT does not improve with two or four times its own steps. More than half of the 7.9-point accuracy gap was therefore due to the fixed step count. The completion gap narrows from 13 to 8.5 points. The revision gives 7.9 points as a result at fixed update counts, together with the figures at 600 steps.

W3. Narrow evidence for the rule. We added the following.

  • A second family, InternVL3. On Geometry3K its teacher completes 97% of its solutions, and OPD from the SFT student ends 3.0 completion points below SFT, against 13.8 for Qwen. On ChartQA at 300 questions OPD leads SFT by 3.95 points [1.35, 6.55] (seed-paired; SFT at the protocol's fixed length, which we did not tune for this family).
  • A 4B student. OPD from the base student ends 5.3 completion points below SFT. OPD from the 4B SFT student lowers completion by 3.9 points [1.8, 6.1] in every seed, against 13.8 for the 2B student. We had fixed 5 points in advance as the threshold for a replication, so we report a smaller effect in the same direction.
  • The 32B teacher on Geometry3K, and two InternVL3 teachers for the Qwen student (W1 above).
  • ScienceQA as a third task (exploratory). The teacher completes 96% of its solutions and OPD still trails SFT by 3.4 points. This contradicts the rule in the submitted abstract, and we removed the rule (W5).

W5 and S1. The positive half of the rule, and the wording. We agree. On ChartQA, OPD's lead over a length-tuned SFT is 1.1 to 1.5 points with intervals that include zero, and on ScienceQA OPD trails SFT although the teacher completes its solutions. The revised abstract no longer states that OPD is effective only with a reliable teacher. It reports the ChartQA and the Geometry3K results as measured, and the teacher replacement. The introduction no longer gives a rule. It says that OPD was competitive with SFT at small budgets on ChartQA, that this advantage is small, and that on ScienceQA OPD trails SFT. On S1: the within-task test now exists, so the abstract states the effect of the teacher as a result and not as an association.

W4. What is new relative to DAgger, Qiu et al. and Sun et al. We agree that the premise is not new. DAgger's guarantee assumes that the expert's action on the learner's states is worth imitating. Qiu et al. and Sun et al. both argue that teacher feedback on erroneous student prefixes is weak, and each proposes a remedy (Qiu et al. compare a teacher with and without privileged visual evidence; Sun et al. propose step-level distillation in text-only settings). Fu et al. (arXiv:2603.25562) list unreliable teacher guidance on student-generated prefixes among the failure modes of sampled-token OPD. Li et al. (arXiv:2604.13016) and Xie et al. (arXiv:2606.22600) find that the accuracy advantage of a teacher that continues a student's prefix shrinks as the prefix grows.

What we add is a set of measurements. (1) A standard setting in which the assumption fails for one specific behaviour, finishing a solution, although the teacher looks adequate by its own evaluation: it completes 78% of its own solutions and is 20 points more accurate than the student. (2) OPD under this teacher removes a behaviour that SFT had already given the student. (3) A probe that needs only inference detects the problem. Li et al. and Xie et al. report the accuracy of the teacher's continuation (Li et al. as a gain over the student's own continuation), which does not distinguish a wrong answer from no answer. Our probe records whether the teacher concludes. (4) Changing the teacher changes the outcome.

On how general the failure is: in our runs the completion gap to SFT appears for 2B and 4B students, under the 8B and the 32B Qwen teachers, and from both starting points (3.9 points for the 4B student from its SFT model, 13.8 for the 2B student). It is absent or small (3 points) under InternVL3-8B. Text-only studies, which the revision now cites, report closely related behaviour: responses that lengthen under OPD until they reach the generation limit (Luo et al., arXiv:2604.08527; Yang et al., arXiv:2609.20511; Wang et al., arXiv:2607.13399), with most rollouts truncated in Luo et al. Luo et al. attribute this to a self-reinforcing loop on repetitive tokens, Yang et al. in part to end-of-sequence tokens that differ between student and teacher, and Wang et al. to a sequence-averaged objective. Zhao et al. (arXiv:2605.16826) report that a mixture weighted toward the forward KL keeps response length stable where reverse-heavy mixtures do not. Our setting differs in three respects. The student and the teacher both put 0.98 of their probability on the same end token where an output stops. The loss is a full-distribution KL at every position and not a sampled-token advantage. Three changes to the objective do not restore completion (the loss on ended rollouts only, removal of the end-of-turn and answer tokens, and the forward KL), whereas replacing the teacher does.

S2. Repetition loops. Among the OPD students' outputs without a parsed answer, 14% end in a verbatim loop. The share is 49% for the zero-shot student and 1% for SFT. By a looser measure, 43% of OPD's unfinished outputs have low 5-gram diversity in their last 1,500 characters (87% for the zero-shot student, 1% for SFT). Where the evaluation recorded the stop reason (300 unfinished outputs of OPD and SFT students), every unfinished output ran to the token budget. OPD therefore removes most of the base student's verbatim loops. What remains is a student that continues to derive without concluding, which is also what its teacher does on these states. The numbers are in Appendix S.

S3. What is specific to vision-language models. We do not claim that the effect is specific to vision-language models, and we have not tested whether it depends on image inputs. Replacing the teacher also replaces the vision encoder through which the teacher sees the image, so the experiment does not isolate the contribution of vision. Text-only studies also report responses that run to the generation limit under OPD (Luo et al., arXiv:2604.08527; Yang et al., arXiv:2609.20511), so the symptom is not confined to image inputs. The causes proposed there (repetition, mismatched end-of-sequence tokens) may differ from ours.

Changes in the revised paper for this review. Section 4.3: Table 3 and the teacher replacement, step scaling, and OPD-term from the SFT student. Abstract and introduction: the rule is removed and the effect of the teacher is stated as a result. Related work: positioning against the cited and the newly cited papers. Appendix S: repetition statistics.

Reviewer_Q2QeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper conducts a controlled experimental comparison between OPD and filtered SFT, analyzing how factors such as data budget, teacher reliability, and training strategies influence student model accuracy and response completion rates.

Critical strengths
  1. The paper addresses the practically significant issue of distillation reliability, providing robust experimental insights through the use of multiple random seeds, budget control, and analysis of response completion rates.

  2. The paper reports experimental results based on four random seeds and employs paired difference analysis, confidence intervals, and grouped statistical analysis, reflecting a rigorous overall experimental approach.

  3. The paper focuses on the high cost of model parameter updates during few-shot domain adaptation in the visual modality; the research objective is well-defined and holds practical application value.

Critical weaknesses
  1. Because the two methods utilize different numbers of training steps—and differ in their supervision formats, data filtering processes, and teacher-querying mechanisms—a strictly fair comparison between them is difficult to achieve.

  2. Efficiency evaluations—covering total GPU time, VRAM usage, and FLOPs—should be included.

  3. All core conclusions are derived exclusively from experiments using Qwen3-VL-Instruct; the reproducibility of these findings across other VLM families—such as LLaVA-OneVision, InternVL, Phi-3-Vision, and DeepSeek-VL—remains entirely unknown.

  4. The dataset scale and diversity are severely limited. The study covers only two types of VQA tasks (ChartQA and Geometry3K), and the out-of-distribution evaluations on CharXiv and ChartQAPro remain heavily skewed toward chart comprehension.

Questions for authors

For further details, please refer to the "Critical Weaknesses" section.

4 comments

Thank you for the positive assessment and for the four concrete requests. We ran an experiment for each. Intervals are 95% and question-clustered over four seeds unless stated otherwise.

W1. Fairness of the comparison (step counts, supervision format, filtering, teacher queries). We varied the number of steps of both methods on Geometry3K at 900 problems.

StepsAccuracyCompletion
OPD150 (paper)0.4020.628
OPD3000.4240.639
OPD4500.4470.655
OPD6000.4580.675
SFT302 (paper)0.4810.760
SFT6040.4710.765
SFT1,2080.4570.770

OPD improves with more steps and SFT does not. The fixed-step protocol therefore understated OPD's accuracy. At 600 OPD steps SFT leads by 2.4 accuracy points [-0.1, 5.0] instead of 7.9. The completion gap is still 8.5 points [6.0, 11.1] at 600 steps. The revision reports both numbers and describes the 7.9 points as a result at fixed update counts.

On the other differences you list: both methods use the same teacher, the same student with the same LoRA configuration, the same pool of training questions at each budget (SFT uses those the teacher answered correctly), and the same evaluation. The form of supervision is what the paper compares, so it differs by design. For filtering we added a control. SFT on complete-only teacher solutions, matched in number and length to the correctness-filtered set, reaches the same completion (0.760 against 0.760). For teacher queries, the cost table below includes SFT's one-off teacher generation and OPD's per-step teacher scoring.

W2. Efficiency. Each run uses one GPU (L40S, 48 GB). The trainers log wall time and peak memory. FLOPs are an analytic estimate, and the appendix states its assumptions.

RunTraining timeGPU-hours incl. evaluationPeak memoryEstimated training FLOPs
Geometry3K, SFT, 900 problems15.5 min + 5 min teacher data generation0.3619.5 GB6.6e16
Geometry3K, OPD, 150 steps212 min3.5639.1 GB1.1e17
Geometry3K, OPD, 600 steps816 min13.639.0 GB4.4e17
ChartQA, SFT, 300 questions10.8 min + 0.4 min teacher data generation0.1911.3 GB4.2e16
ChartQA, OPD, 150 steps50 min0.8427.1 GB5.5e16

On Geometry3K, OPD takes about ten times the GPU time of SFT and 1.7 times the estimated FLOPs. Most of the time goes to token-by-token rollouts (70 of 85 seconds per step). At 600 steps, where OPD comes within 2.4 accuracy points of SFT, it takes about 38 times the GPU time of SFT. On ChartQA the ratio is 4.4. The revised paper gives this table in Appendix P and states the cost ratio together with the data-efficiency result in Section 4.1.

W3. Other model families. We added InternVL3 (2B student, 8B teacher), with four seeds per arm.

  • ChartQA, 300 questions. SFT reaches 0.647 and OPD 0.687. OPD leads by 3.95 points [1.35, 6.55] (seed-paired). The lead for Qwen is 2.7 points.
  • Geometry3K, 900 problems. The InternVL3-8B teacher completes 97% of its solutions. Completion is 0.996 for SFT, 0.962 for OPD and 0.966 for OPD from the SFT student. The drop from SFT is 3.0 points, against 13.8 for Qwen. The accuracies are 0.401, 0.390 and 0.397. For these trained students the answer-line rate and the rate of an answer in any form are the same.

We also ran a Qwen3-VL-4B student with the 8B teacher. OPD from the base student ends 5.3 completion points below SFT. OPD from the 4B SFT student lowers completion by 3.9 points [1.8, 6.1], against 13.8 for the 2B student.

Finally, we trained the Qwen student with two InternVL3 teachers. With InternVL3-8B the student's completion rises from 0.62 to 0.97 (global comment, Sections 1 and 2). We did not test the other families you name (Limitations).

W4. Dataset scale and diversity. We agree that two tasks are a narrow basis. We chose them because the same teacher completes 99% of its solutions on one and 78% on the other. We added ScienceQA as a third task, which we treat as exploratory (one budget, four seeds, fixed step counts for both methods). SFT reaches 0.896 accuracy and OPD 0.862, a deficit of 3.4 points [1.75, 5.1], although the teacher completes 96% of its solutions. A teacher that completes its solutions is therefore not sufficient for OPD to match SFT, and we removed the statement in the abstract that implied it. The out-of-distribution sets are still chart-centred (Limitations).

Changes in the revised paper for this review. Section 4.3 and Appendix O: step scaling for both methods. Section 4.1 and Appendix P: compute cost. Section 4.4 and Appendix R: the second family, the 4B student and ScienceQA. Abstract: the rule on teacher reliability is removed. Limitations: rewritten.

I appreciate the detailed response and the additional experiments provided by the authors. The new experiments address some of my previous concerns. However, two issues still warrant attention: first, the significant computational overhead introduced by OPD; and second, the variation in results across different models and datasets. The authors have clarified the scope of their conclusions and explicitly acknowledged these limitations. I will maintain my initial rating: 3 (marginally accept).

Thank you for considering the new experiments. We agree with both points. OPD takes about ten times the GPU time of SFT on Geometry3K and 4.4 times on ChartQA. Appendix P reports these costs, and Section 4.1 states the ChartQA cost next to the data-efficiency result.

The results do vary across teachers and tasks. The limitations section describes the settings we have not tested.

Reviewer_oBNgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical investigation of on-policy distillation (OPD) for vision-language models, focusing on how the teacher's failure modes may affect the student's learning behavior. The authors compare OPD against correctness-filtered supervised fine-tuning (SFT) using Qwen3-VL models on ChartQA and Geometry3K. The results suggest that OPD can achieve competitive data efficiency when the teacher reliably completes its responses, but may substantially underperform filtered SFT when the teacher frequently fails to produce a final answer.

A particularly interesting observation is that OPD can degrade the completion behavior of an SFT-initialized student, reducing its completion rate from approximately 76% to 62%. Through teacher continuation probes, termination probability analysis, and several OPD variants, the authors further investigate the relationship between teacher reliability on student-generated prefixes and the observed performance degradation. The paper also finds that completeness-based filtering alone can recover much of the benefit of correctness-filtered SFT.

Critical strengths
  1. Important and practically relevant research question.

The paper investigates an underexplored limitation of OPD: teacher supervision may be unreliable on the states visited by the student, even when the teacher performs reasonably well under standard evaluation. This is particularly relevant to long-horizon reasoning and agentic model training, where student-generated trajectories can deviate substantially from the teacher's typical behavior.

  1. Interesting empirical findings on completion degradation.

The observation that OPD can substantially reduce the completion rate of an already SFT-trained student is compelling. In particular, the degradation from approximately 76% to 62% suggests that OPD does not necessarily preserve previously acquired task-completion behaviors. The decomposition of the accuracy gap into completion-related and answer-correctness-related components provides additional insight into the failure mode.

  1. Useful diagnostic analyses and ablations.

The authors conduct teacher continuation probes, analyze end-of-turn probabilities, and evaluate several OPD variants, including forward KL, lower rollout temperature, termination-based filtering, sequence-level loss normalization, and SFT initialization. These experiments help characterize the observed phenomenon and rule out several simple explanations.

Critical weaknesses
  1. The central causal claim is not sufficiently supported.

The paper argues that OPD inherits the teacher's failure modes, particularly its inability to complete responses on student-generated prefixes. While the experimental observations are consistent with this hypothesis, they do not establish a direct causal relationship.

The teacher continuation probes demonstrate that the teacher frequently fails to complete responses when conditioned on truncated student trajectories. However, such trajectories may correspond to intrinsically more difficult questions or already-degraded reasoning states. Therefore, the low teacher completion rate could partly reflect differences in task difficulty or prefix quality rather than a failure mode directly transferred through OPD.

  1. The experimental scope is relatively limited.

The main experiments focus on a single model family, Qwen3-VL, with a 2B student and primarily an 8B teacher, evaluated on only two main benchmarks, ChartQA and Geometry3K. Importantly, these benchmarks differ not only in teacher completion rate but also in task complexity, reasoning length, answer characteristics, and teacher accuracy. Consequently, it is difficult to isolate teacher completion reliability as the primary factor explaining the different OPD outcomes.

  1. The proposed gradient-based explanation lacks direct empirical validation.

The analysis of reverse-KL gradients with respect to the end-of-turn token provides an interesting possible explanation for the observed behavior. However, the paper does not directly measure the student's termination probabilities or the corresponding gradient contributions during training. As a result, it remains unclear whether the observed completion degradation is primarily driven by weak termination gradients, the teacher's low termination probability, or other optimization dynamics.

Questions for authors
  1. Can the authors provide a controlled experiment in which the teacher's completion reliability is explicitly improved, while the remaining OPD training setup is unchanged? Would this intervention recover the student's completion rate?

  2. To what extent does the teacher continuation failure persist when controlling for question difficulty and comparing successful and unsuccessful prefixes from the same question?

  3. In the completeness-only SFT experiments, how do the results change when the number and length distribution of demonstrations are matched to those of correctness-filtered SFT?

5 comments

Thank you for the careful review. Your main concerns were the causal reading of the teacher probe (W1, Q1) and the gradient explanation that we had not validated (W3). We ran an experiment for each, and the second one changed our explanation. Intervals are 95% and question-clustered over four seeds unless stated otherwise.

W1 and Q1. The causal claim, and a controlled change of the teacher. We changed the teacher within Geometry3K and kept the student, the data, the recipe and the objective fixed. The result has two parts.

A teacher that concludes recovers the student's completion. InternVL3-8B completes 97% of its own solutions and concludes from 66 to 83% of the student's truncated outputs. With this teacher the student's completion is 35.1 points higher [31.5, 38.8] from the SFT student and 35.6 points higher [31.9, 39.2] from the base student, in every seed. From the SFT student it ends at 0.974. The SFT student itself is at 0.760, and OPD under the paper's teacher ends at 0.623.

Improving the Qwen teacher did not change the student's completion. Qwen3-VL-32B is more accurate than the 8B teacher and completes more of its own solutions, but it concludes from only 25 to 43% of the student's truncated outputs. The student's completion stays at 0.62. We also fine-tuned the 8B teacher to conclude from student prefixes, with a control teacher fine-tuned on unselected continuations. The teacher's conclusion rate rose by 11 points, and we had required 15 in advance. The student's completion did not change (0.650 against 0.652 from the SFT student, 0.635 against 0.627 from the base student). Sampled completion at temperature 1 rose by about 5.5 points in both arms, but we had not specified that readout in advance.

TeacherTeacher alone: accuracy / completionTeacher concludes from truncated outputs of the OPD / SFT / zero-shot studentStudent completion after OPD, from the SFT studentStudent completion after OPD, from the base student
Qwen3-VL-8B (paper)0.570 / 0.780.15 / 0.15 / 0.210.6230.628
Qwen3-VL-8B, fine-tuned to conclude (a)0.559 / 0.830.16 / 0.12 / 0.240.6500.635
Qwen3-VL-32B0.658 / 0.820.29 / 0.25 / 0.430.6180.615
InternVL3-8B0.456 / 0.9740.79 / 0.83 / 0.660.9740.984
InternVL3-2B, by the answer line0.194 / 0.5180.46 / 0.46 / 0.560.3620.372
InternVL3-2B, by an answer in any form (b)0.322 / 0.9480.77 / 0.79 / 0.740.9720.978

Before OPD the SFT student completes 0.760 of its solutions and the base student 0.648. "Teacher concludes" is the share of the student's truncated test outputs that the teacher completes when it continues them greedily from token 256 for 512 more tokens. Row (a) was measured with sampled continuations and in a second environment, and readout (b) was added after we saw the outputs. The global comment (Section 1) gives the details of both.

The same student, trained on the same data with the same objective, keeps or loses its completion depending on the teacher. The results do not show which property of the teacher is responsible, because the two teachers under which the student completes its solutions belong to one family. For all five teachers, the student's completion follows what the teacher does when it continues the student's unfinished outputs. The InternVL3-2B rows show in addition that the student adopts the teacher's answer format (global comment, Section 2).

Q2. Controlling for question difficulty. We compared prefixes from the same question. 275 test questions have both a concluded and a truncated student output. On these questions the 8B teacher concludes 59.6% of its continuations from the concluded prefixes and 26.5% from the truncated ones, a paired difference of 33.1 points [28.2, 37.8]. On the questions that the teacher itself answers correctly, the two rates are 85.1% and 26.3%. Without the control the rates are 77.9% and 15.8%, a difference of 62 points. Question difficulty therefore accounts for part of the gap, and about half of it remains within question. The truncated prefixes can also be completed by another teacher. InternVL3-8B concludes 79% of the OPD students' truncated prefixes (its continuations are correct in 22% of cases), and Qwen3-VL-8B concludes 15%. These results are in Appendix S.

W2. Scope. We made four additions.

  1. The teacher is varied within one task (W1 above). The task, its solution length and its answer format are then fixed while the teacher changes, which removes the confound you describe.
  2. A second model family, InternVL3 (2B student, 8B teacher). On Geometry3K its teacher completes 97% of its own solutions, and OPD from the SFT student ends 3.0 completion points below SFT [1.2, 4.8] (seed-paired). For Qwen the drop is 13.8 points. On ChartQA at 300 questions OPD leads SFT by 3.95 points [1.35, 6.55] (seed-paired).
  3. A larger student, Qwen3-VL-4B, with the 8B teacher. OPD from the base student trails SFT by 5.3 completion points and 5.1 accuracy points. OPD from the 4B SFT student lowers completion by 3.9 points [1.8, 6.1], against 13.8 for the 2B student.
  4. A third task, ScienceQA, for the Qwen pair (exploratory; global comment, Section 5).

Each family was still run on two main tasks only (Limitations).

W3. The gradient explanation. The gradient explanation was not validated, and the measurements you asked for do not support it. At every OPD step and on the student's own rollouts, we log the probability that the student and the teacher assign to the end-of-turn token and to the tokens that open the answer line, and the loss gradient on those logits (two seeds from the SFT student, four from the base student).

  • Termination probabilities. On rollouts that run to the budget, both models assign about 1e-6 to stopping at every position. On rollouts that conclude, the teacher opens the answer line with probability 0.78 to 0.81. The student, started from SFT, does so with probability 0.91 in the first 25 steps and 0.86 to 0.87 afterwards.
  • Gradient contributions. The end-of-turn and answer-opening tokens together carry less than 0.5% of the gradient magnitude at the logits.
  • Intervention. We removed these tokens from both distributions in the KL by masking their logits in the student and the teacher, so the loss has no gradient on them. Completion is then 0.607 against 0.623 (-1.6 [-3.5, 0.4]).

You list three candidates. A weak termination gradient is not the driver, because completion falls just as much when the termination terms are removed from the loss. The teacher's termination probability on the student's overlong prefixes is indeed negligible, which is why the loss contains almost no signal toward stopping there. These prefixes are not required either: completion also falls when the loss is computed only on rollouts that end (0.613). The change in the student is therefore produced by the loss on the remaining tokens of the solution. We have not identified which of these tokens are responsible. Section 4.3 now reports these measurements in place of the gradient argument, and Appendix Q gives the details.

Q3. Complete-only SFT matched in number and length. We subsampled the complete-only demonstrations to the number and the response-length distribution of the correctness-filtered set. At 900 problems the matched set gives 0.467 accuracy and 0.760 completion. Correctness-filtered SFT gives 0.481 and 0.760. The differences are 0.0 points in completion [-2.3, 2.2] and -1.5 points in accuracy [-4.0, 0.9]. The number and length of the demonstrations therefore do not explain the completion level.

Changes in the revised paper for this review. Section 4.3: Table 3 and the paragraph on replacing the teacher, the token-level measurements, and the matched complete-only control. Appendix S: the same-question control. Appendices N, Q and R: per-seed results, token-level details, the second family and the 4B student. Abstract and introduction: the causal wording.

Reviewer_oBNgOpenReview ↗

Response to the authors

Thanks a lot to your detailed response. Most of my concerns have been resolved and I would raise my score. But I suggest adding such experiments into the final verison. (if possible, models such Qwen3.5 series are better considered to ensure a broad applicability for your conclusion)

Comments on the paper

4 comments

We thank the three reviewers. All three reviews point to the same gap. The paper attributed the completion failure to the teacher, but the teacher's reliability varied only between two tasks. We have now varied the teacher within Geometry3K while keeping the student, the data and the training recipe fixed. Replacing the Qwen3-VL-8B teacher with InternVL3-8B raises the student's completion from 0.62 to 0.97, from both starting points and in all four seeds.

Sections 1 and 2 report this experiment. Sections 3 to 5 report the other experiments the reviews asked for, and Section 6 lists the changes to the paper. Every training arm has four seeds. Intervals are 95% and question-clustered unless stated otherwise. We fixed the readout of each new experiment before running it; the exceptions are marked.

1. Replacing the teacher (oBNg W1/Q1, t8bh W1/Q1). Every row uses the Qwen3-VL-2B student and 900 Geometry3K problems. Only the teacher differs. Before OPD, the SFT student completes 0.760 of its solutions and the base student 0.648.

TeacherTeacher alone: accuracy / completionTeacher concludes from truncated outputs of the OPD / SFT / zero-shot studentStudent completion after OPD, from the SFT studentStudent completion after OPD, from the base student
Qwen3-VL-8B (paper)0.570 / 0.780.15 / 0.15 / 0.210.6230.628
Qwen3-VL-8B, fine-tuned to conclude (a)0.559 / 0.830.16 / 0.12 / 0.240.6500.635
Qwen3-VL-32B0.658 / 0.820.29 / 0.25 / 0.430.6180.615
InternVL3-8B0.456 / 0.9740.79 / 0.83 / 0.660.9740.984
InternVL3-2B, by the answer line0.194 / 0.5180.46 / 0.46 / 0.560.3620.372
InternVL3-2B, by an answer in any form (b)0.322 / 0.9480.77 / 0.79 / 0.740.9720.978

"Teacher alone" is the teacher's zero-shot result on the test set. "Teacher concludes" is the share of the student's truncated test outputs that the teacher completes when it continues them greedily from token 256 for 512 more tokens (four seeds per trained student). A completed continuation need not be correct: InternVL3-8B's continuations are correct in 13 to 24% of cases. InternVL3 and Qwen3-VL assign the same token ids to generated text, so the OPD objective is the same for all teachers.

(a) This row was measured differently. The teacher's own numbers are on 900 held-out training problems (unmodified teacher: 0.543 / 0.79). Its conclusion rate is the share of 8 sampled continuations per prefix that reach an answer line (unmodified teacher: 0.06 / 0.04 / 0.10; over all prefixes, 6.8% before and 17.9% after fine-tuning). Its student arms were trained in a second compute environment (other library versions and GPUs), where the unmodified teacher gives 0.652 from the SFT student and 0.627 from the base student.

(b) In the paper, a completed output is one that contains the answer line. InternVL3-2B often gives its answer as a boxed value or in a sentence instead. We added the any-form readout after seeing these outputs. For all other teachers the two readouts differ by at most 1.2 points.

2. What the replacement shows.

(a) Replacing the teacher restores the student's completion. With InternVL3-8B as the teacher, the student's completion is 35.1 points higher [31.5, 38.8] from the SFT student and 35.6 points higher [31.9, 39.2] from the base student than with the paper's teacher. This holds in every seed. Before OPD the two starting points are 11 points apart. After OPD they are at most 2.5 points apart under any teacher, while the level ranges from 0.36 to 0.98 across teachers. Under InternVL3-8B the student reaches the teacher's own level in accuracy (0.443 to 0.449) and in completion (0.974 to 0.984). For the SFT student this is 21 points more completion and 3.3 points less accuracy [0.2, 6.5] than before OPD.

(b) A teacher that is better by its own evaluation does not restore completion. The 32B teacher is more accurate than the 8B teacher and completes more of its own solutions. Restricting OPD to the problems that the 8B teacher completes (t8bh Q1) removes the problems it leaves unfinished. Neither change affects the student's completion (-1.4 points [-3.6, 0.9] and +0.4 [-1.7, 2.4], from the base student). The continuation probe is consistent with this. On the student's truncated outputs, both Qwen teachers conclude in at most 43% of cases, and InternVL3-8B concludes in 66 to 83%. The 32B teacher does raise the student's accuracy, by 4.5 points [2.2, 6.9]. Accuracy and completion therefore depend on different properties of the teacher.

(c) The student adopts the teacher's answer format. InternVL3-2B stops on its own in 97% of its solutions. It writes the requested answer line in only 52% of them and otherwise gives a boxed value or a sentence such as "the answer is ...". Its student does the same from both starting points. 97% of the student's outputs stop on their own, and 36 to 37% contain the answer line, although the SFT student wrote the answer line in 76% of its outputs before OPD. By the answer line, this student is 61 points below the students of InternVL3-8B. By an answer in any form, its completion is 0.97 and does not differ from theirs (-0.2 [-1.2, 0.8] and -0.7 [-1.6, 0.2]). Its accuracy is 7 to 9 points lower; InternVL3-2B is also the less accurate teacher (0.322 against 0.456).

(d) What the experiment does not show. The experiment shows that the choice of teacher changes the student's completion. It does not show which property of the teacher is responsible, because the two teachers under which the student completes its solutions are both InternVL3 models. We tried to change one candidate property within the Qwen family by fine-tuning the 8B teacher to conclude from student prefixes. Its conclusion rate rose by 11 points, below the 15 points we had required in advance, and the student's completion did not change against the unmodified teacher (-0.15 points [-4.1, 3.8] from the SFT student, +0.85 [-4.0, 5.7] from the base student; seed-paired). The revised paper reports the effect of the teacher as a result, and the teacher's behaviour on the student's states as an explanation that agrees with all five teachers.

(e) A check before training. The continuation probe needs only inference. In our runs, the teachers under which the student lost completion conclude from 15 to 43% of the student's truncated outputs. The teachers under which it kept completion conclude from 66 to 83% (any answer form). On the zero-shot student's outputs alone, which exist before any training, the rates are 21 and 43% against 66 and 74%. The teachers' own accuracy orders them differently.

3. The stopping tokens do not explain the drop in completion (oBNg W3). During OPD we now log, at every step, the probability that the student and the teacher assign to the end-of-turn token and to the tokens that open the answer line, and the loss gradient on those logits (two seeds from the SFT student, four from the base student). On rollouts that run to the token budget, both models assign about 1e-6 to stopping at every position. The teacher does not stop on these states, so the gradient toward stopping is negligible under both the reverse and the forward KL. Two interventions test this. Removing these tokens from the KL leaves completion at 0.607 against 0.623 (-1.6 [-3.5, 0.4]). Computing the loss only on rollouts that end, starting from the SFT student (t8bh Q2), leaves it at 0.613. In both cases completion still falls from the SFT student's 0.760, so neither the stopping tokens nor the rollouts without a stop are needed for the drop. The loss on the remaining tokens of the solution therefore produces the change. We have not identified which of those tokens are responsible. In Section 4.3, these measurements replace the paragraph that attributed the failure to an attenuated stopping gradient.

4. Further controls.

  • Training steps (t8bh Q3, Q2Qe W1). With 450 OPD steps instead of 150, SFT leads by 3.5 accuracy points [1.0, 6.1] and 10.5 completion points [7.9, 13.2]. With 600 steps it leads by 2.4 accuracy points [-0.1, 5.0] and 8.5 completion points [6.0, 11.1]. SFT does not improve with two or four times its own steps. More OPD steps therefore close most of the accuracy gap and about a third of the completion gap.
  • Question difficulty (oBNg Q2). 275 test questions have both a concluded and a truncated student output. On these questions the teacher concludes 59.6% of its continuations from the concluded prefixes and 26.5% from the truncated ones (+33.1 points [28.2, 37.8]).
  • Matched complete-only SFT (oBNg Q3). Complete-only SFT with the same number of demonstrations and the same length distribution as correctness-filtered SFT reaches the same completion (0.760 against 0.760).
  • Cost (Q2Qe W2). On Geometry3K one OPD run takes 3.6 GPU-hours and 39 GB. One SFT run takes 0.36 GPU-hours and 19.5 GB.

5. Scope (Q2Qe W3/W4, oBNg W2, t8bh W3). Under both Qwen teachers the Qwen student ends at 0.62 completion from both starting points, against 0.76 for SFT, and a 4B Qwen student shows a smaller drop. Under InternVL3-8B it ends at 0.97 to 0.98, and an InternVL3 student also keeps its completion.

  • Second model family. With an InternVL3 2B student and 8B teacher on Geometry3K, the teacher completes 97% of its own solutions. OPD from the SFT student ends 3.0 completion points below SFT (0.966 against 0.996). The same comparison for Qwen gives 13.8 points. On ChartQA at 300 questions OPD leads SFT by 3.95 points [1.35, 6.55] (seed-paired; SFT at the protocol's fixed length).
  • Larger student. With a Qwen3-VL-4B student and the 8B teacher, OPD from the base student ends 5.3 completion points below SFT. OPD from the 4B SFT student lowers completion by 3.9 points [1.8, 6.1] and accuracy by 3.5 points [1.2, 5.7], in every seed. The completion drop for the 2B student is 13.8 points. We had fixed 5 points in advance as the threshold for a replication, so we report a smaller effect in the same direction.
  • Third task (exploratory). On ScienceQA the teacher completes 96% of its own solutions, and OPD at the protocol's 150 steps trails SFT by 3.4 accuracy points [1.75, 5.1]. A teacher that completes its solutions is therefore not sufficient for OPD to match SFT. The submitted abstract implied that it was, and we removed that statement.

6. Changes in the revised PDF (new and changed text is in blue).

  • Abstract and introduction: the statement that OPD is effective only with a reliable teacher is removed. The teacher replacement is reported as a result. The 7.9-point accuracy gap is described as a result at fixed update counts, next to the figure at 600 steps.
  • Section 4.3: the teacher replacement (Table 3), step scaling, the token-level measurements (which replace the gradient argument), and the matched complete-only control.
  • Section 4.4: the second family, the 4B student and ScienceQA.
  • Appendices N to S: per-seed results of all new arms, the fine-tuned teacher and the criterion it missed, step scaling, compute cost, token-level measurements, and further probe results.
  • Related work and limitations are rewritten.
  • To keep the main text within ten pages, some details of the probes and of the SFT data controls moved from Section 4.3 to Appendices D and F. The list of contributions in the introduction is merged into the list of findings, and several sentences are shortened without a change of content.

We thank the three reviewers for the discussion and the Area Chair for handling the paper. Since our first response, we have narrowed the conclusion, reported the control teacher's conclusion rate, added a Qwen3.5 pair, and corrected the InternVL3-8B continuation probe. All updates are in the revised PDF.

  1. Conclusion. It now says that the student's completion depends on the teacher, but which property of the teacher drives this difference remains open. The introduction and the limitations section say the same.
  2. Control teacher, Appendix N. The control and the unmodified teacher have similar conclusion rates on student prefixes (7.1% and 6.8%), but the control teacher lowers the student's completion. This comparison is therefore not evidence for the continuation probe within the Qwen family, and the paper says so.
  3. Qwen3.5 pair, Section 4.4 and Appendix R. With a Qwen3.5-2B student and a Qwen3.5-9B teacher, starting OPD from the SFT student reduces completion by 6.0 percentage points (95% question-clustered bootstrap interval for the drop: [3.8, 8.2]), compared with a 13.8-point drop for the Qwen3-VL pair.

Correction to our first response. We reran the InternVL3-8B continuation probe on exactly the same student outputs used for the other teachers. Its row in the teacher table is now 0.79 / 0.81 / 0.74 (outputs of the OPD / SFT / zero-shot student). Our first response reported 0.79 / 0.83 / 0.66, with part of the outputs generated in a second compute environment (different library versions and GPUs). Where the first response said that the two InternVL3 teachers conclude from 66 to 83% of the truncated outputs (any answer form), the range is 74 to 81%. InternVL3-8B's continuations are correct in 18 to 23% of cases (first response: 13 to 24%). The corrected rates remain well above the 15 to 43% of the Qwen teachers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.