- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper compares on-policy distillation (OPD) with SFT on correct teacher solutions for distilling Qwen3-VL-8B into Qwen3-VL-2B, and finds OPD competitive on ChartQA. On Geometry3K, where the teacher leaves 22% of its solutions unfinished, OPD students finish less often and trail SFT by up to 7.9 points.
Critical strengths
-
The design is careful, and the main effect is robust. The Geometry3K gap at 900 and 2,101 problems holds under every interval type and numeric tolerance tested.
-
The completion analysis shows a failure that accuracy alone hides. Starting OPD from the SFT student drops completion from 76% to 62%, a useful warning now that OPD is a standard post-training stage.
Critical weaknesses
-
The title's explanation is not tested directly, and this is the main reason for my rating. Teacher reliability only changes between the two tasks, which also differ in token budget, solution length, and difficulty. The paper itself notes that the causal question "awaits a teacher trained to conclude from such states".
-
Some results seem to fit a simpler reading, that OPD teaches little about finishing rather than passing on a teacher failure. From the base student, plain OPD (0.63) and OPD-term (0.64), which drops truncated rollouts from the loss, stay near the zero-shot level (0.65). Also, more than half of the completion gap to SFT comes from the 391 problems the teacher itself finishes (Table 9).
-
The evidence seems narrow for the abstract's rule that OPD is effective only with a reliable teacher. The rule rests on one task where the teacher is reliable and one where it is not, with one teacher-student pair. The 32B teacher was only tried on ChartQA, where completion is not an issue.
-
The high-level lesson seems somewhat expected, so the paper's value rests on how general the completion failure is. DAgger [1] already assumes that the expert's action on the learner's states is worth imitating (Section 2.1). Qiu et al. [2] and Sun et al. [3], cited by the paper, also report weaker teacher feedback on student prefixes with errors.
-
The positive half of the rule also seems weak. On ChartQA, OPD's lead over a length-tuned SFT is 1.1 to 1.5 points, which the paper itself calls not robust. At full data, SFT is 1.5 points ahead.
References:
[1] Stéphane Ross et al. A reduction of imitation learning and structured prediction to no-regret online learning. AISTATS, 2011.
[2] Chenhao Qiu et al. OPD-Aha: From linguistic momentum to visual reflection in multimodal on-policy distillation. arXiv preprint arXiv:2609.16459, 2026.
[3] Changhui Sun et al. Step-level on-policy distillation: Interpolating between on-policy distillation and supervised fine-tuning. arXiv preprint arXiv:2608.16333, 2026.
Questions for authors
Questions:
- Can teacher reliability be changed within Geometry3K, for example with a larger rollout budget, the 32B teacher, or OPD only on problems the teacher finishes? If completion then tracks teacher reliability, the title claim would be much better supported, and I could raise my score.
- Does OPD-term started from the SFT student also lose completion? If it keeps completion, that would point to the teacher's feedback on overlong prefixes and would ease my main concern.
- Does the 7.9-point accuracy gap at 900 problems hold if OPD's step count, now fixed at 150, grows with the data? If it holds, the accuracy claim would be more convincing to me.
Suggestions:
- Without a within-task test, the title and abstract could present the link to the teacher as an association.
- Showing how many truncated OPD outputs are repetition loops would help readers see what the student learned.
- With the vision encoder frozen, a short note on what is specific to vision-language models would help.
4 comments
W2. "OPD teaches little about finishing." The teacher replacement separates this reading from ours. From the same zero-shot student, OPD reaches 0.984 completion under InternVL3-8B and 0.628 under the paper's teacher. OPD therefore does teach the student to finish when the teacher finishes. Under the paper's teacher the two readings predict the same numbers, which is why the submitted paper could not distinguish them.
On Table 9 (Table 10 in the revised PDF): the 391 problems are those that the teacher finishes when it solves them alone. On these same problems the teacher concludes only 24% of the student's truncated prefixes, against 84% of the student's concluded prefixes. A completion gap on these problems is therefore also expected if the student follows the teacher.
Two further results bear on this point. OPD from the SFT student removes a behaviour the student already had: the termination rate of its rollouts falls from 0.79 in the first five steps to 0.54 in steps 21 to 40. And an InternVL3-2B student, which zero-shot writes the answer line in only 52% of its outputs, writes it in 96% after OPD under InternVL3-8B.
Q2. OPD-term started from the SFT student. It also loses completion. It ends at 0.613, against 0.623 for standard OPD from the SFT student (-1.0 [-2.9, 0.9]) and 0.760 for the SFT model. The loss of completion therefore does not require scoring unterminated rollouts. In these runs only the rollouts that ended carry loss (63% of all rollouts), and the termination rate of the student's rollouts still falls from 0.77 in the first ten steps to 0.56 in steps 21 to 40. Under standard OPD it falls from 0.77 to 0.54. By the criterion you propose, this result does not point to the teacher's feedback on overlong prefixes. The token-level measurements agree. Removing the end-of-turn and answer tokens from the loss leaves completion at 0.607 (global comment, Section 3). Under InternVL3-8B, standard OPD from the same SFT student ends at 0.974. The choice of teacher therefore changes the outcome, and we have not identified which tokens of the solution carry the effect.
Q3. Does the 7.9-point gap hold when OPD's step count grows with the data? Only in part. With 450 steps, three times the protocol's 150, SFT leads by 3.5 accuracy points [1.0, 6.1] and 10.5 completion points [7.9, 13.2]. With 600 steps it leads by 2.4 accuracy points [-0.1, 5.0] and 8.5 completion points [6.0, 11.1]. SFT does not improve with two or four times its own steps. More than half of the 7.9-point accuracy gap was therefore due to the fixed step count. The completion gap narrows from 13 to 8.5 points. The revision gives 7.9 points as a result at fixed update counts, together with the figures at 600 steps.
W3. Narrow evidence for the rule. We added the following.
- A second family, InternVL3. On Geometry3K its teacher completes 97% of its solutions, and OPD from the SFT student ends 3.0 completion points below SFT, against 13.8 for Qwen. On ChartQA at 300 questions OPD leads SFT by 3.95 points [1.35, 6.55] (seed-paired; SFT at the protocol's fixed length, which we did not tune for this family).
- A 4B student. OPD from the base student ends 5.3 completion points below SFT. OPD from the 4B SFT student lowers completion by 3.9 points [1.8, 6.1] in every seed, against 13.8 for the 2B student. We had fixed 5 points in advance as the threshold for a replication, so we report a smaller effect in the same direction.
- The 32B teacher on Geometry3K, and two InternVL3 teachers for the Qwen student (W1 above).
- ScienceQA as a third task (exploratory). The teacher completes 96% of its solutions and OPD still trails SFT by 3.4 points. This contradicts the rule in the submitted abstract, and we removed the rule (W5).
W5 and S1. The positive half of the rule, and the wording. We agree. On ChartQA, OPD's lead over a length-tuned SFT is 1.1 to 1.5 points with intervals that include zero, and on ScienceQA OPD trails SFT although the teacher completes its solutions. The revised abstract no longer states that OPD is effective only with a reliable teacher. It reports the ChartQA and the Geometry3K results as measured, and the teacher replacement. The introduction no longer gives a rule. It says that OPD was competitive with SFT at small budgets on ChartQA, that this advantage is small, and that on ScienceQA OPD trails SFT. On S1: the within-task test now exists, so the abstract states the effect of the teacher as a result and not as an association.
W4. What is new relative to DAgger, Qiu et al. and Sun et al. We agree that the premise is not new. DAgger's guarantee assumes that the expert's action on the learner's states is worth imitating. Qiu et al. and Sun et al. both argue that teacher feedback on erroneous student prefixes is weak, and each proposes a remedy (Qiu et al. compare a teacher with and without privileged visual evidence; Sun et al. propose step-level distillation in text-only settings). Fu et al. (arXiv:2603.25562) list unreliable teacher guidance on student-generated prefixes among the failure modes of sampled-token OPD. Li et al. (arXiv:2604.13016) and Xie et al. (arXiv:2606.22600) find that the accuracy advantage of a teacher that continues a student's prefix shrinks as the prefix grows.
What we add is a set of measurements. (1) A standard setting in which the assumption fails for one specific behaviour, finishing a solution, although the teacher looks adequate by its own evaluation: it completes 78% of its own solutions and is 20 points more accurate than the student. (2) OPD under this teacher removes a behaviour that SFT had already given the student. (3) A probe that needs only inference detects the problem. Li et al. and Xie et al. report the accuracy of the teacher's continuation (Li et al. as a gain over the student's own continuation), which does not distinguish a wrong answer from no answer. Our probe records whether the teacher concludes. (4) Changing the teacher changes the outcome.
On how general the failure is: in our runs the completion gap to SFT appears for 2B and 4B students, under the 8B and the 32B Qwen teachers, and from both starting points (3.9 points for the 4B student from its SFT model, 13.8 for the 2B student). It is absent or small (3 points) under InternVL3-8B. Text-only studies, which the revision now cites, report closely related behaviour: responses that lengthen under OPD until they reach the generation limit (Luo et al., arXiv:2604.08527; Yang et al., arXiv:2609.20511; Wang et al., arXiv:2607.13399), with most rollouts truncated in Luo et al. Luo et al. attribute this to a self-reinforcing loop on repetitive tokens, Yang et al. in part to end-of-sequence tokens that differ between student and teacher, and Wang et al. to a sequence-averaged objective. Zhao et al. (arXiv:2605.16826) report that a mixture weighted toward the forward KL keeps response length stable where reverse-heavy mixtures do not. Our setting differs in three respects. The student and the teacher both put 0.98 of their probability on the same end token where an output stops. The loss is a full-distribution KL at every position and not a sampled-token advantage. Three changes to the objective do not restore completion (the loss on ended rollouts only, removal of the end-of-turn and answer tokens, and the forward KL), whereas replacing the teacher does.
S2. Repetition loops. Among the OPD students' outputs without a parsed answer, 14% end in a verbatim loop. The share is 49% for the zero-shot student and 1% for SFT. By a looser measure, 43% of OPD's unfinished outputs have low 5-gram diversity in their last 1,500 characters (87% for the zero-shot student, 1% for SFT). Where the evaluation recorded the stop reason (300 unfinished outputs of OPD and SFT students), every unfinished output ran to the token budget. OPD therefore removes most of the base student's verbatim loops. What remains is a student that continues to derive without concluding, which is also what its teacher does on these states. The numbers are in Appendix S.
S3. What is specific to vision-language models. We do not claim that the effect is specific to vision-language models, and we have not tested whether it depends on image inputs. Replacing the teacher also replaces the vision encoder through which the teacher sees the image, so the experiment does not isolate the contribution of vision. Text-only studies also report responses that run to the generation limit under OPD (Luo et al., arXiv:2604.08527; Yang et al., arXiv:2609.20511), so the symptom is not confined to image inputs. The causes proposed there (repetition, mismatched end-of-sequence tokens) may differ from ours.
Changes in the revised paper for this review. Section 4.3: Table 3 and the teacher replacement, step scaling, and OPD-term from the SFT student. Abstract and introduction: the rule is removed and the effect of the teacher is stated as a result. Related work: positioning against the cited and the newly cited papers. Appendix S: repetition statistics.
Thank you for the detailed review and for stating which experiments would change your assessment. We ran the within-task test (Q1) and the two other experiments you asked for (Q2, Q3). Intervals are 95% and question-clustered over four seeds unless stated otherwise.
W1 and Q1. Changing the teacher within Geometry3K. Within Geometry3K the student's completion changes with the teacher. It is 0.62 under the paper's teacher and 0.97 under InternVL3-8B, with the student, the data and the recipe fixed. We ran two of the three routes you suggest and added three teachers.
Completion therefore tracks the teacher, but not the teacher's reliability on its own solutions: the 32B teacher and the restriction to finished problems both improve that measure and leave the student unchanged. The continuation probe agrees with all the results. On the student's truncated outputs the two Qwen teachers conclude in 15 to 43% of cases and the two InternVL3 teachers in 66 to 83% (any answer form). The student's completion is 0.62 under the former and 0.97 to 0.98 under the latter (any answer form). In practice, a stronger teacher of the same family does not fix the problem.
One limitation remains. The two teachers under which the student completes its solutions are both InternVL3 models, and the relation is a step between two groups of teachers, not a gradual one. The experiment therefore shows the effect of the teacher and does not identify the property of the teacher that produces it.