acceptodds
Under review as a conference paper at ICLR 2027

OVD: On-policy Verbal Distillation

Abstract

Knowledge distillation transfers reasoning capabilities from large teachers to efficient students. However, token-level on-policy distillation (OPD) constrains student exploration and requires teacher token probabilities, precluding distillation from black-box teachers that provide only text outputs. We introduce On-policy Verbal Distillation (OVD), a framework that uses verbal scores from black-box teachers to rank student-generated sub-trajectories, retaining high-scoring ones and replacing low-scoring ones with teacher-generated continuations. We analyze when ranking induced by verbal scores can guide distribution approximation: under a density-ratio calibration condition on acceptance probabilities and bounded teacher-replacement error, we bound the approximation error between the resulting mixed trajectory distribution and a teacher-preferred target. On Web Q&A, OVD achieves 41.09% average EM with teacher feedback at inference, exceeding the strongest evaluated baseline by 5.89 percentage points. On AMC23, OVD-FR improves accuracy over RLVR by 10.0 percentage points (52.5% to 62.5%) after 600 training steps on 128 problems. Further experiments suggest that retaining student-generated prefixes helps preserve exploration and mitigate trajectory-level entropy collapse. OVD also improves training efficiency: resampling selected suffixes rather than entire responses reduces mean per-step training time by 10.2% in the 128-problem setting. Code is available at https://anonymous.4open.science/r/OVD-review-3BDA/OVD_math/README.md.

Primary area:
reinforcement learning
Keywords:
On-policy distillation, Verbal feedback, Knowledge distillation, Large language models, Reinforcement learning, Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wxhiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents On-policy Verbal Distillation (OVD), a framework designed for black-box LLM teachers that provide only text outputs rather than full-vocabulary token distributions. OVD operates by having the teacher score student-generated sub-trajectories using discrete verbal score tokens (e.g., on a 0–9 scale). High-scoring sub-trajectories are retained, while low-scoring ones are rejected and replaced with teacher continuations, after which the student resumes rollout.

Critical strengths

Black-Box Logit-Free Formulation: OVD circumvents the massive memory and access bottlenecks of token-level reverse-KL distillation by relying strictly on discrete verbal feedback, enabling distillation from black-box teacher interfaces.

Prefix Preservation and Exploration: By replacing only low-scoring suffixes or sub-trajectories rather than entire responses, OVD preserves student-initiated reasoning paths, demonstrating empirically improved solution coverage (Pass@16) and reduced rollout compute.

Critical weaknesses

Weak Justification of Density-Ratio Assumptions: The distribution-matching guarantees require verbal acceptance probabilities to approximate density ratios (aτ∝pT/πSa_\tau \propto p_T/\pi_S). In practice, verbal scores from an LLM critic are subjective heuristics rather than calibrated importance weights, making the theoretical total variation bounds more aspirational than strictly operational.

Benchmark Selection and Stability: Acknowledging post-hoc selection of math benchmarks in Appendix A.5 weakens confidence in general mathematical reasoning gains. Furthermore, Table 6 shows that random-suffix resampling (OVD-RS) occasionally matches or exceeds high-entropy suffix selection (OVD-HES), making the value of the entropy-split heuristic somewhat ambiguous.

Questions for authors

Post-Hoc Selection: Could you provide the aggregate results across the entire, uncurated pool of 23 math benchmarks for both OVD-HES and OVD-FR to confirm that gains are consistent without subset filtering?

How sensitive is the framework if the verbal score prompt or scale is modified, and does the observed correlation with the true density ratio hold across non-math domains?

What is the overall wall-clock training overhead when accounting for teacher verbal-scoring inferences compared to standard RLVR or offline SFT distillation?

Reviewer_sqYpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes On-policy Verbal Distillation (OVD). OVD trains a small student model with a teacher model that gives only text. Thus, OVD does not need the teacher's token probabilities, unlike token-level on-policy distillation (OPD). OVD trains a small model (the student) with help from a larger model (the teacher). The teacher scores each part of the student's work. Then the teacher rewrites only the weak parts. The teacher scores each part of the student's work from 0 to 9. Then the teacher rewrites only the weak parts. GRPO with an outcome reward then trains the student on the resulting mixed trajectories of student and teacher parts. Experiments cover Web Q&A with a search tool (3B students, a 7B teacher, eight benchmarks) and tool-free math (a 1.5B student, a 7B teacher). Without a test-time teacher, OVD reaches 37.51% average EM in Web Q&A, against 35.20% for the best baseline. In math, OVD-HES has the highest best-checkpoint average over nine benchmarks (Table 3).

Critical strengths
  1. OVD improves agentic search on every benchmark. OVD needs only a verbal score and replacement text from the teacher. The method therefore suits teachers that expose no token probabilities. Without a teacher at test time, OVD (τ_tr = 5, τ_te = 0) reaches 37.51% average EM, against 29.69% for Search-R1 and 31.07% for ZeroSearch (Table 1). OVD is higher than both baselines on every benchmark.

  2. A direct test of the central mechanism and a clearly scoped theory. Figure 2 isolates the ranking signal of the verbal scores. On 4,096 fixed student prefixes, continuations from prefixes with a score of at least 8 reach 61.22% EM. A random choice of the same number of prefixes per problem reaches 49.06%. The scores therefore reliably separate good and weak prefixes within a problem. The authors also transparently explain what the approximation bounds (Appendix A.2) do not establish, for example, convergence of the trained student (Appendix A.4). The appendix reports unfavorable results openly, including the random-suffix control and the weaker 14B teacher. This is welcome alongside the reported stronger results.

Critical weaknesses
  1. The math experiments do not support the math claims in the abstract and in Findings 5. The authors chose eight of the nine benchmarks post hoc from 23, to maximize the gain of OVD-HES over RLVR (Appendix A.5). The main text then compares the best of four checkpoints, selected on the same test benchmarks.

Even under these choices, the gains are small. With 128 problems, the best OVD-HES average exceeds RLVR by 1.75 percentage points (57.40% against 55.65%). Gaokao-II contributes 1.59 of these points. Because Gaokao-II has only 14 problems, this contribution equals two additional correct answers. Averaged over the four checkpoints, OVD-HES is below RLVR (54.72% against 54.95%). The paper reports no repeated runs or confidence intervals to show whether this gain is reliable.

The AMC23 gain of 10.0 points in the abstract corresponds to four of 40 problems. At the same checkpoint, the nine-benchmark average of OVD-FR is below RLVR (54.82% against 55.15%). The evidence for Findings 5 is a five-benchmark subset in Figure 6. AIME24 alone contributes 2.66 of the 3.24-point gain (13 against 9 solved problems, out of 30). Over all ten benchmarks in Table 7, OVD-HES is only 0.7 points above OVD-FR at training step 800. At the same step, the random-suffix variant has a higher average than OVD-HES (56.34% in Table 6 against 53.77% in Table 3). The paper therefore does not show that the high-entropy split itself helps.

Because the math setting is the only tool-free test of OVD, the general claims currently depend on one agentic task. Results on all 23 benchmarks, preferably with confidence intervals, would resolve this concern.

  1. The threshold results for Web Q&A in the appendix contradict the main text and may contradict the central claim of the paper. Section 3.4 says that τ_te = 5 gives better results than τ_te = 10. Tables 10 and 12 show the opposite (41.94% against 39.04%, and 43.96% against 42.79%). At τ_te = 10, the teacher rewrites every search query at test time. The best test-time results therefore reflect teacher-written queries more than the trained student.

Section 3.4 also says that full replacement during training (τ_tr = 10) does not necessarily give better results. However, Table 12 (τ_tr = 10) reports 40.97% without a test-time teacher, against 37.51% for τ_tr = 5 in Table 1. If the two tables use the same teacher, full replacement is better at every τ_te. This result would contradict the central argument that partial replacement preserves useful student exploration. The paper should name the teacher and thresholds behind each table, and reconcile Section 3.4 with the appendix.

  1. GRPO is written as "Generalized Reward Preference Optimization" instead Group Relative Policy Optimization
Questions for authors

Q1: OPD baselines. Which teacher model gave the token probabilities when you trained the two OPD baselines in Table 1, and do these runs use the outcome reward? Both baselines are below RAG (14.14% and 11.51% against 22.63%). A fair test would give OPD the same outcome reward as OVD, for example GRPO with a reverse-KL term to the teacher.

Q2: Teacher role. In Appendix A.10, one model (Simulation_LLM_google_7B_V2) simulates search and also scores the Web Q&A queries. This model is fine-tuned to generate search results, not to write queries. Why is this model a good judge of query quality? Also, the 14B teacher gives lower EM than the 7B teacher at every τ_te (Table 12). Is the gain from teacher knowledge, or from the extra query rewrites during training? Please name the teacher for each of Tables 1, 10, 11 and 12 (at τ_tr = 10 and τ_te = 5, Table 11 gives 22.82% and Table 12 gives 42.79%).

Q3: Math model selection. Section 3.5 and the abstract use the best checkpoint by test accuracy (Weakness 1). Please report results at a fixed training step, or at a checkpoint that a validation set selects. Please also give the scores on all 23 benchmarks, with confidence intervals where possible.

Reviewer_HdZSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Online Verbal Distillation (OVD), a teacher-guided training framework in which a teacher scores student-generated reasoning segments and replaces low-quality segments before the student continues generation. The resulting trajectories are used for reward-based training. The method is evaluated on Web QA and mathematics.

Critical strengths
  1. The paper addresses a practical setting where teacher logits are unavailable or expensive to store.

  2. Online, segment-level correction is a potentially useful alternative to imitating complete teacher trajectories.

  3. Experiments provide evidence that teacher-guided training can improve standalone student performance.

Critical weaknesses
  1. The main evaluation is not well aligned with the paper’s central claim. The mathematics experiments provide a direct training-method comparison against RLVR, supporting the benefit of teacher-guided training. However, the primary Web QA evaluation mainly compares against large-model, RAG, and search-based systems, rather than matched distillation baselines. These comparisons demonstrate system-level performance but do not isolate the contribution of OVD’s training mechanism.

  2. The efficiency trade-off is insufficiently evaluated. Avoiding full-vocabulary teacher logits may reduce memory overhead, but teacher scoring and regeneration introduce additional computation and sequential interaction. The reported efficiency experiments do not establish an end-to-end memory–time comparison against token-level OPD.

  3. The presentation needs improvement. Several figures are crowded, and the connections among the theoretical analysis, training procedure, and evaluation settings are not sufficiently clear.

Questions for authors
  1. Why does the main Web QA evaluation lack matched distillation baselines? Can the authors compare OVD with token-level distillation under the same student, teacher, and training data, or clarify why this comparison is infeasible?

  2. How's the trade-off between memory efficiency and computational efficiency with such teacher guidance procedure?

Comments on the paper

1 comment
Reviewer_sqYpOpenReview ↗

No rebuttal received

No rebuttal received. Score remains the same.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.