acceptodds
Under review as a conference paper at ICLR 2027

OmniOPD: Logit-Free On-Policy Distillation via Speculative Verification

Abstract

On-Policy Distillation (OPD) trains a student model on its own generative trajectories under dense token-level feedback from a stronger teacher, mitigating both the off-policy distribution shift of Supervised Fine-Tuning (SFT) and the sparse credit assignment of outcome-based Reinforcement Learning (RL). However, standard OPD faces two coupled limitations. First, it requires direct access to the teacher's token-level logits, excluding a broad class of capable proprietary models. Second, the token-level logit signal itself is brittle, depending on a narrow overlap of plausible next tokens between teacher and student, and prone to amplifying degenerate patterns such as repetition loops. We introduce OmniOPD, a novel framework that addresses both limitations through a logit-free, chunk-level supervision signal. OmniOPD replaces deterministic logit matching with Monte Carlo rollouts that approximate the teacher's preferences through a continuous semantic similarity metric over multi-token chunks, and concentrates this supervision via a peak-entropy scheduler that audits the student only at its high-uncertainty reasoning forks. Across mathematical reasoning and competitive programming benchmarks, OmniOPD outperforms SFT by up to +45.31% relative on math and exceeds standard white-box OPD by up to +28.64% on mathematical reasoning, demonstrating that effective on-policy teacher supervision can be recovered from text alone without access to teacher logits.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on-policy distillation, large language models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_moPuOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes OmniOPD, a logit-free on-policy distillation method that obtains teacher feedback through sampled textual continuations. It selects high-entropy chunks from student-generated trajectories, compares them with multiple teacher continuations, and uses a similarity score smoothed with a student-policy prior to weight the likelihood of the student’s own tokens. A reference-policy KL penalty anchors unaudited positions. Evaluation covers mathematical reasoning and code generation with Qwen3-1.7B/4B students and open-weight or hosted teachers. The paper reports improvements over sequence-level distillation and includes component ablations, teacher-proxy diagnostics, failure analyses, and an analytical teacher-inference cost model.

Critical strengths

The paper addresses a concrete access constraint. Strong teachers may expose generated text without providing token-level probabilities. OmniOPD supplies local feedback on student-generated prefixes using sampled continuations, without requiring teacher logits or a separately trained discriminator. This is a meaningful practical setting, and the inclusion of hosted-teacher experiments directly addresses the motivating constraint.

Critical weaknesses
  1. Unresolved inconsistencies between the evaluation protocol and reported results undermine the reliability of the experimental evidence.

Lines 277–281 specify 100 problems per coding benchmark and Pass@1 averaged over 16 independent sampling trials. With one binary outcome per problem per trial and no additional averaging, each percentage must equal K/16K/16, for an integer KK, before rounding. However, Table 6 reports values such as 46.78 for Seq-KD and 54.28 for OmniOPD on APPS, neither of which can arise under this protocol through ordinary two-decimal rounding.

Under this interpretation, at least 21 of the 24 individual benchmark entries are incompatible with the stated setup, affecting all three methods. Related discrepancies arise in the mathematical results under standard benchmark-size assumptions. With 30 AIME problems and 16 binary trials per problem, percentages must equal 5K/245K/24; the OmniOPD AIME-2024 score of 68.67 in Table 1 cannot be obtained through ordinary rounding or truncation. Likewise, assuming the commonly used 40-problem AMC23 split, the scores of 90.44 in Table 1 and 92.69 in Table 2 require clarification.

These are substantive reproducibility concerns, not merely formatting issues. Additional averaging, different effective sample counts, or non-binary scoring could explain some discrepancies, but the manuscript does not document such procedures sufficiently to reconstruct the results. If partial test-case credit was used for coding, it must be distinguished from binary program-level Pass@1. Until the discrepancies are reconciled with a reproducible aggregation procedure and supporting records, I cannot regard the affected results as reliable evidence for the paper’s conclusions.

  1. The claimed semantic advantage and the conditions under which the surrogate improves reasoning are insufficiently established.

The default edit-distance metric and the ROUGE alternative primarily measure surface overlap. Theorem B.4 correctly transfers invariances already possessed by the similarity function; it does not establish that the implemented metrics tolerate meaning-preserving paraphrases while distinguishing small but consequential reasoning errors. For an identity-separating edit similarity, the induced equivalence classes can be trivial. The existing correlation analyses do not directly establish the semantic robustness claimed in the motivation. The optimization evidence also leaves an important gap. Appendix E.7 reports increasing compatibility scores and decreasing training loss alongside reasoning collapse without KL anchoring. Table 11 shows severe sensitivity to chunk length: configurations with (M,N,C)=(10,10,25)(M,N,C)=(10,10,25)and (5,10,50)(5,10,50) have equal nominal audited-token and teacher-output-token budgets but substantially different outcomes. This does not invalidate the full method, and the authors appropriately disclose these failures. It does, however, leave the operating conditions of the central surrogate insufficiently understood. The conditional variance and drift bounds do not establish when these updates improve reasoning rather than reinforce superficial agreement or existing student preferences.

Questions for authors
  1. How were the Table 6 APPS scores of 46.78 and 54.28 obtained from 100 problems and 16 sampling trials? Was there additional sampling, averaging, or filtering?

  2. Does coding evaluation use binary program-level correctness or partial test-case credit? If partial credit is used, what are the corresponding binary Pass@1 results?

  3. Which AIME and AMC23 subsets and sample counts were used? How were the AIME score of 68.67 and AMC23 scores of 90.44 and 92.69 computed?

  4. Can you provide the per-problem evaluation records, scorer configuration, and aggregation script needed to reproduce these entries?

  5. What direct evidence shows that the implemented similarity metrics tolerate meaning-preserving changes while distinguishing consequential reasoning errors?

  6. How does the method compare with constant or shuffled teacher weights, keeping the update structure, KL regularization, and sampling budget unchanged?

  7. Why does (M,N,C)=(10,10,25)(M,N,C)=(10,10,25) fail so severely relative to (5,10,50)(5,10,50), despite equal nominal audited-token and teacher-output-token budgets? Does it exhibit the same collapse as the no-KL variant?

Reviewer_ayn3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OmniOPD, a logit-free framework for on-policy distillation from both open-weight and proprietary teachers. The student generates its own reasoning trajectories, and high-entropy regions are selected for chunk-level supervision. The teacher samples multiple continuations from the corresponding student prefixes, and their textual similarity to the student chunk provides a compatibility weight for training. The method additionally uses student-prior smoothing and a reference-policy KL penalty on unaudited tokens. Experiments cover mathematical reasoning and competitive programming with two student sizes and several teachers. OmniOPD improves over sequence-level distillation across the reported settings and outperforms standard white-box OPD in some mathematical reasoning configurations. The paper also provides component ablations and analyses of teacher-prefix bias, signal quality, training collapse, and inference cost.

Critical strengths
  1. A useful approach to local on-policy supervision from black-box teachers. The method obtains feedback directly from sampled teacher continuations without requiring teacher logits or an additional learned discriminator. Although several components are established techniques, their use in this concrete feedback mechanism offers a practical contribution.

  2. Meaningful gains supported by a substantial experimental analysis. The improvements are sizeable in several mathematical reasoning settings. For example, with a Qwen3-4B student and Qwen3-32B teacher, OmniOPD improves average accuracy from 63.80 to 69.08 over Seq-KD and from 64.16 to 69.08 over standard OPD. With Claude-4.5-Haiku, it improves over Seq-KD from 67.52 to 74.92. I also appreciate the analyses of erroneous student prefixes and the failure mode without KL anchoring, which help clarify the method’s limitations.

Critical weaknesses
  1. The contribution of the compatibility signal could be clarified further. The paper provides useful evidence that textual similarity captures teacher preferences, although its relationship with reasoning correctness remains limited. Controls using the same audit selection and KL penalty, but constant or student-prior-only weights, would help isolate the benefit of teacher feedback.

  2. The practical advantage varies across settings. The mathematical results are encouraging, but improvements over GRPO and standard OPD are not consistent across student sizes and tasks. Since the cost estimates depend on prefix reuse, comparisons under matched teacher-inference budgets would further strengthen the practical claims.

Questions for authors
  1. When sampled teacher continuations have zero compatibility with a student chunk, what is the learning benefit of retaining a positive self-imitation update through the student prior? Does this behavior help on incorrect chunks, or can it reinforce confident mistakes?

  2. For the proprietary-teacher experiments, what were the actual teacher input/output volumes and prefix-reuse conditions for OmniOPD and Seq-KD? Would the accuracy advantage remain under matched teacher-inference budgets?

  3. How does the entropy-based scheduler differ from the high-entropy token selection studied in Beyond the 80/20 Rule and TIP? In particular, how does the method handle confident but incorrect reasoning regions that entropy-only selection may miss?

Reviewer_iHJiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OmniOPD, a logit-free on-policy distillation (OPD) method for black-box (text-output-only) teacher models. Concretely, it first samples the student's reasoning trajectories and uses entropy to select local chunks with higher uncertainty; it then has the teacher model generate a short continuation conditioned on the corresponding student-generated prefix, and computes a chunk-level compatibility weight via text similarity; finally, this weight is combined with the student's probability prior through Bayesian smoothing and used for a weighted-likelihood update, together with KL regularization to suppress training drift. The whole scheme requires no access to teacher logits and no additional discriminator training. The authors test student models of different sizes on math and coding tasks, covering both open-weight and closed-source API teachers, and provide ablation, hyperparameter sensitivity, and cost analyses. Experiments show that the method performs well overall, outperforming Seq-KD and white-box OPD in some settings.

Critical strengths
  1. The framing is very practical, naturally extending on-policy distillation to the black-box API teacher setting. Without access to teacher logits, having the teacher continue a short segment along the student's own reasoning prefix to construct local feedback both places the training signal on the state distribution actually sampled by the student and avoids the overhead of training an additional discriminator, allowing direct integration with closed-source large models. The overall training pipeline is designed cleanly and practically.

  2. The experiments have broad coverage, and the gains in the core settings are solid. The paper validates on two mainstream reasoning task types—math and code—and on student models of different parameter sizes. For example, in Table 1, with a 4B student paired with Claude-4.5-haiku and Gemini-2.5-flash, the math average scores reach 74.92 and 75.67, respectively, 7.40 and 2.16 points higher than the corresponding Seq-KD; under some open-weight teacher configurations it also beats white-box OPD. In addition, the appendix provides fairly detailed diagnostic analyses of erroneous prefixes, low-overlap feedback, and training collapse.

Critical weaknesses
  1. It remains unclear how much independent gain the teacher's textual feedback actually contributes. Equation (5) mixes the teacher similarity with the student's own confidence prior, while the surrogate update in Equation (7) itself carries a distribution-sharpening effect. As a simplified example: fixing the prefix and ignoring the prior, chunk selection, and KL, the expected objective under exact match is in fact ∑ypθ(y)q(y)\sum_y p_\theta(y)q(y), whose optimum is naturally biased toward the mode of the teacher distribution. However, the current experiments lack several key controls with all other conditions held constant: prior-only, constant-weight, and shuffled-feedback (perturbing the correspondence between teacher feedback and student chunks). The correlation analysis in Appendix E.1 also does not rule out interference from the student's own prior and scale differences. It is therefore difficult to judge whether the current gains come from substantive guidance of the teacher continuation or from self-reinforcement of the student's high-confidence trajectories.

  2. Training stability depends heavily on KL regularization, and the fairness and attribution of some baseline comparisons are not sufficiently rigorous. In Table 2, removing KL causes the average score to plummet from 69.08 to 8.28, whereas removing entropy-driven selection or Bayesian smoothing drops it by only 0.63 and 0.45 points. This indicates that the current scheme is mainly backed by a strong KL constraint; however, the paper does not state whether the white-box OPD baseline also uses reference regularization of comparable strength, and module gains of less than 1 point lack multi-seed results to rule out training fluctuations. In addition, with C=25C=25 in Table 11, the average score drops to 24.48, which the authors attribute to insufficient semantic information in short chunks; however, changing the chunk length CC simultaneously changes the total number of audited tokens and the effective ratio of the two losses, and this explanation is of limited persuasiveness before these confounders are controlled.

Questions for authors
  1. On the independent role of teacher feedback: Holding the chunk selection strategy, KL strength, optimizer, and training budget fixed, how large an advantage does real teacher feedback provide over prior-only, constant-weight, and shuffled-feedback (randomly perturbing the chunk–feedback pairing) baselines? We suggest adding these control groups under one representative configuration, and reporting the mean and standard deviation over multiple training seeds for the small-magnitude ablation items in Table 2.

  2. On the regularization alignment of the white-box OPD baseline: What reference regularization or stabilization techniques does the white-box OPD in the experiments specifically use, and how are the hyperparameters selected? If the white-box OPD is given reference regularization of a comparable level, or if the text compatibility weight is replaced directly by the true teacher likelihood within the same chunk-update framework as in this paper, would the currently reported performance advantage still hold?

  3. On the cause of the performance collapse at C=25C=25: How are the chunk loss and the KL term in Equation (7) normalized in the implementation? When CC changes, do the effective gradient ratio between the two and the total number of audited tokens change accordingly? If the total number of audited tokens is controlled or the effective weights of the two losses are realigned, would such severe degradation still occur at C=25C=25?

  4. On the computation details of the evaluation scores in Table 6: Section 3.1 describes the coding evaluation as 100 problems per category with 16 samples to compute the average Pass@1. If one directly averages these 100×16=1600100 \times 16 = 1600 binary outcomes, the smallest step in percentage should be 0.0625%, which cannot be rounded to obtain 54.28% in Table 6 (868/1600=54.25%868/1600=54.25\%, 869/1600=54.3125%869/1600=54.3125\%). Please clarify the actual evaluation denominator, the aggregation scheme, and whether any additional subset weighting or problem filtering is involved.

Reviewer_WjEGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes OmniOPD, a method for on-policy distillation without access to the teacher’s token probabilities. The student first generates an answer. OmniOPD selects short chunks where the student is uncertain and compares them with several teacher continuations. Text similarity scores guide student training. A smoothing step reduces noise, while a KL penalty limits unwanted changes outside the selected chunks. Experiments on math and programming show improvements over standard fine-tuning. The method also outperforms white-box OPD in some settings and works with closed teachers such as Claude and Gemini.

Critical strengths
  • Practical motivation. Many closed models only provide text outputs. OmniOPD allows students to learn from these models without requiring teacher probabilities or an additional trained discriminator.

  • Good experimental coverage. The paper evaluates two student sizes, several teachers, and both math and programming tasks. The improvements over standard fine-tuning are consistent across the reported settings.

  • Useful analysis. The paper studies its main components, different chunk sizes, and errors caused by student prefixes. The analysis of training collapse without KL regularization helps explain why this component is needed.

Critical weaknesses
  • Text similarity does not always reflect reasoning quality. Edit distance and ROUGE mainly measure surface similarity. A small change in a number or sign can make a reasoning step wrong while keeping its similarity score high.

  • Strong sensitivity to chunk size. Reducing chunk length from 50 to 25 tokens lowers average math accuracy from 69.08% to 24.48%. This large drop raises concerns about how reliably the method works under different settings.

  • Some improvements need stronger evidence. OmniOPD does not always outperform white-box OPD or GRPO. The gains from entropy selection and smoothing are small, and the paper does not report variation across independent training runs.

Questions for authors
  • Why does the 25-token setting perform so poorly? Does this problem remain when the total number of checked tokens is kept the same across different chunk lengths?

  • Can the authors repeat training with different random seeds? This would help show whether the small gains from entropy selection and smoothing are reliable.

  • What is the actual training cost compared with standard fine-tuning and OPD? The cost estimate depends on reusing teacher prefixes. Measured training time and total teacher input/output tokens would make the comparison clearer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.