acceptodds
Desk rejected at ICLR 2027

Learning from a Thoughtful Teacher: Adaptive On-Policy Self-Distillation for Mathematical Reasoning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

On-policy self-distillation (OPSD) trains a question-only student with token-level feedback from a teacher given training-only privileged information (PI). OPSD therefore provides dense, on-policy supervision, and is free of a larger external teacher, but its effectiveness rests on how PI is designed and utilized. Our preliminary diagnostics suggest a significant gap between teacher utility and student learnability, where a small fraction of high-disagreement tokens dominate the distillation signal, and short teacher continuations at these positions further expose more explicit PI leakage than transferable correction cues, indicating a strong intent on injecting PI-conditioned shortcuts. We propose Adaptive On-Policy Self-Distillation (AOPSD), which adapts what information the teacher receives and how strongly its feedback influences learning. AOPSD encodes each solution as a reasoning DAG, orders problems by the student’s evolving capability, and reveals only the affordable subgraph and its next frontier as PI. For high-disagreement tokens, AOPSD utilizes short teacher continuations as probes to encourage useful guidance while mitigating PI-conditioned shortcuts among teacher supervisions. On HMMT25, AIME24, AIME25, and BRUMo25, AOPSD achieves 72.5% Pass@8, which is 6.7 percentage points above OPSD and 4.2 above the strongest competing baseline while reducing 15 percentage points of training time at lower cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language model, on policy self-distillation

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_xndFOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Adaptive On-Policy Self-Distillation (AOPSD), which consists of three main components: (1) representing the reasoning steps and their dependencies in reference solutions as Reasoning DAGs, and adaptively disclosing Privileged Information (PI) based on the Student's current progress; (2) dynamically adjusting the curriculum according to the Student's reasoning progress; and (3) using Teacher continuations and Student NLL to compute token-level weights, thereby adjusting the strength of supervision for high-disagreement tokens. The authors evaluate the effectiveness of these components through ablation studies, hyperparameter analysis, and semantic audits.

Critical strengths
  1. The paper has a clear motivation, showing that PI can significantly improve the Teacher's conditional problem-solving ability.
  2. The design of Adaptive PI Disclosure is intuitive and easy to understand.
  3. Continuation-based Selective Distillation provides a lightweight way to adjust the strength of supervision.
Critical weaknesses
  1. I'm not sure how well the DAG alignment works when the Student takes a different but valid reasoning path.
  2. I'm a bit concerned about using the Student to judge its own progress, as mistakes here could affect both PI Disclosure and Curriculum Learning.
Questions for authors

The method is interesting and the results are promising, but I still have some questions about the key design choices.

  1. For problems with multiple valid solution paths, how does the proposed DAG alignment handle cases where the Student follows a reasoning path different from the reference DAG? Could this lead to an underestimation of the Student's reasoning progress?
  2. Reliability of progress estimation. Have the authors compared Student-based judgments with those of an independent LLM judge or human annotators? If some checkpoints are incorrectly marked as reached, how much would this affect the performance of Adaptive PI Disclosure and Curriculum Learning?
  3. The authors assume that Teacher continuations with moderate NLL are more likely to provide useful guidance, while those with high NLL may indicate PI leakage. Could the authors further explain the rationale behind this assumption? For example, is NLL-based weighting more effective than simply adjusting token weights based on Teacher–Student disagreement?
1 comment

We thank the reviewer for the positive assessment and the pointed questions.

Q1 – Different valid paths. A different path is still credited as long as it reaches the same key results; a path that shares none can be under-credited, which is the safe direction. (1) Alignment compares results, not steps. DAG nodes are intermediate results (e.g., "a = 2, b ≤ 2, c ≤ 1"), and a node is credited whenever the response establishes that result, however it was derived. (2) If a route shares no results with the reference, progress is underestimated. The effect is mild: the teacher sees only the first nodes, i.e., less PI, which is the safer direction because the risk we address is that a teacher given excessive PI may inadvertently leak it through its feedback; and the problem is treated as harder, which only changes when it is visited within an epoch (every problem is still visited every epoch, and competence is averaged over attempts). Supporting several reference routes per problem is a natural extension.

Q2 – Reliability of self-judging. Same as our reply to Reviewer HHn3: errors are bounded by design (a node counts only if its prerequisites are reached; a single false positive moves the disclosure by at most one step; the answer is stripped from the DAG; continuation weights suppress PI-dependent feedback anyway), the judgment is a comparatively easy recognition task, and on 299 of 300 frozen rollouts that we re-judged with an independent stronger model, the student's judgments agree with it on 76.4% of node decisions, with the residual disagreement erring toward slightly earlier disclosure (+0.130 progress on the 0–1 scale).

Q3 – Why NLL rather than disagreement alone? Disagreement says how much PI changes the teacher's preference at a token, not whether the student can learn from it. A correction grounded in the problem and a jump that only PI explains can show the same gap; they differ in what the teacher says next. Because the teacher is the student plus PI, the student's NLL on the teacher's continuation measures how much of it is reachable without PI (up to the teacher's own small NLL, it is exactly the log-likelihood ratio, i.e., the information PI adds). Predictable → nothing new; extremely surprising → only PI explains it; in between → learnable. A weight based on disagreement alone cannot tell these apart; NLL can: among events that all pass the same |g| ≥ 2 threshold, the share of PI leakage rises from 0.09% to 91% across NLL bins (NLL-blind audit, Table 8).

Ethics flag. We are not aware of a specific concern: the study uses only public datasets and no human subjects, and LLM use is disclosed in the paper. We would be glad to clarify if the reviewer can point us to one.

Reviewer_HHn3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper uncovers a critical flaw in On-Policy Self-Distillation (OPSD) for mathematical reasoning: providing a teacher with Privileged Information (PI) (like a verified solution) yields a high-utility conditional solver, but this advantage fails to transfer to a question-only student. The authors find that standard OPSD causes the teacher to introduce PI-conditioned shortcuts or leakage. To bridge this "utility-transfer gap" the authors propose Adaptive On-Policy Self-Distillation (AOPSD), which regulates what the teacher sees and how strongly the student learns from it via three components:

  1. Reasoning DAGs as PI: Solutions are structured as directed acyclic graphs of discrete mathematical checkpoints, with the terminal answer layer stripped out to avoid direct answer leakage.
  2. Adaptive PI Disclosure & Curriculum: The teacher is only shown the sub-graph the student has successfully reached in its rollout, plus its immediate frontier. Simultaneously, the problem sequence is organized dynamically as an easy-to-hard curriculum based on the student's moving competence.
  3. Continuation-Based Selective Distillation: For high-disagreement tokens, the teacher generates a short continuation probe. The question-only student scores this probe using negative log-likelihood (NLL), and a band-pass filtering weight suppresses both highly predictable (redundant) and highly surprising (PI leakage) feedback. Evaluating on competition math benchmarks (HMMT25, AIME24, AIME25, BRUMo25) with a Qwen3-4B base model, AOPSD achieves a final macro-average Pass@8 of 72.5% (outperforming OPSD by 6.7 percentage points) while reducing training time by 15.6% due to a lightweight sampled-token policy gradient objective.
Critical strengths
  • Deconstruction of the Utility-Transfer Gap: The paper presents an insightful diagnostic observation: maximizing a teacher's information context can actively hurt student training if the teacher passes down unlearnable, context-dependent shortcuts rather than transferable strategies.
  • Prerequisite-Closed Disclosure: Representing solutions as a reasoning DAG and dynamically limiting teacher visibility to the student's current "frontier" is a clever way to keep feedback within the student's zone of proximal development.
  • Effective Use of Continuation Surprisal: Utilizing short text continuations to audit teacher intent via student NLL is a powerful heuristic. The empirical validation (using Claude Opus 5 to perform an NLL-blind semantic label of 15,043 events) convincingly demonstrates that extremely high student surprise correlates tightly with unlearnable PI leakage.
  • Substantial Computational Efficiency: By changing from full-distribution KL matching to a sampled-token policy gradient target, the method reduces supervision storage overhead by up to 94.9% and decreases overall wall-clock training time by 10% to 15.6% compared to Top-16/Top-32 KL metrics.
Critical weaknesses
  • Heavy Offline Graph Construction Overhead: While the training phase is highly optimized, the pipeline requires an expensive offline generation step using another LLM (glm5.2-fp8) acting as both a producer and an iterative judge (up to 5 iterations) to yield dense and sparse reasoning DAGs for every problem. The cost and scalability of this preprocessing layer are not deeply explored.
  • Dependency on Automated Alignment: During training, the student model itself acts as the judge to align its own rollout text against the reference DAG to determine what subgraph to unlock . If the student makes false positives or improperly judges its own flawed step as "sufficiently established" , it will prematurely expose downstream components of the DAG, reinstating the shortcut risk.
  • Diminishing Margins on Smaller Models: While the results on Qwen3-4B are very strong , the extensions to Qwen3.5-2B (Appendix H) show tighter margins—the final average improvement over the baseline is only 1.7 pp, and the base model occasionally matches or beats the distilled student on individual benchmarks (e.g., HMMT25)
Questions for authors
  1. DAG Judge Robustness: Since the student model acts as an inference-only judge to check off achieved steps in the DAG during training, how often does it make alignment errors? Did you evaluate the alignment accuracy compared to a human or a stronger model like GPT-4?
  2. DAG construction is also time-consuming. AOPSD shifts the complexity from live training compute to static offline data curation. Do you know the exact time cost in the trade-off?
  3. Generalization Beyond Math: The reasoning DAG architecture fits mathematical chains and multi-stage branch/merge problems naturally. How easily can this structure be adapted to code generation or multi-step logic puzzles where checkpoints are less explicitly formulaic?
1 comment

We thank the reviewer for the careful assessment.

W1/Q2 – Offline DAG cost. DAGs are built once per problem, offline and in parallel, and reused across all runs, backbones and ablations. To quantify the cost we re-instrumented the saved dense-to-sparse construction pipeline and re-ran it on 100 reference-length-stratified training problems with an independent API model (at most five producer–judge rounds per stage, a 20,000-token completion cap, failed attempts included): 98/100 problems passed, consuming 535 API calls and 3.50M tokens (2.09M input / 1.41M output) in 26.3 minutes of wall-clock time at concurrency 5. Extrapolating with the stratification weights, building the full problem set costs on the order of 3.9K API calls and 25.4M tokens (95% bootstrap CI 23.1–28.1M). These are supplementary reconstruction measurements rather than a recovered log of the original run, and we do not convert API latency into GPU-hours. Nothing is added at training or inference time. We will report this in the appendix.

W2/Q1 – Judge errors. Errors are bounded by design: a node counts only if all its prerequisites are reached, so a single false positive advances the disclosure by at most one step; the final answer is stripped from the DAG offline; and the continuation weights suppress PI-dependent feedback regardless of what was disclosed. The judgment is also comparatively easy: checking whether a stated intermediate result appears in a response is recognition, not problem solving. We measured it: we re-judged 300 frozen question-only rollouts (100 stratified training problems × 3) with an independent stronger model under the identical prompt, parser and prerequisite-closure rules. On the 300 rollouts where both judges returned valid output, the student's judgments agree with the stronger model on 76.4% of node decisions (95% problem-cluster bootstrap CI 72.3–80.0%). Agreement is thus substantial but not perfect, and the residual gap is one-sided: the student credits 0.130 more progress on average on the 0–1 scale, i.e. it errs toward disclosing slightly earlier. Since agreement with a stronger model is not ground-truth accuracy, it is the design bounds above that keep this level of judge noise tolerable. We will add the full audit to the appendix.

W3 – Smaller models. Self-distillation is particularly demanding for small models, because the teacher is the same small model. In this regime, existing methods degrade: on Qwen3.5-2B, OP²SD and DASH fall below the base model (35.8 and 27.5 vs 36.7). AOPSD is the only one that improves on it (38.3); at Epoch 3 it is at or above the base on every benchmark, and matches or exceeds both baselines on all four.

Q3 – Beyond math. Very promising. We use math because its established benchmarks and verifiable answers make comparison clean; nothing in the method is math-specific. The DAG only needs a solution that can be broken into steps, and most tasks can be cast this way, directly or indirectly (code: milestones or unit tests; puzzles: deduction steps; proofs: lemmas; agent tasks: sub-goals). The continuation weighting needs no steps at all: it acts at teacher–student disagreement points and uses only the two models' likelihoods, so it is independent of the task. We will add this discussion.

Reviewer_yTuPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies a gap between teacher utility and student learnability, showing that providing more information to a teacher can improve its performance while reducing knowledge transfer to the student.

AOPSD addresses this by representing solutions as reasoning DAGs, revealing checkpoints according to student progress, and weighting high disagreement tokens using short teacher continuations. Experiments on math reasoning benchmarks show improvements over many existing distillation methods.

Critical strengths

S1) By revealing only the checkpoints reached by the student and the next reasoning frontier, the method provides a structured way to control privileged information. This differs from existing approaches that use fixed teacher contexts and allows supervision to adapt to the student's demonstrated progress.

Critical weaknesses

W1) It is unclear how much of the gain comes from the method rather than giving the teacher less information. Concurrent work by Tian et al. (2026), What Should a Self-Teacher See?, finds that reduced privileged contexts can outperform full solutions.

W2) Each benchmark contains 30 problems and the paper does not report multiple training seeds or confidence intervals. This makes it difficult to assess the robustness of the reported improvements, as several seem to correspond to only one or two additional solved problems.

Questions for authors

Q1) How does AOPSD compare to a fixed reduced context like the method independent framing context from Tian et al (2026), trained under the same setup? See W1.

Q2) Am I right that AOPSD uses the sampled token objective (Eq. 8) while OPSD, OP2SD and DASH use top-K forward KL? If so, what does OPSD get when trained with the Eq. 8 objective? If it comes close to AOPSD the gain may come from the objective rather than the method.

Q3) Could you rerun AOPSD, OPSD and GRPO with 3 seeds and report mean and std? See W2.

I am happy to reconsider my score if these issues are addressed.

3 comments

We thank the reviewer for the precise attribution questions and for pointing us to Tian et al. We provide both requested controls and two additional training seeds. Both controls use sampled-token PG with w=1, disabling continuation weighting.

W1/Q1 — Fixed reduced context. Under the same training setup, Tian et al.'s fixed method-independent framing context (L3), trained with PG, reaches 66.7%, versus 72.5% for AOPSD at the same seed (42). Full-solution + PG also reaches 66.7%. These are final Epoch-3 four-benchmark averages. Fixed context reduction therefore does not reproduce AOPSD's gain in this setting.

The distinction is how information relates to the current attempt. L3 fixes the context per problem. AOPSD reveals the checkpoints established by the current rollout and the next frontier whose prerequisites have been established. Thus, the same problem can receive different guidance as the student progresses. Context length does not specify which next inference the student's current reasoning supports; the dependency frontier does.

Q2 — Objective. AOPSD uses sampled-token PG, while the OPSD comparison uses top-K forward KL. Replacing OPSD's loss with the PG update, with w=1, gives 66.7%, compared with 65.9% for OPSD and 72.5% for AOPSD. Thus, PG alone does not recover AOPSD's gain. Independently, Table 6 holds the OPSD objective fixed: changing full-DAG disclosure to frontier disclosure raises the final three-benchmark average from 62.2% to 68.9%. These comparisons establish a contribution from adaptive disclosure beyond the objective change.

W2/Q3 — Three training seeds. Each Pass@8 aggregates eight samples per problem, and pooled over the 120 problems AOPSD solves 87 vs 79–82 for the other methods. As requested, we will add 3-seed mean ± std for AOPSD, OPSD and GRPO. Final Epoch-3 four-benchmark macro-average Pass@8 (%):

MethodSeed 42Seed 43Seed 44Mean ± std
OPSD65.965.966.766.2 ± 0.5
GRPO68.465.967.567.3 ± 1.3
AOPSD72.571.772.572.2 ± 0.5

The fixed reduced context baseline and the objective matched OPSD address W1 and Q2, and the three-seed results address W2. The three controls all report exactly 66.7%. Could you give the per-benchmark numbers for each, so it is clear they are distinct runs? I am raising my rating to 3.

Thanks for your careful reading. We apologize for the unclear naming: Full-solution + PG in W1/Q1 and the OPSD experiment using PG with w=1w=1 in Q2 refer to the same experiment.

Below are the detailed results for L3 + PG and Full-solution + PG. Both experiments use training seed 42 and sampled-token PG.

MethodHMMT25 (E3)AIME24 (E3)AIME25 (E3)BRUMo25 (E3)Avg. E1Avg. E2Avg. E3
L3 + PG40.083.366.776.766.768.366.7
Full-solution + PG43.380.066.776.765.065.966.7
Reviewer_5JHgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes adaptive on-policy self-distillation for mathematical reasoning, combining selective disclosure of a reasoning DAG, a competence-based curriculum, and continuation-based feedback weighting.

Critical strengths
  1. The paper studies an important problem: a teacher’s benefit from privileged information does not necessarily translate into better student learning.

  2. The reported results and component ablations suggest that adaptive disclosure and feedback weighting can improve upon standard OPSD.

Critical weaknesses
  1. Limited novelty. The method mainly combines established ideas in selective information disclosure, curriculum learning, and difficulty-based weighting. The paper does not sufficiently establish a substantial conceptual advance beyond this integration.
  2. Unclear presentation. The core idea and end-to-end training procedure are difficult to follow. Several concepts are introduced before being adequately explained, and the rationale for key design choices—particularly continuation-NLL weighting—remains insufficiently clear.
  3. Limited experimental reliability. Each benchmark contains only 30 evaluated problems, and the main results lack repeated-run uncertainty estimates. The differences between the main and ablation results also require clarification.
Questions for authors

NA

1 comment

We thank the reviewer for the valuable feedback.

W1 — Conceptual contribution. The central distinction is between helping a teacher solve with PI and helping a student learn to solve without it. AOPSD makes privileged supervision relative to the student's evolving capabilities: dependency-linked checkpoints determine what the teacher sees, and continuation probes determine how strongly its feedback is used. The DAG enables disclosure of the next complete reasoning frontier; the curriculum reuses the same progress signal for scheduling. Our contribution lies in this formulation, rather than novelty claims for curriculum learning or weighting in isolation. With the OPSD objective unchanged, frontier disclosure raises the final three-benchmark average from 62.2% to 68.9% (Table 6).

W2 — Procedure and NLL. The student first solves without PI. Its response is aligned to the DAG, and the frozen teacher, initialized from the same base model, scores that response using the selected subgraph. At high-disagreement positions, a short teacher continuation is scored by the question-only student; its NLL sets a weight for updating the original student token. These continuations supply weights, not extra training targets. Low NLL often indicates little new guidance; extreme NLL raises the risk of an unsupported jump; intermediate NLL balances novelty and current support.

W3 — Reliability and table consistency. We now have three training seeds (42/43/44). Final Epoch-3 macro-average Pass@8 over the four benchmarks is:

MethodMean ± std across seeds (%)
OPSD66.2 ± 0.5
GRPO67.3 ± 1.3
AOPSD72.2 ± 0.5

AOPSD improves over both baselines in every matched run.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.