- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Adaptive On-Policy Self-Distillation (AOPSD), which consists of three main components: (1) representing the reasoning steps and their dependencies in reference solutions as Reasoning DAGs, and adaptively disclosing Privileged Information (PI) based on the Student's current progress; (2) dynamically adjusting the curriculum according to the Student's reasoning progress; and (3) using Teacher continuations and Student NLL to compute token-level weights, thereby adjusting the strength of supervision for high-disagreement tokens. The authors evaluate the effectiveness of these components through ablation studies, hyperparameter analysis, and semantic audits.
Critical strengths
- The paper has a clear motivation, showing that PI can significantly improve the Teacher's conditional problem-solving ability.
- The design of Adaptive PI Disclosure is intuitive and easy to understand.
- Continuation-based Selective Distillation provides a lightweight way to adjust the strength of supervision.
Critical weaknesses
- I'm not sure how well the DAG alignment works when the Student takes a different but valid reasoning path.
- I'm a bit concerned about using the Student to judge its own progress, as mistakes here could affect both PI Disclosure and Curriculum Learning.
Questions for authors
The method is interesting and the results are promising, but I still have some questions about the key design choices.
- For problems with multiple valid solution paths, how does the proposed DAG alignment handle cases where the Student follows a reasoning path different from the reference DAG? Could this lead to an underestimation of the Student's reasoning progress?
- Reliability of progress estimation. Have the authors compared Student-based judgments with those of an independent LLM judge or human annotators? If some checkpoints are incorrectly marked as reached, how much would this affect the performance of Adaptive PI Disclosure and Curriculum Learning?
- The authors assume that Teacher continuations with moderate NLL are more likely to provide useful guidance, while those with high NLL may indicate PI leakage. Could the authors further explain the rationale behind this assumption? For example, is NLL-based weighting more effective than simply adjusting token weights based on Teacher–Student disagreement?
We thank the reviewer for the positive assessment and the pointed questions.
Q1 – Different valid paths. A different path is still credited as long as it reaches the same key results; a path that shares none can be under-credited, which is the safe direction. (1) Alignment compares results, not steps. DAG nodes are intermediate results (e.g., "a = 2, b ≤ 2, c ≤ 1"), and a node is credited whenever the response establishes that result, however it was derived. (2) If a route shares no results with the reference, progress is underestimated. The effect is mild: the teacher sees only the first nodes, i.e., less PI, which is the safer direction because the risk we address is that a teacher given excessive PI may inadvertently leak it through its feedback; and the problem is treated as harder, which only changes when it is visited within an epoch (every problem is still visited every epoch, and competence is averaged over attempts). Supporting several reference routes per problem is a natural extension.
Q2 – Reliability of self-judging. Same as our reply to Reviewer HHn3: errors are bounded by design (a node counts only if its prerequisites are reached; a single false positive moves the disclosure by at most one step; the answer is stripped from the DAG; continuation weights suppress PI-dependent feedback anyway), the judgment is a comparatively easy recognition task, and on 299 of 300 frozen rollouts that we re-judged with an independent stronger model, the student's judgments agree with it on 76.4% of node decisions, with the residual disagreement erring toward slightly earlier disclosure (+0.130 progress on the 0–1 scale).
Q3 – Why NLL rather than disagreement alone? Disagreement says how much PI changes the teacher's preference at a token, not whether the student can learn from it. A correction grounded in the problem and a jump that only PI explains can show the same gap; they differ in what the teacher says next. Because the teacher is the student plus PI, the student's NLL on the teacher's continuation measures how much of it is reachable without PI (up to the teacher's own small NLL, it is exactly the log-likelihood ratio, i.e., the information PI adds). Predictable → nothing new; extremely surprising → only PI explains it; in between → learnable. A weight based on disagreement alone cannot tell these apart; NLL can: among events that all pass the same |g| ≥ 2 threshold, the share of PI leakage rises from 0.09% to 91% across NLL bins (NLL-blind audit, Table 8).
Ethics flag. We are not aware of a specific concern: the study uses only public datasets and no human subjects, and LLM use is disclosed in the paper. We would be glad to clarify if the reviewer can point us to one.