- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a training-free, per-token diagnostic for on-policy distillation. It derives an ideal per-node gradient, ∂L/∂z_j = P_θ(j)(P_succ(j) − P̄_succ), the logit direction that maximally raises the student's probability of reaching a correct answer from that node, and shows Dr. GRPO recovers it in expectation. It then shows GKD, the single-sample estimator and MiniLLM all produce per-node gradients of the shared form P_θ(j)(f_j − f̄), differing only in f, which makes them comparable by direction. The gradient alignment score is the cosine between the ideal and distillation gradients at each branching node. Success probabilities come from targeted rollouts under exponentially growing depth windows, with one enriched generation tree shared across all teacher configurations. The framework is applied to Qwen3-0.6B and Qwen3-1.7B across eight teacher configurations on BoolQ and MMLU (around 264K decision points) plus AIME 2025 case studies, and the resulting teacher ranking is checked against real distillation runs.
Critical strengths
-
The unification is clean and correct. Showing that reward-based and distillation objectives share the local structure P_θ(j)(f_j − f̄) and differ only in f gives one frame for comparing GRPO-style and KD-style updates at token granularity. The corollary the authors draw from the shared P_θ(j) factor, that the teacher can amplify continuations the student already partly believes in but cannot inject new ones in a single step, is a useful mechanistic statement and it rationalises several scattered findings about overly capable teachers.
-
The finding that distillation guidance is more aligned with the ideal on incorrect rollouts than on correct ones, significant under the SR-weighted cosine in all four settings, is concrete and directly actionable: weight distillation toward failing rollouts. The teacher-independent tree sharing that makes an eight-configuration comparison affordable is a well-judged piece of experimental engineering.
Critical weaknesses
-
The score's dynamic range is too narrow for the ranking claims placed on it. In Table 1 the eight configurations span .000 to .053 with 95% CIs of roughly ±.02 to ±.04 that overlap almost entirely. At 0.6B on BoolQ, Qwen3-4B, 8B and 14B score .018, .018 and .016: two exact ties inside a confidence interval twice the size of the spread. In Figure 4 the fraction of aligned decision points runs 0.501 to 0.533 against a 0.500 chance line. "The teacher ranking largely agrees with the score" is then an ordering over quantities that are not statistically distinguishable, and no pairwise significance is reported.
-
The validation against training is thin and partly self-defeating. Table 2 covers three external teachers, one dataset and two learning rates. The ranking flips at 1e-6. MMLU cannot validate anything because every teacher lowers accuracy. The paper itself notes the learning-rate effect is two to three times the between-teacher spread. Meanwhile the self-distillation configurations, which carry the most interesting claims in Table 1 and Figure 4, are validated only as "preliminary" runs in the appendix, so the headline conclusion about self-distillation versus external teachers is not checked against training at all.
-
The cost works against the stated purpose, and the benchmarks do not match the use case. The pitch is avoiding expensive training runs, but the diagnostic needs 45K to 200K targeted rollouts per question, which is why AIME appears only as case studies. That limits practical use to short-chain tasks. BoolQ (True/False) and MMLU (four-way multiple choice) are exactly where OPD matters least, and where P_succ sits against a high chance floor, so the per-node success estimates are less informative than they would be in the long-chain regime the method is motivated by.
Questions for authors
- What is the actual GPU cost of scoring one BoolQ question relative to one distillation training run at the same scale? A direct comparison would settle whether the diagnostic is cheaper than what it replaces.
- With S_u typically holding only two or three tokens, how sensitive are the aggregate scores to N_sig and to the τ = 0.02 inclusion threshold?
- Does the correct/incorrect alignment gap survive conditioning on nodes with comparable P̄_succ, or is part of it explained by correct and incorrect paths sitting at different points on the success-probability scale?
We are very grateful for an exceptionally careful and constructive review that captures the paper precisely. We appreciate the recognition of the clean unification, the mechanistic corollary of the shared P_theta(j) factor, the actionable incorrect versus correct finding, and the teacher-independent tree sharing. We address each point below.
Small magnitude. Empirically, the mean cosine and the fraction of aligned nodes move together (SELF-1C, 0.6B BoolQ: .526 aligned, mean .047), so a score near 0.05 means the teacher favors the better branch at slightly more than half of the outcome-relevant forks. That this is only slightly above chance is the paper's message: not all teacher guidance is useful. The useful signal concentrates on failing rollouts (Figure 3) and on a subset of nodes whose selection raises the mean signal 10 to 15 times (Table 8).
Self-distillation. The main claim, that correct-only contexts beat wrong-demonstration contexts, holds in all 8 training comparisons, and the score orders all 8 the same way (one-sided sign test p of about 0.004); within-setting Spearman is 0.67 to 1.00. Summarization is confirmed too: for 1.7B, the score ranks SELF-SUM-1C best on both datasets, and it trains best on both. We agree the two teacher families were trained under different protocols (LoRA versus full fine-tuning); we will add matched runs and move this validation to the main text.
MMLU. We agree MMLU cannot validate the ranking, as every teacher lowers accuracy there. The score ranks configurations by per-token signal quality; absolute gain also depends on the training setup and the factors above. We will state this scope explicitly.
Q1: We will add a per-question cost comparison with a training run and a ranking versus budget analysis.
Q2: As noted in Section 2.5, raising N_sig to 40 or 60 did not meaningfully change our conclusions. We will add the full table for N_sig in {20, 40, 60} and stricter tau, reporting ranking stability and the incorrect versus correct gap.
Q3: An excellent question. Existing results suggest the gap is not a scale artifact: the cosine is invariant to the smaller ideal-gradient magnitude, the gap appears for nearly every teacher (Table 12), and it persists on AIME. We will add the analysis stratified by mean P_succ. The practical recipe holds either way: rollout correctness is observable at training time while mean P_succ is not, so weighting by outcome is the implementable version.
Thank you again; this review will directly improve our statistical presentation and the scope of our claims.