- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper defines token utility in on-policy distillation through the maximum first-order decrease in a position's distillation loss under an equal local KL budget on the student's predictive distribution. The resulting utility reduces to log-ratio variance for reverse KL and Pearson chi-squared divergence for forward KL. Fisher Utility Weighting (FUW) converts these quantities into detached, normalized token weights. Experiments on three Qwen3 student–teacher pairs and seven mathematical-reasoning benchmarks show improvements over matched uniform weighting. Additional analyses compare the effective support of existing importance criteria and examine the effects of the metric, weight assignment, and continuous weighting.
Critical strengths
-
The mathematical account is coherent and its scope is stated accurately. The reverse- and forward-KL identities provide an interpretable connection between the distillation objective and a distributional notion of local improvement (Sections 2–3; Appendix A). The finite-radius analysis further clarifies the infinitesimal result. The paper explicitly distinguishes this analysis from globally optimal weighting under shared transformer parameters (lines 216–219).
-
The empirical study includes useful controls beyond final benchmark scores. FUW and uniform OPD share the candidate support and loss, and the reported reverse-KL pooled gains of approximately 4.6–4.7 points on the two 1.7B configurations are meaningful (Section 5.1). The uniform-training trajectory used for the main support analysis, together with Euclidean, shuffled-weight, and hard-selection controls, makes the study informative about similarities and differences among weighting rules. Computing FUW requires no additional rollout or outcome-verification stage.
Critical weaknesses
-
The distinctive theoretical contribution and its connection to actual training remain insufficiently established. TRB, Appendix F, Eq. 23 already derives the same leading-order reverse-KL improvement coefficient under a student-centered KL budget. Although TRB changes rollout behavior rather than loss weights, this overlap narrows the novelty of the reverse-KL derivation. Furthermore, the local analysis optimizes each position's predictive distribution independently; it does not establish that weighting losses by the square root of utility improves shared-parameter training (lines 216–219, 252–268). The authors appropriately acknowledge this limitation, but the empirical bridge remains weak: Fisher and Euclidean weighting differ by only 0.27 pooled points in Table 6, with no significant paired difference (p = .527). Lower weight concentration is not linked to demonstrated stability or efficiency benefits. The forward-KL comparison against uniform weighting likewise does not establish that matching the utility to the objective matters. Thus, the framework's consequential value beyond simpler weighting rules remains unclear.
-
The reliability and scope of the practical gains need stronger support. Each configuration is trained once, and bootstrap uncertainty covers evaluation problems rather than training variability (lines 506–507). The pooled difference from TIP includes zero in its confidence interval on 1.7B ← 8B (lines 343–344); TIDE scores slightly higher on the 4B pair (Table 2), where the forward-KL pooled gain over uniform weighting is only 0.25 points (Table 7). These findings leave the advantage over established alternatives and its persistence with stronger students unresolved. There is also a concrete reproducibility ambiguity: Section 5.4 (lines 405–408) clips log(q/p) without specifying the resulting utility formula. After clipping, E_p[(q/p − 1)^2] and E_p[(q/p)^2] − 1 cease to be equivalent, and the latter can be negative. This requires clarification before the forward-KL implementation can be assessed; it is not evidence of an error in the unclipped theorem.
Questions for authors
-
How does your reverse-KL derivation relate to TRB Appendix F, Eq. 23? Beyond using this quantity to compare positions, what new theoretical or empirical conclusion follows from the proposed framework?
-
What evidence connects the independently optimized local utility to beneficial shared-parameter updates? Given the Euclidean result in Table 6, does Fisher utility better predict realized loss improvement, or does its lower concentration produce measurable stability or efficiency benefits?
-
What exact forward-KL score is computed after clipping (lines 405–408)? For and differ. For example, p = (0.999, 0.001) and q = (0.5, 0.5) give approximately 0.6135 and −0.3463, respectively. Please specify the implemented expression, any nonnegativity handling, and its relationship to the theoretical utility.
-
Does matching the utility to the loss matter empirically? A crossed comparison of reverse- and forward-KL utilities, or a matched continuous gradient/disagreement baseline under forward KL, would distinguish objective-specific value from generic nonuniform weighting.
-
How reproducible are the key contrasts across independently trained runs? Permuted FUW still reaches 38.15 pooled accuracy versus 34.5 for uniform OPD, compared with 39.17 for FUW (Tables 1 and 6). Why does such a large gain survive removal of score-to-position alignment, and is this pattern stable across seeds?
-
What evidence identifies when gains persist or diminish as student capability increases? The smaller gains in the 4B setting are difficult to interpret because the teacher also changes. Evidence holding the teacher and training protocol fixed would materially strengthen the scope of the findings.
4 comments
Response to Reviewer akN1 (2/2)
(Part 2 of 2.)
Connection to shared-parameter updates. Please see the General Response (Analysis of local utility to realized loss reduction), where we ran the suggested test. On a uniform-OPD trajectory, the Fisher utility predicts each position's realized loss decrease after 1–5 shared-parameter steps better than the Euclidean score at every horizon (Spearman 0.50 vs 0.46–0.47; difference +0.035, 95% CI within [0.029, 0.040]). We have not measured a stability or efficiency benefit from lower weight concentration; we will present the concentration statistics in §5.3 and B.3 as descriptive.
Training variance and the permuted-weight control. We retrained FUW, permuted FUW, and uniform reverse-KL OPD on Qwen3-1.7B-Base ← Qwen3-8B with independent rollout seeds. All runs are evaluated at the final checkpoint (step 279), with no checkpoint selection (Avg@8, macro / pooled; paired bootstrap over problems):
| Seed 0 | Seed 1 | Seed 2 | Mean ± s.d. | |
|---|---|---|---|---|
| Uniform OPD | 22.8 / 34.5 | 21.9 / 34.3 | 21.7 / 34.1 | 22.1 ± 0.6 / 34.3 ± 0.2 |
| Permuted FUW | 25.9 / 38.2 | 24.8 / 38.0 | 24.9 / 37.9 | 25.2 ± 0.6 / 38.0 ± 0.15 |
| FUW | 26.8 / 39.2 | 25.7 / 39.4 | 25.7 / 39.1 | 26.1 ± 0.6 / 39.2 ± 0.15 |
| FUW − permuted, pooled [95% CI] | +1.02 [0.11, 1.92] | +1.41 [0.56, 2.26] | +1.13 [0.27, 1.99] | |
| FUW − permuted, macro [95% CI] | +0.90 [−0.30, 2.10] | +0.85 [−0.36, 2.06] | +0.84 [−0.80, 2.48] |
Both observations are stable across seeds. FUW's score is reproducible: pooled within 0.3 points across three seeds, with the macro spread concentrated in the 30-problem AIME sets. The permutation pattern also replicates. In each seed, permuted FUW remains well above uniform OPD, and FUW leads it by a similar margin that is individually significant on the pooled average. The macro differences point the same way but are within noise for single runs, since macro gives the 30-problem sets equal weight.
The permutation preserves FUW's weight distribution while removing its alignment with positions. Permuted FUW retains much of the gain over uniform OPD, while utility-aligned FUW achieves approximately one additional pooled point across seeds. This is consistent with contributions from both non-uniform weighting and utility-based assignment.
Ablating student models for a fixed teacher. We conducted an additional comparison using the same Qwen3-14B teacher and reverse-KL training protocol with two students. With Qwen3-1.7B-Base, FUW improves over uniform OPD by 3.1 macro points (23.9 → 27.0; Table 3). With the stronger Qwen3-4B, FUW improves by 0.8 points (42.8 → 43.6). Both comparisons use single training runs evaluated at the final checkpoint.
The absolute gain is smaller for the stronger student, which also has substantially less headroom relative to the teacher (46.9): approximately 4 points, compared with 23 points for the 1.7B student. The 4B student is post-trained, whereas the 1.7B student is a base model. We read the smaller gain as consistent with reduced headroom available for improvement.
Follow-up on Author Rebuttal
Thank you again for your review. We would like to kindly check whether our response has addressed your concerns. We would be happy to clarify any remaining questions before the discussion period ends.
Thanks for the clarifications. They resolve my implementation concern and strengthen the evidence. I updated my rating.
Response to Reviewer akN1 (1/2)
We thank the reviewer for the detailed and constructive review, and especially for the precise questions on TRB and on the forward-KL implementation. (Part 1 of 2.)
Q1: Relation to TRB (Appendix F, Eq. 23), and what the framework adds. Thank you for pointing us to Plyusov et al., Appendix F, Eq. 23. Their fixed-prefix, small-budget improvement is 2εs+O(ε), where s is the student-weighted variance of the teacher–student log-ratio. Reversing the sign of the log-ratio leaves the variance unchanged. We will explicitly credit TRB for this coefficient and add it to the related work section.
TRB optimizes the behavior distribution used to collect rollouts under KL(x∥p)≤ε, where p is the student. The optimum is a geometric blend, and TRB obtains its mixing strength by solving the finite-budget constraint. Our Appendix A.5 solves the corresponding finite-budget optimization under the opposite orientation, KL(p∥x)≤ε, yielding a Lambert-W solution.
Both orientations give the same local Fisher metric. Our contribution is to use the attainable improvement scale to compare and weight losses at student-visited positions, to derive the corresponding forward-KL utility, and to test those choices in training.
Reference: Plyusov et al., Trust-Region Behavior Blending for On-Policy Distillation, Appendix F, Eq. (23), https://arxiv.org/html/2605.31159v1#A6
Forward-KL score after clipping. Let p and q be the student and teacher distributions, each renormalized on the union of their top-16 candidates. For the forward-KL loss, the exact local Fisher utility is χ2(q∥p). With ρ=q/p, this equals Varp(ρ) because Ep[ρ]=1. Our practical score clips the ratio at e3 before computing the variance under p. The variance is non-negative by construction.
Implemented score. For R=min(q/p,e3), the score is Varp(R)=Ep[R2]−(Ep[R])2. Clipping applies to the utility calculation only. The forward-KL loss uses the restricted distributions directly, and the square root of the score supplies the detached, mean-normalized weight in Eqs. 10–11.
After clipping, Ep[R]≤1, with strict inequality when the cap changes a ratio of positive student probability. In the two-outcome example with p=(0.999,0.001) and q=(0.5,0.5), the variance is 0.38319. The squared deviation Ep[(R−1)2] is 0.61351, while Ep[R2]−1 is −0.34632. We will specify the centered clipped variance explicitly in §5.4.
Does matching the utility to the objective matter? We ran the crossed comparison on Qwen3-1.7B-Base ← Qwen3-8B, changing only the weighting statistic; the loss, support, clip, and training budget are held fixed (Macro Avg@8 over seven benchmarks, mean ± s.e. over problems):
Under reverse KL, the matched utility achieved 2.5 points higher macro accuracy than the crossed utility. Under forward KL, the matched score led by 0.5 macro points, a difference within evaluation uncertainty. On identical forward-KL training tokens, the scores were strongly rank-correlated (Spearman ρ=0.83) and had substantial top-20% overlap (Jaccard = 0.73). The clearest downstream evidence for objective matching therefore comes from reverse KL on this model pair.
Revision. We will add the crossed comparison and token-overlap analysis to the appendix.