acceptodds
Under review as a conference paper at ICLR 2027

TOKEN UTILITY FOR ON-POLICY DISTILLATION VIA LOCAL TRUST-REGION OPTIMIZATION

23 views

Abstract

Token-selection methods for on-policy distillation (OPD) prioritize student-visited positions using diverse signals, including uncertainty, teacher–student disagreement, and mismatch direction. We instead derive token importance from the distillation objective itself: how much can each position's loss decrease under an equal local KL budget on the student's predictive distribution? The answer is the utility , where is the logit gradient and is the categorical Fisher matrix. For reverse-KL it equals the student-weighted variance of the student–teacher log-probability ratio, and for forward-KL it equals the Pearson divergence, so the appropriate utility follows from the choice of objective. On shared training checkpoints, several independently motivated criteria, including student entropy, KL disagreement, and TIP, concentrate their importance on nearly the same positions as the derived utility at a 50% selection fraction, while diverging at stricter fractions and for direction-specific criteria. We instantiate the utility as Fisher Utility Weighting (FUW), a continuous, mean-normalized token weighting. Across seven mathematical reasoning benchmarks and three Qwen3 student–teacher pairs, FUW improves over matched uniform OPD under both reverse-KL (1.35–4.0 macro points) and forward-KL (1.08–2.71 macro points), and is competitive with existing selective methods using a single closed-form score derived from the training objective.

Primary area:
foundation or frontier models, including LLMs
Keywords:
on policy distillation, token utility, forward and reverse KL

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_akN1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper defines token utility in on-policy distillation through the maximum first-order decrease in a position's distillation loss under an equal local KL budget on the student's predictive distribution. The resulting utility reduces to log-ratio variance for reverse KL and Pearson chi-squared divergence for forward KL. Fisher Utility Weighting (FUW) converts these quantities into detached, normalized token weights. Experiments on three Qwen3 student–teacher pairs and seven mathematical-reasoning benchmarks show improvements over matched uniform weighting. Additional analyses compare the effective support of existing importance criteria and examine the effects of the metric, weight assignment, and continuous weighting.

Critical strengths
  1. The mathematical account is coherent and its scope is stated accurately. The reverse- and forward-KL identities provide an interpretable connection between the distillation objective and a distributional notion of local improvement (Sections 2–3; Appendix A). The finite-radius analysis further clarifies the infinitesimal result. The paper explicitly distinguishes this analysis from globally optimal weighting under shared transformer parameters (lines 216–219).

  2. The empirical study includes useful controls beyond final benchmark scores. FUW and uniform OPD share the candidate support and loss, and the reported reverse-KL pooled gains of approximately 4.6–4.7 points on the two 1.7B configurations are meaningful (Section 5.1). The uniform-training trajectory used for the main support analysis, together with Euclidean, shuffled-weight, and hard-selection controls, makes the study informative about similarities and differences among weighting rules. Computing FUW requires no additional rollout or outcome-verification stage.

Critical weaknesses
  1. The distinctive theoretical contribution and its connection to actual training remain insufficiently established. TRB, Appendix F, Eq. 23 already derives the same leading-order reverse-KL improvement coefficient under a student-centered KL budget. Although TRB changes rollout behavior rather than loss weights, this overlap narrows the novelty of the reverse-KL derivation. Furthermore, the local analysis optimizes each position's predictive distribution independently; it does not establish that weighting losses by the square root of utility improves shared-parameter training (lines 216–219, 252–268). The authors appropriately acknowledge this limitation, but the empirical bridge remains weak: Fisher and Euclidean weighting differ by only 0.27 pooled points in Table 6, with no significant paired difference (p = .527). Lower weight concentration is not linked to demonstrated stability or efficiency benefits. The forward-KL comparison against uniform weighting likewise does not establish that matching the utility to the objective matters. Thus, the framework's consequential value beyond simpler weighting rules remains unclear.

  2. The reliability and scope of the practical gains need stronger support. Each configuration is trained once, and bootstrap uncertainty covers evaluation problems rather than training variability (lines 506–507). The pooled difference from TIP includes zero in its confidence interval on 1.7B ← 8B (lines 343–344); TIDE scores slightly higher on the 4B pair (Table 2), where the forward-KL pooled gain over uniform weighting is only 0.25 points (Table 7). These findings leave the advantage over established alternatives and its persistence with stronger students unresolved. There is also a concrete reproducibility ambiguity: Section 5.4 (lines 405–408) clips log(q/p) without specifying the resulting utility formula. After clipping, E_p[(q/p − 1)^2] and E_p[(q/p)^2] − 1 cease to be equivalent, and the latter can be negative. This requires clarification before the forward-KL implementation can be assessed; it is not evidence of an error in the unclipped theorem.

Questions for authors
  1. How does your reverse-KL derivation relate to TRB Appendix F, Eq. 23? Beyond using this quantity to compare positions, what new theoretical or empirical conclusion follows from the proposed framework?

  2. What evidence connects the independently optimized local utility to beneficial shared-parameter updates? Given the Euclidean result in Table 6, does Fisher utility better predict realized loss improvement, or does its lower concentration produce measurable stability or efficiency benefits?

  3. What exact forward-KL score is computed after clipping (lines 405–408)? For Rtilde=min(q/p,exp(3)),Ep[(Rtilde−1)2]R_tilde = min(q/p, exp(3)), E_p[(R_tilde − 1)^2] and Ep[Rtilde2]−1E_p[R_tilde^2] − 1 differ. For example, p = (0.999, 0.001) and q = (0.5, 0.5) give approximately 0.6135 and −0.3463, respectively. Please specify the implemented expression, any nonnegativity handling, and its relationship to the theoretical utility.

  4. Does matching the utility to the loss matter empirically? A crossed comparison of reverse- and forward-KL utilities, or a matched continuous gradient/disagreement baseline under forward KL, would distinguish objective-specific value from generic nonuniform weighting.

  5. How reproducible are the key contrasts across independently trained runs? Permuted FUW still reaches 38.15 pooled accuracy versus 34.5 for uniform OPD, compared with 39.17 for FUW (Tables 1 and 6). Why does such a large gain survive removal of score-to-position alignment, and is this pattern stable across seeds?

  6. What evidence identifies when gains persist or diminish as student capability increases? The smaller gains in the 4B setting are difficult to interpret because the teacher also changes. Evidence holding the teacher and training protocol fixed would materially strengthen the scope of the findings.

4 comments
AuthorsOpenReview ↗

Response to Reviewer akN1 (1/2)

We thank the reviewer for the detailed and constructive review, and especially for the precise questions on TRB and on the forward-KL implementation. (Part 1 of 2.)

Q1: Relation to TRB (Appendix F, Eq. 23), and what the framework adds. Thank you for pointing us to Plyusov et al., Appendix F, Eq. 23. Their fixed-prefix, small-budget improvement is 2εs+O(ε)\sqrt{2\varepsilon s} + O(\varepsilon), where ss is the student-weighted variance of the teacher–student log-ratio. Reversing the sign of the log-ratio leaves the variance unchanged. We will explicitly credit TRB for this coefficient and add it to the related work section.

TRB optimizes the behavior distribution used to collect rollouts under KL(x∥p)≤ε\mathrm{KL}(x \| p) \le \varepsilon, where pp is the student. The optimum is a geometric blend, and TRB obtains its mixing strength by solving the finite-budget constraint. Our Appendix A.5 solves the corresponding finite-budget optimization under the opposite orientation, KL(p∥x)≤ε\mathrm{KL}(p \| x) \le \varepsilon, yielding a Lambert-W solution.

Both orientations give the same local Fisher metric. Our contribution is to use the attainable improvement scale to compare and weight losses at student-visited positions, to derive the corresponding forward-KL utility, and to test those choices in training.

Reference: Plyusov et al., Trust-Region Behavior Blending for On-Policy Distillation, Appendix F, Eq. (23), https://arxiv.org/html/2605.31159v1#A6

Forward-KL score after clipping. Let pp and qq be the student and teacher distributions, each renormalized on the union of their top-16 candidates. For the forward-KL loss, the exact local Fisher utility is χ2(q∥p)\chi^2(q \| p). With ρ=q/p\rho = q/p, this equals Varp(ρ)\mathrm{Var}_p(\rho) because Ep[ρ]=1\mathbb{E}_p[\rho] = 1. Our practical score clips the ratio at e3e^3 before computing the variance under pp. The variance is non-negative by construction.

Implemented score. For R=min⁡(q/p,e3)R = \min(q/p, e^3), the score is Varp(R)=Ep[R2]−(Ep[R])2\mathrm{Var}_p(R) = \mathbb{E}_p[R^2] - (\mathbb{E}_p[R])^2. Clipping applies to the utility calculation only. The forward-KL loss uses the restricted distributions directly, and the square root of the score supplies the detached, mean-normalized weight in Eqs. 10–11.

After clipping, Ep[R]≤1\mathbb{E}_p[R] \le 1, with strict inequality when the cap changes a ratio of positive student probability. In the two-outcome example with p=(0.999,0.001)p = (0.999, 0.001) and q=(0.5,0.5)q = (0.5, 0.5), the variance is 0.38319. The squared deviation Ep[(R−1)2]\mathbb{E}_p[(R-1)^2] is 0.61351, while Ep[R2]−1\mathbb{E}_p[R^2] - 1 is −0.34632. We will specify the centered clipped variance explicitly in §5.4.

Does matching the utility to the objective matter? We ran the crossed comparison on Qwen3-1.7B-Base ← Qwen3-8B, changing only the weighting statistic; the loss, support, clip, and training budget are held fixed (Macro Avg@8 over seven benchmarks, mean ± s.e. over problems):

Loss \ WeightsReverse-KL utility Var[r]\mathrm{Var}[r]Forward-KL utility χ2\chi^2
Reverse KL26.8 ± 1.4 (matched, FUW)24.3 ± 1.2 (crossed)
Forward KL22.3 ± 1.1 (crossed)22.8 ± 1.1 (matched)

Under reverse KL, the matched utility achieved 2.5 points higher macro accuracy than the crossed utility. Under forward KL, the matched score led by 0.5 macro points, a difference within evaluation uncertainty. On identical forward-KL training tokens, the scores were strongly rank-correlated (Spearman ρ=0.83\rho = 0.83) and had substantial top-20% overlap (Jaccard = 0.73). The clearest downstream evidence for objective matching therefore comes from reverse KL on this model pair.

Revision. We will add the crossed comparison and token-overlap analysis to the appendix.

AuthorsOpenReview ↗

Response to Reviewer akN1 (2/2)

(Part 2 of 2.)

Connection to shared-parameter updates. Please see the General Response (Analysis of local utility to realized loss reduction), where we ran the suggested test. On a uniform-OPD trajectory, the Fisher utility predicts each position's realized loss decrease after 1–5 shared-parameter steps better than the Euclidean score at every horizon (Spearman 0.50 vs 0.46–0.47; difference +0.035, 95% CI within [0.029, 0.040]). We have not measured a stability or efficiency benefit from lower weight concentration; we will present the concentration statistics in §5.3 and B.3 as descriptive.

Training variance and the permuted-weight control. We retrained FUW, permuted FUW, and uniform reverse-KL OPD on Qwen3-1.7B-Base ← Qwen3-8B with independent rollout seeds. All runs are evaluated at the final checkpoint (step 279), with no checkpoint selection (Avg@8, macro / pooled; paired bootstrap over problems):

Seed 0Seed 1Seed 2Mean ± s.d.
Uniform OPD22.8 / 34.521.9 / 34.321.7 / 34.122.1 ± 0.6 / 34.3 ± 0.2
Permuted FUW25.9 / 38.224.8 / 38.024.9 / 37.925.2 ± 0.6 / 38.0 ± 0.15
FUW26.8 / 39.225.7 / 39.425.7 / 39.126.1 ± 0.6 / 39.2 ± 0.15
FUW − permuted, pooled [95% CI]+1.02 [0.11, 1.92]+1.41 [0.56, 2.26]+1.13 [0.27, 1.99]
FUW − permuted, macro [95% CI]+0.90 [−0.30, 2.10]+0.85 [−0.36, 2.06]+0.84 [−0.80, 2.48]

Both observations are stable across seeds. FUW's score is reproducible: pooled within 0.3 points across three seeds, with the macro spread concentrated in the 30-problem AIME sets. The permutation pattern also replicates. In each seed, permuted FUW remains well above uniform OPD, and FUW leads it by a similar margin that is individually significant on the pooled average. The macro differences point the same way but are within noise for single runs, since macro gives the 30-problem sets equal weight.

The permutation preserves FUW's weight distribution while removing its alignment with positions. Permuted FUW retains much of the gain over uniform OPD, while utility-aligned FUW achieves approximately one additional pooled point across seeds. This is consistent with contributions from both non-uniform weighting and utility-based assignment.

Ablating student models for a fixed teacher. We conducted an additional comparison using the same Qwen3-14B teacher and reverse-KL training protocol with two students. With Qwen3-1.7B-Base, FUW improves over uniform OPD by 3.1 macro points (23.9 → 27.0; Table 3). With the stronger Qwen3-4B, FUW improves by 0.8 points (42.8 → 43.6). Both comparisons use single training runs evaluated at the final checkpoint.

The absolute gain is smaller for the stronger student, which also has substantially less headroom relative to the teacher (46.9): approximately 4 points, compared with 23 points for the 1.7B student. The 4B student is post-trained, whereas the 1.7B student is a base model. We read the smaller gain as consistent with reduced headroom available for improvement.

AuthorsOpenReview ↗

Follow-up on Author Rebuttal

Thank you again for your review. We would like to kindly check whether our response has addressed your concerns. We would be happy to clarify any remaining questions before the discussion period ends.

Thanks for the clarifications. They resolve my implementation concern and strengthen the evidence. I updated my rating.

Reviewer_E5pWOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper derives token importance for on-policy distillation from how much can each position’s loss decrease under an equal local KL budget on the student’s predictive distribution. The resulting utility is instantiated as Fisher Utility Weighting (FUW), a continuous, mean-normalized token weighting. Across seven mathematical reasoning benchmarks and three Qwen3 student–teacher pairs, FUW improves over matched uniform OPD under both reverse-KL and forward-KL objectives.

Critical strengths
  1. The motivation is clear, mathematical derivation connects token utility to distillation objectives.

  2. Comprehensive ablations and comparisons are conducted to show better performance than existing token-importance criteria.

  3. Consistent improvements over existing OPD variants.

Critical weaknesses
  1. The gains over simpler weighting methods are not consistently significant in various benchmarks.

  2. Single-run experiments are limited to Qwen3 models and mathematical reasoning.

Questions for authors

See weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer E5pW

Thank you for recognizing the motivation and the connection between the distillation objective and token utility. We agree that the strength and scope of the empirical conclusions should be stated more precisely.

W1: Significance of gains over simpler weighting methods. We agree, and we do not claim significant superiority over simpler weighting or selection methods; the abstract states that FUW is competitive with them. TIP, student entropy, KL disagreement, and the Euclidean gradient norm identify substantially overlapping high-importance positions with FUW (Section 5.2; General Response, Comparison with other scoring methods). This overlap is consistent with their similar downstream accuracy. Our contribution is to derive the utility from the distillation objective and to explain why these independently motivated criteria largely agree.

With standard errors of approximately 3–8 points on the four smaller benchmarks (30–83 problems each), differences between weighting methods are difficult to resolve reliably at the individual-benchmark level. We therefore emphasize macro and pooled aggregates across the seven benchmarks (1,620 problems), while acknowledging that these aggregates do not establish significant superiority over competing weighting methods.

W2: Single runs, Qwen3 only, mathematical reasoning only. We agree that single training runs are a limitation, and the paper states it explicitly: the Reproducibility Statement notes that each configuration is trained once and that the reported uncertainty reflects paired bootstrap resampling over evaluation problems, not training runs. To address training variability, we conducted two additional matched-seed runs for both uniform OPD and FUW on the main reverse-KL configuration (Qwen3-1.7B-Base ← Qwen3-8B). Across three matched rollout seeds, FUW improves over uniform OPD by +4.0, +3.8, and +4.0 macro Avg@8 points and +4.7, +5.1, and +5.0 pooled points. The mean paired improvements are +3.9 macro and +4.9 pooled points. All methods are evaluated at the final checkpoint (step 279), with no checkpoint selection. The General Response (Variation across training seeds) reports the complete results. We will add these experiments and revise the Reproducibility Statement to distinguish training-run variability from the evaluation-problem bootstrap uncertainty reported in the submission.

The theoretical derivation (Section 2; Appendix A) does not depend on the model family or task domain: the utility follows from the distillation objective and the categorical Fisher geometry, and we derive it for both reverse and forward KL. We chose mathematical reasoning because it is a standard testbed for OPD on reasoning models and enables direct comparison with TIP and TIDE under their evaluation protocols. Nevertheless, empirical generalization to other model families and domains remains untested, and we will emphasize this limitation in Section 7.

AuthorsOpenReview ↗

Follow-up on Author Rebuttal

Thank you again for your review. We would like to kindly check whether our response has addressed your concerns. We would be happy to clarify any remaining questions before the discussion period ends.

Reviewer_fbwBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines token importance in OPD as how much each position's loss can drop under the same KL budget. The derived score is the variance of the log-probability ratio under reverse KL and the chi-squared divergence under forward KL. The authors use sqrt(s) of this score as a token weight and call the method FUW. On three Qwen3 student-teacher pairs and seven math benchmarks, FUW improves over uniform OPD by 1 to 4 points and performs on par with TIP and TIDE.

Critical strengths
  1. The same framework gives different closed-form scores for reverse KL and forward KL. The score costs only O(V) to compute and needs no extra model.
  2. Uniform OPD and FUW use the same loss implementation and differ only in the weights. Shuffling the weights lowers accuracy by 1.02 points (p = 0.028), which shows that where the weights go matters. The authors also report honestly that the gap to TIP is not significant and that TIDE is better on the 4B setting.
Critical weaknesses
  1. The derivation measures how fast each position's loss can drop when updated on its own. It ignores parameter sharing and how a token's error affects the rest of the trajectory and the final answer. In reasoning tasks, positions with large KL may not affect the result, while branching points that decide the result may have small KL. The paper also does not explain why a larger possible improvement should mean a larger weight, or why the weight is sqrt(s).
  2. The paper's own analysis shows that at a 50% fraction, entropy, KL, gradient norm and TIP select almost the same positions as FUW, and replacing the Fisher score with the Euclidean gradient norm gives nearly the same accuracy (38.9 vs 39.2). The gap to TIP is not significant, and TIDE is better on 4B←30B. So deriving the score from the objective does not yet bring extra benefit.
Questions for authors
  1. How does FUW compare with using KL directly as a continuous weight?
  2. Why use sqrt(s) as the weight? Did you compare it with using s directly or other transforms?
2 comments
AuthorsOpenReview ↗

Response to Reviewer fbwB

We thank the reviewer for the constructive review and for recognizing the objective-specific closed forms, the matched controls, and our transparent reporting of non-significant comparisons.

W1(a): Locality, parameter sharing, and trajectory-level effects. We agree with this characterization, and it reflects a deliberate scope. FUW's weights are derived from a local analysis of the OPD loss itself: they measure where the distillation loss can be reduced under an equal local change in the student's distribution. Like OPD, FUW uses no outcome signal. It therefore does not model how a token's error propagates through the rest of the trajectory or affects the final answer. A large-disagreement position that does not change the outcome can still receive high weight, and an outcome-critical branching point where student and teacher nearly agree may receive little weight. Uniform OPD assigns a constant unit weight at such a position, although its distillation gradient may also be small. Accounting for downstream effects requires a trajectory-level signal; we view outcome-based credit assignment and FUW's distillation weights as complementary. We will state this scope explicitly in Section 7. The General Response (Analysis of local utility to realized loss reduction) reports a new analysis of whether local utility predicts realized loss reduction under shared-parameter updates.

W1(b) and Q2: Why larger improvement implies larger weight, and why s\sqrt{s}. Under the equal budget ε\varepsilon, the maximum first-order loss decrease at position tt is Δt∗=2εst\Delta^*_t = \sqrt{2\varepsilon s_t} (Section 2.2). The weight wt∝stw_t \propto \sqrt{s_t} is this attainable improvement, normalized across positions. Since sts_t and st\sqrt{s_t} rank positions identically, the choice affects only how strongly weight is concentrated, not which positions are prioritized. We did not compare it with sts_t or other transforms. We do not claim that this is the optimal weighting under shared-parameter training; we treat the transform as a design choice motivated by the improvement scale (Section 3, Scope).

Q1: KL as a continuous weight. We ran this baseline on Qwen3-1.7B-Base ← Qwen3-8B, identical to FUW except for the weighting statistic. Each position is weighted by its per-token KL on the candidate set, transformed and mean-normalized exactly as the FUW utility (w∝KL1/2w \propto \mathrm{KL}^{1/2}). Avg@8:

MethodMacroPooled
Uniform OPD22.8 ± 1.234.5 ± 1.0
TIP (Soft-OR, 50%)25.3 ± 1.438.4 ± 1.0
KL weighting26.2 ± 1.338.5 ± 0.9
FUW (ours)26.8 ± 1.439.2 ± 1.1

KL weighting recovers most of FUW's gain over uniform OPD. The FUW advantage of 0.6 macro and 0.7 pooled points is a point estimate, and we do not claim a statistically significant difference. This is consistent with the relationship between the scores: reverse KL is the student-weighted mean of the log-ratio whose variance defines Fisher utility (Table 4). In the shared-parameter update analysis, Fisher utility is a stronger predictor of realized loss decreases than KL (Spearman 0.50 versus 0.41–0.43). Linear weighting directly proportional to KL remains untested.

W2: Whether deriving the score brings extra benefit. We agree that several criteria select nearly the same positions at the 50% operating point, that the Fisher–Euclidean and FUW–TIP differences are not statistically significant, and that TIDE performs slightly better on 4B ← 30B-A3B. Our claim is that FUW is competitive with existing selective methods, rather than consistently superior, and we will clarify this in Section 5.1.

Beyond downstream accuracy, the derivation provides a principled explanation for the similarities and differences among existing criteria. Several methods rely on related statistics of the student–teacher log-ratio, helping explain why they often identify similar positions, while methods such as TIDE-excess emphasize a particular direction of mismatch. More importantly, the framework extends naturally across distillation objectives: applying the same principle to forward KL yields the χ2(pT∥pS)\chi^2(p_T \| p_S) utility, which improves over uniform forward-KL training on all three model pairs.

The framework also provides a continuous measure of attainable local improvement rather than only a selection criterion. This distinction matters empirically: permuting FUW's weights across positions reduces pooled accuracy by 1.02 points (p=0.028p = 0.028; Table 6), indicating that the assignment of weights contributes to performance. Furthermore, despite similar downstream accuracy, Fisher utility predicts realized loss reduction under shared-parameter updates more strongly than the Euclidean gradient norm (Spearman 0.50 versus 0.46–0.47). Together, these results support the value of an objective-derived utility measure, even when its downstream performance is comparable to existing methods.

AuthorsOpenReview ↗

Follow-up on Author Rebuttal

Thank you again for your review. We would like to kindly check whether our response has addressed your concerns. We would be happy to clarify any remaining questions before the discussion period ends.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2)

We thank the reviewers for their careful assessment. At a fixed student-visited prefix, our analysis asks how much the distillation loss can decrease under a common small KL budget on the student's predictive distribution. The maximum first-order decrease is proportional to the square root of Fisher utility, providing a common scale for comparing positions. Whether weighting by this scale improves training with shared parameters is an empirical question. We will clarify this distinction between the local derivation and its practical application.

End-to-end accuracy

These experiments test whether the locally derived weighting rule improves distillation when token positions are coupled through shared model parameters. Utility weighting increased macro and pooled mean accuracy over matched uniform OPD in all six evaluated combinations of model pair and objective. The macro gains below are percentage points. Macro gives each of the seven benchmarks equal weight, and pooled gives each problem equal weight.

Student ← teacherReverse-KL macro gainForward-KL macro gain
Qwen3-1.7B ← 8B+4.00+2.71
Qwen3-1.7B ← 14B+3.10+1.08
Qwen3-4B ← 30B-A3B+1.35+1.16

FUW is competitive with existing weighting and selection methods, although the reported comparisons do not establish consistent superiority over them. Our common response below addresses the analysis of the local utility to the realized loss reduction, comparison with alternative position-scoring methods for OPD, and variability in training runs.

Analysis of local utility to realized loss reduction

We trained Qwen3-1.7B-Base from Qwen3-8B with uniform reverse-KL OPD. At step 20 we collected two responses for each of 64 prompts, covering about 235,000 positions. We fixed each prefix and its candidate set. At the starting checkpoint we measured Fisher utility, squared Euclidean logit-gradient norm, and the restricted reverse-KL loss. We then measured the realized loss decrease at those positions after one, two, and five further optimizer updates.

Updates after step 20FisherEuclideanReverse KL
10.500.470.41
20.500.460.42
50.500.470.43

Spearman rank correlation between each starting score and the subsequent fixed-prefix loss decrease.

Fisher utility is a moderate but consistent predictor of realized loss reduction under shared-parameter updates, with the strongest correlation among the three scores at every measured horizon. Its rank-correlation advantage is 0.034–0.035 over Euclidean (95% CIs within [0.029, 0.040]) and 0.074–0.090 over KL (95% CIs within [0.059, 0.104]), based on 200 bootstrap resamples over responses. Unlike the local derivation, which considers an optimally directed infinitesimal change at one prefix, these measurements reflect shared-parameter updates.

Revision. We will add the shared-parameter realized-loss analysis to the appendix.

AuthorsOpenReview ↗

General Response (2/2)

General response (2/2)

Comparison with other scoring methods

Several weighting criteria achieve similar downstream performance. FUW scores 39.2 pooled versus 38.9 for Euclidean gradient-norm weighting (p=0.53p = 0.53; Table 6). In an additional experiment, square-root KL weighting achieves 38.5 pooled and 26.2 macro, compared with 39.2 pooled and 26.8 macro for FUW. These similarities are partly explained by the framework: Fisher utility is the student-weighted variance of the student–teacher log-ratio rr, VarpS[r]\mathrm{Var}_{p_S}[r]; the squared Euclidean gradient norm uses the same centered deviations with different probability weighting; and reverse KL is the mean of rr. These scores are related by construction but not equivalent.

The containment analysis shows substantial agreement in practice. At the 50% operating fraction, gradient magnitude, KL disagreement, student entropy, TIP Soft-OR and TIDE-deficit show near-complete bidirectional weight-mass containment with FUW (B0.5≥0.985B_{0.5} \ge 0.985 from step 61 under uniform OPD on 1.7B ← 14B; Section 5.2), and the same structure appears on 1.7B ← 8B along the FUW trajectory (Appendix B.2). While the criteria overlap in their effective support, they differ in weight allocation within the shared support (Table 5) and at stricter selection fractions (ρ=0.2\rho = 0.2). These differences are not resolved by downstream accuracy at the current training budget.

We therefore present FUW as an objective-derived weighting criterion that is competitive with existing methods, rather than consistently superior to them. The framework helps explain their overlapping emphasis and yields corresponding utilities for other distillation objectives without separate heuristics.

Variation across training seeds

To assess training variability, we repeated the main reverse-KL configuration (Qwen3-1.7B-Base ← Qwen3-8B) using three matched rollout seeds for uniform OPD and FUW. All methods are evaluated at the final checkpoint (step 279), with no checkpoint selection. The original paper's bootstrap intervals quantify uncertainty over evaluation problems rather than training runs.

MethodSeed 0 (paper)Seed 1Seed 2Mean ± SD
Uniform OPD, macro22.821.921.722.1 ± 0.6
FUW, macro26.825.725.726.1 ± 0.6
Uniform OPD, pooled34.534.334.134.3 ± 0.2
FUW, pooled39.239.439.139.2 ± 0.15

Macro and pooled Avg@8 accuracy (%); SD across three training seeds.

FUW outperforms uniform OPD across all three matched seeds, with paired gains of +4.0, +3.8, and +4.0 macro points and +4.7, +5.1, and +5.0 pooled points. The mean improvements are +3.9 macro and +4.9 pooled points. These results show that the improvement over uniform OPD persists across the tested seeds, although other model pairs and objectives remain single-run comparisons.

Revision. We will add the matched-seed results and clarify the distinction between evaluation-problem uncertainty and training-run variability in the Reproducibility Statement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.