- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a combined approach to address an important question of in an on policy distillation regime, which supervision is useful for a student at its current training stage. More specifically, it leverages two main components i.e. group alignment and token calibration to achieve this. The main gains are supported via evaluation across three training seeds , while single seed ablation studies analyze the effect of each component and their combination.
Critical strengths
1- The paper addresses an important question of which supervision is useful for a student at its current training state. 2- Useful ablations have been provided to better understand the role of different components i.e. group alignment and token calibration
Critical weaknesses
1- First of all, the paper in its current shape is really hard to follow and transition from the diagnosis to the method section requires substantial improvement. Better clarifying the problem. intuition and assumptions before explaining the components and formulation make the paper easier to follow. 2-My main concern is that the paper has not established significant technical contribution over existing approaches and is more of an effective combination of existing approaches. 3-There are missing comparisons with the closely related works such as reference gradient reweighing family of methods which may help to better justify the novelty of the proposed method.
Questions for authors
-
Improve the transition from the diagnosis to the method section Section 2 shows that the benefit of fixed supervision changes with model capacity and training state. However, it is not clear how these observations lead to Attune's specific design. The diagnosis uses SFT on fixed arithmetic reasoning trajectories, while Attune uses token probability gaps and controller-gradient alignment for on-policy distillation. Why should the token gate in eq. (2) and the alignment score in eq. (3) capture the changes observed in the diagnosis? Explaining the problem, intuition, and assumptions behind each component before introducing the formulation and methodology would make the paper easier to follow.
-
Expand experiments to show whether the controlled findings extend beyond arithmetic and Qwen. The main paired-intervention study evaluates four Qwen2.5-Instruct sizes on a 500 example arithmetic probe. Although the appendix adds three domain experiments and Attuneis evaluated on both Qwen and Llama, the motivating study that is the main motivation for the proposed method is not as broad as the overall evaluation. So, it would be useful to repeat the controlled interventions on another model family and a natural reasoning task. Evaluating the full 5,000-example held-out probe would also help check the stability of the patterns. Please clearly differentiate between observations specific to this setup and conclusions that apply more generally.
-
Explain the controller choices and test their sensitivity. The controller construction has been well-described, but it is not clear why you use a fixed pool of 512 prompts and a minibatch of 32? Since its gradient determines which training groups receive more weight, the choice of controller prompts could be significantly important. I would like to see how the results change with the pool size, minibatch size, source mixture, and random choice of prompts. A few of such experiments would establish a better understanding of the stability of the gains across reasonable controller configs.
-
Justify the choice of gradient-probe layers. It is not clear why you compare gradients only in the LM head and final two Transformer blocks? Table 3 supports combining the head and top-block, but does not explain why these two blocks were chosen. Since the final update trains all student parameters, other layers may provide useful information about update quality. Comparing a few layer subsets and, in a smaller setting, full-model alignment would help justify the choice. Please report both the quality of the allocation scores and the computational cost.
-
Include a more closely relevant allocation baseline. The paper covers a useful range of on-policy distillation methods, but the closest alternatives also include reference-gradient reweighting and learner-adaptive allocation. An adapted baseline based on ideas from Ren et al., LESS, or GradAlign would help show what Attune adds over existing related work. Since some of these methods are directly applicable, the comparison should preserve the same distillation objectives, with matched compute if possible. If a meaningful adaptation is not feasible, please explain the difference in setting. DMC and PACED are also relevant to the broader comparison with student-adaptive distillation.
Related references
a- Ren et al. Learning to Reweight Examples for Robust Deep Learning. ICML 2018. b-Xia et al. LESS: Selecting Influential Data for Targeted Instruction Tuning. ICML 2024. c-Yang et al. GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning. 2026. d-Huang et al. Tailoring the Curriculum: Student-Centered Reasoning Distillation via Dynamic Data-Model Compatibility. 2026. e-Xu et al. PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student Competence. 2026.
-
Separate local loss improvement from capability improvement Table 4 measures one-step reductions in unweighted Generalized Jensen-Shannon divergence loss. This is a useful evidence of improved local teacher-student agreement, but it does not directly imply better reasoning or instruction following. Moreover, it is not clear whether the allocation scores predict capability gains beyond a single update. Testing longer update horizons and generation-based outcomes would strengthen that paper. Reporting the starting probe loss and relative reduction would also make the interpretation of the size of the local gains much easier.
-
State the methodological novelty more specifically Reference-gradient reweighting is already established. Applying token calibration to both candidate and controller gradients appears to be Attune's key contribution, and table 3 provides useful support for that. However, it is not clear how this differs from prior methods and what the rank-fusion step adds. The closer baseline reported above would help understand whether the contribution is a new allocation mechanism or an effective combination of existing ideas for on-policy distillation.
-
Establish the added benefit of the coupled design across seeds. Table 3 suggests that group alignment accounts for much of the aggregate improvement i.e. TaskAvg rises from 42.24 for GKD to 43.45 with group alignment alone, compared with 44.30 for Full Attune. Adding token calibration still improves TaskAvg by 0.85 and CatAvg by 0.76 but does not help every task. On TruthfulQA-MC2, group alignment alone scores 50.94 vs 50.60 for Full Attune (probably wrongly bolded in the table). Since these components comparisons use only seed 10, please repeat the ablations across training seeds and report uncertainty for the paired gains. This would help establish whether the coupled design reliably improves over group alignment alone. Also, please clarify whether the bold Full Attune row highlights the method or the best result in every column.
12 comments
We sincerely thank Reviewer xUjk for the insightful comments on establishing multi-seed support for our coupled design and clarifying both our methodological novelty and Table 3's formatting. Following your suggestions, we repeated all Table 3 ablations across three training seeds and added a Controller-only calibration variant to complete the full factorial comparison.
Part 2: Multi-Seed Verification of the Coupled Design and Methodological Novelty — Addressing W2, Q7 & Q8
To test whether our coupled design reliably improves over Group-only and partial calibration variants, we evaluated all Table 3 configurations across Seeds 10, 20, and 30 in Qwen2.5 7B-to-1.5B distillation. Each per-seed column below reports TaskAvg / CatAvg.
| Setting | Seed 10 | Seed 20 | Seed 30 | TaskAvg | dTask | CatAvg | dCat |
|---|---|---|---|---|---|---|---|
| Full Attune | 44.30 / 41.81 | 44.02 / 41.54 | 44.18 / 41.75 | 44.17±0.14 | - | 41.70±0.14 | - |
| Controller-only | 43.85 / 41.23 | 43.67 / 41.07 | 43.76 / 41.15 | 43.76±0.09 | -0.41 | 41.15±0.08 | -0.55 |
| Candidate-only | 43.23 / 40.84 | 43.12 / 40.69 | 43.46 / 41.05 | 43.27±0.17 | -0.90 | 40.86±0.18 | -0.84 |
| Uncalibrated | 43.01 / 40.64 | 42.65 / 40.32 | 43.10 / 40.72 | 42.92±0.24 | -1.25 | 40.56±0.21 | -1.14 |
| Group-only | 43.45 / 41.05 | 43.36 / 40.98 | 43.60 / 41.21 | 43.47±0.12 | -0.70 | 41.08±0.12 | -0.62 |
| Token-only | 42.68 / 40.28 | 42.33 / 40.03 | 42.58 / 40.32 | 42.53±0.18 | -1.64 | 40.21±0.16 | -1.49 |
| Vanilla GKD | 42.24 / 39.81 | 41.91 / 39.49 | 42.23 / 39.89 | 42.13±0.19 | -2.04 | 39.73±0.21 | -1.97 |
| Ref: TA-OPD | 42.68 / 40.37 | 42.61 / 40.31 | 42.58 / 40.35 | 42.62±0.05 | -1.54 | 40.34±0.03 | -1.35 |
- Reliable multi-seed gain over group alignment alone and Table 3 formatting: Across all three seeds, Full Attune consistently improves over Group-only with low variance, confirming that the coupled design provides a stable advantage beyond group alignment alone. Regarding Table 3's formatting, the original table bolded the Full Attune row to highlight the proposed method; we will revise it to bold strictly the column-best values so that TruthfulQA-MC2 is unambiguous.
- Why Attune is not a naive combination of existing components: A straightforward uncoupled combination of group alignment and token gating corresponds to Uncalibrated alignment, which applies the token gate to the update loss while scoring uncalibrated gradients. Across all three seeds, this naive combination degrades performance below Group-only and doubles variance, because candidate groups are selected on large-gap outlier tokens that are subsequently downweighted during the actual update.
- Super-additive synergy of coupled calibration: Relative to Vanilla GKD, the three-seed gain of Full Attune exceeds the linear sum of Token-only and Group-only. Calibrating both controller and candidate scoring gradients aligns the selection geometry with the token-gated update step, nearly doubling the marginal benefit of token calibration compared to using it in isolation.
We sincerely thank Reviewer xUjk for highlighting the need for a clearer transition from our Section 2 diagnosis to the Section 3 methodology. We outline below the problem intuition, underlying assumptions, and analytical formulation connecting our diagnostic observations to Equations (2)–(5).
Part 3: Problem Intuition and Analytical Link from Diagnosis to Method — Addressing W1 & Q1
Our Section 2 diagnosis (and Appendices A.2–A.4) establishes two empirical properties of supervision utility as a student evolves: (i) Local fit vs. global generation divergence: forcing a capacity-constrained student to fit complex trajectories can improve intermediate-step accuracy or teacher-forced token NLL while degrading end-to-end free-generation accuracy (Section 2.4; Table 11); and (ii) State-conditioned utility: the return of fixed supervision varies across student capacities and training states (Figure 1), and cannot be reliably predicted by raw token-level teacher–student disagreement alone (Table 4A, where candidate JSD and positive CE gap achieve Spearman of only 0.14 and 0.19). To operationalize these findings in on-policy distillation without costly counterfactual branch rollouts at every step, Attune couples two state-dependent mechanisms around the current student :
- Why the token gate in Eq. (2) provides state-adaptive local pacing: In on-policy rollouts , tokens with a large positive log-probability gap mark local positions where the teacher strongly favors a transition that the current student state assigns low probability. Motivated by Observation (i), Eq. (2) uses as a student-conditioned discrepancy signal, mapping to bounded within-group weights . Rather than relying on static difficulty labels, softly tempers the gradient influence of large positive-gap tokens early on and automatically relaxes toward
1as masters those transitions (), while retaining full corrective weight whenever the student is over-confident (). - Why controller alignment in Eq. (3) serves as a local proxy for state-dependent utility: Motivated by Observation (ii), we need an online surrogate to evaluate which candidate groups () benefit the student at its current state . Under a first-order Taylor approximation around with fixed trajectories and detached token weights (left equation above), the local reduction in the calibrated controller loss induced by candidate update is governed by the inner product . Normalizing by subspace norms within the LM head and top Transformer blocks (Eq. 3) and fusing percentile ranks (Eq. 4) provides a scale-invariant proxy for state-conditioned utility, refreshed on current student rollouts at every step.
- Analytical decomposition of why Eq. (2) and Eq. (3) should be coupled: Decomposing any uncalibrated gradient into its calibrated counterpart plus the residual from token reweighting, where , reveals why scoring uncalibrated gradients conflicts with the calibrated parameter update in Eq. (5) (right equation above). Because Eq. (5) updates along , scoring groups via introduces three discrepancy terms driven by and —allowing tokens that are downweighted during the actual update to distort the allocation scores.
- Direct consistency with our three-seed ablation in Part 2: This decomposition directly accounts for the empirical ordering reported in Part 2: leaving both sides uncalibrated retains all three discrepancy terms and degrades TaskAvg to
42.92 ± 0.24(below even uniformGroup-onlyat43.47 ± 0.12); calibrating only the candidate side (,43.27 ± 0.17) or only the shared controller side (,43.76 ± 0.09) removes a subset of these mismatch terms; and calibrating both sides in Full Attune (,44.17 ± 0.14) aligns the scoring geometry with the actual parameter update.
We sincerely thank Reviewer xUjk for your insightful feedback and for recognizing the importance of studying state-dependent supervision utility and the value of our component ablations. As you rightly noted, controller-aligned group allocation drives a major share of Attune's empirical gains and serves as the core mechanism of our framework. Because establishing the stability and boundary conditions of the controller is central to our claims, we prioritized this comprehensive investigation first and present the results below as Part 1 of our response.
Part 1: Controller Design Choices and Sensitivity Analysis — Addressing Q3
Keeping the 96,000-prompt training stream fixed in Qwen2.5 7B-to-1.5B distillation, we evaluated 10 additional configurations varying the per-step controller minibatch from 8 to 64 with the pool size fixed at 512, varying the reference pool size from 128 to 1024 with the per-step minibatch fixed at 32, excluding individual capability domains from the 512-prompt pool, and folding the cyclic controller trajectories directly into the GKD loss instead of using them for group allocation.