- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies catastrophic forgetting during continual adaptation with Self-Distillation Fine-Tuning (SDFT). The authors first bound old-task drift by the expected divergence between the target distribution and the base policy, and show empirically that although SDFT reduces average target divergence relative to SFT, its per-sample divergence remains heavy-tailed across models and tasks. A diagnostic intervention that down-weights high-KL samples reduces forgetting on IFEval without hurting new-task accuracy. Motivated by this observation, the authors propose Trust-Region Constrained Self-Distillation Fine-Tuning (TRSD), which projects the demonstration-conditioned teacher onto an f-divergence ball around the current student policy. Instantiating TRSD with forward KL, reverse KL, and squared Hellinger distance yields closed-form geometric, arithmetic, and square-root teacher interpolations governed by a sample-wise coefficient calibrated from local curvature. Experiments on Qwen2.5-7B-Instruct and InternLM2.5-7B-Chat across Tool Use, Science Q&A, and Medical tasks in both single-task and sequential settings show consistent improvements in IFEval retention and task accuracy over matched SDFT baselines.
Critical strengths
- The empirical diagnosis of SDFT is well-executed. By inspecting per-sample KL distributions across two model families and three domains (Figure 1 and Figure 5), and pairing this with the batch quantile down-weighting intervention (Figure 2), the paper establishes a clear link between the right tail of teacher-student divergence and instruction-following degradation.
- The f-divergence trust-region formulation is computationally practical. Deriving the closed-form interpolation paths and local curvatures for forward KL, reverse KL, and squared Hellinger (Table 1) allows TRSD to adjust the teacher distribution directly in logit or probability space without requiring extra model forward passes, replay buffers, or separate teacher networks.
- The empirical evaluation covers both single-task and three-stage sequential continual learning across three divergence geometries. In addition, the qualitative analysis of sequential failure modes in Appendix E.7.2 usefully separates format-wrapper collapse from genuine scientific reasoning errors after Medical adaptation.
Critical weaknesses
- There is a noticeable gap between the theoretical trust-region problem and the implemented online calibration rule. Equations (4) and (8) define a token-level constraint with an absolute budget C, and Lemma 3.1 bounds forgetting relative to the fixed initialization theta_0 under a local target-matching assumption. In practice, Algorithm 1 uses an evolving EMA teacher, averages local curvature across all tokens in a completion, replaces C with a batch-relative budget proportional to the batch mean curvature (Equation 37), and clips beta_i to [beta_min, beta_max]. Because the peer-relative budget scales with the batch mean curvature and beta_i is lower-bounded by beta_min, the actual target divergence is not strictly bounded when an entire batch shifts or when extreme divergences are localized to a few tokens in a long sequence.
- The paper does not directly compare TRSD against the simpler high-KL down-weighting prototype (DH-KL) or against a fixed global interpolation coefficient (constant beta < 1). Because beta_max is often set to 0.75, 0.80, or 0.85, TRSD applies uniform shrinkage even to low-curvature samples. Without a direct comparison against DH-KL and a constant-beta baseline in Table 2, it is difficult to tell how much of the gain comes from the smooth f-divergence interpolation path and curvature-based sample adaptation versus simple global dampening of the teacher.
- The hyperparameter selection protocol for the main results is not clearly specified. TRSD introduces three calibration hyperparameters (rho, beta_min, beta_max) whose selected values in Tables 5 and 11 vary considerably across tasks, models, and geometries. While the 311-run sweep in Figure 4 and Tables 18 to 26 demonstrates that most configurations improve a combined score over SDFT, individual settings still exhibit nontrivial variance in task accuracy (for example, up to an 8.25-point drop on Tool Use in Table 18). It is unclear how the single reported configuration for each task in Table 2 and Figure 3 was selected without using the test benchmarks.
Questions for authors
- How does TRSD compare directly against the DH-KL baseline from Section 3.3 and against a constant interpolation baseline (fixing beta_i = beta_0 across all samples without local curvature scaling) under the exact evaluation setup of Table 2?
- What validation criterion and data split were used to select the specific (rho, beta_min, beta_max), epoch, and batch-size configurations reported in Table 2 and the sequential trajectories in Figure 3? If a single default hyperparameter tuple (e.g., rho = 0.10, [beta_min, beta_max] = [0.05, 0.85]) is fixed across all tasks, models, and stages, how much of the reported gain is retained?
- Since token-level teacher-student divergence within a completion is itself often concentrated on a small subset of decision tokens, why does TRSD average local curvature s_f over the entire sequence rather than calibrating beta_{i,t} at the token level as derived in Equations (5) and (9)? Did you observe instability with token-wise beta_{i,t}?
6 comments
Response to Reviewer LBeZ (2/4): DH-KL Comparison
W2–Q1-DH-KL Comparison: How does TRSD compare with DH-KL and constant- baselines under the exact Table 2 setup?
We thank the reviewer for requesting these direct comparisons and for raising the possibility that TRSD’s gains may primarily result from global teacher dampening. To separate this effect from curvature-based sample adaptation, we evaluated DH-KL using the configurations from Section 3.3 across all nine Qwen2.5-7B-Instruct task–divergence settings in Table 2. We also trained a constant- control for each setting, fixing the coefficient for all samples to the mean realized over the corresponding TRSD training trajectory.
We first compare DH-KL with TRSD:
Forward KL
| Task | DH-KL Acc / IFEval | TRSD Acc / IFEval | Acc | IFEval | |
|---|---|---|---|---|---|
| Tool Use | 71.13 / 49.63 | 65.98 / 57.81 | −5.15 | +8.18 | +1.52 |
| Science Q&A | 70.41 / 44.67 | 68.24 / 53.06 | −2.17 | +8.39 | +3.11 |
| Medical | 41.70 / 58.44 | 40.90 / 60.91 | −0.80 | +2.47 | +0.84 |
Reverse KL
| Task | DH-KL Acc / IFEval | TRSD Acc / IFEval | Acc | IFEval | |
|---|---|---|---|---|---|
| Tool Use | 56.70 / 47.74 | 67.01 / 55.62 | +10.31 | +7.88 | +9.10 |
| Science Q&A | 66.07 / 50.07 | 67.65 / 55.88 | +1.58 | +5.81 | +3.70 |
| Medical | 34.20 / 52.83 | 42.30 / 54.73 | +8.10 | +1.90 | +5.00 |
Squared Hellinger
| Task | DH-KL Acc / IFEval | TRSD Acc / IFEval | Acc | IFEval | |
|---|---|---|---|---|---|
| Tool Use | 65.98 / 49.01 | 72.16 / 58.08 | +6.18 | +9.07 | +7.63 |
| Science Q&A | 63.51 / 50.16 | 68.05 / 53.40 | +4.54 | +3.24 | +3.89 |
| Medical | 39.70 / 52.30 | 40.20 / 62.77 | +0.50 | +10.47 | +5.49 |
All values are computed as TRSD minus DH-KL. Here, uses the same retention-aligned joint score, , as in our robustness analysis. Task accuracy and IFEval are reported separately, while summarizes their trade-off. As shown in the results, TRSD achieves higher retention in each case, and consistently achieves higher joint score than DH-KL across all nine settings.
Response to Reviewer LBeZ (3/4): Constant-Beta Controls
W2–Q1-Constant-Beta Controls: How does TRSD compare with DH-KL and constant- baselines under the exact Table 2 setup?
We next address the reviewer’s concern about uniform shrinkage from . We agree that clipping can shrink the teacher even for low-curvature samples. To test whether this global dampening alone explains the gains, we remove local curvature scaling and apply one constant coefficient to every sample. For each setting, this coefficient equals the mean realized across samples and updates during the corresponding TRSD trajectory.
| Geometry | Task | Constant | Constant- Acc / IFEval | TRSD Acc / IFEval | Acc | IFEval |
|---|---|---|---|---|---|---|
| Forward KL | Tool Use | 0.2810 | 62.89 / 44.97 | 65.98 / 57.81 | +3.09 | +12.84 |
| Forward KL | Science Q&A | 0.3348 | 65.09 / 52.16 | 68.24 / 53.06 | +3.15 | +0.90 |
| Forward KL | Medical | 0.3267 | 39.10 / 51.21 | 40.90 / 60.91 | +1.80 | +9.70 |
| Reverse KL | Tool Use | 0.2663 | 64.95 / 48.38 | 67.01 / 55.62 | +2.06 | +7.24 |
| Reverse KL | Science Q&A | 0.4665 | 66.67 / 53.45 | 67.65 / 55.88 | +0.98 | +2.43 |
| Reverse KL | Medical | 0.2285 | 39.70 / 52.68 | 42.30 / 54.73 | +2.60 | +2.05 |
| Squared Hellinger | Tool Use | 0.4571 | 63.92 / 52.01 | 72.16 / 58.08 | +8.24 | +6.07 |
| Squared Hellinger | Science Q&A | 0.5267 | 62.72 / 52.33 | 68.05 / 53.40 | +5.33 | +1.07 |
| Squared Hellinger | Medical | 0.3245 | 38.70 / 59.83 | 40.20 / 62.77 | +1.50 | +2.94 |
TRSD outperforms the matched constant- control on IFEval in all nine settings, while learning comparable or higher task accuracy. The retention differences are larger on Tool Use and Medical, while the three Science Q&A settings show smaller but consistently positive improvements. Because the control preserves the same interpolation geometry and matches the average interpolation strength but removes local curvature scaling, these results indicate that simple global dampening does not explain the gains.
Because the matched constants can only be obtained after observing the complete TRSD trajectory, we additionally sweep constant under forward KL. Here, is the original SDFT baseline.
| Task | Metric | TRSD | |||||
|---|---|---|---|---|---|---|---|
| Tool Use | Acc. | 65.98 | 63.92 | 63.92 | 70.10 | 67.01 | 63.92 |
| IFEval | 57.81 | 44.14 | 41.75 | 42.81 | 43.80 | 50.40 | |
| Science Q&A | Acc. | 68.24 | 65.88 | 67.85 | 70.41 | 66.27 | 65.29 |
| IFEval | 53.06 | 45.60 | 46.96 | 44.35 | 52.27 | 54.27 | |
| Medical | Acc. | 40.90 | 40.20 | 39.70 | 40.90 | 38.20 | 38.00 |
| IFEval | 60.91 | 48.56 | 55.37 | 55.95 | 53.44 | 53.80 |
TRSD generally achieves higher IFEval in constant- comparisons. These comparisons are now reported in Section 5.3, with the pre-specified constant- sweep in Appendix E.12.
Response to Reviewer LBeZ (1/4)
We thank the reviewer for recognizing the empirical diagnosis of SDFT’s heavy-tailed teacher–student divergence, the computational practicality of our f-divergence formulation, and the breadth of our single-task, sequential, and qualitative evaluations. Below, we respond to your questions and concerns in detail.
We thank the reviewer for the question. The proximal teachers used in our final training objectives—Eq. (7) for forward KL and Eq. (11) for general f-divergences—are derived through Lagrangian and KKT analysis from the idealized token-level absolute-budget problems in Eqs. (4) and (8), respectively (the derivation in Appendix D.1-D.4).
During online training, however, the scale of an appropriate absolute budget varies across tasks, divergence geometries, and adaptation stages. We therefore use the peer-relative budget in Section 4.3 and Appendix D.5 as an adaptive online alternative. The reason why we average curvature over the completion is that token-level estimates are noisier, and varying β across semantically coupled tokens might make teacher corrections inconsistent within a response. The completion is therefore treated as the practical calibration unit.
Clipping is a practical regularization of the online calibration. It limits extreme interpolation weights and preserves a minimum teacher signal, but relaxes strict per-sample budget satisfaction.
For the reason that we use EMA reference, we kindly refer the reviewer to Appendix E.2, Training Procedure. We use it because Shenfeld et al. (2026) show that it tracks learning progress while stabilizing on-policy training, unlike frozen-base and current-student teachers. Our complete sweeps show that the practical implementation consistently follows the intended acquisition–retention direction. This distinction is now clarified in Section 4.3, with empirical characterization in Appendix D.6.
We kindly refer the reviewer to Appendix D.1. Equation (25) implicitly characterizes the exact β-parameterized teacher path as
q^*_{f,β}(v) = G^{-1}_{f,β,v}(−η), with the normalization conditionΣ_v q^*_{f,β}(v) = 1.Equation (26), together with β=1/(1+λ), applies complementary slackness: λ[Df(qf,β⋆∥pθt)−C]=0.
Thus, β=1 when the original teacher is feasible; otherwise, the constraint is active and β is the largest point on the path satisfying the budget.
For each instantiated divergence, the exact β is determined by solving the boundary equation Df(qβ∥p)=C. This generally requires per-token numerical root finding, so we use the second-order expansion in Eq. (27) to obtain the explicit online calibration in Eq. (10).