acceptodds
Under review as a conference paper at ICLR 2027

Trust-region Constraint Improves Continual Learning of Self-distillation Fine-tuning

Abstract

Continually adapting large language models to new domains risks erasing capabilities acquired during pretraining. Self-Distillation Fine-Tuning (SDFT) mitigates this problem by using a demonstration-conditioned version of the model as an on-policy teacher. We show, however, that although this teacher is substantially closer to the student than one-hot supervision on average, its teacher–policy divergence remains strongly heavy-tailed across model families and adaptation domains. A controlled down-weighting intervention further shows that high-divergence teachers contribute disproportionately to forgetting. Motivated by this finding, we introduce Trust-Region Constrained Self-Distillation Fine-Tuning (TRSD), which constructs a sample-wise trust-region-constrained teacher distribution that remains faithful to the demonstration-conditioned teacher while limiting its divergence from the current student. Formulated over the -divergence family, TRSD induces distinct teacher-distribution geometries for forward KL, reverse KL, and squared Hellinger distance without requiring an additional model forward pass beyond SDFT. Across tool use, scientific question answering, and medical reasoning, TRSD improves instruction-following over SDFT, generally without requiring weaker task accuracy. These gains persist under sequential adaptation and broad hyperparameter sweeps. Our results establish teacher calibration as an important principle for continual self-distillation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Self-distillation, Fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LBeZOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies catastrophic forgetting during continual adaptation with Self-Distillation Fine-Tuning (SDFT). The authors first bound old-task drift by the expected divergence between the target distribution and the base policy, and show empirically that although SDFT reduces average target divergence relative to SFT, its per-sample divergence remains heavy-tailed across models and tasks. A diagnostic intervention that down-weights high-KL samples reduces forgetting on IFEval without hurting new-task accuracy. Motivated by this observation, the authors propose Trust-Region Constrained Self-Distillation Fine-Tuning (TRSD), which projects the demonstration-conditioned teacher onto an f-divergence ball around the current student policy. Instantiating TRSD with forward KL, reverse KL, and squared Hellinger distance yields closed-form geometric, arithmetic, and square-root teacher interpolations governed by a sample-wise coefficient calibrated from local curvature. Experiments on Qwen2.5-7B-Instruct and InternLM2.5-7B-Chat across Tool Use, Science Q&A, and Medical tasks in both single-task and sequential settings show consistent improvements in IFEval retention and task accuracy over matched SDFT baselines.

Critical strengths
  1. The empirical diagnosis of SDFT is well-executed. By inspecting per-sample KL distributions across two model families and three domains (Figure 1 and Figure 5), and pairing this with the batch quantile down-weighting intervention (Figure 2), the paper establishes a clear link between the right tail of teacher-student divergence and instruction-following degradation.
  2. The f-divergence trust-region formulation is computationally practical. Deriving the closed-form interpolation paths and local curvatures for forward KL, reverse KL, and squared Hellinger (Table 1) allows TRSD to adjust the teacher distribution directly in logit or probability space without requiring extra model forward passes, replay buffers, or separate teacher networks.
  3. The empirical evaluation covers both single-task and three-stage sequential continual learning across three divergence geometries. In addition, the qualitative analysis of sequential failure modes in Appendix E.7.2 usefully separates format-wrapper collapse from genuine scientific reasoning errors after Medical adaptation.
Critical weaknesses
  1. There is a noticeable gap between the theoretical trust-region problem and the implemented online calibration rule. Equations (4) and (8) define a token-level constraint with an absolute budget C, and Lemma 3.1 bounds forgetting relative to the fixed initialization theta_0 under a local target-matching assumption. In practice, Algorithm 1 uses an evolving EMA teacher, averages local curvature across all tokens in a completion, replaces C with a batch-relative budget proportional to the batch mean curvature (Equation 37), and clips beta_i to [beta_min, beta_max]. Because the peer-relative budget scales with the batch mean curvature and beta_i is lower-bounded by beta_min, the actual target divergence is not strictly bounded when an entire batch shifts or when extreme divergences are localized to a few tokens in a long sequence.
  2. The paper does not directly compare TRSD against the simpler high-KL down-weighting prototype (DH-KL) or against a fixed global interpolation coefficient (constant beta < 1). Because beta_max is often set to 0.75, 0.80, or 0.85, TRSD applies uniform shrinkage even to low-curvature samples. Without a direct comparison against DH-KL and a constant-beta baseline in Table 2, it is difficult to tell how much of the gain comes from the smooth f-divergence interpolation path and curvature-based sample adaptation versus simple global dampening of the teacher.
  3. The hyperparameter selection protocol for the main results is not clearly specified. TRSD introduces three calibration hyperparameters (rho, beta_min, beta_max) whose selected values in Tables 5 and 11 vary considerably across tasks, models, and geometries. While the 311-run sweep in Figure 4 and Tables 18 to 26 demonstrates that most configurations improve a combined score over SDFT, individual settings still exhibit nontrivial variance in task accuracy (for example, up to an 8.25-point drop on Tool Use in Table 18). It is unclear how the single reported configuration for each task in Table 2 and Figure 3 was selected without using the test benchmarks.
Questions for authors
  1. How does TRSD compare directly against the DH-KL baseline from Section 3.3 and against a constant interpolation baseline (fixing beta_i = beta_0 across all samples without local curvature scaling) under the exact evaluation setup of Table 2?
  2. What validation criterion and data split were used to select the specific (rho, beta_min, beta_max), epoch, and batch-size configurations reported in Table 2 and the sequential trajectories in Figure 3? If a single default hyperparameter tuple (e.g., rho = 0.10, [beta_min, beta_max] = [0.05, 0.85]) is fixed across all tasks, models, and stages, how much of the reported gain is retained?
  3. Since token-level teacher-student divergence within a completion is itself often concentrated on a small subset of decision tokens, why does TRSD average local curvature s_f over the entire sequence rather than calibrating beta_{i,t} at the token level as derived in Equations (5) and (9)? Did you observe instability with token-wise beta_{i,t}?
6 comments
AuthorsOpenReview ↗

Response to Reviewer LBeZ (1/4)

We thank the reviewer for recognizing the empirical diagnosis of SDFT’s heavy-tailed teacher–student divergence, the computational practicality of our f-divergence formulation, and the breadth of our single-task, sequential, and qualitative evaluations. Below, we respond to your questions and concerns in detail.

W1-1: How does the practical implementation differ from the theoretical trust-region formulation?

We thank the reviewer for the question. The proximal teachers used in our final training objectives—Eq. (7) for forward KL and Eq. (11) for general f-divergences—are derived through Lagrangian and KKT analysis from the idealized token-level absolute-budget problems in Eqs. (4) and (8), respectively (the derivation in Appendix D.1-D.4).

During online training, however, the scale of an appropriate absolute budget varies across tasks, divergence geometries, and adaptation stages. We therefore use the peer-relative budget in Section 4.3 and Appendix D.5 as an adaptive online alternative. The reason why we average curvature over the completion is that token-level estimates are noisier, and varying β across semantically coupled tokens might make teacher corrections inconsistent within a response. The completion is therefore treated as the practical calibration unit.

Clipping is a practical regularization of the online calibration. It limits extreme interpolation weights and preserves a minimum teacher signal, but relaxes strict per-sample budget satisfaction.

For the reason that we use EMA reference, we kindly refer the reviewer to Appendix E.2, Training Procedure. We use it because Shenfeld et al. (2026) show that it tracks learning progress while stabilizing on-policy training, unlike frozen-base and current-student teachers. Our complete sweeps show that the practical implementation consistently follows the intended acquisition–retention direction. This distinction is now clarified in Section 4.3, with empirical characterization in Appendix D.6.

W1-2: Why do we use a local quadratic approximation to calibrate β?

We kindly refer the reviewer to Appendix D.1. Equation (25) implicitly characterizes the exact β\beta-parameterized teacher path as q^*_{f,β}(v) = G^{-1}_{f,β,v}(−η), with the normalization condition Σ_v q^*_{f,β}(v) = 1.

Equation (26), together with β=1/(1+λ)\beta=1/(1+\lambda), applies complementary slackness: λ[Df(qf,β⋆∥pθt)−C]=0\lambda\left[\mathcal D_f\left(q^\star_{f,\beta}\Vert p_\theta^t\right)-C\right]=0.

Thus, β=1\beta=1 when the original teacher is feasible; otherwise, the constraint is active and β is the largest point on the path satisfying the budget.

For each instantiated divergence, the exact β is determined by solving the boundary equation Df(qβ∥p)=C\mathcal D_f(q_\beta\Vert p)=C. This generally requires per-token numerical root finding, so we use the second-order expansion in Eq. (27) to obtain the explicit online calibration in Eq. (10).

AuthorsOpenReview ↗

Response to Reviewer LBeZ (2/4): DH-KL Comparison

W2–Q1-DH-KL Comparison: How does TRSD compare with DH-KL and constant-β\beta baselines under the exact Table 2 setup?

We thank the reviewer for requesting these direct comparisons and for raising the possibility that TRSD’s gains may primarily result from global teacher dampening. To separate this effect from curvature-based sample adaptation, we evaluated DH-KL using the configurations from Section 3.3 across all nine Qwen2.5-7B-Instruct task–divergence settings in Table 2. We also trained a constant-β\beta control for each setting, fixing the coefficient for all samples to the mean βi\beta_i realized over the corresponding TRSD training trajectory.

We first compare DH-KL with TRSD:

Forward KL

TaskDH-KL Acc / IFEvalTRSD Acc / IFEvalΔ\Delta AccΔ\Delta IFEvalΔJ\Delta J
Tool Use71.13 / 49.6365.98 / 57.81−5.15+8.18+1.52
Science Q&A70.41 / 44.6768.24 / 53.06−2.17+8.39+3.11
Medical41.70 / 58.4440.90 / 60.91−0.80+2.47+0.84

Reverse KL

TaskDH-KL Acc / IFEvalTRSD Acc / IFEvalΔ\Delta AccΔ\Delta IFEvalΔJ\Delta J
Tool Use56.70 / 47.7467.01 / 55.62+10.31+7.88+9.10
Science Q&A66.07 / 50.0767.65 / 55.88+1.58+5.81+3.70
Medical34.20 / 52.8342.30 / 54.73+8.10+1.90+5.00

Squared Hellinger

TaskDH-KL Acc / IFEvalTRSD Acc / IFEvalΔ\Delta AccΔ\Delta IFEvalΔJ\Delta J
Tool Use65.98 / 49.0172.16 / 58.08+6.18+9.07+7.63
Science Q&A63.51 / 50.1668.05 / 53.40+4.54+3.24+3.89
Medical39.70 / 52.3040.20 / 62.77+0.50+10.47+5.49

All Δ\Delta values are computed as TRSD minus DH-KL. Here, ΔJ\Delta J uses the same retention-aligned joint score, J=12(Acc+IFEval)J=\frac{1}{2}(\mathrm{Acc}+\mathrm{IFEval}), as in our robustness analysis. Task accuracy and IFEval are reported separately, while ΔJ\Delta J summarizes their trade-off. As shown in the results, TRSD achieves higher retention in each case, and consistently achieves higher joint score than DH-KL across all nine settings.

AuthorsOpenReview ↗

Response to Reviewer LBeZ (3/4): Constant-Beta Controls

W2–Q1-Constant-Beta Controls: How does TRSD compare with DH-KL and constant-β\beta baselines under the exact Table 2 setup?

We next address the reviewer’s concern about uniform shrinkage from βmax⁡<1\beta_{\max}<1. We agree that clipping can shrink the teacher even for low-curvature samples. To test whether this global dampening alone explains the gains, we remove local curvature scaling and apply one constant coefficient to every sample. For each setting, this coefficient equals the mean βi\beta_i realized across samples and updates during the corresponding TRSD trajectory.

GeometryTaskConstant β\betaConstant-β\beta Acc / IFEvalTRSD Acc / IFEvalΔ\Delta AccΔ\Delta IFEval
Forward KLTool Use0.281062.89 / 44.9765.98 / 57.81+3.09+12.84
Forward KLScience Q&A0.334865.09 / 52.1668.24 / 53.06+3.15+0.90
Forward KLMedical0.326739.10 / 51.2140.90 / 60.91+1.80+9.70
Reverse KLTool Use0.266364.95 / 48.3867.01 / 55.62+2.06+7.24
Reverse KLScience Q&A0.466566.67 / 53.4567.65 / 55.88+0.98+2.43
Reverse KLMedical0.228539.70 / 52.6842.30 / 54.73+2.60+2.05
Squared HellingerTool Use0.457163.92 / 52.0172.16 / 58.08+8.24+6.07
Squared HellingerScience Q&A0.526762.72 / 52.3368.05 / 53.40+5.33+1.07
Squared HellingerMedical0.324538.70 / 59.8340.20 / 62.77+1.50+2.94

TRSD outperforms the matched constant-β\beta control on IFEval in all nine settings, while learning comparable or higher task accuracy. The retention differences are larger on Tool Use and Medical, while the three Science Q&A settings show smaller but consistently positive improvements. Because the control preserves the same interpolation geometry and matches the average interpolation strength but removes local curvature scaling, these results indicate that simple global dampening does not explain the gains.

Because the matched constants can only be obtained after observing the complete TRSD trajectory, we additionally sweep constant β∈{1.0,0.8,0.6,0.4,0.2}\beta\in\{1.0,0.8,0.6,0.4,0.2\} under forward KL. Here, β=1\beta=1 is the original SDFT baseline.

TaskMetricTRSDβ=1.0\beta=1.0β=0.8\beta=0.8β=0.6\beta=0.6β=0.4\beta=0.4β=0.2\beta=0.2
Tool UseAcc.65.9863.9263.9270.1067.0163.92
IFEval57.8144.1441.7542.8143.8050.40
Science Q&AAcc.68.2465.8867.8570.4166.2765.29
IFEval53.0645.6046.9644.3552.2754.27
MedicalAcc.40.9040.2039.7040.9038.2038.00
IFEval60.9148.5655.3755.9553.4453.80

TRSD generally achieves higher IFEval in constant-β\beta comparisons. These comparisons are now reported in Section 5.3, with the pre-specified constant-β\beta sweep in Appendix E.12.

Reviewer_byRNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the authors look at forgetting in self-distillation fine-tuning (SDFT).

The authors find that a small number of examples have very large teacher–student divergence. They then propose TRSD to limit these large shifts. They validate it on tool use, science Q&A, and medical reasoning, and report better retention than SDFT.

My main concern is that they do not compare TRSD with their simpler DH-KL baseline. I also worry that some of the reported forgetting is caused by output format errors.

Critical strengths
  • S1: The method is well-motivated. The heavy-tailed divergence observation and DH-KL intervention give a clear motivation for the proposed method.
  • S2: Proposed TRSD has a straightforward formulation across several divergences. The experiments include both single-task and sequential adaptation.
Critical weaknesses
  • W1: In fact, the missing comparison with DH-KL is my biggest concern. Fig 2 shows that simply down-weighting high-divergence examples already improves retention without hurting the new task. But this baseline is not included in the main TRSD comparisons, either for single-task adaptation or for sequential learning.

  • W2: The results do not clearly separate forgetting the answer format from forgetting the underlying knowledge. The Science Q&A accuracy of SFT drops to 0.99% after later adaptation, which looks like severe catastrophic forgetting. However, the task requires answers inside <answer> tags, and the appendix shows that the models often stop producing this format. The paper should quantify how much of the accuracy drop comes from missing tags and how much comes from wrong answers.

  • W3: I would like to know what the high-divergence examples actually look like. The whole method is motivated by a small tail of unusual training samples, but the paper does not really investigate these examples. Are they difficult but useful demonstrations, noisy labels, or out-of-distribution prompts? If many are simply bad or irrelevant examples, a data-cleaning baseline could already address part of the problem.

  • W4: I am not fully convinced by the hyperparameter robustness claim. Some configurations in the appendix perform much worse; for example, Table 18 includes a forward-KL setting that loses 8.25 points on Tool Use relative to SDFT. The sequential experiments also choose different configurations for different stages.

  • W5: The online calibration depends on statistics of the current mini-batch, including the mean divergence curvature. To me, this means the constraint for one example may change depending on what other examples happen to be in its batch. The paper does not show how stable the method is across batch sizes or different random batch compositions.

Questions for authors

Please kindly find the comments in the weaknesses section.

6 comments
AuthorsOpenReview ↗

Response to Reviewer byRN (1/3)

We thank the reviewer for recognizing the motivation provided by our heavy-tailed divergence analysis and DH-KL intervention, the straightforward formulation of TRSD across multiple divergences, and the coverage of both single-task and sequential adaptation. Below, we respond to your concerns in detail.

W1: How does TRSD compare directly with DH-KL in single-task and sequential adaptation?

We thank the reviewer for emphasizing this missing comparison. Following the reviewer’s suggestion, we evaluated DH-KL using the same configurations as in Section 3.3 across all nine Qwen2.5-7B-Instruct task–divergence settings in Table 2.

Forward KL

TaskDH-KL Acc / IFEvalTRSD Acc / IFEvalΔ\Delta AccΔ\Delta IFEvalΔJ\Delta J
Tool Use71.13 / 49.6365.98 / 57.81−5.15+8.18+1.52
Science Q&A70.41 / 44.6768.24 / 53.06−2.17+8.39+3.11
Medical41.70 / 58.4440.90 / 60.91−0.80+2.47+0.84

Reverse KL

TaskDH-KL Acc / IFEvalTRSD Acc / IFEvalΔ\Delta AccΔ\Delta IFEvalΔJ\Delta J
Tool Use56.70 / 47.7467.01 / 55.62+10.31+7.88+9.10
Science Q&A66.07 / 50.0767.65 / 55.88+1.58+5.81+3.70
Medical34.20 / 52.8342.30 / 54.73+8.10+1.90+5.00

Squared Hellinger

TaskDH-KL Acc / IFEvalTRSD Acc / IFEvalΔ\Delta AccΔ\Delta IFEvalΔJ\Delta J
Tool Use65.98 / 49.0172.16 / 58.08+6.18+9.07+7.63
Science Q&A63.51 / 50.1668.05 / 53.40+4.54+3.24+3.89
Medical39.70 / 52.3040.20 / 62.77+0.50+10.47+5.49

All Δ\Delta values are computed as TRSD minus DH-KL. Here, ΔJ\Delta J uses the same retention-aligned joint score, J=12(Acc+IFEval)J=\frac{1}{2}(\mathrm{Acc}+\mathrm{IFEval}), as in our robustness analysis. Task accuracy and IFEval are reported separately, while ΔJ\Delta J summarizes their trade-off. As shown in the results, TRSD achieves higher retention in each case, and consistently achieves higher joint score than DH-KL across all nine settings.

We further compare DH-KL and TRSD under sequential Tool Use →\rightarrow Science Q&A →\rightarrow Medical adaptation. The TRSD configurations are those reported in Figure 3 under Sequential Continual Learning. DH-KL values are reported in absolute percentage, while the TRSD rows report percentage-point differences from DH-KL.

StageConfigurationToolScienceMedicalIFEval
After ToolDH-KL71.13––49.63
TRSD Δ\Delta−5.15––+8.18
After ScienceDH-KL59.7970.41–39.24
TRSD Δ\Delta+3.10−3.15–+8.63
After MedicalDH-KL56.7065.4838.5036.44
TRSD Δ\Delta+1.03+1.98−1.30+3.20

Throughout sequential adaptation, TRSD achieves higher IFEval than DH-KL at every stage while generally providing stronger retention of previously learned tasks. The single-task and sequential DH-KL comparisons are now reported in Section 5.3, with the complete stage-wise results in Appendix E.12.

AuthorsOpenReview ↗

Response to Reviewer byRN (2/3)

W2: How much of the sequential Science Q&A decline reflects answer-format loss versus incorrect underlying decisions?

Thanks for the question. We audited 400 Science Q&A responses using Grok 4.7 with human verification: 100 from each of SFT and DFT after Science adaptation, and another 100 from each method after Medical adaptation. Each response was first assessed without the gold answer to identify the model's intended choice, and the resulting label was then human-checked before comparison with the gold answer.

We grouped the responses into four cases: strictly correct means the answer and format are both correct; format-only means the intended choice is correct but the required format is not; wrong decision means the response makes a clear but incorrect choice; and unresolved means that no usable final choice can be identified.

MethodCheckpointStrictly correctFormat-onlyWrong decisionUnresolvedContent-normalized correct
SFTAfter Science65034165
SFTAfter Medical12371524
DFTAfter Science69031069
DFTAfter Medical232941752

Here, content-normalized correct combines strictly correct and format-only responses.

The results show that the accuracy decline cannot be attributed entirely to formatting errors. Even when format-only responses are counted as correct, accuracy after Medical adaptation still drops by 41 points for SFT and 17 points for DFT. The decline therefore reflects losses in both format adherence and the task understanding.

This analysis is now reported in Appendix E.9.2, and the four audited 100-example subsets will be provided in the supplementary material.

W3: What do high-divergence examples look like, and can data cleaning explain the tail?

Thanks for the concern. Using the data underlying Figures 1 and 5, we ranked the training examples by the initialization-time divergence Ki,0=KL(qo∥p0)K_{i,0}=\mathrm{KL}(q_o\Vert p_0) for Qwen2.5-7B-Instruct and InternLM2.5-7B-Chat on Tool Use, Science Q&A, and Medical. For every model–task pair, we randomly sampled 10 examples from each of the low (0–20th percentile), typical (40–60th), and high (80–100th) divergence ranges, yielding 180 inspected instances.

We conducted a blinded human-in-the-loop inspection of the sampled examples (shuffled and blind the divergence label), assessing annotation validity as either No obvious issue or Invalid, whether the generation was OOD (unrelated to the prompt), and whether its format was garbled.

Divergence rangeInspectedInvalid demonstrationMalformed generationIrrelevant/OODAny obvious issue
Low (0–20%)602204 (6.7%)
Typical (40–60%)603003 (5.0%)
High (80–100%)604004 (6.7%)

The high-divergence group does not show a concentration of obvious data-quality problems based on its flagged rate. The only two malformed generations occur in the low-divergence range, no irrelevant or out-of-domain examples are identified in any range. Since this analysis suggests that high divergence here is not a proxy for a class of defective examples, a simple data cleaning would not directly substitute for adaptive teacher calibration. The audit protocol and aggregate analysis are now reported in Appendix B.1 (Data-Quality Audit Across Divergence Ranges), and we will provide the sampled JSON records in the supplementary material.

AuthorsOpenReview ↗

Response to Reviewer byRN (3/3)

W4: How robust is TRSD to calibration hyperparameters and stage-specific configurations? Table 18 includes a forward-KL setting that loses 8.25 points on Tool Use.

We thank the reviewer for raising this concern. To clarify, our robustness claim is intended to describe the trend across the sweep, rather than uniform dominance at every individual point.

As detailed in Section 5.3, the complete Qwen2.5-7B-Instruct sweep contains 311 configurations across all nine task–divergence settings. Among them, 251 configurations (80.7%) improve the retention-aligned joint score over matched SDFT. More importantly, the sweep median improves IFEval in all nine settings, by 3.57 points on average, while median task accuracy remains comparable to matched SDFT, with no reduction exceeding 1.58 points. Thus, the retention trend persists beyond the individually displayed configurations, although the unfavorable settings show that the method is not completely insensitive to calibration choices.

For sequential adaptation, we kindly refer the reviewer to the stage-wise Cartesian-product grids for Qwen in Appendix E.9.3 and for InternLM in Appendix E.10, which cover all three divergence geometries. Across the grids, TRSD consistently improves IFEval and generally strengthens retention of previously learned tasks over matched SDFT, while incoming Medical accuracy remains comparable. The results in the grids support the effectiveness of TRSD in sequential adaptation.

W5: How sensitive is peer-relative online calibration to batch size and random batch composition?

We thank the reviewer for the question. To measure this sensitivity, we analyzed all 18 main settings (two models, three tasks, and three divergence geometries). We split the training by training steps into early, middle, and late stage. We used the saved samples from a run with the original batch size set to 32. At every step, we shuffled the same 32 curvature values, repartitioned them into virtual batches of 8 or 16, and recomputed each sample's coefficient using the original ρ\rho and clipping interval. The implementation details are in Appendix D.6.

We summarize the results with three measures. Within-sample SD and the 90% β\beta range measure how much the same sample's coefficient changes across peer assignments. Mean-shift magnitude compares its average resampled coefficient with the original batch-32 value, showing whether a smaller batch systematically shifts the calibration. Each entry is the median of the corresponding setting-level statistic across the 18 settings, and parentheses report the minimum and maximum setting-level median within-sample SD.

Training stageVirtual batchWithin-sample β\beta SD90% β\beta rangeMedian mean-shift magnitude
Early80.0155 (0.0069–0.0418)0.05100.0034
Early160.0092 (0.0041–0.0248)0.03040.0011
Middle80.0171 (0.0076–0.0429)0.05570.0038
Middle160.0104 (0.0046–0.0256)0.03400.0013
Late80.0194 (0.0086–0.0485)0.06360.0044
Late160.0118 (0.0052–0.0287)0.03860.0015

As expected, smaller virtual batches increase composition-level variation. However, this variation remains centered near the original calibration: the signed shift is below 0.002 in magnitude at every stage for batch size 8 and below 0.001 for batch size 16. The median absolute mean shift is at most 0.0044. We also observe no qualitative change between the early, middle, and late stages.

These results indicate that changing the peer composition introduces bounded coefficient-level variation but does not systematically move the overall calibration regime, even when the original batch is reduced by a factor of four. This batch dependence and the scope of the analysis are now clarified in Appendix D.6, and we will provide the complete per-setting results and analysis code in the supplementary material.

Reviewer_TQHyOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies why Self-Distillation Fine-Tuning (SDFT) still forgets. They measure the per-example KL between the demonstration-conditioned teacher and the student, find a long right tail, and show that down-weighting the tail of each batch improves IFEval retention without hurting the task. Based on this, they propose TRSD, which replaces the SDFT teacher at each token with its projection into an f-divergence ball around the current student. For forward KL, reverse KL and Hellinger this becomes a geometric, arithmetic or square-root interpolation between student and teacher, with a per-sample coefficient set from a local curvature statistic. Experiments on Qwen2.5-7B-Instruct and InternLM2.5-7B-Chat over Tool Use, Science Q&A and Medical, in single-task and sequential settings, show better IFEval retention than their SDFT runs at similar task accuracy.

Critical strengths

1.The observation that the SDFT teacher divergence is long-tailed, and that down-weighting the tail alone improves retention, is a genuine and useful finding about SDFT that stands independent of the proposed method. 2.The method is cheap to adopt. It reuses the distributions SDFT already computes, adds no forward passes, and every TRSD variant is compared with SDFT under the same divergence. 3.They have reported all 311 sweep configurations with nothing removed, and the IFEval improvement over their own SDFT runs holds in a large majority of them, so I believe the direction of the effect is real.

Critical weaknesses

The SDFT baseline looks over-stepped, and the paper cannot separate its mechanism from a smaller effective step. SDFT is run at learning rate 5e-5 for 2 to 3 epochs with 1024 completion tokens and retains only 67 to 74% of base IFEval under forward KL, while the original SDFT paper reports around 90 to 97% retention on the same model and tasks with a lower learning rate, fewer epochs and 2048 tokens. Under the proposed calibration a typical sample gets beta = sqrt(rho), which is 0.22 to 0.50 for the reported rho values, and for the reverse-KL path the TRSD gradient is beta times the SDFT gradient to first order. So TRSD may mostly be undoing an over-large step. A lower-learning-rate SDFT baseline and a fixed global beta control are needed to attribute the gain, and neither is present. The headline numbers are best-of-grid single runs. Seven of the nine Qwen entries in Table 2 are the maximum of their sweep on the reported metric, compared against a single SDFT run, with no seeds, no validation split and no stated selection rule. The Tool Use test set appears to be 97 examples, so several of the reported gains amount to one or two questions. The IFEval direction probably survives this, but the task accuracy claims do not. Forgetting is measured only by IFEval, and the paper's own analysis shows that IFEval and the Science score are sensitive to output format. The large InternLM sequential gains are against SDFT runs that collapsed (IFEval 9.91, Science 0.00), and the cause of the collapse is never diagnosed. I'd like to see retention on a broader suite, as in the original SDFT paper.

Questions for authors

1- Why does your SDFT baseline retain 67 to 74% of base IFEval when Shenfeld et al. report around 90 to 97% on the same model and tasks? Could you report SDFT and TRSD at lower learning rates (1e-5 and 5e-6) with one epoch and three seeds? If a tuned SDFT matches TRSD, the premise of the paper changes. 2- Under the same protocol, how does TRSD compare with a fixed global beta equal to the batch mean of your beta_i, and with the DH-KL down-weighting from Section 3 placed in Table 2? A positive answer here would move my score. 3- What is the selection rule for Table 2 and the sequential results? I'd suggest reporting the sweep median next to the best cell, and stating the test-set sizes. 3- What is the empirical distribution of beta_i and the realized divergence? Section 4.1 says teachers already close to the student remain unchanged, but with rho at most 0.25 and beta_max below 1 in most configurations the rule alters every sample, and clipping at a positive beta_min can push a sample past its own budget.

6 comments
AuthorsOpenReview ↗

Response to Reviewer TQHy (1/5)

We thank the reviewer for recognizing the value of the heavy-tailed teacher-divergence finding, the low computational overhead of TRSD, and the consistent IFEval-retention trend across the complete 311-configuration sweep. Below, we respond to your concerns and questions in detail.

W1, Q1: Why does the SDFT baseline retain 67–74% of base IFEval when Shenfeld et al. report around 90–97%, and how does it compare with TRSD under lower learning rates, fewer epochs, and longer completion length?

We thank the reviewer for raising this important concern. Shenfeld et al. report a sweep over learning rates and epochs, but do not identify the exact combination corresponding to each post-adaptation IFEval result. We therefore followed their released default reproduction commands for Tool Use and Science on forward KL, which use a learning rate of 5e-5 and 2 epochs. Meanwhile, although we use a maximum completion length of 1,024 rather than 2,048, truncation is not activated very often, so this setting would not affect the performance too much. For example, the mean rates across our nine Qwen SDFT and nine matched TRSD (in Table 2) configurations are 1.02% and 0.93%, respectively.

Following the reviewer’s suggestion, we additionally evaluated forward-KL SDFT and TRSD on Tool Use with Qwen2.5-7B-Instruct using 1 epoch, learning rates {1e-5, 5e-6}, and three pre-specified seeds {42, 43, 44}. Each row reports the mean and standard deviation across the three seeds and the Tool Use test set contains 97 examples. The retention rate reports the proportion of the base model’s IFEval score (65.78%) retained after adaptation.

MethodlrToolIFEval avgRetention rate
SDFT1e-562.89% ± 1.7860.05% ± 1.2791.3%
TRSD1e-565.29% ± 1.5763.77% ± 0.7596.9%
SDFT5e-659.79% ± 2.7361.25% ± 0.2993.1%
TRSD5e-662.54% ± 1.5864.19% ± 0.3597.6%

With smaller learning rate, SDFT indeed retains a larger proportion, 91.3%–93.1% here, of the base IFEval score. However, the adjustment of training configuration does not eliminate the observed retention advantage of TRSD. TRSD achieves higher IFEval than SDFT for every seed at both learning rates, with the three-seed mean improvements of 3.72 and 2.94 points, respectively. New-task performance is comparable, with TRSD achieving higher mean task accuracy at both learning rates.

We have added these three-seed results in Section 5.3, explicitly noting the smaller margin in this setting. The complete per-seed breakdown and four-metric IFEval results are provided in Appendix E.12.

AuthorsOpenReview ↗

Response to Reviewer TQHy (2/5)

W1, Q2: How does TRSD compare with DH-KL and matched fixed-global-β\beta controls?

We thank the reviewer for suggesting DH-KL and fixed-global-β\beta as additional baselines. Following this suggestion, we evaluated DH-KL using the same configurations as in Section 3.3 across all nine Qwen2.5-7B-Instruct task–divergence settings in Table 2. We also trained a fixed-β\beta control for each setting, where the global coefficient is set to the mean βi\beta_i realized over the corresponding TRSD training trajectory.

We first compare DH-KL with TRSD:

Forward KL

TaskDH-KL Acc / IFEvalTRSD Acc / IFEvalΔ\Delta AccΔ\Delta IFEvalΔJ\Delta J
Tool Use71.13 / 49.6365.98 / 57.81−5.15+8.18+1.52
Science Q&A70.41 / 44.6768.24 / 53.06−2.17+8.39+3.11
Medical41.70 / 58.4440.90 / 60.91−0.80+2.47+0.84

Reverse KL

TaskDH-KL Acc / IFEvalTRSD Acc / IFEvalΔ\Delta AccΔ\Delta IFEvalΔJ\Delta J
Tool Use56.70 / 47.7467.01 / 55.62+10.31+7.88+9.10
Science Q&A66.07 / 50.0767.65 / 55.88+1.58+5.81+3.70
Medical34.20 / 52.8342.30 / 54.73+8.10+1.90+5.00

Squared Hellinger

TaskDH-KL Acc / IFEvalTRSD Acc / IFEvalΔ\Delta AccΔ\Delta IFEvalΔJ\Delta J
Tool Use65.98 / 49.0172.16 / 58.08+6.18+9.07+7.63
Science Q&A63.51 / 50.1668.05 / 53.40+4.54+3.24+3.89
Medical39.70 / 52.3040.20 / 62.77+0.50+10.47+5.49

All Δ\Delta values are computed as TRSD minus DH-KL. Here, ΔJ\Delta J uses the same retention-aligned joint score, J=12(Acc+IFEval)J=\frac{1}{2}(\mathrm{Acc}+\mathrm{IFEval}), as in our robustness analysis. Task accuracy and IFEval are reported separately, while ΔJ\Delta J summarizes their trade-off. As shown in the results, TRSD achieves higher retention in each case, and consistently achieves higher joint score than DH-KL across all nine settings.

We next compare TRSD with the reviewer-suggested fixed-global-β\beta control. For each setting, we average the realized βi\beta_i values over samples and training updates, and then train a new model using that value globally.

GeometryTaskFixed β\betaFixed-β\beta Acc / IFEvalTRSD Acc / IFEvalΔ\Delta AccΔ\Delta IFEval
Forward KLTool Use0.281062.89 / 44.9765.98 / 57.81+3.09+12.84
Forward KLScience Q&A0.334865.09 / 52.1668.24 / 53.06+3.15+0.90
Forward KLMedical0.326739.10 / 51.2140.90 / 60.91+1.80+9.70
Reverse KLTool Use0.266364.95 / 48.3867.01 / 55.62+2.06+7.24
Reverse KLScience Q&A0.466566.67 / 53.4567.65 / 55.88+0.98+2.43
Reverse KLMedical0.228539.70 / 52.6842.30 / 54.73+2.60+2.05
Squared HellingerTool Use0.457163.92 / 52.0172.16 / 58.08+8.24+6.07
Squared HellingerScience Q&A0.526762.72 / 52.3368.05 / 53.40+5.33+1.07
Squared HellingerMedical0.324538.70 / 59.8340.20 / 62.77+1.50+2.94

TRSD outperforms the matched fixed-β\beta control IFEval in all nine settings, and learns comparable or higher task accuracy. The retention gains are larger on Tool Use and Medical, while the three Science Q&A settings show smaller but consistently positive IFEval improvements. This comparison controls for the average interpolation strength and indicates that TRSD’s benefit is not explained solely by applying a smaller global update.

Because these matched fixed coefficients can only be obtained after observing the complete TRSD trajectory, we additionally sweep fixed β∈{1.0,0.8,0.6,0.4,0.2}\beta\in\{1.0,0.8,0.6,0.4,0.2\} under forward KL. Here, β=1\beta=1 is the original SDFT baseline.

TaskMetricTRSDβ=1.0\beta=1.0β=0.8\beta=0.8β=0.6\beta=0.6β=0.4\beta=0.4β=0.2\beta=0.2
Tool UseAcc.65.9863.9263.9270.1067.0163.92
IFEval57.8144.1441.7542.8143.8050.40
Science Q&AAcc.68.2465.8867.8570.4166.2765.29
IFEval53.0645.6046.9644.3552.2754.27
MedicalAcc.40.9040.2039.7040.9038.2038.00
IFEval60.9148.5655.3755.9553.4453.80

TRSD generally achieves higher IFEval. These baseline comparisons are now reported in Section 5.3 (Robustness and Matched Control Baselines), with the pre-specified fixed-β\beta sweep in Appendix E.12.

AuthorsOpenReview ↗

Response to Reviewer TQHy (3/5)

W2, Q3a: What is the selection protocol for Table 2 and the sequential results, and how representative are the retention and task-accuracy trends across the complete sweeps and evaluation sets? What are the test-set sizes, and how should the task-accuracy differences be interpreted?

We thank the reviewer for raising this important question about the representativeness of the reported results. We kindly refer the reviewer to Section 5.3, where we evaluate the complete Qwen2.5-7B-Instruct calibration sweep across all nine domain–divergence settings with 311 configurations. Among them, 251 (80.7%) improve the retention-aligned joint score over matched SDFT. These complete-sweep results indicate that the retention trend is not confined to the individual configurations displayed in Table 2.

For sequential adaptation, we kindly refer the reviewer to the stage-wise Cartesian-product grids for Qwen in Appendix E.9.3, Cross-Task Grid Configurations, and for InternLM in Appendix E.10, InternLM2.5 Sequential Adaptation. Across the sequential grids, TRSD consistently improves IFEval and generally strengthens retention of previously learned tasks over matched SDFT. Incoming Medical accuracy remains comparable to matched SDFT on both model families. This supports the overall continual-learning trend beyond the individual trajectories shown in the main text.

Following the reviewer’s suggestion, we additionally report the median TRSD result over the complete sweep for each Table 2 cell:

CellnSDFT AccMedian TRSD AccSDFT IFEvalMedian TRSD IFEval
Forward Tool4063.92%63.92%44.14%47.42%
Forward Science4065.88%67.65%45.60%47.115%
Forward Medical2440.20%39.00%48.56%55.185%
Reverse Tool4064.95%64.41%43.80%50.185%
Reverse Science4067.06%66.50%49.82%53.345%
Reverse Medical2438.60%40.20%49.01%54.82%
Hellinger Tool4062.89%64.95%53.14%55.52%
Hellinger Science4065.88%64.30%49.72%51.59%
Hellinger Medical2337.60%39.70%55.73%56.49%

Median TRSD improves IFEval in all nine cells, by 3.57 points on average. The median task accuracy remains comparable to matched SDFT, with no reduction exceeding 1.58 points. These median comparisons are now reported alongside the selected configurations in Table 2 and discussed in Section 5.3.

We also thank the reviewer for noting that the evaluation-set sizes were not sufficiently visible. The evaluation sets contain 541 prompts for IFEval, 97 examples for Tool Use, 507 for Science Q&A, and 1,000 for Medical. Therefore, small Tool accuracy differences should not be interpreted as statistically decisive. However, our central claim throughout is not that TRSD universally improves task accuracy, but that it improves prior-capability retention while maintaining comparable new-task acquisition. The task-accuracy results are intended to show that the retention gain generally does not require substantially weaker learning of the incoming task.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response and Summary of Revisions

General Response

Review Summary

We thank the Area Chair and reviewers for their careful reading and constructive feedback. As language models are adapted to new domains, it becomes increasingly important to learn new skills while preserving existing capabilities. Self-distillation fine-tuning helps mitigate forgetting, yet we find that a small fraction of training examples produces unusually large gaps between the teacher and student predictions and contributes disproportionately to forgetting.

Our paper addresses this challenge by introducing Trust-Region Constrained Self-Distillation Fine-Tuning (TRSD). Across tool use, scientific question answering, and medical reasoning, TRSD better preserves instruction-following while generally maintaining or improving new-task accuracy.

We appreciate the reviewers’ recognition of the following strengths.

  1. An important empirical finding on the underlying cause of forgetting in SDFT.

"The observation that the SDFT teacher divergence is long-tailed, and that down-weighting the tail alone improves retention, is a genuine and useful finding" - Reviewer TQHy

"The empirical diagnosis of SDFT is well-executed." - Reviewer LBeZ

"The heavy-tailed divergence observation and DH-KL intervention give a clear motivation for the proposed method." - Reviewer byRN

  1. A principled, computationally efficient, and well-motivated framework.

"The method is cheap to adopt. It reuses the distributions SDFT already computes, adds no forward passes" - Reviewer TQHy

"Proposed TRSD has a straightforward formulation across several divergences." - Reviewer byRN

"The f-divergence trust-region formulation is computationally practical." — Reviewer LBeZ

  1. Extensive empirical evaluations across diverse adaptation settings.

"The IFEval improvement over their own SDFT runs holds in a large majority of them, so I believe the direction of the effect is real." - Reviewer TQHy

"The empirical evaluation covers both single-task and three-stage sequential continual learning across three divergence geometries." - Reviewer LBeZ

Summary of Manuscript Modification

Based on the reviewers' feedback, we have revised our manuscript, with changes highlighted in color. Below, we summarize the key revisions:

[TQHy, byRN, LBeZ] Added direct comparisons with DH-KL and matched fixed-global-β\beta controls in Section 5.3, together with a sequential DH-KL comparison and a pre-specified fixed-β\beta sweep in Appendix E.12.

[TQHy] Added three-seed results under two milder learning rates with one-epoch adaptation in Section 5.3 and Appendix E.12, and added sweep-median TRSD results to Table 2 to summarize the complete calibration sweep.

[TQHy, LBeZ] Clarified the distinction between the exact trust-region formulation and the practical peer-relative, clipped implementation in Section 4.3. Appendix D.6 now reports the empirical βi\beta_i and realized-divergence distributions, batch-composition sensitivity, and the motivation for completion-level calibration.

[TQHy, byRN] Expanded the appendix with retention results on the broader SDFT benchmark suite, a 400-response audit separating format errors from incorrect decisions, and a 180-example data-quality audit across divergence ranges.

[TQHy, LBeZ] Clarified evaluation-set sizes and the configuration-reporting protocol in Appendix E.1, and added in Appendix E.11 an analysis of fixed forward-KL calibration tuples applied unchanged across all three tasks.

We believe these revisions have significantly strengthened our manuscript and addressed the key concerns raised.

Overall Response

In addition, we greatly appreciate the constructive feedback from the reviewers, which further strengthens our work. Altogether, these contributions position TRSD as a practical solution for continual self-distillation in language models, with promising implications for future research.

Below, we address each reviewer’s comments point by point.

I thank the authors for their thorough explanation and for providing further experimental results in response to my questions. The newly provided analyses address my concerns regarding SDFT, DH-KL baselines, and comparisons at fairly matched configs. The sweeps on \beta and different objective geometries are very helpful and answered my questions. Their forgetting measurement on a broader eval suite also suggests that TRSD improves retention beyond IFEval. Overall, my main concerns are largely addressed, and I would like to increase my score to 3.

Dear Reviewer TQHy,

Thank you very much for your thoughtful follow-up and for updating your score. We are glad that our additional analyses and experiments addressed your main concerns. We sincerely appreciate your time and constructive feedback, which helped us strengthen the paper.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.