acceptodds
Under review as a conference paper at ICLR 2027

Tether: Decision Retention for Continual Fine-Tuning of LLMs

Abstract

Continual fine-tuning can cause catastrophic forgetting in language models. Changes in old-task accuracy reflect a net effect: repaired errors offset lost correct predictions. We instead measure and constrain decision drift, the fraction of old-task inputs whose decisions differ from those of the model at the end of that task, the teacher. Drift bounds forgetting, both lost correct answers and the loss in accuracy, whereas accuracy does not bound drift; it needs no labels and has an exact finite-sample upper bound on held-out inputs. We propose Tether, a framework that stores real or synthetic inputs with the model's outputs as records and regularises later training with any continuous loss that upper-bounds record drift. For forward-KL distribution matching on records, we derive the exact per-record critical radius below which the recorded decision is preserved; it shows that standard KL also penalises distributional changes that leave the recorded decision intact. Two new instances, Tether-Margin and thresholded KL (Tether-TKL), are zero on a region around the teacher and skip satisfied records before back-propagation. On the O-LoRA benchmark with T5-large and Llama-2-7B-chat and on six generation tasks with Llama-3-8B-Instruct, without retaining per-example ground-truth labels, the three instances reach final average scores comparable to DER++'s, from 1.1 points below to 1.5 above. Tether-TKL and KL change fewer earlier decisions than DER++, KL losing about half as many correct answers, and on generation all three keep 50–79% of earlier free-form outputs unchanged, against 38% for DER++. On the O-LoRA benchmark, with fewer retained records in aggregate than the 2% replay and DER++ buffers, all three also preserve more of the model's earlier behaviour than labelled replay, changing fewer decisions in every setting.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Large Language Models, Catastrophic Forgetting, Decision Retention, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ar1WOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Tether, which extends the retention goal in continual fine-tuning from old-task accuracy to decision drift relative to earlier models. The authors derive an exact per-record KL radius for decision preservation and introduce Margin and thresholded KL losses with zero-loss regions. Classification and generation experiments show that the framework can reduce changes in earlier behavior while achieving similar final scores.

Critical strengths

The paper clearly explains the often-overlooked issue that unchanged accuracy does not imply unchanged behavior, and provides a label-free, measurable, and auditable decision drift metric with practical value.

The exact KL critical radius provides a clear theoretical basis for preserving recorded decisions.

Critical weaknesses
  1. Table 7 shows that, over the fifteen-task sequences, Tether-KL has substantially higher total training time than Replay and DER++ on both T5-large and Llama-2.

  2. The statement “Neither provides the independent task-distribution draws of Assumption 1, so the theorem does not cover the reported runs” highlights a gap between the theory and experiments. The authors should explain why the experimental setup does not match the assumptions required by the theory.

  3. The admission rule excludes inputs with low margins or invalid teacher answers, yet these inputs account for a large share of the remaining drift. In the Amazon→Yelp experiment, excluded inputs contribute 12.0 percentage points out of Margin’s total drift of 17.8%. The method provides little direct protection for this vulnerable region, and the authors should discuss how to cover these inputs.

  4. Tether-KL, which achieves the best retention, uses standard output distillation related to DER/LwF, with a different application setting. The new Margin and TKL instances have higher aggregate drift than KL. In the generation experiment, TKL takes about the training time longer than the baselines. The authors should explain its practical value.

  5. The paper states that “IncLoRA, replay and DER++ are local; other baselines are published results,” meaning that several baseline scores are taken from their original papers rather than reproduced under a shared setup. Table 9 also shows large gaps between reproduced and published scores, such as 79.1 versus 57.8 for Replay, although these settings are not directly comparable; this makes cross-paper rankings difficult to interpret. The main local baselines are also relatively old, including DER++ from 2020, so comparisons with more recent methods under a shared protocol are needed.

  6. Both classification and generation experiments use LoRA, which restricts the form of parameter updates. It remains unclear whether the retention performance and computational benefits hold under full-parameter fine-tuning. At least one full-parameter fine-tuning experiment would help establish broader applicability.

  7. The authors should include open-domain question answering, multi-turn dialogue, or long-context question answering to test the method on more complex outputs and knowledge-update scenarios.

  8. The budget of 48 records per task is an empirical choice, with no theoretical justification for this specific value and no record-count ablation on the main benchmark. The only record-count study, MNLI→RTE with T5, shows that drift decreases substantially as the number of records increases, leaving the quality–cost trade-off at 48 unclear. The authors should report retention performance and computational cost across record budgets on the main benchmark.

Questions for authors

See above

4 comments
AuthorsOpenReview ↗

Response to Reviewer ar1W (Part 1/3)

We thank the reviewer for the detailed review and for recognizing the label-free, auditable measure of decision drift and the exact KL critical radius, which is the central technical result of the paper. The main new result is that, at matched training time and similar accuracy, DER++ still changes more of the earlier decisions than Tether-KL on both models. We respond to the eight points in order and will include the new experiments in the revision.

Throughout this response, drift is the percentage of earlier-task test inputs on which the final model's decision differs from the teacher's, and Acc is the final average accuracy. Unless stated otherwise, each new result is the mean of nine runs, the three task orders (Orders 4–6) with three seeds each. Paired comparisons are between runs that share a task order and a seed, so each comparison has nine pairs.

1. Training time of Tether-KL (Table 7)

The T5-large numbers for Tether-KL in Table 7, including the 9.0× after fifteen tasks that also appears in Table 3, are a mistake. They were timed on an earlier variant that back-propagates through all 48 records of every earlier task at every step, whereas all Tether-KL results in Tables 2 and 3 use 16 of the 48 records per step. The correct cost is 2.5× that of IncLoRA for training alone, which is the basis of the other entries in these tables, or 3.1× when record construction is included. On Llama-2, the 12.1× is correct for the configuration in the paper, but using 4 records per step instead of 16 gives similar drift and accuracy and lowers the cost to 4.1×.

With these numbers, Tether-KL takes about twice the training time of DER++ at its default setting. To examine whether the lower drift is simply an effect of the longer training time, we gave DER++ approximately the same training time as Tether-KL by using more replay batches per step and a proportionally larger buffer. In the table below, Time is the training time including record construction, relative to IncLoRA.

ModelMethodTimeAccDrift
T5-largeDER++ 2%1.6×76.212.7
T5-largeDER++, matched training time3.0×78.310.5
T5-largeTether-KL3.1×78.17.0
Llama-2DER++ 2%1.9×82.97.6
Llama-2DER++, matched training time4.1×83.07.0
Llama-2Tether-KL4.1×83.23.6

Even with the additional training time, DER++ changes more of the earlier decisions than Tether-KL on both models, at similar final accuracy. The paired difference in drift, computed as Tether-KL minus DER++ at matched training time, is −3.5 points on T5-large (95% CI: −4.5 to −2.4) and −3.4 points on Llama-2 (95% CI: −4.0 to −2.8), and Tether-KL has the lower drift in every pair. We will correct Tables 3 and 7 in the revision and add this comparison.

2. Assumption of Theorem 1 and the experimental setup

Assumption 1 of Theorem 1 requires the records of each class to be independent draws from the task distribution. The benchmark provides no inputs for a task beyond its training and test sets, so in the submission we took the records from the training inputs, which are the teacher's own training data. This is why the paper states that the theorem does not cover the reported runs, and we agree that the paper should explain this more clearly. Remark 1 of the paper says that the assumption is met when the records are drawn from inputs outside the teacher's training data (lines 946 to 948). In the new experiments, we draw the records in this way for 13 of the 15 tasks. For each of these tasks, the candidate inputs are sampled uniformly at random from the rows of the original dataset's training split that appear in neither the benchmark's training file nor its test file. After the teacher is trained, the records are drawn at random from the admitted candidates within each class that the teacher predicts. This sampling design is the one that Assumption 1 describes. The two smallest tasks have no such rows and still use training-set records. The results stay close to those of the submission. The drift of Tether-KL on T5-large is 7.0%, compared with 6.7% in the submission. The per-record guarantees of Propositions 1 and 2 and the audit bound of Proposition 5 do not depend on this assumption.

The admission experiment in point 3, the Tether-TKL runs on Llama-2 in point 4, and the record-count ablation in point 8 instead use records held out of the training file, so these runs train on 5% to 13% fewer rows.

AuthorsOpenReview ↗

Response to Reviewer ar1W (Part 2/3)

3. Low-margin inputs excluded by the admission rule

We tested directly whether admitting the low-margin inputs as records would help, using Tether-KL on T5-large. Removing the admission rule did not produce a clear improvement in overall drift in this experiment. The drift is 7.1% without the admission rule and 7.2% with it, and the 95% confidence interval of the paired difference, −0.7 to +0.6 points, includes zero. The low-margin inputs are those on which the teacher itself is uncertain, and they are where every method changes decisions most often. In the main T5-large runs, the replay methods change 44% to 47% of the decisions on the excluded inputs, and Tether-KL changes 37%, which is the lowest. These results suggest that continued distribution matching may help reduce drift on low-margin inputs. In the Amazon to Yelp experiment that the reviewer cites, the excluded inputs contribute 8.2 points of drift under Tether-KL and 12.0 under Tether-Margin (Appendix D). These inputs remain the least protected under every method, and we will state this in the limitations section of the revision.

4. Relation to DER and LwF, and the practical value of Margin and TKL

The loss of Tether-KL is the standard forward-KL distillation loss, as the paper says (line 71). The paper contributes the exact KL radius within which a recorded decision cannot change; the Tether framework, in which a continuous loss bounds the record drift of every earlier task (Eqs. 7 and 8); and two new losses that vanish on decision-preserving regions. Tether-KL differs from LwF in that it distills on stored inputs of the earlier tasks and not on new-task inputs, and from DER in that every earlier task has its own term in the objective at every step, whereas DER draws its replay batches from one buffer that pools all earlier tasks.

To compare Tether-KL with DER directly, we ran our DER++ baseline without its label term. We call this variant label-free DER. The drift of Tether-KL is lower on both models (7.0% vs. 13.8% on T5-large, and 3.6% vs. 5.6% on Llama-2) and in every pair, and its final accuracy is higher. We also gave label-free DER the additional replay batches of the matched DER++ runs in point 1, while keeping its buffer at 2%. Its drift then falls to 10.1% on T5-large and 4.8% on Llama-2, both still above the 7.0% and 3.6% of Tether-KL.

Tether-KL has the lowest drift of the three instances, so we recommend it whenever the teacher's distribution can be stored. With the corrected times, Tether-TKL no longer trains faster than Tether-KL, and the revision will not present it as the cheaper option. The value of Tether-Margin and Tether-TKL is that they show what the guarantees on the recorded decisions achieve on their own (Propositions 1 and 2). Both instances stop penalizing a record once its decision is guaranteed, and this alone yields a drift close to or below that of DER++ (Table 3). At matched training time, Tether-TKL changes fewer of the earlier decisions than DER++ on Llama-2 (4.8% vs. 7.0%). In the generation experiment, Tether-TKL takes 3.2× the time of IncLoRA, compared with 2.1× for DER++, but it keeps 53% of the earlier free-form outputs unchanged, whereas DER++ keeps 38% (Appendix F.4). Tether-Margin is intended for cases where the teacher's distribution cannot be stored or where no real input may be kept. It stores only the recorded answer and its margin. With synthetic records, which retain no real task inputs, its drift is 12.6% on T5-large and 9.6% on Llama-2, lower on both models than that of labeled replay with a 2% buffer (Table 5).

5. Source and recency of the baselines

The paper's conclusions about drift rest only on baselines that we ran ourselves under one protocol. These are IncLoRA, Replay, and DER++, and now also label-free DER and DER++ at matched training time. The published numbers are listed for reference. They cannot be used to compare drift, because drift requires the per-example decisions of each model. The gap that the reviewer cites in Table 9 (79.1 vs. 57.8 for Replay) is a difference between protocols, and we agree with the reviewer that the published rows are not directly comparable with ours.

As for more recent baselines, DER and DER++ are the replay methods closest to ours, and our DER++ is the adaptation to LLMs by He et al. (2024). We also reproduced O-LoRA (Wang et al., 2023) and LwF on T5-large under the same protocol (Tables 5 and 9). On the fifteen-task sequences, O-LoRA changes 21.4% of the earlier decisions, compared with 6.7% for Tether-KL (Table 5).

AuthorsOpenReview ↗

Response to Reviewer ar1W (Part 3/3)

6. Full fine-tuning

We ran full fine-tuning of T5-large on Order 4 with three seeds, as the reviewer suggests.

MethodAccDrift
Sequential fine-tuning71.425.5
Replay 2%78.817.0
DER++ 2%79.112.9
Label-free DER 2%78.08.6
Tether-KL78.27.6

Tether-KL again has the lowest drift. It changes 7.6% of the earlier decisions, compared with 12.9% for DER++ and 8.6% for label-free DER.

7. Open-domain question answering, multi-turn dialogue, and long context

We have not evaluated the method on open-domain question answering, multi-turn dialogue, or long-context question answering. Beyond the classification benchmark, the paper evaluates the method on six generation tasks with Llama-3-8B-Instruct, where the same losses are applied to the teacher's token-level decisions (Appendix F). On these tasks, Tether-KL leaves 79% of the earlier free-form outputs identical to the teacher's, whereas DER++ leaves 38%. The paper states that this experiment covers only short-prompt tasks and does not validate long-context generation (lines 482 to 483), and that the method does not yet handle a new task that contradicts an old rule (lines 483 to 485). We will make this scope more explicit in the revision.

8. Number of records per task

The table below gives the record-count ablation on the main benchmark that the reviewer asks for, with Tether-KL on T5-large and Order 4. Each entry is the mean of three seeds. Training time includes record construction and is given relative to the setting with 48 records.

Records per task12244896192
Acc76.876.677.477.677.6
Drift11.18.48.26.26.1
Training time0.97×0.93×1.00×0.91×0.92×

More records per task give lower drift. Using more records barely changed training time in this experiment because we kept the number replayed per step fixed. A larger budget therefore costs storage and not training time. The value 48 is an empirical choice, as the reviewer says. On this benchmark, we chose it so that Tether stores fewer examples than the baselines. With 48 records for each of the 15 tasks, the memory holds 720 records, compared with 1,113 examples in the 2% buffer, and the table shows that this choice is conservative for Tether.

We would also welcome pointers to the passages that were hard to follow, so that we can revise them. We would be glad to discuss any of these points further.

Dear Reviewer ar1W, thank you again for your feedback. We would appreciate hearing whether our response and the revised paper have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification.

Reviewer_3MZFOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes using decision drift (the portion of predictions changed in either direction after model updates) as an anchor to mitigate forgetting. The paper derives critical radius below which their won't be decision drift and incorporate into loss functions and proposed Tether-TKL and Tether-Margin.

The experiment results do not show clear improvement over DER++ in term of final accuracy, but improves it on the drift metrics, which aligns with the purpose of the approach; the drift metrics is also properly motivated in introduction.

Critical strengths
  • The derivation of the critical radius that preserves earlier predictions is sound, and the approach is well-motivated from the proposed approach.
  • The resulting approach is simple and computationally efficient compared to baselines such as DER++.
Critical weaknesses

Baselines are not sufficient; the advantages claimed are factually true about DER++, but not DER.

The authors compared with DER++ with uses labels, but does not discuss DER (which is proposed in the same paper as DER++). If DER performs equally well, then it weaken's the label-free advantage of the proposed approach claimed in L74 of the introduction.

I am not sure what is the novelty of Tether-KL compared to DER (other than the MSE loss for logit differences in the original DER, which is itself uncommon in distillation), which achieves lowest drift metrics in Table 3. The other variants Tether-TKL and Tether-Margin underperforms the approach. If DER achieves similarly lower drift, I wonder what is the edge of the proposed approach.

Limited Novelty

In addition to DER, the author also cited Jiang et al. Churn reduction via distillation, where "churn" is similarly defined as the difference to the model predictions before update rather than absolute accuracy. They also show distillation can reduce "churn", which is close to the drift metrics reported in the paper.

Questions for authors

Please answer how the Tether-KL approach is novel, and how it performs, compared to DER.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 3MZF (Part 1/3)

We thank the reviewer for the comments, and especially for pointing out that the paper compares Tether-KL with DER++ but not with DER, which is its closest baseline. We have now run this comparison on T5-large and Llama-2, with a DER baseline that we call label-free DER. Tether-KL changes fewer of the earlier decisions than label-free DER on both models, at DER's default budget (7.0% vs. 13.8% on T5-large, and 3.6% vs. 5.6% on Llama-2) and when DER is given three to four times as much replay (7.0% vs. 10.1%, and 3.6% vs. 4.8%). On the question of novelty relative to DER, the loss of Tether-KL is the standard forward-KL distillation loss, which the LLM adaptation of DER also uses. The paper presents this loss as an existing retention penalty (line 71), and we claim no novelty for it. The paper contributes the exact KL radius within which a recorded decision cannot change; the Tether framework, in which a continuous loss bounds the record drift of every earlier task (Eqs. 7 and 8); and two new losses that vanish on decision-preserving regions.

1. How Tether-KL performs compared with DER

Label-free DER is the paper's DER++ baseline with the label term removed. For each task, it stores 2% of the training inputs, drawn uniformly at random, together with the answer generated by the model at the end of training on that task and that model's full-vocabulary logits at the answer positions. No ground-truth labels are stored in the buffer or used in the replay loss, and the replay schedule, the total replay weight, and all other settings are those of DER++. Like our DER++ baseline, which follows He et al. (2024) as described in Appendix B.1, it departs from the original DER (Buzzega et al., 2020) in ways that bring it closer to Tether-KL. The logits are recorded at the end of each task rather than during training, so the stored targets come from the teacher against which drift is measured. Because its loss is the cross-entropy to the stored distribution, not the squared error on the logits, the baseline, like Tether-KL, minimizes a forward KL divergence, here taken over the vocabulary at each answer position.

In the two tables below, Acc is the final average accuracy, Acq is the accuracy on each task right after it is learned, and Drift is the percentage of earlier-task test inputs on which the final model's decision differs from the teacher's, as in Table 3 of the paper. "Stored examples" is the total number of examples held in memory after the fifteen tasks. Time is the training time relative to IncLoRA, including record construction. The rows with additional replay use four times as many replay batches per step on T5-large and three times as many on Llama-2, with the 2% buffer unchanged. The last row of each table gives Tether-KL a 2% record buffer, which part 2 discusses. Every row is the mean of nine runs, the three task orders with three seeds each.

In both tables, Tether's records are inputs that the teacher was not trained on. For Tether-KL with 48 records per task, the records are rows of the original datasets that appear in neither the benchmark's training file nor its test file; the two smallest tasks have no such rows and keep their training-set records. For Tether-KL with 2% records, the records are rows held out of the training file, so these runs train on 13% fewer rows.

T5-large, Orders 4–6.

MethodStored examplesAccAcqDriftTime
Label-free DER 2%1,11375.778.213.81.5×
Label-free DER 2%, additional replay1,11376.578.310.12.7×
DER++ 2%1,11376.278.212.71.6×
Tether-KL, 48 records per task72078.178.37.03.1×
Tether-KL, 2% records1,11377.578.07.02.6×

Llama-2-7B-chat, Orders 4–6.

MethodStored examplesAccAcqDriftTime
Label-free DER 2%1,11382.783.05.62.0×
Label-free DER 2%, additional replay1,11383.183.14.84.2×
DER++ 2%1,11382.983.07.61.9×
Tether-KL, 48 records per task72083.283.13.64.1×
Tether-KL, 2% records1,11382.482.33.93.8×

To compare two methods, we pair the runs that share a task order and a seed, which gives nine pairs. At the default budget, the paired difference in drift, computed as Tether-KL minus label-free DER, is −6.8 points on T5-large (95% CI: −8.8 to −4.8) and −2.0 points on Llama-2 (95% CI: −2.5 to −1.5), and Tether-KL drifts less in every pair. As the Acq column shows, the two methods reach about the same acquisition accuracy, so the lower drift does not come at the expense of learning the new tasks. Tether-KL also has the higher final accuracy on both models.

AuthorsOpenReview ↗

Response to Reviewer 3MZF (Part 2/3)

Tether-KL replays records of every earlier task at every step, so it trains for roughly twice as long as label-free DER at its default budget. To test whether the difference in drift is simply a matter of more replay, we ran label-free DER with the additional replay described above, which brings its training time close to that of Tether-KL. The additional replay narrows the gap from 6.8 to 3.1 points on T5-large and from 2.0 to 1.2 points on Llama-2 without closing it in any pair, and the acquisition accuracy stays about the same.

As for the concern about line 74, DER does not perform equally well on the paper's benchmark on either model, whether at the default budget or with additional replay. Line 74 itself is accurate for DER++, as the reviewer notes, but it leaves out DER, which retains no per-example labels either. We will revise the sentence to say that Tether-KL shares this property with DER. In the revision, we will also add label-free DER to the main tables at both budgets and discuss DER as the closest baseline in the introduction. Label-free DER will be part of the released code.

2. What is new relative to DER

As a training procedure, Tether-KL differs from DER in its objective and in the distribution that it matches. Every earlier task has its own retention term at every step (Eq. 8 and Appendix B.1), and the matched distribution is over complete answers (Eq. 1) rather than over the vocabulary. The paper also adds a guarantee that DER does not have: by Proposition 2 and Eq. 12, the loss on a task's records bounds the share of its recorded decisions that may have changed.

In Tether-KL, the amount of replay and the retention weight grow with the number of earlier tasks, because every earlier task keeps its own term, each with its own bound on record drift (Eq. 7). In DER, one replay batch and one replay weight are divided among all earlier tasks in proportion to their share of the buffer, so each task's portion shrinks as the sequence grows.

Part 1 shows that additional replay for DER accounts for part of the gap but not for all of it. We also gave Tether-KL the allocation of DER, a 2% record buffer divided across tasks in the same way (the rows with 2% records in the tables of part 1). Its drift stays at 7.0% on T5-large and moves from 3.6% to 3.9% on Llama-2, and both values remain below those of label-free DER with additional replay (10.1% and 4.8%). This comparison changes the training data as well as the record allocation, because the 2% records are held out of the training file, so it does not isolate the effect of allocation alone.

We also ran a control with Tether-KL on T5-large and Order 4, with three seeds, in which the training data, the record bank, and the number of records replayed per step are the same in every setting. Pooling the separate replay terms into one term did not produce a clear change in drift in this experiment (7.0% pooled vs. 7.3% separate at the higher total weight, and 8.9% vs. 8.9% at the lower one). Lowering the total weight from one unit for each earlier task to one unit in total raised drift in every seed, by 1.7 points on average. Finally, we gave label-free DER with additional replay the same growing weight, so that its total replay weight also equals the number of earlier tasks. Over the nine runs on T5-large, its drift fell from 10.1% to 9.0%. With the same weight, the drift of Tether-KL (7.0%) is still lower by 2.0 points on average (95% CI: 0.4 to 3.6), in seven of the nine pairs, and its final accuracy is higher in every pair, by 0.9 points on average.

The second difference is the matched distribution. Tether-KL matches the distribution over complete answers, the one that defines the decision and the radius, whereas the LLM adaptation of DER matches token logits along one generated answer. The two coincide for a classifier head but not when a generative model scores a finite answer set. Proposition 2 gives, for each stored input, the exact divergence from the teacher below which the recorded decision cannot change, and no larger radius has this property. For Tether-KL, it says how far the loss on a record can rise before its decision can change.

AuthorsOpenReview ↗

Response to Reviewer 3MZF (Part 3/3)

3. Tether-Margin and Tether-TKL

We agree that Tether-KL has the lowest drift of the three instances. Tether-Margin and Tether-TKL answer a different question: how much retention the guarantee on the records provides on its own. They stop penalizing a record once the model is in a region where its decision is guaranteed. In the sweep of Section 3.3, all three instances preserve every recorded decision, so their differences in drift arise on inputs that are not stored (lines 356 to 362). On Llama-2, where the three instances share the same coefficient, Table 3 reports a drift of 19.9% for unprotected training (IncLoRA), 7.8% for Tether-Margin, and 4.9% for Tether-TKL, compared with 4.1% for Tether-KL, which keeps penalizing the divergence inside these regions. Tether-Margin needs the least information from the teacher, since it stores only the recorded answer and its margin (Table 1), and it is the instance that the paper also runs with synthetic records, which retain no real task inputs (Tables 2 and 5).

4. Relation to churn reduction (Jiang et al., 2022)

Decision drift is prediction churn, measured in continual fine-tuning against the model at the end of each earlier task. The paper introduces it through the work on churn and negative flips (lines 48 to 52), cites Jiang et al. for controlling churn by distillation (line 462), and states that Tether builds on this work (lines 464 to 466). What the paper adds, as the related-work section states (lines 466 to 468), is the exact forward-KL boundary for preserving each recorded decision, and objectives with zero-loss regions that let the output distribution change without changing the decision.

Jiang et al. show that distillation from the base model is equivalent to a constraint on a soft churn, the expected divergence between the output distributions of the two models. They adopt this soft quantity for tractability, in place of the fraction of predictions that change, and their analysis does not say how small the divergence must be for a prediction to stay the same. Proposition 2 answers exactly this question. For each input, it gives the largest divergence from the teacher below which the decision cannot change. As a result, the loss on a task's stored inputs bounds the share of their decisions that may have changed (Eq. 12). Our two new losses go one step further and stop penalizing an input once its decision is guaranteed (Propositions 1 and 2), whereas distillation keeps pulling the whole output distribution toward the teacher.

The setting differs as well. Jiang et al. retrain one classifier on the old and the new labeled data of a single task. In continual fine-tuning, the earlier data is no longer available, every earlier task has its own teacher, and the loss can be applied only to a small stored set. We will add this comparison to the introduction.

We thank the reviewer again for these comments.

Dear Reviewer 3MZF, thank you again for your feedback. We would appreciate hearing whether our response and the revised paper have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification.

Reviewer_kZ29OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes decision drift, the fraction of old-task inputs on which the current model's decision differs from the teacher's (the model at the end of that task), as the retention target for continual fine-tuning (eq. 3). Drift is label-free and upper-bounds both the harmful-flip rate and the accuracy loss (eq. 4). Tether stores a small set of records per task (inputs plus teacher outputs, 48 per task here) and adds any continuous loss that upper-bounds record drift (eqs. 7-8). Three instances: Tether-KL (forward KL to the stored teacher distribution over the answer set plus a residual outcome), Tether-Margin (hinge on the recorded answer's probability margin, Proposition 1), and Tether-TKL (KL thresholded at a fraction eta of a per-record critical radius). Proposition 2 gives that radius in closed form, eps*_v = a log(2a/(a+b)) + b log(2b/(a+b)) >= m_v^2/2, the smallest forward KL at which the recorded decision can change, and shows it is sharp. Proposition 3 separates a record term from a transfer term for unrecorded inputs; Proposition 5 gives an exact Clopper-Pearson audit of task drift on held-out inputs; Theorem 1's population guarantee requires an independent stratified record design that the reported runs do not satisfy (stated, lines 478-480).

Validation on the O-LoRA benchmark (T5-large, Llama-2-7B-chat; three orders x three seeds) and six generation tasks (Llama-3-8B-Instruct). Final average accuracy ranges from 1.1 points below to 1.5 above DER++ (Table 2: 77.9 vs 76.4 on T5 O4-6, 83.4 vs 82.9 on Llama, 78.9 vs 80.0 on T5 O1-3). Drift on up to 2,000 held-out inputs per task (Table 3): KL 6.7 / 4.1, TKL 10.4 / 4.9, Margin 10.4 / 7.8 vs DER++ 12.9 / 7.6 and replay 18.0 / 12.7, at 2.3-12.1x the unprotected training time vs 1.6-1.9x for DER++. Propositions 1, 2, 3 and 5 are correct as stated and the code implements the losses exactly; the empirical claims are stated precisely, and the contribution is weakened by the absence of the label-free baseline Tether-KL most resembles (DER without labels) and by unmatched compute.

Critical strengths
  • Soundness: Proposition 2 is a correct, sharp, closed-form result with a clean proof idea: the nearest decision-changing distribution moves mass only between the recorded answer and its strongest competitor to a tie at (a+b)/2, giving eps* = a log(2a/(a+b)) + b log(2b/(a+b)), and Pinsker on the two-point restriction gives eps* >= m_v^2/2. It yields the explicit drift bounds of eqs. 12 and 14 and the thresholded-KL instance, and the shipped code (tkl.py, radius_from_ab) implements exactly this formula. Proposition 3 (eq. 15) is also correct: a flip at X with its record intact requires m_theta_t(X; c_X) <= (1 - kappa) m_v(X) + Delta(v(X)) - Delta(X), which the absolute-value transfer term covers. The paper is also precise about what is guaranteed: record retention deterministically (Props. 1-2), task drift only via the held-out audit (Prop. 5), and Theorem 1 not at all for these runs (lines 478-480), with one disclosed exception noted below.
  • Significance: Decision drift exposes a failure of the standard metric. Table 3 shows labelled replay with a 2% buffer changing 18.0% (T5) and 12.7% (Llama) of earlier decisions while old-task accuracy moves by 2.5 and 1.6 points, and DER++ changing 12.9% / 7.6% at a 2.2 / 0.2 point accuracy change. For deployed systems where downstream consumers depend on earlier outputs, this is the quantity that matters, and the paper gives a label-free way to measure it (Prop. 5) and to constrain it without retaining per-example labels; note that the reported runs do use aggregate training-label statistics to set record weights (lines 298-299, 1148-1150; benchmark.py build_records passes labels to records.class_prior), so "label-free" applies to the audit and to what is stored, not to record construction.
Critical weaknesses
  • Significance and soundness of the headline comparison (central): The baseline Tether-KL most closely resembles is not run, and the compute is unmatched. Tether-KL is forward-KL distillation to stored teacher output distributions on stored past inputs; that is DER (Buzzega et al. 2020, logit replay without labels) restricted to the answer set plus bot with 48 records per task, and the exemplar-distillation term of iCaRL. The experiments compare against DER++ (labelled CE plus logit matching, --der-kd 0.5 with a 2% buffer, scripts/run_benchmark_llama2.sh line 27) and labelled replay only; DER appears solely in the related-work sentence (line 447) and functional regularisation on stored inputs (Pan et al. 2020, line 446) is likewise not run, and the shipped buffer stores gold targets for every row (replay.py lines 17-33), so a genuinely label-free DER would store only past inputs and teacher outputs. At the same time Table 3 gives KL at 9.0x (T5) and 12.1x (Llama) the unprotected training time against 1.6x / 1.9x for DER++ and TKL at 2.7x / 3.9x, and line 440 states the comparisons "follow the stated training schedules rather than matched compute budgets." "KL changes about half as many decisions as DER++" (line 319) is therefore a comparison against the wrong baseline at an unequal budget; TKL at 4.9 vs 7.6 on Llama at roughly twice DER++'s cost is the fairest evidence in the paper and should be the headline.
  • Soundness (secondary): Appendix B.5 (lines 1193-1203) states that T5-large Tether-TKL's lambda was selected on the same Amazon and Yelp test inputs used for its audit, so Proposition 5's exact coverage does not apply to those two cells. This is disclosed, but those cells should be marked in Table 8 and, if possible, re-audited on an independent sample.
Questions for authors

Questions that could change my assessment

  1. Please add a genuinely label-free DER (a buffer of past inputs and stored teacher output distributions only, no gold targets; the current replay buffer stores y and tgt, replay.py lines 17-33) with the same 2% buffer, and Tether-KL with a 2% record buffer, so that the label-free comparison is like-for-like in both directions. Score-change criterion: Rating rises to 4 only if Tether-KL at 48 records drifts less than label-free DER at 2% on both models (and question 2 is also satisfied). If DER matches Tether-KL, the framework's empirical contribution reduces to Margin/TKL's skipping and the Rating stays at 3.
  2. Please report DER++ (or replay) at a compute budget matched to Tether-TKL and to Tether-KL (larger buffer or more replay steps), with drift and final accuracy. Score-change criterion: Soundness rises to 3 if the compute-matched comparison is provided and the drift claims survive it; if matched-compute DER++ matches KL's drift, the "half as many decisions" claim must be qualified.

Clarifications and minor points (do not affect my assessment)

  1. State the aggregate record count vs buffer size for the four-task orders O1-3, not only O4-6 (lines 338-339).
  2. Table 3: Margin on Llama drifts 7.8 vs DER++ 7.6; the abstract's "Margin's drift is broadly comparable" is fair, but keep "replay" distinct from DER++ in the sentence at lines 30-31.
  3. tkl.py floors a and b at 1e-9 and the radius at 1e-6; mention these guards in Appendix B.
  4. Add the drift numbers for the synthetic-record variant to Table 3; it is the variant a practitioner without stored data would use.
  5. The MNLI-to-RTE conflict experiment shows retention needs a release mechanism for contradicted records (line 485); a simple criterion would be a natural ablation for future work.
  6. Cite iCaRL (Rebuffi et al. 2017) for distillation on stored exemplars.
5 comments
AuthorsOpenReview ↗

Response to Reviewer kZ29 (Part 1/3)

We thank the reviewer for the careful reading of the paper and the code, and for identifying the two comparisons that the submission was missing: a DER baseline that stores no labels, and DER++ at a compute budget matched to Tether's. We ran both on T5-large and Llama-2, with the results shown in Tables A and B. The main result is that Tether-KL keeps its drift advantage at approximately matched training time and similar final accuracy. When training time is matched, DER++ changes 10.5% of the earlier decisions on T5-large and Tether-KL changes 7.0%; on Llama-2, the two figures are 7.0% and 3.6%. Tether-KL also drifts less than label-free DER on both models (7.0% vs. 13.8% on T5-large, and 3.6% vs. 5.6% on Llama-2).

In both tables, Acc is the final average accuracy, and Acq is the accuracy on each task right after it is learned. Drift is the percentage of earlier-task test inputs on which the final model's decision differs from the teacher's. Acq and Drift are computed as in Table 3 of the paper. The lowest drift in each table is in bold. "Stored examples" is the total number of examples held in memory after the fifteen tasks. Time is the training time relative to IncLoRA, including record construction. All methods were timed on the same server. Every row of the two tables is the mean of nine runs, the three task orders with three seeds each. Each paired difference reported below is computed between one run of each method, where the two runs share the same task order and seed, so each comparison has nine pairs; the intervals are two-sided 95% Student-t intervals.

Table A. T5-large, Orders 4–6.

MethodStored examplesAccAcqDriftTime
Replay 2%1,11375.978.516.51.6×
DER++ 2%1,11376.278.212.71.6×
Label-free DER 2%1,11375.778.213.81.5×
DER++, matched training time (8% buffer, 4× replay batches)4,45278.378.410.53.0×
Tether-KL, 48 records per task72078.178.37.03.1×
Tether-KL, 2% records1,11377.578.07.02.6×

Table B. Llama-2-7B-chat, Orders 4–6.

MethodStored examplesAccAcqDriftTime
Replay 2%1,11381.482.812.72.0×
DER++ 2%1,11382.983.07.61.9×
Label-free DER 2%1,11382.783.05.62.0×
DER++, matched training time (6% buffer, 3× replay batches)3,33983.082.87.04.1×
Tether-KL, 48 records per task72083.283.13.64.1×
Tether-KL, 2% records1,11382.482.33.93.8×
Tether-TKL, 48 records per task72082.382.34.84.2×

Training time of Tether-KL in Table 3

Thank you for pointing this out. The 9.0× reported for Tether-KL on T5-large in Table 3 is a mistake in that entry. It was timed on an earlier variant that back-propagates through all 48 records of every earlier task at every step, whereas all Tether-KL results in Tables 2 and 3 use 16 of the 48 records per step. The correct cost is 3.1× when record construction is included, as in Table A, or 2.5× for training alone, which is the basis of the other entries in Table 3. We will correct this entry, and the corresponding row of Table 7, in the revision.

On Llama-2, the runs in Table B use 4 records per step instead of the 16 used in the paper. Their drift is 3.6%, compared with 4.1% in Table 3, and their acquisition accuracy is about the same (83.1 and 83.2). The cost is 4.1× instead of 12.1×.

Records and baseline reruns in Tables A and B

In Tables A and B, Tether's records are inputs that the teacher was not trained on. For Tether-KL with 48 records per task, the records are rows of the original datasets that appear in neither the benchmark's training file nor its test file; the two smallest tasks have no such rows and keep their training-set records. For Tether-KL with 2% records and for Tether-TKL, the records are rows held out of the training file, so these runs train on 5% to 13% fewer rows. With the training-set records used in the submission, the drift of Tether-KL on T5-large is 6.7%, as in Table 3.

The Replay and DER++ rows of Table A are from reruns with the amount of replay specified in Appendix B.1. Their drift is 16.5% for Replay (18.0% in the submission) and 12.7% for DER++ (12.9%), and we will update these entries in the paper.

AuthorsOpenReview ↗

Response to Reviewer kZ29 (Part 2/3)

Q1. Label-free DER with a 2% buffer, and Tether-KL with a 2% record buffer

Tether-KL with 48 records per task drifts less than label-free DER with a 2% buffer on both models. The same holds when Tether-KL uses a 2% record buffer.

We implemented label-free DER by removing the label term from our DER++ baseline. For each task, it stores 2% of the training inputs, drawn uniformly at random, as reservoir sampling does in the original DER. For each stored input, it keeps the answer generated by the model at the end of training on that task, and that model's logits at the answer positions. No ground-truth labels are stored in the buffer or used in the replay loss. During replay, the loss is the cross-entropy between the stored distribution and the current output at the same positions. The replay schedule and the total replay weight are the same as in DER++, but all of the weight is placed on the logit term. Our implementation differs from the original DER (Buzzega et al., 2020) in two ways: the logits are recorded at the end of each task rather than during training, and the loss is the cross-entropy rather than the mean squared error. We will add this baseline and the matched-budget settings to the released code.

On T5-large, the drift is 7.0% for Tether-KL and 13.8% for label-free DER. The paired difference is −6.8 points (95% CI: −8.8 to −4.8), and Tether-KL has the lower drift in every pair. Its final accuracy is also higher (78.1 vs. 75.7). On Llama-2, Tether-KL drifts 3.6% and label-free DER drifts 5.6%, a paired difference of −2.0 points (95% CI: −2.5 to −1.5). Tether-KL again has the lower drift in every pair, and its final accuracy is 83.2, compared with 82.7 for label-free DER.

With a 2% record buffer, Tether-KL stores as many examples as label-free DER (1,113). Its drift is then 7.0% on T5-large and 3.9% on Llama-2, still below that of label-free DER on both models (13.8% and 5.6%). In this variant, the records are allocated in proportion to task size, as in the 2% buffer.

Q2. DER++ at a matched compute budget

When DER++ is given a matched compute budget, Tether-KL still has the lower drift on both models, at similar final accuracy. We measure compute as wall-clock training time on the same server. To match it, we gave DER++ more replay batches per step and a proportionally larger buffer, until it trained for about as long as Tether-KL with 48 records per task. These runs are the rows labeled "matched training time" in Tables A and B, and we call them matched DER++ below.

On T5-large, Tether-KL trains for 162 minutes and matched DER++ for 156 minutes. With about the same training time, matched DER++ still changes 10.5% of the earlier decisions, while Tether-KL changes 7.0%. The paired difference is −3.5 points (95% CI: −4.5 to −2.4), the drift of Tether-KL is lower in every pair, and the two methods reach about the same final accuracy (78.1 and 78.3). The picture is the same on Llama-2, where Tether-KL trains for 355 minutes and matched DER++ for 350 minutes. Tether-KL changes 3.6% of the earlier decisions and matched DER++ changes 7.0%, a paired difference of −3.4 points (95% CI: −4.0 to −2.8). Tether-KL again has the lower drift in every pair, and the final accuracies are 83.2 and 83.0. Tether-KL also stores fewer examples than matched DER++ (720 vs. 4,452 on T5-large, and 720 vs. 3,339 on Llama-2).

The question also asks for a budget matched to Tether-TKL. Because Tether-TKL takes 362 minutes on Llama-2, nearly the same as Tether-KL, the same matched DER++ runs serve as its comparison. Tether-TKL changes 4.8% of the earlier decisions, compared with 7.0% for matched DER++, and its drift is lower in every pair. The trade-off is that Tether-TKL has a lower acquisition accuracy than matched DER++ (82.3 vs. 82.8). On T5-large, the drift of Tether-TKL (10.4% in Table 3, and 9.7% with the new records) is about the same as that of matched DER++ (10.5%), so on this model we claim lower drift at matched training time for Tether-KL only.

We also gave label-free DER the same additional replay as matched DER++, while keeping its 2% buffer. Its drift falls from 13.8% to 10.1% on T5-large and from 5.6% to 4.8% on Llama-2, but Tether-KL remains lower on both models (7.0% and 3.6%).

We will add Tables A and B to the revision, and we will restate the "half as many decisions" statement at line 319 with the matched numbers. At matched training time, Tether-KL changes 33% fewer decisions than DER++ on T5-large (7.0% vs. 10.5%) and 49% fewer on Llama-2 (3.6% vs. 7.0%). Against DER++ at its default budget, the reductions are 45% (7.0% vs. 12.7%) and 52% (3.6% vs. 7.6%). We also agree that the headline should be the fairest comparison. Now that the budgets are matched, we will lead with the Llama-2 comparison at matched training time in Table B, which covers both Tether-KL and Tether-TKL.

AuthorsOpenReview ↗

Response to Reviewer kZ29 (Part 3/3)

Weakness 1 (central): the headline comparison and the relation to DER

The new runs under Q1 and Q2 provide the label-free baseline and the matched compute that this point calls for. We agree with the reviewer about the relation to prior work, and we will state explicitly in the revision that Tether-KL is distillation on stored inputs and belongs to the same family as DER, functional regularization on stored inputs (Pan et al., 2020), and the exemplar-distillation term of iCaRL. The paper contributes the exact KL radius within which a recorded decision cannot change; the Tether framework, in which a continuous loss bounds the record drift of every earlier task (Eqs. 7 and 8); and two new losses that vanish on decision-preserving regions. As a training procedure, Tether-KL differs from DER in its objective and in the distribution that it matches. In Tether-KL, every earlier task has its own retention term at every step (Eq. 8 and Appendix B.1), whereas DER shares one replay batch and one replay weight among all earlier tasks. The distribution that Tether-KL matches is over complete answers (Eq. 1) rather than over the vocabulary.

Weakness 2 (secondary): Appendix B.5 and Table 8

As suggested, we re-audited the affected rows on an independent sample. Of the 153 task rows behind the T5-large Tether-TKL entry of Table 8, 27 are Amazon or Yelp rows. For each of them, we used 4,600 test inputs that had been used neither for the reported audit nor for selecting λ. Averaged over the 27 rows, the drift on the independent sample is 12.6%, close to the 12.4% reported, and the drift on admitted inputs is 4.6% on both samples. In the revision, Table 8 will use the rates and the bounds from the independent sample for these rows, so the coverage of Proposition 5 applies to them as well.

The term "label-free" (raised under Strengths)

We agree. The record weights use class proportions that are computed from the training labels, so we will restrict the term "label-free" to the audit and to what is stored.

Minor points

We respond to the six minor points in the reviewer's order.

  1. On Orders 1–3, Tether stores 192 records, and the 2% buffer holds 660 examples. On Orders 4–6, the two numbers are 720 and 1,113. We will state both at lines 338–339.
  2. We will revise lines 30–31 so that Replay and DER++ are named separately. In Table 3, the drift on Llama-2 is 7.8% for Tether-Margin, 7.6% for DER++, and 12.7% for Replay, so Tether-Margin is close to DER++ and below Replay.
  3. We will state the numerical floors on a and b (1e-9) and on the radius (1e-6) in Appendix B.
  4. We will add the synthetic-record variant to Table 3. It retains no real task inputs, yet it drifts less than Replay on both models (12.6% vs. 16.5% on T5-large, and 9.6% vs. 12.7% on Llama-2).
  5. We agree that a release criterion for contradicted records would be a natural ablation, and we will note it as future work at line 485.
  6. We will cite iCaRL (Rebuffi et al., 2017) for distillation on stored exemplars.

We thank the reviewer again for the detailed and constructive review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.