- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Tether, which extends the retention goal in continual fine-tuning from old-task accuracy to decision drift relative to earlier models. The authors derive an exact per-record KL radius for decision preservation and introduce Margin and thresholded KL losses with zero-loss regions. Classification and generation experiments show that the framework can reduce changes in earlier behavior while achieving similar final scores.
Critical strengths
The paper clearly explains the often-overlooked issue that unchanged accuracy does not imply unchanged behavior, and provides a label-free, measurable, and auditable decision drift metric with practical value.
The exact KL critical radius provides a clear theoretical basis for preserving recorded decisions.
Critical weaknesses
-
Table 7 shows that, over the fifteen-task sequences, Tether-KL has substantially higher total training time than Replay and DER++ on both T5-large and Llama-2.
-
The statement “Neither provides the independent task-distribution draws of Assumption 1, so the theorem does not cover the reported runs” highlights a gap between the theory and experiments. The authors should explain why the experimental setup does not match the assumptions required by the theory.
-
The admission rule excludes inputs with low margins or invalid teacher answers, yet these inputs account for a large share of the remaining drift. In the Amazon→Yelp experiment, excluded inputs contribute 12.0 percentage points out of Margin’s total drift of 17.8%. The method provides little direct protection for this vulnerable region, and the authors should discuss how to cover these inputs.
-
Tether-KL, which achieves the best retention, uses standard output distillation related to DER/LwF, with a different application setting. The new Margin and TKL instances have higher aggregate drift than KL. In the generation experiment, TKL takes about the training time longer than the baselines. The authors should explain its practical value.
-
The paper states that “IncLoRA, replay and DER++ are local; other baselines are published results,” meaning that several baseline scores are taken from their original papers rather than reproduced under a shared setup. Table 9 also shows large gaps between reproduced and published scores, such as 79.1 versus 57.8 for Replay, although these settings are not directly comparable; this makes cross-paper rankings difficult to interpret. The main local baselines are also relatively old, including DER++ from 2020, so comparisons with more recent methods under a shared protocol are needed.
-
Both classification and generation experiments use LoRA, which restricts the form of parameter updates. It remains unclear whether the retention performance and computational benefits hold under full-parameter fine-tuning. At least one full-parameter fine-tuning experiment would help establish broader applicability.
-
The authors should include open-domain question answering, multi-turn dialogue, or long-context question answering to test the method on more complex outputs and knowledge-update scenarios.
-
The budget of 48 records per task is an empirical choice, with no theoretical justification for this specific value and no record-count ablation on the main benchmark. The only record-count study, MNLI→RTE with T5, shows that drift decreases substantially as the number of records increases, leaving the quality–cost trade-off at 48 unclear. The authors should report retention performance and computational cost across record budgets on the main benchmark.
Questions for authors
See above
4 comments
Response to Reviewer ar1W (Part 2/3)
3. Low-margin inputs excluded by the admission rule
We tested directly whether admitting the low-margin inputs as records would help, using Tether-KL on T5-large. Removing the admission rule did not produce a clear improvement in overall drift in this experiment. The drift is 7.1% without the admission rule and 7.2% with it, and the 95% confidence interval of the paired difference, −0.7 to +0.6 points, includes zero. The low-margin inputs are those on which the teacher itself is uncertain, and they are where every method changes decisions most often. In the main T5-large runs, the replay methods change 44% to 47% of the decisions on the excluded inputs, and Tether-KL changes 37%, which is the lowest. These results suggest that continued distribution matching may help reduce drift on low-margin inputs. In the Amazon to Yelp experiment that the reviewer cites, the excluded inputs contribute 8.2 points of drift under Tether-KL and 12.0 under Tether-Margin (Appendix D). These inputs remain the least protected under every method, and we will state this in the limitations section of the revision.
4. Relation to DER and LwF, and the practical value of Margin and TKL
The loss of Tether-KL is the standard forward-KL distillation loss, as the paper says (line 71). The paper contributes the exact KL radius within which a recorded decision cannot change; the Tether framework, in which a continuous loss bounds the record drift of every earlier task (Eqs. 7 and 8); and two new losses that vanish on decision-preserving regions. Tether-KL differs from LwF in that it distills on stored inputs of the earlier tasks and not on new-task inputs, and from DER in that every earlier task has its own term in the objective at every step, whereas DER draws its replay batches from one buffer that pools all earlier tasks.
To compare Tether-KL with DER directly, we ran our DER++ baseline without its label term. We call this variant label-free DER. The drift of Tether-KL is lower on both models (7.0% vs. 13.8% on T5-large, and 3.6% vs. 5.6% on Llama-2) and in every pair, and its final accuracy is higher. We also gave label-free DER the additional replay batches of the matched DER++ runs in point 1, while keeping its buffer at 2%. Its drift then falls to 10.1% on T5-large and 4.8% on Llama-2, both still above the 7.0% and 3.6% of Tether-KL.
Tether-KL has the lowest drift of the three instances, so we recommend it whenever the teacher's distribution can be stored. With the corrected times, Tether-TKL no longer trains faster than Tether-KL, and the revision will not present it as the cheaper option. The value of Tether-Margin and Tether-TKL is that they show what the guarantees on the recorded decisions achieve on their own (Propositions 1 and 2). Both instances stop penalizing a record once its decision is guaranteed, and this alone yields a drift close to or below that of DER++ (Table 3). At matched training time, Tether-TKL changes fewer of the earlier decisions than DER++ on Llama-2 (4.8% vs. 7.0%). In the generation experiment, Tether-TKL takes 3.2× the time of IncLoRA, compared with 2.1× for DER++, but it keeps 53% of the earlier free-form outputs unchanged, whereas DER++ keeps 38% (Appendix F.4). Tether-Margin is intended for cases where the teacher's distribution cannot be stored or where no real input may be kept. It stores only the recorded answer and its margin. With synthetic records, which retain no real task inputs, its drift is 12.6% on T5-large and 9.6% on Llama-2, lower on both models than that of labeled replay with a 2% buffer (Table 5).
5. Source and recency of the baselines
The paper's conclusions about drift rest only on baselines that we ran ourselves under one protocol. These are IncLoRA, Replay, and DER++, and now also label-free DER and DER++ at matched training time. The published numbers are listed for reference. They cannot be used to compare drift, because drift requires the per-example decisions of each model. The gap that the reviewer cites in Table 9 (79.1 vs. 57.8 for Replay) is a difference between protocols, and we agree with the reviewer that the published rows are not directly comparable with ours.
As for more recent baselines, DER and DER++ are the replay methods closest to ours, and our DER++ is the adaptation to LLMs by He et al. (2024). We also reproduced O-LoRA (Wang et al., 2023) and LwF on T5-large under the same protocol (Tables 5 and 9). On the fifteen-task sequences, O-LoRA changes 21.4% of the earlier decisions, compared with 6.7% for Tether-KL (Table 5).
Response to Reviewer ar1W (Part 3/3)
6. Full fine-tuning
We ran full fine-tuning of T5-large on Order 4 with three seeds, as the reviewer suggests.
| Method | Acc | Drift |
|---|---|---|
| Sequential fine-tuning | 71.4 | 25.5 |
| Replay 2% | 78.8 | 17.0 |
| DER++ 2% | 79.1 | 12.9 |
| Label-free DER 2% | 78.0 | 8.6 |
| Tether-KL | 78.2 | 7.6 |
Tether-KL again has the lowest drift. It changes 7.6% of the earlier decisions, compared with 12.9% for DER++ and 8.6% for label-free DER.
7. Open-domain question answering, multi-turn dialogue, and long context
We have not evaluated the method on open-domain question answering, multi-turn dialogue, or long-context question answering. Beyond the classification benchmark, the paper evaluates the method on six generation tasks with Llama-3-8B-Instruct, where the same losses are applied to the teacher's token-level decisions (Appendix F). On these tasks, Tether-KL leaves 79% of the earlier free-form outputs identical to the teacher's, whereas DER++ leaves 38%. The paper states that this experiment covers only short-prompt tasks and does not validate long-context generation (lines 482 to 483), and that the method does not yet handle a new task that contradicts an old rule (lines 483 to 485). We will make this scope more explicit in the revision.
8. Number of records per task
The table below gives the record-count ablation on the main benchmark that the reviewer asks for, with Tether-KL on T5-large and Order 4. Each entry is the mean of three seeds. Training time includes record construction and is given relative to the setting with 48 records.
| Records per task | 12 | 24 | 48 | 96 | 192 |
|---|---|---|---|---|---|
| Acc | 76.8 | 76.6 | 77.4 | 77.6 | 77.6 |
| Drift | 11.1 | 8.4 | 8.2 | 6.2 | 6.1 |
| Training time | 0.97× | 0.93× | 1.00× | 0.91× | 0.92× |
More records per task give lower drift. Using more records barely changed training time in this experiment because we kept the number replayed per step fixed. A larger budget therefore costs storage and not training time. The value 48 is an empirical choice, as the reviewer says. On this benchmark, we chose it so that Tether stores fewer examples than the baselines. With 48 records for each of the 15 tasks, the memory holds 720 records, compared with 1,113 examples in the 2% buffer, and the table shows that this choice is conservative for Tether.
We would also welcome pointers to the passages that were hard to follow, so that we can revise them. We would be glad to discuss any of these points further.
Dear Reviewer ar1W, thank you again for your feedback. We would appreciate hearing whether our response and the revised paper have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification.
Response to Reviewer ar1W (Part 1/3)
We thank the reviewer for the detailed review and for recognizing the label-free, auditable measure of decision drift and the exact KL critical radius, which is the central technical result of the paper. The main new result is that, at matched training time and similar accuracy, DER++ still changes more of the earlier decisions than Tether-KL on both models. We respond to the eight points in order and will include the new experiments in the revision.
Throughout this response, drift is the percentage of earlier-task test inputs on which the final model's decision differs from the teacher's, and Acc is the final average accuracy. Unless stated otherwise, each new result is the mean of nine runs, the three task orders (Orders 4–6) with three seeds each. Paired comparisons are between runs that share a task order and a seed, so each comparison has nine pairs.
1. Training time of Tether-KL (Table 7)
The T5-large numbers for Tether-KL in Table 7, including the 9.0× after fifteen tasks that also appears in Table 3, are a mistake. They were timed on an earlier variant that back-propagates through all 48 records of every earlier task at every step, whereas all Tether-KL results in Tables 2 and 3 use 16 of the 48 records per step. The correct cost is 2.5× that of IncLoRA for training alone, which is the basis of the other entries in these tables, or 3.1× when record construction is included. On Llama-2, the 12.1× is correct for the configuration in the paper, but using 4 records per step instead of 16 gives similar drift and accuracy and lowers the cost to 4.1×.
With these numbers, Tether-KL takes about twice the training time of DER++ at its default setting. To examine whether the lower drift is simply an effect of the longer training time, we gave DER++ approximately the same training time as Tether-KL by using more replay batches per step and a proportionally larger buffer. In the table below, Time is the training time including record construction, relative to IncLoRA.
Even with the additional training time, DER++ changes more of the earlier decisions than Tether-KL on both models, at similar final accuracy. The paired difference in drift, computed as Tether-KL minus DER++ at matched training time, is −3.5 points on T5-large (95% CI: −4.5 to −2.4) and −3.4 points on Llama-2 (95% CI: −4.0 to −2.8), and Tether-KL has the lower drift in every pair. We will correct Tables 3 and 7 in the revision and add this comparison.
2. Assumption of Theorem 1 and the experimental setup
Assumption 1 of Theorem 1 requires the records of each class to be independent draws from the task distribution. The benchmark provides no inputs for a task beyond its training and test sets, so in the submission we took the records from the training inputs, which are the teacher's own training data. This is why the paper states that the theorem does not cover the reported runs, and we agree that the paper should explain this more clearly. Remark 1 of the paper says that the assumption is met when the records are drawn from inputs outside the teacher's training data (lines 946 to 948). In the new experiments, we draw the records in this way for 13 of the 15 tasks. For each of these tasks, the candidate inputs are sampled uniformly at random from the rows of the original dataset's training split that appear in neither the benchmark's training file nor its test file. After the teacher is trained, the records are drawn at random from the admitted candidates within each class that the teacher predicts. This sampling design is the one that Assumption 1 describes. The two smallest tasks have no such rows and still use training-set records. The results stay close to those of the submission. The drift of Tether-KL on T5-large is 7.0%, compared with 6.7% in the submission. The per-record guarantees of Propositions 1 and 2 and the audit bound of Proposition 5 do not depend on this assumption.
The admission experiment in point 3, the Tether-TKL runs on Llama-2 in point 4, and the record-count ablation in point 8 instead use records held out of the training file, so these runs train on 5% to 13% fewer rows.