- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces a method called "Task Vector Descent" (TVD), which performs partial merging of parameter updates and AdamW optimizer states during local post-training on temporal data. Various experiments across various training settings demonstrate that using coefficients derived from intermediate values enhances the performance of continual learning models. This improvement is often more pronounced when models process ordered sequences drawn from a uniform distribution. Since the method simultaneously reinforces temporal effects and reduces merging frequency, it is difficult to isolate which specific component drives the performance gains.
Critical strengths
This paper presents an interesting research topic.
Critical weaknesses
-
The study investigates whether partial integration becomes more effective when the same data distribution is observed for a longer period. However, increasing (L) has two simultaneous effects. It makes the data more non-IID through longer periods of temporal clustering, and it reduces how frequently task-vector integration is performed. The authors acknowledge that (L) controls both factors, but this coupling makes it difficult to determine which factor accounts for the observed results. Can the authors provide experiments that disentangle the effects of temporal clustering and integration frequency?
-
The proposed method scales not only the task-vector parameters but also the AdamW optimizer state, including the first moment, second moment, and step count. In particular, the second moment is interpolated in square-root space. The role of each component remains unclear. How does the method perform when only the parameters are scaled while the optimizer state is left unchanged, when the optimizer state is reset, or when only the first moment (m) is interpolated? Is interpolating the step count necessary? Ablations addressing these alternatives would help clarify which parts of the optimizer-state interpolation are essential.
-
The proposed parameter update appears closely related to Lookahead and sequential checkpoint averaging, yet the evaluation focuses mainly on comparisons with full integration and learning-rate scaling. Without comparisons to these closer algorithmic baselines, it is difficult to assess whether TVD introduces a distinct method or offers a new empirical characterization of an existing interpolation mechanism. The authors should clarify the methodological distinction and, ideally, include these baselines in the evaluation.
-
The formatting of this paper is very poor. This paper does not compare the proposed approach with a wide range of baselines.
Questions for authors
Please see the weakness section.
3 comments
We hope that our rebuttal addressed the reviewers' questions about our work. Should you have any further questions, we will be more than happy to answer them during the ongoing discussion phase.
The response.
Thanks for the author's response, which addresses some of my concerns. I still think the novelty of this work is limited.
First, we want to thank the reviewer for the time and effort put into reviewing our submission. We also thank the reviewer for finding the research question interesting.
Weakness 1 (temporal clustering versus integration frequency) We thank the reviewer for pointing this out. We agree that it is important to investigate the effects of segment length $L$ and integration interval $M$ independently. We therefore added an experiment that varies $L$ and $M$ separately while holding the training batches and budget fixed. We added a paragraph discussing the results in the main text (Fig. 5): At a fixed merging interval $M$, the final-checkpoint advantage of partial integration increases with $L$ for every tested interval. At a fixed segment length $L>1$, its run-average advantage increases as $M$ approaches $L$. When $M>L$, the run-average difference is negative in every tested setting. Thus, longer same-distribution segments increase the final-checkpoint advantage even at a fixed integration interval, while the run-average advantage of partial integration grows as integration becomes less frequent up to once per segment and worsens when integration intervals exceed segment length.
Weakness 2 (optimizer state) We added an ablation using the same three seeds, comparing interpolation of the full AdamW state, parameter-only interpolation with the optimizer state retained, and optimizer reset (Table 7). Interpolating the state gives the highest mean result in the tested settings. Its advantage over retaining the state is generally small, while resetting is substantially worse in several settings. For continual pretraining at $L=M=32$, full-state interpolation improves BPT gain by $0.002$ over parameter-only interpolation and by $0.059$ over reset. We have not separately tested first-moment-only or step-count-only interpolation, so these results do not establish that either component is individually necessary. They support full-state interpolation as the default among the procedures tested.
Regarding Weakness 3 (novelty), we agree that the parameter interpolation rule in itself is algorithmically not new: if the merge schedule matches the task boundaries, the update rule takes the same form as in Lookahead and Reptile. Our contribution is to empirically study and characterize when continual model merging improves language-model training as the data stream ranges from IID mixing to long same-distribution segments.
Lookahead and SFA provide the closest precedents for the update used in our study, but examine parameter interpolation in the context of different research questions:
We summarize these relationships in the new Table 1 (Appendix A), which compares the methods’ update rules, merge schedules, and optimizer-state handling.
A Lookahead run matched to our merge interval, coefficient, and optimizer-state handling would reproduce TVD’s update, rather than provide a distinct baseline. We instead compare TVD directly with SFA on the same controlled streams (Appendix E.2). We also vary same-distribution segment length while holding the training data fixed. Our second-order SGD analysis distinguishes partial integration from training at a matched learning rate: their updates agree to first order, but their within-segment interaction terms have coefficients $\lambda\eta^2$ and $\lambda^2\eta^2$, respectively.
Weakness 4 (baseline breadth). We added comparisons in two experimental settings:
Presentation We have revised the figure spacing, legends, error bars, and captions. We would welcome specific examples of any remaining formatting issues, so we can address them in the revision.