acceptodds
Under review as a conference paper at ICLR 2027

Task Vector Descent: Learning from Non-IID Batches

12 views

Abstract

A central challenge in continual learning is to acquire new knowledge without forgetting what the model has already learned. This challenge appears in language model training when training data comes from various domain-, user-, or task-specific distributions that are encountered unevenly over time. In such settings, successive minibatches are temporally clustered by distribution instead of being sampled i.i.d. from the overall data mixture. Training on temporally clustered data induces a stability-plasticity tradeoff. Adapting the model to the active distribution can improve the model on the active distribution but may lead to a performance degradation on data it previously trained on. We find that this tradeoff intensifies with longer exposure to the same distribution. We therefore ask if the parameter displacement produced by such a sequence (the task vector) should be fully retained or applied partially. We compare applying the full displacement () with partial integration, which scales the task vector by before applying it to the continuing model and scale the optimizer state by the same coefficient. Across continual pretraining, pretraining from random initialization, supervised post-training, and reinforcement post-training, we find that intermediate values of often improve average continuing-model performance relative to full integration, particularly after longer same-distribution sequences. In continual-pretraining experiments with both controlled streams and naturally defined adaptation sequences, task-vector scaling outperforms full integration at the matched learning rate, showing that its benefits are not reproduced by learning-rate scaling alone.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, Non-IID data, Model merging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pP12OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces a method called "Task Vector Descent" (TVD), which performs partial merging of parameter updates and AdamW optimizer states during local post-training on temporal data. Various experiments across various training settings demonstrate that using coefficients derived from intermediate values enhances the performance of continual learning models. This improvement is often more pronounced when models process ordered sequences drawn from a uniform distribution. Since the method simultaneously reinforces temporal effects and reduces merging frequency, it is difficult to isolate which specific component drives the performance gains.

Critical strengths

This paper presents an interesting research topic.

Critical weaknesses
  1. The study investigates whether partial integration becomes more effective when the same data distribution is observed for a longer period. However, increasing (L) has two simultaneous effects. It makes the data more non-IID through longer periods of temporal clustering, and it reduces how frequently task-vector integration is performed. The authors acknowledge that (L) controls both factors, but this coupling makes it difficult to determine which factor accounts for the observed results. Can the authors provide experiments that disentangle the effects of temporal clustering and integration frequency?

  2. The proposed method scales not only the task-vector parameters but also the AdamW optimizer state, including the first moment, second moment, and step count. In particular, the second moment is interpolated in square-root space. The role of each component remains unclear. How does the method perform when only the parameters are scaled while the optimizer state is left unchanged, when the optimizer state is reset, or when only the first moment (m) is interpolated? Is interpolating the step count necessary? Ablations addressing these alternatives would help clarify which parts of the optimizer-state interpolation are essential.

  3. The proposed parameter update appears closely related to Lookahead and sequential checkpoint averaging, yet the evaluation focuses mainly on comparisons with full integration and learning-rate scaling. Without comparisons to these closer algorithmic baselines, it is difficult to assess whether TVD introduces a distinct method or offers a new empirical characterization of an existing interpolation mechanism. The authors should clarify the methodological distinction and, ideally, include these baselines in the evaluation.

  4. The formatting of this paper is very poor. This paper does not compare the proposed approach with a wide range of baselines.

Questions for authors

Please see the weakness section.

3 comments

First, we want to thank the reviewer for the time and effort put into reviewing our submission. We also thank the reviewer for finding the research question interesting.

Weakness 1 (temporal clustering versus integration frequency) We thank the reviewer for pointing this out. We agree that it is important to investigate the effects of segment length $L$ and integration interval $M$ independently. We therefore added an experiment that varies $L$ and $M$ separately while holding the training batches and budget fixed. We added a paragraph discussing the results in the main text (Fig. 5): At a fixed merging interval $M$, the final-checkpoint advantage of partial integration increases with $L$ for every tested interval. At a fixed segment length $L>1$, its run-average advantage increases as $M$ approaches $L$. When $M>L$, the run-average difference is negative in every tested setting. Thus, longer same-distribution segments increase the final-checkpoint advantage even at a fixed integration interval, while the run-average advantage of partial integration grows as integration becomes less frequent up to once per segment and worsens when integration intervals exceed segment length.

Weakness 2 (optimizer state) We added an ablation using the same three seeds, comparing interpolation of the full AdamW state, parameter-only interpolation with the optimizer state retained, and optimizer reset (Table 7). Interpolating the state gives the highest mean result in the tested settings. Its advantage over retaining the state is generally small, while resetting is substantially worse in several settings. For continual pretraining at $L=M=32$, full-state interpolation improves BPT gain by $0.002$ over parameter-only interpolation and by $0.059$ over reset. We have not separately tested first-moment-only or step-count-only interpolation, so these results do not establish that either component is individually necessary. They support full-state interpolation as the default among the procedures tested.

Regarding Weakness 3 (novelty), we agree that the parameter interpolation rule in itself is algorithmically not new: if the merge schedule matches the task boundaries, the update rule takes the same form as in Lookahead and Reptile. Our contribution is to empirically study and characterize when continual model merging improves language-model training as the data stream ranges from IID mixing to long same-distribution segments.

Lookahead and SFA provide the closest precedents for the update used in our study, but examine parameter interpolation in the context of different research questions:

  • Lookahead introduces the same parameter-interpolation update as a general optimizer, motivated by improved stability and reduced variance. It reports gains on standard training benchmarks. It does not systematically study how the update’s effect changes with temporal clustering of the data stream.
  • SFA studies sequential fine-tuning on distinct tasks that do not recur in its evaluated streams. It merges toward a fixed task-start checkpoint and restarts the optimizer at the next task.

We summarize these relationships in the new Table 1 (Appendix A), which compares the methods’ update rules, merge schedules, and optimizer-state handling.

A Lookahead run matched to our merge interval, coefficient, and optimizer-state handling would reproduce TVD’s update, rather than provide a distinct baseline. We instead compare TVD directly with SFA on the same controlled streams (Appendix E.2). We also vary same-distribution segment length while holding the training data fixed. Our second-order SGD analysis distinguishes partial integration from training at a matched learning rate: their updates agree to first order, but their within-segment interaction terms have coefficients $\lambda\eta^2$ and $\lambda^2\eta^2$, respectively.

Weakness 4 (baseline breadth). We added comparisons in two experimental settings:

  • Controlled Pile streams: We compare TVD with Sequential Fine-tuning with Averaging (SFA) on the same streams (Appendix E.2). TVD has higher run-average gain across the tested settings. This compares the complete procedures, including their different optimizer-state handling.
  • PG-19: We added experience replay and unweighted L2 regularization to the main-text comparison (Fig. 10). The best evaluated mean BPT gains are 0.0427 for TVD, 0.0408 for replay, 0.0374 for L2, and 0.0372 for full integration.

Presentation We have revised the figure spacing, legends, error bars, and captions. We would welcome specific examples of any remaining formatting issues, so we can address them in the revision.

We hope that our rebuttal addressed the reviewers' questions about our work. Should you have any further questions, we will be more than happy to answer them during the ongoing discussion phase.

Reviewer_pP12OpenReview ↗

The response.

Thanks for the author's response, which addresses some of my concerns. I still think the novelty of this work is limited.

Reviewer_PZz6OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies language models' continual learning capabilities with non-i.i.d. batches. In a more realistic setting for language models, data do not arrive in i.i.d. batches. Instead, they arrive in temporally clustered batches. The core of this paper focuses on the strength (λ\lambda) of a "task vector," which is defined as the parameter displacement between tasks.

This paper answers four research questions through extensive empirical studies: (1) Does task vector scaling outperform full integration? (2) If so, what are the trade-offs associated with different values of λ\lambda? (3) Does it improve continual pretraining from random initialization, without pretrained weights? (4) How does it behave with natural data?

The paper examines settings including continual pretraining, pretraining from random initialization, supervised post-training, and reinforcement learning-based post-training, and finds that intermediate values of λ\lambda consistently improve performance.

Critical strengths

(1) The overall writing, as well as the visualizations, tables, and equations, is very organized, consistent, and easy to follow.

(2) For each of the research questions, the authors conduct extensive ablation studies; the presented results are very easy to follow.

(3) This paper studies an empirical research regime that is closer to real-world settings and shows that task vector scaling of partial integration is always better than full integration.

Critical weaknesses

(1) This paper is mostly an empirical study. I'm a little concerned about the paper's novelty and currently consider it slightly below the bar for acceptance.

(2) This paper studies a continual learning problem for language modeling; It shows the trade-off between stability and plasticity; however, most of the experiments are empirical ablations of the strength of the task vector. I'm wondering if some existing continual learning algorithms could help with this task.

Questions for authors

Following the 'Critical Weaknesses' section, have you tried using existing continual learning algorithms for this setting?

3 comments

First, we want to thank the reviewer for the time and effort put into reviewing our submission. We also thank the reviewer for recognizing the paper’s clear presentation, extensive ablations, and the practical relevance of studying temporally clustered language-model data.

Regarding Weakness 1 (novelty), we agree that the parameter interpolation rule in itself is algorithmically not new: if the merge schedule matches the task boundaries, the update rule takes the same form as in Lookahead and Reptile. Our contribution is to empirically study and characterize when continual model merging improves language-model training as the data stream ranges from IID mixing to long same-distribution segments.

Lookahead and SFA provide the closest precedents for the update used in our study, but examine parameter interpolation in the context of different research questions:

  • Lookahead introduces the same parameter-interpolation update as a general optimizer, motivated by improved stability and reduced variance. It reports gains on standard training benchmarks. It does not systematically study how the update’s effect changes with temporal clustering of the data stream.
  • SFA studies sequential fine-tuning on distinct tasks that do not recur in its evaluated streams. It merges toward a fixed task-start checkpoint and restarts the optimizer at the next task.

We summarize these relationships in the new Table 1 (Appendix A), which compares the methods’ update rules, merge schedules, and optimizer-state handling.

Together, these methods motivate studying interpolation across a spectrum of temporal structure, from i.i.d. minibatches to sequential fine-tuning on distinct tasks, which is the focus of our study: Lookahead evaluates conventional fixed-dataset training, while SFA evaluates successive, nonrecurring task stages including substantial domain shifts. We systematically vary the length of recurring same-distribution segments between these regimes while holding the training data fixed. Our second-order SGD analysis further distinguishes partial integration from standard SGD training with a matched learning-rate: the updates agree to first order, but their within-segment interaction terms have coefficients $\lambda\eta^2$ and $\lambda^2\eta^2$, respectively. This offers an analytical rationale for the empirical difference we observe.

Regarding Weakness 2 (comparisons with existing continual-learning methods), we thank the reviewer for the suggestion. We added a comparison with SFA on the controlled Pile streams (Appendix E.2), and comparisons with experience replay and L2 regularization on PG-19 (Fig. 10).

Across the tested controlled-stream settings, TVD at $\lambda=0.6$ achieves higher run-average gain than either tested SFA coefficient. These are comparisons of the complete procedures: SFA resets the optimizer at segment boundaries, and for $M<L$, it holds its reference fixed throughout the segment while TVD refreshes its reference after each merge. On PG-19, the best evaluated mean BPT gains are $0.0427$ for TVD, $0.0408$ for replay, $0.0374$ for L2, and $0.0372$ for full integration. Replay performs close to TVD and improves over full integration.

Thank you for the update! The rebuttal has addressed my concerns to some extent, and I have raised my rating from 2 to 3. However, please note that I have not worked in this field for several years, so my review may have some limitations. I would therefore encourage the AC and SACs to take this into consideration.

Thank you for taking the time to consider our response and revised paper! We appreciate your feedback and updated assessment.

Reviewer_DtoYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies continual learning for LLMs under non-IID data streams. The authors empirically show that Task Vector Descent (TVD), which partially integrates task-specific parameter updates, improves performance across different LLM training regimes, particularly for longer same-domain sequences. They further provide a second-order analysis showing that TVD preserves more within-task gradient interactions than learning-rate scaling.

Critical strengths

Strength: The paper provides solid empirical verification across diverse LLM training regimes, including pretraining, SFT, and RL. The experiments are carefully designed, and comparisons with learning-rate scaling further strengthen the empirical findings.

Critical weaknesses

Weakness: The paper lacks sufficient novelty for a top-tier conference like ICLR. The core method closely resembles existing approaches like Reptile [1], and the theoretical analysis provides limited new insights beyond explaining the difference from learning-rate scaling. While the empirical results are solid, the work introduces neither fundamentally new ideas nor substantial conceptual insights for the community. It may be better suited to conferences or journals that place greater emphasis on empirical contributions.

[1] Nichol, Alex, Joshua Achiam, and John Schulman. "On first-order meta-learning algorithms." arXiv preprint arXiv:1803.02999 (2018).

Questions for authors

See weaknesses.

3 comments

We thank the reviewer for the time and effort put into reviewing our submission. We also thank the reviewer for recognizing the breadth and care of our empirical evaluation across pretraining, SFT, and RL, and for highlighting the comparison with learning-rate scaling.

We agree that the parameter interpolation rule in itself is algorithmically not new: if the merge schedule matches the task boundaries, the update rule takes the same form as in Lookahead and Reptile. Our contribution is to empirically study and characterize when continual model merging improves language-model training as the data stream ranges from IID mixing to long same-distribution segments.

Reptile, Lookahead, and SFA provide the closest precedents for the update used in our study, but examine parameter interpolation in the context of different research questions:

  • Reptile uses the model-merging update across sampled tasks to meta-learn an initialization that can adapt quickly to a new task.
  • Lookahead introduces the same parameter-interpolation update as a general optimizer, motivated by improved stability and reduced variance. It reports gains on standard training benchmarks. It does not systematically study how the update’s effect changes with temporal clustering of the data stream.
  • SFA studies sequential fine-tuning on distinct tasks that do not recur in its evaluated streams. It merges toward a fixed task-start checkpoint and restarts the optimizer at the next task.

We summarize these relationships in the new Table 1 (Appendix A), which compares the methods’ update rules, merge schedules, and optimizer-state handling.

Together, these methods motivate studying interpolation across a spectrum of temporal structure, from i.i.d. minibatches to sequential fine-tuning on distinct tasks, which is the focus of our study: Lookahead evaluates conventional fixed-dataset training, while SFA evaluates successive, nonrecurring task stages including substantial domain shifts. We systematically vary the length of recurring same-distribution segments between these regimes while holding the training data fixed. Our second-order SGD analysis further distinguishes partial integration from standard SGD training with a matched learning-rate: the updates agree to first order, but their within-segment interaction terms have coefficients $\lambda\eta^2$ and $\lambda^2\eta^2$, respectively. This offers an analytical rationale for the empirical difference we observe.

We hope that our rebuttal addressed the reviewers' questions about our work. Should you have any further questions, we will be more than happy to answer them during the ongoing discussion phase.

Thank you to the reviewer for the explanation. However, it does not change my assessment that the paper offers neither fundamentally novel ideas nor substantial conceptual insights, focusing instead on a collection of scattered empirical contributions. Therefore, I will maintain my score of 2.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We thank the reviewers and Area Chair for their time and consideration of our revised submission.

We appreciate the reviewers’ recognition of “solid empirical verification across diverse LLM training regimes” (DtoY), “extensive ablation studies” (PZz6), and presentation that is “very organized, consistent, and easy to follow” (PZz6).

The revision addresses the following main points raised:

  • Novelty and relationship to prior work (DtoY, PZz6, pP12): The parameter-merging based algorithm we study has precedents in Lookahead, Reptile, and SFA (Table 1), but those works do not systematically examine when partial integration improves continuing-model performance over full integration on data streams spanning the spectrum from IID mixing to long, recurring same-distribution segments.
  • Temporal clustering versus merge frequency (pP12): The original setup coupled longer same-distribution segments with a lower merging-frequency. We now vary segment length and merge interval independently (new main-text Fig. 5). At each tested fixed merge interval, partial integration’s final-checkpoint advantage grows with the length of the same-distribution segments.
  • Optimizer-state handling (pP12): We added ablations on the optimizer state handling, comparing full-state interpolation, parameter-only interpolation, and optimizer reset. Full-state interpolation has the highest mean result, though its advantage over parameter-only interpolation is small (Appendix Table 7).
  • Continual-learning comparisons (PZz6, pP12): Appendix E.2 compares SFA on controlled streams; Fig. 10 adds replay and L2 regularization on PG-19. TVD has higher run-average gain than the tested SFA settings, while replay performs close to TVD on PG-19.

We appreciate your consideration of these additions in assessing the revised paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.