acceptodds
Under review as a conference paper at ICLR 2027

Not All Forgetting is Equal: Objective Structure in Temporal Continual Learning

Abstract

Catastrophic forgetting is often treated as a uniform cost of sequential updates, yet on structured temporal streams its severity can depend on the prediction objective. We study Domain-IL fine-tuning on three matched univariate corpora—glucose, electricity, and weather—under five objectives (next-value and short-/long-horizon regression; dense and rare within-horizon events) and five architectures. Event objectives forget far more than value objectives in every domain-architecture cell. Controls rule out rarity, horizon, and the monthly protocol; a reference swap shows the gap follows how the target is defined (entity-fixed thresholds vs. segment-relative value targets), not events as such. Joint-vs-sequential and recurrence diagnostics then split failures into preventable interference and a recurrence gap invisible to single-transition forgetting. Savings, recalibration, and mechanistic probes further show that the loss is often recoverable for recurrent learners, while transformer damage reaches the representation. We demonstrate that not all forgetting on streams is equal. Rather, it is objective-conditioned and diagnostically typed.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
catastrophic forgetting, continual learning, time series data, objective-dependent forgetting, event detection, sequential fine-tuning, interference

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_qHBfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper empirically examines various continual-learning metrics (e.g., forgetting) in time series analysis, in 3 datasets, 5 objectives, and 5 models.

Critical strengths

The paper is asking interesting questions, e.g., how does the objective affect forgetting? How much of the damage of forgetting can be fixed?

Critical weaknesses
  1. I found the paper very hard to read, starting from the introduction. The main issue for me is that many sentences are difficult to interpret unless you understand the context for them, which appears only later (this context awareness is something the authors can do easily, but not a new reader). For example, Q2 on line 048 reads: "Is event forgetting a single failure mode or do diagnostics split (Q2)?" I do not understand what "failure mode" or "diagnostic split" mean here. As another example, it is hard to understand the setting because the notation is not clear; e.g., in the first displayed equation on line 118, k,K, and t_k are not defined.

  2. The results are purely empirical; this is fine, but the authors need to convince the reader they are representative, at least in some specific practical domain. However, the authors do not establish this: the models are rather small, and it is not clear that they have optimized hyperparameters to be competitive within their domain. Therefore, it is hard for me to understand whether the findings reflect properties of the objective or architecture, as claimed, or are specific to the particular models with their hyperparameter configurations.

Questions for authors

Please address the above weaknesses.

Reviewer_snuHOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a gap in the continual learning (within univariate numeric time series) literature where the problem is only studied through the lens of loss in performance when sequentially updated. The authors expand on this by studying the effect of objective functions, architectures and capacity, and how those factors affect forgetting severity, the type of forgetting and recoverability.

The paper empirical grid spans 3 different datasets, 5 prediction objectives (3 regression - next value, short and long horizon forecasting, 2 classification - dense and rare event classification), and 5 architecture classes (LSTM, GRU, RNN, CNN, transformers). The authors introduce several metrics, and diagnostic tools to study forgetting across multiple dimensions.

The results show that event based classification objectives forget substantially more than the value based regression objectives. The experiment with reference swap shows that this gap is almost entirely due to the reference frame for the event based objectives (references moving from global thresholds to local thresholds). Additionally, the degree of forgetting tracks the base rate shift, as the base rate shift is a direct consequence of data distribution shifts in univariate time series data and the considered labelling strategy. The authors additionally introduce failure model classification based on two diagnostic tools, preventable interference (whether joint training can avoid the damage) and the recurrence gap (whether a model continually trained through a full year of segments underperforms a fresh model when a calendar month recurs). Those metrics are utilized to study failure modes in across different objectives and architectures, where the recurrence gap results from non stationary distribution of the labels and gradual drift that the immediate forgetting metrics miss. On recoverability, the paper demonstrates that the damage is architecture dependent. With transformers showing the largest representation damage identified through linear probing of the learned representation to recover the performance, and negative savings that quantify the amount of training required to re-learn a specific regime.

The paper concludes that the continual learning on stream should be evaluated per objective and caractorized using different diagnostic tools and their outcomes.

Critical strengths

The experimental design is rigorous within the scope identified by the authors, and the results are extremely informative. Specifically, P5 reference swap experiment answers a very logical question regarding the definition of the labels and the cause for forgetting. I appreciate the authors being self critical regarding the claims made with respect to "label" framing as compared to the actual event classification objective.

Each metric and diagnostic tool designed answers specific question with the output values easy to interpret. the failure mode classifications with relevant matched diagnostics and remedies are useful for the domain.

Critical weaknesses

The papers framing and the headline claims made deviate or over-state the findings. For instance, the claims made regarding event classification and catastrophic forgetting are not originating from the objective itself, but how the labels are constructed and normalization used, with an open follow up question. (Please see the question below)

Limited technical novelty and scope of the domain. Although I find the experiments conducted to be rigorous, and findings useful. The majority of the technical contributions are in the form of carefully designed metrics and diagnostics. Further, the application is limited to a specific domain (univariate time series data). In my opinion, this limits the overall contributions.

Questions for authors

Is catastrophic forgetting measured on an entity grain with each entity getting a custom model? Is that common in practice where 1 model is productionalized for each entity or a common model is used?

Follow up experiment - Can the authors conduct an experiment where the event classification has global normalization statistics across the entire training dataset or at-least some earlier time window greater than 1 month? Global normalization statistics align more closely with continual learning environments.

Reviewer_PgAPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary
  • The paper analyzes forgetting in continual learning from three perspectives:
  1. Is forgetting caused because of the prediction objective, or architecture and capacity?
  2. Whether forgetting is homogenuous across different domains?
  3. Can the a model recover from forgetting and if so what part of the architecture is responsible for the recovery
  • The paper uses a domain incremental setting, with each domain being a univariate time series with a fixed prediction objective.
  • A stream is partitioned into temporal segments, each having a different distribution PmP_m
  • The paper uses 5 prediction objectives: next-value regression, short-horizon value regression, long-horizon value regression, dense-event classification, rare-event classification
  • The analysis reveals:
  1. Event objectives forget more than value objectives
  2. Forgetting is non-homogenuous across datasets, where some datasets can avoid forgetting if trained jointly on old and new tasks.
  3. The forgetting in LSTMs is largely in the output head, whereas in CNNs and Transformer, the forgetting occurs in the intermediate representations.
Critical strengths
  • Analyzes forgetting in a controlled setting on time series data across multiple domains.
  • The analysis of where the forgetting occurs in different model architectures is interesting.
Critical weaknesses
  • The description of the data sequence construction is very brief. Since the datasets used in the paper are non-standard, it should include details of what each segment is constructed. Does it contain multiple entities? This makes it hard to understand what exactly the prediction tasks are doing.
  • The paper clearly written using a coding agent and is hard to parse. It will benefit from a rewrite such that the results and analysis are easier to read and grasp.
  • The paper does not consider pretrained time-series models such as MOIRAI/MOIRAI2. The continual learning research on standard benchmarks has largely shifted to using pretrained models, and it will be interesting to see if the analysis holds in that domain.
Questions for authors

None other than those in the weaknesses.

Reviewer_G4FzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how prediction objectives affect forgetting during sequential fine-tuning on glucose, electricity, and weather streams. It compares five objectives and five model architectures, and finds substantially greater loss-based forgetting for event prediction than value regression under the default protocol. Reference-swap experiments indicate that much of this difference follows entity-fixed event thresholds versus segment-relative value targets.

Critical strengths

The reference-swap experiment is the strongest part of the paper. Rather than stopping at the observation that event objectives forget more, the authors manipulate how the targets are defined. For example, segment-relative event thresholds reduce weather dense-event forgetting from 2.31 to 0.06. This substantially sharpens the explanation and avoids attributing the effect simply to class imbalance, event rarity, or architecture.

Critical weaknesses

In the main comparison, regression targets are normalized separately for each segment, while event labels use fixed thresholds for each entity in Section 4.2. So the two objectives handle shifts in the data level differently.

Questions for authors

na

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.