acceptodds
Under review as a conference paper at ICLR 2027

When Representation Change Misleads: Evaluating Encoder Adaptation in Time-Series Foundation Models

Abstract

Fine-tuning a time-series foundation model reshapes its internal representations, and that reshaping is routinely read as catastrophic forgetting to be prevented. Preventing it presupposes something worth preserving. **Centered kernel alignment (CKA) tells us that the representation changed; a fine-tune-versus-frozen-encoder intervention tells us whether that change was valuable. Across the cells we evaluate, the former is not an actionable substitute for the latter.** Screening **32 cells** (three backbones: Moirai-S/B/L, Chronos-T5, TimesFM-2.5; six datasets; two horizons) against a lookback-96 linear regression *fit on the target data* and scored on windows disjoint from every outcome we report, only **7 of the 31** scored cells show the pre-trained model beating that baseline, and all seven are Moirai on two of the six datasets. On those seven, full fine-tuning *improves* MSE in five. **No cell meets our prespecified criterion for harmful encoder adaptation**: under multiplicity-adjusted paired intervals, the 31 cells split **2 freezing-better, 4 adaptation-better, 25 inconclusive**, and both cells where freezing decisively wins are cells our own screen rejects; a pre-registered seed top-up adds a third it *admits*, and the pre-registered unanimity form returns zero throughout. The outcome nonetheless spans to against zero-shot, and under an analysis fixed before it was run we find **no evidence** that CKA improves out-of-sample prediction of it beyond backbone identity: across 15 leave-one-cluster-out folds it *lowers* from to . In sample CKA **does not provide a reliable, actionable ordering** either: within Moirai, the only backbone with enough cells to ask, (clustered CI ), and acting on it as prescribed (freeze where drift is largest) is worth  pp against always adapting. Two measurement choices decide this: the frozen-encoder comparison must be scored on **held-out** windows (4 of 31 cells otherwise reverse sign, every one of them flattering the frozen encoder, and two of the four are cells that pass the screen), and the baseline must be **fitted**, since an unfitted per-window trend extrapolation that a constant predictor beats puts 16 of 21 Moirai cells on the wrong side of the screen.

Primary area:
learning on time series and dynamical systems
Keywords:
time series foundation models, time series forecasting, fine-tuning, catastrophic forgetting, representation similarity, centred kernel alignment, negative results, pre-registration

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_sq16OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether representation changes during fine-tuning of time-series foundation models (TSFMs) can serve as reliable indicators of harmful encoder adaptation or catastrophic forgetting. The authors propose an evaluation framework combining baseline-relative screening of pretrained capabilities, CKA-based representation analysis, and controlled comparisons between full fine-tuning and frozen-encoder adaptation. Experiments across Moirai, Chronos-T5, and TimesFM-2.5 suggest that CKA does not reliably predict the relative benefits of encoder adaptation in the evaluated settings, while no experimental configuration satisfies the authors' prespecified criterion for harmful adaptation.

Critical strengths
  1. Important methodological question. The paper highlights a useful distinction between representation changes and their consequences for downstream performance. Evaluating whether encoder adaptation actually improves or degrades forecasting performance, rather than relying solely on representation similarity metrics, is a meaningful research direction.

  2. Careful experimental design. The authors separate model selection from held-out evaluation and compare full fine-tuning with frozen-encoder adaptation using paired experimental runs. The inclusion of multiple TSFM backbones, statistical significance analysis, and sensitivity studies on baseline selection and freezing strategies strengthens the empirical evaluation.

  3. Transparent reporting of negative results and limitations. The paper reports that most experimental configurations are statistically inconclusive and acknowledges that its findings do not establish the universal safety of fine-tuning or the general uselessness of CKA. The analysis of evaluation protocols and freezing boundaries also provides useful methodological insights.

Critical weaknesses
  1. The central motivation is not sufficiently supported by the cited literature. The paper argues that representation changes during fine-tuning are routinely interpreted as catastrophic forgetting, motivating techniques such as encoder freezing, regularization, and LoRA. However, the cited literature primarily establishes that fine-tuning can cause forgetting or that certain adaptation techniques may preserve pretrained capabilities. These findings do not necessarily imply that researchers generally treat representation drift, particularly low CKA, as direct evidence of harmful forgetting. Moreover, freezing and LoRA can be motivated by parameter efficiency, optimization stability, or generalization rather than representation preservation. The authors should provide concrete examples of prior work that explicitly equates representation drift with catastrophic forgetting or uses such measurements to guide adaptation decisions. Otherwise, the paper risks overstating the limitations of existing research.

  2. The experiments do not directly evaluate catastrophic forgetting of pretrained capabilities. The models are fine-tuned and evaluated on different windows from the same target datasets, rather than being evaluated on their original pretraining domains or previously mastered tasks after adaptation. Consequently, the experiments primarily measure the effect of encoder updates on target-task performance, rather than the preservation of previously acquired capabilities. Importantly, a model may exhibit substantial representation changes and improved target-task performance while simultaneously losing its ability to generalize to other domains. Although the authors acknowledge that this is not a continual-learning study, the experimental evidence remains narrower than the broad motivation concerning catastrophic forgetting. Evaluating pretrained and fine-tuned checkpoints on independent forecasting benchmarks would help establish whether representation drift is related to genuine forgetting.

  3. The lack of confirmed harmful adaptation cases limits the conclusions. None of the 31 evaluated configurations satisfies the authors' prespecified criterion for harmful encoder adaptation, and 25 configurations remain statistically inconclusive regarding the relative benefits of freezing versus fine-tuning. Therefore, the results mainly demonstrate that CKA does not provide a reliable ordering of adaptation outcomes under the current experimental settings, rather than establishing its inability to identify genuinely harmful forgetting. The within-Moirai correlation also has a wide confidence interval, leaving considerable uncertainty about the underlying relationship. Controlled positive cases involving clear degradation of previously established capabilities would make the analysis more convincing.

  4. The conclusions are sensitive to the definition of frozen-encoder adaptation. The primary frozen-encoder condition still permits updates to the input projection and mask encoding, meaning that the encoder representations are not strictly preserved. The appendix reports that switching to strict freezing reverses the sign of the estimated adaptation effect in four of sixteen configurations. This sensitivity is particularly important because the full-versus-frozen comparison is the central intervention used to assess whether encoder updates are beneficial or harmful. A more systematic evaluation using strictly frozen encoders would strengthen the reliability and interpretation of the results.

Questions for authors

See weakness.

2 comments

Thank you. Each of your points led to a change in the revision.

W1 (literature). Agreed. §1 now cites work that uses representation change as forgetting evidence: CKA drops to locate forgetting (Ramasesh et al., 2021), similarity to predict how much a foundation model forgets (Mian et al., 2026), and feature distortion (Kumar et al., 2022). It also says freezing and LoRA are motivated by efficiency too.

W2 (forgetting not measured). We ran the experiment you describe (§7.2, Figure 2, appendix A4). The 7 gate-passing cells were fine-tuned under full fine-tuning (B), the frozen encoder and the strict freeze (H), and every checkpoint was scored on the 4 benchmarks it was not trained on.

  • Fine-tuning does forget: in 11 of 28 pairs, B loses at least 10 pp more than H in every seed. Neither freeze ever loses more than 6.5%.
  • CKA does not order the loss: ρ = −0.229, CI [−0.390, +0.137]. CKA's level is set by model size; Moirai-Small barely drifts yet has 7 of the 11 harmful pairs.
  • Within Moirai-Small (not registered) ρ = +0.488, which we report: drift may rank damage within one checkpoint, but its level does not carry across checkpoints.

W3 (no harm cases). The 11 pairs above are those cases, and CKA fails the registered test on them. On the wide interval, §6 now states what the design could detect (ρ = 0.5 at 80% power).

W4 (freezing boundary). The strict freeze now runs on all 22 Moirai cells and is the primary contrast (§5.2, Table 1). It gives 0 freezing-better, 3 adaptation-better and 19 inconclusive cells. The 4 sign reversals are all ETTm2, and all move towards adaptation, so none is a hidden case of harm.

Thank you for the additional experiments, which address several of my concerns. However, CKA’s within-checkpoint diagnostic value remains unresolved, and the broader conclusions are still insufficiently supported. I therefore maintain my score of 2.

Reviewer_VxaKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigates the use of Centered Kernel Alignment (CKA) as a predictor of performance when fine-tuning a Time Series Foundation Model. They test this by fine-tuning Moirai, Chronos, and TimesFM-2.5 family of models on six datasets and comparing against a fixed lookback linear model. They find that the linear model trained on the target data outperforms the pretrained models. On the Moirai models that beat out the linear model, they find that fine-tuning mostly improves performance and no result showed harm from encoder adaptation. They conclude that CKA does not reliably predict performance change.

Critical strengths

None

Critical weaknesses
  • The paper is extremely poorly written with technical contradictions
    • One-sixth of the paper’s text is bolded and is written as a revision logbook with many mentions to “earlier versions” and findings being “retracted” or “withdrawn”
    • Missing many citations. The main motivation for the paper in the first lines of the introduction (L43-44) claims “a large literature treats that reshaping as damage to be prevented" but no work is ever cited for this centralizing claim.
    • The authors change their experimental setup despite claiming it is fixed. For example, L174-175 they say they use no early stopping but then in their appendix they describe how they used early stopping L841-842 in their central sweep.
  • The paper’s claims are too weak and are mainly failing to reject nulls from their results
    • Their measured correlation between CKA and Δenc\Delta_{\text{enc}} in L390 has a 95% CI of
−0.469,+0.717-0.469, +0.717

. This interval is too wide to draw a conclusion as to whether CKA can accurately predict encoder adaptation performance delta.

  • Their main claim in L534-535 is that “CKA is not an actionable substitute for intervention.” However, the authors acknowledge that even if CKA could be used as a predictor it is not helpful as a decision rule since to compute CKA you must already have fine-tuned the model (L211-213).

This work is not at the level for this conference. The writing lacks professionalism and the results do not push the research frontier.

Questions for authors

None

2 comments

Thank you. We agree with each point; the revision has been substantially rewritten.

Writing. Revision-log prose is removed, bold text is cut by more than half, and the PDF is 44 pages instead of 77.

Citations (L43–44). "A large literature" is now "often", with citations to work that reads representation change as forgetting (§1) and a note that freezing and LoRA also have efficiency motivations.

Early stopping. A real inconsistency: one trainer's protocol was stated as the paper's. §3 and Table 6 now gives each trainer's protocol. The 21 Moirai matrix cells use the final epoch; ILI, TimesFM, Chronos and the n = 10k point select an epoch on selection windows. For n = 10k, all four protocols we ran agree on held-out windows (Table 5).

Weak claims, wide CI (L390). §6 states what the interval excludes and what the design could detect. Two results go beyond failing to reject:

  • adding CKA lowers out-of-sample R² (+0.201 to +0.176);
  • where fine-tuning causes real out-of-domain forgetting (11 of 28 pairs), CKA orders it with the wrong sign (§7.2).

CKA is post-treatment (L534 vs. L211). Agreed, and "not an actionable substitute" is removed. CKA needs the fine-tuned model, so it cannot save the comparison run. The revised claim: a CKA drop is not reliable evidence that encoder adaptation was harmful.

Reviewer_VxaKOpenReview ↗

Reviewer Rebuttal Response

Thank you for the major revision. Unfortunately, the paper still does not meet the expectation of this conference and my score will remain unchanged at Clear Rejection (1).

Reviewer_u3rtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks whether representation drift can substitute for the interventional comparison between full fine-tuning and a frozen encoder when deciding whether to protect a time-series foundation model’s encoder. The authors build 32 cells across three backbones and six datasets, using a fitted linear baseline to screen whether the pre-trained model has a capability worth preserving. Only 7 of 31 cells pass the screen, and no cell meets the prespecified definition of harmful encoder adaptation. Within Moirai, CKA does not reliably predict the intervention outcome, which the authors take as evidence that a post-treatment similarity statistic cannot replace the intervention. Overall, the paper is not well organized, and the core argument is hard to follow because key evidence is scattered across many appendices and buried in repeated numbers and revision notes.

Critical strengths

1.The paper is transparent in experiments: it uses held-out evaluation data, pre-registers key analyses, and openly reports its own earlier errors. 2. It offers a useful takeaway that whether fine-tuning beats freezing can flip depending on how the evaluation data are split and how strong the baseline is.

Critical weaknesses
  1. The paper is very hard to follow. The main argument depends heavily on the appendices, forcing constant back-and-forth.
  2. I have strong concerns about the reliability of the results. The paper itself acknowledges that many numbers in earlier versions could not be traced to any run record.
  3. In addition, the main finding, that representation similarity does not predict whether fine-tuning the encoder helps, rests on very limited data: essentially one model family, tested on only a handful of independent dataset groups.
Questions for authors
  1. The core motivation is that practitioners want a cheap diagnostic to avoid running the frozen-encoder comparison. However, the paper notes that once full fine-tuning has been run, the frozen-encoder run is just one additional training run of similar or lower cost. If the intervention is this cheap, why would practitioners need a substitute in the first place?
  2. Almost all meaningful evidence comes from Moirai, while the other two models are evaluated under different settings and cannot be compared directly. How do the authors justify generalising the conclusion beyond this single model family ? Adding more model families or datasets may change the outcome.
1 comment

Thank you. We agree on organisation and reliability.

Organisation. The paper is rewritten around four questions, each answered in the body with its own table or figure. It is 44 pages instead of 77, with no revision-log prose.

Reliability. Every table is generated from committed run records by one script. A checker verifies all 287 numerical claims in the prose against those records. Claims that could not be traced to a record are removed from the paper and listed in a supplementary errata file. The three new experiments were pre-registered before their first run.

Q1 (why a substitute?). You are right, and the revision drops that framing. CKA measured after fine-tuning is widely reported as evidence of forgetting (§1). We test whether that reading is warranted, and find it is not, on the target task or out of domain (§7.2). As you suggested, the recommendation now is: run the comparison.

Q2 (beyond Moirai). Traffic is the only screened dataset where Chronos-T5 and TimesFM-2.5 beat a fitted baseline. A pre-registered arm there (§5.3, Table 2) finds full fine-tuning better than the strict freeze in every seed on both. Chronos does so at a CKA below that of an untrained encoder. Each backbone keeps its own trainer, so we do not pool across backbones, and §8 states exactly which settings were tested.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Answer to all reviewers

We thank the reviewers. We agree with the shared criticisms, and we have uploaded a revision that addresses them. Changes, with where to find them:

  1. Rewritten as an argument, not a revision log. No revision-log prose; one pointer to a supplementary errata file. 77 pages down to 44; the appendix holds only what the body cites.
  2. Motivation cited (§1): work that reads CKA or representation change as forgetting, and the efficiency motivations for freezing and LoRA.
  3. Training protocol stated per trainer (§3, Table 6), including where early stopping applies.
  4. Strict freeze on all 22 Moirai cells (§5.2, Table 1). Only the head trains. Calls: 0 freezing-better, 3 adaptation-better, 19 inconclusive.
  5. Other backbones where there is something to protect (§5.3, Table 2). On Traffic, full fine-tuning beats the strict freeze in every seed on Chronos-T5 (−39.3 pp) and TimesFM-2.5 (−10.2 pp).
  6. Out-of-domain forgetting (§7.2, Figure 2, pre-registered). Fine-tuning forgets in 11 of 28 (cell, dataset) pairs. CKA does not order it: ρ = −0.229, CI [−0.390, +0.137], against a registered ≥ 0.5.
  7. Claim narrowed (Ethics Statement): a CKA drop is not reliable evidence that encoder adaptation was harmful. We do not propose CKA as a substitute for the full-versus-frozen comparison, which costs one run.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.