- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether representation changes during fine-tuning of time-series foundation models (TSFMs) can serve as reliable indicators of harmful encoder adaptation or catastrophic forgetting. The authors propose an evaluation framework combining baseline-relative screening of pretrained capabilities, CKA-based representation analysis, and controlled comparisons between full fine-tuning and frozen-encoder adaptation. Experiments across Moirai, Chronos-T5, and TimesFM-2.5 suggest that CKA does not reliably predict the relative benefits of encoder adaptation in the evaluated settings, while no experimental configuration satisfies the authors' prespecified criterion for harmful adaptation.
Critical strengths
-
Important methodological question. The paper highlights a useful distinction between representation changes and their consequences for downstream performance. Evaluating whether encoder adaptation actually improves or degrades forecasting performance, rather than relying solely on representation similarity metrics, is a meaningful research direction.
-
Careful experimental design. The authors separate model selection from held-out evaluation and compare full fine-tuning with frozen-encoder adaptation using paired experimental runs. The inclusion of multiple TSFM backbones, statistical significance analysis, and sensitivity studies on baseline selection and freezing strategies strengthens the empirical evaluation.
-
Transparent reporting of negative results and limitations. The paper reports that most experimental configurations are statistically inconclusive and acknowledges that its findings do not establish the universal safety of fine-tuning or the general uselessness of CKA. The analysis of evaluation protocols and freezing boundaries also provides useful methodological insights.
Critical weaknesses
-
The central motivation is not sufficiently supported by the cited literature. The paper argues that representation changes during fine-tuning are routinely interpreted as catastrophic forgetting, motivating techniques such as encoder freezing, regularization, and LoRA. However, the cited literature primarily establishes that fine-tuning can cause forgetting or that certain adaptation techniques may preserve pretrained capabilities. These findings do not necessarily imply that researchers generally treat representation drift, particularly low CKA, as direct evidence of harmful forgetting. Moreover, freezing and LoRA can be motivated by parameter efficiency, optimization stability, or generalization rather than representation preservation. The authors should provide concrete examples of prior work that explicitly equates representation drift with catastrophic forgetting or uses such measurements to guide adaptation decisions. Otherwise, the paper risks overstating the limitations of existing research.
-
The experiments do not directly evaluate catastrophic forgetting of pretrained capabilities. The models are fine-tuned and evaluated on different windows from the same target datasets, rather than being evaluated on their original pretraining domains or previously mastered tasks after adaptation. Consequently, the experiments primarily measure the effect of encoder updates on target-task performance, rather than the preservation of previously acquired capabilities. Importantly, a model may exhibit substantial representation changes and improved target-task performance while simultaneously losing its ability to generalize to other domains. Although the authors acknowledge that this is not a continual-learning study, the experimental evidence remains narrower than the broad motivation concerning catastrophic forgetting. Evaluating pretrained and fine-tuned checkpoints on independent forecasting benchmarks would help establish whether representation drift is related to genuine forgetting.
-
The lack of confirmed harmful adaptation cases limits the conclusions. None of the 31 evaluated configurations satisfies the authors' prespecified criterion for harmful encoder adaptation, and 25 configurations remain statistically inconclusive regarding the relative benefits of freezing versus fine-tuning. Therefore, the results mainly demonstrate that CKA does not provide a reliable ordering of adaptation outcomes under the current experimental settings, rather than establishing its inability to identify genuinely harmful forgetting. The within-Moirai correlation also has a wide confidence interval, leaving considerable uncertainty about the underlying relationship. Controlled positive cases involving clear degradation of previously established capabilities would make the analysis more convincing.
-
The conclusions are sensitive to the definition of frozen-encoder adaptation. The primary frozen-encoder condition still permits updates to the input projection and mask encoding, meaning that the encoder representations are not strictly preserved. The appendix reports that switching to strict freezing reverses the sign of the estimated adaptation effect in four of sixteen configurations. This sensitivity is particularly important because the full-versus-frozen comparison is the central intervention used to assess whether encoder updates are beneficial or harmful. A more systematic evaluation using strictly frozen encoders would strengthen the reliability and interpretation of the results.
Questions for authors
See weakness.
2 comments
Thank you for the additional experiments, which address several of my concerns. However, CKA’s within-checkpoint diagnostic value remains unresolved, and the broader conclusions are still insufficiently supported. I therefore maintain my score of 2.
Thank you. Each of your points led to a change in the revision.
W1 (literature). Agreed. §1 now cites work that uses representation change as forgetting evidence: CKA drops to locate forgetting (Ramasesh et al., 2021), similarity to predict how much a foundation model forgets (Mian et al., 2026), and feature distortion (Kumar et al., 2022). It also says freezing and LoRA are motivated by efficiency too.
W2 (forgetting not measured). We ran the experiment you describe (§7.2, Figure 2, appendix A4). The 7 gate-passing cells were fine-tuned under full fine-tuning (B), the frozen encoder and the strict freeze (H), and every checkpoint was scored on the 4 benchmarks it was not trained on.
W3 (no harm cases). The 11 pairs above are those cases, and CKA fails the registered test on them. On the wide interval, §6 now states what the design could detect (ρ = 0.5 at 80% power).
W4 (freezing boundary). The strict freeze now runs on all 22 Moirai cells and is the primary contrast (§5.2, Table 1). It gives 0 freezing-better, 3 adaptation-better and 19 inconclusive cells. The 4 sign reversals are all ETTm2, and all move towards adaptation, so none is a hidden case of harm.