acceptodds
Under review as a conference paper at ICLR 2027

What Do Time-Series Foundation Models Share? An Input-Kernel Accounting of Representational Agreement

Abstract

Do time-series foundation models organize input histories in a shared way? We compare the matrices of pairwise history similarities produced by ten variants from seven families on 17 datasets. Released pretrained weights (PT) agree more than independent architecture-matched initializations (INIT). On the full inventory, more of the additional agreement lies in the input kernels for mean pooling than for last-position pooling. The last-position residual stays positive across depth, reference expansion and family deletion. The mean-linear endpoint interval crosses zero; mean-RBF and intermediate mean-linear depths stay positive. On later Wikipedia records the mean-linear residual is larger than the last-linear residual. The last-linear remainder is a difference in residual direction: PT and INIT have similar residual norms, and permuting each released parameter tensor removes the additional agreement. Small MLP and CNN predictors that improve within-dataset forecasting also develop additional last-position agreement outside the same input span. The balance between the input span and the residual depends on the readout and the population.

Primary area:
learning on time series and dynamical systems
Keywords:
time series foundation models, representational similarity, centered kernel alignment, Platonic representation hypothesis, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_QxP2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies representations across time series foundation models using CKA. It compares pretrained and randomly initialized models, separates their agreement into input-reference and residual components, and investigates these components through controlled experiments. The findings depend on the readout, and greater agreement does not consistently imply better forecasting.

Critical strengths

The research question is meaningful. Compared with representation studies in foundation models for other modalities, time series representations remain a worthwhile direction to explore.

The experimental design is careful. The paper includes many small experiments and complementary controls that examine different explanations for the observed agreement.

Critical weaknesses

The new contribution needs clearer positioning. The paper lacks a clear central experiment and reads more like an accumulation of small studies. Please identify the main claim and explain which experiments provide its strongest evidence.

The connection to forecasting ability remains unclear. Please state what direct, concrete guidance these findings provide for TSFM pretraining or downstream applications.

Questions for authors

Classification is a common way to evaluate representation quality, and classification results could provide additional evidence that the representations are useful. Could you add a classification related evaluation, such as random forest on frozen features?

Tabby uses deep supervision at intermediate layers, making its representations an interesting case for the depth analysis. I would be interested in whether this supervision changes the balance between input-reference and residual agreement. I explicitly acknowledge that Tabby appeared in September 2026: whether it is included will not affect my score.

4 comments

Thank you for the careful reading and for suggesting the classification and Tabby analyses. All references below are to the revised manuscript.

The central experiment compares agreement among pretrained models from different families with agreement among fresh, architecture-matched initializations, using the same histories across the full model and dataset inventory. The input reference is a span of similarity matrices computed from the histories; the residual is the part of model agreement outside that span. With linear CKA on the final-layer features at the last observed position, the pretrained-minus-initialized increases in both components mainly reflect closer matrix alignment; norm contributions remain uncertain. Stronger cross-family sharing is not accompanied by a detected average increase in how much of each model's geometry the input reference covers. At the final layer, mean pooling instead gives a residual increment in linear CKA whose interval includes zero. Our central result is stronger sharing through alignment, without a detected average coverage gain; evidence for a residual increment depends on the readout. The revised introduction makes this comparison and its directional decomposition the main evidence; the smaller studies test its interpretation and limits (Sections 1 and 3.1–3.3, pp. 1–4; Figures 1–3, pp. 4–5).

For practical use, we recommend selecting models or ensemble weights by forecasting loss on an independent calibration window, then testing the locked choice at later forecast origins. We tested separately fitted model/ensemble selection rules with and without the residual increment as an input. Including it slightly worsened historical performance on held-out sources and left choices unchanged at separately scored origins (Section 6.2, p. 9; Appendices K.2, p. 45, and J.3, p. 42). For frozen-feature applications, include inexpensive input statistics as baselines and state the readout clearly. A general forecasting benefit from maximizing agreement during pretraining remains unverified (Section 8, p. 10).

We added the requested random forests on frozen features using a patient-disjoint PTB-XL classification split. Window statistics outperformed Chronos-Bolt and TimesFM 2.5 features by macro-AUROC on this split, supporting the recommendation to check inexpensive baselines (Section 6.2, p. 9; Appendix J.3, p. 42). The forests evaluate complete feature vectors; the contribution of the geometric residual to classification remains unmeasured. These results come from one patient split with unknown pretraining-data overlap.

The Tabby analysis describes how the input-reference and residual increments vary with depth. At the last-position linear readout, the estimated pretrained-minus-initialized residual increment is positive at the measured depths. From quarter-depth to the final layer, the input-reference increment decreases, while the residual increment's change remains uncertain. At the final layer, the mean-pooled linear residual-increment interval includes zero (Section 3.2, p. 4; Appendix M and Figure 11, p. 53).

Your question helped us connect the extra alignment to a testable interpretation: forecast training can change which input relations models share, depending on the training and evaluation distributions. The full-inventory comparison remains central; the two tests below probe this interpretation.

In the revised transfer test, we fit combinations of input kernels, excluding both the test source and model family, then freeze the combination weights. On the same held-out pretrained targets, weights fitted to pretrained models predict last-position geometry better on average than weights fitted to fresh architecture-matched initializations (Section 5.1, Table 2). The joint 95% interval for the mean gain in centered Frobenius alignment excludes zero.

In a controlled study, a small Transformer and CNN first train on one synthetic law, then continue on a version with changed seasonal and trend structure. At the final checkpoints, the mean squared distance between their similarity matrices projected onto the input reference is lower on the new distribution but higher on the original than before continuation. After continuation, we rescale each projected matrix to its norm before continuation and recompute distances; the decrease on the new distribution remains (Section 6.3; Appendix N.2).

For model selection, retain the independent forecast-loss evaluation from our first reply; an agreement-based pretraining objective still requires evidence of forecast improvement.

Thank you for the substantial revisions and additional experiments, particularly the classification evaluation, Tabby depth analysis, and controlled training study. These additions clarify the central claim and substantially address my concerns about the paper’s focus and representation evaluation. Although the implications for improving pretraining remain limited, the functional tests provide useful evidence about the limits of using representational agreement as a proxy for forecasting ability. I will update my assessment accordingly.

Thank you for taking the time to consider our revisions and additional experiments. We appreciate your thoughtful assessment and feedback.

Reviewer_uzrFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether TSFMs share representational structure and how much of their additional agreement, relative to architecture-matched random initializations, can be accounted for by input-derived reference kernels. Across ten variants from seven model families and 17 datasets, the authors find that much of this additional agreement lies within the reference span, while residual agreement remains in several settings. The balance depends on pooling, depth, and the evaluated population. Further experiments show that ordinary forecasting training can also produce residual agreement, and that encouraging a common temporal relation does not consistently improve forecasting.

However, the organization of the content in this paper is not good.. Although its central subject is shared representations in pretrained time-series models, the introduction begins with a general discussion of representation comparison and provides insufficient background on TSFMs themselves. I would expect a clearer account of these models and why their shared representations warrant investigation. The related-work section should also provide a more substantive discussion of TSFM research.

Furthermore, the representation-level investigation closely resembles existing work in CV and NLP, where pretrained models have a longer research history. Questions concerning (1) whether different models learn similar representations, (2) whether similarity originates from the input data, (3) whether models converge toward shared representations, and (4) whether representational similarity reflects model capability have already been investigated in that literature. The experimental methodology and broad findings are largely consistent with these established lines of research. In these respects, I do not find particularly significant new conceptual insights. Much of the contribution characterizes general properties of pretrained neural networks in the time-series setting, without establishing a substantially new theoretical account.

The paper does include investigations directed at TSFMs, such as “Ordinary Forecasting Training and Residual Agreement,” “What Changes When Pooling Changes?”, and “Does Favoring a Common Relation Improve Transfer?” These experiments indicate that shared relations depend strongly on pooling and the evaluated population, that residual agreement outside the chosen reference span is not exclusive to foundation models, and that alignment with a common temporal relation does not consistently yield forecasting benefits. However, I am not convinced that these findings substantially advance our understanding of TSFMs. In particular, the study does not establish what forecasting-relevant information the shared residual contains or how it contributes to predictive capability. In my assessment, the paper remains primarily a description of shared relations and does not yet provide a convincing explanation of forecasting ability.

Therefore, I currently believe that the experiments in this paper are relatively complete. However, whether the results are sufficient to form important new insights and whether they truly explain the predictive power of TSFM remains unconvincing.

Critical strengths
  1. This paper features a broad experimental scope and includes targeted controls. This is its most significant strength.

  2. It yields new evidence that can correct over-interpretation. These results remind us that common residuals cannot be directly considered as knowledge specific to the base model, and similarity cannot be directly used as a substitute for predictive ability. This concrete evidence gives it value for time-series research.

  3. The overall structure is clear, making the main content easy for readers to understand.

Critical weaknesses
  1. In my personal opinion, the understanding of new concepts is still insufficient compared to existing research. Cross-model representation similarity, confounding effects caused by input relationships, and the distinction between similarity and functional behavior have all been well-studied. This paper adds a temporal reference kernel and more comprehensive experiments, but it does not fully demonstrate which previously unknown and significant characteristics of TSFM these findings reveal.

  2. This paper identified common residuals in TSFM, but failed to explain what information these residuals carry or why they are useful. The authors did attempt future relation kernels and ACF interventions, but the former failed to identify the main implications of the residuals, and the latter did not establish stable predictive returns. From describing the geometry of the representation to explaining predictive power, key evidence is still lacking. This is the shortcoming I value most.

  3. I strongly hope that the authors can add an introduction to the research subject TSFM itself and related work.

  4. I believe the focus of this paper should be on exploring how these representations of TSFM can improve the predictive performance of our model. In other words, the experimental conclusions presented in this paper should provide guidance and inspiration for our subsequent efforts to improve model performance.

  5. The title on OpenReview is inconsistent with the title in the PDF. But the meaning is similar.

Questions for authors

For specific Questions, please refer to the Critical Weaknesses and Paper Summary sections.

5 comments

Thank you for reading the paper closely and for asking us to connect shared residuals to forecasting ability.

All section references below are to the revised manuscript.

1. Contribution beyond earlier representation studies

The main TSFM finding is that the additional agreement in pretrained models primarily reflects closer alignment of their patterns of similarities between histories. We establish this comparison against architecture-matched initializations using linear similarity matrices from the final representations at the last valid historical position. The input reference is a space of similarity matrices computed directly from the histories. The directional effect appears both inside and outside this space. Within it, pretrained models share relationships more closely across families, while average coverage, the proportion of each model's relation matrix captured by the reference, shows no detected gain (Section 3.3; Appendix H).

Two methodological contributions make this account possible. Our signed decomposition separates reference and residual contributions that add exactly to total CKA on its original scale. We also fit input-kernel combinations using model relation matrices from other sources and families, then predict held-out model relations with the weights frozen. On identical targets, weights fitted to pretrained models better predict pretrained relations, and weights fitted to initialized models better predict initialized relations, on average. This provides a transferable, training-state-specific account of model relations (Sections 2.2 and 5.1).

2. Residual content and forecasting value

We now identify a concrete component of shared residual agreement: relative daily amplitude explains part of the Chronos-2/Moirai residual increment on NASA temperature histories, supported by matched controls. This result ties a measurable temporal property to agreement left outside the original input reference. The property was selected during development and checked across dates at shared sites, with a near-zero average contribution on other hourly sources (Section 3.4; Appendix J.2).

The revision also extends the earlier future/forecast-shape and autocorrelation experiments with native-accuracy, model-selection and activation-intervention tests. Pretrained residual agreement has a weak positive association with final accuracy; in a separate eligible model set, its change from initialization poorly tracks pretraining gains. Fixed-input activation steps shift residual agreement in the intended directions on the tested panels, with dataset-dependent forecasting effects. Adding the residual increment to a matched model/ensemble selection rule gives no average gain in the reported tests (Sections 6.1–6.2; Appendices I, J.3–J.4 and K.2).

3. TSFM background and related work

We have reorganized the introduction around TSFM pretraining and zero-shot forecasting, and expanded Section 7 to cover TSFM training, evaluation, temporal concepts and interventions. This gives the representation analysis a clearer time-series context (Sections 1 and 7).

4. Practical guidance

For forecasting applications, we recommend choosing models or ensemble weights by forecast loss on an independent calibration window, then testing the frozen rule at later origins. For frozen-feature applications, include inexpensive input-statistic baselines; for a proposed temporal explanation, fix the property before evaluating new histories with matched controls. These recommendations follow directly from the functional tests, including cases where forecasts improve as coverage or sharing declines (Sections 6.2–6.3 and 8).

5. Paper-title consistency

The revised PDF title now matches the title registered on OpenReview: “What Do Time-Series Foundation Models Share? An Input-Kernel Accounting of Representational Agreement.”

Thank you again for asking what the shared residual means for forecasting. Two further studies address its content and, separately, how forecast learning changes the input-explained part of model relations.

Across the original TSFM families, we tested additional input similarity matrices built from transformed versions of the existing histories. The transformations were fixed in advance: one set smooths local fluctuations, and the other changes the influence of extreme values, for example by clipping. Both additions explained part of the PT–INIT residual increment, with a larger contribution from the extreme-value transformations. We fitted weights combining these added matrices on other sources, then froze them. Each addition also improved average prediction of pretrained models’ similarity matrices on held-out sources over the original reference (Appendix N.1).

The controlled study used a small Transformer and a causal CNN in twelve independent training repetitions. Both first learned a harmonic-and-trend task. They then adapted to a modified task with changed daily amplitude and phase drift, while each history’s mean and variance remained fixed. At the planned endpoint, both passed the prespecified forecasting criteria in every repetition, including at least halving their pre-adaptation error on the new task. Their input-explained relation matrices became closer on the new distribution than before adaptation. The distance also shrank after rescaling the final projections to their pre-adaptation norms, supporting stronger directional alignment. Residual agreement decreased on average even as forecasts improved. On the original distribution, the same training moved the input-explained matrices farther apart and worsened forecasts (Section 6.3; Appendix N.2).

This supports an empirical hypothesis about planned endpoints: with the input reference and training protocol fixed, successful learning of a common forecasting task brings the input-explained relations closer on that task’s distribution, on average across training repetitions. The input-explained distance did not decrease at every checkpoint. Whether explicitly training models to align this component improves forecasting remains open (Appendix B.6).

Thank you for the detailed responses and additional experiments. I appreciate the improvements, but many central findings including input geometry influencing agreement, alignment depending on training and evaluation conditions, and similarity not necessarily reflecting predictive ability, are already well studied in CV and NLP. Although the signed PT–INIT decomposition and frozen input-kernel transfer provide useful quantitative extensions, I remain unconvinced that the resulting observations establish sufficiently significant new insights into temporal representation learning. In addition, the time series specific content appears strongest in the residual-content analysis and adaptation to altered temporal laws, while much of the remaining analytical framework is applicable to foundation models in other domains. So, I would welcome more systematic analyses of properties directly relevant to time-series forecasting, such as temporal order, long-range dependencies, and multi-scale features, to clarify what these shared representations reveal beyond general neural-network behavior. My concern remains the conceptual contribution beyond existing representation studies for TSA, so I maintain my rating.

Reviewer_egVZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper compares history-similarity matrices from ten model variants across seven families and 17 datasets. It decomposes the difference between pretrained and initialized models into agreement within an input-kernel span and a residual. The study examines readout and population sensitivity, ordinary forecasting models, and recovery on held-out sources. Synthetic experiments further examine temporal factors and adaptation toward a common relation.

Critical strengths
  1. The paper asks a useful question: which relationships between histories do the models share? Comparing pretrained weights with architecture-matched initializations and separating agreement into an input-aligned component and a residual makes this question more concrete.

  2. The depth, family-deletion, and later-collected-data checks give the findings more support than a single similarity score would. The paper also distinguishes descriptive accounting from held-source prediction and states its limitations clearly.

  3. The motivation and progression of the experiments are easy to follow.

Critical weaknesses
  1. The input-kernel decomposition explains some shared relations, but the temporal properties underlying the remaining agreement are not identified. The checks show that it persists across several settings, without identifying the temporal relationships behind it.

  2. The sinusoid experiment explains pooling effects on a separate population, and the ordinary-predictor experiment shows that the effect is not specific to foundation models. Neither explains the real-data residual, which is central to the paper's account of shared representations.

Questions for authors
  1. Can the authors connect part of the real-data residual to a specific temporal property, or distinguish plausible explanations for it?
  2. How does the similar finding in ordinary predictors change the interpretation of foundation-model convergence?
4 comments

Thank you for the careful reading and concrete suggestions on explaining the residual and interpreting foundation-model convergence.

For your first question, we identify relative daily amplitude as a temporal contributor to the residual increment for Chronos-2 and Moirai on NASA temperature histories. This measures the strength of a history's daily cycle relative to its overall variation. Adding its kernel to the input reference explains part of the extra residual agreement in pretrained models relative to their initialized versions at four dates at the same sites. Matched-kernel and geographic-reassignment controls support this attribution (Section 3.4, pp. 4–6; Appendix J.2, pp. 34–36). Its average contribution is near zero across the four other hourly sources tested.

We also tested change in autocorrelation between the two halves of a history as a second candidate, finding near-zero average contribution across the original panels with complete 256-step histories (Appendix J.4, p. 43). Our interpretation of the remaining agreement is that forecasting models may emphasize temporal relations that are incompletely represented by the current reference (Discussion, Section 8, p. 10). The daily-amplitude result gives a concrete example of how to identify such relations.

The sinusoid study explains the temporal basis of pooling sensitivity: mean pooling emphasizes frequency, while the last position also retains phase and frequency–phase interaction (Section 6.1, Figure 6, p. 8). This connects the readout dependence in our comparisons to named temporal factors.

For your second question, the ordinary-predictor result shows that increased residual agreement can emerge through forecasting supervision. In the within-dataset experiment, an independently trained MLP–CNN pair produces a last-position residual increment comparable in magnitude to that of TSFM pairs on the same histories, using forecasting loss alone (Section 4, Figure 4, p. 6; Appendix C, pp. 22–23). This places TSFM residual convergence in the broader context of learning to forecast.

For TSFMs, last-position linear CKA shows stronger agreement between pretrained models than between their architecture-matched initializations. Our decomposition shows that stronger directional alignment is the main contributor to this increase, both within the input reference and in the residual (Section 3.3, p. 4; Figure 3, p. 5). Together with the daily-amplitude result, this gives a quantitative account of stronger shared geometry and a concrete example of its temporal content.

All references above are to the printed pages of the revised manuscript.

Thank you again. We have added a broader real-data test and a controlled follow-up on convergence.

Across all original datasets and model families, we tested prespecified input similarities after smoothing histories or changing the influence of extreme values, for example through clipping. We used fixed history subsets. Both sets account for part of the last-position linear residual increment on average. The set including clipping contributes more: 0.0062 CKA units (nominal 95% source-bootstrap interval 0.0017–0.0123). Each addition also reduces the average held-source error in predicting pretrained models’ similarity matrices relative to the original input bank, with combination weights fitted only on other sources. Smoothing has no established advantage over controls that shuffle time order before smoothing and then restore the original order. Whether the smoothing contribution reflects temporal filtering therefore remains unresolved (Section 3.4; Appendix N.1, pp. 54–55).

For your second question, we ran 12 independent training blocks, each containing a small Transformer and a CNN. After training on one synthetic data-generating process, each model branched into matched-budget continuation on that process and on a version with different seasonal structure and trend. Both architectures met the prespecified forecasting-performance criteria for the modified process. In the modified-process branch, from the pre-switch to the final checkpoint, the average distance between the models’ similarity matrices projected onto the input-reference span decreased on histories from the modified process but increased on histories from the original process. On the modified-process histories, the final distance was also smaller than after matched continuation on the original process. The average trajectory was nonmonotonic across checkpoints, and the contraction persisted after rescaling each projected component to its pre-switch norm (Section 6.3; Appendix N.2, pp. 55–57).

The training study supports distribution-dependent alignment within the input reference. Applying this explanation to agreement among released TSFMs remains a hypothesis.

Thank you for your responses and additional experiments. They address some of my concerns, but the identified factors explain only part of the real-data residual and are specific to the data source. I will keep my score.

Thank you for taking the time to consider our response and the additional experiments. We agree that the temporal properties underlying the remaining residual agreement are only partly understood, and that our current attribution is limited and varies across sources.

We would like to clarify one complementary finding: our decomposition distinguishes whether the increase in residual agreement comes primarily from larger residual norms or from closer alignment of residual directions. In the comparisons studied, the main contribution comes from the latter. This characterizes how the representations become more similar, without claiming to identify all the temporal properties those directions encode.

Comments on the paper

2 comments
Area_Chair_S2w1OpenReview ↗

Reminder: Please Participate Actively in the Discussion

Dear Reviewers,

Please actively participate in the discussion by responding to the authors’ replies and engaging with your fellow reviewers. This discussion is essential for clarifying concerns, resolving differences, and reaching a well-supported consensus on the final recommendation.

Thank you for your time and valuable contributions.

Best regards, AC

Dear Reviewers,

Thank you again for your thoughtful reviews and suggestions. We have responded to each of your comments and revised the paper, including additional analyses addressing the questions raised.

If there are any remaining concerns or points that would benefit from further clarification, we would be grateful for the opportunity to address them during the discussion period.

Thank you for your time and consideration.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.