acceptodds
Under review as a conference paper at ICLR 2027

Finding the Zeitgeist in Time Series Foundation Models

Abstract

Time series foundation models (TSFMs) achieve strong zero-shot and transfer performance across diverse tasks, yet their internal representations remain poorly understood. Although sparse autoencoders (SAEs) have emerged as a powerful tool for mechanistic interpretability in language and vision models, this has not yet been established with TSFMs. We conduct the first large-scale exploration of their internal representations using SAEs, uncovering sparse features that align with interpretable temporal patterns. Moving beyond only probing for hypothesized concepts, SAEs cast opaque model internals into disentangled, often monosemantic features. To structure their large latent space, we establish *Zeitgeist*, an unsupervised method for fully automated characterization of individual latent features. Zeitgeist enables creating a characterizing atlas of activating geometries, captions, frequency, locality, and positional affinity, describing the inner workings of TSFMs. We demonstrate Zeigeist's generality across 9 TSFMs, spanning 3 architectural families, and provide new insights into their systematic differences in reasoning. Moreover, we show that Zeitgeist's internal representations are not only a tool for interpretability but can also be leveraged for targeted interventions on a TSFM's activations, providing a strong foundation for steering downstream applications.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series, Time Series Foundation Models, Interpretability, SAE

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wAUTOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework that constructs a "feature manual" for the hidden representations of time-series foundation models and then attempts to control their predictions by manipulating these features.

Critical strengths
  1. The research topic is valuable, with a clearly defined object of investigation.
  2. The paper provides a relatively broad and systematic analysis.
Critical weaknesses
  1. The specific advantages and necessity of using Sparse Autoencoders (SAEs) need to be better justified.
  2. The five proposed dimensions are relatively abstract and rely on manually designed rules.
Questions for authors

1. Would directly analyzing the hidden states without using SAEs achieve comparable results? Alternatively, would other self-supervised feature extraction methods yield similar performance? The authors need to clarify what makes SAEs particularly necessary or advantageous in this framework.

2. The five proposed dimensions appear rather abstract. Even with these five dimensions identified, it remains difficult for humans to understand what the time-series foundation model has actually learned. From a practical perspective, what concrete insights or benefits can ZEITGEIST provide?

3. The core interpretability metrics rely on a series of manually defined choices and evaluation rules. Consequently, the reported "interpretability" may partly reflect the preferences embedded in the evaluation criteria rather than the intrinsic structure of the model's learned representations.

4. The fact that activation samples exhibit similar patterns does not necessarily imply that the corresponding feature has only one semantic meaning. The proposed method does not seem to sufficiently account for the possibility that a single feature may encode multiple distinct semantics.

2 comments

Dear Reviewer wAUT,

We thank you for acknowledging the importance of TSFM interpretability and the systematic analysis of Zeitgeist. We carefully address each concern and question in the following:

Specific Advantages and Necessity of SAEs (W1/Q1):

We first use supervised linear probes to verify that the selected hidden states encode recognizable temporal structure (Sec. 4.1, Tab. 2). These probes perform better on the predefined concept-recognition task. However, supervised probing requires specifying target concepts and corresponding labels in advance [1], so this initial analysis is restricted to the concept inventory supplied by the researcher.

To explore beyond this inventory, we train SAEs through self-supervised reconstruction: the hidden activations themselves provide the reconstruction targets, without requiring concept labels [2]. Our T-SAE additionally derives its contrastive signal from consecutive patches, following [3], rather than from annotated concepts. Feature interpretation takes place after dictionary learning, when Zeitgeist characterizes the learned features through their activating examples, geometric properties, and captions (Sec. 3.3). The synthetic labels used for evaluation do not supervise dictionary learning.

Hence, our choice of SAEs is thereby motivated by their combination of label-free dictionary learning, sparse activations, reconstruction, and explicit decoder directions. These properties connect feature discovery, inspection through activating examples, and intervention within one framework.

The Five Feature Properties and Interpretability of Zeitgeist (W2/Q2/Q3):

The five descriptors answer concrete questions about a learned feature: how often it fires, where in the context it fires, whether activation survives changes to the surrounding context, which shapes recur among its activating examples, and how those shapes can be summarized in language (Sec. 3.3). See Sec. 4.4 for a concrete example. Together, the five features make large feature dictionaries inspectable and help assess the limits of individual interpretations.

Zeitgeist's atlas therefore exposes both the diversity of the SAE's features activating shapes and its contextual dependence that a single caption or selected patch visualization would miss. At the model level, these descriptors reveal systematic differences across layers and architectures, such as the decreasing share of patch-local features with depth across all nine TSFMs (Sec. 4.3).

To further support Zeitgeist's utility for interpretability, we conducted a human evaluation of the captions. See our response to Reviewer kyjG and App. F. This provides evidence that the generated descriptions correspond to the displayed temporal patterns.

Semantics and Heterogeneity (Q4):

We agree that similar activating shapes do not establish a unique semantic meaning. Furthermore, we argue, that semanticity overall is not a useful concept for general time series foundation models, as the explicit notion of a domain and its context are not present. Even within one geometric cluster, similar standardized shapes can have different meanings across domains or contexts. We, therefore, propose geometricity, a measurable concentration of observed shapes.

The Zeitgeist atlas explicitly captures the inherent heterogeneity of time series data and their models. A feature is classified as mono-geometric when its largest cluster contains at least 70% of sampled activating patches. This does not imply that strictly all activations share the same shape or meaning (Sec. 3.3), and we do surface diversity where it is observed.

The example in Sec. 4.4 illustrates this directly. Feature 2735 is mono-geometric because smooth declines account for 77.4% of its sampled activating patches, but broad humps account for another 11.8%. Both patterns are retained in its caption.

References:

  • [1] John Hewitt et al. A Structural Probe for Finding Syntax in Word Representations, NAACL-HLT, 2019.
  • [2] Trenton Bricken et al. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, Transformer Circuits, 2023.
  • [3] Usha Bhalla et al. Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretability, ICLR, 2026.
Reviewer_wAUTOpenReview ↗

Thank you

I will retain my scores.

Reviewer_4SoYOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces Zeitgeist, an end-to-end mechanistic interpretability protocol for interpreting time series foundation models. The authors test numerous TSFMs against several SAE approaches and present empirical results on hyperparameter choices, downstream implications of concepts, and preliminary analyses on steering forecasting results. The authors present a taxonomy of concepts to consider for mapping from SAE features to concepts that are motivated in time series literature and foundations. Overall, this work is an interesting empirical exploration of mech interp for TSFM’s, and there are many interesting insights to be gained from this work.

Critical strengths
  1. The presented motivation is strong and does a great job drawing distinction to previous work. This is an important problem in the TSFM community.
  2. I believe the taxonomy of feature usage is well-defined and comprehensive. One challenge in TSFM interpretability is that of defining what concepts mean, and I believe the authors did this well. Nothing in their set of concepts is necessarily novel with respect to literature, but the breakdown is useful for these purposes.
  3. The authors test a broad scope of TSFM’s, giving credibility to their analyses. They also tightly guard against overgeneralizing results to any single model, which is appreciated and stands in stark contrast to prior work.
  4. There are many generalizable lessons to be taken from these experiments for SAE training on downstream TSFMs.
  5. The work is very well-written and well-presented. This is a great example of a well-presented and scoped paper.
Critical weaknesses
  1. This work is best seen as a broad empirical study rather than introducing core algorithmic novelties. The novelties are limited to the mapping of SAE features to concepts in time series which, while an important step for time series interpretability, is not a massive leap forward.
  2. On Table 2, there are no error bars reported; we need to understand this variance in scores as it seems that no SAE method is consistently strongest. Error bars throughout the paper are highly encouraged.
  3. The steering analyses are quite limited and anecdotal, standing in stark contrast to the broad empirical work that authors did to understand SAE hyperparameter choices.
  4. None of the insights from the work are especially striking. There were no surprising insights that deviated from standard SAE takeaways for language models. While authors went about presenting their findings very well, the paper could use more focus on surprising results counter to current literature.
  5. Some of the code is missing from your submission, including code to run the steering experiments. More details in this regard would be helpful for release, but the released code now is fairly comprehensive otherwise.
Questions for authors
  1. How do you pick the features matching the SAE features for Table 2 results? Is this done on held-out data?
  2. The paper considers all time series domains as homogenous; however, time series is a heterogeneous data modality, and discovered concepts might differ from financial to weather domain, for example. It’s unclear right now if these SAEs can help with discovering domain-specific concepts. This would be a useful application of the Zeitgeist method. How would authors apply these SAEs to different domains? Could concept vocabularies be expanded to account for domain-specific effects?
  3. What are some of your most striking findings? How should this paper guide future SAE application to language models, and how do practitioners know when to trust their SAE outputs?
3 comments

Dear Reviewer 4SoY,

We thank you for acknowledging the importance of interpretability and Zeitgeist for the TSFM community. We carefully address each concern and question in the following:

Methodological Contributions of Zeitgeist (W1/Q3):

Transferring SAE-based interpretability from language to time series requires methodological adaptations beyond training an SAE on different data. Numerical patches require an explicit treatment of temporal geometry that is not present in textual domains of related prior works.

For instance, we cluster standardized activating patches with tolerance to small temporal shifts, then caption representative examples while preserving distinct recurring patterns within a feature (Sec. 3.3).

Locality testing also requires care. For models using series-level normalization, perturbing the context can change the normalized values of an otherwise unchanged patch. Zeitgeist therefore preserves whole-series mean and variance while testing activation survival under context perturbations (App. A.2).

With regard to the connection to LLMs in Q3, we see Zeitgeist's atlas of textually captioned temporal features as a step towards future, multimodal TSFMs that, e.g., jointly process text and numerical time series.

Error Bars and Variance (W2):

We thank the reviewer for highlighting the missing variability estimates. We have repeated the main cross-model analyses over three seeds (Figs. 1 and 4) and added variability estimates to Tables 2 and 3 and the per-concept results in Fig. 9. This improved rigor confirmed the findings of our work.

Thorough Analysis of Steering (W3):

Please refer to our response to Reviewer kyjG (W2/Q2), as well as Table 8 and App. E in the paper.

Surprising Key Insights (W4/Q3)

Two findings deserve greater emphasis, and we reworded Sec. 4.3 to make this more clear. First, geometric concentration and context dependence follow different depth trends: Hop-0 locality decreases across all nine TSFMs, while mono-/duo-geometricity remains broadly stable in Transformers and recurrent models but decreases in TTM-R3 (Sec. 4.3). A feature can therefore exhibit consistent activating shapes while depending on broader context. This shows why isolated patch visualizations can give an incomplete interpretation.

Second, reconstruction fidelity and forecasting impact require separate assessment. For Chronos-2, later layers have higher reported reconstruction error but smaller forecasting degradation under sparse reconstruction (Tab. 1 and 3). These measurements use different evaluation corpora, so we interpret this as motivation to assess both quantities rather than as a general inverse relationship.

Together, these findings provide practical guidance: inspect activating examples alongside contextual dependence, evaluate reconstruction and downstream behavior, and distinguish descriptive coherence from semantic or causal validation. A readable caption or concentrated activating geometry alone is insufficient to establish a faithful explanation. For language-model SAEs, our results motivate similarly complementary evaluations; whether the specific depth and architecture trends transfer remains an empirical question.

We foreground these findings and their implications more clearly in Sec. 4.3.

Code for Steering (W5):

Thank you for this hint. The linked anonymous repository now contains the missing code for the steering experiment, including additions made for the rebuttal experiments.

Synthetic Shapes Dataset and Experiments (Q1):

We selected the diagnostic shape concepts shown in App. C to represent a minimal set of patterns to expect as localized features. Therefore, they differ from works such as FreqShapes [1] we initially explored, or the now reproduced Mishra shapes [2] in App. G.

We evaluated on hold-out data, as stated in the Experimental Details in App. B.2, last sentence.

Differences Across Domains (Q2):

We have added a breakdown by data domain in App. D.2 (Fig. 10), reporting reconstruction fidelity, geometricity, locality, and forecasting impact. The differences are substantial: at the middle layer, healthcare and sales both have high reconstruction error, but healthcare exhibits a much larger forecasting penalty. This motivates examining features in the domain where the model is used.

For domain-specific analysis, the TSFM and SAE can remain fixed while Zeitgeist characterizes features using activating examples from the target domain. The 14 synthetic concepts are only a diagnostic evaluation set; dictionary learning and atlas descriptions are not restricted to this set. The vocabulary can therefore expand through descriptions of additional recurring patterns. Domain metadata and expert knowledge can then connect these observed patterns to domain-specific interpretations.

AuthorsOpenReview ↗

Rebuttal (continued)

References:

  • [1] Queen et al. Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistency, NeurIPS, 2023.
  • [2] Anurag Mishra. Dissecting Chronos: Sparse Autoencoders Reveal Causal Feature Hierarchies in Time Series Foundation Models, ICLR Workshop on Time Series in the Age of Large Models (TSALM), 2026.
Reviewer_4SoYOpenReview ↗

Thanks for your responses

Thank you for the carefully-crafted responses! I think this work is impactful, and I appreciate that you expanded the steering analysis to give some interesting comparisons across different types of steering methods. I'll retain my score at 3, but I think this is a strong paper that should be considered for acceptance.

Reviewer_kyjGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper trains sparse autoencoders on frozen hidden states from nine time-series foundation models and proposes Zeitgeist, an atlas that characterizes each learned feature by firing frequency, positional affinity, locality, activation geometry, and automated captions. It compares sparse-autoencoder choices and model families, probes synthetic temporal concepts, and edits sparse directions to steer forecasts in synthetic and electricity-demand case studies

Critical strengths
  • This is a broad and practically useful analysis pipeline. The treatment of locality, activation geometry, and positional affinity is more concrete than a collection of selected feature visualizations, and the paper studies multiple model families rather than a single Chronos checkpoint

  • The paper makes a serious effort to separate reconstruction, sparsity, and interpretability heuristics, and the intervention experiments go beyond purely descriptive atlases. It also positions itself against recent time-series sparse-autoencoder work, including TimeSAE and the Chronos SAE study.

Critical weaknesses
  • The evidence does not validate the central interpretability claims independently of the pipeline’s own heuristics. Mono/duo-geometricity, locality, and caption quality are defined and optimized by choices within Zeitgeist; lower reconstruction quality and higher sparsity can improve these scores without showing that a feature is monosemantic or causally explanatory. The synthetic concept evaluation selects the best matching feature after training and remains far below the supervised linear probe (mean F1 0.731 for the best SAE versus 0.999). There is no blinded human assessment of captions, no robustness analysis of the automated captions, and no test that the same semantic feature is recovered across SAE seeds.

  • The steering evidence is too narrow to support “targeted interventions” as a general capability. The electricity result selects among 16 features, five gains, and two widths on a 2013 selection split and evaluates one Christmas effect in 2014; it does not establish that the chosen direction has the captioned causal meaning rather than functioning as a useful forecast correction. The synthetic intervention gains are modest and depend on known rule-specific effects. Broader real-data interventions, negative directions, seed replicates, and comparisons to simple activation directions are missing.

Questions for authors
  1. Please add independent validation of captions and feature semantics: blinded expert ratings, seed-to-seed feature matching, and tests on an out-of-domain real corpus. How stable are the reported atlas-level conclusions?

  2. Can the authors evaluate steering over multiple prespecified real-world events and multiple TSFMs, with held-out selection and simple baselines such as residual, random decoder, PCA, and matched-norm activation directions?

  3. Please clarify the contribution relative to TimeSAE and the Chronos SAE work through direct, matched comparisons rather than narrative positioning alone.

3 comments

Dear Reviewer kyjG,

We thank you for recognizing the utility and comprehensiveness of the Zeitgeist pipeline and its feature atlas, as well as proposing important experiments that further support our claims and contributions. We carefully address each concern and question in the following:

Empirical Validation and Stability (W1/Q1):

Supervised linear probe has higher mean F1 score.

We agree that linear probes outperform Zeitgeist with regard to F1 score. Nevertheless, supervised probes test predefined temporal concepts, whereas self-supervised SAEs learn dictionaries without hand-crafted concept labels. Zeitgeist uses SAEs to find structure in TSFMs beyond these pre-defined shapes, which makes the solved task inherently more difficult and lower scores are expected.

The evidence does not validate the central interpretability claims independently of the pipeline’s own heuristics. No blinded human assessment of captions.

We agree that to show interpretability, human studies are highly valuable and therefore ran two such studies and 33 overall participants (19 + 14) for the textual captions and their summaries. To the best of our knowledge, Zeitgeist is the first proposed method with such captioning, hence in each study, we included shuffled and semantically inverted controls (App. F) as comparative baselines. On a Likert scale, Zeitgeist's captions received a mean score of 4.3 out of 5 versus 2.0 for controls; and its feature summaries received 4.5 versus 2.9.

No robustness analysis. No test that the same semantic feature is recovered across SAE seeds.

Language-model studies show that individual SAE features can differ across seeds [1], while reproducible structure can persist at the subspace level [2]. Substantial feature consistency is nevertheless achievable under suitable configurations [3], and architectural and training changes can improve stability [4]. These findings motivate treating feature correspondence as a distinct evaluation orthogonal to reconstruction statistics and atlas validation.

How stable are the reported atlas-level conclusions?

However, we agree that confirming the efficacy and interpretability claims of Zeitgeist on additional seeds is crucial. Therefore, we repeated the atlas comparisons across all nine TSFMs at three depths, together with the Chronos-2 concept-recognition and sparsification evaluations, over three seeds. The updated Figs. 1, 4, and 9 and Tab. 2-3 confirm the existing findings in the paper and substantially improve its rigor.

AuthorsOpenReview ↗

Rebuttal (continued)

Feature-based Steering (W2/Q2):

Steering evidence is too narrow to support "targeted interventions" as a general capability. Evaluate over multiple TSFMs, on held-out selection, and simple baselines.

Following the reviewer's suggestion, we have expanded the evaluation with an additional TSFM (TimesFM-3), positive and negative interventions, repetitions across SAE training seeds, and PCA and random-direction baselines. These additions are highlighted in cyan in Sec. 4.5 as well as App. E and disclosed below.

Sec. 4.5 begins with experiments on synthetic data, which we now expanded to three independently trained Chronos-2 SAEs on 2,048 held-out pairs (see also App. E.1).

Separately, we expanded the synthetic steering evaluation for one Chronos-2 SAE checkpoint and tested the resulting policies on 6,144 fresh pairs per setting across three new data-generation seeds (newly added App. E.2). Each method independently selects its intervention strength and, where applicable, feature budget on validation. The synthetic steering results below (Table 8 in the paper) show that the SAE lowers the effect MAE by approximately 5.4% relative to the best PCA variant in both settings (lower is better).

MethodIndependent effect MAE ↓Reversal effect MAE ↓
SAE0.041890.04348
PCA variance0.045050.04724
PCA contrast0.044270.04594
Raw hidden contrast0.042750.04502
Residual contrast0.046840.05070
Full SAE direction0.043310.04532
Random decoder (10-draw mean)0.058080.05946
Isotropic (10-draw mean)0.065020.06517

Multiple prespecified real-world events. The synthetic intervention gains are modest.

We expanded our existing real-world evaluation (App E.3) to additionally include forecasts on the ERCOT WEST datasets with Chronos-2 and TimesFM-3.0 (App E.4).

For the selected ERCOT WEST example, the table below reports MAE and nWQL over the same 48-hour forecast for both intervention signs, using one Chronos-2 feature and three TimesFM features. The paired plots in App. E.4 of the paper show the resulting changes to forecast trajectories and quantiles alongside matched-norm PCA and random-decoder controls.

ModelMethodSubtraction, α = −0.5Addition, α = +0.5
Chronos-2SAE70.76 / 0.04760439.30 / 0.25967
Chronos-2PCA variance132.95 / 0.07675295.04 / 0.16519
Chronos-2Random decoder, draw 0265.33 / 0.14234276.95 / 0.15773
TimesFM-3SAE153.38 / 0.09643385.36 / 0.22411
TimesFM-3PCA variance331.24 / 0.18182161.56 / 0.10884
TimesFM-3Random decoder, draw 0402.33 / 0.24484294.81 / 0.15356

Related Work (Q3):

In order to expand on Zeitgeist's positioning within the TSFM interpretability literature, we have expanded the manuscript with App. G. Here, we report concept association on the reproduced corpus of Mishra [6], locality, and reconstruction. Summarizing, we find that Zeitgeist learns more specific and relevant representations.

Please note that these closely related prior works on TimeSAE and Chronos-T5 dissection [5, 6] are short workshop papers that are limited in scope and reproducibility.

References:

  • [1] Gonçalo Paulo and Nora Belrose. Sparse Autoencoders Trained on the Same Data Learn Different Features, ICLR, 2026.
  • [2] Gleb Gerasimov et al. Unstable Features, Reproducible Subspaces: Understanding Seed Dependence in Sparse Autoencoders, arXiv preprint arXiv:2606.12138, 2026.
  • [3] Xiangchen Song et al. Mechanistic Interpretability Should Prioritize Feature Consistency in Sparse Autoencoders, ACL, 2026.
  • [4] Walter Nelson, Theofanis Karaletsos, and Francesco Locatello. Toward Identifiable Sparse Autoencoders, ICML, 2026.
  • [5] Vedika Sanjeev Lakhanpal. TimeSAE: Mechanistic Interpretability for Time-Series Foundation Models, ICLR Workshop on Time Series in the Age of Large Models (TSALM), 2026.
  • [6] Anurag Mishra. Dissecting Chronos: Sparse Autoencoders Reveal Causal Feature Hierarchies in Time Series Foundation Models, ICLR Workshop on Time Series in the Age of Large Models (TSALM), 2026.

Dear Reviewer,

The discussion phase is coming to an end soon. It will be helpful if you could engage and let us know if we have clarified all your concerns. We have worked very hard to provide the answers to all your questions and indeed the review have been very helpful to make our paper stronger.

Let us know if we can alleviate any remaining concerns.

Regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.