acceptodds
Under review as a conference paper at ICLR 2027

TSAE: STRUCTURED SPARSE AUTOENCODERS FOR INTERPRETING TIME-SERIES FORECASTING MODELS

Abstract

Time-series forecasting informs critical decisions in energy dispatch, industrial operations, and environmental monitoring; understanding the patterns models rely on is essential for assessing reliability and identifying failures. Input attribution identifies important variables and time segments but offers limited insight into internal features, while standard sparse autoencoder (SAE) objectives do not directly constrain cross-variable structure or temporal continuity. We introduce TSAE, a structured sparse autoencoder for forecasting representations that decomposes hidden states into individually inspectable features. TSAE organizes cross-variable structure through shared and variable-routed private dictionaries, separates feature detection from magnitude estimation with gated encoding, and constrains neighboring sparse-code changes according to raw-segment similarity. These mechanisms support analysis of variable context, activation strength, and temporal evolution. Forecast-consistency fine-tuning further improves preservation of the frozen forecaster’s outputs. The accompanying TSEVAL protocol separately audits fidelity, feature coherence, and physical calibration to ground feature interpretation. In three-seed experiments with frozen PatchTST on ETTh1, ETTh2, and ETTm1, TSAE achieves the lowest hidden-state reconstruction error, normalized forecast-reconstruction error (NFRE), and feature transition rate among five SAEs at comparable per-token activity. NFRE decreases by 4.3–26.4% relative to the next-best mean. Dataset-dependent tradeoffs in selectivity, physical correlation, and calibration show that fidelity and temporal-stability gains require independent semantic validation to support feature interpretation.

Primary area:
interpretability and explainable AI
Keywords:
Time-Series, Sparse Autoencoders (SAEs), Mechanistic Interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AyZUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

TSAE applies structured sparse autoencoders to hidden representations of time-series forecasting models. It introduces shared/private dictionaries, gated feature encoding, and conditional temporal-continuity regularization, together with a multi-axis evaluation protocol for sparse features.

Critical strengths

The paper raises an interesting question: how to interpret internal representations of forecasting models rather than only attributing predictions to input variables. The attempt to evaluate sparse representations beyond reconstruction error is conceptually useful.

Critical weaknesses
  1. The paper does not sufficiently connect its analysis to existing understanding of spatial and temporal heterogeneity in multivariate forecasting. Prior MTS benchmarking work has already shown that spatial and temporal heterogeneity are central to model behavior and to differences between forecasting architectures. TSAE introduces shared/private dictionaries, but does not convincingly demonstrate that these features recover or explain such known heterogeneity. This substantially weakens the scientific value of the interpretation: the method reconstructs hidden states, but it remains unclear what new forecasting insight is obtained.
  2. The strongest empirical result is reconstruction fidelity rather than interpretability. TSAE achieves lower reconstruction/NFRE in the main PatchTST experiments, but it is not consistently superior in physical correlation, selectivity, or calibration. The authors themselves report such trade-offs. Thus, the evidence supports a better structured hidden-state compressor more strongly than a substantially better interpretation method.
  3. The comparison is not fully controlled. The main experiment matches per-token code width rather than total routed dictionary parameter count, and only TSAE receives the forecast-consistency refinement stage while baseline SAEs retain their original objectives. This makes the reconstruction advantage difficult to attribute specifically to the proposed structured dictionary.
  4. The evaluated forecasting representations are too narrow and dated for an ICLR 2027 interpretability paper. The headline evidence is based on PatchTST and three ETT datasets; iTransformer and DLinear are only secondary checks, and the fidelity advantage already fails to transfer uniformly to DLinear. Modern forecasting models increasingly use pretrained and hierarchical representations, including Sundial and Zeus. Sundial was an ICML 2025 oral model trained on a trillion time points, while Zeus provides a 2026 multi-scale foundation-model representation. It is unclear whether TSAE's proposed structure remains meaningful for these stronger representations.
  5. The case-study visualizations are too limited to support general representation-level conclusions. The paper does show selected feature examples in the appendix, but these are manually selected descriptive cases rather than a systematic geometry-level analysis. More global analysis, such as UMAP/clustering of channel or regime representations and correspondence to known spatial/temporal structures, would be much more convincing.
Questions for authors

Do TSAE features recover known spatial/temporal heterogeneity or channel groups in MTS datasets? Do the reported advantages remain when all SAE baselines receive the same forecast-consistency objective and are matched in total parameters? Does TSAE remain useful on modern pretrained forecasting representations rather than PatchTST/iTransformer? Can the authors provide a global representation-space analysis rather than selected atom examples?

Reviewer_UPttOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TSAE interprets frozen forecasting representations using shared and variable-specific sparse dictionaries, gated encoding and similarity-weighted temporal continuity. Forecast-consistency refinement preserves the original model's outputs. The accompanying six-axis TSEval protocol audits fidelity and interpretation-related properties. Three-seed PatchTST comparisons on ETT and Weather show improved reconstruction, NFRE and temporal stability, with mixed semantic metrics and backbone-dependent transfer.

Critical strengths
  1. The paper moves beyond input-level attribution to investigate forecasting models' hidden representations. Shared/private routing, gated detection–magnitude separation and conditional temporal continuity address variable structure, magnitude preservation and temporal continuity. Exploring how different temporal patterns affect forecasts gives the work conceptual depth.
  2. The paper effectively adapts the core SAE framework to time-series forecasting representations. TSEval diagnoses reconstruction, purity, physical calibration, forecast sensitivity, hidden controls and input-level baselines, supporting reproducibility and providing a useful contribution to the community.
Critical weaknesses
  1. Evaluation is limited to ETTh1, ETTh2, ETTm1 and Weather, with the core comparison centered on PatchTST. Cross-backbone baseline comparisons cover iTransformer and DLinear only on ETTh1, and Gated SAE outperforms TSAE on DLinear. Direct comparison with closely related time-series sparse explanation methods such as TimeSAE (Oublal et al., 2026) is missing, limiting evidence for generality and comparative advances.
  2. The most consistent gains concern reconstruction, NFRE and transition rate. Metrics closer to semantic interpretation do not uniformly improve: L1 often performs better on dead-feature ratio and calibration error, while TopK or JumpReLU can lead selectivity or physical correlation. The results therefore support effective structured sparse representation more strongly than improved model explanation.
  3. Private atoms can activate only on their assigned variables because of the routing architecture. Variable specificity is therefore an imposed prior, rather than an entirely data-discovered result. It cannot alone establish meaningful private structure; stricter controls or comparative analyses are needed.
Questions for authors

Could the method improve time-series foundation models through direct adaptation or fine-tuning? The forecasting backbone currently remains frozen, and forecast-consistency fine-tuning updates only SAE parameters. Could end-to-end or lightweight fine-tuning establish practical forecasting benefits?

Reviewer_1xmdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TSAE learns sparse features from frozen forecasting representations using shared and variable-specific dictionaries, gated encoding, and conditional temporal regularization. A subsequent forecast-consistency stage preserves the original model's outputs. TSEval separately examines fidelity, coherence, physical calibration, and intervention sensitivity. Experiments compare five SAE variants on ETT and Weather, with supporting checks on additional backbones.

Critical strengths
  • TSEval is a useful multi-axis protocol: it separates reconstruction fidelity, feature coherence, physical calibration, forecast intervention sensitivity, hidden-space controls, and input-level baselines. This is more informative than reconstruction-only SAE evaluation.
  • The structural biases are well motivated: shared/private variable routing targets cross-variable motifs, gated detection/magnitude targets sparsity-induced shrinkage, and conditional temporal continuity targets feature flicker on similar adjacent patches.
  • The PatchTST comparison is carefully run: validation-selected operating points, common data splits, three independently trained SAE seeds, and test-split evaluation. The activity-fidelity sweep is a good complement to a single operating point.
  • The paper is honest about tradeoffs. TSAE is not uniformly best on selectivity, physical correlation, or calibration, and the limitations section says so.
  • The reproducibility and AI use statements are detailed, and TSEval's adapter design covers multiple forecasting models (as smoke tests). The released artifact includes run-level results and configurations.
Critical weaknesses
  • The headline comparison is not apples-to-apples on the training objective. TSAE receives a forecast-consistency fine-tuning stage, while the baselines retain their original SAE objectives. NFRE is essentially the normalized MSE between the reconstructed and original forecasts—the same quantity the fine-tuning term optimizes—so the NFRE advantage is partly by construction. Baselines should receive the same fine-tuning, or the comparison should use a metric TSAE does not directly optimize.
  • Total dictionary capacity is not matched. Per-token code width is matched at 2,048, but TSAE uses shared plus variable-routed private dictionaries, so its total routed parameter count is larger and can vary with the number of variables. The paper acknowledges this but does not control it. The reconstruction/NFRE advantage may therefore reflect capacity rather than structure.
  • The ablations do not isolate the components. Removing routing changes capacity, removing gating changes realized activity from 203–268 to 46–108, and the fine-tuning stage is not ablated at all. The paper itself calls these "component diagnostics, rather than capacity- or activity-matched causal estimates." As written, the mechanism claims are not established.
  • The interpretability advantage is not established. Physical correlation and calibration are mixed or worse than baselines on most datasets; the transition rate is the behavior the temporal term is designed to suppress, so a lower rate is not independent evidence; and the shared/private separation is illustrated with selected atoms rather than a quantitative per-path analysis. The selected-feature diagnostics use one SAE seed.
  • Activity matching and generality are limited. The admissible activity band is [192, 320], but realized active counts still differ across methods, so fidelity comparisons may be confounded by activity. The main evidence is one backbone and the ETT datasets; the 18-model adapter coverage is smoke-tested, not scientifically evaluated. The TopK setting is also inconsistent: Appendix C.2 says k=256 for ETT, while Table 6 uses k=320 for ETTh1.
Questions for authors
  1. What happens when the baselines receive the same forecast-consistency fine-tuning? How much of the NFRE and reconstruction advantage remains, and does it survive on a metric TSAE does not directly optimize?

  2. Can you match total routed dictionary parameters (or explicitly control capacity) and report actual parameter counts and per-path utilization? Does the advantage remain when total capacity is equalized?

  3. Can you report capacity- and activity-matched ablations, including the fine-tuning stage, so that routing, gating, and temporal continuity can be isolated rather than confounded with capacity or activity?

  4. Do the interpretability results persist across seeds with a fixed feature-selection rule and matched baseline training? Please report shared-versus-private utilization and a quantitative cross-variable analysis, not only selected atoms.

  5. Can you reconcile the TopK k=256/320 inconsistency, report exactly activity-matched fidelity curves, and provide a controlled transfer check to another backbone? Which TSEval axes are actually supported for each model, rather than smoke-tested?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.