- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
TSAE applies structured sparse autoencoders to hidden representations of time-series forecasting models. It introduces shared/private dictionaries, gated feature encoding, and conditional temporal-continuity regularization, together with a multi-axis evaluation protocol for sparse features.
Critical strengths
The paper raises an interesting question: how to interpret internal representations of forecasting models rather than only attributing predictions to input variables. The attempt to evaluate sparse representations beyond reconstruction error is conceptually useful.
Critical weaknesses
- The paper does not sufficiently connect its analysis to existing understanding of spatial and temporal heterogeneity in multivariate forecasting. Prior MTS benchmarking work has already shown that spatial and temporal heterogeneity are central to model behavior and to differences between forecasting architectures. TSAE introduces shared/private dictionaries, but does not convincingly demonstrate that these features recover or explain such known heterogeneity. This substantially weakens the scientific value of the interpretation: the method reconstructs hidden states, but it remains unclear what new forecasting insight is obtained.
- The strongest empirical result is reconstruction fidelity rather than interpretability. TSAE achieves lower reconstruction/NFRE in the main PatchTST experiments, but it is not consistently superior in physical correlation, selectivity, or calibration. The authors themselves report such trade-offs. Thus, the evidence supports a better structured hidden-state compressor more strongly than a substantially better interpretation method.
- The comparison is not fully controlled. The main experiment matches per-token code width rather than total routed dictionary parameter count, and only TSAE receives the forecast-consistency refinement stage while baseline SAEs retain their original objectives. This makes the reconstruction advantage difficult to attribute specifically to the proposed structured dictionary.
- The evaluated forecasting representations are too narrow and dated for an ICLR 2027 interpretability paper. The headline evidence is based on PatchTST and three ETT datasets; iTransformer and DLinear are only secondary checks, and the fidelity advantage already fails to transfer uniformly to DLinear. Modern forecasting models increasingly use pretrained and hierarchical representations, including Sundial and Zeus. Sundial was an ICML 2025 oral model trained on a trillion time points, while Zeus provides a 2026 multi-scale foundation-model representation. It is unclear whether TSAE's proposed structure remains meaningful for these stronger representations.
- The case-study visualizations are too limited to support general representation-level conclusions. The paper does show selected feature examples in the appendix, but these are manually selected descriptive cases rather than a systematic geometry-level analysis. More global analysis, such as UMAP/clustering of channel or regime representations and correspondence to known spatial/temporal structures, would be much more convincing.
Questions for authors
Do TSAE features recover known spatial/temporal heterogeneity or channel groups in MTS datasets? Do the reported advantages remain when all SAE baselines receive the same forecast-consistency objective and are matched in total parameters? Does TSAE remain useful on modern pretrained forecasting representations rather than PatchTST/iTransformer? Can the authors provide a global representation-space analysis rather than selected atom examples?