acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Embeddings in Time Series Forecasting from Latent Space and Data Perspective

Abstract

Embeddings serve as the fundamental representation units in time series forecasting and impose critical inductive biases on model learning, substantially influencing forecasting performance. Despite their importance, existing studies on embeddings largely overlook the distinction between direct mapping and high-order interactions on embedding effectiveness. In this work, we systematically investigate the effectiveness of embeddings across diverse datasets, ranging from first-order mapping to high-order interactions. Through extensive large-scale hyperparameter exploration and evaluation-theoretic analysis, we find that embedding-based models often degenerate into simple linear mappings without high-order interactions. In contrast, under higher-order interactions, embeddings exhibit distinct sensitivities to specific data characteristics and demonstrate distinct behaviors under different encoder architectures, suggesting that embeddings cannot be studied independently of higher-order interaction mechanisms. Our forecasting comparison covers attention, MLP mixing, recurrent, state-space, frequency-based, and convolutional encoders. Dataset characteristics, spectral rank, and frozen linear probes connect view preferences to representation evolution. The spectral and probe analyses cover MLPMixer, Transformer, GRU, Mamba, and FFT. With fixed-budget probes, encoder outputs reduce readout error in 72 of 98 configurations, with gains varying by dataset and token view. A multiscale data-rank rule further selects embedding–encoder pairs before target-model training and recovers six source-derived reference choices under leave-one-dataset-out calibration. Together, these results connect view construction, representation evolution, and practical module selection.

Primary area:
learning on time series and dynamical systems
Keywords:
Time series analysis, Latent space analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_g8EnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates the impact different embedding methodologies have on the performance of different forecasting architectures (referred to as encoder by the paper and in the following) on given datasets. Under the assumption of linearity of the embedding and decoding blocks, the paper shows that a model without an encoder degenerates to a linear mapping irrespective of the chosen embedding type. Extensive experiments show that an individual selection of the embedding method given the dataset and encoder can improve the model's performance. The paper further investigates predictive performance of the model under compression of the encoder's outputs and finds that SVD-based rank compression leads to lower MSE values in two thirds of the configurations measured. Finally, they propose a function selecting the best embedding method and encoder combination based on dataset characteristics.

Critical strengths

None

Critical weaknesses

Presentation

The organization of the paper makes it difficult to follow the arguments with some reasons given below. Notions used in explanations should be defined beforehand or at the same place where it first occurs. Here are some examples for which this is not the case

  • line 079: "predictive readability"
  • line 088: "access to forecasting targets"
  • line 372: "interaction families"
  • line 016: "high-order interactions". The abstract mentions them four times, but only on line 095 the reader can deduce that the additional complexity introduced to the model by the encoder is the meaning of this term.
  • line 459: "mean relative MSE regret". What is meant by regret?

The appendix should be the place for additional information and not be imperative to the ability to understand the paper's main part. The main part left out important details, some of them listed under Questions For Authors.

The description of the module-selection rule does not clearly state the needed information for its application hindering its possible adoption.

  • Why have H and the same three squares in each cell of figure 1, and overload H with the encoder representations and the prediction horizon length?
  • Why introduce f when the paper does not use it anymore?
  • Link to Appendix A in line 271 points to the theoretical analysis of the embedding methods instead of the significance tests of the embedding types.

Informativeness

By omitting the encoder component and assuming linearity, it directly follows that the model (linear embedding layer + linear decoder) reduces to a linear projection; the paper presents this, however, as one of the main findings. The empirical analysis of this linear model case extends over three quarters of a page reporting no significant differences despite Appendix C.1 showing a significant difference for the Exchange dataset. A theoretical analysis motivated by the maximum rank of the linear projection could help explain the results. Namely, in two thirds of possible configurations (d∈{128,256}d \in \{128, 256\}) the embedding dimension is larger than the input size, in those cases every linear input-output map is reachable making the respective configurations equally expressive.

Soundness

The paper does not do a great job of justifying the absence of non-linearities in the embedding layers. Although most of the cited methods do use linear embedding layers, iTransformer [1] does utilize an MLP with the GeLU activation function.

Furthermore, the paper is imprecise about how methods are applied to generate the presented results, making it difficult to reproduce the results. The paragraph "Sensitivity to retained spectral directions" does not specify the truncation level, making it hard to interpret the results.

[1] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. (2024, May). itransformer: Inverted transformers are effective for time series forecasting. In International conference on learning representations (Vol. 2024, pp. 11116-11140).

Summary

The paper should be rejected for the following reasons:

(1) Essential details necessary to follow and understand the experiments are pushed into the appendix (see Presentation and questions below). Without them, the main claims cannot be assessed from the main text alone.

(2) The linear assumption on the embedding component is too restrictive to support generalizable claims (see Soundness). Under this assumption, the first-order finding (F1) follows directly from the fact that a composition of linear maps is linear, yet the paper presents it as one of its main findings (see Informativeness).

(3) Inconsistent training budgets across configurations (e.g., five-epoch searches for some encoders on ECL and Traffic, Table 2) limit the comparability of the reported predictive performance, which undermines the view-preference conclusions of RQ2.

Questions for authors
  • Why is the fact that a composition of linear maps is linear presented as a research question and key finding (F1), rather than as a sanity check?
  • What was the experimental setup? Which datasets were used for this study?
  • What exactly constitutes a "configuration"? Encoder type, embedding method and dataset supposedly constitute one configuration, but enumerating all possible combinations leads to 162 (9 datasets, 6 encoders, 3 embeddings) instead of 98 as mentioned in the abstract or line 116 in section 5.3.That is to say: Which configurations were selected and why?
  • Why do 18 configurations not possess a checkpoint? Lines 334-336
  • Why were models trained for a different number of epochs (Table 2)?
  • How many layers were used for the Transformer, MLP, TimeMixer encoders?
  • In which evaluations is the Illness dataset included, from which is it excluded and why? Line 369 says it is included in part of the "rank evolution and predictive readability" evaluation.
  • Section 5.3 replaces the decoder with a linear probe to compute the GprobeG_\mathrm{probe} values. How does replacing the decoder with the linear probe after the encoder differ from just using the decoder?
  • Section 5.4 fits the encoder-embedding selection rule using the datasets. Can the metrics be computed directly from the time series themselves? How transferable to new datasets is that rule?
Reviewer_X6t2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the role of embedding in time series forecasting problem. It specifically shows that this role changes with the order of feature interaction. A key result is that in a certain regime, different embeddings can be equivalent to a linear mapping.

Critical strengths
  • The idea of exploring the role of embedding and viewing it as a dynamic structure rather a static case is important.
  • The investigation on the role of higher order interactions is meaningful.
  • The analysis of latent geometry is useful.
Critical weaknesses
  • A core concern is that the analysis in Section 4 appears to be based on conjecture rather than on rigorous proofs. The discussion in that section is at a very high level and seems rather heuristic and not rigorously grounded.
  • The authors do not clarify why the proposed framework and ensuing results are uniquely tailored towards time-series forecasting. What features of time series forecasting make the results specifically adapted to them?
  • The datasets used mostly have simple time evolution. It is not clear if the results extend to complex dynamics (e.g., time series we see in networks, robotics, etc.)
  • There seems to be an overlap between the contribution of this work and Han et Al. 2024, Chen et al. 2025. The differences appear relatively incremental.
  • The scope of the contribution is very limited, and it is not clear whether the observations made on embeddings can help build better time-series forecasting solutions.
Questions for authors
  • Can you provide rigorous mathematical proofs of the claims made in the paper, particularly Section 4?
  • Can you expand the experimental results to complex time-series datasets?
  • What is the most significant contribution, particularly compared to the prior work surveyed?
  • How do the observations made on embedding behavior help to perform better time-series forecasting?
Reviewer_mD91OpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper systematically revisits the role of embeddings in time-series forecasting from both latent-space and data perspectives. The authors show that, without high-order interactions, different embedding strategies largely collapse to equivalent linear mappings and exhibit limited performance differences. Once an encoder is introduced, however, embedding effectiveness becomes strongly dependent on both the dataset and the encoder architecture. The paper further analyzes representation evolution using effective rank and frozen linear probes, and proposes a multiscale data-rank based selector that can choose embedding–encoder combinations before target-model training.

Critical strengths

Systematic analysis of embedding mechanisms. The paper studies embeddings from first-order mappings to high-order interactions, providing a clearer explanation of when and why different embedding strategies matter.

Broad evaluation across encoder families. Experiments cover Transformer, MLPMixer, GRU, state-space, FFT-based, and CNN encoders, showing that embedding effectiveness strongly depends on the interaction mechanism.

Useful representation-level diagnostics. Effective rank, frozen linear probes, and the multiscale rank-based selector connect embedding behavior with latent geometry, predictive readability, and practical module selection.

Critical weaknesses

Effective rank is not strongly predictive of representation quality by itself. Both rank compression and rank expansion can accompany improved probe performance, making the practical interpretation of effective rank less straightforward.

The practical gain of the analysis framework is not yet fully demonstrated. The paper provides extensive diagnostics of embeddings and latent representations, but it remains less clear how much these insights improve real forecasting systems beyond the relatively small module-selection experiment.

The rank-based selector depends on manually calibrated decision boundaries. Although leave-one-dataset-out calibration avoids directly fitting on the target dataset, the thresholds and encoder ordering are still derived from a small source collection.

Questions for authors

How robust is the proposed DRS-RG selector when applied to datasets or forecasting horizons that differ substantially from the six source datasets used for calibration?

Since effective-rank changes are not consistently aligned with probe improvements, could the authors clarify when rank should be interpreted as a reliable indicator for embedding–encoder selection?

1 comment

Since the author did not give any reply, I think the author has given up the submission.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.