- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigates the impact different embedding methodologies have on the performance of different forecasting architectures (referred to as encoder by the paper and in the following) on given datasets. Under the assumption of linearity of the embedding and decoding blocks, the paper shows that a model without an encoder degenerates to a linear mapping irrespective of the chosen embedding type. Extensive experiments show that an individual selection of the embedding method given the dataset and encoder can improve the model's performance. The paper further investigates predictive performance of the model under compression of the encoder's outputs and finds that SVD-based rank compression leads to lower MSE values in two thirds of the configurations measured. Finally, they propose a function selecting the best embedding method and encoder combination based on dataset characteristics.
Critical strengths
None
Critical weaknesses
Presentation
The organization of the paper makes it difficult to follow the arguments with some reasons given below. Notions used in explanations should be defined beforehand or at the same place where it first occurs. Here are some examples for which this is not the case
- line 079: "predictive readability"
- line 088: "access to forecasting targets"
- line 372: "interaction families"
- line 016: "high-order interactions". The abstract mentions them four times, but only on line 095 the reader can deduce that the additional complexity introduced to the model by the encoder is the meaning of this term.
- line 459: "mean relative MSE regret". What is meant by regret?
The appendix should be the place for additional information and not be imperative to the ability to understand the paper's main part. The main part left out important details, some of them listed under Questions For Authors.
The description of the module-selection rule does not clearly state the needed information for its application hindering its possible adoption.
- Why have H and the same three squares in each cell of figure 1, and overload H with the encoder representations and the prediction horizon length?
- Why introduce f when the paper does not use it anymore?
- Link to Appendix A in line 271 points to the theoretical analysis of the embedding methods instead of the significance tests of the embedding types.
Informativeness
By omitting the encoder component and assuming linearity, it directly follows that the model (linear embedding layer + linear decoder) reduces to a linear projection; the paper presents this, however, as one of the main findings. The empirical analysis of this linear model case extends over three quarters of a page reporting no significant differences despite Appendix C.1 showing a significant difference for the Exchange dataset. A theoretical analysis motivated by the maximum rank of the linear projection could help explain the results. Namely, in two thirds of possible configurations () the embedding dimension is larger than the input size, in those cases every linear input-output map is reachable making the respective configurations equally expressive.
Soundness
The paper does not do a great job of justifying the absence of non-linearities in the embedding layers. Although most of the cited methods do use linear embedding layers, iTransformer [1] does utilize an MLP with the GeLU activation function.
Furthermore, the paper is imprecise about how methods are applied to generate the presented results, making it difficult to reproduce the results. The paragraph "Sensitivity to retained spectral directions" does not specify the truncation level, making it hard to interpret the results.
[1] Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., & Long, M. (2024, May). itransformer: Inverted transformers are effective for time series forecasting. In International conference on learning representations (Vol. 2024, pp. 11116-11140).
Summary
The paper should be rejected for the following reasons:
(1) Essential details necessary to follow and understand the experiments are pushed into the appendix (see Presentation and questions below). Without them, the main claims cannot be assessed from the main text alone.
(2) The linear assumption on the embedding component is too restrictive to support generalizable claims (see Soundness). Under this assumption, the first-order finding (F1) follows directly from the fact that a composition of linear maps is linear, yet the paper presents it as one of its main findings (see Informativeness).
(3) Inconsistent training budgets across configurations (e.g., five-epoch searches for some encoders on ECL and Traffic, Table 2) limit the comparability of the reported predictive performance, which undermines the view-preference conclusions of RQ2.
Questions for authors
- Why is the fact that a composition of linear maps is linear presented as a research question and key finding (F1), rather than as a sanity check?
- What was the experimental setup? Which datasets were used for this study?
- What exactly constitutes a "configuration"? Encoder type, embedding method and dataset supposedly constitute one configuration, but enumerating all possible combinations leads to 162 (9 datasets, 6 encoders, 3 embeddings) instead of 98 as mentioned in the abstract or line 116 in section 5.3.That is to say: Which configurations were selected and why?
- Why do 18 configurations not possess a checkpoint? Lines 334-336
- Why were models trained for a different number of epochs (Table 2)?
- How many layers were used for the Transformer, MLP, TimeMixer encoders?
- In which evaluations is the Illness dataset included, from which is it excluded and why? Line 369 says it is included in part of the "rank evolution and predictive readability" evaluation.
- Section 5.3 replaces the decoder with a linear probe to compute the values. How does replacing the decoder with the linear probe after the encoder differ from just using the decoder?
- Section 5.4 fits the encoder-embedding selection rule using the datasets. Can the metrics be computed directly from the time series themselves? How transferable to new datasets is that rule?
Since the author did not give any reply, I think the author has given up the submission.