- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper analyzes the benefits and shortcomings of channel-dependent and channel-independent forecasting. It employs iTransformer and SAMFormer in experiments across two datasets, Solar and ECL (except 4.1 where ETTm1 and Exchange are also covered). A series of experiments analyzes types of interactions and their impact on the forecasting performance measured by MSE. Most experiments focus on the log-MSE ratio of the models using an identity attention mask (no-cross) and dense (full mixing). The main results show that short-horizon (Short) forecasts tend to benefit from cross-channel access, while long-horizon (Far) forecasts do not show much change in no-cross vs dense settings. The last set of experiments explores five arbitrary strategies of removing cross-channel edges and finds that removing up to 90% of them does not significantly hurt MSE (according to the introduced noninferiority criterion).
Overall, the paper focuses on an interesting topic and runs relevant experiments. However, the methodology, the depth of experiments, and the analyses are significantly limited, causing little to no generalizability of the observed behaviors. Moreover, the paper is hard to follow at times because there are plenty of undefined terms and vague sentences; paragraphs are loaded with numbers, and a lot of important context paramount for understanding the contents of a given section is moved to the appendix (for instance, section 6 is difficult to read without constantly looking into the appendix).
Critical strengths
- The paper asks interesting questions about cross-channel interactions in multivariate forecasting and analyzes them to a certain extent.
- The Short vs Far division is a smart idea and the results favoring channel-interactions for Short are insightful (for iTransformer).
Critical weaknesses
- The majority of experiments are restricted to only two datasets (Solar, ECL). It is critical to run this analysis across various datasets and domains because they exhibit diverse patterns of cross-channel interactions.
- Only one model type is evaluated (iTransformer; SAMFormer is based on the same architecture). This is a significant limitation of this work because iTransformer-style models compress each variate into a single token and only after that apply cross-variate attention. Multivariate forecasting is by no means dominated by only iTransformer-style models, and the analysis performed by the authors carries no generalizable information about multivariate forecasting.
- The fixed 96 context-horizon setup is a severe limitation. It is not unlikely that different datasets would exhibit different spurious/useful cross-variate correlations, which would impact the observed results. Furthermore, the division into Short (1-48) and Far (49-96) is also arbitrary, and some sort of sensitivity analysis on it should be performed.
- Analyses mostly focus on numbers and confidence intervals while omitting detailed explanations of results and the experimental setup. Moreover, some of the analyses seem to be logically flawed. For instance, in Section 4.1: “No-cross changes MSE relative to Dense by -1.31% and -1.13% with one-sided 95% upper bounds of -0.87% and - 0.51%.” This is a contradiction because the CI doesn’t cross zero, so the CIs actually suggest that the improvement of Dense vs No-Cross on ETTm1 and Exchange is also there, it's just that its magnitude is small. Moreover, I don’t see how they satisfy the 1% non-inferiority criterion since the means of the CIs are larger than -1%.
Writing/formatting:
- L194, L200: “receivers” is a vague term that hasn’t been clearly defined. Presumably, receivers mean series/channels/variates, but this is not clear.
- I think it would improve the formatting and clarity if standard terms like joint-96, Dense, No-cross, Far, and Short were set in a distinct font.
- Undefined
- compact iTransformer.
- Random-95%.
- test worlds
- Random mask
- original-zero receiver
- improvements in all 20 families
- DGP
- n = 5, seeds 2–6 (Figure 3) // there are 6 seeds in this interval but n=5
Questions for authors
- What does “20 paired Dense-No-cross training blocks for each of four datasets” precisely refer to? (L183).
- How was instance normalization applied during training? Specifically, were model predictions inverse-scaled back to the original scale for loss computation?