- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies feature selection in time series forecasting. The authors first propose Conditional Distribution Difference (CDD) and examine its properties and implications using simulated data. CDD empirically shown to be a metric that reflects temporal instability, and Lower CDD-std refers to greater gains from exogenous features. Based on this finding, the authors proposed CDD-std hard filtering and Timeless, a CDD-std guided learning method. Various results were conduced and CDD-std based achieve overall the best performance.
Critical strengths
Pros:
- This paper is well-motivation and easy to follow.
- The proposed method is simple and effective. The hard filtering method can also easily apply to retrained foundation models.
Critical weaknesses
Cons:
- In Figure 4b, the test error shows a decrease and then increase pattern with the increase of number of layers. It would be interesting to also explain why this pattern exists.
- It would be helpful to analyse the computational complexity given Eqs 1 and 5 and how the number of bins affects runtime.
- The title of sec. 3.2, it seems not appropriate calling the method as meta learning.
Questions for authors
For multivariate forecasting, can you explain how TIMELESS is applied to forecast each variate?
What is the (possible) relationship between TIMELESS performance and correlations with covariates?
2 comments
Thank the authors for the explanation, which helped me understand more. I would keep my score.
Rebuttal to Reviewer G6ds
Thank you for the valuable assessment and constructive suggestions.
W1: Error trend with network depth
We believe the reviewer refers to Fig. 2(b). The initial decrease in test error reflects improved modeling capacity, while the subsequent increase is mainly due to overfitting. In Fig. 2(b), the varying-frequency trend and added noise make the underlying pattern harder to distinguish, causing deeper networks to overfit noisy patterns. In contrast, the fixed-frequency trend in Fig. 2(a) is easier to learn, making overfitting less pronounced.
W2: Complexity analysis
Let T be the training length, d the number of exogenous features, w and h the window and horizon lengths, s the sliding step, and K the number of temporal bins. With N=(T-w-h)/s windows, directly evaluating Eq. 1 requires comparing all pairs of windows, resulting in O(dN^2) Wasserstein distance computations. Our bin-wise construction compares each window only with the reference of its corresponding bin, reducing the number of comparisons to O(dN). The resulting computational complexity is O(dN(w+h+(m+T/K) log T)).
Increasing K reduces the reference size T/K and therefore does not increase the computational cost. CDD-std is computed only once on the training split, with marginal overhead. For example, on Traffic (862 features), computing dataset-level CDD-std takes less than 2 minutes.
For Eq. 5, the CDD-std vector is precomputed during dataset initialization, before model training. The MLP therefore only introduces a small additional computational overhead. Its architecture and computational cost are independent of the number of temporal bins.
W3: Terminology
We agree that "meta learning" could be inaccurate. We will use "Adaptive Feature Weighting" to better reflect how TimeLess uses CDD-std as feature-level metadata to guide feature weighting.
Q1: Multivariate forecasting
Timeless can be extended to multivariate forecasting following the multi-univariate forecasting paradigm, as also discussed in [1]. First, CDD-std is computed for each target-feature pair to obtain target-specific feature descriptors. Second, the descriptors are organized into a batched tensor with an additional variate dimension (e.g., [batch, n_variates, ...], instead of [batch, ...]) and jointly processed in a single MLP forward pass to generate target-specific feature weights and selection masks. Finally, the resulting weights and masks are applied to the corresponding target's forecasting process.
Q2: TimeLess performance and correlation with covariates
To investigate the relationship between feature-target correlation and TimeLess gains, we analyze the eight datasets in Fig. 1(b) for which TimeLess results are available with the four main models (Tables 11, 12, and 16). We compute the Spearman correlation between each dataset's overall feature-target correlation and the MSE reduction of TimeLess over Base, averaged across horizons and models. The correlation is -0.19 (p=0.66); restricting the analysis to the datasets used in Table 1 yields 0.49 (p=0.36). Neither result shows a clear relationship between feature-target correlation and TimeLess gains.
This is consistent with our intuition in Fig. 1(b): overall feature-target correlation is not a reliable indicator of exogenous feature usefulness, as it does not capture temporal stability. TimeLess instead uses CDD-std to identify and down-weight unstable features.
[1] Time-Series Forecasting for Out-of-Distribution Generalization Using Invariant Learning. ICML