- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether time series similarity based retrieval is a useful metric for time series retrieval. The paper has two conclusions: (i) although similarity-based retrieval is useful and benefits the downstream model, there are cases where random retrieval is equally beneficial. (ii) the retrieval should be domain-aware in that the utility function of retrieving certain series in a domain should depend on the domain itself. While getting to that conclusion, the paper establishes a useful analysis framework. My main criticism with respect to this is that benchmark (LTSF benchmark) set is limited in diversity and the paper would benefit greatly from a more encompassing benchmark set.
Critical strengths
- The paper investigates the utility of similarity-based selection, which is quite important in time where there are lots of retrieval methods in time series forecasting.
- Overall, I find the analytical framework in this paper valuable.
Critical weaknesses
- The conclusion of this paper depends on LTSF datasets, which are quite limited in diversity. Especially, exchange dataset is known to be near random walk, and hence it is obvious to me that random selection will yield competitive if not better results. Similarly, ETT datasets are known to be highly periodic.
Questions for authors
I suggest authors add Gift-eval or Monash datasets in the analysis. In the current form, experiments are my main concern and addressing this issue will reflect in a change in my assessment.
Response to the Review [1/1]
Thank you for asking us to test across a wider range of domains and series characteristics.
1. Do RQ1–RQ3 hold on broader benchmarks?
We evaluated 43 dataset entries across GIFT-Eval (24 datasets, with 209 sampled time series) and Monash (19 datasets, with 100 sampled time series). The two benchmarks cover a wide range of domains and sampling frequencies (see Tables 14–17 of Appendix E, pp. 20–25). Specifically,
Overall, these results demonstrate that the three main findings persist across datasets covering a substantially wider range of domains and sampling frequencies. The weak correlation between similarity and predictive utility, together with the interface dependence of candidate utility, is therefore not limited to the periodic ETT datasets or the Exchange dataset in our original evaluation.