- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Structural Scaling Response (SSR), a metric used to measure how much the metric improvement by scaling the size of a time series foundation model is attributed to the temporal structure of the evaluation data. SSR is measured by evaluating the difference between the improvement on the downstream dataset and the improvement on randomized Fourier features as the model scales up. By maintaining the power spectrum of the Fourier phases, the SSR metric can be used to distinguish between the gains from the original temporal structure and the gains that persist under phase randomization. The paper then applies SSR to many classes of time series foundation models, including Toto, Chronos, and Moirai to show the scaling behavior of each class of models.
Critical strengths
- The question studied in the paper is interesting and novel. It does not only look at the "scaling law" of time series foundation models by analyzing the standard metrics, but also asks where the gain is coming from. I think this is a more interesting problem that will otherwise be overlooked if one only focuses on the metrics.
- SSR provides a straightforward, training-free, and intuitive diagnostic that can be applied to existing pretrained models. It does not depend on the architecture of the model class per se, which makes it widely applicable.
- The empirical evaluation, while not completely self-explanatory (see my comments in the weakness section), leads to interesting findings on different time series foundation model classes, which potentially suggest revealing distinctions between them.
Critical weaknesses
- The paper offers one potential approach to analyze the attribution of gains from a larger time series foundation model via SSR. The proposed method is intuitive, and provides a reasonable way to distinguish between different sources of the gain, but it is unclear why this particular analysis should be considered as the appropriate test. In particular, the paper would benefit from a discussion of the limitations of SSR, and how it would compare to other possible evaluation metrics which are not proposed in the paper.
- While the paper shows interesting findings related to the different classes of time series foundation models, it does not totally explain why such a distinction exists. I think the paper can be strengthened by a brief discussion or explanation of this distinction, followed by some stronger evidence beyond the SSR measurement to demonstrate this difference.
Questions for authors
- While the randomized Fourier feature preserves the spectral information, it would change some other properties of the time series. What makes SSR particularly informative compared with alternative ways of modifying temporal structure? Could other interventions lead to qualitatively different conclusions about the same model upgrades?
- Do the authors have hypotheses about which attributes, including the model architecture and training datasets, contribute to the contrasting behavior of Toto, Chronos, and Moirai?
- How sensitive are the findings to forecasting horizon, context length, and temporal resolution? It would be interesting to see whether the SSR scores and the identified frequency ranges remain stable under different forecasting configurations.