- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an inference-time scaling law for time series foundation models, and a theory of time series learning. The former relates a normalized MASE error to context length, target horizon, and parameter count, while the latter argues that specialized models learn by storing information about the target task in the weights, while foundation models do so by storing that information in the activations as a function of the input context. Experimental results show the proposed scaling law can explain the relationship between the considered variables to an extent, and it compares favorably to some more complex alternatives. The paper supports claims about the learning theory with experiments that show task information can be partially transfered by acting on model weights or activations, and that task-specific tuning can improve performance at fixed input lengths.
Critical strengths
- The work studies the compelling problem of understanding inference-time empirical relationships connecting time series foundation models performance with their parameter, input and horizon scale. Advancing scientific understanding in this direction can help practicioners pick among pre-trained models. Overall, the structure of the proposed scaling laws appears in line with the general understanding that additional context helps performance, longer target horizons are harder to predict, and increasing model size can be beneficial. Previous work does not seem to have directly focused on the specific type of scaling law proposed in this paper.
Critical weaknesses
- The proposed scaling law, while reasonable in its structure, is not accompanied by sufficiently convincing empirical evidence to support the claims of a general scaling law. In particular, the predicted error does not exhibit a strong linear correlation to the true error when evaluated out-of-sample (Figure 3), and the law appears to systematically underestimate error on the Toto evaluation for shorter input lengths (Figure 7). Moreover, extrapolation does not seem to be sufficiently tested: its unclear whether the law extrapolates well to held-out input lengths or horizons, and extrapolation to held-out models or parameter counts is only partially tested through the Toto-specific study. As such, it is unclear whether the scaling law can reliably guide general decision making. The authors claim to use an Occam's Razor approach to select a parsimonious law parametrization, yet results in Table 5, suggest that the contribution of the capacity parameter is marginal and possibly detrimental, hence it is unclear if a parsimonious law should include it. For instance, in Figure 5b, the law only predicts a marginal change in the preferable input length across parameter counts that differ by orders of magnitude. Finally, the claim that the proposed law can support decision making for model building seems at least partially unsubstantiated, as the law deals with inference time relationships, and completely neglects the influence of training budget in terms of compute and data.
- I do not believe the evidence provided in this work supports the claims of development of a unified theory of time series learning. First, the theory itself appears mostly incremental, as the intuition that foundation models may infer the task from the input context is already discussed in prior works, and the principle that task-specific models embed the forecasting rule in their weights is the fundamental way in which they are intended to work. The link to Gaussian regression is unclear, as the paper does not demonstrate a connection between that and how the studied models solve the forecasting task, hence I am unsure what it provides beyond a conceptual intuition. In regards to experimental evidence: swapping task-specific model parameters (Sec. 6.3) amounts essentially to swapping models, so the accuracy results after the swap are expected. The experiment with activation patching (Sec. 6.4) is suggestive, yet the breadth of the evaluation is narrow and the experiment seems to only support the claim that activations may contain information about the target task inferred from context, which is a plausibly expected result and not sufficiently strong to support claims of a unified learning theory.
Questions for authors
- Could the authors explain why the law including the capacity dependency was preferred, given the parsimonious approach?
- How does the law account for "parameter count" not having a homogeneous meaning across different models?
- Does the law mix error on both univariate and multivariate tasks? If so, how does it account for the discrepancy in the amount of information for a multivariate vs univariate task at the same input length?
- Could the authors explain how the law can support model building choices given the lack of relationship to training time effects (e.g., data budget, maximum pre-training length and horizon length)?
- Can the authors explain the purpose of the Gaussian regression discussion in Sec. 6.1?
- Can the authors explain how the experiments in Sec. 6 support the proposed unified learning theory being more than an intuition which is in line with intuitions present in prior works?
Minor questions:
- At line 916 you mention "an aggregated long-context coordinate can include datasets with shorter available histories". Does this mean that error at longer context may potentially include samples that had shorter actual context?
- Is there a reason for using the ratio of MASE as score, given the MASE is already normalized by the naive error? Would the results change if a probabilistic score like CRPS was used?
- Why were in Figure 2 the coefficients not re-fitted for the alternative laws?
- Appendix C.6 does not seem to report any actual result for the horizon-interaction extension.