- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies predictive multiplicity in time-series forecasting and introduces a trajectory-level perspective termed temporal predictive multiplicity. The authors organize predictive multiplicity into a hierarchy of pointwise, horizon-wise, and temporal notions, and characterize temporal disagreement using several trajectory properties including magnitude, increments, volatility, direction, shape, spectral structure, and timing. The paper further provides theoretical results showing that similar aggregate forecasting performance, and even tight horizon-wise agreement, do not necessarily imply similar forecast trajectories. Experiments across 19 forecasting architectures and 11 datasets demonstrate that near-optimal models can exhibit substantial trajectory-level disagreement and may lead to different downstream conclusions.
Critical strengths
i) The paper provides a careful formalization of predictive multiplicity for time-series forecasting. The distinction between pointwise, horizon-wise, and temporal multiplicity is clearly defined and theoretically characterized.
ii) The theoretical analysis is extensive. The paper derives bounds showing that models with similar predictive performance can still differ substantially in trajectory-level properties, and the appendices provide detailed proofs and constructions.
iii) The empirical study is broad and carefully designed. The experiments cover many forecasting architectures, datasets, random seeds, and hyperparameter settings, and the Rashomon set is selected using validation performance while multiplicity is evaluated on test forecasts.
Critical weaknesses
i) The central observation of the paper appears somewhat intuitive. Similar aggregate predictive performance does not generally imply similar prediction behavior. Analogous phenomena are common in other predictive settings, where models with the same overall accuracy may exhibit substantially different class-wise performance, instance-wise predictions, or error distributions. While the paper provides a careful temporal formalization of this phenomenon, it remains unclear whether the underlying insight itself is sufficiently novel.
ii) The methodological novelty of the trajectory measures used in the empirical analysis is limited. Measures such as RMS differences, increments, volatility, sign disagreement, Pearson correlation, Fourier spectra, and lag-based alignment are largely based on established tools. Therefore, much of the empirical framework appears to organize and apply existing measures rather than introduce fundamentally new trajectory metrics.
iii) The paper primarily diagnoses temporal multiplicity but provides limited guidance on how it should be addressed. Given that models with similar predictive accuracy can reasonably be expected to produce different trajectories when aggregate performance is the sole selection criterion, a stronger practical contribution would be to propose a principled model-selection criterion that incorporates trajectory-level behavior, or a learning or regularization method that controls undesirable temporal multiplicity while preserving predictive accuracy.
iv) Some of the theoretical results appear to formalize behavior that is already qualitatively expected from aggregate loss functions. Since an aggregate loss constrains only the total error budget and does not directly control how errors are distributed across forecast horizons, substantial differences in trajectory shape, direction, or timing are not entirely surprising. The theoretical analysis is rigorous, but the paper could better clarify which results provide genuinely new insights beyond formalizing this intuition.
Questions for authors
i) Similar aggregate predictive performance is generally expected to allow different prediction behaviors, as seen in other predictive settings where models with the same overall accuracy may have different class-wise or instance-wise error patterns. What aspects of temporal predictive multiplicity do the authors view as fundamentally new beyond this general phenomenon?
ii) Many of the trajectory distances used in the paper are based on standard measures such as RMS differences, correlation, Fourier spectra, and lag-based alignment. Can the authors clarify which components of the temporal multiplicity framework should be considered methodologically novel rather than a structured application of existing measures?
iii) The paper demonstrates that near-optimal models can exhibit substantial temporal multiplicity, but it remains unclear how practitioners should act on this finding. Could the authors discuss or evaluate a model-selection criterion or learning objective that explicitly accounts for trajectory-level multiplicity while maintaining predictive accuracy?
iv) Several theoretical results formalize the fact that aggregate loss does not constrain how errors are distributed across forecast horizons. Could the authors clarify which theoretical findings go beyond this intuitive consequence and provide new implications for forecasting practice or learning theory?
We thank the reviewer for their thoughtful comments.
1. That aggregate accuracy underdetermines behavior is a general phenomenon, and we do not claim it as our contribution. The reviewer’s examples, instance-wise and class-wise differences, correspond to the pointwise and horizon-wise levels of our hierarchy, which is what prior work measures. What is new is specific to forecasts being structured objects: the H predictions are used jointly as one trajectory, and properties such as direction of change, shape and peak timing exist only at that joint level. We show that disagreement on these properties is (a) provably invisible to pointwise and horizon-wise statistics, which are invariant to permuting origins within a horizon (Definition 3.4 and Appendix C), and unconstrained even under tight horizon-wise control (Theorem 4.2 and Appendix D); and (b) empirically only weakly related to them (rank correlations of 0.04–0.47, Table 2). In extending multiplicity to time series, we also generalize it beyond forecasts to revision and error trajectories.
The core insight is therefore not that equally accurate models disagree, but that how much they disagree at individual horizons says little about whether they disagree as trajectories. The size of the effect is also not predictable from the general phenomenon: on 80% of forecast cells, at least one near-optimal model predicts an increase while another predicts a decrease (Table 3). To our knowledge, this has not been formalized or measured for forecasting.
2. The distances are deliberately standard, and we do not claim them as a contribution. Definition 3.3 leaves the distance d unspecified, and Section 5 instantiates it with familiar choices so that each statistic has a clear interpretation for practitioners. The methodological contributions are: (i) the general definition of temporal multiplicity (Definition 3.3), which lifts range, ambiguity and discrepancy from scalar predictions to trajectories under any distance, together with the diameter, which is new; (ii) the three-level hierarchy and its permutation-invariance characterization (Definition 3.4, Appendix C), which establishes what pointwise and horizon-wise statistics cannot detect; (iii) the bounds each distance inherits from Theorems 4.1 and 4.2 (Appendix D); and (iv) the application to revision and error trajectories alongside forecast trajectories, which to our knowledge has no precedent in the multiplicity literature.
3. Selection and regularization methods are a natural next step, but they are outside the scope of this paper for two reasons.
First, any such method needs a target, and the appropriate one is application-dependent: disagreement in peak timing matters for demand planning, disagreement in direction for trading. This is why Definition 3.3 leaves the distance unspecified, and why a single criterion would be premature.
Second, temporal multiplicity is not always undesirable: as in the broader multiplicity literature (Black et al., 2022; Rudin et al., 2024), the freedom among equally accurate models can be used to select on secondary objectives, so whether to reduce or exploit it is a choice that first requires measuring it.
Our framework provides that measurement, and it already supports practice: a practitioner can audit near-optimal candidates on the trajectory property relevant to their task and justify the model they deploy. We added Section 7, a discussion section with practical guidance on using the statistics for model selection, and listed trajectory-aware selection and regularization as future work.
4. The intuition the reviewer describes, that an aggregate loss fixes only a total error budget, corresponds to Theorem 4.1, where our addition is that the bound is attained and that the freedom grows with the horizon as H1/p. Theorem 4.1 is just the start of our theoretical analysis; our main theoretical results concern a much stronger constraint than aggregate loss. Definition 3.4, Theorem 4.2 and the consequent corollaries in Appendix D show that even when disagreement is controlled at every individual horizon, direction, shape and timing remain unconstrained, while magnitude, increments and volatility are bounded. This split between what horizon-wise control does and does not determine does not follow from the error-budget argument, and it is what we confirm empirically (Table 2). As noted in our response to point (1), this is the central contribution, and we have revised Section 4 to make the distinction explicit.