acceptodds
Under review as a conference paper at ICLR 2027

Temporal Predictive Multiplicity: Equally Accurate Time Series Models Yield Different Forecast Trajectories

Abstract

Models with near-identical predictive performance can yield substantially different predictions, a phenomenon known as _predictive multiplicity_. Prior work has mostly studied this at the level of individual scalar outputs. In time-series forecasting, however, predictions across horizons jointly define a trajectory, and horizon-wise comparisons can hide important differences in predictive behavior. To address this problem, we introduce _temporal predictive multiplicity_, a framework that characterizes disagreement over complete forecast trajectories among models with near-identical predictive performance. We show that constraining predictive performance alone can still admit a broad range of different trajectories. We further show that constraining multiplicity at individual horizons partially reduces, but does not eliminate, trajectory-level multiplicity. Experiments with 19 neural forecasting architectures on 11 datasets confirm that near-optimal models can exhibit substantial variability in the forecast trajectories they produce, and trajectory-level disagreement is largely unrelated to horizon-wise disagreement. Our framework, therefore, exposes a gap in existing multiplicity studies: models with indistinguishable predictive performance imply fundamentally different temporal trajectories, with consequential downstream effects.

Primary area:
learning on time series and dynamical systems
Keywords:
Predictive Multiplicity, Rashomon Effect, Trustworthy AI

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_A2wCOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies predictive multiplicity in time-series forecasting and introduces a trajectory-level perspective termed temporal predictive multiplicity. The authors organize predictive multiplicity into a hierarchy of pointwise, horizon-wise, and temporal notions, and characterize temporal disagreement using several trajectory properties including magnitude, increments, volatility, direction, shape, spectral structure, and timing. The paper further provides theoretical results showing that similar aggregate forecasting performance, and even tight horizon-wise agreement, do not necessarily imply similar forecast trajectories. Experiments across 19 forecasting architectures and 11 datasets demonstrate that near-optimal models can exhibit substantial trajectory-level disagreement and may lead to different downstream conclusions.

Critical strengths

i) The paper provides a careful formalization of predictive multiplicity for time-series forecasting. The distinction between pointwise, horizon-wise, and temporal multiplicity is clearly defined and theoretically characterized.

ii) The theoretical analysis is extensive. The paper derives bounds showing that models with similar predictive performance can still differ substantially in trajectory-level properties, and the appendices provide detailed proofs and constructions.

iii) The empirical study is broad and carefully designed. The experiments cover many forecasting architectures, datasets, random seeds, and hyperparameter settings, and the Rashomon set is selected using validation performance while multiplicity is evaluated on test forecasts.

Critical weaknesses

i) The central observation of the paper appears somewhat intuitive. Similar aggregate predictive performance does not generally imply similar prediction behavior. Analogous phenomena are common in other predictive settings, where models with the same overall accuracy may exhibit substantially different class-wise performance, instance-wise predictions, or error distributions. While the paper provides a careful temporal formalization of this phenomenon, it remains unclear whether the underlying insight itself is sufficiently novel.

ii) The methodological novelty of the trajectory measures used in the empirical analysis is limited. Measures such as RMS differences, increments, volatility, sign disagreement, Pearson correlation, Fourier spectra, and lag-based alignment are largely based on established tools. Therefore, much of the empirical framework appears to organize and apply existing measures rather than introduce fundamentally new trajectory metrics.

iii) The paper primarily diagnoses temporal multiplicity but provides limited guidance on how it should be addressed. Given that models with similar predictive accuracy can reasonably be expected to produce different trajectories when aggregate performance is the sole selection criterion, a stronger practical contribution would be to propose a principled model-selection criterion that incorporates trajectory-level behavior, or a learning or regularization method that controls undesirable temporal multiplicity while preserving predictive accuracy.

iv) Some of the theoretical results appear to formalize behavior that is already qualitatively expected from aggregate loss functions. Since an aggregate loss constrains only the total error budget and does not directly control how errors are distributed across forecast horizons, substantial differences in trajectory shape, direction, or timing are not entirely surprising. The theoretical analysis is rigorous, but the paper could better clarify which results provide genuinely new insights beyond formalizing this intuition.

Questions for authors

i) Similar aggregate predictive performance is generally expected to allow different prediction behaviors, as seen in other predictive settings where models with the same overall accuracy may have different class-wise or instance-wise error patterns. What aspects of temporal predictive multiplicity do the authors view as fundamentally new beyond this general phenomenon?

ii) Many of the trajectory distances used in the paper are based on standard measures such as RMS differences, correlation, Fourier spectra, and lag-based alignment. Can the authors clarify which components of the temporal multiplicity framework should be considered methodologically novel rather than a structured application of existing measures?

iii) The paper demonstrates that near-optimal models can exhibit substantial temporal multiplicity, but it remains unclear how practitioners should act on this finding. Could the authors discuss or evaluate a model-selection criterion or learning objective that explicitly accounts for trajectory-level multiplicity while maintaining predictive accuracy?

iv) Several theoretical results formalize the fact that aggregate loss does not constrain how errors are distributed across forecast horizons. Could the authors clarify which theoretical findings go beyond this intuitive consequence and provide new implications for forecasting practice or learning theory?

1 comment

We thank the reviewer for their thoughtful comments.

1. That aggregate accuracy underdetermines behavior is a general phenomenon, and we do not claim it as our contribution. The reviewer’s examples, instance-wise and class-wise differences, correspond to the pointwise and horizon-wise levels of our hierarchy, which is what prior work measures. What is new is specific to forecasts being structured objects: the H predictions are used jointly as one trajectory, and properties such as direction of change, shape and peak timing exist only at that joint level. We show that disagreement on these properties is (a) provably invisible to pointwise and horizon-wise statistics, which are invariant to permuting origins within a horizon (Definition 3.4 and Appendix C), and unconstrained even under tight horizon-wise control (Theorem 4.2 and Appendix D); and (b) empirically only weakly related to them (rank correlations of 0.04–0.47, Table 2). In extending multiplicity to time series, we also generalize it beyond forecasts to revision and error trajectories.

The core insight is therefore not that equally accurate models disagree, but that how much they disagree at individual horizons says little about whether they disagree as trajectories. The size of the effect is also not predictable from the general phenomenon: on 80% of forecast cells, at least one near-optimal model predicts an increase while another predicts a decrease (Table 3). To our knowledge, this has not been formalized or measured for forecasting.

2. The distances are deliberately standard, and we do not claim them as a contribution. Definition 3.3 leaves the distance d unspecified, and Section 5 instantiates it with familiar choices so that each statistic has a clear interpretation for practitioners. The methodological contributions are: (i) the general definition of temporal multiplicity (Definition 3.3), which lifts range, ambiguity and discrepancy from scalar predictions to trajectories under any distance, together with the diameter, which is new; (ii) the three-level hierarchy and its permutation-invariance characterization (Definition 3.4, Appendix C), which establishes what pointwise and horizon-wise statistics cannot detect; (iii) the bounds each distance inherits from Theorems 4.1 and 4.2 (Appendix D); and (iv) the application to revision and error trajectories alongside forecast trajectories, which to our knowledge has no precedent in the multiplicity literature.

3. Selection and regularization methods are a natural next step, but they are outside the scope of this paper for two reasons.

First, any such method needs a target, and the appropriate one is application-dependent: disagreement in peak timing matters for demand planning, disagreement in direction for trading. This is why Definition 3.3 leaves the distance unspecified, and why a single criterion would be premature.

Second, temporal multiplicity is not always undesirable: as in the broader multiplicity literature (Black et al., 2022; Rudin et al., 2024), the freedom among equally accurate models can be used to select on secondary objectives, so whether to reduce or exploit it is a choice that first requires measuring it.

Our framework provides that measurement, and it already supports practice: a practitioner can audit near-optimal candidates on the trajectory property relevant to their task and justify the model they deploy. We added Section 7, a discussion section with practical guidance on using the statistics for model selection, and listed trajectory-aware selection and regularization as future work.

4. The intuition the reviewer describes, that an aggregate loss fixes only a total error budget, corresponds to Theorem 4.1, where our addition is that the bound is attained and that the freedom grows with the horizon as H1/pH^{1/p}. Theorem 4.1 is just the start of our theoretical analysis; our main theoretical results concern a much stronger constraint than aggregate loss. Definition 3.4, Theorem 4.2 and the consequent corollaries in Appendix D show that even when disagreement is controlled at every individual horizon, direction, shape and timing remain unconstrained, while magnitude, increments and volatility are bounded. This split between what horizon-wise control does and does not determine does not follow from the error-budget argument, and it is what we confirm empirically (Table 2). As noted in our response to point (1), this is the central contribution, and we have revised Section 4 to make the distinction explicit.

Reviewer_jiFBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies temporal predictive multiplicity in time-series forecasting: models with near-identical predictive performance may produce different forecast trajectories, while pointwise or horizon-wise disagreement statistics cannot fully characterize differences in their joint behavior across forecast horizons. The paper considers the Rashomon set of models with near-optimal aggregate loss and assumes that the loss is the average of nonnegative loss terms over forecast cells. On this basis, it develops a framework for predictive multiplicity at the pointwise, horizon-wise, and temporal levels and concludes that constraining aggregate predictive loss alone still permits substantial differences in forecast trajectories. Even when differences between models at each forecast horizon are bounded by a given threshold, trajectories may still differ substantially in properties such as direction of change, shape, and spectral behavior. Experiments with 19 neural forecasting architectures on 11 datasets support these conclusions.

Critical strengths
  1. This paper studies differences in forecast trajectories among models with similar predictive performance, highlighting that selecting models solely based on aggregate loss may overlook predictive behavior relevant to downstream decisions.
  2. The paper evaluates 19 neural forecasting architectures on 11 datasets, demonstrating differences in forecast trajectories among near-optimal models, and uses a case study to show that these differences may affect downstream decisions.
Critical weaknesses
  1. The main findings and conclusions of this paper are relatively intuitive and closely resemble existing observations in related work. For example, models with similar predictive performance may produce different forecast trajectories; this phenomenon is unsurprising when predictions are high-dimensional vectors but performance is measured only through a single aggregate scalar loss. Likewise, horizon-wise disagreement statistics cannot fully characterize the joint dependence structure across forecast horizons. This observation conceptually parallels the classical fact that marginal distributions do not uniquely determine a joint distribution. “Uncertainty Quantification in Complex Simulation Models Using Ensemble Copula Coupling” [1] has already investigated the importance of preserving temporal dependence in joint probabilistic forecasting.
  2. Beyond identifying these issues, related work has also proposed methods to address them. For example, “On Forecast Stability” [2] studies the temporal stability of forecasts and proposes methods for controlling the trade-off between accuracy and stability. “Horizon-Constrained Rashomon Sets for Chaotic Forecasting” [3] studies how predictive multiplicity within Rashomon sets varies across forecast horizons in chaotic systems and proposes a model selection method based on downstream decision utility. Compared with these works, the framework in this paper remains primarily focused on describing and quantifying relatively intuitive phenomena.
  3. The theoretical contributions of the main theorems are also limited. The upper bound in Theorem 4.1 follows simply from the fact that an individual loss term cannot exceed the total loss budget, together with the inverse mapping of the loss function and the triangle inequality. Its construction attaining equality relies on the shell-richness assumption to concentrate each of the two models’ entire error budget at the same forecast cell, with errors of opposite signs. The upper-bound part of Theorem 4.2 is essentially a restatement of its premise by definition.

[1] Schefzik, Roman, Thordis L. Thorarinsdottir, and Tilmann Gneiting. “Uncertainty quantification in complex simulation models using ensemble copula coupling.” (2013): 616–640.

[2] Godahewa, Rakshitha, et al. “On forecast stability.” International Journal of Forecasting (2025).

[3] Kale, Gauri, et al. “Horizon-constrained Rashomon sets for chaotic forecasting.” AIP Advances 16.4 (2026).

Questions for authors

Please see the weaknesses.

3 comments

We thank the reviewer for their thoughtful comments.

1. Our claim is not that near-optimal models differ, but how: trajectory-level disagreement is largely unrelated to horizon-wise disagreement. Models that agree closely at every horizon can still disagree on direction, shape and timing. We show this theoretically (Definition 3.4, Theorem 4.2, Appendices C and D) and empirically (Section 6, Appendices B and D.3), for forecast trajectories as well as revision and error trajectories.

We thank the reviewer for the pointer to Schefzik et al. [1], which we now cite. As the reviewer notes, the common ground is the classical fact that marginals do not determine a joint distribution. [1] applies this fact to a different problem: restoring the dependence structure of a single probabilistic ensemble forecast after univariate postprocessing. It does not study disagreement among the trajectories of competing models. Moreover, although the method is formulated generally, its quantitative evaluation is limited to three dimensions (one variable at three locations, Section 5.2) and concerns dependence across locations only. None of its experiments concerns dependence across lead times, that is, temporal trajectories, which the authors explicitly defer to future work (Section 5.4).

2. We discuss both works in Sections 2 and 3. Both propose remedies, but each targets one specific, pre-chosen kind of disagreement (the changes between a single model’s successive forecasts in [2], a horizon-separable utility in [3]), and neither addresses disagreement between the trajectories of near-optimal models.

"On Forecast Stability" [2] stabilizes a single model's forecasts, across origins or within the output window, by interpolating between them. Disagreement between models at the same origin and target is named ("replicability") but explicitly set aside as an ensembling problem, without being defined, measured, or remediated. It corresponds to our pointwise level.

Kale et al. [3] define a separate Rashomon set at each lead time and do not reduce multiplicity: they select one near-optimal model using a user-supplied utility that is a weighted sum of per-horizon terms. Their own experiments confirm that the members of a Rashomon set are not interchangeable: a model picked at random from the set achieves lower decision utility than the one their rule selects. Their rule, however, can only separate models on what a horizon-separable utility measures.

Any remedy of either kind needs a target, and the appropriate one is application-dependent: disagreement in peak timing matters for demand planning, disagreement in direction for trading. A selection rule can only separate models on properties its utility depends on, and Theorem 4.2, the consequent corollaries in Appendix D, and the experiments in Table 2 show that horizon-wise quantities leave direction, shape and timing largely unconstrained. A horizon-separable utility or a pointwise average, therefore, cannot distinguish the two groups of models in Figure 3. This is why Definition 3.3 leaves the distance unspecified: our statistics show on which properties near-optimal models actually disagree, and hence what a selection or stabilization criterion must capture for a given task.

Moreover, temporal multiplicity is not always undesirable. As in the broader multiplicity literature (Black et al., 2022; Rudin et al., 2024), and as [3] itself illustrates, the freedom among equally accurate models can be used to select on secondary objectives. Whether to reduce or exploit it is a choice that first requires measuring it, and that characterization and measurement is our contribution. We added Section 7, a discussion section with practical guidance on using the statistics for model selection, and listed trajectory-aware selection and regularization as future work.

3. Theorems 4.1 and 4.2 are tightness results: their role is to establish limits. Because both bounds are attained, neither near-optimal loss nor horizon-wise control, the two constraints used by existing multiplicity analysis, can guarantee more about trajectories without further assumptions on the model class. The theoretical analysis then continues in Appendix D, where nine corollaries instantiate the two theorems for each of our seven trajectory distances (Table 21). They show that both constraints bound magnitude, increment and volatility disagreement, with horizon-wise control sharpening the bounds to functions of the per-horizon tolerances, but that direction, shape, spectral and timing disagreement can remain maximal under either constraint, for any nonzero tolerance. These corollaries are what predict the split confirmed experimentally in Table 2, and Appendix D.3 measures how close the empirical Rashomon set comes to each bound.

Since the extremal construction is a worst case, the paper does not stop at the theory. Across 19 architectures and 11 datasets, some pair of near-optimal models moves in opposite directions on 84% of increments at a typical origin (Table 1). The predicted split also appears empirically: horizon-wise disagreement correlates almost perfectly with magnitude (0.98–0.99), moderately with increments and volatility (0.32–0.68), and weakly with direction, shape, spectrum and timing (0.04–0.47) (Table 2).

We believe this empirical finding is a valuable contribution in its own right. Machine learning practitioners routinely choose among forecasting models with near-identical accuracy, and our results show that this choice can change the predicted direction of change or the timing of a peak, while the diagnostics currently used to audit multiplicity give no warning of it.

Thank you for clarifying the relationship to prior work. My main concern remains whether the paper offers sufficient novelty beyond the familiar observation that aggregate loss and horizon-wise disagreement summaries do not fully characterize trajectory-level differences. The response also leaves unclear whether the proposed characterization provides enough additional insight or practical value relative to existing stability and utility-based selection approaches. Therefore, I maintain my original score.

Reviewer_72NTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies temporal predictive multiplicity in time series forecasting, namely that models with similar predictive performance may still produce different forecast trajectories. Under the Rashomon set framework, the authors distinguish forecast disagreement at three levels: point-wise, forecast-step-wise, and trajectory-wise. They also provide experiments to support the proposed claims.

Critical strengths

1, The research question is practically meaningful. 2, The authors distinguish the point-wise, forecast-step-wise, and trajectory levels, and use permutation invariance to show that different statistics retain different information. The unified treatment of forecast trajectories, revised trajectories, and error trajectories expands the analytical scope of the framework.

Critical weaknesses

1, The trajectory definition in this paper remains a fairly conventional mathematical quantity. The authors need to further clarify what substantive insights the proposed formal framework offers beyond this basic observation, relative to existing analyses of forecast multiplicity. 2, The authors devote considerable space to arguing that models with similar predictive performance can indeed produce different forecast trajectories, but they do not explain what these findings are actually useful for.

Questions for authors

1, Keeping the validation-set selection rule unchanged, what is the test MASE distribution of the selected Rashomon set members? Do pronounced trajectory disagreements also appear between model pairs with very similar test performance? 2, If the 5% tolerance is changed to a stricter or looser threshold, or if the candidate set is subsampled to an equal size, do the main conclusions still hold?

2 comments

We thank the reviewer for their thoughtful comments and for the concrete experimental questions.

1. We agree that the trajectory is a conventional object: the vector of H forecasts issued at one origin, together with its errors and its revisions. The definition is not the contribution. The contribution is what the framework establishes about this object, which existing analyses of forecast multiplicity cannot show: how much near-optimal models disagree at individual horizons says little about whether they disagree as trajectories.

Existing analyses work one horizon at a time. The closest, Kale et al. (2026), define a separate Rashomon set at each lead time and select one near-optimal model using a user-supplied utility that is a weighted sum of per-horizon terms. Such an analysis only sees horizon-wise quantities.

Our framework shows what these quantities miss, in three ways. (a) Formally, pointwise and horizon-wise statistics are invariant to permuting origins within each horizon (Definition 3.4, Appendix C), so they cannot distinguish the three panels of Figure 1. Horizon-wise control bounds magnitude, increments and volatility, but leaves direction, shape, spectrum and timing unconstrained (Theorem 4.2 and Appendix D). (b) Empirically, across 19 architectures and 11 datasets, the same split appears: horizon-wise disagreement tracks magnitude almost perfectly (Spearman rank correlation 0.98–0.99) but is only weakly related to direction, shape, spectrum and timing (0.04–0.47, Table 2). (c) Definition 3.3 makes this disagreement measurable under any distance and, as the reviewer notes, covers revision and error trajectories as well as forecast trajectories.

We would be glad to engage with any other analysis the reviewer has in mind.

2. The findings are useful in two ways.

First, they show that choosing among equally accurate models is not a harmless choice. Practitioners routinely train many models, find that several have almost the same validation error, and deploy one of them, often arbitrarily. Our results show that this pick can change the forecast that is acted on: for Queensland electricity demand, 165 of the 332 near-optimal models place the peak in the evening and 167 on the following morning (Figure 3), so any decision that depends on peak timing is determined by which model happened to be picked, not by accuracy.

Second, they show that existing multiplicity checks give false reassurance, because the gap between two models at each horizon says little about whether they agree on direction, shape or timing (Table 2). Our trajectory-level statistics close this gap: practitioners can audit candidates on the property relevant to their task, report it alongside accuracy, and justify the model they deploy. We now state these uses explicitly in the introduction.

Q1. We ran both analyses and report them in a new Appendix B.5.

(a) Test MASE of the selected members. Keeping the validation-set selection rule unchanged, Table 17 reports the validation and test MASE of the members of each Rashomon set, and Figure 4 shows their full distribution, each relative to the best member on the same split. The selected models remain close to the best on the test split: the 90th percentile of the members' gap to the best member is 4.5% on average on validation and at most 7.1% on test for 15 of the 25 targets, and 8% to 15% on 9 others. The one clear exception is Gasoline, whose test period contains the 2020 COVID-19 demand collapse, which is absent from its training and validation periods; we discuss it in the appendix.

(b) Disagreement between models with very similar test performance. We kept, in each Rashomon set, only the members that are also within 5% of the best test MASE, so that every remaining pair has very similar performance on the very forecasts on which its disagreement is measured. These sets are smaller, and a smaller set disagrees less for that reason alone, so we compare them with validation-selected sets subsampled to the same size.

Trajectory disagreement persists. For example, some pair of these models still moves in opposite directions on 76% of the forecast increments at a typical origin, against 80% for validation-selected sets of the same size. Across all temporal aspects, the disagreement is on average between 0.7 and 1.0 times that of the validation-selected sets (Table 18). The whole, non-subsampled sets are reported for completeness (Table 19).

Q2. Thank you for raising this; we ran both analyses and report them in a new Appendix B.4.

We selected the Rashomon sets with a stricter (2.5%) and a looser (10%) tolerance, in addition to the 5% headline figure of the main text.

A looser tolerance admits more models, so we again compare the sets at equal size, with the same subsampling as in Q1(b) and as the reviewer suggests: the 2.5% set is measured as is, and the 5% and 10% sets are subsampled to its size.

The conclusions hold at all three tolerances. At equal size (Table 14), the average direction range is 77%, 78% and 81% at 2.5%, 5% and 10% respectively; the shape range is 74%, 77% and 82%; the spectral range 62%, 65% and 68%; and the timing range 87%, 90% and 94%. The magnitude-type ranges grow by about 10% on average between 2.5% and 5% and, excluding Exchange Rate, by about 15% between 5% and 10%. On Exchange Rate, they more than double between 5% and 10%, as the looser tolerance admits models whose forecasts lie far from the others; a possible explanation is that exchange rates are close to a random walk, so that the loss gives little guidance toward a common forecast structure. We discuss it in the appendix. For completeness, the appendix also reports the results for the whole, non-subsampled sets at 2.5% and 10% (Tables 15 and 16).

Comments on the paper

1 comment

We thank all reviewers for their careful reading and their constructive comments. We have uploaded a revised manuscript in which every change is marked in blue. In summary:

  • Introduction. We now state explicitly how the findings are meant to be used (auditing near-optimal candidates on the trajectory property relevant to the task, and reporting trajectory-level multiplicity alongside accuracy), and we cite Schefzik et al. (2013) and additional related work.

  • Sections 4 and 5. We made explicit the distinction between the two constraints we study (near-optimal aggregate loss and horizon-wise control), and which temporal aspects horizon-wise control bounds (magnitude, increments, volatility) and which it leaves unconstrained (direction, shape, spectral behavior, timing).

  • New Section 7 (Discussion). We discuss why measuring temporal multiplicity is a prerequisite for reducing it, give practical guidance on using our statistics for model selection, and discuss when multiplicity should be reduced or exploited.

  • Conclusions. We list trajectory-aware model selection and regularization as future work.

  • New Appendix B.4 (Rashomon tolerance). We repeat the main experiment at a stricter (2.5%) and a looser (10%) tolerance, comparing the sets at equal size (Tables 14-16).

  • New Appendix B.5 (test performance). We report the test performance of the members of each Rashomon set (Table 17, Figure 4), and repeat the main experiment keeping only the members that are near-optimal on both the validation and the test set (Tables 18-19).

The new experiments confirm the conclusions of the original submission. We respond to each reviewer individually below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.