acceptodds
Under review as a conference paper at ICLR 2027

Forecasting the Forecasters: Training-Free Two-Phase Aggregation of Frozen Time-Series Foundation Models

Abstract

Frozen time-series foundation models differ in accuracy across datasets and over time, making it difficult to decide which forecasts to trust. Combination weights fixed from historical backtests cannot adapt to new outcomes, while deployment-only aggregation has little feedback when few forecasts are scored. Learned routers require additional training. To address these limitations, we introduce TIMEHEDGE, a training-free aggregation procedure that combines unscored historical backtests, called practice, with deployment feedback. Practice sets initial weights from pooled and series-specific errors; deployment feedback updates them. The weights combine forecasts at each quantile level, followed by sorting. We extend classical expert-path analysis to this two-phase protocol and ordered quantile levels. An exact identity quantifies how practice changes a deployment regret bound, where regret measures cumulative excess loss relative to a sequence of model choices. Convexity and rearrangement bound forecast pinball loss by weighted member losses. On GIFT-Eval, TIME and fev-bench, TIMEHEDGE outperformed all 22 tested same-pool baselines: eight individual models and 14 selection or aggregation methods. These gains held for both mean absolute scaled error and continuous ranked probability score on all three benchmarks. Ablation studies indicate that practice drives most of the gains: pooled and series-specific evidence each help, and the same protocol benefits other online learners.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, time-series foundation models, forecast aggregation, online learning, probabilistic forecasting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5He8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TIMEHEDGE, a training-free framework for aggregating forecasts from frozen time-series foundation models. The method combines two sources of feedback: historical backtesting and newly observed outcomes during deployment. It uses pooled practice errors to construct a prior over models, adapts the weights using each target series' own practice history, and applies an expert-path Hedge algorithm for online updates. Separate weights are maintained for different quantile levels, followed by rearrangement to ensure valid quantile ordering. The paper also extends regret analysis to account for the practice phase. Experiments on GIFT-Eval, TIME, and fev-bench show improvements over the tested model selection and aggregation baselines.

Critical strengths
  1. Practical motivation and coherent design. Combining historical backtesting with online feedback addresses a relevant problem in foundation-model forecasting. The use of pooled and series-specific practice information is intuitive, particularly when only a few deployment outcomes are available.

  2. Extensive empirical evaluation. The paper evaluates three benchmarks and a broad range of aggregation baselines. The ablations in Tables 2 and 3 help identify the contributions of different practice sources, while the learner substitutions in Table 9 provide useful evidence beyond the main comparisons.

Critical weaknesses
  1. The methodological novelty is somewhat limited. TIMEHEDGE mainly combines established techniques, including exponential weighting, Fixed-Share expert tracking, and quantile aggregation. The practice correction in Theorem 1 follows directly from the exponential-weights posterior identity.

  2. The contribution of the proposed Hedge mechanism is not clearly established. Tables 2 and 3 indicate that most improvements come from incorporating practice information. In Table 9, when other online learners receive the same practice information, their performance becomes very close to TIMEHEDGE. AdaHedge even achieves a slightly better MASE on GIFT-Eval (0.6494 vs. 0.6506). These results suggest that the main benefit comes from the information protocol.

  3. The model pool selection raises concerns about evaluation independence. Appendix E states that the inclusive pool was selected as the primary setting after both pool rules had been evaluated on scored outcomes. Although both results are disclosed, selecting the primary configuration based on test performance may introduce optimistic bias.

  4. The practical efficiency gains are not sufficiently convincing. Although TIMEHEDGE requires no parameter training, it performs additional inference over multiple models and practice windows. Table 6 shows that for a single scored window, TIMEHEDGE takes 13.31 seconds compared with 1.88 seconds for uniform aggregation. Meanwhile, the accuracy improvements over strong practice-informed baselines are relatively modest on some benchmarks.

Questions for authors
  1. Algorithmic contribution: What is the specific advantage of the proposed expert-path Hedge mechanism over standard Hedge or AdaHedge when all methods receive identical practice information? Can the authors further isolate the benefit of switching paths?

  2. Theoretical novelty: Beyond the posterior identity used in Theorem 1, what new theoretical insight does the two-phase analysis provide? How should the regret guarantee on normalized and capped losses be interpreted relative to the reported CRPS?

  3. Pool selection: Since the inclusive pool was chosen after evaluating both pool rules, how does this decision affect the statistical interpretation of the main results? Would the conclusions remain unchanged under a pool selection procedure independent of scored outcomes?

  4. Computational efficiency: Can the authors provide a more direct comparison of accuracy versus inference cost, particularly against strong practice-informed baselines, across different numbers of deployment windows?

4 comments

Thank you for the constructive review. We conducted four additional experiments and three new analyses, each marked below.

W2 & Q1: Expert-path learner under identical practice

That the main benefit comes from the information protocol is the main claim of the paper (lines 1449-1450). Under identical practice, the specific advantage of the expert-path learner is a better probabilistic forecast. In Table 9 it has the lowest CRPS on all three benchmarks, and it reduces pooled CRPS by 0.57% and 0.59% relative to practice-informed Hedge and AdaHedge. A new analysis confirms the significance (Holm-adjusted p < 0.001). The MASE difference on GIFT-Eval that you cite is not significant (p ≥ 0.3, lines 1445-1446), and in pooled MASE both learners are behind, by 0.35% and 0.39% (lines 1447-1449).

The advantage comes from the path prior, which hedges over two unknowns, whether the pooled prior suits the target series (pooled or uniform start) and whether the best model will change (static or switching paths). Three additional experiments isolate its contribution.

  1. Path prior. With practice, losses and learning rate held fixed (215 configurations of Table 11), TIMEHEDGE reduces CRPS by 0.51% (95% CI [0.14, 0.87]) relative to Hedge with the pooled prior alone (static paths) and by 0.30% [0.09, 0.53] relative to switching paths alone.
  2. Simulation (512 random seeds). Switching paths lowered CRPS by 26% after a sustained change of the best model, against the learner without them.
  3. Reserved data. On six public sources absent from our development records, TIMEHEDGE is ahead of practice-informed Hedge in both metrics on all six tasks, and the switch step lowered CRPS by 0.74% when added to Hedge started from TIMEHEDGE's first-forecast weights, with a gain on every task.

W1 & Q2: Novelty, theory and the relation to CRPS

The new element is the two-phase protocol, which starts the online learner from a prior pooled across series and runs it through unscored backtests before deployment. Without it, an online learner has at most three scored windows to learn from on half of GIFT-Eval (lines 149-150). The identity makes the effect of practice explicit and signed (it lowers the bound for paths that practice favored and raises it for the others), consistent with the permuted-label control of Table 3. Beyond it, the analysis provides three results.

  1. Proposition 4 shows when the bound is approached. The future-path code length is attained up to a residual.
  2. By Corollary 1, uniform-start paths make the initial cost of every model independent of the pooled prior, which is why we keep both pooled and uniform starts.
  3. Proposition 6 lower-bounds the value of the scored updates over a learner frozen after practice.

The guarantee is on normalized, capped pinball losses, each round in units of its median model loss. Exact identities link it to the reported CRPS (Appendix A.9). Raw regret adds a clipping residual (Table 8) and weights each round by its scale. The cap is active on 0.23% to 0.79% of the losses (lines 1697-1699).

W3 & Q3: Pool selection

The main results do not hinge on this decision. The conclusions remain unchanged under the pool rule declared first (one checkpoint per lineage, five models), which was fixed before its evaluation (Appendix E).

  • Additional experiment. Under this rule we recomputed the same-pool comparison of Tables 1 and 17, with rules and tests fixed beforehand. TIMEHEDGE has lower pooled MASE and CRPS than all 13 non-router rules of Table 17, and none is ahead on any benchmark. After Holm correction, all 26 comparisons are significant with dataset clusters and 24 with source clusters. The two routers need retraining and are not included. We will add the table to Appendix F.6.
  • Table 14 removes each lineage in turn. TIMEHEDGE stays ahead of the uniform average and the practice pick in every pool, benchmark and metric (pooled, joint Holm p < 0.001).

W4 & Q4: Accuracy versus inference cost

Against strong practice-informed baselines, TIMEHEDGE adds no inference and is more accurate. It uses the same forecasts as Hedge + own practice (13.3, 21.0 and 49.4 s against 13.2, 21.0 and 49.2 s at W=1, 5 and 20 in Table 6).

Against the uniform average, the additional cost is fixed. The six practice backtests add about 11.5 s per series for any W (the uniform average takes 1.9, 9.5 and 37.9 s).

In a new analysis of the 215 configurations of Table 11, TIMEHEDGE's CRPS is lower in every bin of W (2-5, 6-19 and ≥ 20). The reduction is 0.3% to 0.9% relative to practice-informed Hedge and 3.4% to 8.3% relative to the uniform average, the latter with 95% intervals above zero (13 of the 215 have W=1, too few for a reliable estimate).

In a further new analysis, one practice round keeps 89.5% of the pooled MASE gain over the uniform average (Tables 1 and 16), while the fixed cost falls from 48 to 8 forecasts. We will add these analyses to Appendix C.6.

I appreciate the author's efforts. However, the following issues remain insufficiently addressed:

  • TIMEHEDGE primarily integrates existing techniques—namely Exponential Weights, Fixed-Share, and Forecast Combination—without providing sufficient evidence of distinct methodological innovation compared to these established approaches.
  • While the expert-path mechanism yields a statistically significant improvement in CRPS, the relative gain is only about 0.5%.
  • The actual pooled prior incorporates observations from the target sequence itself, which violates the prior independence condition required for Proposition 3. Consequently, the PAC-Bayes risk interpretation derived for the practice phase cannot be directly applied to the actual implementation.

Thank you for your response. For the third issue we ran an additional experiment.

1. Innovation compared to the established approaches

Our contribution lies both in the two-phase protocol, including the design of its practice phase, and in the expert-path learner that runs in it (see 2). We built the update rule of the learner from exponential weights and Fixed-Share, as stated in the paper (lines 88-90). The protocol does not depend on the learner. We applied it to five established online learners and compared them with their scored-only forms (Table 10, reductions copied below).

LearnerMASE reductionCRPS reduction
Hedge2.52%2.35%
AdaHedge0.96%0.78%
BOA4.26%4.24%
FTL1.28%1.03%
BOA + fixed smoothing4.28%4.25%

All five improved in both metrics, which shows that the protocol is not limited to our learner and can be applied to existing online learners.

The design of the practice phase is also important. The forecast combinations of Table 1 (stacking, BMA and the Best-3 mixture) also fit their weights on the practice rounds, and they are 1.2% to 2.2% behind TIMEHEDGE in pooled MASE, with no 95% interval containing zero (Table 17). Within the protocol, the pooled prior and the series' own practice are complementary, since each improves the results alone and using both is the best on all three benchmarks. With permuted labels, practice is even worse than no practice (Table 3).

2. Relative gain of the expert-path mechanism

The expert-path mechanism further improves the results on top of the protocol. With the same practice, TIMEHEDGE has lower pooled CRPS than each of the five established learners of Table 9, by 0.57% to 1.38% (Holm-adjusted p < 0.001 for each in the analysis of our first response), and this mechanism adds no inference (Table 6). The gain of about 0.5% that you cite is against Hedge with our protocol, which is the strongest of these five learners. Relative to Hedge in its established form (Table 1), the protocol lowers pooled CRPS by 2.35% (Table 10) and the expert-path mechanism lowers it by an additional 0.57% (Table 9), which is 2.9% in total.

3. Prior independence condition required for Proposition 3

You are right that the pooled prior includes the target series' own practice rows. We state this in the remark after Proposition 3 (lines 644-647), where we also note that the deployment guarantee does not depend on the proposition. The guarantee for the actual implementation (Proposition 1 and Theorem 1) is deterministic and conditions on the prior that the implementation uses (lines 155-156), so it does not require independence between the prior and the target series.

We conducted an additional experiment in which TIMEHEDGE uses the leave-one-series-out prior of the remark, for which the prior independence condition holds. Each series gets the prior pooled over the other series of its configuration (the uniform prior if it is alone, which applies to 508 of the 293,693 series), and all other settings are kept the same.

MASE / CRPSGIFT-EvalTIMEfev-bench
Best value among the other rows of Table 10.6543 / 0.74040.6675 / 0.67950.6188 / 0.7753
TIMEHEDGE (Table 1)0.6506 / 0.73290.6653 / 0.67700.6092 / 0.7643
TIMEHEDGE with leave-one-series-out prior (additional experiment)0.6506 / 0.73270.6657 / 0.67720.6086 / 0.7638

The scores are within 0.1% of those in Table 1 on all three benchmarks, and TIMEHEDGE still outperforms all other rows of that table. Across the 293 configurations, MASE changes by -0.01% (95% CI [-0.08, 0.05]) and CRPS by -0.02% ([-0.08, 0.02]). We will add this experiment to Appendix F.5 and refer to it in the remark after Proposition 3.

Dear Reviewer 5He8, the discussion closes on November 18 (AoE). Our reply above answers the three remaining issues, and for the third it adds an experiment with the leave-one-series-out prior, for which the prior independence condition holds. We would be glad to hear whether it resolves them, and we can still answer further questions before the deadline.

Reviewer_gCwEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to aggregate forecasts from frozen time-series foundation models without training the constituent models or a separate routing network. It proposes TIMEHEDGE, which combines two sources of information: historical in-context backtests, termed “practice,” and feedback revealed during deployment. Practice losses pooled across series define an initial prior over models. The target series’ own practice losses then adapt this prior before the first scored forecast. During deployment, an exponential-weights learner continues updating a mixture of static and switching expert paths. Aggregation weights are specific to quantile levels, and the resulting quantiles are sorted to restore monotonicity. An equal-weight ensemble is included as an additional expert. The theoretical analysis characterizes how practice changes a deployment regret bound through an exact posterior-conditioning identity. Convexity and rearrangement connect weighted member losses to the combined quantile forecasts, with additional accounting for loss normalization and clipping. Experiments on GIFT-Eval, TIME, and fev-bench compare TIMEHEDGE with individual models, forecast combinations, online learners, and learned routers. The main empirical finding is that historical practice supplies most of the improvement, and its benefit transfers to several established online learners.

Critical strengths
  1. The paper is generally clear and well organized. The distinction between unscored practice and scored deployment is introduced early, and the four-stage description makes the procedure understandable. The main text also distinguishes complete-protocol comparisons from learner substitutions using matched practice information.
  2. The paper addresses a practical limitation of deployment-only aggregation: many forecasting settings provide very few scored updates. Historical backtests offer information before the first deployment forecast. Combining pooled evidence with target-specific adaptation is a sensible way to use this information while retaining the ability to respond to subsequent outcomes.
  3. The two-by-two practice ablation, learner substitutions, and permutation controls are particularly valuable. They show that pooled and target-specific practice each contribute, and that misleading practice can be harmful. Improvements obtained by giving other learners the same practice information establish that the proposed protocol has value beyond one update rule.
Critical weaknesses
  1. The main learner combines established exponential weights, Fixed-Share tracking, quantile-wise aggregation, and rearrangement. Theorem 1 provides useful accounting for practice, but its central identity follows directly from Gibbs posterior conditioning and the standard deployment regret bound. Consequently, the strongest contribution appears to be the integrated protocol and its empirical validation, rather than a substantially new online-learning principle. Table 9 reinforces this interpretation. Once other learners receive the same pooled prior and target-specific practice, the performance gaps become much smaller. On GIFT-Eval, practice-informed Hedge and AdaHedge even obtain numerically lower MASE than TIMEHEDGE: 0.6497 and 0.6494 versus 0.6506, although TIMEHEDGE retains the best CRPS and these MASE differences are not significant. The authors acknowledge much of this, which is a strength. Nevertheless, the paper should explain more sharply what the particular expert-path construction contributes beyond historical initialization of established online quantile aggregators. The direct author-package comparison is helpful, but those package variants do not receive the same pooled prior.
  2. Appendix E transparently describes sequential development decisions. In particular, the inclusive pool became primary after both pool rules had been scored, and learner-development shots were compared using scored-test outcomes. Registering each run before execution improves traceability, but does not make the final design independent of earlier benchmark feedback. This is distinct from using unrevealed targets to issue an individual forecast: the causal forecasting protocol can remain valid while design selection introduces optimism into the final benchmark results. Reporting both pool variants and the development trajectory mitigates this concern, but does not fully resolve it. An evaluation of the completely frozen final pipeline on an untouched benchmark or reserved set of sources would provide stronger evidence that the observed advantages generalize beyond the development process.
Questions for authors
  1. What is the central methodological advantage of the four-component expert-path learner over practice-informed Hedge or AdaHedge? Can this advantage be linked to identifiable regimes, such as sustained changes in the best member, rather than only pooled average performance?
  2. Can the direct CRPS-Learning author-package comparison incorporate the same pooled practice prior? If this is technically difficult, can you quantify how much of the remaining difference is attributable to that initialization advantage?
  3. Which evaluations, if any, were untouched by the learner and pool-selection process described in Appendix E? Could the final pipeline be evaluated on reserved data sources after all design choices are frozen?
2 comments

Thank you for the constructive review. We conducted five additional experiments and one new analysis, each marked below.

W1 & Q1: Advantage of the four-component learner over practice-informed Hedge or AdaHedge

The central methodological advantage is that the path construction hedges over two things that are unknown before deployment, whether the pooled prior suits the target series (pooled or uniform start) and whether the best model will change (static or switching paths). Practice-informed Hedge commits to one answer to each. We agree that the contribution is the integrated protocol with its analysis. See our response to Reviewer 5He8 (W1 & Q2) for what the analysis adds beyond the identity.

The result is a better probabilistic forecast under identical practice. CRPS evaluates all nine quantiles, including the tails, whereas MASE evaluates the median alone (lines 137-138). TIMEHEDGE has the lowest CRPS on all three benchmarks in Table 9, and it reduces pooled CRPS by 0.57% and 0.59% relative to practice-informed Hedge and AdaHedge. A new analysis confirms the significance (Holm-adjusted p < 0.001). The reduction recurs in the additional experiments below, under the five-model pool (0.52% and 0.57%, Holm-adjusted p < 0.001) and on six reserved sources (1.2% to 1.5% against practice-informed Hedge, on every task). The MASE differences on GIFT-Eval that you note are not significant (p ≥ 0.3, lines 1445-1446), and in pooled MASE both learners are behind, by 0.35% and 0.39% (lines 1447-1449).

The advantage is linked to two identifiable regimes.

  1. A pooled prior that points to the wrong model, as for the 29% of GIFT-Eval series whose first-window best model sits at the floor of the prior (line 1665). In a simulation of this regime (additional experiment, 512 random seeds), uniform-start paths lowered CRPS by 34% over the first eight windows.
  2. A sustained change of the best model. In the same simulation, switching paths lowered CRPS by 26% after such a change. On the six reserved sources, the switch step lowered CRPS by 0.74% when added to Hedge started from TIMEHEDGE's first-forecast weights, with a gain on every task.

Pooled over 215 benchmark configurations (Table 11), the two mechanisms together reduce CRPS by 0.51% (95% CI [0.14, 0.87]) relative to Hedge with the pooled prior alone, with practice, losses and learning rate held fixed (additional experiment).

Q2: CRPS-Learning author package with the same pooled prior

The comparison can incorporate the same pooled practice prior. In an additional experiment, we gave the author package (profoc 1.3.4, BOA with the 18-cell grid of Appendix F.4 and six practice rounds) this prior on the 215 configurations of Table 11. TIMEHEDGE still reduces CRPS by 0.64% (95% CI [0.31, 1.00], Holm-adjusted p ≤ 0.002) and MASE by 0.43% [0.03, 0.91], so the advantage over the author package does not come from the initialization alone. Under the sharp prior a rounding error in the package gives non-finite learning rates, which we removed with an algebraically equivalent stable form in all runs with this prior (to be documented in Appendix F.4).

W2 & Q3: Evaluations untouched by development

The evaluation untouched by the learner and pool selection is a new one on reserved data sources, which we ran after all design choices were frozen. We also recomputed the benchmark comparisons under the pool rule declared first. Both are additional experiments.

  1. Reserved sources. The frozen pipeline was evaluated under both pool rules on six public sources absent from our development records. These are CTA and MTA ridership, EPA county air quality, London cycle hires, TSA passenger counts and NSIDC sea-ice extent (14 daily series, 32 weekly windows each), fixed before any of their forecasts was computed.
Pool ruleCRPS vs practice-informed HedgeCRPS vs author BOA, same prior
Five models (one per lineage)1.18% lower1.53% lower
Eight models (primary)1.51% lower1.82% lower

TIMEHEDGE is ahead of both matched learners in MASE and CRPS on all six tasks under both pool rules. The advantage therefore holds on sources outside the development process.

  1. Pool rule declared first. The five-model rule was declared before its evaluation (Appendix E). Under it we recomputed the same-pool comparison of Tables 1 and 17 and the learner substitutions of Table 9, with rules and tests fixed beforehand. TIMEHEDGE has lower pooled MASE and CRPS than all 13 non-router rules of Table 17, and none is ahead on any benchmark. After Holm correction, all 26 comparisons are significant with dataset clusters and 24 with source clusters. The two routers need retraining and are not included. A further pool check (lineage removals, Table 14) is in our response to Reviewer 5He8 (W3 & Q3).

We will add the five experiments with full tables to Appendix F (F.2, F.4 and F.6 for the path prior, the author package and the pool rule).

Dear Reviewer gCwE, the discussion period closes on November 18 (AoE), so we wanted to check whether our response of November 11 answers your questions. It has five additional experiments, including the comparison with practice-informed Hedge and AdaHedge and the evaluation on sources untouched by development. Please let us know if anything remains unclear.

Reviewer_eJ5QOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces TIMEHEDGE, a framework for combining predictions from multiple frozen time-series foundation models using historical backtesting and online feedback. Specifically, the proposed framework estimates model weights from historical forecasting errors, and subsequently updates these weights as new observations roll in. The tools for aggregation/combination of forecast are well established, including online expert tracking algorithms, quantile aggregation monotone rearrangement to produce the probabilistic forecasts. On the theory front, the paper provides regret bounds for the effect of historical information. Performance evaluation of the framework is based on experiments across three widely used benchmarks. The results show consistent improvements over both individual and several established forecasting combination approaches.

Critical strengths
  • The paper addresses a number important and always timely issue in forecasting. Given the increasing availability of foundation models, efficient methods for leveraging their complementary strengths are useful to practitioners.

  • Extensive performance evaluation across three benchmarks, eight individual foundation models and 14 methods of selection and aggregation. Further, TIMEHEDGE achieves the best results across all three benchmarks and both MASE and CPRS evaluation metrics.

  • A number of carefully designed ablation studies provide insights into the contributions of the various components of the TIMEHEDGE framework.

Critical weaknesses
  • There is a very large body of literature on forecast combination, prediction with expert advice and adaptive ensemble weighting. The proposed framework leverages several of them, which is interesting and produces good performance, but its methodological advance is rather incremental. The same holds true for the regret bounds established.

  • On the performance front, the gains are marginal and in all likelihood not statistically significant, compared to competing methods. Since the proposed framework is computational significantly more demanding than simpler aggregation methods, it is doubtful that the very small performance improvements justify the additional computing cost.

  • The ablation studies indicate that most of the performance improvements can be attributed to incorporating historical practice information, rather than the specific online learning algorithm used.

Questions for authors
  • Based on the ablation studies, it would be useful to quantify how much of the performance improvements is attributable to the proposed algorithm rather than the availability of additional historical information.

  • Computational considerations are not provided. The author should provide a more explicit accuracy vs computational cost comparison with some of the simpler approaches considered in the performance assessment.

  • How does TIMEHEDGE perform under distribution shifts?

2 comments

Thank you for the constructive review. We conducted three additional experiments and two new analyses, each marked below.

W2: Statistical significance of the gains

The gains are statistically significant, and the tests were reported in the submission. Pooled over the 293 configurations, every competing rule in Table 1 is behind TIMEHEDGE at p < 0.001 (lines 320-323). This holds under Holm and Bonferroni correction with configurations that share a data source merged into 117 clusters (adjusted p ≤ 0.044, lines 1813-1822). No pooled 95% cluster-bootstrap interval in Table 17 contains zero.

The gains are also not marginal. The margin over the closest combiners is 1.2% to 1.6% in MASE. For scale, the published zero-shot, pretrained and fine-tuned single-model scores on the official GIFT-Eval harness span 1.3% (0.6668 to 0.6582, lines 374-377), and TIMEHEDGE reaches 0.6502 without training.

W2 & Q2: Accuracy versus computational cost

Computational considerations are provided in Table 6 (p. 23) and Appendix C.6, where all practice forecasts are charged. To set cost against accuracy in one place, we will add the following table to Appendix C.6. It includes a new analysis of TIMEHEDGE with one practice round.

RuleModel forecasts per seriesMASE relative to TIMEHEDGE
Uniform averagePW+4.4% [2.5, 6.7]
AdaHedge, deployment onlyPW+1.4% [0.6, 2.2]
Practice pick6P + W+1.5% [0.9, 2.2]
TIMEHEDGE, one practice roundP(1 + W)+0.5% [0.1, 0.8]
TIMEHEDGE, six practice roundsP(6 + W)reference

(P = 8 models, W scored windows, pooled 95% intervals from Table 17. The one-round row is the new analysis.)

In Table 6, TIMEHEDGE takes 13.3 s against 1.9 s for the uniform average at W=1 and 49.4 s against 37.9 s at W=20. The additional cost is fixed, about 11.5 s per series for the six practice backtests, and each later forecast costs the same as for the uniform average. In a second new analysis, on the 215 configurations of Table 11, TIMEHEDGE reduces CRPS relative to the uniform average by 3.4% to 8.3% for W ≥ 2 (13 of the 215 have W=1, too few for a reliable estimate).

W1, W3 & Q1: Novelty, and the algorithm versus historical information

The contribution is the two-phase formulation, not its components. The online learners of Table 1 start from uniform weights and learn from scored outcomes alone, and half of the GIFT-Eval configurations offer at most three scored windows (lines 149-150). They are 1.4% to 5.6% behind TIMEHEDGE in pooled MASE (Table 17). TIMEHEDGE creates its feedback before deployment by running backtests inside the observed history, and we propose how to pool and process them. The improvement that you attribute to practice information is therefore the improvement of the proposed method.

Starting from standard Hedge, the submitted tables allow two orders of substitution (293 configurations, CRPS). Practice first lowers CRPS by 2.35% and the learner then lowers it by a further 0.57% (Tables 10 and 9). The learner first lowers it by 1.13% and practice then lowers it by a further 1.78% (Tables 1 and 16). The learner accounts for 19% to 39% of the CRPS improvement.

We also conducted an additional experiment that isolates the learner. With practice, losses and learning rate held fixed, TIMEHEDGE reduces CRPS by 0.51% (95% CI [0.14, 0.87]) relative to Hedge started from the same pooled prior (215 configurations of Table 11), so the learner contributes beyond the historical information.

The protocol also does more than warm up a learner on its own series. Against the CRPS-Learning authors' package with six practice rounds, TIMEHEDGE's MASE is 0.5% to 3.9% lower (Table 11), and the pooled prior adds to own-series practice on all three benchmarks (Table 3). The analysis is specific to the protocol too, since Theorem 1 gives the sign and size of the effect of practice on the deployment bound.

Q3: Distribution shifts

TIMEHEDGE is built to follow distribution shifts. Its regret guarantee, stated for normalized and capped losses, holds against arbitrary sequences of models (expert paths) and assumes no stationarity (Section 5).

We conducted two additional experiments, which we will add to Appendix F.

  • In a simulation (512 random seeds), switching paths lowered CRPS by 26% after a change of the best model, relative to the same learner without them. Under a pooled prior that favored the wrong model, uniform-start paths lowered it by 34% over the first eight windows.
  • On six public sources absent from our development records, including transit ridership, air quality and sea-ice extent, the switch step lowered CRPS by 0.74% over 32 weekly windows when added to Hedge started from TIMEHEDGE's first-forecast weights, with a gain on all six.

These results indicate that TIMEHEDGE follows a change of the best model and limits the effect of a misleading prior.

Dear Reviewer eJ5Q, we were wondering whether you had a chance to read our response of November 11. It added three experiments and two analyses on the points you raised (significance of the gains, accuracy against cost, novelty, distribution shifts). The discussion closes on November 18 (AoE), and we are happy to answer any remaining questions before then.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of reviews and responses

Dear Area Chair and Reviewers,

We summarize the reviews and our responses below. The additional results are in a new Appendix H of the revised version (pages 37-38, blue text). We did not change the main text, so the line and table numbers cited in the reviews still apply.

The reviewers found the problem practical and timely (eJ5Q, gCwE, 5He8), the evaluation extensive (eJ5Q, 5He8) and the ablations valuable (eJ5Q, gCwE). Their main concerns and our answers are as follows.

  1. Novelty (all reviewers). The new element is the two-phase protocol. TIMEHEDGE creates its feedback before deployment by running unscored backtests (practice) inside the observed history, and we propose how to pool and process them. Without practice, an online learner has at most three scored windows on half of GIFT-Eval (lines 149-150). The improvement that the reviewers attribute to practice is therefore the improvement of the proposed method. All five established online learners also improve with the protocol (Table 10), which in the words of Reviewer gCwE shows that "the proposed protocol has value beyond one update rule".

  2. Learner versus practice (all reviewers). With the same practice for every learner, TIMEHEDGE has the lowest CRPS on all three benchmarks (Table 9). The reduction is significant against each of the five established learners (Holm-adjusted p < 0.001, H.2) and against the CRPS-Learning author package with the same prior (p ≤ 0.002, H.5). In a simulation, switching paths lowered CRPS by 26% after a change of the best model, and uniform-start paths by 34% under a misleading prior (H.3).

  3. Size of the gains and cost (eJ5Q, 5He8). Every competing rule in Table 1 is behind TIMEHEDGE at p < 0.001 (lines 320-323). The margin over the closest combiners is 1.2% to 1.6% in MASE. For scale, the published single-model scores on GIFT-Eval, from zero-shot to fine-tuned, span 1.3% (lines 374-377). Against practice-informed baselines TIMEHEDGE adds no inference (13.3 s against 13.2 s for Hedge with practice, Table 6). Against the uniform average the additional cost is fixed, about 11.5 s per series (H.1).

  4. Independence of the evaluation (gCwE, 5He8). On six data sources evaluated after all design choices were frozen, TIMEHEDGE is ahead of both matched learners on all six (H.4). Under the pool rule declared first it has lower pooled MASE and CRPS than all 13 non-router rules (H.6).

  5. Prior and target series (5He8, follow-up). With the leave-one-series-out prior the scores are within 0.1% of Table 1 (H.7).

We thank the Area Chair and the reviewers for their time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.