- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TIMEHEDGE, a training-free framework for aggregating forecasts from frozen time-series foundation models. The method combines two sources of feedback: historical backtesting and newly observed outcomes during deployment. It uses pooled practice errors to construct a prior over models, adapts the weights using each target series' own practice history, and applies an expert-path Hedge algorithm for online updates. Separate weights are maintained for different quantile levels, followed by rearrangement to ensure valid quantile ordering. The paper also extends regret analysis to account for the practice phase. Experiments on GIFT-Eval, TIME, and fev-bench show improvements over the tested model selection and aggregation baselines.
Critical strengths
-
Practical motivation and coherent design. Combining historical backtesting with online feedback addresses a relevant problem in foundation-model forecasting. The use of pooled and series-specific practice information is intuitive, particularly when only a few deployment outcomes are available.
-
Extensive empirical evaluation. The paper evaluates three benchmarks and a broad range of aggregation baselines. The ablations in Tables 2 and 3 help identify the contributions of different practice sources, while the learner substitutions in Table 9 provide useful evidence beyond the main comparisons.
Critical weaknesses
-
The methodological novelty is somewhat limited. TIMEHEDGE mainly combines established techniques, including exponential weighting, Fixed-Share expert tracking, and quantile aggregation. The practice correction in Theorem 1 follows directly from the exponential-weights posterior identity.
-
The contribution of the proposed Hedge mechanism is not clearly established. Tables 2 and 3 indicate that most improvements come from incorporating practice information. In Table 9, when other online learners receive the same practice information, their performance becomes very close to TIMEHEDGE. AdaHedge even achieves a slightly better MASE on GIFT-Eval (0.6494 vs. 0.6506). These results suggest that the main benefit comes from the information protocol.
-
The model pool selection raises concerns about evaluation independence. Appendix E states that the inclusive pool was selected as the primary setting after both pool rules had been evaluated on scored outcomes. Although both results are disclosed, selecting the primary configuration based on test performance may introduce optimistic bias.
-
The practical efficiency gains are not sufficiently convincing. Although TIMEHEDGE requires no parameter training, it performs additional inference over multiple models and practice windows. Table 6 shows that for a single scored window, TIMEHEDGE takes 13.31 seconds compared with 1.88 seconds for uniform aggregation. Meanwhile, the accuracy improvements over strong practice-informed baselines are relatively modest on some benchmarks.
Questions for authors
-
Algorithmic contribution: What is the specific advantage of the proposed expert-path Hedge mechanism over standard Hedge or AdaHedge when all methods receive identical practice information? Can the authors further isolate the benefit of switching paths?
-
Theoretical novelty: Beyond the posterior identity used in Theorem 1, what new theoretical insight does the two-phase analysis provide? How should the regret guarantee on normalized and capped losses be interpreted relative to the reported CRPS?
-
Pool selection: Since the inclusive pool was chosen after evaluating both pool rules, how does this decision affect the statistical interpretation of the main results? Would the conclusions remain unchanged under a pool selection procedure independent of scored outcomes?
-
Computational efficiency: Can the authors provide a more direct comparison of accuracy versus inference cost, particularly against strong practice-informed baselines, across different numbers of deployment windows?
4 comments
I appreciate the author's efforts. However, the following issues remain insufficiently addressed:
- TIMEHEDGE primarily integrates existing techniques—namely Exponential Weights, Fixed-Share, and Forecast Combination—without providing sufficient evidence of distinct methodological innovation compared to these established approaches.
- While the expert-path mechanism yields a statistically significant improvement in CRPS, the relative gain is only about 0.5%.
- The actual pooled prior incorporates observations from the target sequence itself, which violates the prior independence condition required for Proposition 3. Consequently, the PAC-Bayes risk interpretation derived for the practice phase cannot be directly applied to the actual implementation.
Thank you for your response. For the third issue we ran an additional experiment.
1. Innovation compared to the established approaches
Our contribution lies both in the two-phase protocol, including the design of its practice phase, and in the expert-path learner that runs in it (see 2). We built the update rule of the learner from exponential weights and Fixed-Share, as stated in the paper (lines 88-90). The protocol does not depend on the learner. We applied it to five established online learners and compared them with their scored-only forms (Table 10, reductions copied below).
| Learner | MASE reduction | CRPS reduction |
|---|---|---|
| Hedge | 2.52% | 2.35% |
| AdaHedge | 0.96% | 0.78% |
| BOA | 4.26% | 4.24% |
| FTL | 1.28% | 1.03% |
| BOA + fixed smoothing | 4.28% | 4.25% |
All five improved in both metrics, which shows that the protocol is not limited to our learner and can be applied to existing online learners.
The design of the practice phase is also important. The forecast combinations of Table 1 (stacking, BMA and the Best-3 mixture) also fit their weights on the practice rounds, and they are 1.2% to 2.2% behind TIMEHEDGE in pooled MASE, with no 95% interval containing zero (Table 17). Within the protocol, the pooled prior and the series' own practice are complementary, since each improves the results alone and using both is the best on all three benchmarks. With permuted labels, practice is even worse than no practice (Table 3).
2. Relative gain of the expert-path mechanism
The expert-path mechanism further improves the results on top of the protocol. With the same practice, TIMEHEDGE has lower pooled CRPS than each of the five established learners of Table 9, by 0.57% to 1.38% (Holm-adjusted p < 0.001 for each in the analysis of our first response), and this mechanism adds no inference (Table 6). The gain of about 0.5% that you cite is against Hedge with our protocol, which is the strongest of these five learners. Relative to Hedge in its established form (Table 1), the protocol lowers pooled CRPS by 2.35% (Table 10) and the expert-path mechanism lowers it by an additional 0.57% (Table 9), which is 2.9% in total.
3. Prior independence condition required for Proposition 3
You are right that the pooled prior includes the target series' own practice rows. We state this in the remark after Proposition 3 (lines 644-647), where we also note that the deployment guarantee does not depend on the proposition. The guarantee for the actual implementation (Proposition 1 and Theorem 1) is deterministic and conditions on the prior that the implementation uses (lines 155-156), so it does not require independence between the prior and the target series.
We conducted an additional experiment in which TIMEHEDGE uses the leave-one-series-out prior of the remark, for which the prior independence condition holds. Each series gets the prior pooled over the other series of its configuration (the uniform prior if it is alone, which applies to 508 of the 293,693 series), and all other settings are kept the same.
| MASE / CRPS | GIFT-Eval | TIME | fev-bench |
|---|---|---|---|
| Best value among the other rows of Table 1 | 0.6543 / 0.7404 | 0.6675 / 0.6795 | 0.6188 / 0.7753 |
| TIMEHEDGE (Table 1) | 0.6506 / 0.7329 | 0.6653 / 0.6770 | 0.6092 / 0.7643 |
| TIMEHEDGE with leave-one-series-out prior (additional experiment) | 0.6506 / 0.7327 | 0.6657 / 0.6772 | 0.6086 / 0.7638 |
The scores are within 0.1% of those in Table 1 on all three benchmarks, and TIMEHEDGE still outperforms all other rows of that table. Across the 293 configurations, MASE changes by -0.01% (95% CI [-0.08, 0.05]) and CRPS by -0.02% ([-0.08, 0.02]). We will add this experiment to Appendix F.5 and refer to it in the remark after Proposition 3.
Dear Reviewer 5He8, the discussion closes on November 18 (AoE). Our reply above answers the three remaining issues, and for the third it adds an experiment with the leave-one-series-out prior, for which the prior independence condition holds. We would be glad to hear whether it resolves them, and we can still answer further questions before the deadline.
Thank you for the constructive review. We conducted four additional experiments and three new analyses, each marked below.
W2 & Q1: Expert-path learner under identical practice
That the main benefit comes from the information protocol is the main claim of the paper (lines 1449-1450). Under identical practice, the specific advantage of the expert-path learner is a better probabilistic forecast. In Table 9 it has the lowest CRPS on all three benchmarks, and it reduces pooled CRPS by 0.57% and 0.59% relative to practice-informed Hedge and AdaHedge. A new analysis confirms the significance (Holm-adjusted p < 0.001). The MASE difference on GIFT-Eval that you cite is not significant (p ≥ 0.3, lines 1445-1446), and in pooled MASE both learners are behind, by 0.35% and 0.39% (lines 1447-1449).
The advantage comes from the path prior, which hedges over two unknowns, whether the pooled prior suits the target series (pooled or uniform start) and whether the best model will change (static or switching paths). Three additional experiments isolate its contribution.
W1 & Q2: Novelty, theory and the relation to CRPS
The new element is the two-phase protocol, which starts the online learner from a prior pooled across series and runs it through unscored backtests before deployment. Without it, an online learner has at most three scored windows to learn from on half of GIFT-Eval (lines 149-150). The identity makes the effect of practice explicit and signed (it lowers the bound for paths that practice favored and raises it for the others), consistent with the permuted-label control of Table 3. Beyond it, the analysis provides three results.
The guarantee is on normalized, capped pinball losses, each round in units of its median model loss. Exact identities link it to the reported CRPS (Appendix A.9). Raw regret adds a clipping residual (Table 8) and weights each round by its scale. The cap is active on 0.23% to 0.79% of the losses (lines 1697-1699).
W3 & Q3: Pool selection
The main results do not hinge on this decision. The conclusions remain unchanged under the pool rule declared first (one checkpoint per lineage, five models), which was fixed before its evaluation (Appendix E).
W4 & Q4: Accuracy versus inference cost
Against strong practice-informed baselines, TIMEHEDGE adds no inference and is more accurate. It uses the same forecasts as Hedge + own practice (13.3, 21.0 and 49.4 s against 13.2, 21.0 and 49.2 s at W=1, 5 and 20 in Table 6).
Against the uniform average, the additional cost is fixed. The six practice backtests add about 11.5 s per series for any W (the uniform average takes 1.9, 9.5 and 37.9 s).
In a new analysis of the 215 configurations of Table 11, TIMEHEDGE's CRPS is lower in every bin of W (2-5, 6-19 and ≥ 20). The reduction is 0.3% to 0.9% relative to practice-informed Hedge and 3.4% to 8.3% relative to the uniform average, the latter with 95% intervals above zero (13 of the 215 have W=1, too few for a reliable estimate).
In a further new analysis, one practice round keeps 89.5% of the pooled MASE gain over the uniform average (Tables 1 and 16), while the fixed cost falls from 48 to 8 forecasts. We will add these analyses to Appendix C.6.