- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates training-free, label-free ensemble adaptation for time series foundation models (TSFMs). The authors challenge conventional quantile aggregation approaches, such as mean and median averaging, and propose HuberLOP, a robust extension of the linear opinion pool (LOP). HuberLOP penalizes disagreement from models whose forecasts deviate substantially from the collective center. The results demonstrate consistent improvements over equally weighted LOP and other ensemble approaches, including quantile mean and median aggregation.
Critical strengths
Practical and relevant problem. The paper addresses an important practical question: how to effectively combine forecasts from different TSFMs. Given the diversity of model architectures, pretraining data, and capabilities, developing more effective ensemble methods is valuable to the time series forecasting community.
Extensive empirical evaluation. The authors evaluate the proposed method on two prominent benchmarks, TIME and GIFT-EVAL, providing evidence of its performance across a range of forecasting settings.
Simple and generalizable approach. HuberLOP is a plug-and-play method that is training-free, parameter-free, and adaptable to arbitrary sets of forecasting models. This simplicity makes it easy to integrate into existing forecasting pipelines without additional training or supervision.
Critical weaknesses
Reliance on collective agreement: The method assumes that forecasts close to the collective center are more reliable, penalizing models that disagree with the majority. However, a model may be more accurate than most other models in particular scenarios, and the proposed method does not explicitly identify or account for such cases.
Limited empirical justification for the motivation: The paper makes strong claims about the need for a label-free and training-free ensemble method, but provides insufficient empirical justification for these claims. In particular, the authors do not compare the proposed method with context-based approaches such as Synapse and TimeCopilot, which may also support label-free adaptation. Comparisons with their ensemble strategies would help establish the value of HuberLOP relative to existing alternatives.
Marginal improvements over simple baselines: The improvements over basic approaches such as quantile averaging and quantile median appear relatively modest. Although the results in Table 2 suggest a more substantial difference between LOP and HuberLOP, the overall practical significance of the improvements requires further investigation, particularly against stronger, context-based alternatives.
Insufficient justification for excluding context-based adaptation: The paper argues that ensemble weights should be determined solely from member forecasts in a label-free and training-free manner, but this design choice is not sufficiently justified. The claim that no useful supervision is available when a new series arrives overlooks the potential of historical context for backtesting, in-context learning, or model adaptation. The authors acknowledge some of these possibilities but do not sufficiently establish why they are inadequate for the problem considered.
Limited comparison with existing methods and computational overhead: The paper does not provide empirical comparisons with relevant context-based approaches, such as Synapse and TimeCopilot, including their ensemble strategies where applicable. These methods may offer alternative ways to combine forecasts without requiring externally labeled data. Furthermore, the lack of a computational overhead analysis makes it difficult to assess whether HuberLOP provides a meaningful practical advantage over these alternatives; this might even uncover HuberLOP's strength.
Questions for authors
Definition of labeled data: Could the authors clarify what constitutes labeled data in their setting and distinguish it from contextual information, such as sampling frequency, domain, and sequence length? The current terminology may confuse readers about what information is available during adaptation.
Consistency of reported results: Why do the results in Table 2 for a pool size of seven differ from those in Figure 2 for the same pool size? Is this discrepancy due to Figure 2 evaluating every fifth forecast window, or are there other differences in the evaluation protocol?
5 comments
Author's Response to Reviewer L1b6 (2/3)
W2: Limited empirical justification for the motivation: The paper makes strong claims about the need for a label-free and training-free ensemble method, but provides insufficient empirical justification for these claims. In particular, the authors do not compare the proposed method with context-based approaches such as Synapse and TimeCopilot, which may also support label-free adaptation. Comparisons with their ensemble strategies would help establish the value of HuberLOP relative to existing alternatives.
W4: Insufficient justification for excluding context-based adaptation: The paper argues that ensemble weights should be determined solely from member forecasts in a label-free and training-free manner, but this design choice is not sufficiently justified. The claim that no useful supervision is available when a new series arrives overlooks the potential of historical context for backtesting, in-context learning, or model adaptation. The authors acknowledge some of these possibilities but do not sufficiently establish why they are inadequate for the problem considered.
Thanks for your thoughtful feedback, which has helped us better position our paper compared to the related works. Scoring members on the context complements a label-free rule rather than replacing it. Backtest weights and Synapse with a backtest history need twice the TSFM calls and gain significantly only on clean pools, and for a new series from an unseen domain there is no way to confirm in advance that the pool is clean. With naive forecasters or a member failing after the backtest, HuberLOP_con has lower error than both on GIFT-Eval and TIME, and on fev-bench [5], a third benchmark, it beats both under failure (Tables R2 and R3). Our reimplementation of Synapse [1] without history reads the same forecasts as HuberLOP_con, yet its error is more than 20% higher on GIFT-Eval and TIME, both with naive forecasters and when a member fails (Table R1). With a backtest budget, HuberLOP_con + backtest gives the lowest error in Table R2, so the two approaches are complementary (Appendices C.7, C.10, and C.13).
Synapse with archived history needs scored past forecasts of the same series, which a new series lacks, and under failure HuberLOP_con has 4 to 9% lower CRPS than it (Appendix C.12). Models adapted by in-context learning or fine-tuning can join our pool as members, and the ensemble of TimeCopilot [2] is the quantile median, which Table R1 includes.
Table R1: With the same forecasts as input, HuberLOP_con is more robust than Synapse and the TimeCopilot ensemble. CRPS on GIFT-Eval and TIME and scaled quantile loss on fev-bench (↓), in that order in each cell. "Failure" means that one TSFM outputs its last observation after the weights are set. None of these methods runs a backtest. The TimeCopilot ensemble is the quantile median of the members' quantiles, and Synapse is our reimplementation started from uniform weights.
| Method | Naive added | Failure |
|---|---|---|
| TimeCopilot ensemble [2] | 0.4754 / 0.5557 / 0.5244 | 0.4693 / 0.5492 / 0.5096 |
| Synapse [1] | 0.5908 / 0.6796 / 0.5767 | 0.5972 / 0.6648 / 0.5733 |
| HuberLOP_con (ours) | 0.4701 / 0.5504 / 0.5167 | 0.4670 / 0.5469 / 0.5077 |
Table R2: When a member fails after the weights are set, HuberLOP_con beats both backtest-based methods with half the TSFM calls, and HuberLOP_con + backtest, which multiplies backtest weights by those of HuberLOP_con, gives the lowest error. Error when one TSFM outputs its last observation after the weights are set (↓).
| Method | Backtest | TSFM calls | GIFT-Eval | TIME | fev-bench |
|---|---|---|---|---|---|
| Synapse [1] | ✓ | 14 | 0.4711 | 0.5559 | 0.5132 |
| Backtest weights | ✓ | 14 | 0.4774 | 0.5631 | 0.5296 |
| HuberLOP_con (ours) | ✗ | 7 | 0.4670 | 0.5469 | 0.5077 |
| HuberLOP_con + backtest (ours) | ✓ | 14 | 0.4659 | 0.5464 | 0.5065 |
Author's Response to Reviewer L1b6 (3/3)
W5: Limited comparison with existing methods and computational overhead: The paper does not provide empirical comparisons with relevant context-based approaches, such as Synapse and TimeCopilot, including their ensemble strategies where applicable. These methods may offer alternative ways to combine forecasts without requiring externally labeled data. Furthermore, the lack of a computational overhead analysis makes it difficult to assess whether HuberLOP provides a meaningful practical advantage over these alternatives; this might even uncover HuberLOP's strength.
We thank the reviewer for this suggestion, which indeed highlights a strength of HuberLOP_con. HuberLOP_con adds only 3.6 ms to the 123.5 ms of TSFM inference on TIME, needs no labels or stored histories, and is the most accurate of these methods when naive forecasters enter (Table R3). Backtesting runs every TSFM a second time, stored histories need per-series bookkeeping that a new series or model lacks, and the TimeCopilot agent also calls an LLM five or more times per run (Appendices C.9 and C.12).
Table R3: HuberLOP_con needs no labels, no stored history, and no extra TSFM calls, yet it is the most accurate when naive forecasters are added. Requirements and average time per forecast on TIME, and CRPS with three naive forecasters added (↓), with GIFT-Eval before TIME. Times add the combination time to 123.5 ms per inference pass, and the backtest rows assume two passes, with compiled exact-mixing Synapse. Bold marks no requirement or the best CRPS.
| Method | TSFM calls | Labels | Time (ms) | CRPS, naive added |
|---|---|---|---|---|
| TimeCopilot ensemble [2] | 7 | No | 123.6 | 0.4754 / 0.5557 |
| Synapse [1] | 14 | Yes | ≈ 250 | 0.4759 / 0.5579 |
| Backtest weights | 14 | Yes | ≈ 250 | 0.4736 / 0.5571 |
| HuberLOP_con (ours) | 7 | No | 127.0 | 0.4701 / 0.5504 |
W3: Marginal improvements over simple baselines: The improvements over basic approaches such as quantile averaging and quantile median appear relatively modest. Although the results in Table 2 suggest a more substantial difference between LOP and HuberLOP, the overall practical significance of the improvements requires further investigation, particularly against stronger, context-based alternatives.
Thanks for letting us know your concern regarding performance. HuberLOP_con improves on simple baselines even on clean pools, and the gap widens when bad models enter, on fev-bench as well. On the clean pool it is significantly better than the quantile mean, by about 2% on GIFT-Eval and TIME, and than the quantile median on all three benchmarks. These clean-pool gaps are modest because, when every member is sound, a label-free rule has little to correct, and its larger value appears when bad models enter, which no one can rule out on a new series. With naive forecasters, its error rises by less than 1% on GIFT-Eval and TIME and under 3% on fev-bench, less than the median's everywhere and a fraction of the quantile mean's (Table R4).
Table R4: HuberLOP_con degrades the least when three naive forecasters are added. Error (↓) and its increase (%) from the clean pool. Bold marks the best results.
| Method | GIFT-Eval | TIME | fev-bench | Increase (%) |
|---|---|---|---|---|
| Quantile mean | 0.6271 | 0.7205 | 0.7601 | 31.98 / 29.75 / 46.26 |
| LOP | 0.5308 | 0.6140 | 0.6063 | 13.98 / 11.98 / 19.55 |
| Quantile median | 0.4754 | 0.5557 | 0.5244 | 1.75 / 1.52 / 3.89 |
| HuberLOP_con (ours) | 0.4701 | 0.5504 | 0.5167 | 0.93 / 0.87 / 2.79 |
Once again, we appreciate the reviewer highlighting valuable discussion points in our study. We hope our response has clarified the contents of the paper, and we are more than willing to engage in further discussion regarding any aspects that remain unclear.
References
[1] "Synapse: Adaptive Arbitration of Complementary Expertise in Time Series Foundational Models.", In Transactions on Machine Learning Research, 2026
[2] "TimeCopilot.", In arXiv preprint arXiv:2509.00616v3, 2025
[3] "Trimmed Opinion Pools and the Crowd's Calibration Problem.", In Management Science, 2014
[4] "Probabilistic Forecast Aggregation with Statistical Depth.", In European Journal of Operational Research, 2026
[5] "fev-bench: A Realistic Benchmark for Time Series Forecasting.", In Advances in Neural Information Processing Systems (NeurIPS), Evaluations & Datasets Track, 2026
Author's Response to Reviewer L1b6 (1/3)
We sincerely thank the reviewer for your insightful comments, which have helped us improve the paper quality significantly. Below, we are more than willing to clarify the reviewer's concerns one by one.
Q1: Definition of labeled data: Could the authors clarify what constitutes labeled data in their setting and distinguish it from contextual information, such as sampling frequency, domain, and sequence length? The current terminology may confuse readers about what information is available during adaptation.
We appreciate the reviewer for providing a valuable discussion point. This distinction is central to our method, because HuberLOP_con never scores a forecast and therefore needs no labels of the target series. Labeled data are observed values used to score the members' forecasts, from the forecast horizon or from a held-out part of the context, whereas metadata such as frequency, domain, and length are not labels. The lookback window is model input and becomes labels only when part of it is held out for scoring, as in backtesting. We added this definition to Section 3.1.
W1: Reliance on collective agreement: The method assumes that forecasts close to the collective center are more reliable, penalizing models that disagree with the majority. However, a model may be more accurate than most other models in particular scenarios, and the proposed method does not explicitly identify or account for such cases.
We appreciate the reviewer for providing a crucial discussion point. From the current forecasts alone, no rule can tell a correct minority from a wrong one, so we chose the side that keeps the combined forecast from collapsing when bad models enter. Every agreement-based rule shares this limitation, including trimmed [3] and depth pools [4], and Synapse [1] states it in its Limitations section, "This self-referential loop risks confirmation bias, potentially favoring consensus models over correct outliers." When one TSFM is shifted by one context standard deviation and is wrong, HuberLOP_con has the lowest CRPS of nine robust rules, about 5% below LOP on GIFT-Eval and TIME. HuberLOP_con therefore follows the majority. If the disagreeing model is correct, the combination gives up what it would have added, and if it is wrong, it can pull the combination only a little, whereas under LOP a wrong model keeps its full share however far it goes. We discuss this limitation in Section 5 (Appendices C.2 and C.4).
Q2: Consistency of reported results: Why do the results in Table 2 for a pool size of seven differ from those in Figure 2 for the same pool size? Is this discrepancy due to Figure 2 evaluating every fifth forecast window, or are there other differences in the evaluation protocol?
Thanks for your comments, which have helped improve the clarity of our paper. We redrew Figure 2 using all forecasts, and it now matches Table 2. As the reviewer suspected, Figure 2 scored every fifth forecast while normalizing by seasonal naive scores computed on all forecasts. The conclusions of Section 4.3 are unchanged.