- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Backtest-Gated Augmentation Averaging (BGAA), an inference-time method for improving frozen time-series foundation models. It constructs structured views of an observed context, scores them using historical rolling-origin backtests, retains the best positively scored views, and aggregates their predictive distributions with the base forecast. Selection is performed once per dataset configuration using observations available before evaluation. Experiments cover four forecasters and three benchmarks, with compute ladders, view/aggregation ablations, matched random/noise controls, and verification-headroom diagnostics. The paper also gives restricted theoretical results for aggregation and a prior-transport interpretation.
Critical strengths
the selection signal is usable in practice. The method tests candidate transformations on old forecasting problems whose outcomes are already known. It can therefore choose useful transformations before seeing the real evaluation future. Aggregate results improve across all 12 model–benchmark combinations.
extra predictions alone do not explain the improvement. Randomly selecting transformations hurts performance, and averaging forecasts from tiny input perturbations helps very little. Historical selection performs better, supporting the central idea.
Critical weaknesses
how much improvement does the extra computation buy? On GIFT-Eval, the largest budget reduces MASE by approximately 0.50–1.14% while using approximately 7.87–9.40× baseline pass-equivalent computation. That could still be worthwhile, but actual runtime, memory, and the number of forecasts sharing the historical selection cost determine its practical value.
it doesn't seem to compare with other TTC methods, are there no TTC methods available for TS forecasting tasks?
Questions for authors
is the broad transformation pool necessary? The full pool performs better on average than lookback-only or cleaning-only alternatives in the reported subset. However, the paired comparisons are inconclusive. Larger comparisons that also measure cost would clarify if we need so many transformations, or could a few simple ones give similar results with less computation?
5 comments
3. Is the broad transformation pool necessary?
We agree that reducing the candidate pool could make BGAA cheaper. Our question is whether a smaller pool chosen on one model or benchmark transfers without sacrificing accuracy. We added a systematic restricted-pool study in Appendix C.3.5, Tables 14–15, on the same 90 model–configuration pairs as the matched-combiner experiment (three TSFMs, 15 GIFT-Eval and 15 fev-bench configurations per model). We change only the candidate set, while keeping backtest windows, selection and aggregation rules, and evaluation forecasts fixed.
We compare a 22-view pool curated from Chronos-2/GIFT-Eval history with leave-one-configuration-out restricted pools and random pools. These comparisons also quantify the corresponding forward-pass savings.
Table R2. Restricted candidate pools across all 90 pairs. MASE improvement is positive when better; paired log-ratio gaps are to the full pool (positive favors full). Cost is relative to the full pool.
| Candidate pool | Views | MASE improvement | Full gain retained | Gap to full pool [95% CI] | Relative cost |
|---|---|---|---|---|---|
| Full shared pool | 51 | 3.84% | 100% | — | 1.00× |
| Chronos-2/GIFT curated | 22 | 2.89% | 75% | +0.99 [0.36, 1.84] | 0.60× |
| Same-pair leave-one-config-out | 20 | 3.09% | 80% | +0.78 [0.25, 1.45] | 0.53× |
| Other-pairs leave-one-config-out | 20 | 3.03% | 79% | +0.84 [0.28, 1.51] | 0.58× |
| Random 20-view pool | 20 | 2.40% | 62% | +1.49 [0.67, 2.35] | 0.47× |
We find that smaller pools reduce verification cost substantially but lose accuracy relative to the full pool. We find the transfer result particularly informative: when the 22-view Chronos-2/GIFT-Eval pool is applied unchanged to the other 75 configurations, it retains only 69% of the full-pool MASE gain, with a +1.18 [0.44, 2.20] paired disadvantage. Excluding the three largest gaps still leaves a disadvantage of +0.47 [0.23, 0.76]. We also see this loss in Table 15: across the other model–benchmark pairs, 30–41% of the views selected by the full pool are absent from the source-model pool.
We do not claim all 51 views are necessary for every model or configuration. The nested thinking modes already show how much a small pool recovers, and the answer depends on the model (Table 5). With the 9-view low budget (2.4–3.1× base passes), PatchTST-FM-R1 recovers 92% of its xhigh MASE gain (−0.45% versus −0.49%), Chronos-2 45% (−0.44% versus −0.97%), and Toto-2-313M 40% (−0.27% versus −0.68%), whereas TiRex-2 gains nothing (+0.02%). A few simple views therefore suffice for some models but not others, consistent with the model-dependent family preferences in Section 6.
We instead observe a measurable accuracy–compute–transfer trade-off: model-specific pruning saves passes but can discard views useful elsewhere. Our aim is to show that one shared candidate pool and historical selection rule improve diverse frozen models without per-model or per-benchmark tuning. Learning a portable, cost-aware restricted pool remains a promising direction.
4. How does BGAA compare with existing test-time compute approaches?
We appreciate the question and agree that comparisons must respect the information available at inference time. To our knowledge, only two other methods spend inference-time compute on a frozen TSFM, both discussed in Section 2, and neither can serve as a baseline in our setting, which requires a training-free method that uses no information beyond each series' observed history:
- Hua et al. (2026) diversify sampled forecasts by perturbing inputs. Their primary aggregator combines all generated views without rejecting individual harmful views, and their selective variant ranks views using ground-truth labels, i.e., the future observations that a causal forecaster cannot access.
- Pandey et al. (2026) edit a single context by suppressing its most disruptive patch when a meta-feature gate fires. The gate is fit by cross-validation on labeled GIFT-Eval data, and the method is evaluated on TimesFM-2.5 over a subset of GIFT-Eval under a different protocol.
A head-to-head comparison would therefore require either future labels or a gate trained on labeled benchmark data, both of which violate the causal setting BGAA is designed for. Instead, our ablations test the core mechanism of each method inside our pipeline, with the same backbones, candidate pool, causal cutoffs, and metrics. These are not reimplementations, but they isolate the ideas each method relies on:
| Mechanism in prior work | Closest ablation in our pipeline | MASE change vs. base |
|---|---|---|
| Aggregating all views without filtering (Hua et al., primary) | Ungated averaging over the full 51-view pool; Chronos-2, GIFT-Eval, 97 configs (Fig. 4a) | +4.97% (worse), vs. −0.95% for BGAA |
| Diversifying forecasts with input perturbations (Hua et al.) | Averaging jittered copies at matched compute (noise-k); Chronos-2, GIFT-Eval (Table 11) | +0.03% (p = 0.079), vs. −0.76% for gated top-2 |
| Editing the context to remove disruptive segments (Pandey et al.) | Cleaning-only restricted pool; Chronos-2, 15-config subset (Table 10). Anomaly cleaning and despiking were also evaluated and rejected (Sections B and F) | +0.53% (worse), vs. −1.60% for the full pool |
| A meta-feature gate fit on other datasets (Pandey et al.) | Feature-based routers trained leave-one-configuration-out; 3 models, 30 configs (Table 28) | None of 36 variants beats historical top-1 |
The selective variant of Hua et al. corresponds to our label-ranked gate and per-window hindsight selection (Tables 22–23), which we report only as diagnostics because they read future labels. Random view selection at matched compute also worsens MASE by 4.82% and CRPS by 10.73% (Table 11), confirming that additional inference compute is not useful without selection.
Beyond test-time compute methods, the applicable baselines for choosing among candidate forecasts with historical evidence come from classical forecast combination and selection. We compare directly against these, with identical candidate forecasts and backtest windows (Appendix C.3.4, Table 13). On a preselected, high-opportunity 30-configuration subset, BGAA improves aggregate MASE by 3.84%, versus 1.79% for historical top-1 and 2.68% for base + top-1, and significantly outperforms both. Its mean advantage over L2-regularized combinations is not statistically significant, but BGAA shows fewer severe regressions and needs fewer application forecasts. Beyond these, we are not aware of any other baselines that apply to this setting.
We believe these experiments support our main distinction: BGAA's contribution is a causal verifier that makes additional inference computation selective and useful, not merely a larger ensemble.
We hope these responses and the additional experiments address your concerns. If so, we would be grateful if you would consider updating your score, and we are happy to answer any further questions during the discussion period.
Author Response to Reviewer cgNk
We thank the reviewer for the constructive feedback. As described in our general clarification post, we corrected an implementation bug in our codebase. Two of the numbers cited in the review changed: at the largest budget, BGAA reduces GIFT-Eval MASE by 0.45–0.97% (previously 0.50–1.14%) at 8.32–9.54× baseline pass-equivalent computation (previously 7.87–9.40×). The corrected results still support the strengths highlighted in the review (that the historical selection signal is usable in practice and that extra predictions alone do not explain the improvement). We agree that the value of test-time compute depends on both its accuracy gains and its inference cost.
1. Are the improvements worth the additional computation?
We evaluate four thinking budgets with nested 9/19/31/51-view candidate pools (Figures 2–3, Tables 5–6). We observe diminishing returns: on GIFT-Eval, the high rung recovers approximately 61–96% of the xhigh MASE gain at 54–78% of its estimated compute, depending on the model. We do not claim that the largest budget is always preferable.
We find the comparison with deploying a larger frozen model more informative. On GIFT-Eval, TiRex-2 + BGAA (high) achieves 0.6945 MASE at ~89 PFLOPs, versus 0.6956 at ~121 PFLOPs for Toto-2-2.5B (Figure 2): lower MASE with approximately 26% less estimated inference compute. This is a pairwise MASE/estimated-FLOPs result, not a CRPS or latency dominance claim.
Separately, on BOOM, Toto-2-313M + BGAA reaches 0.575 MASE, outperforming the published 1B/2.5B family members at approximately 0.582/0.581 (Tables 1–2, 18); on CRPS, it is on par with them (0.350 versus 0.349). We use this BOOM comparison to establish accuracy, not matched inference cost.
2. How does the compute trade-off look in actual latency and memory?
We additionally measured wall-clock inference on an ** NVIDIA H100**, including cold historical verification (Appendix C.2, Tables 7–8).
Table R1. Accuracy, estimated compute, and measured throughput. PFLOPs refer to the full GIFT-Eval sweep; latency separately uses representative synthetic contexts.
At 10,000 forecasts, Chronos-2 + BGAA is 2.0–4.1× faster than Toto-2-2.5B and uses about 21× less GPU memory, including verification. However, Toto-2-2.5B remains more accurate than Chronos-2 + BGAA; TiRex-2 xhigh is more accurate in MASE but slower. We interpret these as different accuracy–efficiency operating points, not a claim of joint accuracy–latency dominance. Also, TiRex-2's checkpoint pads every input to 9,120 steps (285 patches), so its per-call computation does not fall when views shorten the observed context. For the same reason, the estimated-FLOPs advantage of TiRex-2 + BGAA (high) over Toto-2-2.5B does not carry over to wall-clock time: TiRex-2's base model alone takes 17.7 s per 10,000 forecasts.