acceptodds
Under review as a conference paper at ICLR 2027

A Verifier from the Past: Test-Time Compute for Time Series Foundation Models

Abstract

Test-time compute (TTC) improves language models by spending additional inference compute to generate, verify, and select among candidate outputs, but this recipe has not transferred directly to time series forecasting. A language model can verify a proof or vote across samples, but a forecaster must be causal; it cannot use future information to score a candidate. We propose Backtest-Gated Augmentation Averaging (BGAA), a TTC algorithm for time series forecasting that builds a causal verifier from the series’s own history, using rolling-origin backtests to score structured views of the context, discard those that underperform the untransformed context, and average the survivors’ predictive distributions. BGAA improves every tested TSFM–benchmark combination, reducing MASE by 0.5–1.1% on GIFT-Eval, 3.2–5.9% on fev-bench, and 1.9–5.1% on BOOM, where a 313M model with TTC outperforms the 1B and 2.5B models in its family. We further show that useful views have structured, model-dependent effects consistent with a prior-transport interpretation. On the 53 cross-fittable GIFT-Eval configs, the deployed policy gains only 0.4–1.5% CRPS, versus 1.4–4.1% when choosing the best view for each dataset configuration and 17–19% when choosing the best view separately for each test window. This gap suggests that a major remaining challenge is window-level view selection.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Foundation Models, Test-time Compute, Time Series Forecasting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_cgNkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Backtest-Gated Augmentation Averaging (BGAA), an inference-time method for improving frozen time-series foundation models. It constructs structured views of an observed context, scores them using historical rolling-origin backtests, retains the best positively scored views, and aggregates their predictive distributions with the base forecast. Selection is performed once per dataset configuration using observations available before evaluation. Experiments cover four forecasters and three benchmarks, with compute ladders, view/aggregation ablations, matched random/noise controls, and verification-headroom diagnostics. The paper also gives restricted theoretical results for aggregation and a prior-transport interpretation.

Critical strengths

the selection signal is usable in practice. The method tests candidate transformations on old forecasting problems whose outcomes are already known. It can therefore choose useful transformations before seeing the real evaluation future. Aggregate results improve across all 12 model–benchmark combinations.

extra predictions alone do not explain the improvement. Randomly selecting transformations hurts performance, and averaging forecasts from tiny input perturbations helps very little. Historical selection performs better, supporting the central idea.

Critical weaknesses

how much improvement does the extra computation buy? On GIFT-Eval, the largest budget reduces MASE by approximately 0.50–1.14% while using approximately 7.87–9.40× baseline pass-equivalent computation. That could still be worthwhile, but actual runtime, memory, and the number of forecasts sharing the historical selection cost determine its practical value.

it doesn't seem to compare with other TTC methods, are there no TTC methods available for TS forecasting tasks?

Questions for authors

is the broad transformation pool necessary? The full pool performs better on average than lookback-only or cleaning-only alternatives in the reported subset. However, the paired comparisons are inconclusive. Larger comparisons that also measure cost would clarify if we need so many transformations, or could a few simple ones give similar results with less computation?

5 comments
AuthorsOpenReview ↗

Author Response to Reviewer cgNk

We thank the reviewer for the constructive feedback. As described in our general clarification post, we corrected an implementation bug in our codebase. Two of the numbers cited in the review changed: at the largest budget, BGAA reduces GIFT-Eval MASE by 0.45–0.97% (previously 0.50–1.14%) at 8.32–9.54× baseline pass-equivalent computation (previously 7.87–9.40×). The corrected results still support the strengths highlighted in the review (that the historical selection signal is usable in practice and that extra predictions alone do not explain the improvement). We agree that the value of test-time compute depends on both its accuracy gains and its inference cost.

1. Are the improvements worth the additional computation?

We evaluate four thinking budgets with nested 9/19/31/51-view candidate pools (Figures 2–3, Tables 5–6). We observe diminishing returns: on GIFT-Eval, the high rung recovers approximately 61–96% of the xhigh MASE gain at 54–78% of its estimated compute, depending on the model. We do not claim that the largest budget is always preferable.

We find the comparison with deploying a larger frozen model more informative. On GIFT-Eval, TiRex-2 + BGAA (high) achieves 0.6945 MASE at ~89 PFLOPs, versus 0.6956 at ~121 PFLOPs for Toto-2-2.5B (Figure 2): lower MASE with approximately 26% less estimated inference compute. This is a pairwise MASE/estimated-FLOPs result, not a CRPS or latency dominance claim.

Separately, on BOOM, Toto-2-313M + BGAA reaches 0.575 MASE, outperforming the published 1B/2.5B family members at approximately 0.582/0.581 (Tables 1–2, 18); on CRPS, it is on par with them (0.350 versus 0.349). We use this BOOM comparison to establish accuracy, not matched inference cost.

2. How does the compute trade-off look in actual latency and memory?

We additionally measured wall-clock inference on an ** NVIDIA H100**, including cold historical verification (Appendix C.2, Tables 7–8).

Table R1. Accuracy, estimated compute, and measured throughput. PFLOPs refer to the full GIFT-Eval sweep; latency separately uses representative synthetic contexts.

SystemGIFT MASEEstimated PFLOPsTime for 10,000 forecastsGPU memory
TiRex-2 + BGAA xhigh0.6940148.8114 s5.96 GiB
Chronos-2 + BGAA low0.701253.07.02 s0.48 GiB
Chronos-2 + BGAA xhigh0.6975208.714.3 s0.48 GiB
Toto-2-2.5B base0.695612128.5 s10.30 GiB

At 10,000 forecasts, Chronos-2 + BGAA is 2.0–4.1× faster than Toto-2-2.5B and uses about 21× less GPU memory, including verification. However, Toto-2-2.5B remains more accurate than Chronos-2 + BGAA; TiRex-2 xhigh is more accurate in MASE but slower. We interpret these as different accuracy–efficiency operating points, not a claim of joint accuracy–latency dominance. Also, TiRex-2's checkpoint pads every input to 9,120 steps (285 patches), so its per-call computation does not fall when views shorten the observed context. For the same reason, the estimated-FLOPs advantage of TiRex-2 + BGAA (high) over Toto-2-2.5B does not carry over to wall-clock time: TiRex-2's base model alone takes 17.7 s per 10,000 forecasts.

3. Is the broad transformation pool necessary?

We agree that reducing the candidate pool could make BGAA cheaper. Our question is whether a smaller pool chosen on one model or benchmark transfers without sacrificing accuracy. We added a systematic restricted-pool study in Appendix C.3.5, Tables 14–15, on the same 90 model–configuration pairs as the matched-combiner experiment (three TSFMs, 15 GIFT-Eval and 15 fev-bench configurations per model). We change only the candidate set, while keeping backtest windows, selection and aggregation rules, and evaluation forecasts fixed.

We compare a 22-view pool curated from Chronos-2/GIFT-Eval history with leave-one-configuration-out restricted pools and random pools. These comparisons also quantify the corresponding forward-pass savings.

Table R2. Restricted candidate pools across all 90 pairs. MASE improvement is positive when better; paired log-ratio gaps are to the full pool (positive favors full). Cost is relative to the full pool.

Candidate poolViewsMASE improvementFull gain retainedGap to full pool [95% CI]Relative cost
Full shared pool513.84%100%—1.00×
Chronos-2/GIFT curated222.89%75%+0.99 [0.36, 1.84]0.60×
Same-pair leave-one-config-out203.09%80%+0.78 [0.25, 1.45]0.53×
Other-pairs leave-one-config-out203.03%79%+0.84 [0.28, 1.51]0.58×
Random 20-view pool202.40%62%+1.49 [0.67, 2.35]0.47×

We find that smaller pools reduce verification cost substantially but lose accuracy relative to the full pool. We find the transfer result particularly informative: when the 22-view Chronos-2/GIFT-Eval pool is applied unchanged to the other 75 configurations, it retains only 69% of the full-pool MASE gain, with a +1.18 [0.44, 2.20] paired disadvantage. Excluding the three largest gaps still leaves a disadvantage of +0.47 [0.23, 0.76]. We also see this loss in Table 15: across the other model–benchmark pairs, 30–41% of the views selected by the full pool are absent from the source-model pool.

We do not claim all 51 views are necessary for every model or configuration. The nested thinking modes already show how much a small pool recovers, and the answer depends on the model (Table 5). With the 9-view low budget (2.4–3.1× base passes), PatchTST-FM-R1 recovers 92% of its xhigh MASE gain (−0.45% versus −0.49%), Chronos-2 45% (−0.44% versus −0.97%), and Toto-2-313M 40% (−0.27% versus −0.68%), whereas TiRex-2 gains nothing (+0.02%). A few simple views therefore suffice for some models but not others, consistent with the model-dependent family preferences in Section 6.

We instead observe a measurable accuracy–compute–transfer trade-off: model-specific pruning saves passes but can discard views useful elsewhere. Our aim is to show that one shared candidate pool and historical selection rule improve diverse frozen models without per-model or per-benchmark tuning. Learning a portable, cost-aware restricted pool remains a promising direction.

4. How does BGAA compare with existing test-time compute approaches?

We appreciate the question and agree that comparisons must respect the information available at inference time. To our knowledge, only two other methods spend inference-time compute on a frozen TSFM, both discussed in Section 2, and neither can serve as a baseline in our setting, which requires a training-free method that uses no information beyond each series' observed history:

  • Hua et al. (2026) diversify sampled forecasts by perturbing inputs. Their primary aggregator combines all generated views without rejecting individual harmful views, and their selective variant ranks views using ground-truth labels, i.e., the future observations that a causal forecaster cannot access.
  • Pandey et al. (2026) edit a single context by suppressing its most disruptive patch when a meta-feature gate fires. The gate is fit by cross-validation on labeled GIFT-Eval data, and the method is evaluated on TimesFM-2.5 over a subset of GIFT-Eval under a different protocol.

A head-to-head comparison would therefore require either future labels or a gate trained on labeled benchmark data, both of which violate the causal setting BGAA is designed for. Instead, our ablations test the core mechanism of each method inside our pipeline, with the same backbones, candidate pool, causal cutoffs, and metrics. These are not reimplementations, but they isolate the ideas each method relies on:

Mechanism in prior workClosest ablation in our pipelineMASE change vs. base
Aggregating all views without filtering (Hua et al., primary)Ungated averaging over the full 51-view pool; Chronos-2, GIFT-Eval, 97 configs (Fig. 4a)+4.97% (worse), vs. −0.95% for BGAA
Diversifying forecasts with input perturbations (Hua et al.)Averaging jittered copies at matched compute (noise-k); Chronos-2, GIFT-Eval (Table 11)+0.03% (p = 0.079), vs. −0.76% for gated top-2
Editing the context to remove disruptive segments (Pandey et al.)Cleaning-only restricted pool; Chronos-2, 15-config subset (Table 10). Anomaly cleaning and despiking were also evaluated and rejected (Sections B and F)+0.53% (worse), vs. −1.60% for the full pool
A meta-feature gate fit on other datasets (Pandey et al.)Feature-based routers trained leave-one-configuration-out; 3 models, 30 configs (Table 28)None of 36 variants beats historical top-1

The selective variant of Hua et al. corresponds to our label-ranked gate and per-window hindsight selection (Tables 22–23), which we report only as diagnostics because they read future labels. Random view selection at matched compute also worsens MASE by 4.82% and CRPS by 10.73% (Table 11), confirming that additional inference compute is not useful without selection.

Beyond test-time compute methods, the applicable baselines for choosing among candidate forecasts with historical evidence come from classical forecast combination and selection. We compare directly against these, with identical candidate forecasts and backtest windows (Appendix C.3.4, Table 13). On a preselected, high-opportunity 30-configuration subset, BGAA improves aggregate MASE by 3.84%, versus 1.79% for historical top-1 and 2.68% for base + top-1, and significantly outperforms both. Its mean advantage over L2-regularized combinations is not statistically significant, but BGAA shows fewer severe regressions and needs fewer application forecasts. Beyond these, we are not aware of any other baselines that apply to this setting.

We believe these experiments support our main distinction: BGAA's contribution is a causal verifier that makes additional inference computation selective and useful, not merely a larger ensemble.

We hope these responses and the additional experiments address your concerns. If so, we would be grateful if you would consider updating your score, and we are happy to answer any further questions during the discussion period.

Reviewer_uB9AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Backtest-Gated Augmentation Averaging (BGAA), which improves a frozen forecaster by generating forecasts from transformed versions of its input. Historical rolling-origin backtests score these views, and the best positively scoring candidates are combined with the original forecast. Selection is performed once per dataset configuration and reused across its evaluation windows. Experiments cover four foundation models and three benchmarks, with additional analyses of compute, aggregation, view behavior, and the gap between historical selection and hindsight selection.

Critical strengths
  1. The empirical results is convincing. A fixed pool and policy improve both point and probabilistic metrics across all 12 model–benchmark combinations. Random-view, jitter, ungated-ensemble, and restricted-pool controls provide useful evidence that the gains depend on selecting informative views rather than merely increasing the number of forecasts.

  2. The paper is careful about what its analysis establishes. It separates historical-loss guarantees from future performance, acknowledges that the deployed CDF-mixture policy is not covered by the main backtest guarantee, and treats hindsight selection as a diagnostic. The configuration-level versus window-level ranking analysis also identifies a concrete limitation of the current verifier.

Critical weaknesses
  1. The gains can be expensive relative to their magnitude. On GIFT-Eval, the highest-budget setting costs approximately 7.9–9.4 times the baseline across models for roughly 0.5–1.1% MASE improvement. The FLOPs accounting is useful and explicitly qualified, but it does not establish the practical trade-off in latency or throughput. Because selection cost is amortized across configurations, the value may also depend strongly on how many subsequent forecasts reuse the selected views.

  2. The comparisons establish the value of historical selection over random or indiscriminate aggregation more clearly than the advantage of BGAA’s particular selection rule. The paper includes several aggregation variants, but a stronger comparison would use the same historical forecasts to select one best view or fit a regularized forecast combination. The cross-fitted best-view diagnostic uses a different information setting and does not answer this deployment comparison. This matters because the methodological contribution rests substantially on how historical evidence is used.

Questions for authors
  1. How does BGAA compare with historical best-single-view selection and regularized convex combination weights fitted on exactly the same gate windows?
  2. What are the measured selection latency and per-forecast latency for the reported budget levels? How does total cost change when a selected configuration is reused for only a small number of forecasts?
  3. How are gate windows sampled across series within a configuration? Does the benefit persist when selection is restricted to the history of each individual series?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer uB9A

We thank the reviewer for the careful assessment and concrete suggestions. As described in our general clarification post, we corrected an implementation bug in our codebase. Two numbers cited in the review changed: at the highest (xhigh) budget, BGAA now costs 8.3–9.5× the base in forward passes (previously 7.9–9.4×), and its GIFT-Eval MASE improvement is 0.45–0.97% (previously 0.5–1.14%). The strengths highlighted in the review, including improvements across all 12 model–benchmark combinations and controls showing that the gains depend on selecting informative views, remain supported by the corrected results. All new experiments below use the corrected pipeline.

1. How does BGAA compare with historical top-1 selection and regularized forecast combinations?

We added this experiment in Appendix C.3.4, Table 13. We compare BGAA against historical top-1 selection, base + top-1 averaging, and two L2-regularized, nonnegative convex combinations. We give all methods identical candidate forecasts and historical backtest windows, without evaluation labels. For the L2 methods, we fit per-quantile weights with five-fold historical cross-validation; in one variant, we penalize deviation from identity weights.

We evaluate 30 configurations (15 GIFT-Eval, 15 fev-bench) × three models = 90 model–configuration pairs. We selected the configurations ranked by the deployed gate's MASE gain averaged over the three models with at most two per dataset family, because view effectiveness can only be compared across series characteristics where views actually change the forecast. This oversamples datasets where views matter, so effect sizes are larger than on a typical configuration. Thus, this is a targeted subset, not a representative full-benchmark estimate.

Table R1. Matched historical-combiner comparison. MASE/WQL are changes versus frozen base (%; negative improves). Positive paired MASE gaps favor BGAA; intervals are 95% bootstrap CIs.

MethodMASE Δ%WQL Δ%Paired MASE gap to BGAA [95% CI]Worst regressionConfigs >3% worseExtra forecasts
BGAA−3.84−3.55—+1.38%02.8
Historical top-1−1.79−0.49+2.11 [0.45, 4.04]+27.39%61.0
Base + top-1−2.68−1.97+1.20 [0.22, 2.35]+6.54%31.0
L2-convex−2.97−2.45+0.91 [−0.48, 2.61]+12.30%813.2
L2-anchored−3.01−2.33+0.86 [−0.57, 2.59]+12.30%56.3

We find that BGAA has the largest aggregate improvements in both metrics and significantly outperforms historical top-1 and base + top-1 in paired MASE comparisons. We explicitly acknowledge that its mean advantage over the L2-regularized combinations is not statistically significant. Nevertheless, we observe no >3% regressions for BGAA on this targeted set, a smaller worst regression, and substantially fewer application forecasts than the regularized combinations.

We believe this comparison better isolates the practical value of admitting a small, historically verified set of views and aggregating them, rather than selecting only one view or fitting more detailed weights from the same historical evidence. The much larger best-view and per-window hindsight results in Appendix D.3, Tables 22–23 use held-out test labels for diagnosis, and are not deployable baselines.

2. What are the actual latency and memory costs, and how does amortization change them?

We added H100 wall-clock latency and peak GPU-memory measurements (Appendix C.2, Tables 7–8). We include historical backtesting, candidate forecasting, and aggregation in cold BGAA timings. We can then reuse the chosen views for subsequent forecasts within the same configuration. We measure representative synthetic GIFT-Eval contexts, not a timed full-benchmark sweep.

Relative to its own base model, BGAA's cost depends strongly on how many forecasts reuse a selection. For Chronos-2 at the xhigh budget, selection takes 4.49 s, so a single forecast costs about 313× the base forecast; at 10,000 forecasts, the total including selection falls to 8.4× the base (Table 8). The low budget costs 4.1× the base at 10,000 forecasts (7.02 s versus 1.71 s) and recovers 45% of the xhigh MASE gain (−0.44% versus −0.97%). BGAA is therefore expensive for isolated, one-off forecasts and best suited to settings where many forecasts share a configuration.

Table R2. Measured inference costs. GIFT-Eval MASE comes from the full benchmark; times are measured on the representative workload and, for BGAA, include view selection.

SystemGIFT MASETime, F = 1Time, F = 100Time, F = 10,000Peak GPU memory
Chronos-2 base0.70430.0143 s0.0203 s1.71 s0.48 GiB
Chronos-2 + BGAA low0.70120.125 s0.160 s7.02 s0.48 GiB
Chronos-2 + BGAA xhigh0.69754.49 s4.58 s14.3 s0.48 GiB
TiRex-2 + BGAA xhigh0.694040.4 s41.1 s114 s5.96 GiB
Toto-2-2.5B base0.69560.133 s0.319 s28.5 s10.30 GiB

For context, we also compare against deploying a larger frozen model. At 10,000 forecasts, Chronos-2 + BGAA takes 7.02–14.3 s across thinking modes, compared with 28.5 s for Toto-2-2.5B, with about 21× less GPU memory. We note that Toto-2-2.5B nevertheless has better absolute accuracy than Chronos-2 + BGAA. Conversely, TiRex-2 + BGAA is more accurate in MASE but slower at the measured xhigh budget. We therefore do not claim joint accuracy–latency dominance. TiRex-2 also pads inputs to a fixed 9,120-step window, so lookback cropping does not reduce its per-call model computation (Section C.2).

3. How are gate windows sampled, and would independent per-series selection work better than pooling across a configuration?

Sampling. We uniformly sample windows per series, and origins are placed at roughly equal intervals. Each gate window provides n_f = 3 rolling origins, or one under the light gate for series with fewer than 8 usable gate windows.

Per-series selection. We added a matched comparison on 41 multi-series GIFT-Eval configurations containing 1,972 series (Appendix D.4, Table 24). We use the same candidate pool and sampled historical evidence for the pooled and independent per-series selectors; only the selection granularity differs.

Table R3. MASE change versus base (%) and paired per-series-minus-pooled log gap [95% CI]. Negative MASE change is better; positive gap favors pooling.

ModelConfiguration-pooledPer-seriesPaired gap [95% CI]
Chronos-2−0.91−0.17+0.74 [−0.32, 2.34]
TiRex-2−0.65−0.25+0.40 [−0.01, 0.89]
Toto-2-313M−0.66+0.22+0.89 [0.23, 1.67]

Pooling across a configuration performs better on average for all three models, though the aggregate difference is statistically reliable only for Toto-2. In the most series-dense configurations (118–156 series), pooling significantly improves on per-series selection for Chronos-2 (+1.50 [0.50, 2.70]) and TiRex-2 (+0.48 [0.08, 0.88]). We take this as evidence for pooling historical verification when each series alone offers too little for stable ranking. However, we do not rule out better localized selectors in future work.

We hope these responses and the additional experiments address your concerns. If so, we would be grateful if you would consider updating your score, and we are happy to answer any further questions during the discussion period.

AuthorsOpenReview ↗

Additional Clarification on Evaluation Soundness

We would also like to clarify the scope of the implementation correction discussed in our general clarification post, particularly to ensure confidence in the experimental evidence supporting our approach.

We sincerely apologize again for the original implementation error. We recognize that the reviewers invested considerable time in evaluating our original results, and we take full responsibility for ensuring that the corrected manuscript provides a reliable and consistent evaluation.

First, we have fixed and validated the rolling-evaluation mechanism to preserve strict causality. For configuration-level selection, the historical gate uses only observations available before the first evaluation forecast window. We have additionally tested the corrected implementation to ensure that future evaluation targets cannot influence the selection of candidate views.

Second, we have independently regenerated every affected result using the corrected implementation, rather than adjusting the previously reported scores. This includes all affected gate-dependent experiments, supporting appendix tables, figures, and statistical analyses. We have ensured that no results affected by the original issue remain in the updated evaluation. Importantly, the corrected results continue to support all our central claims, including consistent improvements across all 12 model–benchmark combinations, accuracy-compute Pareto claims, and the effectiveness of historical verification.

Third, the updated manuscript contains all corrected numerical results, together with the additional experiments requested during the rebuttal. In particular, we have added information-matched comparisons against historical top-1 selection and regularized forecast combinations, measured inference latency and GPU memory, and a direct comparison between configuration-pooled and per-series historical selection. These experiments use the corrected evaluation pipeline and are reported in the revised appendix.

We believe these additional results substantially strengthen the empirical evidence supporting BGAA, while directly addressing the reviewer's original concerns regarding stronger baselines, computational efficiency, and the reliability of configuration-level selection. The corrected findings also continue to support the main strengths recognized in the original review.

We fully understand the importance of this correction and sincerely regret any confusion or additional effort it has caused. We hope the reviewer will consider the complete corrected evaluation, including the newly added experiments, when assessing the overall soundness and contributions of our work.

We thank the reviewer again for the careful assessment and constructive feedback, which have been valuable in improving the manuscript.

Reviewer_5wN9OpenReview ↗
Rating
4(+1)
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a test-time computation method to improve the performance of general time-series foundation models across several benchmarks without violating causality constraints. The method generates multiple transformed views of the input, evaluates them through backtesting on the observed history, and aggregates the best selected forecasts. The paper also shows that, with this method, smaller models can achieve comparable or better performance than larger models, both within and across model families.

Critical strengths
  • The method is carefully designed to avoid information leakage and respect causality constraints.
  • Evaluation covers three benchmarks and uses both point-forecasting and probabilistic metrics.
  • Comparisons include recent TSFMs of different model families and scales.
Critical weaknesses
  • The paper analyzes which transformations work best for different model types, but the results appear inconclusive. I would have liked to see a more fine-grained analysis relating the effectiveness of each transformation to both the model and the statistical characteristics of each series. Intuitively, the best transformation may depend on both factors, allowing each model to benefit from a suitable input representation for a given series.
  • Could the method incorporate a criterion to avoid aggregating transformations when the historical evidence for improvement is too weak to suggest that the gains will transfer to future observations? Table 3 shows that, on GIFT-Eval, 21% of configurations regress under the deployed policy. This could help explain the more modest aggregate gains on this benchmark and raises the question of whether the selection mechanism could be made more conservative.
Questions for authors
  • Why are the improvements on GIFT-Eval more limited than on the other benchmarks? Could this be related to the characteristics or representation of the series in this benchmark?
  • How well do the gains persist at longer forecast horizons? Intuitively, I would expect the benefit of transformations selected using the observed history to weaken as the forecast horizon increases and the predictions extend further beyond the context used for selection.
5 comments
AuthorsOpenReview ↗

Author Response to Reviewer 5wN9

We thank the reviewer for the positive assessment and constructive suggestions. As described in our general clarification post, we corrected an implementation bug in our codebase. One statistic cited in the review changed: under the corrected deployed policy, 39% of GIFT-Eval configurations regress for Chronos-2 (previously 21%), although 95% show no statistically significant regression (Table 3). We address this directly in point 2. The strengths highlighted in the review, including evaluation on three benchmarks with point and probabilistic metrics and comparisons across model families and scales, remain unaffected. Below, we address the concerns using the corrected results and additional experiments.

1. How does transformation effectiveness vary across models and series?

We agree that useful transformations should depend on both the forecasting model and the series' properties. In our existing family analysis (Section 6, Tables 25–26), we find weak agreement across models: Kendall's τ between family-benefit rankings ranges from 0.01 to 0.35. For example, differencing is strongest for Chronos-2 (−0.85% MASE) but slightly hurts TiRex-2 (+0.01%), whereas lookback is particularly useful for PatchTST-FM-R1 and Toto-2-313M. We also observe differences in admitted families across domains (Figure 10a).

We extended this analysis to 30 configurations (15 GIFT-Eval, 15 fev-bench) across Chronos-2, Toto-2-313M, and TiRex-2. We compare the lowest and highest terciles of seasonality, trend strength, lag-1 autocorrelation, and volatility (Appendix E.2, Table 27).

Table R1. Hindsight-best family by series characteristic. Cells report family MASE change versus base (%) [95% bootstrap CI], followed by BGAA's change on the same windows. Negative is better.

CharacteristicTercileChronos-2Toto-2TiRex-2
SeasonalityLowdiff −6.4 [−12.2, −0.9]; BGAA −6.9winsor −3.2 [−13.7, +4.4]; BGAA −1.7offset −0.4 [−1.7, +0.5]; BGAA −2.4
Highdecomp −4.3 [−7.6, −1.7]; BGAA −5.6decomp −1.5 [−3.2, +0.1]; BGAA −1.2multideseason −2.0 [−4.4, +0.6]; BGAA −3.4
Trend strengthLowdecomp −2.6 [−5.4, −0.6]; BGAA −4.1decomp −1.2 [−2.1, −0.2]; BGAA −0.9multideseason −1.4 [−3.2, +0.4]; BGAA −2.0
Highdecomp −2.9 [−6.0, −0.5]; BGAA −8.1winsor −2.5 [−11.0, +3.7]; BGAA −2.1multideseason −0.9 [−3.7, +0.8]; BGAA −4.1
Lag-1 autocorr.Lowdecomp −2.0 [−3.9, −0.3]; BGAA −2.0winsor −1.8 [−5.5, +0.3]; BGAA −1.3multideseason −0.7 [−1.8, +0.2]; BGAA −1.7
Highdiff −5.7 [−10.9, −0.8]; BGAA −7.7decomp −1.2 [−3.2, +0.6]; BGAA −1.7multideseason −2.6 [−6.3, +0.5]; BGAA −5.8
VolatilityLowdiff −5.4 [−10.9, −0.3]; BGAA −7.6winsor −1.9 [−11.9, +5.1]; BGAA −1.5offset −0.3 [−1.5, +0.4]; BGAA −2.6
Highdecomp −3.0 [−5.1, −1.2]; BGAA −4.0decomp −1.8 [−3.9, −0.1]; BGAA −2.7winsor −0.4 [−1.0, −0.0]; BGAA −1.7

For example, we find that Chronos-2 favors differencing on low-seasonality or highly autocorrelated series, but decomposition on high-seasonality series. BGAA, which selects views causally, also matches or exceeds the hindsight-best single family in 18 of 24 cells (all 8 for TiRex-2, 7 plus one tie for Chronos-2, and 2 for Toto-2), although this comparison is not budget-matched, since BGAA aggregates up to three views. We emphasize that these are retrospective associations: families are selected using realized outcomes, and their CIs are not adjusted for that selection. We use them to explain variation in effectiveness, not to define a deployable selector.

1. Can series characteristics replace historical verification? (continued)

We tested this directly with leave-one-configuration-out feature routers trained on the other configurations, without using outcomes from the evaluated configuration (Appendix E.2, Table 28). We match historical top-1 and the routers at the same one-view application budget; we include BGAA with three selected views only as a reference.

Table R2. MASE change (%) versus base. Parentheses give the router's paired log-ratio disadvantage relative to historical top-1 [95% bootstrap CI]. Lower MASE change is better.

Selection methodChronos-2Toto-2TiRex-2
Historical top-1 + identity−4.4−1.7−1.9
Global view (no characteristics)−2.8 (+1.7 [0.3, 3.3])+2.0 (+3.7 [1.5, 6.4])+1.3 (+3.2 [1.4, 5.5])
Four-characteristic router−1.5 (+3.0 [0.0, 7.8])−0.7 (+1.1 [−0.7, 2.7])+2.8 (+4.6 [1.6, 9.0])
Best of 12 routers (post hoc)−4.3 (+0.2 [−1.1, 1.2])−1.6 (+0.1 [−2.6, 2.1])−0.7 (+1.3 [−0.8, 3.7])
BGAA (k=3; reference only)−6.1−2.2−3.2

We find that none of the 36 router variants improves average MASE over historical top-1. Even when we select the best of 12 routers in hindsight, we do not consistently match the gate. We find that series statistics help explain which transformations work, but these simple feature routers do not reliably replace historical verification.

2. Can more conservative admission reduce regressions?

We agree that historical improvement does not guarantee improvement on a future forecast. BGAA already requires historical evidence before admitting a view: a view must improve on the base on average over the usable gate windows (ε̂(g) > 0), and the standard gate requires at least m_min = 8 usable windows (Section 3). Where a configuration's history is too short for three rolling origins per gate window, we use a lighter gate with one rolling origin.

Beyond these defaults, we study two more conservative policies.

Margin rule (Section A.1, Table 19). Requiring a minimum mean improvement and a minimum win fraction makes BGAA abstain on most configurations. On Chronos-2/GIFT-Eval, this reduces the number of regressing configurations from 38 to 9 (win/loss 17/9, versus 55/38 for the deployed policy), but retains only about a third of the MASE gain (−0.35% versus −0.97%), and its single worst regression is larger (+12.1% versus +7.9%). Abstention therefore reduces how often BGAA regresses, but not necessarily how badly.

Spread-routed CRPS gate (routemix05-crps). This policy changes both gating and aggregation (Table 3; Appendix D.1, Tables 16–17).

Table R3. Chronos-2 regression risk (Table 3). The safer policy is routemix05-crps.

BenchmarkEqual/better configs: deployed → saferWorst regression: deployed → safer
GIFT-Eval61% → 67%8% → 6%
BOOM69% → 85%199% → 98%
fev-bench70% → 77%7% → 6%

The largest BOOM regressions come from a small number of near-constant series, for which the scaled metric is unstable (Section 5).

We also find that the safer policy generally improves win/loss ratios across the evaluated models: for example, on GIFT-Eval, Chronos-2 improves from 55/38 to 63/29 and TiRex-2 from 54/38 to 62/28; on fev-bench, from 56/38 to 71/23 and 54/39 to 64/27, respectively (Tables 16–17).

We observe a cost to average accuracy: on GIFT-Eval, Chronos-2 achieves 0.6975 MASE with the deployed policy versus 0.6996 with spread-routing (base: 0.7043). We also observe that the fraction with no statistically significant regression decreases slightly from 95% to 93% on GIFT-Eval. We interpret these results as a useful reliability–accuracy trade-off, not a guarantee against future regressions.

3. Why are the improvements smaller on GIFT-Eval?

We believe limited accuracy headroom and the amount of historical backtest evidence are plausible contributing factors, although neither is established as the sole cause. On GIFT-Eval, even substantial model-size increases yield relatively small differences.

Table R4. Published Toto-2 GIFT-Eval results. Lower MASE is better.

ModelParametersMASEImprovement vs. 313M
Toto-2-313M313M0.7028—
Toto-2-1B1B0.6992+0.50%
Toto-2-2.5B2.5B0.6956+1.01%

We note that increasing model size by approximately 8× improves MASE by about 1%, comparable in magnitude to BGAA's corrected 0.45–0.97% gains on GIFT-Eval. Moreover, some GIFT-Eval configurations offer relatively few historical windows; the benchmark-level discussion in Section D.1 and the BOOM evidence-bucket analysis in Figure 4c, where gains grow with the number of backtest windows, show why history length is relevant, without isolating it from domain effects. The benchmarks also aggregate differently: GIFT-Eval reports a geometric mean of ratios to seasonal-naive, whereas fev-bench reports raw arithmetic means, which weight tasks with large errors more heavily, so relative gains are not directly comparable across benchmarks.

We observe substantially larger corrected BGAA gains on fev-bench (1.3–5.0%) and BOOM (1.7–5.9%) (Table 1). We therefore do not regard the GIFT-Eval gains as representative of all evaluation settings. We also find the compute comparison meaningful: on GIFT-Eval, high-budget TiRex-2 + BGAA achieves 0.6945 MASE at ~89 PFLOPs, versus 0.6956 at ~121 PFLOPs for Toto-2-2.5B (Figure 2). This is a MASE/estimated-FLOPs comparison, not a claim of latency dominance.

4. Does BGAA remain effective at longer horizons?

We added a direct evaluation for long-horizon forecasting on ETTh1, ETTh2, ETTm1, ETTm2, and Weather, the standard benchmarks for the task, with fixed context length 2,048, identical forecast origins, and horizons H = 96, 192, 336, 720 (Section D.2, Table 21). Errors use training-split channel normalization.

Table R5. Mean normalized error across the five datasets. Parentheses show BGAA improvement versus base (%); positive is better.

ModelHMSE: base → BGAA (gain)MAE: base → BGAA (gain)
Chronos-2960.2405 → 0.2362 (+1.79%)0.2723 → 0.2707 (+0.59%)
1920.2920 → 0.2870 (+1.71%)0.3105 → 0.3083 (+0.71%)
3360.3328 → 0.3232 (+2.88%)0.3402 → 0.3371 (+0.91%)
7200.3901 → 0.3863 (+0.97%)0.3827 → 0.3815 (+0.31%)
Toto-2-313M960.2386 → 0.2392 (−0.25%)0.2673 → 0.2677 (−0.15%)
1920.2867 → 0.2852 (+0.52%)0.3055 → 0.3049 (+0.20%)
3360.3241 → 0.3139 (+3.15%)0.3354 → 0.3333 (+0.63%)
7200.3830 → 0.3806 (+0.63%)0.3793 → 0.3786 (+0.18%)

Across the 20 dataset–horizon settings per model, BGAA improves both normalized MSE and MAE in 18/20 for Chronos-2 and 15/20 for Toto-2-313M. Both models continue to improve at H=720, but gains are not monotonic, and Toto-2 slightly regresses at H=96. We also use these tests to complement Table 12, which separately evaluates how performance changes when the historical evidence is taken up to 4H further into the past.

We thank the reviewer again for these suggestions, which have helped us characterize both the strengths and limitations of historical verification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Clarification Regarding Experimental Results

We would like to bring an issue to the reviewers' and the Area Chair's attention that we identified while preparing our rebuttal.

While preparing our rebuttal, we identified an edge case in our evaluation code. In a subset of configurations, the gate rolled forward across a few evaluation windows, so these test windows contributed to view scoring. While we averaged view selection across all backtest windows, this error led to overly optimistic view selection for those configurations. The majority of the configurations were unaffected. We fixed the issue, validated it with additional tests, reran the affected experiments, and updated the corresponding tables and figures. In the corrected evaluation, view selection uses only information available before the first evaluation window, without access to any future observations, and all new rebuttal experiments use the corrected pipeline.

The updated aggregate MASE improvements are:

  • GIFT-Eval: 0.45–0.97% (previously 0.5–1.14%)
  • fev-bench: 1.3–5.0% (previously 3.2–5.9%)
  • BOOM: 1.7–5.9% (previously 1.9–5.1%)

The changes are primarily concentrated in the main benchmark results and the per-configuration regression statistics. We have also verified the remaining experiments and updated the affected tables and figures accordingly.

Importantly, none of our central conclusions change. BGAA continues to improve all 12 model–benchmark combinations, and our key Pareto-frontier findings remain intact. This includes the results showing that smaller models augmented with TTC can outperform larger models, both within and across model families. The proposed methodology and its underlying motivation are also unchanged. We believe the corrected results continue to substantiate the central strengths highlighted by the reviewers, including the novelty of historical verification, its applicability across diverse frozen TSFMs, and the demonstrated accuracy–compute trade-offs.

We apologize for this oversight, particularly because the original results formed part of the evidence on which the reviewers based their assessments. We recognize the time and effort the reviewers have invested in carefully evaluating our work, and we deeply regret that this correction requires them to reconsider some of the numerical evidence. We recognize the importance of reporting accurate experimental results and take full responsibility for the error.

We believe our central contribution remains significant: to our knowledge, BGAA introduces the first training-free, causally grounded verifier for test-time compute in time series foundation model forecasting. We remain confident that this contribution opens an important new direction for inference-time scaling in time series forecasting and could significantly impact future research.

We wanted to be fully transparent about this issue rather than leave the original results uncorrected. We also took this opportunity to substantially strengthen the experimental evaluation with additional analyses addressing the reviewers' concerns, as detailed in our individual responses and revised manuscript. While some numerical results have changed, we believe the corrected results, together with these additional experiments, still provide strong evidence for the main contributions and merits of our work.

We hope that the reviewers and Area Chair will consider the corrected evidence and additional experiments when assessing the overall contributions of our work. We deeply regret the error and any confusion or additional effort it may have caused, and sincerely thank them for their time, careful consideration, and understanding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.