- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents an empirical study of test-time scaling for frozen time series foundation models. It investigates three sources of potential improvements: repeated sampling, multi-view inference through input transformations, and history-based selection and aggregation. Experiments on four TSFMs across eight datasets show diminishing returns from repeated sampling and model-dependent benefits from multi-view aggregation, while highlighting the importance of strong raw-input baselines, historical selection reliability, and inference costs.
Critical strengths
Well-designed controls that help identify the sources of performance improvements.
The paper goes beyond comparing multi-view inference with a single raw-input baseline. By introducing historically selected raw configurations, raw-input ensembles, and matched residual-predictor controls, it provides a more rigorous assessment of the incremental benefits of input transformations and aggregation, and is valuable for identifying potentially misleading improvements in prior or future studies.
Comprehensive empirical analysis with informative negative findings.
The paper examines multiple aspects of test-time inference, including sampling saturation, input transformations, aggregation strategies, historical selection, candidate-pool expansion, and computational costs. It reports both positive and negative results rather than focusing exclusively on favorable comparisons. In particular, the observations that historical improvements may not transfer to test data and that expanding the candidate pool can degrade performance provide useful practical insights into the limitations of test-time inference strategies.
Critical weaknesses
Limited novelty and insufficient new insights beyond established principles.
Although systematically evaluating these phenomena in TSFMs has empirical value, the paper does not establish sufficiently distinctive insights into why or when test-time scaling is effective. For example, it does not identify model or data characteristics that explain the effectiveness of multi-view inference, nor does it establish generalizable relationships between additional inference computation and forecasting improvements.
Insufficient attribution and limited generalizability of the main positive results.
The strongest positive evidence is concentrated in Moirai, which achieves a 7.42% average MAE reduction over raw top-k ensembles on 48 additional series. The remaining models show considerably smaller or inconsistent improvements. Besides, the relationship between performance improvements and additional test-time computation is not fully characterized. Different sampling budgets lack directly measured runtime comparisons, and a substantial portion of the multi-view overhead comes from historical calibration rather than online prediction.
Questions for authors
Explain the gains from multi-view aggregation while the individual transformed view families generally underperform the raw baseline.
Table 8 shows that none of the three transformed view families improves average MAE over selected raw on Moirai's 48 additional series, yet their joint top-3 ensemble achieves substantial improvements. More analysis is needed to establish whether the gains arise from specific complementary representations or from generic ensembling effects.
5 comments
Response to Reviewer SHJ7 (2/3)
... the paper does not establish sufficiently distinctive insights into why or when test-time scaling is effective. ... model or data characteristics that explain the effectiveness of multi-view inference ...
Response: The findings connect the representation supplied to a model with the forecasts worth retaining. Structured views provide alternative forecasts even for direct-quantile models, whose repeated calls on an unchanged input produce no new candidates. On eight Chronos-2 KDD series with two forecast origins, aligned cycle reduces MAE by 9.50% versus raw and 5.26% versus a half-cycle shift, while centering and scale matching have approximately zero effect. This control identifies a benefit from aligned temporal structure in that panel (Figure 2; Appendix E.4).
Compact aggregation makes the gains more consistent across models. With the same single-selection rule applied to multi-view and raw-only pools, aggregate MAE improves in three of seven models; with top-3 applied to both pools, it improves in all seven (Table 1(b)). Within the multi-view pool, top-3 also has lower aggregate MAE than single selection and full-pool averaging in every model, with overall reductions of 1.90% and 3.22%, respectively. The results support generating structured candidates and combining a few that perform well historically.
For a new model and dataset, we use historical trials to determine which views help that model at the intended horizon and loss. Table 4 provides a concrete example: on the same Solar source, eight pilot series select raw+cycle for Chronos-2, raw+trend for TimesFM, and raw only for Moirai-2. The chosen pools are evaluated on 56 disjoint series per model. Chronos-2 retains a 15.08% MAE reduction versus default, close to the full pool's 15.16%. This model–view profile gives a practical choice based on measured forecasting performance; seasonality alone does not determine the choice, as the M_DENSE controls demonstrate. Sections 3.2 and 5 describe how to select candidates on earlier history, validate their aggregate on later history, and retain a compact pool when its accuracy and cost justify reuse.
The strongest positive evidence is concentrated in Moirai ... The remaining models show considerably smaller or inconsistent improvements.
Response: The expanded evaluation shows gains beyond Moirai-1. We evaluate seven models on all 97 GIFT-Eval tasks, including Chronos-2, Moirai-2, and Toto-2. All seven improve aggregate MAE and MASE over historically selected raw top-3. Removing Moirai-1 also preserves aggregate MAE gains over both raw references:
| Evaluation | Model–task pairs | MAE reduction vs original default | MAE reduction vs raw-only top-3 |
|---|---|---|---|
| All seven models | 679 | 2.10% | 4.76% |
| Excluding Moirai-1 | 582 | 0.56% | 3.40% |
Reductions use geometric means of paired task-error ratios. The default reference measures improvement over the designated inference configuration; raw-only top-3 controls for historical context selection and ensembling. Gains vary across models and tasks, and TimesFM and Toto remain slightly worse than default in aggregate MAE. Section 4.1/Tables 1–2 and Appendix E.2 report both references across the full benchmark.
Response to Reviewer SHJ7 (3/3)
... Different sampling budgets lack directly measured runtime comparisons, and a substantial portion of the multi-view overhead comes from historical calibration rather than online prediction.
Response: We now measure sampling budgets directly and include pilot, calibration, and online prediction costs when evaluating compact view pools. For Moirai, independent N=20 and N=128 calls use fixed historical patch choices, three seeds, and seven source panels. Increasing the budget reduces median-source MAE by 4.24% using interpolated medians, with online runtime ratios of 0.996–1.027. Patch-discovery cost is reported separately. Sampling can therefore remain useful at low marginal cost in this implementation; the original sub-0.6% finding applies specifically to N=64→128 on the eight original OT series (Appendix E.7).
For multi-view inference, a compact pool can retain the accuracy gain while reducing the measured calibration and prediction time, including its pilot cost. We execute a Solar workload with 56 evaluation series and 19 forecast origins per model:
| Model | Chosen pool | MAE reduction vs default: full → compact | Time in seconds: full → pilot + compact |
|---|---|---|---|
| Chronos-2 | Raw + cycle | 15.16% → 15.08% | 86.33 → 67.50 |
| Moirai-2 | Raw only | 0.17% → 0.29% | 79.58 → 58.01 |
These are time reductions of 21.80% and 27.11% relative to full-pool inference. Online prediction alone also becomes faster, by 19.06% and 14.95%, respectively. Moirai-2 saves work by pruning transformed candidates; Chronos-2 retains the useful cycle view. Comparisons use the same GPU within each model and include transformations and online aggregation, excluding loading, warm-up, data reading, and selector CPU work (Section 4.5/Figure 3; Appendix E.5).
Across the broader five-model, seven-source profiling study, historical pool selection retains 84.6% of the full pool's MAE point-estimate gain with 17.4% fewer prediction branches, including pilot and calibration work. These branch counts are distinct from the measured runtimes above. Fixed raw+cycle achieves similar aggregate accuracy with fewer branches, and the source-bootstrap intervals for both full and historically chosen pools versus default include zero (Table 3; Appendix E.5).
The smaller Solar workload, with eight series and two origins, incurs higher total time because of pilot overhead. The two workloads show why the intended amount of reuse matters: a pool should be validated for accuracy and evaluated on the combined pilot, calibration, and forecasting cost of its intended workload.
Updated paper. We have updated the paper, with key additions highlighted in yellow. The full benchmark data and results are available in the linked anonymous repository.
We hope these clarifications and revisions address your concerns, and we would be happy to discuss any remaining questions.
Response to Reviewer SHJ7 (1/3)
Thank you for recognizing the value of our controlled comparisons and for asking what drives the multi-view gains. Our framework changes the temporal representation presented to a frozen TSFM, then uses historical forecasts to select and combine useful candidates. The updated paper connects this design to three findings: cycle views supply better candidates in the Moirai study, compact aggregation improves accuracy across seven models, and historical view selection can retain most of the gain while reducing prediction work.
Response: Cycle views supply useful candidates, and historical selection plus compact aggregation improves on raw-input ensembling. On the original 48 additional Moirai series, selecting just one forecast from raw+cycle reduces MAE by 4.52% relative to the historically selected raw top-k Median ensemble. Combining the top three candidates increases this reduction to 7.67%. The gain is already present under single selection and increases with aggregation.
The original family table restricts every series to forecasts from one transformed family. The mixed pool keeps raw forecasts available and lets each series select candidates using its own history. To identify which views contribute, we compare the following pools on those same 48 series, matching historical scoring, Median aggregation, and available ensemble size. Reductions average per-series relative MAE changes with equal dataset weights; positive values indicate improvement.
Removing cycle reduces the gain from 7.42% to 0.42%, while raw+cycle retains the full pool's improvement. Keeping raw available matters: forcing a cycle-only winner increases MAE by 0.88% against the same raw reference. These ablations identify the value of cycle candidates under matched aggregation; pool sizes and calibration work can differ.
Cycle forecasts appear in 46/48 selected sets, and 24/48 sets contain only cycle candidates with different context lengths or periods. The Median ensemble outperforms every selected constituent on 25/48 series. Useful alternatives therefore also arise within a view family.
The Mean-ensemble diagnostic gives a further answer to the attribution question: better selected members dominate the improvement. Relative to the matched raw Mean ensemble, improved member quality contributes 21.3 percentage points, while weaker error cancellation offsets 5.4 points, yielding a net 15.9% MAE reduction after rounding. These numbers concern Mean aggregation; the 7.42% result above uses Median. Sections 4.2–4.3/Figure 2 and Appendix E.3 report these controls.