- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ECHO is a black-box forecast-correction framework for frozen time-series foundation models. It combines two sources of historical evidence: AnalogE retrieves context–future analogues and learns a gated correction toward a historical template, while RecentE learns residual corrections from past forecasting errors and updates them as fully observed feedback becomes available. TempRA combines the two branches using their historical performance over multiple temporal views. The evaluation covers five foundation models and nine datasets, alongside long-horizon forecasting and component analyses. The principal strength is the consistency of the MSE improvements under a carefully specified evaluation protocol. The main questions concern the incremental value of the framework’s complexity and the scope of its benefits across error metrics.
Critical strengths
- Broad and consistent empirical gains. ECHO improves MSE across all 45 backbone–dataset settings, supported by long-horizon and multi-seed evaluations.
- Practical design and rigorous evaluation. Black-box access makes the framework broadly applicable, while fully matured feedback and matched-input baselines strengthen the credibility of the results.
Critical weaknesses
- Complexity exceeds the demonstrated marginal gains. Dynamic aggregation and online updates contribute relatively small improvements, leaving it unclear whether a simpler correction pipeline could achieve most of the benefit.
- Benefits are less consistent beyond MSE. MAE improvements are smaller and sometimes negative, while RecentE alone can degrade forecasts, limiting claims of uniformly reliable correction.
Questions for authors
- How much of the overall gain remains with fixed fusion and a plain residual regressor, without online updates or robust reweighting?
- Can the headline results present MAE alongside MSE and explicitly distinguish point-forecast improvements from probabilistic forecasting quality?
- Does TempRA reliably down-weight RecentE in settings where that branch underperforms the backbone?
- What are the complete backbone-specific pretraining and deployment costs, including upstream forecasts, retrieval encoding, and memory requirements?
- Can you compare full ECHO against strong end-to-end black-box adaptation baselines under matched inputs, tuning budgets, and causal feedback availability, rather than primarily comparing individual branches?
- Can you extend these competing-method comparisons across additional foundation-model backbones and datasets to determine whether ECHO’s advantage generalizes beyond the current comparison subset?
3 comments
Thank you for this thorough rebuttal and the substantial additional experiments. The complete method comparison against ELF and ORCA on all nine datasets with two backbones was exactly what was missing, and ECHO's lowest macro MSE and MAE there meaningfully strengthens the framework-level claim. I also appreciate the concrete cost accounting and the Toto-2 extension, which answer the practical-deployment questions I raised.
But two concerns are not fully resolved by the new evidence. First, on complexity versus marginal benefit: your own simplification answer shows that on ETTh1 a fixed fusion with a plain ridge regressor is essentially as good as full ECHO (0.342 versus 0.346), so the elaborate dynamic aggregation and online-update machinery is only clearly justified on weight-sensitive datasets such as Exchange and PEMS08. This leaves my concern that a much simpler pipeline recovers most of the gain, and I would ask that the simplified baseline be reported as a full row in the main comparison rather than for a few datasets. Second, on the MSE–MAE asymmetry: the long-horizon macro MAE improvements are only -0.63% to -1.33%, and Solar's MAE worsens at every horizon. Clarifying that ECHO targets point forecasts does not address this; it suggests the gains are concentrated in suppressing large errors rather than uniformly improving point accuracy, which is an important limitation for a correction method and should be stated explicitly, ideally with MAE reported as a first-class headline metric.
On balance, the rebuttal resolves my missing-comparison and cost questions well, but my two central weaknesses stand. I maintain my rating of weak reject, and would only reconsider if you can further substantiate that the aggregation and online machinery are necessary beyond a few datasets and that the benefit is not specific to MSE. Thank you again for the careful and constructive response.
Thank you for your careful review and further feedback. We acknowledge that the benefits of dynamic aggregation and online updates are dataset-dependent, and the current evidence does not establish their necessity across all datasets. We also recognize that ECHO’s strengths lie primarily in reducing large forecast errors. These remain limitations of ECHO. In future work, we will further investigate when dynamic fusion can deliver consistent gains and how to achieve more consistent improvements in MAE.
Thank you for reading our paper carefully and for your constructive feedback. We address your questions below.
Q1. How much gain remains after simplification?
On TimesFM 2.5, fixed fusion with plain residual regression, without RecentE online updates or robust reweighting, produces different results across datasets.
On ETTh1, the simplified version has slightly lower MSE than ECHO (.342 vs .346). On Exchange, it retains only 33.4% of the MSE gain, while MAE rises from .164 to .170, exceeding the frozen backbone's .168. On PEMS08, it retains 91.9% of the MSE gain but only 66.2% of the MAE gain.
These results indicate that the dynamic model helps preserve gains on datasets sensitive to fusion weights, such as Exchange and PEMS08, where simplification can sacrifice a substantial portion of the benefit.
Q2. Point versus probabilistic forecasting
ECHO is designed to improve point forecasts. For quantile-based backbones such as Chronos-2, we use the median (q50) as the point forecast. Both MSE and MAE therefore measure point-forecast accuracy.
Q3. Does TempRA down-weight RecentE when it underperforms?
We define the recent loss difference as ΔL=LRecentE−Lbackbone. RecentE is classified as recently worse when ΔL>0 and no worse when ΔL≤0. The table reports its average weight in each state. Weights lie in [0,1], with lower values indicating smaller contributions to the final forecast.
Across all nine datasets, RecentE receives a lower average weight when its recent performance is worse.
Q4. Training and deployment costs
Overall, ECHO maintains a low computational overhead across both offline training and downstream deployment, with one-time backbone-specific training and efficient inference-time adaptation. We measured TimesFM 2.5 and Chronos-2 on the same H200. AnalogE is trained once per backbone on 50k samples for three epochs and reused across downstream datasets. The retrieval encoder and candidate bank can be shared across backbones.
Training uses batch size 32. We timed 30 steps after five warm-up steps, including upstream frozen-backbone forecasts, AnalogE forward/backward passes, and optimizer updates.
Deployment uses ETTh1, L=512, H=64, and K=20. Latency includes the backbone forecast, query encoding, exact retrieval, case loading, AnalogE, RecentE, and TempRA. Matured-feedback processing and refitting add approximately 0.15 ms per variate per origin, amortized.
ETTh1 candidate vectors occupy 165.40 MiB, and retained RecentE training features occupy approximately 849.37 MiB. These total approximately 0.991 GiB, separate from model weight storage.
Q5. Complete-method comparisons
We compared complete ELF, ORCA, and ECHO on all nine datasets using Chronos-2 and TimesFM 2.5 (L=512, H=64). ECHO results come from the main evaluation. Macro is the unweighted nine-dataset mean. Wins counts first places using unrounded results.
ECHO achieves the lowest macro MSE and MAE on both backbones, ranking first in MSE in 15 of 18 backbone–dataset settings and in MAE in 10 of 18.
Q6. Extension to additional backbones
We evaluated ECHO on Toto-2 across six datasets. Each cell reports MSE / MAE. The lower value for each metric is bolded based on the unrounded results.
ECHO reduces MSE on all six datasets and MAE on four. MAE increases slightly on ETTh1 and ETTh2.
We hope these additions address your concerns and would appreciate your reconsideration of the score. Thank you for your constructive feedback.