- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose ResRAF, a framework that uses historical forecast errors in a retrieval-augmented setting to correct the predictions of existing forecasting models. The authors empirically demonstrate that applying the framework to several supervised models and a time-series foundation model improves forecasting accuracy.
Critical strengths
1. Easy to use. The framework is applied post hoc. It is evaluated on a diverse set of backbones (linear, Transformer-based, retrieval-based, multiscale), including a TSFM.
2. Sensible ablations. The ablations are informative: random retrieval removes almost all of the gain, dropping the residual similarity term removes most of it, and state-adaptive calibration mainly reduces the number of degraded tasks. I also appreciate that the authors report the median improvement, the number of degraded tasks, and worst-case performance rather than only averages, and that they specify which data each fitting stage uses.
Critical weaknesses
1. Limited methodological novelty for a complex pipeline. Unfortunately, the central idea seems to have substantial overlap with existing literature. Retrieving historical forecasts that resemble the current forecast and using their corresponding observations or errors for post-processing is well established in analog-based weather forecasting (Delle Monache et al., 2011; 2013). Nearest-neighbor methods for updating existing forecasts have also been studied in hydrology (Brath et al., 2002), while regression-based and regime-dependent forecast calibration have a long history (Glahn & Lowry, 1972; Allen et al., 2019). More directly, ResMem (Yang et al., 2023) already proposes fitting a nearest-neighbor regressor to the residuals of a base predictor and adding the estimated residual to its output. Although ResMem was primarily evaluated outside time-series forecasting, the underlying residual-correction formulation is particularly close to ResRAF.
I expected a more detailed discussion of this literature. The predicted residual representation as an additional retrieval key and the state-dependent correction coefficients may be useful contributions, but relative to these established ideas, the methodological novelty appears rather incremental. At the same time, the framework combines randomized SVD, a CNN-based residual predictor, K-means clustering over 78 handcrafted features, a clipped ridge regression fit, and several manually chosen constants. While each component may be motivated individually, I am not convinced that the reported performance gains justify the complexity of the complete pipeline. Moreover, there is no analysis of fitting time, memory consumption, or additional inference latency. This is especially important for a retrieval-based post-processing method: demonstrating low computational overhead would substantially strengthen its practical contribution and help justify the added complexity.
2. Limited experimental evaluation. The evaluation is restricted to six relatively low-dimensional datasets (7–21 channels), without experiments on higher-dimensional benchmarks such as Electricity or Traffic. I am also missing stronger comparisons that isolate the contributions of the proposed retrieval mechanism. The paper mainly compares its retrieval score against random retrieval and forecast-shape similarity alone. Random retrieval is a useful sanity check, but it is not a particularly competitive alternative.
There are several informative alternatives that are not evaluated: retrieval based on input-window similarity, as in RAFT (Han et al., 2025); retrieval in the frozen forecaster's representation space; simpler nearest-neighbor residual correction, as in ResMem; and stationarity-aware retrieval strategies, as in SARAF (Zhou et al., 2026). These methods are not all directly interchangeable with ResRAF, but comparisons or controlled adaptations would help establish which parts of the proposed retrieval mechanism are necessary.
In particular, there is no baseline that directly decodes the predicted residual representation into a forecast correction without retrieval. Such a comparison would clarify whether the retrieval stage provides additional value beyond the learned residual predictor itself.
I am also concerned by the performance of the two included correction baselines. UEC-STD reportedly increases MSE by approximately 4%–18% on ETTh1, while PIR increases MSE by up to 21.7% on Exchange. These degradations raise questions about the comparability of their configurations and tuning budgets.
Finally, ResRAF fits additional components using target-domain data, so the resulting Chronos-based system is not strictly zero-shot, even though the underlying Chronos model remains frozen. A comparison with lightweight target-domain adaptation or parameter-efficient fine-tuning would therefore be helpful. A simpler alternative would also be to retrieve similar historical samples and oversample them during fine-tuning of the model or its prediction head. SOLID (Chen et al., 2024) follows a related strategy by selecting contextually similar historical samples and adapting a model's prediction layer, but it is not compared. Such an experiment would help determine whether the proposed multi-stage correction pipeline offers a meaningful advantage over more direct adaptation.
Questions for authors
Clarification requests
-
How does ResRAF compare with a simple ResMem-style nearest-neighbor residual corrector, alternative retrieval keys (e.g., input-window or backbone-representation similarity), and direct decoding of the predicted residual representation without retrieval? I am particularly interested in whether the additional residual-similarity retrieval stage improves over a direct learned correction.
-
Have the authors considered selective fine-tuning on retrieved historical samples, as in SOLID, as a simpler alternative? For the Chronos experiments, how would ResRAF compare against lightweight target-domain adaptation or parameter-efficient fine-tuning?
-
How were UEC-STD and PIR configured and tuned? Can the authors explain their relatively large degradations on ETTh1 and Exchange, and whether comparable tuning budgets were used?
-
How are historical residuals in the retrieval memory generated? In particular, are they computed from in-sample predictions or out-of-sample/rolling-origin forecasts? How is it ensured that both the retrieved samples and their corresponding future labels are available at each evaluation forecast origin?
-
How do fitting time, memory requirements, and inference latency scale with the size of the retrieval memory and the number of channels? Results on higher-dimensional datasets such as Electricity or Traffic would be valuable.
Suggestions for presentation
- With , the retrieval weights in Equation 9 appear to be approximately uniform, assuming the similarity scores have moderate magnitude. Could the authors clarify whether this weighting has a meaningful numerical effect? If not, a plain average could simplify the method description.
References
- Allen, Ferro & Kwasniok (2019). Regime-dependent statistical post-processing of ensemble forecasts. Quarterly Journal of the Royal Meteorological Society.
- Brath, Montanari & Toth (2002). Neural networks and non-parametric methods for improving real-time flood forecasting through conceptual hydrological models. Hydrology and Earth System Sciences.
- Chen et al. (2024). Calibration of Time-Series Forecasting: Detecting and Adapting Context-Driven Distribution Shift (SOLID). KDD.
- Delle Monache et al. (2011). Kalman Filter and Analog Schemes to Postprocess Numerical Weather Predictions. Monthly Weather Review.
- Delle Monache et al. (2013). Probabilistic Weather Prediction with an Analog Ensemble. Monthly Weather Review.
- Glahn & Lowry (1972). The Use of Model Output Statistics (MOS) in Objective Weather Forecasting. Journal of Applied Meteorology.
- Han et al. (2025). Retrieval Augmented Time Series Forecasting (RAFT). ICML.
- Yang et al. (2023). ResMem: Learn what you can and memorize the rest. NeurIPS.
- Zhou et al. (2026). Stationarity-Aware Retrieval-Augmented Time Series Forecasting (SARAF).
9 comments
Q2: Have the authors considered selective fine-tuning on retrieved historical samples, as in SOLID, as a simpler alternative? For the Chronos experiments, how would ResRAF compare against lightweight target-domain adaptation or parameter-efficient fine-tuning?
A2: We appreciate the reviewer's suggestion to compare ResRAF with selective fine-tuning and lightweight target-domain adaptation. Inspired by your feedback, we conducted additional experiments with SOLID and Head-only FT on Chronos-Bolt-Small across 6 datasets at a prediction length of 96. For SOLID, we follow its similarity-based retrieval and sample-wise gradient updates, using a causal online pool where retrieved targets are fully observed before each forecast origin. For Head-only FT, we fine-tune the output module (5.29% of Chronos parameters) on the target-domain training set. All methods use the same pretrained checkpoint, data splits, context length (512), and validation-based hyperparameter selection. The results are summarized in Table B.
Table B: Relative MSE/MAE Changes on Chronos-Bolt-Small at a Prediction Length of 96. Negative values indicate improvements over the original Chronos predictions.
| Dataset | Metric | ResRAF(Ours) | SOLID | Head-only FT |
|---|---|---|---|---|
| ETTh1 | MSE | -1.02% | -3.31% | -3.82% |
| MAE | +0.79% | -1.05% | -0.52% | |
| ETTh2 | MSE | -1.06% | 0.00% | -5.32% |
| MAE | +0.62% | 0.00% | +0.31% | |
| ETTm1 | MSE | -7.62% | -0.61% | -3.05% |
| MAE | -1.81% | +0.30% | -0.90% | |
| ETTm2 | MSE | -1.72% | 0.00% | -1.15% |
| MAE | 0.00% | 0.00% | 0.00% | |
| Exchange | MSE | -9.20% | +1.15% | +2.30% |
| MAE | -4.35% | 0.00% | +1.45% | |
| Weather | MSE | -11.05% | -1.16% | -6.98% |
| MAE | -8.25% | -0.49% | -0.97% | |
| Average | MSE | -5.28% | -0.66% | -3.00% |
| MAE | -2.17% | -0.21% | -0.11% |
As shown in Table B, our ResRAF improves MSE on all 6 datasets and achieves the largest average reductions in both MSE (5.28%) and MAE (2.17%), demonstrating its consistent effectiveness across different forecasting scenarios.
Two observations further support the advantage of our approach. (1) SOLID can leverage newly observed samples and perform query-specific gradient updates during testing, whereas our ResRAF relies exclusively on a fixed training-set residual memory. Nevertheless, our ResRAF achieves substantially larger average improvements in both MSE (5.28% vs. 0.66%) and MAE (2.17% vs. 0.21%). This shows that our ResRAF achieves stronger improvements without relying on online adaptation. (2) Although Head-only FT outperforms our ResRAF in terms of MSE on ETTh1 and ETTh2, its effectiveness does not consistently extend across datasets. On Exchange, both Head-only FT and SOLID exhibit negative transfer, increasing MSE by 2.30% and 1.15%, respectively, whereas our ResRAF reduces MSE by 9.20%. This demonstrates the robustness of our ResRAF under distribution shifts, where parameter-based adaptation may suffer from negative transfer.
We will incorporate these additional comparisons and discussions into the revised manuscript.
Q3: How were UEC-STD and PIR configured and tuned? Can the authors explain their relatively large degradations on ETTh1 and Exchange, and whether comparable tuning budgets were used?
A3 (part 1): We sincerely thank the reviewer for raising these important concerns about the fairness of our baseline comparisons. We carefully configure and tune UEC-STD and PIR with reference to their original implementations. Below, we clarify the tuning procedures and provide additional evidence to explain the observed degradations.
Configurations and tuning. For UEC-STD , to ensure a fair comparison, we adapt it from autoregressive to direct multi-step forecasting. Its original autoregressive rollout would produce different base forecasts and introduce additional error accumulation, making it difficult to isolate the effects of the correction methods. We therefore apply UEC-STD to the same frozen forecasts as ResRAF, while preserving its original trend–seasonal decomposition, correction architecture, training objective, and hyperparameter settings. In particular, its correction strength is selected independently for each task from the original grid using the prescribed balanced validation pool. For PIR , we use from the original paper's search space . Additionally, all three retrieval sizes are evaluated, with independently selected based on validation loss for each of the 120 tasks (5 backbones, 6 datasets, and 4 horizons). As shown in Table C, performs best among the three fixed settings on 75/120 tasks and achieves the lowest average MSE degradation. However, validation-based tuning still fails to prevent negative transfer on Exchange. For our proposed ResRAF , we similarly tune the retrieval size and regularization coefficient on validation data. Although the search spaces differ across methods, all hyperparameters are selected using validation data, and no test labels are used for tuning.
Table C: PIR Retrieval-Size Comparison (Relative MSE Change).
| Dataset | K=10 | K=20 | K=50 | Tuned K | ResRAF(Ours) |
|---|---|---|---|---|---|
| ETTh1 | +0.87% | -0.06% | -0.16% | +0.36% | -1.47% |
| ETTh2 | +2.81% | +1.29% | -0.13% | -0.24% | -1.71% |
| ETTm1 | -0.13% | -0.50% | -0.56% | -0.75% | -2.04% |
| ETTm2 | +4.88% | +3.20% | +3.58% | +2.25% | -2.46% |
| Exchange | +70.01% | +23.75% | +9.36% | +25.29% | -10.17% |
| Weather | +2.71% | +0.19% | -0.76% | -0.69% | -5.83% |
| Average | +13.52% | +4.65% | +1.89% | +4.37% | -3.95% |
Note: Results report average relative MSE changes across 5 backbones and 4 horizons. Negative values indicate improvement. Tuned K is selected per backbone–dataset–horizon combination using validation loss. Bold indicates the best result.
Q1: How does ResRAF compare with a simple ResMem-style nearest-neighbor residual corrector, alternative retrieval keys such as input-window or backbone-representation similarity, and direct decoding of the predicted residual representation without retrieval? In particular, does residual-aware retrieval improve on a direct learned correction?
A1: We thank the reviewer for suggesting these comparisons. We evaluate ResMem, three alternative retrieval strategies, and direct decoding with and without SARC on iTransformer across six datasets and four prediction horizons, as summarized in Table A.
Our ResRAF combines residual-aware retrieval (RAR) with state-adaptive residual calibration (SARC). To assess RAR, we replace its retrieval strategy with backbone-representation similarity (ResMem-style + SARC), input-window similarity (RAFT-style + SARC), or SARAF-inspired stationarity and temporal-structure criteria (SARAF-style + SARC), while keeping the aggregation and SARC unchanged. We also compare with the original ResMem, which retrieves historical residuals using backbone representations and applies distance-weighted correction. To assess the value of retrieval itself, Direct uses the same trained residual-representation predictor as ResRAF but decodes its output into a correction without retrieval. Direct + SARC further calibrates this correction.
Table A. Residual correction and retrieval alternatives on iTransformer: relative MSE change.
Note: Each dataset result averages relative MSE changes across four prediction horizons. Negative values indicate improvement over the same frozen iTransformer forecast. Bold indicates the best result in each row.
Our ResRAF achieves the best overall performance among the alternative retrieval strategies, with an average ΔMSE of −3.71%, compared with −1.60% for the strongest alternative (SARAF-style + SARC). This highlights the importance of retrieval criteria and supports the effectiveness of our residual-aware retrieval (RAR). Furthermore, comparing ResMem with ResMem-style + SARC shows that aggregation and calibration help mitigate negative transfer. The average ΔMSE improves from +0.96% to −1.21%, while the Exchange degradation drops from +13.42% to +1.58%. However, ResRAF still outperforms ResMem-style + SARC, indicating that improved aggregation and calibration alone cannot fully compensate for less effective retrieval. Finally, compared with Direct and Direct + SARC, ResRAF achieves better average results on five of the six datasets. This supports the benefit of retrieving historical residuals over directly decoding predicted residual representations.
Implementation details. All methods share the same frozen iTransformer, cached base forecasts, and training and validation splits. For backbone-representation retrieval, we extract the first C variable tokens after the encoder's final LayerNorm and before projection. The original ResMem selects its retrieval size and distance-weighting temperature based on validation MSE. For direct correction, we freeze ResRAF's trained residual-representation predictor and train an MLP decoder to predict the full H-step residual, selecting its checkpoint using validation MSE. Direct + SARC additionally fits calibration coefficients using training data.
We will incorporate these additional comparisons and the corresponding analysis into the revised manuscript.