acceptodds
Under review as a conference paper at ICLR 2027

Past Errors, Better Forecasts: Memory-Guided Calibration for Frozen Time Series Predictors

Abstract

Time series forecasting supports decision making in many data-intensive applications, where forecasting models typically use a fixed-length historical window to predict a future sequence. In standard forecasting pipelines, a single checkpoint is typically selected according to aggregate validation performance, yet strong aggregate performance does not guarantee uniformly small errors across samples, variables, or temporal contexts. Consequently, a frozen forecaster may exhibit structured, input-dependent residual patterns on historical labeled windows, which can provide reusable evidence of how the same forecaster tends to deviate under similar temporal conditions. Motivated by this observation, we propose Memory-Guided Test-Time Calibration (TTC), a non-parametric residual-memory framework that calibrates frozen time-series forecasters at the prediction level. After model selection, TTC constructs an external residual memory that associates compact variable-wise representations of historical look-back windows with their corresponding true-minus-predicted residual trajectories. At inference time, the current look-back window is used to retrieve historically similar variable patterns, and their residual trajectories are similarity-weighted and aggregated to compensate the base prediction. To reduce harmful over-correction, TTC applies a correction-ratio fallback that preserves the base prediction when the relative magnitude of the candidate correction is excessively large. The forecasting backbone remains frozen throughout calibration, and TTC never accesses the future labels of the current test window. Experiments across eight benchmark datasets and five heterogeneous forecasting backbones, together with comparisons against representative test-time adaptation methods and extensive ablation and sensitivity analyses, demonstrate the effectiveness and robustness of TTC across diverse forecasting settings.

Primary area:
learning on time series and dynamical systems
Keywords:
long-term time series forecasting, test-time calibration, residual memory, retrieval-based residual calibration, prediction-level calibration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_h22DOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Memory-Guided Test-Time Calibration (TTC), a prediction-level calibration framework for frozen time-series forecasters. The work is motivated by the observation that forecasting models selected according to aggregate validation performance may still exhibit structured, context-dependent residual patterns that provide useful information for subsequent predictions. TTC constructs an external memory that associates variable-wise representations of historical look-back windows with their corresponding forecasting residual trajectories. At test time, the method retrieves relevant historical patterns and aggregates their residuals to estimate a local correction, with a conservative fallback mechanism to limit excessively large adjustments. The framework is evaluated across eight benchmark datasets, five heterogeneous forecasting backbones, and multiple prediction horizons. The paper also includes comparisons with representative test-time adaptation methods, ablation and sensitivity studies, and evaluations of computational efficiency and scalability.

Critical strengths
  1. TTC uses historical forecasting residuals to correct subsequent predictions. The variable-wise retrieval mechanism selects relevant historical residuals for each variable, and the retrieved residuals are aggregated to calibrate the current forecast. The method provides a practical approach to improving the predictions of frozen forecasting models.
  2. TTC introduces an external calibration mechanism that can be integrated with existing forecasting models. Its variable-wise retrieval representation is constructed directly from observed temporal patterns, enabling a common calibration procedure across heterogeneous forecasting architectures. The combination of context-dependent retrieval, similarity-weighted residual aggregation, and conservative correction is straightforward and technically coherent.
  3. The paper compares TTC with TAFAS and SOLID using the same validation-selected forecasting checkpoints. The ablation studies examine different residual correction strategies and show the effectiveness of context-aware retrieval and multi-neighbor residual aggregation. The paper also reports parameter sensitivity, inference latency, and memory overhead..
Critical weaknesses
  1. The complete retrieval representation combines statistical summaries, normalized temporal shape, and first-order differences. Although the existing comparison between retrieval representations is informative, it does not fully isolate the contribution of each component. A leave-one-block-out ablation would provide a clearer understanding of their individual roles and complementary.
  2. The default residual memory is constructed from both training and validation windows. While this provides broader historical coverage, a train-only memory comparison would help clarify whether TTC retains its calibration benefits with a more restricted memory source and how the additional validation windows influence performance.
  3. The overall methodology is understandable, but some figures contain relatively dense information, and certain methodological details, particularly the fixed-backbone calibration setting, are explained repeatedly across sections. A more concise presentation and clearer visual hierarchy would make the main ideas and empirical findings easier to follow.
Questions for authors
  1. Could the authors provide a leave-one-block-out ablation of the complete retrieval representation by individually removing statistical summaries, normalized temporal shape, and first-order differences? Results on representative model–dataset settings would help clarify the relative importance of these components.

  2. How does TTC perform when the residual memory is constructed using only the training split? A comparison with the default training+validation memory, while keeping the forecasting checkpoint and other calibration settings unchanged, would help clarify the influence of historical memory coverage on calibration performance.

Reviewer_PwBDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Memory-Guided Test-Time Calibration (TTC) to improve time-series forecasting without updating the parameters of the original forecasting model. The key idea is to retrieve similar historical windows, use their past prediction errors to estimate a correction, and apply a fallback rule when the correction is too large.

The problem is relevant, especially when retraining a forecasting model is expensive or not possible. The paper reports promising results: TTC achieves the best MSE in 36 out of 40 backbone–dataset combinations and the best MAE in 34 out of 40 combinations.

However, I am not fully convinced that the paper establishes a clear literature gap. Context-based forecasting calibration, test-time adaptation, and residual correction have already been studied in related work, including SOLID and TAFAS. The paper's potential contribution is the specific combination of a fixed residual memory, variable-wise retrieval, and a correction-ratio fallback. But the paper does not clearly show why this design is better than other contextual correction methods or simpler residual prediction approaches.

The experiments cover eight datasets, five forecasting backbones, multiple prediction horizons, and several ablations. This is a useful evaluation, but it is not sufficient to fully support the broader claims. Four datasets are related ETT benchmarks, the main ablation studies are concentrated on one dataset and horizon, and the evaluation does not adequately test performance under distribution shift or establish the statistical reliability of the improvements. Overall, the method is promising, but its novelty and generality are not yet sufficiently demonstrated.

Critical strengths
  1. Practical and relevant problem. The method aims to improve existing forecasting models without retraining them. Reusing historical prediction errors is intuitive, relatively simple, and potentially useful in real applications.

  2. Encouraging initial experimental results. The paper evaluates multiple datasets and forecasting models, compares against existing calibration methods, and includes ablations and efficiency measurements. The results suggest that residual memory can be a useful direction for improving forecast accuracy.

Critical weaknesses
  1. The paper does not clearly explain what is fundamentally new compared with existing context-based calibration and residual-correction methods. For example, SOLID and TAFAS study contextual forecasting adaptation, while COSA (ICLR 2026) directly corrects frozen forecasting outputs using a context-aware residual adapter and gating. These works make the general idea of correcting frozen forecasters less novel. The authors need to define their contribution more narrowly and explain why a fixed residual memory provides an important advantage over learned correction methods.

  2. TTC assumes that historical windows with similar input characteristics will have similar future prediction errors. This is plausible, but it is not guaranteed, especially when the data distribution changes over time. The current experiments mainly show that TTC improves final forecasting metrics; they do not clearly establish that the retrieved windows actually contain more useful residual information than random or less-similar windows. Without this analysis, it remains unclear whether the proposed retrieval mechanism is responsible for the gains or whether a simpler correction method could achieve similar results.

  3. The paper compares TTC with TAFAS and SOLID, which is useful, but the comparisons do not fully establish TTC's advantage over other lightweight residual-correction approaches. COSA and δ-Adapter (both ICLR 2026) are particularly relevant because they correct frozen forecasting models at the output level. The paper should compare TTC with the closest available methods, or provide a more detailed and evidence-based explanation of their differences.

  4. The paper evaluates eight datasets and five forecasting backbones, and the results are encouraging. However, four datasets are related ETT benchmarks, and the main ablations are concentrated on a limited setting. The evaluation also does not adequately test what happens when the historical residual memory becomes outdated or when the time series experiences abrupt or gradual distribution shifts. In addition, stronger statistical analysis would help establish whether the improvements are reliable across settings. These limitations make it difficult to conclude that TTC is broadly effective in practical deployment.

Questions for authors
  1. Can the authors clarify the main novelty of TTC compared with SOLID, TAFAS, and other output-space residual-correction methods? A direct comparison would help explain the specific advantage of maintaining a fixed residual memory rather than learning or adapting a correction function.

  2. Can the authors directly test whether similar historical windows have more similar residual trajectories?

  3. Can the authors strengthen the evaluation with residual-regression baselines and distribution-shift tests?

Reviewer_imgROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets residual errors of a frozen time-series forecaster. It proposes Memory-Guided Test-Time Calibration (TTC), which builds a bank of pairs of a look-back representation and the true-minus-predicted residual trajectory from historical labeled windows. At test time, each variable's look-back window is embedded with a hand-crafted representation (mean and standard deviation, standardized shape, standardized first differences). The top-K nearest historical windows are retrieved per variable, and their residual trajectories are softmax-weighted and added to the base forecast with a strength lambda. A correction-ratio gate discards the correction when its norm relative to the forecast exceeds a threshold. No gradients or test labels are used. Evaluation covers different datasets and backbones (iTransformer, PatchTST, DLinear, Autoformer, Crossformer) and over 4 horizons, against TAFAS and SOLID from a shared base checkpoint. TTC gets the lowest horizon-averaged MSE in 36 of 40 backbone/dataset pairs.

Critical strengths
  1. Simple, practical and model-agnostic idea with broad empirical coverage. TTC needs no backbone access and no gradients, and adds only milliseconds of overhead. It shows gains across five architectures and many different settings (Table 1, Appendix C.4), so the benefit is not tied to one model family. This breadth is the main reason the work could be useful in practice.

  2. Table 2 shows that a global mean residual and random-K retrieval give little or negative gain, and that 1-NN is unstable. This supports the claim that input-conditioned multi-neighbor aggregation matters.

Critical weaknesses
  1. The source of the gains is not established, and a simple explanation for it is missing in the current form of the manuscript. The memory is built from training and validation windows with stride 8. The backbone was trained on the training windows, so its residuals there are in-sample and likely smaller and differently distributed than test residuals and this is not properly discussed. The largest gains appear on shifted or non-stationary datasets: Exchange (iTransformer MSE 0.365 to 0.264, Autoformer 0.631 to 0.359), Illness (PatchTST 1.937 to 1.720) and Autoformer-Solar (0.693 to 0.377). That pattern suggests TTC may mostly be correcting a persistent level or bias error of weak or poorly fit base models, rather than learning input-conditioned error structure. The mean-residual and random-K controls are run only on Weather H=96, where gains are small (Table 2).

  2. The related work (Section 2, Appendix A.3) covers only TAFAS and SOLID as time-series test-time adaptation methods. It omits COSA (Im, Jeonghwan, and Hyuk-Yoon Kwon, ICLR 2026) and PETSA (Medeiros et al., 2025). Both correct a frozen forecaster with lightweight gated modules at the input or output. COSA in particular applies a gated residual correction to the base prediction, so the claimed novelty of prediction-level calibration with a conservative gate is overstated without a comparison. Neither method is compared experimentally. SOLID equals Base exactly in many cells, and Autoformer-ETTh1 gives MSE 1.425 against Base 0.488. The paper attributes this to "transferred or manually extended configurations" (Appendix C.4), which weakens the comparison. TTC's own configuration selection is also unclear, and sensitivity plots are shown on test performance. All results are single runs with no variance. This is fixable with added experiments and a corrected related-work section, but it affects both the novelty and the empirical claims.

Questions for authors
  1. Were K, τ, λ, δ_r, the α weights and the stride fixed globally, or chosen per dataset, backbone or horizon? On which split were they selected? Please report TTC with a single global default and with a validation-only selection protocol.

  2. How do COSA and PETSA compare with TTC on the same shared-base protocol? What is the conceptual and empirical difference between TTC's retrieved-residual gated correction and COSA's gated residual correction?

  3. How do residuals from training windows (in-sample for the backbone) differ from validation and test residuals? Please report TTC with a memory built from validation windows only.

  4. For Exchange, Illness and Autoformer-Solar, what do the Mean, Random-K, 1-NN and Top-K-U ablations give? What fraction of the gain is explained by a constant or slowly varying bias correction?

  5. Why does SOLID equal Base exactly in so many cells? Were the baselines tuned on validation data per backbone and dataset?

  6. Can results be reported over multiple seeds, with confidence intervals?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.