acceptodds
Withdrawn from ICLR 2027

Calibrate First, Remember the Residual: Outcome-Aligned Memory for Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Online forecasting provides a stream of delayed targets that can be used to correct later predictions. A short residual memory, however, may spend much of its capacity tracking persistent changes in forecast level and scale. We introduce Calibrated-Residual Outcome-Aligned Ridge Memory (CR-OARM), which first applies causal channel-wise affine calibration and then fits a 32-episode ridge memory to the remaining error. The memory groups overlapping forecasts by their target timestamp so that repeated predictions of one outcome do not act as independent evidence. A frozen sample-wise gate controls correction strength. Across six datasets, four horizons, two backbones, and three source seeds, CR-OARM obtains the lowest mean MSE in a 144-case matched comparison (0.3826, versus 0.3865 for calibration, 0.3874 for COSA, and 0.3889 for TAFAS), although paired intervals against the two adapters include zero. In a convergence-controlled replay, a gate retrained on purged validation streams matches calibration (0.3859 versus 0.3858), while the fixed original gate reaches 0.3820 MSE and is reported as a control. A prediction-equivalent compact implementation runs 1.84–2.79× faster than released TAFAS on four GPU cases.

Primary area:
learning on time series and dynamical systems
Keywords:
time-series forecasting, online adaptation, residual memory, ridge regression, delayed feedback, distribution shift TL, DR

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BQxUOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CR-OARM, a method for online time-series forecasting.

The method has three main steps. First, it corrects systematic prediction errors using a simple calibration model. Second, it uses a small memory to learn from past residual errors. Third, a learned gate controls the size of each correction.

The main contribution is a memory that accounts for overlapping forecasts sharing the same target.

The authors evaluate the method on 6 datasets, 4 prediction horizons, and 2 forecasting models, using 3 seeds.

CR-OARM achieves the lowest average MSE in the main experiments. However, the improvements over COSA and TAFAS are not statistically significant.

The authors also propose a faster implementation that reduces computation time.

Critical strengths

The paper addresses an important problem in online forecasting. Several predictions can refer to the same future observation.

The proposed memory accounts for this dependence instead of treating all predictions as independent. This is a useful idea, supported by controlled experiments.

The authors compare their method with recent approaches, including COSA and TAFAS. They also provide several ablation studies.

The paper clearly reports cases where the method does not improve performance. The faster implementation is another useful contribution, especially for online forecasting.

Critical weaknesses

The improvements over COSA and TAFAS are relatively small. The reported confidence intervals include zero.

The method also does not consistently outperform simple calibration across datasets.

This makes it difficult to conclude that the proposed method provides a clear improvement over existing approaches. The results depend heavily on how the gate is trained.

In the convergence-controlled experiments, CR-OARM with a retrained gate achieves almost the same MSE as calibration alone (0.3859 versus 0.3858).

The best results instead come from the original fixed gate, which was trained using residuals from the fitted source models.

This raises an important question about the contribution of the proposed memory. It is unclear whether the gains come mainly from the new memory or from the gate and its training procedure.

This weakens the evidence for the main contribution.

Questions for authors
  1. Why does the original fixed gate perform better than the gate retrained on held-out validation data? Can the authors explain this difference and clarify how the gate should be trained in practice?

  2. Can the authors provide stronger evidence that outcome-aligned memory improves forecasting beyond calibration alone? In particular, does the improvement remain consistent when the gate is trained only on held-out data?

1 comment
AuthorsOpenReview ↗

Author response to Reviewer BQxU

Thank you for asking what the stricter gate comparison shows and how much of the gain comes from the memory.

End-to-end streaming cost (revised Table 5). Compact shared-gate CR-OARM is 1.84–2.79× faster than released TAFAS in median end-to-end latency across four GPU cases. The timing includes source inference, feedback, updates, and prediction, with three repetitions per case. The compact implementation preserves the reference CR-OARM forecast, with a largest observed difference of 2.99 × 10⁻¹¹ in MSE or MAE. Peak allocated memory is lower than TAFAS in two of the four cases. Appendix I gives the timing and equivalence checks.

CaseCR-OARM µs/originTAFAS µs/originSpeedupCR-OARM peak MiBTAFAS peak MiB
DLinear ETTm2 H9668.8174.42.53×42.827.0
PatchTST ETTm2 H96511.21426.22.79×175.4213.9
DLinear Weather H9678.4178.12.27×65.043.3
DLinear ETTm2 H72084.4155.41.84×60.9154.6

Gate training and affine correction. The shared checkpoint uses H96 raw residuals pooled from source-training and validation streams across all six datasets and both backbones. The source-training forecasts are in-sample. The purged-validation gate uses corrected residuals, so the comparison changes both the supervision split and the residual target. It therefore does not identify why the shared gate performs better. In revised Table 3, the purged-validation gate reaches 0.385853 MSE / 0.412226 MAE, versus 0.385784 / 0.410213 for affine alone. The held-out comparison provides no gain over affine. For dataset fine-tuning (revised Table 7), we start from the shared checkpoint, update on cached training examples, select the epoch by validation MSE, and freeze before test. This reaches 0.381053 MSE / 0.408696 MAE. Its MSE difference from the shared gate is −0.000983, with 95% paired task- and dataset-cluster intervals of [−0.001462, −0.000562] and [−0.001737, −0.000294]. The initialization retains its earlier H96 validation exposure.

Memory effect and uncertainty. Revised Table 6 holds gate outputs fixed and matches effective degrees of freedom for the ridge and shuffle controls. With the purged-validation gate, true timestamp groups give 0.385853 MSE, versus 0.387019 for DF-matched ridge and 0.387060 for shuffled groups. With the shared gate, true and shuffled groups give 0.382036 and 0.383315, respectively (95% paired dataset-cluster interval [−0.002880, −0.000148]). This supports the choice of timestamp grouping among memory variants. Across the full pipelines in revised Table 3, paired task- and dataset-cluster intervals against COSA-P, TAFAS, and the lagged ensemble include zero (Appendix D). TAFAS has the lowest mean MAE, 0.406978 versus 0.409326 for the shared gate.

Forecast accuracy on 144 matched cases (revised Tables 3 and 7)

MethodMean test MSEMean test MAE
Affine only0.3857840.410213
CR-OARM with purged-validation gate0.3858530.412226
CR-OARM with shared gate0.3820360.409326
CR-OARM with dataset fine-tuning0.3810530.408696
Released TAFAS0.3826810.406978
Reviewer_1VrmOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes affine calibration followed by ridge-memory correction of the remaining residuals, grouping overlapping forecasts by target timestamp and using a frozen gate to control correction strength. Although the method improves average MSE in some settings, the evidence does not establish stable benefits from its core additions over simple calibration.

Critical strengths

S1. Grouping overlapping forecasts by target timestamp addresses a concrete problem: repeated use of the same realized observation.

S2. The experiments include source-model convergence controls, supervision-source comparisons, and cluster-based confidence intervals, and transparently report negative results.

S3. The compact implementation preserves predictions and provides clear runtime improvements on the evaluated cases.

Critical weaknesses

W1. The core method fails to demonstrate added value over simple calibration under the stricter evaluation. In Table 3, the complete method with a retrained gate obtains an MSE of 0.385853, compared with 0.385784 for calibration alone, while MAE worsens from 0.410213 to 0.412226. The original gate remains the strongest variant, yet the paper does not explain why a gate trained on purged validation streams with supervision matched to the calibrated residual target loses this advantage. This is a central evidence gap concerning whether residual memory justifies its added complexity.

W2. The gate ablation does not establish the need for sample-adaptive gating. Table 4 compares learned gating against applying the full correction without shrinkage. The severe degradation of the latter only demonstrates that the candidate correction requires attenuation; it does not establish that a sample-adaptive neural network is necessary. The critical comparison is against a validation-tuned fixed coefficient applied to the same calibrated-residual correction. Without this control, the paper does not rule out the explanation that the gate mainly performs simple shrinkage.

W3. The mechanism attributed to outcome alignment is not sufficiently isolated in the main real-data experiments. Changing the group precision matrix also changes effective regularization, as the paper itself derives. However, the real-data controls involving matched degrees of freedom, count weighting, and shuffled timestamps are largely confined to four preliminary cases. Although the improvement over ordinary ridge in the main experiment has statistical support, this comparison does not adequately distinguish the benefit of correctly modeling shared outcomes from the effect of altered regularization. Synthetic experiments demonstrate that the mechanism can help under favorable conditions; they do not establish what drives the gains in the main benchmark.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Author response to Reviewer 1Vrm

Thank you for asking us to separate outcome grouping, shrinkage, and gate supervision.

Practical cost. The compact implementation with the shared gate runs 1.84–2.79× faster end to end than released TAFAS on four GPU cases (revised Table 5), while preserving the reference CR-OARM forecast.

Gate supervision and affine correction (W1). Affine alone reaches 0.385784 MSE and 0.410213 MAE. The purged-validation gate reaches 0.385853 and 0.412226, while the shared gate reaches 0.382036 and 0.409326 (revised Table 3). The shared checkpoint was trained on H96 raw residuals pooled across all six source datasets and both backbones, using source-training and validation streams. The source-training predictions are in-sample. The purged-validation control uses a different supervision split and corrected residuals, so these scores do not isolate the cause of the gap. With the purged-validation gate, true timestamp groups improve on the matched ridge and shuffle controls (revised Table 6), while affine alone still gives lower error. As a separate control, we fine-tuned one copy of the shared gate per dataset on cached training examples and selected the epoch by validation MSE before freezing. This reaches 0.381053 MSE (revised Table 7). The paired difference from the shared gate is −0.000983, with a 95% dataset-cluster interval of [−0.001737, −0.000294]. The initialization retains its H96 validation exposure.

Validation-tuned constant beta (W2). We select the MSE-optimal bounded constant separately for each case and memory estimator on the last 40% of validation origins (Appendix K). Constant-beta OARM gives 0.393221 test MSE, compared with 0.382036 for the shared gate and 0.385853 for the purged-validation gate (revised Table 3). The purged-gate minus constant-beta difference is −0.007368, with a 95% paired dataset-cluster interval of [−0.013846, −0.001658].

Timestamp structure and regularization (W3). Revised Table 6 uses all 144 cases, holding the memory, features, refit schedule, and realized beta values fixed. The DF-matched controls also share the same effective degrees of freedom at each refit. With the shared gate, true timestamps yield 0.382036 MSE, compared with 0.383285 for DF-matched ridge, 0.383814 for DF-matched inverse-count weighting, and 0.383315 for shuffled groups. The paired true-minus-shuffled difference is −0.001279, with a 95% dataset-cluster interval of [−0.002880, −0.000148]. With the purged-validation gate, the corresponding true and shuffled scores are 0.385853 and 0.387060. Appendix K gives the five shuffle seeds and pairing details.

Affine and gate controls (revised Tables 3 and 7)

MethodMean test MSEMean test MAE
Affine only0.3857840.410213
OARM with validation-tuned constant beta0.3932210.415850
CR-OARM with purged-validation gate0.3858530.412226
CR-OARM with shared gate0.3820360.409326
CR-OARM with dataset fine-tuning0.3810530.408696

Precision controls at fixed gate outputs (revised Table 6)

PrecisionShared gate MSEPurged-validation gate MSE
True timestamp OARM0.3820360.385853
DF-matched ridge0.3832850.387019
DF-matched inverse-count weighting0.3838140.387063
DF-matched shuffled groups0.3833150.387060

Thank you for the response and additional experiments. However, this paper contribution remains unclear, and the presentation falls short of the standard I would expect at ICLR. Therefore I maintain my recommendation to reject.

Reviewer_HnkzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper designs a ridge-regression bias and amplitude forecast correction, using forecasts created at different origins. It uses a prescribed inverse-covariance weighting to account for dependence among historical forecast residuals that share the same target timestamp.

Critical strengths
  • Substantial effort to isolate mechanisms: ablations, synthetic studies and failure cases are appreciated

  • Candid reporting of negative results

  • Careful design of ridge residual-correction memory that uses target-timestamp covariance

Critical weaknesses
  • The conformal methods are conceptually related, but totally absent from related work and baselines

  • The novelty and the added value of the method compared to Angelopoulos et al, Conformal PID Control for Time Series Prediction is not very clear

  • The method is positioned as a calibration method. calibrate/calibration appear 94 times in the document. However, the method is not capable of producing confidence intervals and all metrics are point prediction metrics. I feel that the paper misrepresents what method does technically and creates a wrong expectation.

  • Additionally, there is a strong link to the weighted-average lagged ensemble. This is a very related literature area, which is completely missing from the related work review.

  • The novelty with respect to DelSole et al., The weighted-average lagged ensemble is unclear

  • Benchmarks used in the study are small scale

Questions for authors
  • Could you please discuss the positioning of the current paper with respect to references [1-3]? While [1] is a more general method reference, [2] is a standard and wide-spread ACI, [3] seems to be most directly related to your work. Why conformal methods are not discussed at all in the literature review?

  • Could you please include one of the conformal methods as a baseline?

  • What is the added value of the current method compared to [3]?

  • Please remove the usage of calibration, because the proposed method does not do any calibration (in statistical sense).

  • Please discuss novelty with respect to [4]

  • Please include [4] as a baseline

  • Please update the empirical results on GiftEval and fev-bench benchmarks. Current benchmarks are small. It is unclear if results are statistically significant.

[1] Chen Xu, Yao Xie, Conformal prediction for time series, https://arxiv.org/pdf/2010.09107

[2] Isaac Gibbs, Emmanuel Candès, Adaptive Conformal Inference Under Distribution Shift, https://arxiv.org/pdf/2106.00170

[3] Anastasios N. Angelopoulos, Emmanuel J. Candes, Ryan J. Tibshirani, Conformal PID Control for Time Series Prediction, https://arxiv.org/pdf/2307.16895

[4] DelSole, T., Trenary, L., & Tippett, M. K. (2017). The weighted-average lagged ensemble. Journal of Advances in Modeling Earth Systems, 9, 2739–2752. https://doi.org/10.1002/2017MS001128

1 comment
AuthorsOpenReview ↗

Author response to Reviewer Hnkz

Thank you for pointing out the links to conformal forecasting and lagged ensembles, and for suggesting a broader evaluation.

Streaming cost. In a separate comparison on four GPU cases, compact shared-gate CR-OARM runs 1.84–2.79× faster end to end than released TAFAS (revised Table 5).

Point forecasts and conformal prediction. The revision calls Eq. 1 affine correction. It adjusts the level and scale of a point forecast, evaluated by MSE and MAE. Section 2 now discusses EnbPI, adaptive conformal inference, and conformal PID. These methods adapt prediction intervals to control coverage. Conformal PID updates interval thresholds using past coverage errors, whereas CR-OARM uses delayed forecast residuals to update the point forecast itself. We did not run a conformal baseline. A direct empirical comparison would require interval outputs and a shared coverage-and-width protocol.

Lagged ensembles. DelSole et al. combine forecasts of the current target issued from different origins. CR-OARM instead fits a residual predictor from completed historical episodes, with target timestamps determining the regression weights in Eq. 5. We implemented the published weighted average using validation-fitted error moments on the same 144 source checkpoints, causal affine inputs, and future-only scoring mask (revised Table 3 and Appendix L). Mean MSE is 0.382036 for shared-gate CR-OARM, 0.383458 for the ensemble, and 0.385784 for affine alone. The paired 95% task-cluster interval for CR-OARM minus the ensemble is [−0.004778, 0.001825]. On PatchTST, the ensemble gives lower MSE than the shared gate, 0.386425 versus 0.387078.

Benchmark scale and uncertainty. We added Electricity (321 channels) and Traffic (862 channels) at H336 and H720 with three DLinear seeds per task. In revised Table 8, we fine-tune one gate per dataset from the shared checkpoint on training residuals, select the epoch on validation, and freeze it before test. Mean MSE across the 12 DLinear cases is 0.432467 for the dataset gate, 0.434273 for the transferred shared gate, 0.431981 for the per-case gate, 0.432116 for affine alone, and 0.430279 for TAFAS. The dataset gate improves on the transferred gate in all 12 cases. Affine and TAFAS both give lower MSE in all 12 cases. The additional PatchTST Electricity-H336 result uses an interim source checkpoint and is shown separately. We have not evaluated GIFT-Eval or FEV-Bench. On the 144-case benchmark, the paired task- and dataset-cluster intervals against COSA-P, TAFAS, and the lagged ensemble include zero (Appendix D).

Matched lagged-ensemble comparison (revised Table 3)

MethodMean MSEPatchTST MSE
Affine only0.3857840.390236
Affine + weighted lagged ensemble0.3834580.386425
Shared-gate CR-OARM0.3820360.387078

Electricity and Traffic test MSE (revised Table 8)

TaskAffineShared gateCase gateDataset gateTAFAS
DLinear Electricity H3360.2170780.2194140.2170070.2174280.214477
DLinear Electricity H7200.2576860.2609030.2585580.2583270.256409
DLinear Traffic H3360.6070980.6087530.6063820.6073200.605610
DLinear Traffic H7200.6466020.6480220.6459790.6467940.644619
DLinear mean (12 cases)0.4321160.4342730.4319810.4324670.430279
PatchTST Electricity H336 (interim)0.2067350.2088900.2067010.2069950.202767

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.