- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CR-OARM, a method for online time-series forecasting.
The method has three main steps. First, it corrects systematic prediction errors using a simple calibration model. Second, it uses a small memory to learn from past residual errors. Third, a learned gate controls the size of each correction.
The main contribution is a memory that accounts for overlapping forecasts sharing the same target.
The authors evaluate the method on 6 datasets, 4 prediction horizons, and 2 forecasting models, using 3 seeds.
CR-OARM achieves the lowest average MSE in the main experiments. However, the improvements over COSA and TAFAS are not statistically significant.
The authors also propose a faster implementation that reduces computation time.
Critical strengths
The paper addresses an important problem in online forecasting. Several predictions can refer to the same future observation.
The proposed memory accounts for this dependence instead of treating all predictions as independent. This is a useful idea, supported by controlled experiments.
The authors compare their method with recent approaches, including COSA and TAFAS. They also provide several ablation studies.
The paper clearly reports cases where the method does not improve performance. The faster implementation is another useful contribution, especially for online forecasting.
Critical weaknesses
The improvements over COSA and TAFAS are relatively small. The reported confidence intervals include zero.
The method also does not consistently outperform simple calibration across datasets.
This makes it difficult to conclude that the proposed method provides a clear improvement over existing approaches. The results depend heavily on how the gate is trained.
In the convergence-controlled experiments, CR-OARM with a retrained gate achieves almost the same MSE as calibration alone (0.3859 versus 0.3858).
The best results instead come from the original fixed gate, which was trained using residuals from the fitted source models.
This raises an important question about the contribution of the proposed memory. It is unclear whether the gains come mainly from the new memory or from the gate and its training procedure.
This weakens the evidence for the main contribution.
Questions for authors
-
Why does the original fixed gate perform better than the gate retrained on held-out validation data? Can the authors explain this difference and clarify how the gate should be trained in practice?
-
Can the authors provide stronger evidence that outcome-aligned memory improves forecasting beyond calibration alone? In particular, does the improvement remain consistent when the gate is trained only on held-out data?
Author response to Reviewer BQxU
Thank you for asking what the stricter gate comparison shows and how much of the gain comes from the memory.
End-to-end streaming cost (revised Table 5). Compact shared-gate CR-OARM is 1.84–2.79× faster than released TAFAS in median end-to-end latency across four GPU cases. The timing includes source inference, feedback, updates, and prediction, with three repetitions per case. The compact implementation preserves the reference CR-OARM forecast, with a largest observed difference of 2.99 × 10⁻¹¹ in MSE or MAE. Peak allocated memory is lower than TAFAS in two of the four cases. Appendix I gives the timing and equivalence checks.
Gate training and affine correction. The shared checkpoint uses H96 raw residuals pooled from source-training and validation streams across all six datasets and both backbones. The source-training forecasts are in-sample. The purged-validation gate uses corrected residuals, so the comparison changes both the supervision split and the residual target. It therefore does not identify why the shared gate performs better. In revised Table 3, the purged-validation gate reaches 0.385853 MSE / 0.412226 MAE, versus 0.385784 / 0.410213 for affine alone. The held-out comparison provides no gain over affine. For dataset fine-tuning (revised Table 7), we start from the shared checkpoint, update on cached training examples, select the epoch by validation MSE, and freeze before test. This reaches 0.381053 MSE / 0.408696 MAE. Its MSE difference from the shared gate is −0.000983, with 95% paired task- and dataset-cluster intervals of [−0.001462, −0.000562] and [−0.001737, −0.000294]. The initialization retains its earlier H96 validation exposure.
Memory effect and uncertainty. Revised Table 6 holds gate outputs fixed and matches effective degrees of freedom for the ridge and shuffle controls. With the purged-validation gate, true timestamp groups give 0.385853 MSE, versus 0.387019 for DF-matched ridge and 0.387060 for shuffled groups. With the shared gate, true and shuffled groups give 0.382036 and 0.383315, respectively (95% paired dataset-cluster interval [−0.002880, −0.000148]). This supports the choice of timestamp grouping among memory variants. Across the full pipelines in revised Table 3, paired task- and dataset-cluster intervals against COSA-P, TAFAS, and the lagged ensemble include zero (Appendix D). TAFAS has the lowest mean MAE, 0.406978 versus 0.409326 for the shared gate.
Forecast accuracy on 144 matched cases (revised Tables 3 and 7)