- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MATE, a test-time adaptation framework for time-series forecasting under delayed feedback. MATE keeps the pretrained forecasting backbone frozen and constructs five residual correction candidates using local and global output structures, historical forecast versions, input calibration, and recent observations. The proposed credit controller uses accumulated second-order statistics from fully matured historical predictions to estimate correction weights. To account for differences across forecast horizons and variables, it combines lead-specific and channel-specific weights regularized toward a global reference.
The method is evaluated on six datasets, four forecasting backbones, and five distribution-shift scenarios. MATE achieves the lowest MSE in 396 out of 460 configurations (86.1%), with additional ablation studies examining the contribution of credit allocation granularity and individual residual sources.
Critical strengths
- Consistent empirical improvements across diverse settings.
The paper demonstrates consistent forecasting improvements across multiple datasets, backbone architectures, prediction horizons, and distribution-shift scenarios. MATE achieves the lowest average MSE in all five scenarios and outperforms the compared methods in a large majority of configurations. The broad evaluation provides reasonably strong empirical evidence that the approach is effective across different forecasting settings rather than being tailored to a particular backbone or dataset.
- A reasonable and well-structured approach to correction allocation under delayed feedback.
Separating residual generation from credit allocation is a sensible design choice. The use of accumulated second-order statistics allows the controller to account for interactions among correction candidates, while the global reference provides regularization for lead- and channel-specific weights when feedback is limited. The full-window feedback protocol also ensures that adaptation relies only on available historical targets. Overall, the proposed framework is technically coherent and addresses a relevant challenge in time-series test-time adaptation.
Critical weaknesses
- The significance of the proposed methodological contribution is not fully established.
While the overall framework is technically reasonable, its main components are relatively straightforward, combining multiple residual corrections, regularized quadratic optimization, and additive lead–channel weight decomposition. The selection of the five residual sources is also primarily motivated by intuition, without a systematic analysis of their complementarity or necessity.
More importantly, the ablation results raise questions about the incremental contribution of the proposed lead–channel allocation, which appears to be a central methodological contribution of the paper. In Table 4, the full method achieves only a marginal improvement over channel-only allocation (0.2258 vs. 0.2263 in average MSE, approximately 0.22%). Although the full method performs best on average, the results do not clearly establish whether the additional lead-dependent allocation provides a sufficiently meaningful advantage over simpler alternatives.
Thus, while the empirical results are encouraging, it remains unclear to what extent the performance gains stem from the proposed credit allocation mechanism itself, rather than the availability of multiple complementary correction candidates. This makes the significance of the methodological contribution somewhat difficult to assess.
Questions for authors
-
[Clarification / Additional evidence] The improvement of full lead–channel allocation over channel-only allocation is relatively small in Table 4. Could the authors elaborate on when lead-dependent allocation provides meaningful additional benefits? Are these benefits consistent across datasets, backbones, and forecasting horizons?
-
[Clarification] What motivates the particular selection of the five residual correction sources? Is there evidence that they provide sufficiently distinct or complementary correction directions? A clearer understanding of their interactions would help assess the importance of the proposed residual dictionary.
-
[Clarification] The credit statistics accumulate historical evidence while the residual-generating calibrators continue to evolve. How does the method account for the potential mismatch between historical residual candidates and those generated by updated calibrators? Additionally, how does accumulating statistics without explicit forgetting affect adaptation to abrupt distribution shifts?
3 comments
Q3. Do historical candidates differ from current candidates, and how does accumulated credit affect adaptation after an abrupt shift?
(1) How historical candidates enter credit updates. When full-window feedback arrives, MATE updates the credit statistics with the candidates saved at prediction time and the corresponding base forecast errors. As the calibrators change, the accumulated statistics still contain evidence from earlier candidates. We recalculated Local, Global, and Input on the same historical inputs using the current calibrators and observed changes in the candidates. See Sections 4.1 and 4.4 and Appendix C.7.
(2) How forgetting affects adaptation after a shift. We compared cumulative credit with two forgetting rates. Faster forgetting increased full-stream error. In Level, slower forgetting reduced mean MSE by 0.452% over the first 256 steps after the first complete post-shift feedback arrived. The gain was close to zero when the observation window was extended to 1,024 steps. We retain cumulative credit in the main experiments based on the full-stream results. See Appendix C.7 and Table C15.
Table C15: Full-stream error reductions of credit forgetting relative to cumulative credit. Positive values favor forgetting.
| Half-life (matured origins) | MSE reduction | MAE reduction | MSE wins |
|---|---|---|---|
| 512 | −0.273% | −0.077% | 74/368 |
| 2,048 | +0.003% | +0.029% | 176/368 |
Thank you for the detailed response. I believe the additional experiments adequately address my concerns. In particular, the findings on Vintage and the complementarity of residual sources are interesting.
However, the performance gains remain relatively modest, especially for the proposed lead–channel allocation and the incremental improvements from adding some residual sources (e.g., Input and Context). I also find the methodological novelty somewhat limited. While the rebuttal strengthens the empirical evidence, I remain unconvinced that the overall contribution is sufficiently significant. Therefore, I will maintain my rating.
Thank you for pointing out that the original manuscript needed to distinguish more clearly between the roles of credit allocation and residual combinations. In response, we added credit controls with fixed candidates, experiments with individual and incrementally combined sources, and analyses of candidate evolution and forgetting. We also revised the associated discussion. We show the revised material in blue text to help you locate the changes. Please see the revision for details.
Thank you for pointing out that the original manuscript needed to distinguish more clearly between the roles of credit allocation and residual combinations. In response, we added credit controls with fixed candidates, experiments with individual and incrementally combined sources, and analyses of candidate evolution and forgetting. We also revised the associated discussion. We show the revised material in blue text to help you locate the changes. Please see the revision for details.
Direct Questions
Q1. When does lead-dependent credit help?
(1) Which configurations benefit. We hold the residual candidates fixed and use Channel-only as the control to assess the value of distinguishing forecast leads in addition to channels. Joint allocation reduces mean paired MSE by 0.27%. The mean gain is positive for all four backbones. It is largest on ETTm1, while the two methods are close on Exchange. See Section 5.3, Table 4, and Appendix C.9, Table C14 of the revision.
(2) Forecast horizons and the correspondence between leads and weights. Across the same 80 dataset–backbone–scenario combinations at each horizon, joint allocation yields an average gain at all four horizons. The gain is larger at shorter horizons, as shown below. With fixed candidates, averaging the lead component while retaining the channel component raises MSE. This also supports varying correction strength by lead. See Appendix C.5, Table C11.
Supplementary table. MSE reduction of Full MATE relative to Channel-only at each horizon
Q2. Why choose these five residual sources, and do they complement one another?
(1) Rationale for the five sources. Local uses relationships among nearby forecast leads and across channels. Global uses smooth structure across the forecast window. Vintage combines earlier forecasts for the same target from different origins. Input captures the effect of input calibration on the forecast, and Context uses the persistence of recent observations. Section 4.2 describes how each source is constructed.
(2) Standalone effects. We allow each of the five residual sources to adapt independently. Each source yields lower mean MSE than the frozen backbone in all four scenarios. Relative to Vintage-only, the single source with the lowest overall mean MSE, Full MATE reduces mean paired MSE by 3.45% and has lower error in 366 of 368 configurations. See Section 5.3, Table 5 for the single-source comparison.
(3) Gains from combining sources. We fix the residual candidates and re-estimate the credit weights after each addition. In the sequence tested in Table 5, each new source lowers mean MSE in all four scenarios. For example, adding Vintage still improves the forecast when Local and Global are already present. This indicates that overlapping historical forecasts provide information that complements output structure. See Section 5.3, Table 5 for MSE and Appendix C.6, Table C13 for MAE.
Table 5: Mean MSE for independent single-source adaptation and fixed candidate subsets. Each scenario has 92 configurations and three test seeds per configuration. Bold marks the lowest displayed MSE within each adaptation group for each scenario.