acceptodds
Under review as a conference paper at ICLR 2027

Consensus Forgets First: Online Adaptation of Frozen Predictors

Abstract

When a pool of frozen models predicts the same target under distribution shift, the right adaptation memory is unknown and itself non-stationary. The Gram matrix of the pool's predictions, however, has a spectrum known before deployment, which acts as a prior on memory. With one covariance-inflation scalar, the dominant direction, the consensus level, carries the shortest memory while contrasts between models keep long memory. We take the ratios between memory lengths from the spectrum and hedge over their common scale by aggregating recursive least-squares experts online. After standardizing by the training scale, the layer has no tuned parameter. We prove deterministic oracle inequalities: safety against the best raw predictor, and path-length tracking against the best time-varying combination. On French electricity load (2019–2023) and NYC taxi ridership, the layer improves on every frozen base and on aggregation of the raw pool in every regime, and on validation-tuned adaptive filters overall. Without the inflation scalar, the same hedge loses to a validation-tuned Kalman filter.

Primary area:
learning on time series and dynamical systems
Keywords:
online learning, distribution shift, prediction with expert advice, recursive least squares, time series forecasting, dynamic regret

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_pHy4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

MELO is an online calibration framework for time-series forecasting that combines recursive least-squares experts with varying forgetting factors and covariance inflation settings, using MLpol to aggregate their predictions alongside those of the backbone models. The key intuition is that models forecasting the same target often capture a shared predictive direction but differ in their calibration errors and responses to distribution shifts. To address this, MELO balances the retention of historical information with adaptation to changing conditions. Covariance inflation enables the experts to adjust more rapidly when predictive relationships shift, while different forgetting factors provide complementary memory horizons. By aggregating multiple experts, MELO aims to improve forecasting robustness and generalization under distribution shift.

Critical strengths
  1. Well-supported design and transparent implementation: The ablation studies show that removing covariance inflation substantially degrades performance and clarify the contributions of memory selection, intercept correction, and ensemble composition. These results help explain the mechanism behind the performance gains. The supplied code is consistent with Algorithm 1 and includes extensive experiments corroborating the paper’s claims, strengthening confidence in the results and supporting reproducibility.

  2. Practical efficiency with clearly defined applicability: The method operates as a lightweight calibration layer on predictions from frozen backbones, requiring no additional backbone fine-tuning. This makes it straightforward to integrate into existing forecasting pipelines with limited computational overhead. The paper also explicitly acknowledges that calibration cannot recover information absent from the original forecasts, giving readers a clear understanding of the method’s capabilities and limitations.

Critical weaknesses
  1. Insufficient comparative evidence to establish MELO’s practical contribution.

    The evaluation omits relevant adaptation methods, including OMPB [1], ELF [2], TAFAS [3], SOLID [4], and OneNet [5]. OMPB is particularly relevant to calibration of a fixed backbone, while ELF and OneNet provide important comparisons for online aggregation. The experiments also rely mainly on small ensembles of conventional forecasting models; the appendix’s tabular foundation-model experiments on a shorter electricity window do not establish consistent benefits for time-series foundation models. Comparisons should align forecast horizons, available information, and label-arrival timing, report computational costs, and distinguish methods that update the backbone from those that preserve it. Without broader comparisons or clear explanations of incompatibility, the current evidence does not adequately establish MELO’s advantage over existing adaptation and ensembling approaches.

  2. Insufficient evidence of robustness across different types of distribution shift.

    The electricity and taxi experiments use plausible event-based regimes, but these do not establish the type or severity of shift or distinguish marginal changes from changes in the conditional input–target relationship. MELO updates an affine combination of frozen forecasts: covariance inflation facilitates adaptation without expanding the available representation. A large demand collapse may therefore be manageable through level or scale corrections, leaving robustness to feature-dependent changes, altered temporal relationships, and shifts in relative predictor quality insufficiently tested. Controlled experiments and diagnostics that account for seasonality and temporal dependence are needed to identify when calibration succeeds or fails, especially because wrapping an already adaptive predictor can worsen performance. Clearer links between the idealized recursion, covariance-inflated analysis, and implementation choices would also help readers assess which guarantees apply to the evaluated method.

References

[1] Chenfeng Huang, Zixuan Ma, and George Michailidis. Model-Agnostic Online Certificate-Driven Calibration for Time Series Forecasting Under Distribution Shift. UAI, 2026. (OMPB)

[2] Thomas L. Lee, William Toner, Rajkarn Singh, Artjom Joosen, and Martin Asenov. Lightweight Online Adaption for Time Series Foundation Model Forecasts. ICML, 2025. (ELF)

[3] HyunGi Kim, Siwon Kim, Jisoo Mok, and Sungroh Yoon. Battling the Non-stationarity in Time Series Forecasting via Test-time Adaptation. AAAI, 2025. (TAFAS)

[4] Mouxiang Chen, Lefei Shen, Han Fu, Zhuo Li, Jianling Sun, and Chenghao Liu. Calibration of Time-Series Forecasting: Detecting and Adapting Context-Driven Distribution Shift. KDD, 2024. (SOLID)

[5] Yi-Fan Zhang et al. OneNet: Enhancing Time Series Forecasting Models under Concept Drift by Online Ensembling. NeurIPS, 2023.

Questions for authors

Based on the weakness, I have the following questions:

  1. Comparison with relevant adaptation methods: How does MELO compare with OMPB, ELF, TAFAS, SOLID, and OneNet under matched forecast horizons, information access, and label-arrival timing? Please report computational costs, distinguish methods that update the backbone from those that preserve it, and explain any incompatibilities that prevent a fair comparison.

  2. Applicability to modern time-series forecasting models: Does MELO consistently improve deep learning forecasters, including recent time-series foundation models, across multiple datasets and shift conditions compared with the same models without calibration? Under what conditions does wrapping an already adaptive predictor degrade performance?

  3. Shift characterization and limits of affine calibration: What evidence distinguishes changes in the input distribution from changes in the conditional input–target relationship in the electricity and taxi experiments, after accounting for seasonality and temporal dependence? Can controlled experiments involving feature-dependent changes, altered temporal relationships, or changes in relative predictor quality clarify when MELO’s affine calibration succeeds or fails?

  4. Robustness to longer feedback delays: How does MELO perform when the label-feedback delay exceeds one time step? Can the authors evaluate forecasting accuracy and adaptation speed across several delay lengths, particularly following distribution shifts, while ensuring that all compared methods receive labels at the same time?

Reviewer_XrKAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper deals with non-stationary regression, where the model may change with time.

The proposed approach is as follows:

(a) Train a set of fixed forecasters and freeze them. In what follows they are considered essentially as feature extractors, providing a feature vector ztz_t at time tt.

(b) Initialize a set of Kalman filters. The state wtw_t of each filter is the vector of regression coefficients, modelled as a random walk with isotropic process noise εI\varepsilon I, and each filter also discounts past observations by a forgetting factor γ\gamma. ztz_t are viewed as the observation functions of the state, so that yt=⟨zt,wt⟩+noisey_t = \langle z_t, w_t \rangle + \text{noise} is the observation and also the target variable to forecast.

The different filters differ in the forgetting factor and in the level of the process noise, one filter per point of a preset grid of parameters.

(c) An expert aggregation algorithm, MLpol in this paper, is used to combine the forecasts from the filters and the raw forecasts of the fixed forecasters. Roughly speaking, with time this should attain the performance of the best convex combination of the filters.

Regret bounds are proved on the scheme, for standard and for dynamic regret.

Experiments on electricity and NYC taxi time series show that the approach outperforms several baseline competitors.

Critical strengths

The problem of model shift, non-stationary regression in this case, is important.

There is an improvement in performance over the baselines chosen.

Critical weaknesses

Novelty.

There are critical issues with novelty, for both the approach and the theory.

Both parts of the approach, modelling non-stationary regression with Kalman filters, and hyperparameter choice via expert aggregation algorithms, are completely standard and widely used in the literature. Moreover, there are several recent papers where exactly this combination is used, with minor variations in how filter parameters are chosen, which features enter the filter, and which aggregating algorithm is used. See the references below.

For the theory, all the results are essentially standard.

Theorem A.3 is a re-derivation of Yuan & Lamperski 2020 (Theorem 1, case 3) with a minor difference (without the projection step, assuming bounded iterates instead). Their argument, in turn, is also a variation on standard arguments. Yuan & Lamperski 2020 should be cited here.

Moreover, Section 6 of the same paper (Algorithm 2, Theorem 7) runs forgetting RLS on a geometric grid of discount factors, including γ=1\gamma = 1, combines them by exponential weights, and obtains the path-length dynamic regret bound without knowledge of the path length. This is the construction of the present paper without the covariance inflation, and the result of Corollary A.15.

Theorem A.8 is an extension of A.3 and its argument is a straightforward modification of the argument for A.3.

All the other results are standard applications of existing well known results.

I thus do not see novelty in this paper.

Evaluation.

It is clear from Tables 1 and 2 that the data favours tracking regression methods, such as the Kalman filter, rather than a static convex combination of the fixed forecasters, like, for instance, MLpol on bases only.

However, only two such baselines are presented: a rolling-window OLS combination, and VIKING, which runs a single Kalman filter and learns its process-noise variance online.

I would expect much broader evaluation of different regression tracking methods here.

References

  • David Obst, Joseph de Vilmarest, and Yannig Goude. Adaptive methods for short-term electricity load forecasting during COVID-19 lockdown in France. IEEE Transactions on Power Systems, 36(5):4754–4763, 2021. arXiv:2009.06527.
  • Joseph de Vilmarest and Yannig Goude. State-space models for online post-covid electricity load forecasting competition. IEEE Open Access Journal of Power and Energy, 9:192–201, 2022. arXiv:2110.00334.
  • Baptiste Abélès, Joseph de Vilmarest, and Olivier Wintenberger. Adaptive time series forecasting with Markovian variance switching. arXiv:2402.14684, 2024.
  • Andrew Jacobsen and Ashok Cutkosky. Online linear regression in dynamic environments via discounting. ICML 2024. arXiv:2405.19175.
  • Jianjun Yuan and Andrew Lamperski. Trading-off static and dynamic regret in online least-squares and beyond. AAAI 2020. arXiv:1909.03118.
Questions for authors

Please discuss the items in the weakness section.

Reviewer_85QBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an online learning algorithm, named MELO for online adaptation of frozen predictors under distribution shift. The method combines covariance-inflated recursive least squares for direction-dependent memory with MLpol aggregation over multiple adaptation scales. The authors provide regret bounds for the algorithm and evaluate it on French electricity load and NYC taxi ridership datasets. The results show improvements over several baselines, especially during periods of substantial distribution shift.

Critical strengths
  1. An important practical problem.

The paper addresses the problem of adapting existing prediction models to distribution shifts without retraining them. This is relevant to many real-world applications where frequent retraining is costly or impractical.

  1. Clear presentation.

The paper is well written and easy to follow. The motivation, algorithm, theoretical results, and experiments are presented clearly.

  1. Good experimental results.

The method performs well on two real-world datasets and outperforms several baselines. The ablation studies also help explain why covariance inflation and online aggregation are useful.

Critical weaknesses
  1. Lack of spectral dependence in the regret bound.

The main idea of the paper is to use the spectral structure of the Gram matrix for adaptation. However, the regret bound does not explicitly depend on the individual eigenvalues. It is therefore unclear from the theoretical results how the spectral structure affects the regret.

  1. Limited asymptotic guarantee for dynamic regret.

The tracking regret bound in Theorem 1 contains terms proportional to dT/hγdT/h_\gamma and cTcT. For fixed finite hγh_\gamma and positive cc, these terms do not vanish after dividing the bound by TT. Thus, the current bound does not guarantee vanishing average dynamic regret.

Questions for authors
  1. Why use isotropic covariance inflation?

The method uses εI\varepsilon I for covariance inflation. Could the authors explain why this particular choice is made? Would it be possible to use a general positive definite matrix QQ, allowing different inflation levels in different directions? How would this affect the algorithm and its performance?

  1. Why assign shorter memory to directions with larger eigenvalues?*

The method assigns shorter memory lengths to directions with larger eigenvalues of the Gram matrix. However, larger eigenvalues do not necessarily mean that the corresponding optimal coefficients change more quickly. Could the authors explain why this memory allocation is appropriate? What happens if the distribution shift mainly occurs in directions with smaller eigenvalues?

Reviewer_k2bhOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Summary The paper addresses online adaptation of a pool of frozen forecasting models under distribution shift. The proposed method, MELO, wraps the pool in a grid of recursive least-squares experts with forgetting and covariance inflation, and aggregates these experts together with the raw base forecasts by MLpol. The main observation is that covariance inflation, applied to the pool's prediction Gram matrix, assigns different memory lengths to different directions. The dominant consensus direction, along which all base predictions agree, receives the shortest memory and recalibrates within days of a break. The contrasts between models keep a long memory. The spectrum fixes the ratios between memory lengths, and the method hedges over their common scale through a grid of forgetting factors and inflation levels.

The main contributions are: (1) an interpretation of covariance inflation as directional memory allocation determined by the training spectrum of the pool; (2) an aggregation scheme over forgetting factors and inflation levels in which no parameter is selected on validation data; and (3) deterministic oracle inequalities, namely a safety bound against the best raw predictor and a path-length tracking bound against the best time-varying affine combination, under a bounded-iterate assumption.

Validation uses two daily one-step-ahead streams with COVID-era breaks: French national electricity load (2019–2023) and NYC yellow-taxi ridership (2019–2023). MELO improves on every frozen base, on MLpol over the raw pool in every regime, on a rolling least-squares combination, and on VIKING, a validation-tuned variational Kalman filter. Ablations remove the inflation, the intercept, and the ridge. Further experiments compare hedging with causal window selection and a hindsight oracle, vary pool composition, and add daily-refit tabular foundation models as references and as pool members.

Critical strengths
  1. A simple, inexpensive mechanism with a clear explanation. The paper shows that a single inflation scalar, acting on the spectrum of the pool's predictions, separates fast recalibration of the consensus level from slow adjustment of the relative model weights. The controlled single-expert experiments (Figures 3, 5, and 9) make this mechanism visible: without inflation, every regime favors the shortest memory, while with inflation the calm regimes recover their long-memory optimum. The layer runs in about 1.5 ms per step and requires no retraining of the base models or knowledge of the regimes. This weighs significantly in my assessment because it offers a practical and interpretable recipe for a common deployment setting.
  2. Careful and honest empirical analysis. The gains over MLpol on the raw pool are large, especially at the breaks (lockdown RMSE 1063 vs. 2244 MW). The improvement over VIKING is significant on both streams (paired bootstrap intervals [+6, +97] MW and [+100, +1319] trips). The experimental design goes beyond headline accuracy. The hedge matches the hindsight per-regime splice within bootstrap noise and beats every causal window selector. The pool-composition sweep shows that the gain persists across all 31 base subsets. The authors also report cases in which MELO loses: VIKING before COVID on taxi, and daily-refit foundation models on the electricity sub-window. This matters for my assessment because it makes the empirical claims credible within their scope.
Critical weaknesses
  1. The novelty is concentrated in the hedge, and the theory does not cover the cells that produce the gains. A memory length of about 1/√(ελᵢ) per eigendirection is the steady-state gain of a local-level Kalman filter, as the authors note with reference to Harvey (1990). Any Kalman filter with isotropic process noise, including VIKING, already has this directional structure. The new element is therefore the hedge over the inflation magnitude, together with its interpretation, rather than the spectral memory allocation itself.

The tracking guarantee is also weaker than its presentation suggests. By Remark A.6, the inflation leakage stays below the finite-memory term only when ε₀B_z² ≲ 2d/h. Under the deployed coupling ε₀ = c/B̂_z², with c ≥ 1 and B_z a uniform bound on ‖z̃_t‖, the left side is of order c or larger. The right side is at most 2·6/20 = 0.6 on the deployed grid. For every inflated cell, the leakage term therefore dominates, and the tracking bound is linear in T. These are precisely the cells that carry the gains: the hindsight-best cells use c = 10² in five of six electricity regimes and c = 10¹⁰ in the lockdown (Appendix G). The guarantees that remain informative for the deployed method are the MLpol safety bound against the best raw predictor and the bound against the best grid cell. The tracking bound also assumes bounded iterates, which the recursion does not guarantee and which is checked only empirically (Appendix B.9).

This is an important limitation because the theory is intended to explain the underlying mechanism, yet the current guarantees establish the reliability of the aggregation procedure without demonstrating the benefits of covariance inflation.

  1. The comparison with VIKING may partly reflect tuning on a calm validation year, and the evaluation scope is narrow. VIKING selects its nuisance parameters on 2018, a year without a break. Its learned inflation corresponds to c ≈ 0.04 on electricity and c ≈ 0.13 on taxi, below the smallest value in MELO's grid, and it barely moves at the breaks (Appendix B.6). The paper itself shows that selecting MELO's own c on 2018 fails on taxi. Part of MELO's advantage over VIKING may therefore come from hedging versus validation on a calm year, not from spectral memory versus isotropic noise. The existing ablations do not separate these two effects. VIKING has the spectral structure with one inflation level (735.2 MW), the γ-grid hedge has neither inflation nor a learned noise level (807.1 MW), and MELO has both (686.2 MW). The NYC validation year has a further problem. Appendix D reports that the March 2018 counts are about twice the neighbouring level, and 2018 is the validation year for VIKING, rolling OLS, and the c-selection study.

Beyond this, the evaluation covers two daily scalar streams whose largest shifts both come from COVID, with pools of five members. The strongest references, daily-refit tabular foundation models, outperform MELO on the electricity sub-window (628.9 vs. 697.2 MW), although at about a thousand times the cost.

This is an important concern because the comparison with VIKING provides the primary evidence for the benefits of inflation hedging over existing adaptive filters.

Questions for authors
  1. Can the authors compare MELO with MLpol over several VIKING instances that use different process-noise priors or random-walk variances? Alternatively, they could run VIKING with an initial noise level matched to the best MELO inflation. This would separate the benefit of hedging over the inflation scale from the benefit of the spectral memory structure, which VIKING already has.
  2. Does any deployed inflated cell satisfy the condition ε₀B_z² ≲ 2d/h of Remark A.6? If not, can the authors state explicitly that the tracking guarantee is linear in T for all inflated cells? Alternatively, can they provide a bound in which the inflation cost is sublinear for the inflation levels that perform best?
  3. How do the selected nuisance parameters of VIKING and rolling OLS, and the c-selection result in Appendix B.5, change when the anomalous March 2018 data are excluded or corrected?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.