acceptodds
Under review as a conference paper at ICLR 2027

Progressive Adaptation with Credit-Aware Experts for Online Time Series Forecasting

Abstract

Online time series forecasting (OTSF) requires continual adaptation to evolving data distributions, while supervision for multi-step forecasts becomes progressively available. Waiting for fully matured feedback delays adaptation, and historical expert credit can become stale as experts continue to evolve. We propose PAVE, a multi-expert framework that organizes online adaptation around feedback maturity and capability validity. As partial feedback arrives, PAVE rapidly adapts expert selection through a feedback-driven online routing correction and local responsibility supervision of a neural router. Once all targets of a forecasting record have matured, PAVE assigns sample-level responsibility from prediction-time expert outputs and validates this historical ownership against the experts' current capabilities. The resulting version-aware evidence determines whether a capability should be preserved, rebased, or recovered through responsibility-weighted subspace protection and targeted replay. Experiments on four multivariate forecasting benchmarks show that PAVE consistently outperforms strong online forecasting baselines, while ablation and diagnostic analyses support the effectiveness of progressive routing, version-aware credit assignment, and preservation–recovery.

Primary area:
learning on time series and dynamical systems
Keywords:
Online Time Series Forecasting, Continual Learning, Mixture of Experts, Delayed Feedback, Concept Drift, Adaptive Routing

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_KaYuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PAVE for online time series forecasting under progressively avaliable supervision and evolving data distributions. The method distinguishes partial feedback from fully matured feedback. It further introduces version-aware capability validation to determine whether historical knowledge should be preserved, rebased or recovered. Experiments on four benchmarks report the best MSE. Ablations and mechanistic analyses provide additional evidence supporting the proposed framework.

Critical strengths
  1. The version-aware expert credit assignment is interesting, as it distinguishes historical expert responsibility from current capability validity, enabling adaptive preservation and recovery of expert knowledge.
  2. PAVE achieves competitive results on four benchmarks. The feedback-matched experiments and ablations further strengthen the empirical evidence.
Critical weaknesses
  1. Although ETTh2 and ETTm1 have different sampling frequencies, they share the same boundaries for their training and validation sets, consisting of 2880 and 720 observations, respectively. These differ substantially from the commonly used ETT protocal.
  2. Equation 10 measures alignment using randomly projected encoder representations, without considering changes in prediction head. An expert could exhibit high representation alignment while its forecasting performance deteriorates.
Questions for authors
  1. Why are the ETT datasets evaluated using the boundaries described in Appendix B.1 rather than the conventional time-based splits?
  2. Can the authors clarify why encoder representation alignment is sufficient to assess expert capability validity?
2 comments
AuthorsOpenReview ↗

Response to Reviewer KaYu: Online ETT Splits and Encoder-Alignment Validity

We thank the reviewer for recognizing PAVE's version-aware expert credit assignment. We respond separately to the two weaknesses and two questions.

W1. Nonstandard ETT split boundaries

We acknowledge that our ETT splits differ from the conventional month-based protocol. Appendix B.1 specifies 2,880/720/10,800 observations for training/validation/online testing on both ETTh2 and ETTm1.

W2. Encoder alignment does not directly measure prediction-head changes

We agree that high encoder alignment cannot guarantee unchanged prediction-head performance. The 32-D sketch measures context-conditioned representational continuity between versions on the same input, not global predictive competence. When alignment is low, PAVE checks current loss to distinguish benign evolution from degradation.

Q1. Why use these ETT splits instead of conventional time-based splits?

The motivation is continual forecasting under progressive feedback: (i) limited initial training puts greater emphasis on adaptation than extensive offline pretraining; (ii) a long online stream allows sustained evaluation of evolving experts, delayed feedback, and knowledge recovery; and (iii) the same number of evaluation steps offers comparable online adaptation budgets—similar opportunities for progressive feedback, expert updates, and evidence revalidation. This controls stream-length differences, not differences in sampling frequency or calendar duration. All methods share identical chronological splits within each dataset, use training-only normalization, and receive no future targets early.

This is a task-specific online evaluation protocol, not a replacement for standard ETT splits; results are not directly comparable to published standard-split results. We have clarified the rationale in Appendix B.1.

Q2. Why is encoder alignment useful for assessing current capability validity?

Appendix E quantifies the possible false-retention case: 95.8% of 10,000 high-alignment records show no degradation, and 1.8% degrade by more than 10%. Adding a loss check on the high-alignment branch only marginally changes MSE: 0.756→0.7540.756\to0.754 (ETTm1-H48) and 0.509→0.5080.509\to0.508 (WTH-H48).

We also evaluated alternative proxies for historical-credit reversal on 600 records (H=48H=48):

ProxyROC-AUCAP
32-D capability sketch0.7930.934
Output discrepancy0.7420.908
Loss-only relative change0.6570.877
Sampled parameter distance0.5910.849

Given 84% reversal prevalence, AP should be interpreted relative to this base rate. The sketch performs best among these tested proxies, supporting its role as a lightweight historical-credit staleness signal, not a guarantee that every prediction-head change is captured. We have clarified this around Eq. (10).

Thank you for the additional experiments and clarifications. I will maintain my original score.

Reviewer_ZDG6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Progressive Adaptation with Version-aware Experts (PAVE) for online multivariate time-series forecasting with progressively revealed supervision. Partial feedback updates a neural router and an online routing correction, while fully observed forecasting records establish historical expert responsibility using stored prediction-time outputs. PAVE subsequently compares historical and current expert representations and predictive losses to decide whether associated capabilities should be preserved, rebased, or recovered. Experiments cover four datasets and three forecasting horizons. PAVE reports the best or tied-best MSE across all twelve settings, with a mean relative improvement of approximately 5.7% over the strongest baseline in each setting. Validation includes component ablations, progressive-feedback controls, capacity-matched comparisons, mechanism diagnostics, and a controlled recurring-concept experiment. The paper also provides a regret analysis for an idealized routing component.

Critical strengths
  1. A well-motivated treatment of delayed evidence for evolving experts.

The paper distinguishes two issues that are easily conflated: incomplete feedback cannot establish full-window responsibility, and full-window historical responsibility may no longer describe the current expert. Connecting progressive routing with subsequent validation of historical ownership provides a coherent approach to online adaptation. The reported increase in historical-credit reversals with forecasting horizon makes this motivation empirically relevant.

  1. The evaluation goes beyond an unmatched ensemble comparison.

The paper includes progressive-feedback controls, a four-expert control with matched initialization and replay/update budget, and a controlled recurring-concept experiment. These experiments address meaningful alternative explanations for the gains. In particular, the capacity-matched comparison shows larger improvements at longer horizons, while the recurrence experiment supports a practical benefit from recovery after an intervening regime.

Critical weaknesses
  1. The central capability-validity criterion is insufficiently isolated from a simpler prediction-loss criterion.

Equation (11) treats high representation alignment as capability retention irrespective of current predictive degradation. However, the sketch measures pooled encoder representations, while both the encoder and prediction head continue to change. A prediction-head change can therefore degrade forecasting without substantially changing the sketch. Appendix E usefully acknowledges and quantifies this failure mode, but its pooled diagnostic does not establish the decision value of alignment relative to directly comparing current and reference losses on the same matured records.

The existing ablations do not resolve this question. "Without Version Awareness" sets alignment to one, which also deactivates direction-aware classification and rebasing and prevents the described harmful-drift admission into Recovery. Its degradation therefore does not isolate the contribution of representation-based validation. A loss-only validation control retaining the remaining memory, rebasing, protection, and recovery machinery is particularly important because version-aware validation is the paper’s central claimed contribution.

  1. The strongest evidence for the full system does not yet establish a robust advantage under jointly matched evaluation conditions.

The main comparison uses adapted baseline implementations, and progressive-feedback matching is reported for only three baselines on three settings. This matters especially for methods such as DSOF, whose original motivation already includes incomplete feedback. The baseline and PAVE online learning-rate configurations also differ substantially, without a clearly described comparable validation search.

The capacity-matched control is valuable, but Table 10 reports only one seed and removes several mechanisms together. Moreover, matching replay/update counts does not match computational cost: on ECL-48, PAVE takes 165 ms per origin versus 125 ms for Matched MoE and 16–52 ms for the reported individual baselines. These results support the effectiveness of the assembled system, but leave uncertainty about how much advantage remains over well-tuned alternatives with comparable feedback access and computational resources. This is consequential because several gains over the strongest baselines are modest.

Questions for authors
  1. Core mechanism: How does PAVE compare with a loss-only validity rule that compares current and reference forecasting losses for every responsible matured record, while retaining the same responsibility assignment, rebasing, Stable/Recovery capacities, subspace protection, and replay budget? This would directly affect my assessment of whether representation alignment provides a distinct benefit.

  2. Reliability of the retention rule: For Appendix E, how are the 10,000 high-alignment records distributed across datasets, horizons, seeds, and stream positions? Does the sample include all eligible high-alignment owners before memory filtering? Please clarify whether high-alignment degradation is concentrated in particular settings and whether applying the loss check to this branch changes forecasting performance.

  3. Comparison robustness: How were baseline online learning rates and other adaptation hyperparameters selected, and were their validation search budgets comparable to PAVE’s? Can the capacity-matched comparison be reported across the same three seeds as the main results, and can you clarify whether DSOF’s main-table implementation retains its native incomplete-feedback learning mechanism? These answers would materially affect my confidence in the reported improvements.

1 comment
AuthorsOpenReview ↗

Response to Reviewer ZDG6: Validity and Fair Comparison

We thank the reviewer for the comments. We address both weaknesses and three questions below.

W1. Alignment versus loss-only validity

w/o Version Awareness removes multiple mechanisms and cannot isolate alignment. We thus test two 5%-tolerant loss-only controls and a high-alignment loss guard (Q1).

W2. Jointly matched feedback, capacity, and compute

We extended PF-matched tests to 12 settings (Q3) and capacity controls to three seeds, adding an ECL-H48-calibrated time-matched four-expert MoE:

MethodETTm1-H48WTH-H48ECL-H48
Uniform MoE0.887±0.0100.651±0.0089.103±0.084
Matched MoE0.844±0.0130.615±0.0058.771±0.072
Time-matched MoE0.815±0.0080.585±0.0078.470±0.061
PAVE0.756±0.0040.509±0.0038.207±0.096

All use 16.565M parameters. On ECL-H48 (A100), Matched/Time-matched/PAVE require 128/161/166 ms/origin and 0.71/0.74/0.75 GB. PAVE achieves lower MSE at comparable compute.

Q1. Direct comparison with loss-only validity

We compare PAVE with two 5%-tolerant loss-only controls and a high-alignment loss guard (MSE mean ±\pm SD):

Validity ruleETTm1-H48WTH-H48
PAVE (alignment-first)0.756±0.0040.756\pm0.0040.509±0.0030.509\pm0.003
Tolerant loss-only gate (δ=5%\delta=5\%)0.777±0.0060.777\pm0.0060.527±0.0050.527\pm0.005
Pure tolerant loss-only (δ=5%\delta=5\%)0.780±0.0070.780\pm0.0070.523±0.0060.523\pm0.006
PAVE + high-alignment loss guard0.754±0.0050.754\pm0.0050.508±0.0040.508\pm0.004

Tolerant loss-only gate replaces alignment-based validity decisions with a 5%-tolerant loss test but retains alignment-dependent evidence weighting. Pure tolerant loss-only removes alignment from both decisions and weighting. Both preserve prediction-time credit, rebasing, Stable/Recovery memory, protection, and replay.

Both loss-only controls perform worse; the guard changes MSE marginally. This supports alignment's additional decision value, without guaranteeing unchanged prediction-head behavior.

Q2. Distribution and reliability of high-alignment retention

Appendix E's 10,000 records were stratified-randomly sampled from eligible high-alignment historical owners before memory admission, not admitted memory alone. Counts with loss degradation $>$10%:

DimensionStrata in orderSamplesDegradation $>$10%
DatasetETTh2 / ETTm1 / WTH / ECL2500 / 2500 / 2500 / 250047 / 43 / 38 / 52
HorizonH1 / H24 / H483332 / 3332 / 333623 / 59 / 98
Seed0 / 1 / 23333 / 3333 / 333457 / 60 / 63
Stream quartileQ1 / Q2 / Q3 / Q42516 / 2469 / 2532 / 248333 / 43 / 48 / 56

Overall, 420 (4.2%) show any degradation, 180 (1.8%) exceed 10%, and 84 (0.84%) exceed 20%. The $>$10% rate increases across horizons (0.69%/1.77%/2.94%) and stream quartiles (1.31%/1.74%/1.90%/2.26%), but remains low overall. The guard in Q1 assesses its forecasting impact. Pre-admission sampling is clarified in Appendix E.

Q3. LR fairness, DSOF feedback, and three-seed robustness

Compute limits prevented exhaustive LR tuning originally. We now apply the same validation-only LR multiplier grid {0.1,0.3,1,3,10}\{0.1,0.3,1,3,10\} to PAVE and seven baselines over three test seeds. PAVE versus the best tuned competitor:

  • ETTh2-H24: 2.407±0.0082.407\pm0.008 vs. 2.431±0.2052.431\pm0.205.
  • WTH-H24: 0.425±0.0010.425\pm0.001 vs. 0.435±0.0010.435\pm0.001.
  • ETTm1-H48: 0.756±0.0040.756\pm0.004 vs. 0.761±0.0100.761\pm0.010.

DSOF: Table 1 preserves native Fast/Slow incomplete-feedback (Partial-TD) learning. In Table 2, DSOF+PF feeds newly observed horizon-level partial losses to Fast/Slow and temporal-distillation updates. DynaME+PF and ER+PF receive the same causal targets. We match feedback access, not internal algorithms.

PF-matched results across 12 settings, with reselected baseline and PAVE LRs (MSE mean ±\pm SD):

SettingPAVEDSOF+PFDynaME+PFER+PF
ETTh2-H10.355±0.0050.754±0.0440.445±0.0650.527±0.013
ETTh2-H242.407±0.0082.843±0.1322.422±0.1272.479±0.161
ETTh2-H484.919±0.0745.337±0.2485.136±0.1734.982±0.152
ETTm1-H10.085±0.0010.114±0.0010.085±0.0020.091±0.002
ETTm1-H240.557±0.0160.566±0.0080.598±0.0110.577±0.009
ETTm1-H480.756±0.0040.765±0.0090.766±0.0080.792±0.011
WTH-H10.156±0.0000.209±0.0000.186±0.0010.171±0.002
WTH-H240.425±0.0010.527±0.0030.434±0.0030.432±0.002
WTH-H480.509±0.0030.584±0.0050.519±0.0040.607±0.004
ECL-H12.301±0.0452.561±0.0523.465±0.0782.371±0.062
ECL-H246.996±0.0757.432±0.10514.604±0.1897.521±0.129
ECL-H488.207±0.0968.744±0.18315.289±0.2178.638±0.137

PAVE achieves 11 wins and one tie, with 4.3% average relative improvement over the strongest PF-matched competitor.

Reviewer_ffvvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PAVE addresses online multi-step time series forecasting where supervision is delayed and progressively revealed: the h-th target of a forecast issued at origin o only becomes observable at o+h. The framework assigns distinct roles to feedback of different maturity within a mixture-of-experts architecture. Partial feedback drives fast adaptation through (i) an online log-weight correction z added to a neural router's contextual prior, and (ii) confidence-weighted local-responsibility supervision of the router. Once a forecasting record fully matures, PAVE assigns sample-level expert responsibility from stored prediction-time outputs, then revalidates this historical ownership against the expert's current version: a lightweight capability sketch plus a loss-direction check classify each responsible expert's evolution as retained, beneficial/neutral, or harmful. Validated stable evidence builds responsibility-weighted predictive subspaces protecting expert prediction heads, while harmful evidence triggers targeted replay with sketch distillation. An idealized delayed-routing regret bound separates neural-prior mismatch from delayed-feedback cost.

Critical strengths
  1. A well-articulated and genuinely under-appreciated problem structure. The observation that multi-step online supervision arrives progressively—and that early partial evidence and fully matured evidence should play different adaptation roles—is simple, correct, and largely unexploited by prior OTSF work. Pairing this with the insight that matured credit can be stale because continuously updated experts have moved on is a genuinely fresh framing.
  2. The version-aware validation step is the paper's real contribution, and it is well designed. Distinguishing historical ownership (computed from prediction-time outputs, so later updates cannot retroactively rewrite credit) from current capability validity, and routing the outcome to preserve/rebase/recover, is a clean mechanism. The asymmetry—alignment as the primary signal, loss direction only after substantial version change—is thoughtfully justified and empirically checked (Appendix E: among high-alignment cases, 95.8% show no degradation).
  3. The evidence lifecycle is internally consistent. The same version-aware criterion governs memory admission, periodic reassessment, and post-replay re-evaluation, so stale evidence does not accumulate unboundedly—a common failure of replay-based continual learning.
Critical weaknesses
  1. Baseline numbers exhibit worrying variance asymmetry. Several baselines have huge standard deviations where PAVE has near-zero ones (DSOF ETTh2-24: 3.552±0.443 vs. PAVE 2.408±0.008; PROCEED ETTh2-48: 17.438±0.831; OneNet WTH-24: 1.362±1.261), and some baseline results look degenerate rather than merely worse (FSNet ETTm1-48: 1.416±0.067 vs. PAVE 0.759). This pattern suggests baselines may be inadequately tuned or unstable under the protocol rather than genuinely weaker. The feedback-matched control addresses feedback access but not adaptation quality. Given that the headline claim is "best or tied-best in all 12 settings," the credibility of the comparison matters.
  2. The capability sketch is a coarse proxy carrying a strong interpretation. "Capability validity" is operationalized as cosine alignment of a random 320→32 projection of pooled encoder features on a single context. Low alignment triggers a loss-direction check, which rescues the mechanism, but the framing ("capability", "version") promises more semantics than a representation-similarity heuristic delivers. Some analysis of what the sketch actually tracks (vs. e.g. parameter distance or output-KD on a probe set) would ground it.
  3. Typos Duplicated sentence in Sec. 4.2 ("We initialize the performance reference..." appears twice with slightly different wording, ll. 564–575), suggesting a leftover from revision.
Questions for authors

See weaknesses above

1 comment
AuthorsOpenReview ↗

Response to Reviewer ffvv: Baseline Robustness and Capability-Sketch Interpretation

We thank the reviewer for the encouraging assessment of PAVE's evidence lifecycle and version-aware validation. The reviewer raises three weaknesses, which we address individually below.

W1. Baseline stability, variance asymmetry, and LR fairness

We agree that the large baseline variances warrant scrutiny: online LR changes subsequent learning trajectories. The original comparison used method-specific reference configurations; computational limits prevented a fine-grained LR search across baselines. We therefore applied the same validation-only relative LR grid {0.1,0.3,1,3,10}\{0.1,0.3,1,3,10\} to PAVE and all seven baselines on three representative settings and evaluated the selected configurations over three test seeds:

SettingMethodOriginal MSELR-tuned MSE
ETTh2-H24DSOF3.552±0.4433.552\pm0.4433.427±0.1683.427\pm0.168
PAVE2.408±0.0082.408\pm0.0082.407±0.0082.407\pm0.008
WTH-H24OneNet1.362±1.2611.362\pm1.2610.722±0.0650.722\pm0.065
PAVE0.425±0.0010.425\pm0.0010.425±0.0010.425\pm0.001
ETTm1-H48FSNet1.416±0.0671.416\pm0.0670.924±0.0460.924\pm0.046
PAVE0.759±0.0040.759\pm0.0040.756±0.0040.756\pm0.004

Retuning improves both mean MSE and variability for the cited baselines. Equal relative LR grids preserve appropriate method-specific absolute LR scales. PAVE still has the lowest mean MSE on all three settings after the common sweep. The strongest tuned competitors are DynaME (2.431±0.2052.431\pm0.205) on ETTh2-H24, ER (0.435±0.0010.435\pm0.001) on WTH-H24, and PROCEED (0.761±0.0100.761\pm0.010) on ETTm1-H48. We have documented the selection protocol and results.

W2. What the capability sketch measures and whether alignment misses head-level degradation

An expert version is the state of the same continuously updated expert at a given time, not a separately stored model. On historical context XoX_o, the sketch is so,e=norm(Rehe(Xo))s_{o,e}=\mathrm{norm}(R_e h_e(X_o)), where heh_e is the pooled 320-D encoder representation and ReR_e is a fixed, expert-specific 320→32320\to32 Gaussian projection. Reference and current sketches are compared on the same context, with alignment ao,e=1+cos⁡(so,eref,so,ecur)2a_{o,e}=\frac{1+\cos(s_{o,e}^{\mathrm{ref}},s_{o,e}^{\mathrm{cur}})}{2}.

The sketch measures representational continuity: a cheap proxy for the applicability of historical expert evidence, not proof of unchanged prediction-head behavior or global predictive competence. The fixed projection provides a consistent measurement basis on the same historical context, so the signal is record-specific, not global. Low alignment therefore triggers a loss-direction check before evolution is classified as benign/neutral or harmful. High alignment can miss head-only degradation; Appendix E reports 95.8% of 10,000 high-alignment historical-owner records without loss degradation (only 1.8% have degradation above 10%). To evaluate that failure mode operationally, we also tested an extra loss guard on the high-alignment branch. It changes MSE from 0.756±0.0040.756\pm0.004 to 0.754±0.0050.754\pm0.005 (ETTm1-H48), and from 0.509±0.0030.509\pm0.003 to 0.508±0.0040.508\pm0.004 (WTH-H48). Adding the high-alignment loss guard yields only modest MSE improvements in both settings.

We also compared proxies for historical-credit reversal (the prediction-time credited expert is no longer the current lowest-loss expert) on 600 records at H=48H=48:

ProxyROC-AUCAP
32-D capability sketch0.7930.934
Output discrepancy0.7420.908
Loss-only relative change0.6570.877
Sampled parameter distance0.5910.849

The positive class is historical-credit reversal, not simply increased absolute loss of the old owner. Reversal prevalence is 84% (504/600), so AP should be read against that high base rate. The sketch has the best ROC-AUC and AP among these tested proxies, supporting its use as a record-specific staleness signal, not a complete capability measure. Appendix E also reports 420/10,000 records with any loss degradation and 84/10,000 above 20%; the high-alignment branch is empirically reliable, not infallible. We have clarified this intended scope.

W3. Duplicated sentence in Sec. 4.2

We have removed the duplicated sentence in Sec. 4.2.

Reviewer_6htcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes PAVE, a mixture-of-experts method for online multi-step forecasting. The targets of each forecast arrive one step at a time, under a leakage-free protocol. PAVE uses four continuously updated experts (two FSNet and two FSNet-Time) and treats the feedback differently depending on how mature it is. Partial feedback is used for fast routing: an exponentiated-gradient correction of the routing weights plus a local credit loss for the neural router. Once all targets of a forecast are observed, the record is assigned to the expert that predicted it best at prediction time. Since the experts keep changing, PAVE then checks if this credit is still valid. It does so by comparing a random projection of the expert representation at prediction time and now. Records whose expert did not change, or changed without getting worse, go to a Stable memory, which is used to protect a gradient subspace of the prediction head. Records whose expert changed and got worse go to a Recovery memory and are replayed. A regret bound is given for an idealized version of the router. Experiments on four datasets and three horizons show the best or tied-best MSE in all 12 settings.

Critical strengths
  • Very relevant problem. The leakage-free protocol, where the targets arrive progressively, is the realistic setting for multi-step online forecasting.
  • The intuition is interesting. Partial feedback is timely but noisy. Fully matured credit is reliable but can become stale, because the experts keep changing.
  • The experimental evaluation is wide.
Critical weaknesses
  • My largest worry is that the method is maybe too complex. PAVE is a stack of a large number of different components, whose extension is impressive looking at Figure 1. This makes PAVE more a final engineered product than a proposed methodological advancement for a scientific paper. For example, the huge amount of hyperparameters (Table 4) makes very difficult to actually learn something from this work, as it makes actually hard to understand if we are looking at genuine scientific advancements or simply good hyperparameter tuning.

  • The experimentation is wide, but suffer from a fundamental weakness. Each method has a different number of hyperparameters and no common strategy to select them fairly. I would argue that the learning rate is one of the most important component in online learning, as it determines the entire future learning path on the data stream. Having different strategies to select it (or not having any strategy at all) makes the experiments very weak.

Questions for authors
  • Can you please describe how the hyperparameters of PAVE and of the baselines were selected? Can you also show the sensitivity to the main ones?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 6htc: Methodological Coherence and Fair Hyperparameter Selection

We thank the reviewer for highlighting progressive feedback and for raising concerns about complexity and tuning. We respond to the two weaknesses and the question in the review's order.

W1. Method complexity and methodological coherence

PAVE follows one evidence lifecycle: feedback maturity determines when evidence can be used, while capability validity determines how historical evidence should affect continuously evolving experts. Partial feedback adapts routing without premature long-term ownership; matured records establish historical responsibility using prediction-time outputs; version-aware validation then preserves, rebases, or recovers that evidence. Stable-subspace protection and targeted replay implement these outcomes. These are connected decisions, not independent performance tricks. Crucially, prediction-time responsibility and current capability validity answer different questions: who earned credit when the forecast was issued, and whether that evidence remains safe to use after the credited expert has evolved. The memory and protection/recovery branches are consequences of this distinction, rather than separate objectives. In the original H=48H=48 ablations, removing online correction, version awareness, and Recovery increases MSE by 12.9%, 16.6%, and 11.0%, respectively (averaged across datasets).

Table 4 combines architecture, fixed implementation choices, and tunable parameters; we have clarified their different roles.

W2. Fair baseline hyperparameter and learning-rate selection

We agree that LR affects the entire online adaptation trajectory. Original baselines used method-specific reference configurations; compute limits prevented fine-grained, uniform LR searches. We therefore swept the same validation-only relative LR grid {0.1,0.3,1,3,10}\{0.1,0.3,1,3,10\} around each reference LR for PAVE and all seven baselines on three settings. Selected LRs were frozen for three test seeds (MSE mean ±\pm SD):

MethodETTh2-H24 LRMSEWTH-H24 LRMSEETTm1-H48 LRMSE
PAVE1×2.407±0.0082.407\pm0.0081×0.425±0.0010.425\pm0.0010.3×0.756±0.0040.756\pm0.004
DSOF0.3×3.427±0.1683.427\pm0.1681×0.531±0.0030.531\pm0.0031×0.763±0.0110.763\pm0.011
DynaME1×2.431±0.2052.431\pm0.2051×0.436±0.0030.436\pm0.0031×0.767±0.0100.767\pm0.010
ER1×2.530±0.2442.530\pm0.2440.3×0.435±0.0010.435\pm0.0011×0.811±0.0120.811\pm0.012
FSNet0.1×2.937±0.1482.937\pm0.1480.3×0.608±0.0030.608\pm0.0030.3×0.924±0.0460.924\pm0.046
OneNet0.3×3.259±0.3043.259\pm0.3040.3×0.722±0.0650.722\pm0.0650.3×1.033±0.0141.033\pm0.014
PROCEED0.3×5.015±0.4545.015\pm0.4541×0.452±0.0010.452\pm0.0011×0.761±0.0100.761\pm0.010
PatchTST-DGrad0.3×3.253±0.2163.253\pm0.2160.3×0.487±0.0020.487\pm0.0020.3×0.931±0.0080.931\pm0.008

Tuning helps several baselines (DSOF ETTh2-H24: 3.552→3.4273.552\to3.427; FSNet ETTm1-H48: 1.416→0.9241.416\to0.924), which we explicitly acknowledge. The relative multipliers give each method the same LR search grid around its own reference scale, rather than imposing an unsuitable common absolute LR. This controls the selection procedure for the central online hyperparameter, although it does not equalize every model-specific adaptation hyperparameter. The strongest tuned competitors are DynaME (2.431±0.2052.431\pm0.205), ER (0.435±0.0010.435\pm0.001), and PROCEED (0.761±0.0100.761\pm0.010), respectively. PAVE retains the lowest mean MSE in all three settings. Validation determines the LR before testing; the test stream never enters LR selection. We report the validation-based search procedure and controlled results here, and incorporate the refined forecasting results into the revision.

Q1. How were hyperparameters selected, and how sensitive are the results?

Key PAVE parameters were selected by grid search on chronological validation data, without selecting on test targets. These tests assess sensitivity, rather than claiming exhaustive tuning of all Table 4 entries. We conducted three-seed, one-factor-at-a-time sensitivity tests (MSE mean ±\pm SD; bold = default):

SettingParameterValueMSE
ETTm1-H48Alignment τa\tau_a0.60.781±0.0090.781\pm0.009
0.70.758±0.0050.758\pm0.005
0.80.756±0.0040.756\pm0.004
0.90.761±0.0050.761\pm0.005
Stable/Recovery capacity8/80.795±0.0100.795\pm0.010
16/160.764±0.0060.764\pm0.006
32/320.756±0.0040.756\pm0.004
64/640.755±0.0060.755\pm0.006
Subspace λsub\lambda_{\mathrm{sub}}10210^20.788±0.0100.788\pm0.010
10310^30.760±0.0050.760\pm0.005
10410^40.756±0.0040.756\pm0.004
10510^50.768±0.0070.768\pm0.007
LR controllerAdaptive0.756±0.0040.756\pm0.004
Fixed0.771±0.0060.771\pm0.006
WTH-H48Alignment τa\tau_a0.60.522±0.0060.522\pm0.006
0.70.510±0.0040.510\pm0.004
0.80.509±0.0030.509\pm0.003
0.90.511±0.0030.511\pm0.003

On ETTm1-H48, the neighboring thresholds 0.7 and 0.9 yield MSEs of 0.758 and 0.761, respectively, close to the default 0.756, while the markedly lower threshold 0.60.6 gives 0.7810.781. Increasing each memory from 32 to 64 changes MSE only from 0.7560.756 to 0.7550.755; reducing each to 8 yields 0.7950.795. The default is not an isolated optimum; sensitivity outside reasonable ranges remains.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary (Part 1/2): Scientific Contribution and Main Evidence

Dear Area Chair,

We thank the reviewers for their detailed feedback. We would like to summarize the central contribution of PAVE and the evidence supporting it, including the new analyses incorporated into the revision.

1. The scientific problem is the lifecycle of evidence in multi-step online forecasting. At forecast origin o, the h-th target becomes available only at o+h. Early partial feedback is timely but insufficient to establish full-window expert responsibility. When that window eventually matures, however, the expert may already have undergone many online updates; the expert that deserved credit at prediction time need not retain that capability. Thus feedback maturity (when evidence is reliable for a particular role) and capability validity (whether historical evidence still applies to the current expert) are distinct and coupled challenges.

2. PAVE addresses these challenges with a unified evidence lifecycle rather than independent add-ons. Its partial-feedback stage adapts expert routing through (i) a feedback-driven log-weight correction based on stored prediction-time outputs and (ii) confidence-weighted local responsibility supervision of a neural router. Only fully matured records establish sample-level historical expert responsibility, also from prediction-time outputs, so subsequent learning cannot retroactively reassign credit. For each historically responsible expert, a fixed-projection, record-specific capability sketch compares its reference and current representations. High alignment retains the evidence; low alignment invokes a predictive-loss direction check to distinguish benign/neutral evolution (rebase the reference) from harmful degradation (targeted Recovery). Validated Stable evidence builds responsibility-weighted, expert-specific prediction-head subspaces, while degraded evidence supports targeted replay. Both memories are periodically re-evaluated as the experts evolve. This separation between historical ownership and current validity is the methodological contribution.

3. The empirical gains persist beyond an unmatched ensemble comparison. Across ETTh2, ETTm1, Weather, and Electricity, and horizons H=1/24/48, PAVE achieves the best or tied-best mean MSE in all 12 settings (11 wins, one tie), averaging 5.7% lower MSE than the strongest competing baseline in each setting (Table 1). To isolate early target access, we extended the progressive-feedback control to all 12 settings for DSOF, DynaME, and ER, providing the same causal horizon-level feedback and validation-reselected baseline LRs while retaining each baseline's own update rule. PAVE again obtains 11 wins and one tie, with 4.3% average reduction relative to the strongest feedback-matched baseline (Table 2). In particular, Table 1's DSOF preserves its native Fast/Slow incomplete-feedback (Partial-TD) mechanism.

4. Ablations and diagnostics explain why the lifecycle matters. At H=48, removing online correction, version awareness, or Recovery increases MSE by 12.9%, 16.6%, and 11.0%, respectively (Figure 2). Historical-credit reversal grows from 21.0% (H=1) to 44.7% (H=24) and 84.1% (H=48) in the sampled diagnostics (Figure 3), supporting the need to revalidate matured credit. Across 120 sampled Recovery events, median normalized loss falls from 1.47 to 1.12, with 97.5% of events improving. In a controlled A1→B→A2 recurrence experiment, Recovery shortens reacquisition time from 32.0 to 19.0 origins and reduces cumulative excess error from 3.59 to 1.73 (Table 10). These results support not only aggregate accuracy but the proposed failure mode and targeted response.

The delayed-routing analysis (Theorem 1 and Appendix C) provides a supporting regret decomposition for an idealized dense, fixed-expert routing core; we do not claim a no-regret guarantee for the complete evolving-expert system.

AuthorsOpenReview ↗

Summary (Part 2/2): Responses to the Main Review Concerns

The revision and rebuttals address the following review concerns.

Methodological coherence and hyperparameter fairness (6htc; also ffvv and ZDG6). We clarified that Table 4 contains architectural choices, fixed implementation settings, and tunable parameters. Key PAVE parameters were selected using chronological validation data without test-target selection. We applied the same validation-only relative LR multiplier grid {0.1, 0.3, 1, 3, 10} around each method's reference LR to PAVE and all seven baselines on three representative settings, then froze the chosen configurations for three test seeds. Retuning helps several baselines; PAVE retains the lowest mean MSE in each setting. These experiments match the LR search procedure, not every method-specific adaptation-hyperparameter budget. Three-seed sensitivity analyses examine alignment threshold, Stable/Recovery capacity, subspace strength, and the adaptive LR controller (Table 13); nearby configurations perform similarly, while larger deviations can degrade accuracy.

Whether representation alignment adds value beyond a loss-only rule (ZDG6). Because the original "w/o Version Awareness" ablation removes several operations together, we added two focused 5%-tolerant loss-only controls with the remaining evidence lifecycle retained. One replaces validity decisions with loss comparisons but retains alignment-based evidence weighting; the other removes alignment from both decisions and weighting. On ETTm1-H48 and WTH-H48, PAVE achieves 0.756/0.509 MSE, versus 0.777/0.527 for the tolerant gate and 0.780/0.523 for pure tolerant loss-only (Table 14; means over three seeds). This supports alignment's incremental decision value in these settings, not a universal validity guarantee.

Sketch semantics and the risk of head-only degradation (ffvv, KaYu, ZDG6). We now explicitly characterize the fixed 320→32 projection as a record-conditioned representation-continuity proxy, not a guarantee that the prediction head remains unchanged. Appendix E samples 10,000 eligible high-alignment historical-owner records before memory admission, stratified by dataset, horizon, seed, and stream position. Of these, 95.8% show no loss increase; 1.8% exceed 10% degradation and 0.84% exceed 20%. An additional high-alignment loss guard changes MSE only marginally (0.756→0.754 and 0.509→0.508 on the two H48 settings). For identifying historical-credit reversal among 600 H48 diagnostic records, the sketch achieves ROC-AUC 0.793 / AP 0.934, ahead of output discrepancy (0.742 / 0.908) and two other tested proxies (Table 15). As reversal prevalence is 84%, the AP is interpreted against that base rate. These are diagnostic results, not a universal capability certificate.

Capacity, update budget, and runtime (ZDG6). We extended four-expert controls to three seeds, with the same 16.565M total parameters. On ETTm1-H48 / WTH-H48 / ECL-H48, PAVE obtains 0.756 / 0.509 / 8.207 MSE, versus 0.815 / 0.585 / 8.470 for a time-matched MoE using FIFO replay (Table 11). Its replay budget was calibrated only on ECL-H48 validation and kept fixed elsewhere. On ECL-H48 (A100), the time-matched MoE and PAVE require 161 vs 166 ms/origin (Table 12). Thus capacity or compute alone is insufficient to explain these results; exact runtime parity elsewhere is not claimed.

ETT splits and current discussion status (KaYu). Appendix B.1 now explains the task-specific chronological ETT protocol (2,880/720/10,800 training/validation/online observations). The extended online segment tests adaptation under progressive feedback; these results should not be directly compared with published standard-split ETT results. All compared methods use identical splits within each dataset and training-only normalization. After reviewing the additional experiments and clarifications, KaYu explicitly stated on November 18 that they would maintain their original score (3). No other reviewer score change is confirmed.

In summary, the revision strengthens the paper at three levels: a coherent scientific distinction between when historical evidence becomes reliable and whether its credited capability remains valid; evidence from feedback-, capacity-, and loss-only controls; and explicit boundaries on the claims. We hope this consolidated evidence assists the Area Chair in evaluating the paper's contribution and the remaining limitations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.