- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PAVE for online time series forecasting under progressively avaliable supervision and evolving data distributions. The method distinguishes partial feedback from fully matured feedback. It further introduces version-aware capability validation to determine whether historical knowledge should be preserved, rebased or recovered. Experiments on four benchmarks report the best MSE. Ablations and mechanistic analyses provide additional evidence supporting the proposed framework.
Critical strengths
- The version-aware expert credit assignment is interesting, as it distinguishes historical expert responsibility from current capability validity, enabling adaptive preservation and recovery of expert knowledge.
- PAVE achieves competitive results on four benchmarks. The feedback-matched experiments and ablations further strengthen the empirical evidence.
Critical weaknesses
- Although ETTh2 and ETTm1 have different sampling frequencies, they share the same boundaries for their training and validation sets, consisting of 2880 and 720 observations, respectively. These differ substantially from the commonly used ETT protocal.
- Equation 10 measures alignment using randomly projected encoder representations, without considering changes in prediction head. An expert could exhibit high representation alignment while its forecasting performance deteriorates.
Questions for authors
- Why are the ETT datasets evaluated using the boundaries described in Appendix B.1 rather than the conventional time-based splits?
- Can the authors clarify why encoder representation alignment is sufficient to assess expert capability validity?
2 comments
Thank you for the additional experiments and clarifications. I will maintain my original score.
Response to Reviewer KaYu: Online ETT Splits and Encoder-Alignment Validity
We thank the reviewer for recognizing PAVE's version-aware expert credit assignment. We respond separately to the two weaknesses and two questions.
W1. Nonstandard ETT split boundaries
We acknowledge that our ETT splits differ from the conventional month-based protocol. Appendix B.1 specifies 2,880/720/10,800 observations for training/validation/online testing on both ETTh2 and ETTm1.
W2. Encoder alignment does not directly measure prediction-head changes
We agree that high encoder alignment cannot guarantee unchanged prediction-head performance. The 32-D sketch measures context-conditioned representational continuity between versions on the same input, not global predictive competence. When alignment is low, PAVE checks current loss to distinguish benign evolution from degradation.
Q1. Why use these ETT splits instead of conventional time-based splits?
The motivation is continual forecasting under progressive feedback: (i) limited initial training puts greater emphasis on adaptation than extensive offline pretraining; (ii) a long online stream allows sustained evaluation of evolving experts, delayed feedback, and knowledge recovery; and (iii) the same number of evaluation steps offers comparable online adaptation budgets—similar opportunities for progressive feedback, expert updates, and evidence revalidation. This controls stream-length differences, not differences in sampling frequency or calendar duration. All methods share identical chronological splits within each dataset, use training-only normalization, and receive no future targets early.
This is a task-specific online evaluation protocol, not a replacement for standard ETT splits; results are not directly comparable to published standard-split results. We have clarified the rationale in Appendix B.1.
Q2. Why is encoder alignment useful for assessing current capability validity?
Appendix E quantifies the possible false-retention case: 95.8% of 10,000 high-alignment records show no degradation, and 1.8% degrade by more than 10%. Adding a loss check on the high-alignment branch only marginally changes MSE: 0.756→0.754 (ETTm1-H48) and 0.509→0.508 (WTH-H48).
We also evaluated alternative proxies for historical-credit reversal on 600 records (H=48):
Given 84% reversal prevalence, AP should be interpreted relative to this base rate. The sketch performs best among these tested proxies, supporting its role as a lightweight historical-credit staleness signal, not a guarantee that every prediction-head change is captured. We have clarified this around Eq. (10).