- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Continual Competence Routing (CCR) to select among frozen time-series imputers under distribution drift. CCR uses multi-scale memories of sparse reconstruction feedback to track changing model competence.
Critical strengths
W1. The method lacks an overview diagram. Multi-window estimation, weight updates, and certification-based switching are described in separate subsections, making it difficult to follow how these components interact within the complete decision process.
W2. The experimental comparisons are not sufficiently convincing. CCR-MS underperforms static selection in some settings and does not consistently improve over fixed-memory routing, leaving the necessity of multi-scale routing unclear. Moreover, comparisons are largely restricted to static, fixed-memory, and discounted baselines, without representative alternative online model selection methods evaluated under the same feedback conditions.
Critical weaknesses
W1. The method lacks an overview diagram. Multi-window estimation, weight updates, and certification-based switching are described in separate subsections, making it difficult to follow how these components interact within the complete decision process.
W2. The experimental comparisons are not sufficiently convincing. CCR-MS underperforms static selection in some settings and does not consistently improve over fixed-memory routing, leaving the necessity of multi-scale routing unclear. Moreover, comparisons are largely restricted to static, fixed-memory, and discounted baselines, without representative alternative online model selection methods evaluated under the same feedback conditions.
Questions for authors
See Weakness.
6 comments
4. Formal justification for adaptive memory.
The original theory established freshness recovery using a known-reset estimator. The revision substantially advances this analysis through CCR-GL, a causal adaptive selector with algorithm-specific guarantees.
Theorem 3 establishes a directional oracle inequality: the routing excess is at most three times the best available sum of directional statistical uncertainty and transported drift/approximation bias across candidate horizons.
Thus CCR-GL adapts to the estimation--drift trade-off without knowing reset times or variation budgets.
Under conditional excitation, Theorem 4 establishes switching, variation-budget and partition-adaptive dynamic-regret bounds. Proposition 5 gives matching information-theoretic dependence on horizon, audit probability, and drift, up to logarithms in the exact, fixed-dimensional setting.
These results provide a formal justification for adaptation under unknown drift, rather than requiring adaptive memory to dominate fixed windows in every finite-sample experiment. The synthetic evaluations explicitly retain cases where fixed windows outperform formal GL.
In conclusion, the revision establishes the mathematical value of adaptive memory under unknown drift and isolates its empirical contribution through matched comparisons. Together with the unified presentation and stronger online baselines, these advances directly address the reviewer's concerns. We respectfully request reassessment in light of the revised results.
We sincerely appreciate your thoughtful and constructive feedback, which has contributed significantly to strengthening our manuscript. The new theoretical guarantees, expanded experimental evidence, and improved presentation directly address the concerns raised and provide a substantially stronger foundation for our contributions.
We welcome further questions and constructive discussion throughout the rebuttal period. Thank you again for your time, expertise, and careful evaluation. We respectfully look forward to your reassessment of the revised manuscript in light of these substantive advances.
Our responses to the weaknesses and questions
We thank the reviewer for identifying two central questions: how the routing components interact and whether adaptive memory adds value beyond simpler alternatives. Our revision addresses both through a unified causal pipeline, new algorithm-specific guarantees and matched experiments.
1. Complete decision process and notation (W1).
Figure 1 now presents the complete routing pipeline in Section 3. A visible incomplete context is mapped to a frozen representation; nested ridge memories use previously revealed audits to produce horizon-specific expert scores and directional leverage.
The original MS branch combines scores using adaptive horizon weights, while CCR-Cert adds a heuristic incumbent-retaining margin gate. The new GL/Lepski branch selects a single horizon using cross-scale disagreement and directional uncertainty. The selected expert is deployed before current audit losses are revealed; subsequent feedback updates the memories and applicable weights.
Figure 1 separates online audit updates from evaluation-only quantities. Algorithm 1 specifies CCR-GL, and the revised text explicitly defines the scores, weights, losses and deployment rules. Formal GL guarantees are distinguished from heuristic CCR-Cert.
2. Stronger online model-selection alternatives (W2).
The rebuilt benchmark uses common frozen predictions, evaluation targets, audit activations and validation-only tuning.
Beyond individual imputers, Static and fixed/discounted ridge, we now compare deterministic and sliding-window Hedge, Fixed Share, restart ridge, stationary contextual selection, uniform multi-scale aggregation, intercept-only controls and Lepski.
We report absolute MAE/RMSE. GL-Tuned improves original CCR-MS micro-MAE on all four primary datasets:
Across 96 primary family-by-rate comparisons, 76 conditional pointwise 95% macro-MAE intervals favor GL-Tuned, none favor MS, and 20 are inconclusive. Switching counts decrease substantially, e.g., from 129.61 to 2.56 on Weather.
GL-Tuned improves validation-selected Static by 7.44% on ETTh1. On exploratory Electricity, GL-Tuned and MS improve even hindsight BestConst by 1.47% and 2.68%, respectively.
Static remains stronger on Weather and both primary clinical datasets, and Hedge is competitive. Fewer switches do not establish switching safety.
3. Is adaptive multi-scale memory necessary?
We designed a matched study specifically to isolate the contribution of adaptive memory from full-pipeline improvements.
GL-Tuned selects intercept-only representation in 1,330/1,440 primary cells (repeated validation choices), making representation simplification a potentially important contributor.
We therefore replay all 1,608 declared protocol cells with both base and intercept-only representations. GL, Lepski, Longest, Uniform and Fixed share ridge settings, radius multipliers and horizon libraries selected through a common validation objective. Independently tuned Fixed is an additional control.
Table 4 reports eight primary pooled macro-MAE comparisons with approximate Bonferroni adjustment:
The favorable clinical micro-MAE reductions are modest (0.330%, 0.312% and 0.038%), and Static remains stronger. GL and Lepski agree on 99.11% of actions on average per cell. Longest is competitive and outperforms GL in two favorable clinical settings in secondary unadjusted comparisons.
These matched results establish selective benefits of adaptive memory while identifying settings where simpler rules suffice. They distinguish the value of adaptation itself from improvements attributable to representation or tuning.