- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies online adaptation of a weather forecasting model over a long chronological stream. It combines updates in historically sensitive model directions with a memory of persistent forecast errors and an adaptive gate. The evaluation reports average improvements over online adaptation baselines and examines how frozen-model error and adaptation gains relate to departure from the historical climate distribution.
Critical strengths
-
The evaluation covers a long period and a broad set of outputs. The appendix includes regressions as well as favorable cases, and the reported aggregate improvements can be recovered from the tables. That is useful reporting, even though internal arithmetic consistency does not independently verify the forecasts.
-
Persistent error correction and state-dependent adaptation are a sensible combination for this application. The variable-wise analysis helps explain why one correction policy would not suit every field. The gating comparison also gives some evidence that always applying residual memory would be a poor choice for particular variables.
Critical weaknesses
-
The chronological deployment claim needs a clearer experimental contract. Appendix C does not make the initialization source sufficiently explicit: it matters whether the backbone was trained from scratch or adapted from weights with later data exposure. The multi-step forecast/update timeline is also unclear. I cannot tell from the description whether an issued forecast keeps its parameters fixed, how corrected states are fed back, or when each verification target becomes available. Hyperparameter selection and the gate's stated timescale raise related questions. I am not asserting leakage, but these details are necessary to establish the claimed historical separation.
-
The additional value of the sensitivity-based mechanism is not isolated. Gradient concentration and moving activation patterns are interesting diagnostics, but they do not show that these directions improve adaptation more than a simpler parameterization. The important frozen-backbone-plus-residual-memory control is also missing from the reported ablation. Consequently, persistent bias correction could explain a substantial part of the gain. The aggregate improvement is encouraging, but its variation across fields and time needs uncertainty estimates and simpler controls before the central mechanism claim becomes convincing.
Questions for authors
- What exact weights initialize the offline model, and what data periods were used before the stated chronological evaluation?
- During an issued multi-step forecast, are parameters fixed? Please describe how forecast feedback, delayed verification, and overlapping initialization times are handled.
- When were adaptation hyperparameters selected, and was that selection separated from the online evaluation stream?
- What gate normalization, clipping, and forgetting rules were actually used? How does the routing timescale in Appendix C relate to the cumulative update in Equation (11)?
- Are correction-only and matched simple-adapter results available from existing runs? If so, can their gains be reported with uncertainty that respects temporal dependence?
3 comments
Dear Reviewer, thank you for your time and thoughtful feedback. We would appreciate any further comments on our responses and clarifications when you have a chance. Your feedback would be very helpful to us.
Thanks to the author for the rebuttal. After reading the rebuttal carefully, I will raise the score.
Weaknesses
W1: Chronological Deployment Protocol
see Q1,Q2,Q3,Q4
W2: Isolating the Sensitivity Mechanism
We agree that gradient concentration and moving activation patterns alone do not establish the benefit of the selected directions. Beyond the six online adaptation baselines, we therefore compare against LoRA and a parameter-matched random-subspace control. At the six-hour lead time, Ours reduces average MSE by 5.58% relative to LoRA and by 2.11% relative to Random Subspace. The latter uses the same adapted layers, coefficient parameterization, and trainable parameter count, directly supporting the value of selecting forecast-sensitive directions over random directions.
W3:Frozen-backbone residual-memory control.
We compare Ours with a frozen Aurora backbone using a seven-day EWMA residual correction at the six-hour lead time. Ours reduces average MSE by 6.53% relative to the residual-memory control, indicating that residual correction alone does not account for the full improvement.
Questions
Q1: Initialization and Data Provenance
As described in Appendix C.1, we use ERA5 data from 1959–1968 for offline pretraining, followed by chronological online adaptation over 1969–2022. During the pre-training phase, we trained the model for 100 epochs to ensure convergence, and then loaded the checkpoint from the end of pre-training using online initialisation.
Q2: Multi-Step Forecasting and Update Timeline
The model parameters for the multi-step forecast/update timeline remain unchanged; in fact, the model parameters are frozen throughout the entire process.
Q3: Hyperparameter Selection Protocol
Hyperparameter search was conducted separately from the online test stream, and the selected settings were fixed before evaluation. Appendix C.3 summarizes the adopted settings. Appendix C.4 reports results across the tested subspace ranks (Table C.1), adapter learning rates (Table C.2), gate learning rates (Table C.3), and residual-memory half-lives (Table C.4).
Q4: Gate Normalization, Clipping, and Forgetting
The seven-day half-life controls the EWMA residual memory in Eq. (7). The gate follows the cumulative exponential-weighting update in Eq. (11), using normalized endpoint losses clipped to [0,1], without discounting its weights. The wording “a seven-day timescale for its running states” in Appendix C.3 is ambiguous, and we will revise it to distinguish the residual-memory timescale from the gate update.
Q5: Control Experiments and Temporal Uncertainty
Existing runs include both a frozen-backbone residual-memory control and a parameter-matched subspace adapter. We compare them with Ours on 39,440 common six-hour forecast initializations across 69 fields. We compute percentile confidence intervals using 10,000 paired circular moving-block bootstrap replicates of monthly errors, with a 12-month block length and forecast-count weighting. The intervals remain above zero for block lengths of 3, 6, and 24 months.