acceptodds
Under review as a conference paper at ICLR 2027

Adapting to Rare Regimes without Forgetting: Layered Neuron Plasticity for Spatiotemporal Forecasting

Abstract

Known, observable regime shifts, including public holidays and severe weather, can cause the largest errors of an otherwise accurate spatiotemporal forecaster. Adapting a deployed model with only a few rare-regime labels creates a stability–plasticity problem: unrestricted fine-tuning can degrade routine forecasts, whereas a localized update may remain poorly coordinated with the rest of the network. We introduce Layered Neuron Plasticity (LNP), a detect-update-restore framework for this setting. LNP identifies a rare-regime-responsive parameter subspace by cross-example consensus, updates only that subspace, and restores coordination by re-optimizing its complement on the full training set while retaining the original predictor as an event-aware fallback. A local second-order analysis motivates limiting update support and magnitude. Across six dataset-regime settings with ten paired seeds, LNP reduces mean rare-window MAE in every setting while maintaining or improving mean general- and all-window performance. LNP therefore provides a reliability-oriented approach to few-label adaptation for known, observable regime shifts, where rare-event gains must not come at the expense of routine forecasts.

Primary area:
learning on time series and dynamical systems
Keywords:
spatiotemporal forecasting, rare-regime adaptation, continual learning, parameter-efficient adaptation, catastrophic forgetting, neural plasticity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rvU4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

LNP adapts forecasting models to observable rare regimes by selecting neuron rows for sparse updates, restoring complementary model parameters, and using a protected inference path to combine the rare-regime expert with the original predictor.

Critical strengths

Rare-regime adaptation is an important practical problem, and the paper explicitly evaluates both rare-regime and regular-regime performance.

Critical weaknesses
  1. The central “layered neuron plasticity” contribution is not supported by the experiments. The matched factorial study shows a relatively consistent benefit from ONR, but the layered selector does not significantly outperform uniform selection. The paper itself reports nonsignificant layered-vs-uniform differences and states that evidence for layered selection is weaker. This substantially undermines the novelty suggested by the title.
  2. The method is very close to PN-Train without a direct protocol-matched comparison to the complete PN-Train method. Cross-probe selection and sparse neuron updating are inherited from PN-Train, while the paper mainly adds depth-dependent budgets, restoration, and deployment calibration. Without a direct full PN-Train baseline, it is difficult to quantify the actual contribution.
  3. The “few-label” framing is potentially misleading. The small number refers only to the PNO update windows. ONR uses the historical training set, including additional rare windows, and inference calibration additionally uses labeled validation data. The authors explicitly acknowledge that the PNO budget is not the total labeled data available to the pipeline.
  4. Preservation of regular performance partly follows from the protected routing architecture rather than retention by the adapted expert itself. The complete system can route back to the baseline for regular regimes, so preserved aggregate performance does not demonstrate that the adapted model itself avoids forgetting.
Questions for authors

Can the authors provide a protocol-matched comparison with the complete PN-Train method? What is the total amount of labeled information used across PNO, ONR, validation-based gating, and residual calibration? Since layered selection is not consistently better than uniform selection, what empirical evidence supports presenting it as the central contribution?

Reviewer_xSkiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates few-label adaptation of deployed spatiotemporal forecasting models to rare but observable regimes, such as public holidays and severe weather, while preserving regular-regime performance. The authors propose Layered Neuron Plasticity (LNP), which combines depth-dependent neuron selection, sparse parameter updates, complementary-parameter restoration, and indicator-conditioned inference calibration. Experiments across six dataset–regime settings demonstrate improvements in rare-regime forecasting while maintaining regular performance, although the independent advantage of layered neuron selection is inconsistent, whereas network restoration shows more reliable benefits.

Critical strengths
  1. Practically important and well-motivated problem. The paper addresses the stability–plasticity dilemma in adapting deployed forecasting models to rare events with limited labeled data. This problem is relevant to applications such as holiday traffic forecasting and severe-weather demand prediction, where rare regimes may disproportionately affect forecasting reliability.

  2. Coherent framework with a meaningful restoration mechanism. LNP combines selective neuron adaptation with complementary-parameter restoration and inference-time protection. In particular, ONR provides a reasonable strategy for restoring network-wide coordination after sparse rare-regime updates while retaining the adapted parameters.

  3. Relatively rigorous and transparent empirical evaluation. The paper uses multiple paired random seeds, statistical tests, factorial experiments, and component ablations to assess the proposed framework. Importantly, the authors distinguish the consistent benefits of ONR from the less reliable improvements of layered selection and explicitly acknowledge several limitations of the experimental setting.

Critical weaknesses
  1. The independent benefit of the proposed layered neuron selection mechanism is not convincingly established. Layered Neuron Detection is presented as a central technical contribution, motivated by the hypothesis that different network depths should have different plasticity budgets. However, the factorial experiments show that layered selection does not significantly outperform uniform selection in any of the three evaluated settings when ONR is enabled. On the second backbone, uniform selection even achieves slightly better rare-regime MAE, although the difference is not statistically significant. Moreover, the observed increase in selected neurons with depth partly follows from the predefined candidate ratios and does not independently establish that deeper neurons are more task-specific or more suitable for rare-regime adaptation.

  2. The incremental contribution over PN-Train and simpler two-stage adaptation strategies remains unclear. The response-based neuron selection, cross-probe intersection, and sparse optimization mechanisms largely build upon PN-Train, yet the paper does not directly compare against the complete original PN-Train protocol. Although ONR demonstrates relatively consistent improvements, it essentially freezes the adapted parameters while further optimizing the remaining network using historical training data. Comparisons with simpler sequential fine-tuning, partial-parameter freezing, and regularization-based restoration strategies are needed to establish whether the proposed restoration mechanism provides a distinctive advantage beyond additional optimization.

  3. Preservation of regular-regime performance is not equivalent to preventing forgetting in the adapted model. The complete framework includes calendar-conditioned residual calibration, conditional expert fusion, and additional residual correction. These mechanisms can preserve regular-regime accuracy by routing predictions toward the original backbone, even if the adapted rare expert itself suffers substantial forgetting. Consequently, the reported regular-regime performance primarily validates the complete routed predictor rather than demonstrating that the internal adaptation mechanism prevents catastrophic forgetting. Separate evaluations of the adapted expert without routing or calibration, together with comparisons against simpler event-conditioned dual-expert systems, would help clarify the source of the retention benefits.

  4. The few-label adaptation setting relies on substantial additional supervision and strong observability assumptions. The reported rare-label budget primarily refers to the windows used during PNO, while the complete pipeline also accesses rare validation probes, historical labeled training data, labeled validation data, and known regime indicators. In the weather and humidity settings, future exogenous values are assumed to be available. Although these assumptions are disclosed, they substantially narrow the scope of the demonstrated adaptation capability and complicate comparisons with methods operating under stricter few-shot or unknown-shift settings.

  5. Limited evidence for cross-backbone generalization and end-to-end efficiency. The additional TimeFilter experiment provides useful evidence of portability to another architecture, but all its runs select ONR epoch zero, so it does not demonstrate the transferability of the restoration mechanism. Furthermore, ONR optimizes the complement of the sparse neuron mask, meaning that the overall adaptation procedure is not parameter-efficient throughout training. A more systematic analysis of end-to-end computational costs and gains across different backbones would strengthen the practical contribution.

Questions for authors

Q1. What is the distinctive benefit of layered selection? Since the layered selector does not significantly outperform uniform selection in the reported factorial experiments, can the authors identify settings in which the depth-dependent candidate schedule provides a consistent advantage? Can this advantage be established beyond the imposed depth-dependent selection ratios, for example through matched-mask-size or neuron-utility comparisons?

Q2. Is ONR more effective than simpler restoration strategies? Can the authors directly compare LNP against the original PN-Train protocol and simpler two-stage adaptation procedures, such as sparse rare-regime fine-tuning followed by regular-data fine-tuning with partial parameter freezing? This would clarify whether the benefits arise from the proposed restoration design rather than from additional optimization using historical data.

Q3. How much of the retention benefit comes from the adaptation mechanism versus inference-time routing? Can the authors report rare- and regular-regime errors of the adapted expert before CaRH, SCF, and CRC, and compare the complete pipeline against a simple indicator-conditioned dual-expert baseline?

Q4. What is the total adaptation cost and effective supervision budget? Can the authors report the complete number of labeled rare windows accessible across PNO, probe selection, ONR, and validation calibration, together with the total computation required by each adaptation stage?

Reviewer_HhuSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces LNP, a framework for adapting pretrained spatiotemporal forecasting models to rare regimes. The proposed method consists of three main components. First, Layered Neuron Detection identifies candidate neurons using layer-dependent activation-based selection and cross-probe intersection. Second, Pattern Neuron Optimization selectively fine-tunes the detected neurons using a small number of rare-regime training windows. Third, Overall Network Restoration freezes the adapted neurons and optimizes the remaining parameters using historical training data. The authors also introduce an inference-time protection pipeline. Experiments across various regime settings show improvements on rare-regime samples, while maintaining regular-regime forecasting performance. The paper also provides some ablation studies which suggest that ONR consistently contributes to the improvements.

Critical strengths
  • The idea of fixing the adapted neuron subset while re-optimizing the complementary parameters is intuitive.
  • The authors evaluate the proposed method across multiple datasets with various settings. The paired-seed evaluation and factorial ablations are useful for understanding the contributions of different components.
  • The paper is clearly written.
Critical weaknesses
  • The novelty is limited. The idea of selectively updating specific layers, neurons, or parameters when facing OOD data or domain-shift has already been a well-established research trajectory over years in domains of NLP and general DL such as [1] and [2]. Even compared with the most relevant work PN-train in the same domain, the technical contributions appear to be depth-dependent candidate selection, complementary-parameter restoration, and inference-time calibration, which is still somewhat incremental.
  • Also, this paper does not provide a direct protocol-matched comparison with PN-Train.
  • The proposed method identifies neurons with consistently high activations on rare inputs. However, this does not imply causal importance for rare-regime prediction. For example, a neuron may be generally active across all inputs including both rare and regular.
  • The entire pipeline appears rather complex which introduces many hyperparameters and design choices. Although providing some sensitivity analyses, this paper does not provide analysis how different number of rare probes and different choice of neuron ranking method can effect the final performance. For example, the proposed signed summation can lead to substantial cancellation between positive and negative activations, potentially overlooking neurons with strong but sign-varying responses. Besides, activation magnitude does not reflect a neuron's contribution to the forecasting loss. More fundamentally, raw activation-based rankings can be sensitive to function-preserving reparameterizations of attention projections, meaning that equivalent models may produce different neuron selections.

[1] Surgical Fine-Tuning Improves Adaptation to Distribution Shifts. Lee et al. ICLR 2023.

[2] LANDeRMT: Dectecting and Routing Language-Aware Neurons for Selectively Finetuning LLMs to Machine Translation. Zhu et al. ACL 2024.

Questions for authors

Please address the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.