acceptodds
Under review as a conference paper at ICLR 2027

From Windows to Regions: Regional Gradients for Controlling Forecasting Trajectories

Abstract

Time-series forecasters train on overlapping sliding windows, where localized temporal segments repeatedly contribute to parameter updates. We reveal that while recurring high-loss regions provide constructive immediate gradients, their directional alignment with the remaining data collapses over training, causing multi-step momentum accumulation to exert severe trajectory drag. Physical data withholding confirms that this phenomenon is strictly anchored to temporal location, yet physically discarding observations risks forfeiting essential structural dynamics. To resolve this dilemma, we introduce Regional First-Moment Memory Control (-short), shifting the intervention from data withholding to optimizer memory. Operating with complete data retention, -short dampens the accumulated momentum staleness of targeted temporal regions within Adam, absorbing current learning signals without inheriting outdated directional bias. Across diverse forecasters and benchmarks, -short consistently alleviates trajectory drag in expressive deep architectures while safeguarding critical physical transitions where data withholding fails, establishing source-aware memory control as a safe, non-destructive paradigm for temporal optimization.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Forecasting, Regional Memory Control, Momentum Accumulation, Trajectory Drag

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_aWuuOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that difficult temporal regions in sliding-window forecasting can provide useful current gradients while their accumulated Adam momentum harms later training. It introduces RAMP, which suppresses these regions’ gradients, and H\mathcal{H}-short, which retains every training window but replaces the selected regions’ momentum history with their current gradient contribution. The central contribution is controlling optimizer memory according to the source of the gradients.

Critical strengths
  1. The paper proposes a well-motivated, technically coherent optimizer intervention. Distinguishing a sample’s immediate usefulness from its accumulated influence is a valuable perspective. H\mathcal{H}-short implements this distinction cleanly through Adam’s additive first-moment decomposition, preserving data participation and the full-gradient second-moment computation.
  2. The experimental setup is generally broad and includes control studies.
Critical weaknesses

My main concern is about the method. For instance, the proposed memory replacement does not distinguish harmful temporal drift from useful stochastic averaging.

H−short\mathcal{H}-short replaces the selected region’s accumulated first moment with its current minibatch gradient contribution. However, the discrepancy between these quantities reflects both parameter-induced drift and minibatch sampling noise, as Appendix E.4 explicitly acknowledges. The algorithm does not distinguish these sources of variation before removing the regional history. Consequently, it can discard useful averaging even when the underlying regional gradient is stationary.

Furthermore, applying different temporal filters to the two regions can disrupt beneficial cancellation between their gradients. A stable full gradient can arise from changing regional contributions that cancel each other; selectively removing one region’s history need not preserve that stability. Keeping the full-data second moment and matching the update norm does not prevent deterioration of the update direction. The proposed intervention therefore introduces a substantive tradeoff between reduced historical lag, increased sampling noise, and altered regional cancellation. The paper does not characterize when this tradeoff is favorable. This limits the justification for describing H\mathcal{H}-short as selectively eliminating harmful historical influence while preserving useful learning signals. A stronger methodological account should identify conditions under which regional memory replacement improves the update despite these costs.

Questions for authors

See weaknesses. I'm happy to raise my score if it's resolved.

Reviewer_xe7HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the impact of recurring high-loss temporal regions on optimization in sliding-window time-series forecasting. The authors observe that these regions provide useful immediate gradients but may cause optimization trajectory drag due to accumulated historical momentum. To investigate this phenomenon, they first introduce RAMP, which identifies and masks high-loss regions during training. They then propose H-short, the main method, which selectively replaces the historical first-moment contributions of these regions in Adam with their current gradients while retaining all training samples.

Critical strengths
  1. The paper investigates the influence of overlapping windows and accumulated gradient momentum on time-series forecasting, providing a useful perspective on training dynamics.

  2. H-short offers a relatively straightforward modification to Adam, with a clear distinction between data withholding and historical momentum control.

  3. The experiments cover multiple forecasting architectures and datasets, with several diagnostic experiments and ablation studies examining regional gradient behavior.

Critical weaknesses
  1. Limited methodological novelty. The core idea of H-short, which separates gradient contributions from different sources and selectively modifies historical momentum, is closely related to existing optimization techniques. Although the region-aware design provides some distinction, the methodological novelty appears incremental.

  2. Missing important related work and competitive baselines. APTF [1] addresses the adverse effects of low-predictability samples through adaptive loss weighting while retaining their supervision, which is highly relevant to the motivation or design of H-short. The authors should discuss their differences and include comparisons to demonstrate the necessity and advantages of the proposed approach.

[1] Amortized Predictability-aware Training Framework for Time Series Forecasting and Classification.

  1. Limited and inconsistent performance improvements. As shown in Section 5.2 and Table 2, H-short underperforms different baselines in many cases, with only marginal overall improvements. These results raise concerns about the practical effectiveness, robustness, and advantages of H-short over simpler alternatives.

  2. Insufficient validation of the proposed mechanism. Section 3 attributes the performance degradation primarily to stale historical momentum. However, the staleness term in Eq. (8) also reflects mini-batch sampling variability, and the experiments do not fully isolate the effects of stale momentum from other factors such as gradient variance or changes in update directions. More controlled experiments are needed to substantiate the proposed causal explanation.

  3. Limited theoretical justification. The theoretical results in Section 4.4 mainly establish basic properties, including scale invariance, a momentum staleness bound, and a conditional local improvement guarantee. These results do not directly establish improved convergence or generalization. The authors should better connect the theoretical conditions to the observed empirical improvements or moderate the theoretical claims.

Questions for authors

Please refer to the Weaknesses.

Reviewer_CsmGOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates forecasting models trained on overlapping windows, where each time step is included in multiple training windows. In the initial epoch, the loss for each window is recorded, and windows with losses exceeding the median plus three median absolute deviations (MAD) are identified. Empirical results indicate that these high-loss windows are not randomly distributed; instead, they form extended contiguous regions that persist across epochs, with substantial agreement observed across different model architectures.

Two methods are built on this observation:

  • RAMP excludes the identified high-loss windows from training after the first epoch. RAMP-S maintains the initial set of flagged windows throughout subsequent epochs, while RAMP-D updates the set of flagged windows at each epoch.
  • H-short presented as the primary contribution in the title and abstract, retains all windows but modifies the Adam optimizer. In this approach, flagged windows contribute to parameter updates solely through their current gradient, excluding their influence via momentum.

The authors attribute the detrimental effects of these regions to the accumulation of momentum and support their approach with five theoretical propositions.

The experiments encompasses five forecasting models (MixLinear, PaiFilter, TimePro, iTransformer, PatchTST), eight standard benchmarks, and four forecast horizons, all evaluated at a lookback of 96.

Critical strengths
  • The central concept is compelling and merits further investigation. Overlapping windows reuse the same time-series segments, allowing clusters of challenging windows to influence model training. The paper shows this is a tangible effect. The identified regions remain stable across epochs, and their specific locations matter; random or shifted masks of equivalent size yield substantially less benefit.
Critical weaknesses
  • Limited novelty: The approach of excluding high-loss examples after a warm start aligns with established methods such as loss truncation (Kang and Hashimoto, 2020) and trimmed-loss training (Shen and Sanghavi, 2019). In forecasting, RobustTSF (Cheng et al., 2024) already omits windows based on anomaly scores, while Selective Learning (Fu et al., NeurIPS 2025) masks high-error time steps using similar model architectures. Notably, the experimental evaluation cites neither loss truncation nor RobustTSF as baselines.

  • Writing and presentation. The manuscript is difficult to read, introduces numerous coined terms (such as “trajectory drag”, “continuation paradox”, “withholding dilemma”, “unrenormalized”), and employs exaggerated language (including “safeguarding critical physical transitions”, “transcends”, “decisively”, “closes the theoretical loop”) to describe effects that are only a fraction of a percent.

    The terminology used throughout the manuscript is confusing: the abstract introduces “Regional First-Moment Memory Control” but refers to it as H-short, without providing an explanation for either H or “short”. For RAMP: the abbreviations S and D are never defined.

    More seriously, Figure 2 is reprinted as Figure 8 and Figure 4 as Figure 10, and the caption of Fig. 2d seem to contradict its own panel title. The pdf displays placeholders such as “Table ??” (p. 9) and “Equation (??)” (p. 32), and the indicator function in Eq. 5 does not render. The bold formatting is incorrect in Table 1 or at least confusing (please check the Traffic dataset). Table 1’s caption states three seeds, whereas the text indicates five.

References

  • Kang, D. and Hashimoto, T. B. (2020). Improved Natural Language Generation via Loss Truncation. Proceedings of the 58th Annual Meeting of the ACL, pp. 718-731.

  • Shen, Y. and Sanghavi, S. (2019). Learning with Bad Training Data via Iterative Trimmed Loss Minimization. Proceedings of the 36th ICML, PMLR 97, pp. 5739-5748.

  • Cheng, H., Wen, Q., Liu, Y. and Sun, L. (2024). RobustTSF: Towards Theory and Design of Robust Time Series Forecasting with Anomalies. ICLR 2024.

  • Fu, Y., Shao, Z., Yu, C., Li, Y., An, Z., Wang, Q., Xu, Y. and Wang, F. (2025). Selective Learning for Deep Time Series Forecasting. NeurIPS 2025.

Questions for authors
  1. Under what circumstances should H-short be preferred over withholding? Table 1 indicates that RAMP-S performs comparably to H-short, with greater gains on the ETT datasets and ECL, while H-short exhibits smaller losses on Solar and Traffic. Additionally, H-short requires computation of the two regional gradients separately at each step (Eqs. 7 to 9). In which scenarios would H-short be recommended over simply withholding the windows? What are the wall-clock time and memory overheads per epoch relative to Adam?

  2. How robust are the reported gains? All results utilize a lookback of 96, and Table 1 presents means over three seeds without reporting variability. What are the standard deviations across seeds for Table 1? Do the observed gains persist when each forecaster employs its own recommended longer lookback, or when evaluated with more advanced recent forecasters such as CycleNet or Chimera?

Reviewer_kxT7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the effect of overlapping sliding windows on gradient dynamics in time series forecasting. It identifies a potential mismatch between the immediate contribution of high loss temporal regions and their accumulated influence through Adam's first moment during prolonged training.

The authors propose Regional First Moment Memory Control (H-short), which selectively replaces the historical momentum of identified regions with their current gradient contributions while retaining full data participation. The method is evaluated across five forecasting architectures and eight benchmark datasets with four prediction horizons.

Critical strengths
  1. The paper offers an interesting perspective on learning from difficult temporal regions.

The distinction between a region's immediate contribution and its influence over subsequent updates is worth studying. H-short provides a concrete way to act on this distinction: it retains the data while modifying how gradients from selected temporal regions contribute to Adam's momentum. The use of fixed forecast origins connects the optimizer modification to the structure of sliding-window forecasting. This is a useful contribution beyond simply excluding high-loss windows from training.

  1. The experiments include several useful controls.

In addition to evaluating multiple models and datasets, the authors compare selected and random regions, shift the selected regions along the time axis, and vary which regions receive the momentum modification. The continuation experiments also control the update norm. These comparisons help assess whether the observed gains depend on the chosen regions and their gradient history, rather than just the number of masked windows or the size of the updates.

Critical weaknesses
  1. It remains unclear how broadly the proposed explanation applies.

The direct evidence that a region helps a single update but hurts a longer continuation comes from MixLinear on ETTh1 (Appendix E.1). The six-configuration momentum study also focuses on settings where RAMP-S already performs well (Appendix E.3). These results are informative, but do not establish how often the same behavior occurs across the full benchmark. In addition, a region's gradient becoming less aligned with the rest of the data does not by itself show that its historical momentum is harmful. The measured difference between historical and current gradients also includes mini-batch sampling variation. The authors should distinguish the observations supported by these experiments from the broader explanation, and either narrow the claims or provide evidence from a more representative set of configurations.

  1. The advantage over existing optimizer modifications needs clearer evidence.

H-short improves mean test MSE in 99 of 160 configurations, with an average relative reduction of 0.57%. The gains vary across models, and MixLinear shows a small average decline. These results suggest that the method can help, but leave open whether identifying temporal regions is necessary to obtain the improvement. Comparisons with a similarly tuned Adam baseline and methods such as Cautious Optimizers [R1] or Grams [R2] would help answer this question. The discussion of forecasting methods could also be more precise: Selective Learning [R3] already discusses the risk of losing rare but important events through static masking and uses dynamic masking to address it. Acknowledging this overlap would make the contribution of regional momentum control clearer.

Questions for authors
  1. Do the authors observe the single-step versus multi-step reversal beyond MixLinear/ETTh1? If broader results are available, do configurations with a larger regional momentum mismatch also benefit more from H-short? Please also clarify how mini-batch sampling variation affects this measurement.

  2. How does H-short compare with a tuned Adam baseline and existing momentum modifications such as Cautious Optimizers [R1] or Grams [R2]? A comparison on a representative subset, using comparable tuning budgets, would help establish the benefit of treating temporal regions separately.

  3. What are the measured training-time and memory costs of H-short relative to Adam? In particular, please account for the additional backward computation used to obtain the regional gradients.

References: [R1] K. Liang et al. Cautious Optimizers: Improving Training with One Line of Code, ICLR, 2026. [R2] Y. Cao et al. Grams: Gradient Descent with Adaptive Momentum Scaling, SCOPE Workshop at ICLR, 2025. [R3] Y. Fu et al. Selective Learning for Deep Time Series Forecasting, NeurIPS, 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.