acceptodds
Under review as a conference paper at ICLR 2027

Learning from Historical Overlap Errors: Adaptive Pointwise Reweighting for Time Series Forecasting

Abstract

Long-term time series forecasting models are commonly trained with sliding windows, so nearby input windows predict many of the same future timestamps. Standard training optimizes these overlapping forecasts independently, leaving the errors of preceding forecasts unused as timestamp-specific evidence for determining the optimization priority of current predictions. We propose Historical Overlap Reliability Weighting (HORW), a model-agnostic method that compares each prediction point’s current error with a statistical reference derived from historical prediction errors of preceding overlapping windows for the same target timestamp, and uses the resulting error gap to adjust the optimization emphasis on each prediction point. Specifically, after each training epoch, HORW holds the epoch-end model parameters fixed during a gradient-free sequential pass over the training set to construct a historical error memory. For each prediction point in the next epoch, it uses the larger of the mean and median errors from preceding windows forecasting the same target timestamp as a conservative reference. When the current error exceeds this reference, HORW maps the normalized gap to a point-wise loss weight. A single signed scalar coefficient specifies the response to this gap: its sign determines whether the corresponding point is relatively emphasized or attenuated, while its magnitude controls the response strength. HORW preserves the original model architecture, applies to diverse forecasting objectives, and adds no inference-time cost. Experiments on seven datasets with three architecturally diverse forecasting models show improvements in the vast majority of evaluated settings, with additional gains when HORW is combined with auxiliary training objectives. Ablations further support the value of timestamp-aligned historical references over global historical statistics and current-error references.

Primary area:
learning on time series and dynamical systems
Keywords:
Time Series Forecasting

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_hAc4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes HORW, a model-agnostic framework that dynamically adjusts point-wise loss weights using historical prediction errors from overlapping forecasting windows. Experiments across seven datasets and three forecasting models demonstrate its effectiveness in improving forecasting accuracy, with further gains when combined with auxiliary training objectives.

Critical strengths
  1. Interesting idea
  2. HORW is a model-agnostic framework
  3. The experimental results have verified HORW's effectiveness.
  4. This paper focuses on an often-overlooked phenomenon in training: overlapping predictions of future timestamps across sliding windows.
Critical weaknesses
  1. The proposed method incurs relatively high training costs, undermining its claimed advantages of simplicity and lightweight design.
  2. The paper lacks direct comparisons with lightweight yet effective methods for optimizing the training process, such as selective learning [2], DropoutTS [3] and curriculum learning [1].
  3. The writing is poorly organized and lacks clarity, making several key aspects of the method difficult to understand intuitively. For example: 3.1. Why can the prediction errors from preceding windows serve as an indicator of the learning value of the current prediction? 3.2. Why does the method consider only preceding windows rather than all overlapping windows? 3.3. If predictions from preceding windows involve longer forecasting horizons and are inherently more difficult, what additional meaningful information do their historical prediction errors provide?
  4. The discussion of prior work precedes a sufficiently intuitive explanation of the central problem, making it difficult to appreciate the method's novelty.
  5. The introduction does not clearly establish why historical errors from preceding windows provide a reliable signal for optimizing the current prediction.
  6. The paper lacks a thorough analysis of the advantages of the proposed method over existing sample- or label-weighting and selection strategies[1,2,3].

[1]MTGNN, KDD2020

[2]Selective Learning, NeurIPS 2025

[3]DropoutTS, ICML 2026

Questions for authors

see weakness

4 comments
AuthorsOpenReview ↗

Response to Reviewer hAc4 (Part 1/2)

We thank the reviewer for the detailed and constructive feedback. Your comments have helped us clarify the intuition behind historical error references, explain our choice of preceding windows, and present the method’s contribution more clearly. They have also prompted us to examine training costs more candidly and strengthen the comparison with related training strategies. We address each point below.

Q1. The proposed method incurs relatively high training costs, undermining its claimed advantages of simplicity and lightweight design.

We have clarified the additional training costs in the Conclusion section of the main text and revised the lightweight claim to refer specifically to the unchanged backbone architecture and absence of additional inference-time computation.

Q2. The paper lacks direct comparisons with lightweight yet effective methods for optimizing the training process, such as selective learning [2], DropoutTS [3] and curriculum learning [1].

Thank you for pointing us to these studies. We have added references to curriculum learning [1], Selective Learning [2], and DropoutTS [3], together with a discussion of their relationship to HORW. For direct comparisons, we prioritized the two more recent methods, Selective Learning [2] and DropoutTS [3], using DLinear as the common backbone. All methods use the same data splits, forecasting horizons, and evaluation protocol. We tuned the comparison methods using search ranges specified in their papers or confirmed by the authors, selecting configurations based on validation performance. The results on six datasets are reported in Appendix N and reproduced in the table below.

Compared with DropoutTS, HORW achieves lower MAE on all six datasets and lower MSE on five, with a tie on ETTm1 at the reported precision. Although Selective Learning achieves lower MSE and MAE on ILI by approximately 9.7% and 8.5% relative to HORW, HORW reduces MSE and MAE on Exchange by approximately 30.0% and 12.0% relative to Selective Learning. HORW also achieves lower MSE and MAE on four of the six datasets.

Results using DLinear as the common backbone. Lower values are better; the best results at the reported precision are bolded.

DatasetHORW MSEHORW MAEDropoutTS MSEDropoutTS MAESelective Learning MSESelective Learning MAE
ETTh10.4280.4380.4420.4550.4300.444
ETTh20.4000.4220.4700.4640.4260.442
ETTm10.3570.3790.3570.3800.3590.380
ETTm20.2700.3300.2830.3450.2680.330
ILI2.1751.0412.3801.0921.9640.952
Exchange0.2500.3580.3650.4220.3570.407

Q3.1. Why can the prediction errors from preceding windows serve as an indicator of the learning value of the current prediction?

As you noted in Q3.3, preceding windows predict the same target at longer horizons, where accurate forecasting is generally more difficult. Their errors provide a target-specific reference for how well the model predicts the same timestamp and variable using earlier observations. If the current prediction has a larger error than this reference despite having access to more recent observations and a shorter forecast horizon, it underperforms relative to the performance summarized by preceding forecasts. We use this relative underperformance as a signal that the current prediction may warrant an adjustment to its training weight.

Q3.2. Why does the method consider only preceding windows rather than all overlapping windows?

Later windows often yield smaller errors because they forecast the same target at shorter lead times with more recent observations. Including them may lower RR and activate additional points that would otherwise meet the preceding-window reference. Through weight normalization, this also changes other points’ relative contributions to training, potentially affecting their optimization.

Q3.3. If predictions from preceding windows involve longer forecasting horizons and are inherently more difficult, what additional meaningful information do their historical prediction errors provide?

Your question certainly gives us much to reflect on. Indeed, predictions from preceding windows involve longer forecasting horizons and are inherently more difficult. However, these errors also reflect how well the model predicts the particular target, providing a target-specific benchmark beyond current error magnitude. When the current error exceeds this benchmark despite a shorter lead time, it indicates underperformance relative to earlier forecasts made with less recent information.

AuthorsOpenReview ↗

Response to Reviewer hAc4 (Part 2/2)

Q4. The discussion of prior work precedes a sufficiently intuitive explanation of the central problem, making it difficult to appreciate the method's novelty.

We have revised the introduction to explain the central problem intuitively before presenting HORW, and moved the discussion of prior work to the Related Work section.

Q5. The introduction does not clearly establish why historical errors from preceding windows provide a reliable signal for optimizing the current prediction.

Thank you for this helpful suggestion. We have revised the introduction to clarify why preceding-window errors provide a useful reference for the current prediction.

Q6. The paper lacks a thorough analysis of the advantages of the proposed method over existing sample- or label-weighting and selection strategies[1,2,3].

These studies are relevant to HORW’s goal of improving forecasting through training optimization, and we have added citations and discussion in the revised manuscript. The detailed experimental comparisons and analysis are provided in our response to Q2.

My concerns have been well addressed. Therefore, I decide to raise my score to 3 and recommend acceptance of this paper.

Dear Reviewer hAc4,

Thank you for taking the time to reconsider our paper and for your constructive feedback. Your questions helped us make the motivation for HORW clearer, improve the organization of the introduction, and add more direct comparisons. We appreciate your careful review and wish you all the best.

Best regards,

The Authors

Reviewer_UuYPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HORW identifies that the same target timestamp can be predicted by multiple overlapping historical windows, while conventional training objectives typically optimize the prediction errors of individual windows independently, without fully exploiting the additional information provided by these overlapping predictions. To address this issue, HORW leverages historical prediction errors from overlapping sliding windows to adaptively reweight the point-wise forecasting loss, without introducing additional trainable parameters.

Critical strengths

HORW leverages historical prediction errors from overlapping sliding windows to adaptively reweight the point-wise forecasting loss, without introducing additional trainable parameters.

Critical weaknesses

none.

Questions for authors

1, HORW aggregates prediction errors from different historical windows corresponding to the same target timestamp to construct a reference error (R). Since these predictions may be made under substantially different forecasting conditions, their errors may not be directly comparable. This discrepancy could introduce bias into the reference error and potentially lead to the accumulation of errors during training. 2, Eq. (5) constructs the reference error using the maximum of the mean and median historical errors. Is this aggregation strategy primarily based on intuition rather than being specifically designed to account for the statistical properties of prediction errors? Furthermore, have the authors considered whether this strategy could misinterpret sudden changes or anomalous events, potentially resulting in inappropriate loss reweighting?

3 comments
AuthorsOpenReview ↗

Response to Reviewer UuYP

Dear Reviewer UuYP, your comments on the comparability of historical errors and the construction of the reference error have been particularly insightful. Your feedback has helped us clarify the rationale and limitations of the aggregation strategy, including its behavior under sudden changes and anomalous events. We address each concern below and wish you all the best.

Q1. HORW aggregates prediction errors from different historical windows corresponding to the same target timestamp to construct a reference error (R). Since these predictions may be made under substantially different forecasting conditions, their errors may not be directly comparable. This discrepancy could introduce bias into the reference error and potentially lead to the accumulation of errors during training.

We agree that historical windows differ in their available observations and forecasting horizons. However, their errors are computed using the same frozen model and error metric against the same ground-truth timestamp and variable, providing a common basis for aggregation. RR summarizes performance across earlier forecast origins, reducing reliance on any single prediction without assuming identical forecasting conditions. The bounded normalized gap limits weighting adjustments for a fixed β\beta, but does not eliminate the risk of reference bias affecting subsequent updates. We will explicitly acknowledge this limitation in the revised manuscript.

Q2. Eq. (5) constructs the reference error using the maximum of the mean and median historical errors. Is this aggregation strategy primarily based on intuition rather than being specifically designed to account for the statistical properties of prediction errors? Furthermore, have the authors considered whether this strategy could misinterpret sudden changes or anomalous events, potentially resulting in inappropriate loss reweighting?

The aggregation is a heuristic design rather than an optimal estimator derived for a particular error distribution. Taking the maximum requires the current error to exceed both the historical mean and median, imposing a stricter activation criterion than either alone. Our ablation results support this choice empirically.

Thank you for highlighting this issue, which we did not explicitly account for in the current design. HORW cannot distinguish inadequate fitting from sudden changes or anomalies, and inappropriate reweighting remains possible. For example, large historical errors can raise the mean and thus RR, suppressing activation. Your comment motivates further investigation into references that better account for abrupt changes and anomalies. We have clarified this limitation in the Conclusion section of the revised manuscript.

My concerns have been addressed.

Dear Reviewer UuYP,

Thank you for taking the time to review our response. Your thoughtful comments helped us clarify the rationale and limitations of the historical reference. We are glad that your concerns have been addressed. We wish you all the best.

Best regards,

The Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer jJ34 (Part 1/2)

Dear Reviewer jJ34, thank you for your detailed and constructive assessment. Your comments have helped us strengthen the reporting of experimental variability, present the practical costs more transparently, and examine when and how HORW is effective. We appreciate your guidance in making the empirical evaluation more rigorous and clarifying the method’s scope and limitations.

Q1. The reported gains are small across much of the experimental grid, and their statistical significance is not established. The main results follow a single-seed protocol, the coefficient is tuned separately for each configuration, and the supplementary multi-seed study still transfers that coefficient while revealing instability on at least one small dataset. The headline comparison would be more convincing with paired significance tests or variance reporting in the main results.

Following your suggestion, we have moved the three-seed results (2021, 2023, and 2025) into Tables 2 and 3 in the main text, reporting mean ± sample standard deviation for MSE and MAE, respectively. All hyperparameters, including β\beta, remain fixed across seeds. In the revised manuscript, we discuss the observed variability and do not claim statistically significant or uniform improvements.

Q2. The practical cost is understated. Q2.1 The epoch-end memory construction and the additional forward passes slow training by several times on some datasets and increase peak memory substantially, while effective use in the default protocol assumes a per-configuration search over the coefficient whose preferred sign differs across backbones. Q2.2 The paper offers little guidance on when each weighting direction should be expected, and Q2.3 the efficiency-oriented variant that reduces the memory cost gives up part of the improvement on some datasets.

Q2.1 Practical Training and Hyperparameter Tuning Costs

You are right that we did not adequately account for hyperparameter tuning costs. We have now explicitly discussed this additional cost in the Conclusion section of the main text.

Q2.2 The paper offers little guidance on when each weighting direction should be expected.

We acknowledge that guidance on weighting direction remains limited, and reliable guidance could reduce the cost of applying HORW. In fact, we explored historical-error statistics, gradient alignment across training blocks, and short-horizon validation lookahead, but have not established a reliable alternative to validation-based selection across configurations. We will explicitly highlight this limitation in the conclusion and identify efficient, adaptive direction selection as future work.

Q2.3 The efficiency-oriented variant that reduces the memory cost gives up part of the improvement on some datasets.

We acknowledge this trade-off. The efficiency-oriented variant reduces memory usage at the cost of part of the improvement on some datasets. It provides an option for memory-constrained settings, rather than guaranteeing the full benefit of the default method. We will make this trade-off explicit in the revised manuscript.

Q3. •Several design choices are justified only empirically. Q3.1 Neither statistic alone is uniformly worse than the combined reference, and shuffling the timestamp correspondence hurts in some settings while helping in others, so the value of timestamp alignment is not uniform. Q3.2 The paper provides no analysis of when historical gaps are informative, for example how they relate to nonstationarity or trend dominance, and no account of the reweighting dynamics. Q3.3 The evaluation is also confined to relatively lightweight backbones and standard benchmarks, leaving the transfer to stronger recent architectures untested.

Q3.1 Neither statistic alone is uniformly worse than the combined reference, and shuffling the timestamp correspondence hurts in some settings while helping in others, so the value of timestamp alignment is not uniform.

We acknowledge that these choices are empirically supported rather than theoretically optimal. They nevertheless have clear mechanistic motivations: the combined reference requires the current error to exceed both the historical mean and median, while timestamp alignment ensures that the reference concerns the same prediction target. Although the ablated variants outperform HORW in some settings, their gains are small, whereas the proposed design performs better overall. In particular, shuffling timestamp correspondence causes substantial degradation in some settings. These results support our default design without establishing uniform superiority.

AuthorsOpenReview ↗

Response to Reviewer jJ34 (Part 2/2)

Q3.2 The paper provides no analysis of when historical gaps are informative, for example how they relate to nonstationarity or trend dominance, and no account of the reweighting dynamics.

We interpret your comment as asking under which data characteristics historical-gap-based weighting provides useful forecasting gains, and how the weights evolve during training.

To investigate the first aspect, we have added an analysis in Appendix K relating ADF statistics to HORW’s relative MSE improvements on DLinear across seven datasets, as shown below. The ADF test is commonly used to assess unit-root nonstationarity, with smaller (more negative) statistics generally indicating stronger evidence against a unit root. In our comparison, higher (less negative) ADF statistics are generally associated with larger gains: Exchange and ETTh2 improve by 32.344% and 14.615%, respectively, whereas ETTm1 and Weather improve by 0.481% and 0.715%. This pattern suggests that historical-gap-based weighting may offer greater benefits on datasets exhibiting stronger nonstationarity under this indicator, although the relationship is descriptive rather than causal.

For the second aspect, Appendix L (Figure 4) now reports the reweighting dynamics on ETTh1. From epoch 3 to epoch 7, the activation rate rises from approximately 30.6% to 39.3%, while the mean normalized weight of activated points decreases from 1.23 to 1.17. Thus, more prediction points are activated but receive milder average emphasis, illustrating how HORW’s allocation of training emphasis evolves even with a fixed β\beta.

If you intended a different aspect of the informativeness of historical gaps, we would welcome clarification and the opportunity to address it.

DatasetADFRelative MSE improvement (%)
ETTh1-5.913.665
ETTh2-4.1314.615
ETTm1-14.980.481
ETTm2-5.664.387
ILI-5.337.326
Exchange-1.9032.344
Weather-26.680.715

Q3.3 The evaluation is also confined to relatively lightweight backbones and standard benchmarks, leaving the transfer to stronger recent architectures untested.

We extended our evaluation to Crisp, a recent backbone with a more complex architecture proposed by Wang et al. at ICML 2026. We kept its backbone architecture unchanged and incorporated HORW through loss reweighting. The results are reported in Appendix M and reproduced below. HORW reduces both MSE and MAE on all six evaluated datasets, providing additional evidence of its applicability to a recent, more complex backbone.

DatasetCrisp MSECrisp MAECrisp+HORW MSECrisp+HORW MAE
ETTh10.4690.4460.4390.435
ETTh20.3760.4000.3690.395
ETTm10.3850.3860.3770.381
ETTm20.2790.3200.2780.319
ILI2.2810.9432.2480.934
Exchange0.4450.4620.4220.437

Dear Reviewer jJ34,

Thank you for your thoughtful feedback. Your comments on variability and the strength of the experimental evidence helped us improve how we present and qualify our results. We have revised the manuscript and provided detailed responses. We would be grateful if you could take another look when convenient, and we welcome any further questions or concerns.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.