acceptodds
Under review as a conference paper at ICLR 2027

Discrepancy-Ranked Temporal Validation for Deep Time-Series Forecasting

Abstract

Validation-set selection in time-series forecasting has largely converged on the computationally scalable chronological tail holdout used in modern deep forecasting benchmarks. This default concentrates validation on recent targets, assuming that the latest regime best represents the test period. Earlier work on time-series performance estimation used repeated holdouts across multiple test periods to reduce dependence on any one period. The modern tail holdout instead uses a single validation period, trading temporal coverage for computational efficiency. We therefore propose Discrepancy-Ranked Temporal Validation (DRTV), a validation-set construction rule for checkpoint selection under a single-training-run budget. DRTV distributes validation positions across the development pool using a deterministic discrepancy-based rank ordering while preserving a separate test set. We formalize three desirable properties of validation-set construction: determinism, nestedness, and temporal coverage. The standard tail holdout satisfies the first two, whereas DRTV satisfies all three. Across 11 datasets and four forecasting models, DRTV achieved lower MAE than the tail holdout in 32 of 44 cases and lower MSE in 33 of 44. Across all 44 cases, the median relative error reductions were 3.50% for MAE and 6.62% for MSE. The gains varied substantially across dataset types; among cases without improvement for each metric, median degradation was 0.74% for MAE and 0.37% for MSE. By including eligible tail observations in the initial training run, DRTV can avoid a separate post-selection retraining step undertaken solely to incorporate them. These results position DRTV as a practical validation strategy for efficient model selection and deployment. Code is available at https://anonymous.4open.science/r/ValSelect-A565/README.md.

Primary area:
learning on time series and dynamical systems
Keywords:
Time-Series Forecasting, Validation Set Construction, Low-Discrepancy Sequences

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AkvGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Discrepancy-Ranked Temporal Validation (DRTV), a deterministic validation split strategy for time-series forecasting under a single-training budget. By using a low-discrepancy sequence, DRTV constructs validation sets with determinism, nestedness, and temporal coverage while keeping the test split unchanged. Experiments on 11 datasets and 4 forecasting models show consistent improvements over chronological tail holdout, achieving median relative reductions of 3.50% in MAE and 6.62% in MSE.

Critical strengths
  1. DRTV provides a simple, deterministic, and computationally efficient validation strategy with theoretical guarantees for nestedness and temporal coverage.

  2. The method is easy to integrate into existing forecasting pipelines and allows more recent observations to remain available for training without requiring additional training runs.

Critical weaknesses
  1. The theoretical contribution is relatively limited, as nestedness follows directly from prefix selection and temporal coverage relies on established discrepancy theory. The choice of the specific low-discrepancy sequence is also insufficiently justified.

  2. Different validation strategies may retain different numbers of training windows due to overlap exclusion. Therefore, the reported improvements may partially result from differences in training data availability rather than better validation selection.

  3. The experimental analysis is incomplete. DRTV shows inconsistent improvements on PEMS datasets, and the paper lacks detailed analysis of these failures, sensitivity to different input and prediction lengths, and statistical significance tests.

Questions for authors
  1. Can the authors provide experiments that strictly control the number of retained training windows across different validation strategies to isolate the effect of validation selection?

  2. How sensitive is DRTV to the choice of low-discrepancy sequence? Have alternative sequences, such as van der Corput or Halton, been evaluated?

  3. Why does DRTV show inconsistent performance on PEMS datasets? Could distribution shifts, periodicity, or other temporal characteristics explain these results?

  4. How does DRTV compare with standard Tail Holdout when models are retrained using all available development data after checkpoint or hyperparameter selection?

Reviewer_YnAAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a method for selecting a validation set in time series forecasting by designing a deterministic procedure to select subsequences from the available sequence of observations. However, this method is merely a heuristic. The discussed motivations are not convincing, and the methodological/technical contributions are limited.

Critical strengths

Estimating the test performance of a time series forecasting model is a difficult and relevant problem.

Critical weaknesses
  • The main objective of a validation procedure is to provide an estimate of the test performance with good statistical properties (e.g., the estimator should be consistent and unbiased) given appropriate assumptions about the characteristics of the data-generating process. The paper does not even attempt to start this discussion in a meaningful way. Without any of this, the methodological contributions of the paper are very limited. With no assumptions on the process, the properties that are identified as "desirable" for data selection are quite arbitrary and do not say much about the quality of the resulting estimator. The conditions under which the method would rank models better than alternatives are not defined. The motivations for preferring the proposed heuristic over uniformly sampling blocks appear arbitrary.

  • Performance improvements are mostly in datasets such ETT that are well known for being highly non-stationary. Forecasting models on ETT, then, benefit from training on the most recent observations that are most likely included in the training set for the proposed approach. Figure 5 supports this, as the performance of the tail-hold-out method gets worse with more recent samples being excluded. In this case, the improvement cannot be attributed to how good the validation method is at model selection.

Questions for authors

See weaknesses.

Reviewer_ht12OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DRTV (Discrepancy-Ranked Temporal Validation) is a way to choose which time points to use for validation when training a deep forecasting model. Instead of taking the last block of data before the test set, it spreads validation points across the whole history. The positions come from a fixed number sequence that is known to spread points very evenly. The first few points of this sequence form a small validation set, and taking more points makes a larger set that still contains the smaller one. The model trains on all remaining windows, leaving out only those that overlap a validation window, so recent data can still be used for training. Validation loss on these scattered points then picks the best checkpoint, all in a single training run.

Critical strengths
  1. The tail holdout has been used in previous research without much thought. Showing that this choice changes results is a useful point for the field.
  2. Determinism, nestedness, and coverage are an intuitive way to compare validation rules, and the paper’s overview figure explains them well.
Critical weaknesses
  1. DRTV changes two things at once: where validation happens, and whether recent data is used for training. The paper does not separate them. The gains may come entirely from training on more recent data rather than from better checkpoint selection.
  2. A standard and obvious fix is to select with the tail holdout and then retrain on all the data. The paper says DRTV avoids this retraining step but never compares against it, so we don’t know if DRTV is actually better than the simple fix.
  3. When validation windows sit between training windows, the model is judged on filling in gaps, not forecasting an unseen future. Nearby time points are strongly related so validation loss may look better than true forecasting skill.
  4. Only checkpoint selection (when to stop training) is studied, with one input and output length. Choosing hyperparameters or comparing models is not tested. Neither are settings where the most recent period really is the best guide to the future.
Questions for authors
  1. How does DRTV compare to selecting with the tail holdout and then retraining on the full development data?
  2. Why do you think Random beats DRTV on all four PEMS datasets, and what does that say about when DRTV should be used?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.