Discrepancy-Ranked Temporal Validation for Deep Time-Series Forecasting

Validation-set selection in time-series forecasting has largely converged on the computationally scalable chronological tail holdout used in modern deep forecasting benchmarks. This default concentrates validation on recent targets, assuming that the latest regime best represents the test period. Earlier work on time-series performance estimation used repeated holdouts across multiple test periods to reduce dependence on any one period. The modern tail holdout instead uses a single validation period, trading temporal coverage for computational efficiency. We therefore propose Discrepancy-Ranked Temporal Validation (DRTV), a validation-set construction rule for checkpoint selection under a single-training-run budget. DRTV distributes validation positions across the development pool using a deterministic discrepancy-based rank ordering while preserving a separate test set. We formalize three desirable properties of validation-set construction: determinism, nestedness, and temporal coverage. The standard tail holdout satisfies the first two, whereas DRTV satisfies all three. Across 11 datasets and four forecasting models, DRTV achieved lower MAE than the tail holdout in 32 of 44 cases and lower MSE in 33 of 44. Across all 44 cases, the median relative error reductions were 3.50% for MAE and 6.62% for MSE. The gains varied substantially across dataset types; among cases without improvement for each metric, median degradation was 0.74% for MAE and 0.37% for MSE. By including eligible tail observations in the initial training run, DRTV can avoid a separate post-selection retraining step undertaken solely to incorporate them. These results position DRTV as a practical validation strategy for efficient model selection and deployment. Code is available at https://anonymous.4open.science/r/ValSelect-A565/README.md.