- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Discrepancy-Ranked Temporal Validation (DRTV), a deterministic validation split strategy for time-series forecasting under a single-training budget. By using a low-discrepancy sequence, DRTV constructs validation sets with determinism, nestedness, and temporal coverage while keeping the test split unchanged. Experiments on 11 datasets and 4 forecasting models show consistent improvements over chronological tail holdout, achieving median relative reductions of 3.50% in MAE and 6.62% in MSE.
Critical strengths
-
DRTV provides a simple, deterministic, and computationally efficient validation strategy with theoretical guarantees for nestedness and temporal coverage.
-
The method is easy to integrate into existing forecasting pipelines and allows more recent observations to remain available for training without requiring additional training runs.
Critical weaknesses
-
The theoretical contribution is relatively limited, as nestedness follows directly from prefix selection and temporal coverage relies on established discrepancy theory. The choice of the specific low-discrepancy sequence is also insufficiently justified.
-
Different validation strategies may retain different numbers of training windows due to overlap exclusion. Therefore, the reported improvements may partially result from differences in training data availability rather than better validation selection.
-
The experimental analysis is incomplete. DRTV shows inconsistent improvements on PEMS datasets, and the paper lacks detailed analysis of these failures, sensitivity to different input and prediction lengths, and statistical significance tests.
Questions for authors
-
Can the authors provide experiments that strictly control the number of retained training windows across different validation strategies to isolate the effect of validation selection?
-
How sensitive is DRTV to the choice of low-discrepancy sequence? Have alternative sequences, such as van der Corput or Halton, been evaluated?
-
Why does DRTV show inconsistent performance on PEMS datasets? Could distribution shifts, periodicity, or other temporal characteristics explain these results?
-
How does DRTV compare with standard Tail Holdout when models are retrained using all available development data after checkpoint or hyperparameter selection?