acceptodds
Withdrawn from ICLR 2027

Learning from Complexity: Exploring Dynamic Sample Pruning of Spatio-Temporal Training

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Spatio-temporal forecasting is fundamental to intelligent systems in transportation, climate science, and urban planning. However, training deep learning models on the massive, often redundant, datasets from these domains presents a significant computational bottleneck. Existing solutions typically focus on optimizing model architectures or optimizers, while overlooking the inherent inefficiency of the training data itself. This conventional approach of iterating over the entire static dataset each epoch wastes considerable resources on easy-to-learn or repetitive samples. In this paper, we explore a novel training-efficiency techniques, namely learning from complexity with dynamic sample pruning, ST-Prune, for spatio-temporal forecasting. Through dynamic sample pruning, we aim to intelligently identify the most informative samples based on the model's real-time learning state, thereby accelerating convergence and improving training efficiency. Extensive experiments conducted on real-world spatio-temporal datasets show that ST-Prune significantly accelerates the training speed while maintaining or even improving the model performance, and it also has scalability and generality.

Primary area:
learning on time series and dynamical systems
Keywords:
Spatio-Temporal Forecasting, Data Pruning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UcoTOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ST-Prune dynamically selects spatiotemporal training samples using prediction error heterogeneity, applies stationarity-aware reweighting, and finishes with full-data training. Experiments report favorable accuracy-runtime trade-offs across multiple datasets, architectures, and model scales.

Critical strengths

The scoring rule addresses a limitation of average loss pruning: localized forecasting errors can be obscured by otherwise easy predictions.

Evaluation across datasets, backbones, optimizers, and a family of foundation models provides useful evidence of practical applicability.

Critical weaknesses

The claimed unbiasedness is unsupported: the weights in Eqs. (5)–(6) do not generally correct the stated inclusion probabilities. Appendix C.2 also disclaims the guarantee asserted in Section 4.2.

Full data annealing produces the largest ablation benefit, but equivalent treatment of the baselines is unclear. Comparisons using fixed epochs therefore leave the pruning-specific contribution and time-to-target advantage insufficiently established.

Questions for authors

Please reconcile the unbiasedness claim with Appendix C.2 and specify the actual sampling probabilities and weighted training objective.

Do random pruning and InfoBatch receive identical full data annealing, and does ST-Prune remain faster at reaching matched validation accuracy?

Reviewer_uAAeOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates dynamic sample pruning for efficient spatio-temporal forecasting and proposes ST-Prune. The framework consists of a complexity-informed scoring function that combines average prediction error with spatial and temporal error heterogeneity, a randomized pruning policy, stationarity-aware gradient rescaling, and a final-stage full-data annealing strategy. The method is evaluated on PEMS08, UrbanEV, and LargeST, with additional experiments across forecasting architectures, optimizers, prediction horizons, and the OpenCity model family. The reported results demonstrate promising training-time reductions while preserving competitive forecasting performance.

Critical strengths
  1. A practically relevant training-efficiency problem with promising empirical results. The paper addresses an important yet relatively underexplored aspect of spatio-temporal forecasting: reducing redundant training computation through dynamic data selection rather than architecture modifications. The wall-clock improvements on LargeST are encouraging. In several settings, ST-Prune achieves substantial training acceleration while maintaining forecasting accuracy close to full-data training.

  2. Broad experimental evaluation and reasonable domain-specific motivation. The proposed complexity score is motivated by the observation that global prediction errors may conceal localized spatio-temporal failures. The evaluation includes multiple pruning ratios, strong static and dynamic selection baselines, different model architectures, and component ablations. These results provide useful evidence of the practical applicability of dynamic pruning to spatio-temporal forecasting.

Critical weaknesses

The theoretical justification and precise implementation of the pruning strategy are insufficiently established:

  1. The paper claims that stationarity-aware gradient rescaling guarantees an unbiased gradient expectation. However, Equation (5) introduces a sample-dependent correction that does not generally satisfy the inverse-inclusion-probability weighting condition for unbiased estimation of the original empirical gradient.

  2. Furthermore, Equation (6) appears inconsistent with the sample-set definitions in Equation (4). The objective explicitly sums over the redundant set, although some samples in this set are supposed to be pruned.

  3. Importantly, Appendix C.2 characterizes the stationarity-aware rescaling as an empirical correction rather than a general guarantee of unbiased optimization, which contradicts the stronger claim in the main text.

  4. The randomized retention policy also requires clarification. Since all above-threshold samples are retained, the achievable retention fraction cannot be lower than the fraction of above-threshold samples. The implementation of the reported 1% and 10% settings is therefore not sufficiently explained. These concerns affect the technical soundness and reproducibility of the core method.

Limited methodological novelty and insufficient isolation of the improvements over InfoBatch:

  1. The overall framework substantially overlaps with InfoBatch (ICLR 2024), including mean-threshold sample partitioning, randomized pruning of low-score samples, gradient rescaling, and final-stage full-data training. The main methodological differences are two domain-specific heuristic modifications: error-heterogeneity-based scoring and temporal-variance-based gradient reweighting.

  2. While these modifications are potentially useful, the current experiments do not sufficiently establish their independent contributions beyond straightforward extensions of InfoBatch. In particular, the ablation results indicate that annealing, an existing component, contributes substantially to the observed performance. More targeted comparisons against InfoBatch with each proposed modification would help establish the significance of the methodological contribution.

Questions for authors

Q1. Can the authors provide a rigorous justification for the unbiased-gradient claim associated with Equation (5)? How does the proposed weighting account for the actual sample inclusion probabilities? Please reconcile the claim with Appendix C.2 and clarify the sample sets in Equation (6).

Q2. How is the target retention ratio enforced when the proportion of informative samples exceeds the desired budget? In particular, how are the 1% and 10% retention settings implemented? Could the authors report the actual per-epoch retention fractions and the sampling probabilities?

Q3. Could the authors compare InfoBatch augmented separately with the proposed spatio-temporal complexity score and stationarity-aware rescaling, under matched computational budgets? This would better isolate the independent contributions of the proposed components.

Q4. Could the authors clarify the efficiency comparisons in Table 2? The reported improvement row appears to combine prediction gains from the 10% retention setting with speedups from the 1% setting. Some relative degradation percentages in Table 1 also do not match the reported absolute values.

Reviewer_n6RJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study proposes ST-Prune, a training-efficient approach for spatio-temporal forecasting based on dynamic sample pruning. ST-Prune reduces data redundancy during training, a different approach from conventional improvement of training efficiency through model architectures or optimization algorithms. By dynamically identifying and selecting informative samples according to model's current learning state, the proposed method avoids unnecessary computation on repetitive or easy-to-learn samples. Experiments on real-world spatio-temporal datasets demonstrate that ST-Prune significantly accelerates training while maintaining or even improving predictive performance. These results support the effectiveness, efficiency, and generalizability of dynamic sample pruning for spatio-temporal forecasting.

Critical strengths
  • Clear and reasonable research motivation: This study addresses an important problem in applying conventional data pruning methods to spatio-temporal forecasting. The Averaging Masking Effect explains why using the mean prediction error can overlook samples that contain localized spatial or temporal anomalies. The predominance of stationary patterns can lead to an unbalanced training distribution after pruning. These observations provide a reasonable motivation for the proposed complexity-informed scoring and stationarity-aware rescaling.
  • Extensive experiments and drastic training time decrease: The experiments include multiple real-world datasets, pruning ratios, forecasting architectures, optimizers, prediction horizons, and foundation model scales. ST-Prune consistently demonstrates competitive predictive performance while substantially reducing training costs. For example, Table 2 reports a reduction from 13.05 to 2.92 hours on LargeST-GBA at 10% retention, with comparable MAE. The ablation and sensitivity studies further support the practical contributions of the proposed components.
Critical weaknesses
  • The claim of unbiased gradient estimation is not explained enough: The authors explain the proposed weighting scheme preserves the expected gradient of full data training. However, it is not clear from Eq.(5). In my understanding, for unbiased estimator, it is necessary to correct contribution of each sampled example according to, for example, importance-weighted estimation (Horvitz and Thompson 1952). It seems that Eq.(5) introduces an additional factor that depends on temporal variance of each variable. It is necessary to further clarify what assumptions are necessary.

  • Difference between equation and the pruning procedure: It seems that Eq.(6) and the pruning procedure in Section4.1.3 are not matched. The authors explain that the redundant set is randomly selected, but Eq.(6) includes a sum over this set without clear explanation of sampling procedure.

  • Novelty and significance of this study should be further clarified: ST-prune is designed for spatio-temporal data, but most of its formulation and procedure are similar to the one in InfoBatch (Qin et al., 2024). ST-prune and InfoBatch employ dynamically updated difficulty scores, randomly less informative samples, rescale gradients, and train with a whole set of data in the end of optimization. In addition, GDeR (Zhang et al., 2024) also considers distribution balancing and robustness in dynamic pruning in a different setting. In my opinion, it seems that main contribution of this study lies in the adaptation of dynamic pruning to spatio-temporal data, in particular, through its complexity score and stationarity-aware weighting, rather than an entirely new pruning framework. The authors have to further clarify the true contribution of ST-prune framework. For example, in ablation study, adding the proposed complexity score and stationarity-aware rescaling to InfoBatch would specify how much each operation contributes under comparable training conditions.

  • Reliability of reported accuracy improvements: In Section5.1, the authors mention that the experiments were repeated five times. However, standard deviation is not reported. Because some improvements over full-data training are quite small, reporting standard deviation or other uncertainty estimate would distinguish whether the differences are meaningful or not.


Horvitz and Thompson(1952). A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260), 663–685.

Questions for authors
  1. Could you derive the expected gradient under Eq.(5) and demonstrate its equivalence to the gradient obtained with a whole set of dataset? How is unbiasedness preserved when sample-dependent temporal variance weights are introduced?

  2. Could you check consistency between the retained sample sets in Section4.1.3 and Eq.(6), and if necessary, correct it?

  3. Could you provide a more controlled comparison that incrementally introduces the spatio-temporal complexity score and stationarity-aware rescaling into InfoBatch, while keeping the pruning policy and training schedule identical?

  4. Could you show the standard deviations in experimental results? In addition, could you show total end-to-end training time including scoring and sample-selection overhead?

  5. How does ST-Prune guarantee a specified retention ratio, in particular, at 1% and 10%, if the informative set exceeds the target percentage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.