- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes evolved sampling for dynamic sampling during training, which performs batch-level data selection based on predicted loss dynamics. The results show that it reduces backpropagation costs while maintaining model performance.
Critical strengths
- It proposes ES and its pruning extension, ESWP, for online data selection at the batch and set level using loss-based updates.
- It establishes a convergence bound for ES-weighted SGD with a constant learning rate.
- The experimental results span various tasks, which is good.
Critical weaknesses
- The proposed method computes sampling weights from loss predictions for selection. Similarly, many methods are based on loss distributions (e.g., Infobatch). While the authors discussed loss-based selection in Sec 2, I think the methodological novelty is limited.
- What is the difference between batch- and set-level selection? Is the batch size of batch-level selection different during training? If not, the so-called batch-level sampling is actually dynamic selection.
- The literature review is limited; more recent works should be discussed, e.g., TDDS (CVPR 24), Data Agent (ICML 26), OPUS (ICML 26), etc.
- The theoretical analysis weakens readability; more details can be moved to the appendix.
- The complexity analysis is trivial. Since the core part of the proposed method is the loss-based sampling probability, the efficiency parts can be easily derived.
- The accuracy improvement is marginal, and comparisons are relatively limited, which weakens the practical effectiveness. Based on the comments above, the current manuscript is not ready for publication. I can only give a Reject at the current stage.
Questions for authors
See weakness above.
3 comments
Response to Reviewer CNDu (Part 2)
(3) Related work and additional comparisons.
The submission includes an extended review in Appendix A and additional baseline comparisons in Appendix D.4. Nevertheless, we agree that TDDS, Data Agent, and OPUS should be discussed explicitly, and will expand the discussion accordingly.
- TDDS (CVPR 2024) is static data selection, which collects trajectories through an initial training stage, constructs a coreset, and then trains on that subset. ES is dynamic data selection, which updates sampling weights within the ongoing training run. The distinction concerns both the scoring rule and the training workflow; TDDS is not adaptive to ongoing training dynamics.
- Data Agent (ICML 2026) learns a selection policy through PPO using model features and training-derived rewards. ES uses a fixed update rule based only on losses, without training an auxiliary selection policy. Data Agent requires an additional policy network and higher compute overhead.
- OPUS (ICML 2026) selects data using optimizer-aware gradient utility relative to a proxy direction. This provides a direct comparison with ES's loss-based scoring. We have therefore included OPUS in the new GPT-2-Large experiment reported under point (6). In this experiment, OPUS improves mean downstream accuracy but takes 1.4% more execution time than Baseline, whereas ES saves 25.2%. OPUS uses gradients for selection, which requires substantially more compute.
(4) Readability of the theoretical analysis.
The theoretical analysis is important for understanding both the design of ES and its distinction from existing loss-based methods. Specifically, Theorem 3.1 motivates why loss-based weighting can accelerate optimization in the quadratic setting (corresponding to the local geometry of the optimization landscape); Proposition 3.2 explains how the proposed rule incorporates loss variations toward predicting future loss evolution and implicitly captures gradient alignment during training using only lightweight loss calculations; and Theorem 3.3 characterizes convergence of the proposed method under the stated assumptions. These results connect the simple weighting rule to its underlying mechanism and directly support the methodological contribution discussed in point (1) with theoretical guarantees.
We have already placed the detailed proofs in Appendices B.1–B.3, retaining the principal results and their interpretation in the main text so that readers can understand the method without consulting the full derivations. We therefore believe that retaining this theoretical foundation is valuable. To improve accessibility, we will sharpen the intuitive explanations and transitions between the results while keeping the technical proof details in the appendix.
(5) Purpose of the complexity analysis.
We agree that the computational accounting is straightforward and included for completeness. Section 3.5 explains the sources and limits of the savings, and its simplicity reflects the lightweight nature of our method. The contribution lies in obtaining effective sampling weights with inexpensive computations. We will connect this section more directly to the measured training-time results.
Response to Reviewer CNDu (Part 3)
(6) Practical effectiveness: reducing training time while preserving performance.
Our primary goal is to improve efficiency: reducing training time while maintaining or improving performance, rather than solely maximizing accuracy regardless of cost. The submitted results demonstrate this efficiency:
- Table 1(b): ESWP reduces ViT-Large fine-tuning time by 40.7%, while increasing ImageNet-1K accuracy from 84.4% to 85.0%.
- Table 2: ESWP with reduces MAE pre-training time by 27.0%, while maintaining 84.9% downstream accuracy.
- Table 3: ES reduces GPT-2-Small/XL pre-training time by 28.2% / 25.6%, while increasing their mean GLUE scores from 78.6 / 83.7 to 79.0 / 85.0.
For further comparison, we additionally pre-trained GPT-2-Large on OpenWebText against five data-selection baselines. We include the table and the training curve in blue in the updated PDF.
Table R1. New GPT-2-Large results: mean zero-shot accuracy across nine benchmarks and execution-time savings.
| Method | MMLU | ANLI | HellaSwag | PIQA | SIQA | W.G. | ARC-E | C.QA | WSC | Mean | Time saved |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Baseline | 29.44 | 33.29 | 48.51 | 69.21 | 41.56 | 49.80 | 49.28 | 35.54 | 39.42 | 44.01 | — |
| InfoBatch | 29.03 | 33.24 | 48.55 | 70.62 | 41.50 | 46.80 | 49.92 | 35.14 | 45.19 | 44.44 | 6.1% |
| LinUpper | 28.85 | 33.74 | 49.13 | 69.42 | 40.53 | 48.70 | 49.12 | 35.14 | 45.19 | 44.42 | 8.8% |
| UCB | 29.14 | 32.47 | 47.99 | 69.48 | 39.82 | 48.22 | 48.65 | 33.91 | 50.96 | 44.52 | 14.6% |
| IES | 29.19 | 33.12 | 49.09 | 69.59 | 41.20 | 49.09 | 49.20 | 33.66 | 48.08 | 44.69 | -22.4% |
| OPUS | 29.12 | 33.23 | 47.18 | 70.24 | 40.17 | 49.09 | 48.99 | 34.48 | 50.00 | 44.72 | -1.4% |
| ES | 28.93 | 33.91 | 47.60 | 68.88 | 41.04 | 49.64 | 48.32 | 35.79 | 50.00 | 44.90 | 25.2% |
Negative savings indicate increased execution time. ES obtains the highest mean accuracy and the largest time saving among all compared methods. Although the accuracy differences are modest, the simultaneous reduction in training time is substantial and supports its practical effectiveness.
Response to Reviewer CNDu
We thank the reviewer for recognizing the breadth of our experiments and the theoretical analysis. We address each concern below.
(1) Methodological novelty: incorporating loss dynamics into sampling weights.
Our contribution is the specific weighting rule motivated by predicted loss dynamics, developed in Section 3.3, lines 176–201. Rather than focusing solely on current training states as in prior work, it emphasizes training samples that will remain difficult in the future. Whereas InfoBatch prunes samples according to their observed loss distribution and rescales the remaining gradients, ES extrapolates current losses using their historical evolution to predict future loss dynamics, with tunable coefficients. Proposition 3.2 shows that this rule incorporates a smoothed loss-variation term and, through a first-order approximation, information about gradient alignment along the training trajectory, without explicitly computing gradients for scoring.
More precisely, in Equation (3.4):
pi(t)∝wi(t)=β1si(t−1)+(1−β1)ℓi(θ(t)),si(t)=β2si(t−1)+(1−β2)ℓi(θ(t)).
Setting β1=β2=0 recovers pure loss-based selection. We will clarify that "loss prediction" denotes this lightweight predictive surrogate.
Table 7 directly tests this distinction. With annealing enabled in all compared variants, ES achieves 58.4 versus 55.8 and 57.6 on CoLA. Tables 1(a), 1(b), 2, and 5 further show consistent gains in both accuracy and training-time savings over InfoBatch across vision and language settings:
(2) Batch-level versus set-level selection.
The reviewer is correct that ES is a form of dynamic data selection. "Batch-level" and "set-level" specify where and when selection occurs within this broader category.
In batch-level selection, ES evaluates a candidate meta-batch of size B and selects a smaller mini-batch of size b<B for backpropagation. During the selection phase, these sizes are fixed hyperparameters; the selected samples and their probabilities change dynamically. During the initial and final annealing phases, training uses the full meta-batch.
In set-level selection, a subset of the dataset is retained at the beginning of each epoch. ESWP combines this epoch-level pruning with selection within each meta-batch. Figure 1 and Algorithm 1 illustrate the distinction.
We will clarify this terminology and emphasize that the core methodological contribution is the weighting rule rather than the level at which it is applied. Applying it at both levels allows their computational benefits to be combined.