acceptodds
Under review as a conference paper at ICLR 2027

Evolved Sampling: Efficient Training through Loss-Dynamics-Aware Data Selection

Abstract

Data selection aims to accelerate training while preserving model performance. To achieve this, a common approach is to identify informative data samples with significant contributions to the training. In this work, we propose **Evolved Sampling** (**ES**), a simple yet effective framework for *dynamic* sampling along the training process. This method conducts batch level data selection based on not only the dynamics of losses but also *correlations between gradients* along the training, while involving lightweight *re-weighting computations only regarding losses*, significantly reducing the back propagation time with maintained model performance. Due to its conciseness, ES is also readily extensible to incorporate set level data selection (to form ES with pruning, **ESWP**) for further accelerations. As a plug-and-play framework, ES(WP) is mathematically proved to converge, and consistently achieves lossless training accelerations across multiple pre-training and post-training tasks with various scales, saving up to 43.7% wall-clock time. Our results motivate further investigations on the data efficiency aspect of modern large-scale machine learning.

Primary area:
optimization
Keywords:
Data selection, dynamic sampling, loss dynamics, data pruning, training acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CNDuOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes evolved sampling for dynamic sampling during training, which performs batch-level data selection based on predicted loss dynamics. The results show that it reduces backpropagation costs while maintaining model performance.

Critical strengths
  1. It proposes ES and its pruning extension, ESWP, for online data selection at the batch and set level using loss-based updates.
  2. It establishes a convergence bound for ES-weighted SGD with a constant learning rate.
  3. The experimental results span various tasks, which is good.
Critical weaknesses
  1. The proposed method computes sampling weights from loss predictions for selection. Similarly, many methods are based on loss distributions (e.g., Infobatch). While the authors discussed loss-based selection in Sec 2, I think the methodological novelty is limited.
  2. What is the difference between batch- and set-level selection? Is the batch size of batch-level selection different during training? If not, the so-called batch-level sampling is actually dynamic selection.
  3. The literature review is limited; more recent works should be discussed, e.g., TDDS (CVPR 24), Data Agent (ICML 26), OPUS (ICML 26), etc.
  4. The theoretical analysis weakens readability; more details can be moved to the appendix.
  5. The complexity analysis is trivial. Since the core part of the proposed method is the loss-based sampling probability, the efficiency parts can be easily derived.
  6. The accuracy improvement is marginal, and comparisons are relatively limited, which weakens the practical effectiveness. Based on the comments above, the current manuscript is not ready for publication. I can only give a Reject at the current stage.
Questions for authors

See weakness above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer CNDu

We thank the reviewer for recognizing the breadth of our experiments and the theoretical analysis. We address each concern below.


(1) Methodological novelty: incorporating loss dynamics into sampling weights.

Our contribution is the specific weighting rule motivated by predicted loss dynamics, developed in Section 3.3, lines 176–201. Rather than focusing solely on current training states as in prior work, it emphasizes training samples that will remain difficult in the future. Whereas InfoBatch prunes samples according to their observed loss distribution and rescales the remaining gradients, ES extrapolates current losses using their historical evolution to predict future loss dynamics, with tunable coefficients. Proposition 3.2 shows that this rule incorporates a smoothed loss-variation term and, through a first-order approximation, information about gradient alignment along the training trajectory, without explicitly computing gradients for scoring.

More precisely, in Equation (3.4):

pi(t)∝wi(t)=β1si(t−1)+(1−β1)ℓi(θ(t)),si(t)=β2si(t−1)+(1−β2)ℓi(θ(t)).p_i(t) \propto w_i(t) = \beta_1 s_i(t-1) + (1-\beta_1)\ell_i(\boldsymbol{\theta}(t)), \qquad s_i(t) = \beta_2 s_i(t-1) + (1-\beta_2)\ell_i(\boldsymbol{\theta}(t)).

Setting β1=β2=0\beta_1=\beta_2=0 recovers pure loss-based selection. We will clarify that "loss prediction" denotes this lightweight predictive surrogate.

Table 7 directly tests this distinction. With annealing enabled in all compared variants, ES achieves 58.4 versus 55.8 and 57.6 on CoLA. Tables 1(a), 1(b), 2, and 5 further show consistent gains in both accuracy and training-time savings over InfoBatch across vision and language settings:

  • Table 1(a) (CIFAR-10/100, ResNet-18/50): ES achieves higher accuracy than InfoBatch in all three settings (95.4 vs. 95.3, 78.8 vs. 78.4, and 81.1 vs. 80.4) with 10%–11% time saved; with set-level pruning, ESWP surpasses InfoBatch in both accuracy (95.3 / 78.6 / 80.8 vs. 95.3 / 78.4 / 80.4) and time saved (24% / 24% / 31% vs. 21% / 24% / 28%).
  • Table 1(b) (ImageNet-1K, ViT-Large fine-tuning): ES matches InfoBatch's accuracy (84.7%) while saving more training time (26.0% vs. 23.5%), and ESWP improves both to 85.0% accuracy and 40.7% time saved.
  • Table 2 (ImageNet-1K, MAE pre-training): ESWP (r=0.3r=0.3) reaches 84.9% vs. InfoBatch's 84.6% while saving 27.0% vs. 21.8% of pre-training time (and 43.7% time saved at 84.6% with r=0.5r=0.5).
  • Table 5 (GLUE, ALBERT-Base fine-tuning): ES achieves a higher mean score than InfoBatch (83.9 vs. 83.6) with 20.2% time saved, while ESWP matches 83.6 and saves more time (33.1% vs. 28.3%).

(2) Batch-level versus set-level selection.

The reviewer is correct that ES is a form of dynamic data selection. "Batch-level" and "set-level" specify where and when selection occurs within this broader category.

In batch-level selection, ES evaluates a candidate meta-batch of size BB and selects a smaller mini-batch of size b<Bb < B for backpropagation. During the selection phase, these sizes are fixed hyperparameters; the selected samples and their probabilities change dynamically. During the initial and final annealing phases, training uses the full meta-batch.

In set-level selection, a subset of the dataset is retained at the beginning of each epoch. ESWP combines this epoch-level pruning with selection within each meta-batch. Figure 1 and Algorithm 1 illustrate the distinction.

We will clarify this terminology and emphasize that the core methodological contribution is the weighting rule rather than the level at which it is applied. Applying it at both levels allows their computational benefits to be combined.

AuthorsOpenReview ↗

Response to Reviewer CNDu (Part 2)

(3) Related work and additional comparisons.

The submission includes an extended review in Appendix A and additional baseline comparisons in Appendix D.4. Nevertheless, we agree that TDDS, Data Agent, and OPUS should be discussed explicitly, and will expand the discussion accordingly.

  • TDDS (CVPR 2024) is static data selection, which collects trajectories through an initial training stage, constructs a coreset, and then trains on that subset. ES is dynamic data selection, which updates sampling weights within the ongoing training run. The distinction concerns both the scoring rule and the training workflow; TDDS is not adaptive to ongoing training dynamics.
  • Data Agent (ICML 2026) learns a selection policy through PPO using model features and training-derived rewards. ES uses a fixed update rule based only on losses, without training an auxiliary selection policy. Data Agent requires an additional policy network and higher compute overhead.
  • OPUS (ICML 2026) selects data using optimizer-aware gradient utility relative to a proxy direction. This provides a direct comparison with ES's loss-based scoring. We have therefore included OPUS in the new GPT-2-Large experiment reported under point (6). In this experiment, OPUS improves mean downstream accuracy but takes 1.4% more execution time than Baseline, whereas ES saves 25.2%. OPUS uses gradients for selection, which requires substantially more compute.

(4) Readability of the theoretical analysis.

The theoretical analysis is important for understanding both the design of ES and its distinction from existing loss-based methods. Specifically, Theorem 3.1 motivates why loss-based weighting can accelerate optimization in the quadratic setting (corresponding to the local geometry of the optimization landscape); Proposition 3.2 explains how the proposed rule incorporates loss variations toward predicting future loss evolution and implicitly captures gradient alignment during training using only lightweight loss calculations; and Theorem 3.3 characterizes convergence of the proposed method under the stated assumptions. These results connect the simple weighting rule to its underlying mechanism and directly support the methodological contribution discussed in point (1) with theoretical guarantees.

We have already placed the detailed proofs in Appendices B.1–B.3, retaining the principal results and their interpretation in the main text so that readers can understand the method without consulting the full derivations. We therefore believe that retaining this theoretical foundation is valuable. To improve accessibility, we will sharpen the intuitive explanations and transitions between the results while keeping the technical proof details in the appendix.


(5) Purpose of the complexity analysis.

We agree that the computational accounting is straightforward and included for completeness. Section 3.5 explains the sources and limits of the savings, and its simplicity reflects the lightweight nature of our method. The contribution lies in obtaining effective sampling weights with inexpensive computations. We will connect this section more directly to the measured training-time results.

AuthorsOpenReview ↗

Response to Reviewer CNDu (Part 3)

(6) Practical effectiveness: reducing training time while preserving performance.

Our primary goal is to improve efficiency: reducing training time while maintaining or improving performance, rather than solely maximizing accuracy regardless of cost. The submitted results demonstrate this efficiency:

  • Table 1(b): ESWP reduces ViT-Large fine-tuning time by 40.7%, while increasing ImageNet-1K accuracy from 84.4% to 85.0%.
  • Table 2: ESWP with r=0.3r=0.3 reduces MAE pre-training time by 27.0%, while maintaining 84.9% downstream accuracy.
  • Table 3: ES reduces GPT-2-Small/XL pre-training time by 28.2% / 25.6%, while increasing their mean GLUE scores from 78.6 / 83.7 to 79.0 / 85.0.

For further comparison, we additionally pre-trained GPT-2-Large on OpenWebText against five data-selection baselines. We include the table and the training curve in blue in the updated PDF.

Table R1. New GPT-2-Large results: mean zero-shot accuracy across nine benchmarks and execution-time savings.

MethodMMLUANLIHellaSwagPIQASIQAW.G.ARC-EC.QAWSCMeanTime saved
Baseline29.4433.2948.5169.2141.5649.8049.2835.5439.4244.01—
InfoBatch29.0333.2448.5570.6241.5046.8049.9235.1445.1944.446.1%
LinUpper28.8533.7449.1369.4240.5348.7049.1235.1445.1944.428.8%
UCB29.1432.4747.9969.4839.8248.2248.6533.9150.9644.5214.6%
IES29.1933.1249.0969.5941.2049.0949.2033.6648.0844.69-22.4%
OPUS29.1233.2347.1870.2440.1749.0948.9934.4850.0044.72-1.4%
ES28.9333.9147.6068.8841.0449.6448.3235.7950.0044.9025.2%

Negative savings indicate increased execution time. ES obtains the highest mean accuracy and the largest time saving among all compared methods. Although the accuracy differences are modest, the simultaneous reduction in training time is substantial and supports its practical effectiveness.

Reviewer_ZFDVOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Evolved Sampling (ES), a lightweight loss-based data-selection method that weights samples by combining current loss with an EMA of historical losses, and samples a smaller mini-batch from each meta-batch for back-propagation. It further introduces ESWP, which applies the same weights to prune a subset of the dataset at each epoch. The paper provides theoretical analyses on loss-weighted gradient flow and ES-weighted SGD, and evaluates ES(WP) on vision and language tasks, including CIFAR, ImageNet, GPT-2, Qwen, and ALBERT, reporting comparable or improved accuracy with reduced training time.

Critical strengths

The idea of combining historical loss variation with the current loss for sample weighting is intuitive and well motivated.

The LLM experiments are practically relevant and substantially broaden the empirical scope of the paper.

The method is simple, lightweight, and easy to integrate into existing training pipelines.

Critical weaknesses

Implicit dependence on multiple epochs. The method, especially ESWP, relies heavily on epoch structure: set-level pruning is performed at epoch boundaries, annealing is defined in epochs, and pruned samples can only be re-evaluated in later epochs. This is particularly problematic for LLM training, where the number of effective epochs is often small. The paper does not adequately discuss or analyze this limitation, nor does it provide ablations under single-pass or few-epoch settings.

Insufficient discussion and comparison with closely related work. EMA-based historical loss weighting has been studied before, e.g., Batch Loss Score for Dynamic Data Pruning. The paper should discuss and compare with this line of work. In addition, recent dynamic data pruning studies such as Inconsistency Biases in Dynamic Data Pruning and Batch Pruning by Activation Stability are not sufficiently discussed or empirically compared.

Unfair or unclear comparison in Table 1. The comparisons should be made under the same wall-clock budget, the same BP budget, or the same effective sample ratio, or alternatively presented as accuracy–time Pareto frontiers. As currently reported, different pruning ratios and annealing schedules make it difficult to attribute the gains to the proposed method itself.

Incomplete accounting of overhead. It is unclear whether the extra forward pass in ES is fully included in the reported wall-clock time. On GPT-2 XL, the theoretical BP saving is about 45%, while the reported wall-clock saving is only 25.6%. This suggests that additional FP, communication, data-loading, and synchronization overheads are non-negligible. The paper should provide a breakdown of these overheads to make the efficiency analysis more transparent.

Large gap between theory and method, with overly strong theoretical claims. The theory does not cover several important components of the actual algorithm: mini-batch sampling without replacement; weights being updated only for samples in the meta-batch, which can leave many wiw_i stale for long periods; and the pruning feedback loop in ESWP, where pruned low-loss samples are no longer updated and may never be re-evaluated. Consequently, the theoretical claims are stronger than what the analysis actually supports.

Insufficient discussion of bias, robustness, and long-tail behavior. Loss-based weighting tends to up-weight high-loss samples, which may amplify label noise, outliers, and hard long-tail samples. Conversely, pruning low-loss samples may mistakenly remove representative samples that have already been learned. The paper would benefit from a more careful discussion and empirical analysis of these risks.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZFDV

We thank the reviewer for recognizing the motivation, practical relevance, and simplicity of ES. We address each concern below.


(1) Applicability to multi-epoch and limited-pass LLM training.

Epoch-level pruning is optional: ES performs selection within each candidate meta-batch; ESWP additionally prunes at epoch boundaries. Our GPT-2-XL experiment uses ES without dataset-level pruning, and specifies its initial ordinary-training phase in optimizer steps (Appendix D.9). Thus, batch-level selection does not require an epoch boundary. Historical information does, however, require repeated observations of a sample.

Repeated data remains relevant to LLM training. Muennighoff et al., Scaling Data-Constrained Language Models (NeurIPS 2023) [1], find that training with up to four epochs of repeated data produces negligible changes relative to fresh data at matched compute in their experiments. More recently, Microsoft's Frontier Pre-training at Microsoft AI [2] introduced MAI-Base-2, with 2.8 trillion total parameters and 80 billion active parameters, trained on 50T tokens. Microsoft's MAI-Base-2 report explicitly accounts for multi-epoch training through "source scaling," which preserves each dataset's repetition count across training scales. These references establish the practical relevance of repeated-data settings, including frontier pre-training.


(2) Relationship to BLS, RePB, and B-PAS.

These are relevant comparisons, and we will expand Appendix A to distinguish their mechanisms. Our novelty claim concerns the specific weighting rule and its interpretation through loss dynamics, rather than the use of historical information or EMA itself.

  • Batch Loss Score (BLS) [3] updates each participating sample's score using an EMA of the batch-mean loss, providing a proxy when individual losses are unavailable or inconvenient to extract. ES uses individual losses and separately controls history accumulation and current-loss weighting. Therefore, we expect ES to more accurately capture individual loss information compared to BLS.
  • Inconsistency Biases in Dynamic Data Pruning (RePB) [4] addresses score-context drift through local comparison windows and temporal gradient bias through rescaling based on historical selection frequency. ES evaluates current candidate losses under the same model state, while deliberately combining them with historical losses. With an EMA of past losses, ES places controlled weight on current loss vs. loss evolution.
  • Batch Pruning by Activation Stability (B-PAS) [5] tracks changes in activation statistics across epochs and removes entire batches with stable representations. ES instead uses scalar loss histories to select individual examples or sequences within a candidate meta-batch. ES therefore can more accurately capture sample-level information.

These distinctions clarify our contribution.


(3) Fairness of the accuracy–time comparisons.

Our objective is to reduce training time while maintaining or improving performance. Table 1 therefore reports both accuracy and measured time savings. As documented in Appendix D.11, we follow the baseline defaults in the original references: for example, InfoBatch uses r=0.5r=0.5 and annealing ratio 0.1250.125, while UCB and KAKURENBO use r=0.3r=0.3. This provides reproducible comparisons at each method's recommended settings.

The submission provides complementary controls:

  • Loss and Order use the same batch sizes as ES (Appendix D.11). Table 7 compares current-loss, moving-average, and ES weighting with annealing enabled throughout, directly testing the weighting rule.
  • Table 6 fixes training time at 250 minutes. ES improves Qwen3.5-9B-Base accuracy from 91.05 to 91.58 on GSM8K, 68.00 to 69.40 on MATH500, and 28.04 to 30.56 on OlympiadBench.
  • Figure 10 plots accuracy against samples used for BP for Baseline, ES, and ESWP. Figures 15–16 examine the performance–cost trade-off as selection and pruning ratios vary.

Together, these results support both the weighting mechanism and the practical accuracy–time objective.

AuthorsOpenReview ↗

Response to Reviewer ZFDV (Part 2)

(4) Accounting for the additional forward pass and other overhead.

The reported training-time measurements include the scoring forward pass. In the GPT-2-XL implementation, the cumulative training timer surrounds data loading and candidate scoring, selection, selected-sample FP/BP, optimizer updates, and training communication/synchronization.

The 45% BP saving refers to update-token count: Appendix D.9 specifies approximately 30+0.5×270=165B30 + 0.5 \times 270 = 165\text{B} update tokens for ES versus 300B300\text{B} for Baseline. Scored but unselected tokens still incur a forward pass, and selected tokens require a training forward/backward pass. Consequently, 45% fewer BP tokens does not imply 45% less total time; the reported 25.6% training-time saving already reflects these costs.

To inspect the components, we additionally profiled one complete 64-candidate / 32-selected cycle of our GPT-2-XL implementation, after warmup, with CUDA synchronization at phase boundaries.

ComponentTime per optimizer update (ms)
Cached host read and transfer to GPU0.175
Candidate-scoring forward passes456.99
Weight update and sampling0.022
Selected-sample forward pass230.19
Backward pass467.63
Optimizer updates and gradient clipping510.77
Total1665.78

(5) Connection between the theoretical analysis and the algorithm.

We clarify that the theory captures the algorithm's central mechanism at the level of expected gradient contributions. In expectation, sample selection produces a weighted gradient, which has the same mathematical structure as the dynamics studied in our analysis. Specifically, conditional on the training history H\mathcal{H} and candidate meta-batch B\mathcal{B}, a selected subset SS of size bb satisfies

E![1b∑i∈Sgi,|,H,B]=∑i∈Bπibgi,\mathbb{E}\\!\left[\frac{1}{b}\sum_{i\in S}g_i \\,\middle|\\, \mathcal{H},\mathcal{B}\right] = \sum_{i\in\mathcal{B}}\frac{\pi_i}{b}g_i,

where πi\pi_i is the inclusion probability of sample ii. This connects stochastic selection to weighted-gradient dynamics.

Moreover, ES refreshes scores for all candidates, including those excluded from BP, and ESWP resamples from the full dataset at each epoch, allowing previously excluded samples to return. These properties distinguish temporary exclusion from permanent removal.


(6) Bias, noisy data, and long-tail coverage.

In this work, we evaluate the proposed method on standard benchmark datasets under their usual training protocols, without deliberately introduced label noise. Our current scope is reducing training time while maintaining or improving predictive performance. Distinguishing valid hard examples, including rare or out-of-distribution samples, from noisy or mislabeled data is a separate, non-trivial problem beyond the current scope. Our results establish effectiveness in the evaluated clean settings, rather than robustness to arbitrary data corruption.

A possible extension is to use the loss history already maintained by ES to identify persistently high-loss samples with little learning progress. Such samples could be flagged as potentially noisy and down-weighted or removed.

Regarding bias and representative low-loss examples, Appendix C already discusses the intentional departure from uniform sampling. Selection is stochastic, and ESWP resamples from the full dataset rather than permanently deleting excluded examples. These design choices allow examples to be revisited.

References

I thank the authors for their response, which addresses most of my concerns. For the closely related work, however, I would like the authors to provide not only discussion but also empirical comparisons. The methods currently compared are somewhat dated. Overall, I maintain my original score of weak acceptance.

Reviewer_iQazOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Evolved Sampling (ES), a loss-only dynamic data-selection rule. Each sample keeps an exponential moving average of its loss, and its sampling weight with linear combination.At the batch level, ES samples a mini-batch b (b/B = 25%) from a meta-batch B according to w and back-propagates only through b; at the set level, ESWP also prunes the epoch's dataset with the same weights. The paper provides an acceleration theorem for loss-proportional gradient flow in a quadratic shared-minimizer model, a convergence/generalization bound for the weighted SGD, and experiments on CIFAR, ViT-Large MAE pre-training and ImageNet fine-tuning, GPT-2-Small/XL pre-training (300B tokens), Qwen2.5-Math-1.5B, Qwen3.5-9B LoRA and ALBERT, reporting up to 43.7% wall-clock savings with preserved accuracy.

Critical strengths

(1) The method is extremely simple, adds negligible memory/compute, and unifies set-level and batch-level selection under one weight, which is practically attractive. (2) Theorem 3.1 is a genuine (if idealized) result: for loss-proportional weighting in the shared-minimizer quadratic model, the instantaneous decay rate along any direction is Σa_i²/Σa_i ≥ μ, with strict improvement when directional curvatures are heterogeneous. (3) The empirical scale is unusually serious for this literature — GPT-2-XL at 300B tokens, 9B-parameter LoRA, ViT-L MAE pre-training — and the efficiency analysis (BP savings, memory, low-resource gradient accumulation) is well done. (4) Implementation details and hyperparameters of baselines are documented in the appendix.

Critical weaknesses

(1) Theorem 3.3's "sharper rate" is not established: R₁ and R₂ are only stated to be non-negative functions with no expression or scaling, the leading O(1/T) term is unchanged, and Eq. (3.8) compares the population risk to a weighted empirical risk rather than the unweighted one, so "reduced generalization gap" does not follow. (2) The analysis assumes the update θ ← θ − ηΣ_{j∈B}p_j∇l_j with normalized weights over B, while Algorithm 1 draws b samples from B and takes an unweighted optimizer step on b; the theory does not cover the algorithm. (3) The "loss prediction" motivation collapses to w = 0.2·EMA + 0.8·current loss, and the ablation shows the differential term gives no gain in one setting (CoLA: 57.5 vs 57.6) and only 0.7–0.8 points in the others, without seeds. (4) All pre-training/LLM results are single-seed (explicitly stated in D.9) yet the abstract claims "consistently achieves lossless accelerations"; several reported improvements (zero-shot +0.17, MRPC +6.8) are within plausible run variance. (5) No limitations section. (6) Minor algebra error in Eq. (B.49).

Questions for authors

he method is simple and the large-scale runs are valuable, but neither headline claim holds as stated: Theorem 3.3's "sharper rate" rests on two undefined non-negative functions, its generalization bound is taken against a weighted rather than the unweighted empirical risk, and the analyzed update (normalized weights over the meta-batch) is not the one implemented in Algorithm 1 (unweighted mean over a sampled mini-batch). The "loss prediction" motivation collapses to 0.2·EMA + 0.8·current loss, and the ablation shows the differential term gives no gain on CoLA without annealing. All pre-training and LLM results are single-seed, and at matched wall-clock the advantage over InfoBatch is 0.0/+0.2/+0.4 points. Explicit R1/R2 (or removal of the claim), a theory for the implemented estimator along OrderDP's surrogate framework (lack refer or compare with the recent dynamic data selection by Jin et al.,ICLR 2026), and ≥3 seeds are required. Raising may be considered if concerns can be fixed.

4 comments
AuthorsOpenReview ↗

Response to Reviewer iQaz

We thank the reviewer for the thoughtful, constructive assessment and for recognizing the simplicity and negligible overhead of ES(WP), the strict acceleration mechanism in Theorem 3.1, and the scale of our empirical evaluation across vision and language pre-training/fine-tuning. Below, we address each concern point by point and detail the corresponding revisions to our manuscript.


1. Theorem 3.3: Explicit Expressions for R1,R2\mathcal{R}_1, \mathcal{R}_2 and Precise Claim Wording

Reframing "sharper convergence rate" to "GD/SGD-like O(1/T)\mathcal{O}(1/T) convergence rate with a sharper upper bound." We fully agree that the asymptotic O(1/T)\mathcal{O}(1/T) order is preserved in the general smooth convex setting (whereas strict exponential rate improvement μw>μ\mu_w > \mu holds in the structured quadratic setting of Theorem 3.1). We have revised the wording in the updated PDF in blue to state accurately that ES enjoys a standard O(1/T)\mathcal{O}(1/T) convergence rate with a tighter optimization upper bound (reduced by −R1(T)≤0-\mathcal{R}_1(T) \le 0).


2. Bridging the Weighted Update and Algorithm 1's Sampled Mini-Batch Estimator & Comparison with OrderDP

2.1 Design rationale. Conceptually, computing informative loss-dynamics weights wi(t)w_i(t) is the core mechanism of ES; sampling a mini-batch bt\mathfrak{b}_t of size b<Bb < B proportional to wi(t)w_i(t) without back-propagating the remaining B−bB-b samples is the practical realization that converts sample weighting into real wall-clock and memory savings.

  • Case A (Multinomial sampling im∼p(t)i_m \sim p(t) i.i.d. on Bt\mathcal{B}_t, m=1,…,bm=1,\dots,b). Conditioned on (Bt,θ(t))(\mathcal{B}_t, \theta(t)), the unweighted mini-batch average is unbiased for the weighted meta-batch gradient:

E[g^t∣Bt,θ(t)]=∑j∈Btpj(t)∇θℓj(θ(t)),\mathbb{E}[\hat{g}_t \mid \mathcal{B}_t, \theta(t)] = \sum_{j \in \mathcal{B}_t} p_j(t)\nabla_\theta \ell_j(\theta(t)),

so the inner-product term I2I_2 in Eq. (B.26) is identical in conditional expectation. For the squared-norm term I1=ηt2∣g^t∣22I_1 = \eta_t^2 \\|\hat{g}_t\\|_2^2 in Eq. (B.23), convexity of ∣⋅∣22\\|\cdot\\|_2^2 (Jensen's inequality) gives:

E[∣g^t∣22∣Bt,θ(t)]≤1b∑m=1bE[∣∇θℓim(θ(t))∣22∣Bt,θ(t)]=∑j∈Btpj(t)∣∇θℓj(θ(t))∣22.\mathbb{E}\big[\\|\hat{g}_t\\|_2^2 \mid \mathcal{B}_t, \theta(t)\big] \le \frac{1}{b}\sum_{m=1}^b \mathbb{E}\big[\\|\nabla_\theta \ell_{i_m}(\theta(t))\\|_2^2 \mid \mathcal{B}_t, \theta(t)\big] = \sum_{j \in \mathcal{B}_t} p_j(t)\\|\nabla_\theta \ell_j(\theta(t))\\|_2^2.

Notice that the right-hand side is the exact same upper bound used in Eq. (B.23). Consequently, Eqs. (B.24)–(B.35) go through verbatim with the same step-size condition ηt≤1/(2L)\eta_t \le 1/(2L) and the exact same remainder R1(T)\mathcal{R}_1(T).

  • Case B (Weighted sampling without replacement, as in Algorithm 1). Let πj(t):=Pr⁡(j∈bt∣Bt,θ(t))\pi_j(t) := \Pr(j \in \mathfrak{b}_t \mid \mathcal{B}_t, \theta(t)) denote the marginal inclusion probability of sample j∈Btj \in \mathcal{B}_t under weighted sampling without replacement with weights wj(t)j∈Bt\\{w_j(t)\\}_{j\in\mathcal{B}_t}. Since ∣bt∣=b|\mathfrak{b}_t| = b, we have ∑j∈Btπj(t)=b\sum_{j \in \mathcal{B}_t} \pi_j(t) = b. Defining normalized effective weights p~j(t):=πj(t)/b∈[0,1/b]\tilde{p}_j(t) := \pi_j(t)/b \in [0, 1/b] (so ∑j∈Btp~j(t)=1\sum_{j \in \mathcal{B}_t} \tilde{p}_j(t) = 1), linearity of expectation and Jensen's inequality yield:

E[g^t∣Bt,θ(t)]=∑j∈Btp~j(t)∇θℓj(θ(t)),\mathbb{E}[\hat{g}_t \mid \mathcal{B}_t, \theta(t)] = \sum_{j \in \mathcal{B}_t} \tilde{p}_j(t)\nabla_\theta \ell_j(\theta(t)),

E[∣g^t∣22∣Bt,θ(t)]≤1bE[∑j∈bt∣∇θℓj(θ(t))∣22,∣,Bt,θ(t)]=∑j∈Btp~j(t)∣∇θℓj(θ(t))∣22.\mathbb{E}\big[\\|\hat{g}_t\\|_2^2 \mid \mathcal{B}_t, \theta(t)\big] \le \frac{1}{b}\mathbb{E}\left[\sum_{j \in \mathfrak{b}_t} \\|\nabla_\theta \ell_j(\theta(t))\\|_2^2 \\,\Big|\\, \mathcal{B}_t, \theta(t)\right] = \sum_{j \in \mathcal{B}_t} \tilde{p}_j(t)\\|\nabla_\theta \ell_j(\theta(t))\\|_2^2.

Because weighted sampling without replacement is strictly monotone in wj(t)w_j(t) (wi(t)≥wj(t)  ⟹  p~i(t)≥p~j(t)w_i(t) \ge w_j(t) \implies \tilde{p}_i(t) \ge \tilde{p}_j(t)), order consistency (p~i(t)−p~j(t))(ℓi(θ(t))−ℓj(θ(t)))≥0(\tilde{p}_i(t)-\tilde{p}_j(t))(\ell_i(\theta(t))-\ell_j(\theta(t))) \ge 0 is preserved whenever Eq. (B.36) holds. Thus, Theorem 3.3 holds directly for Algorithm 1's exact unweighted mini-batch step simply by replacing pj(t)p_j(t) with the induced inclusion distribution p~j(t)=πj(t)/b\tilde{p}_j(t) = \pi_j(t)/b.

2.2 Difference with OrderDP (Jin et al., ICLR 2026). This analysis connects naturally to the surrogate-risk framework of OrderDP:

(i) Score dynamics: OrderDP ranks candidates solely by current/cached instantaneous loss values, whereas ES incorporates predictive future loss dynamics approximated by the combination of current losses and loss evolution. (ii) Soft sampling & unified two-level acceleration: Instead of hard top-qq truncation, ES uses soft probability-proportional sampling and unifies epoch-level pruning with batch-level selection (ESWP).

AuthorsOpenReview ↗

Response to Reviewer iQaz (Part 2)

3. Motivation of Two-State EMA (Dif vs. NonDif) and Role of Annealing in Table 7

3.1 Why wi(t)=0.2,si(t−1)+0.8,ℓi(θ(t))w_i(t) = 0.2\\,s_i(t-1) + 0.8\\,\ell_i(\theta(t)) is NOT a single EMA. Although the update rule is compact and lightweight, si(t)s_i(t) evolves with its own slower decay rate β2=0.9≠β1=0.2\beta_2 = 0.9 \neq \beta_1 = 0.2: si(t)=0.9,si(t−1)+0.1,ℓi(θ(t))s_i(t) = 0.9\\,s_i(t-1) + 0.1\\,\ell_i(\theta(t)). As proved in Proposition 3.2, whenever β1≠β2\beta_1 \neq \beta_2, wi(t)w_i(t) is algebraically equivalent to:

wi(t)=si(t)+β2−β11−β2(si(t)−si(t−1))≈(1−β2)∑k=1tβ2t−kℓi(θ(k))⏟historical loss trend (weight 0.1)+(β2−β1)∑k=1t−1β2t−1−k(ℓi(θ(k+1))−ℓi(θ(k)))⏟EMA of dynamical loss variations (weight 0.7).w_i(t) = s_i(t) + \frac{\beta_2 - \beta_1}{1 - \beta_2}\big(s_i(t) - s_i(t-1)\big) \approx \underbrace{(1-\beta_2)\sum_{k=1}^t \beta_2^{t-k}\ell_i(\theta(k))}_{\text{historical loss trend (weight } 0.1\text{)}} + \underbrace{(\beta_2 - \beta_1)\sum_{k=1}^{t-1} \beta_2^{t-1-k}\big(\ell_i(\theta(k+1)) - \ell_i(\theta(k))\big)}_{\text{EMA of dynamical loss variations (weight } 0.7\text{)}}.

If one removes the differential/loss-prediction component by setting β1=β2\beta_1 = \beta_2 (NonDif), the second term vanishes identically (β2−β1=0\beta_2 - \beta_1 = 0). Figure 6 further confirms across CIFAR-10, CIFAR-100, and CoLA that (β1,β2)=(0.2,0.9)(\beta_1, \beta_2) = (0.2, 0.9) consistently outperforms near-diagonal (NonDif-like) choices such as (0.8,0.9)(0.8, 0.9) (95.37 vs. 95.07 on CIFAR-10; 78.79 vs. 77.88 on CIFAR-100; 58.40 vs. 57.66 on CoLA).

3.2 Why annealing (A) is necessary for Dif on CoLA, and why annealing alone does not explain the gain. At the very beginning of training (especially with a randomly initialized classification head on CoLA), losses and gradients fluctuate abruptly and si(0)=1/ns_i(0)=1/n has not yet accumulated meaningful per-sample history. Consequently, early finite-difference variations ℓi(θ(k+1))−ℓi(θ(k))\ell_i(\theta(k+1))-\ell_i(\theta(k)) are dominated by initialization noise, explaining why Loss + Dif without annealing (57.5) shows no gain over Loss + NonDif (57.6) on CoLA. A brief 5% annealing warmup stabilizes the model and initializes sis_i before selection begins. Crucially:

  • Once early instability is removed by annealing, adding the differential term (ES = Loss + A + Dif) consistently outperforms Loss + A + NonDif on both benchmarks (81.1 vs. 80.4, +0.7 on CIFAR-100; 58.4 vs. 57.6, +0.8 on CoLA), achieving the best performance among all six rows in Table 7.
  • Annealing completely (100% annealing) degenerates to Baseline, which scores only 56.7 on CoLA (Table 5) and 81.1 on CIFAR-100 with 0% time saved—whereas ES reaches 58.4 (+1.7 over Baseline) while saving 20.2% training time (and Table 11 shows that over-annealing from 5% to 10% on CIFAR-100 actually lowers accuracy from 78.79% to 78.20%). Thus, annealing alone cannot produce the improvement; it simply provides clean initial loss states for Dif to work effectively.

3.3 New five-seed ablation on GPT-2-Small. To further evaluate the contribution of the differential term, we conducted an additional nanoGPT / GPT-2-Small experiment with five random seeds, using the same GLUE evaluation protocol as in the paper. The average GLUE scores (mean ±\pm standard deviation across five seeds) are:

MethodAverage GLUE score
Baseline79.14±0.4679.14 \pm 0.46
ES without the differential term (NonDif)79.60±0.4379.60 \pm 0.43
ES79.88±0.6879.88 \pm 0.68

ES achieves the highest mean score, improving over Baseline by 0.74 points and over NonDif by 0.28 points. The NonDif variant also improves over Baseline by 0.46 points. These results support the benefit of loss-history sampling and suggest an additional benefit from the differential term in this setting.


4. Multi-Seed Pre-Training Results, Meaning of "Lossless Acceleration," and Full InfoBatch Comparison

4.1 Clarifying "lossless acceleration." We agree that our phrasing should be precise. Throughout our work, our objective is to substantially reduce wall-clock training time while maintaining or improving task performance relative to full-data training (Baseline). Small positive differences such as +0.17% average zero-shot accuracy on GPT-2-XL (Table 4) are intended as evidence of no accuracy degradation while saving 25.6% of pre-training wall-clock time, rather than as standalone claims of large accuracy gains.

4.2 New five-seed GPT-2-Small evaluation. In addition to the five-seed CIFAR-100 / ResNet-50 comparison in Figure 14 (Appendix D.5), we now report five-seed language-model results using the paper's GLUE evaluation protocol. As shown in Section 3.3 above, ES achieves 79.88±0.6879.88 \pm 0.68, compared with 79.14±0.4679.14 \pm 0.46 for Baseline and 79.60±0.4379.60 \pm 0.43 for ES without the differential term. The 0.74-point improvement in mean GLUE score provides additional evidence that ES maintains downstream performance in this setting, beyond the original single-seed comparison.

AuthorsOpenReview ↗

Response to Reviewer iQaz (Part 3)

5. Dedicated Limitations Section

We agree and will add an explicit Limitations subsection in Section 5, highlighting:

  1. Multi-epoch / repeated-exposure requirement: Because ES tracks per-sample historical loss states si(t)s_i(t) and loss variations across visits, it requires multiple passes. In strictly single-pass streaming pre-training where every document is seen only once, per-sample historical EMA states are unavailable without auxiliary grouping or proxy scoring.
  2. Industrial-scale validation: While we validate ES(WP) up to 300B-token pre-training (GPT-2-Large/XL) and 9B-parameter fine-tuning, deploying and stress-testing ES(WP) in industrial trillion-scale model/token pre-training pipelines (with 3D parallelism and dynamic domain mixtures) remains important future work.

6. Verification and Clarification of Eq. (B.49)

We carefully double-checked Eq. (B.49) in Appendix B.4 and confirm that both Eq. (B.49) and the resulting transfer function in Eq. (B.45) are algebraically consistent under the continuous-time ODE system in Eq. (B.44):

  • Starting from the continuous-time system Eq. (B.44):

w(t)=s(t)+β2−β11−β2,s′(t),s′(t)=(1−β2)(ℓ(t)−s(t)),w(t) = s(t) + \frac{\beta_2 - \beta_1}{1 - \beta_2}\\,s'(t), \qquad s'(t) = (1 - \beta_2)\big(\ell(t) - s(t)\big),

  • Substituting s′(t)=(1−β2)(ℓ(t)−s(t))s'(t) = (1-\beta_2)(\ell(t) - s(t)) directly into the first equation cancels (1−β2)(1-\beta_2) in the numerator and denominator:

w(t)=s(t)+(β2−β1)(ℓ(t)−s(t))=(β2−β1)ℓ(t)+(1−β2+β1)s(t),w(t) = s(t) + (\beta_2 - \beta_1)\big(\ell(t) - s(t)\big) = (\beta_2 - \beta_1)\ell(t) + (1 - \beta_2 + \beta_1)s(t),

which is exactly Eq. (B.49).


We hope these clarifications, theoretical extensions, and additional multi-seed and large-scale baseline comparisons thoroughly resolve your concerns, and we welcome any further questions during the discussion period.

I thank the authors for their thorough answers and revision. Although I find the authors’ discussion a little bit outdated—some recent methods beyond InfoBatch, such as OrderDP, should be compared, and Ordered SGD should not be used as a substitute—during my checking I found a very similar work (Cheng et al., Data-Efficient Training by Evolved Sampling, 2024) that the authors do not cite, which may be a serious issue

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.