acceptodds
Under review as a conference paper at ICLR 2027

Data2Delta: Gradient Geometry for Budgeted Pretraining Data Selection

Abstract

When deduplication leaves more data than a pretraining budget permits, a selector must choose among examples that are not literal copies. We introduce Data2Delta, which measures redundancy in a fixed probe's gradient space: it clusters normalized gradient sketches and globally removes examples with low within-cluster ridge leverage. The cluster-average score is its effective dimension divided by its size, linking deletion allocation to directional redundancy without explicit cluster quotas. Two controlled Pythia-1B comparisons support the representation and membership choices: under the same selector, gradients improve endpoint perplexity (PPL) over dimension-matched BGE-M3 features; with each cluster's deletion count fixed, leverage-ranked membership improves PPL over random membership. End to end, Data2Delta lowers PPL by 0.0154 relative to uniform selection. Curve interpolation translates this gap into about 19% fewer effective prediction tokens to reach uniform selection's endpoint PPL. Low-NLL filtering achieves similar PPL, yet loses over 13 times as many source families in a separate natural-news evaluation at 10% removal. Training-pool provenance analysis also distinguishes the selections, while mean zero-shot accuracy is lower for Data2Delta than for the controls on the perturbed pool. These results support gradient geometry as a candidate-relative selection signal and distinguish its training-fit gains from source coverage and downstream accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pretraining data selection, gradient representations, ridge leverage scores, data redundancy, language model pretraining

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Y9qiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Data2Delta, a pretraining data selection method based on gradient geometry. The method uses a fixed probe model to extract projected gradients for each training example, clusters the gradient representations, and removes samples with low ridge leverage scores. The authors also provide a mathematical explanation connecting leverage scores to the effective dimension of each cluster.

The method is evaluated through continued pretraining of Pythia-1B, with comparisons against random selection, loss-based filtering, and embedding-based methods. The experiments show improvements in held-out PPL over random selection, while achieving similar results to Low NLL filtering. The paper also includes ablation studies on representations and selection strategies, as well as experiments on source coverage and data selection without synthetic perturbations. However, the improvements do not translate into better average zero-shot performance.

Critical strengths

The method has a clear mathematical basis. The authors relate the average ridge leverage score within a cluster to its effective dimension, providing an explanation for the proposed selection strategy.

The paper includes several useful controlled experiments. The authors compare gradient and BGE representations with matched dimensions and test leverage-based selection against random selection while keeping the deletion count fixed within each cluster. The NEWS-COPY analysis is also useful, showing that Data2Delta preserves more source families than Low NLL, although BGE-leverage performs slightly better on this metric.

Critical weaknesses

My main concern is whether the performance gains justify the complexity of the method. Data2Delta achieves lower PPL than random selection, but performs almost the same as the much simpler Low NLL baseline, with no clear gains on downstream zero-shot tasks. The paper also lacks comparisons with stronger data selection methods, particularly other selection strategies using the same gradient features. Since the method requires additional gradient extraction, clustering, and leverage score computation, I think an end-to-end efficiency comparison is necessary to show whether these small improvements are worth the extra cost.

I am also concerned about how well the method works on natural pretraining data. The main PPL improvements come from experiments with artificially injected perturbations. Without these perturbations, the improvement over random selection becomes much smaller and is not statistically significant. The training budgets are different, so the two settings are not directly comparable, but this still leaves questions about generalization. The NEWS-COPY experiments only measure source coverage, without training a model on the selected data. As a result, it is still unclear whether the method provides meaningful training benefits on naturally duplicated data.

Questions for authors

Data2Delta has a higher selection cost than Low NLL, but their PPL results are almost identical. Is there any end-to-end runtime or compute comparison that includes gradient extraction, clustering, and leverage score computation? Does Data2Delta still offer an efficiency advantage over Low NLL after accounting for these costs?

Could the authors provide comparisons with stronger data selection baselines under the same dataset and training budget? The current comparisons are not sufficient to demonstrate the advantages of the proposed method over existing approaches.

The main PPL improvements are observed in the experiments with artificially injected perturbations, while the improvement over random selection is much smaller and not statistically significant in the original-only experiments. Is the improvement mainly driven by removing synthetic noise? Could the authors provide a comparison under matched training budgets to clarify this?

2 comments
AuthorsOpenReview ↗

Comment 1/2 — W1/Q1: end-to-end cost and efficiency

Thank you for the constructive review. Several of your questions can be answered from the saved artifacts of the submitted study; we report these first.

Measured selection cost. From the run metadata (A800 GPUs), gradient extraction for all 659,814 rows took 1.99 GPU-h, and clustering plus ridge scoring at most 0.08 GPU-h on one GPU (this bound also includes two other variants). Low-NLL scoring took 0.71 GPU-h, BGE-M3 encoding 0.86 GPU-h, and one 2.5B-token training run 29.9 GPU-h. Data2Delta's full selection therefore costs 6.9% of one training run, and low NLL 2.4%. End to end, including tokenization and I/O (GPU-h, A800):

StageData2DeltaLow NLLBGE leverageUniform
Model pass (tokenization, model load, scoring)1.990.710.860
Feature I/O, k-means, ridge, sort, write selected set≤0.08 (93 s measured)<0.01≤0.080
Total (share of one 29.9 GPU-h training run)≤2.07 (6.9%)0.71 (2.4%)≤0.94 (3.2%)0

Tokenizing the retained set for training (8.5 CPU-minutes) is the same for every method.

Equal total compute. Charging each method's selection cost by truncating its own training curve (seeds 300–302):

  • Data2Delta vs uniform: −0.0107 PPL in all three seeds, about 12.5% net token savings on the perturbed pool.
  • Data2Delta vs low NLL: +0.0028 PPL (0.02%), in favor of low NLL on this pool.

We confirmed both by actually training further: the saved uniform and low-NLL runs were continued from update 2,384 with their exact optimizer, data-order and RNG state (a restored continuation reproduces an original run to 9.4×10⁻⁶ NLL), by the 165 and 108 extra updates their cheaper selection buys. Uniform still trails Data2Delta by 0.0109–0.0116 PPL in all three seeds; the extra training recovers only 27% of the gap. Low NLL leads by 0.0023–0.0027 PPL (p=0.001, all three seeds; 0.02% of PPL), matching the curve-based estimate.

When the extra cost pays off. Low NLL's efficiency on the perturbed pool relies on its highest-loss rows being corrupted copies. On natural data this is no longer true: on the original-only pool low NLL is worse than uniform selection (+0.0034 in all three replicates; exploratory), because it removes hard, informative chunks (on the main pool it removes 23.0% of the hardest decile of original chunks, versus 6.0% for uniform and 2.4% for Data2Delta), and Data2Delta beats it in all three replicates (−0.0039, raw p=0.021, Holm p=0.064; Table 13). Selection is also computed once per pool and transfers across target sizes: the same original-only masks trained Pythia-410M, 1B and 1.4B (Comment 2/2), amortizing the one-time extraction over a model sweep. The extraction can also be made cheaper without changing the result: the sketch needs only the sketched layer's input and output gradient, so we stopped backpropagation at that layer and formed no other weight gradients. On 2,048 rows the sketches are bit-identical (maximum absolute difference 0) and throughput rises 1.66× (153 vs 92 rows/s per GPU), which puts the same selection at about 1.28 GPU-h (4.3% of a run) and its extra cost over low NLL at 1.9%.

AuthorsOpenReview ↗

Comment 2/2 — Q2 (stronger baselines) and W2/Q3 (natural data, matched budgets)

Q2: baselines on the same data and budget. All runs use the same residual, retained size (586,921), initialization, 2.5B-token budget and seeds 300–302:

  • Same gradient sketches and clusters as Data2Delta: gradient-cosine (SemDeDup rule), exact greedy log-det (the exact greedy optimizer of the objective in Proposition 1), DBP, and DiSF (on 1,024-d sketches, paired with our 1,024-d Data2Delta run).
  • Text-embedding diversity selectors on BGE-M3: SemDeDup and DBP.

Holm correction is applied within each family. Results (Data2Delta 13.2234, uniform 13.2389; Δ = Data2Delta − baseline, 10⁻³ PPL, paired 95% interval):

Baseline (features)PPLΔHolm p
Gradient-cosine (8,190-d gradients)13.2254−1.95 [−2.96, −0.94]0.029
Greedy log-det (8,190-d gradients)13.2257−2.29 [−3.16, −1.41]0.023
DBP (8,190-d gradients)13.2262−2.79 [−4.89, −0.69]0.029
DiSF (1,024-d gradients; vs 1,024-d Data2Delta)13.2326−7.45 [−10.01, −4.88]0.006
SemDeDup (BGE-M3)13.2432−19.73 [−20.73, −18.72]0.0003
DBP (BGE-M3)13.2378−14.36 [−15.61, −13.10]0.0004

Data2Delta is better than every baseline in every seed and at all 30 saved checkpoints. Neither text-embedding selector improves on uniform deletion (13.2389): SemDeDup is 4.32×10⁻³ PPL worse and DBP only level with it (−1.05, n.s.). They spend 79–93% of their deletions on original text and leave most corrupted copies in place. These complement the existing controls: dimension-matched BGE under the same selector (−0.0131 PPL for gradients, Holm p=0.0015) and random membership at fixed cluster budgets (−0.00168, Holm p=0.0023).

Q3: is the main-pool gain driven by the injected copies? Yes, and we now quantify it at matched budgets using the saved checkpoints at 1.0B tokens (the original-only budget):

At 1.0B tokensData2Delta − uniformData2Delta − low NLL
Perturbed pool−0.0137 [−0.0142, −0.0133]−0.0002 [−0.0015, +0.0011]
Original-only pool−0.0005 [−0.0025, +0.0016]−0.0039 [−0.0064, −0.0014]

(paired 95% intervals over three seeds/replicates). On the perturbed pool the deletions that carry the gain are indeed the corrupted copies, and those copies are the redundancy class the method targets: non-lexical duplicates that standard deduplication leaves behind. MinHash catches only 3,136 of the 29,898 word-shuffled copies, and BGE-based selection removes 1,523 of the survivors, whereas Data2Delta removes 23,000 of them using no labels at all (Table 24). On natural data, the relevant comparison is with loss filtering, which becomes harmful, while Data2Delta does not.

Two new runs make this concrete:

  • Original-only pool at the matched 2.5B budget (continuing the saved runs, three replicates; Data2Delta minus control, 10⁻³ PPL): uniform −0.33 (n.s.), BGE leverage −1.36, low NLL −3.34 (lower in all replicates and at all 18 checkpoints; Holm p=0.24 and 0.069). The ordering of the 1B-token results persists at 2.5B, and it also holds when the same masks train Pythia-410M and 1.4B targets (vs low NLL −3.95 and −4.49, all replicates).
  • Oracle composition control: deleting Data2Delta's numbers of originals, template and word-shuffled rows at random within each type, using the true labels, gives −14.67×10⁻³ vs uniform, i.e. 95% of Data2Delta's gain. Without any labels, Data2Delta matches this oracle and improves on it through its within-type choice (−0.74×10⁻³ [−1.40, −0.08], p=0.041; 25 of 30 checkpoints).

NEWS-COPY. With 4,988 and 14,211 texts per split it is too small for a meaningful pretraining run; it serves as a labelled test of selection behavior on natural duplicates, where Data2Delta loses 79 source families and low NLL 1,060 (Table 14).

Reviewer_AkKgOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies pretraining data selection under a fixed budget, using gradient geometry to identify redundancy beyond text duplication. The authors propose Data2Delta, which computes normalized gradient sketches from one layer of a fixed probe model, clusters the examples, and removes those with the lowest within-cluster ridge leverage scores. Low scores indicate that an example’s gradient direction is already well represented by other cluster members. A global ranking determines both how many examples to remove from each cluster and which members to remove. The selected dataset remains fixed throughout subsequent training.

The authors evaluate Data2Delta by continuing pretraining of Pythia-1B on a Pile subset with added synthetic perturbations. They find that gradient features give lower perplexity than semantic embeddings. Using leverage scores to choose which examples to delete also performs slightly better than deleting examples randomly while keeping each cluster’s deletion count unchanged. Data2Delta improves perplexity over random selection, although simple low-loss filtering achieves almost the same result. From the training curves, the authors estimate that Data2Delta needs about 19% fewer tokens to match random selection’s final perplexity, without counting selection costs. A separate NEWS-COPY experiment shows that it preserves more source families than low-loss filtering, but does not evaluate training on the selected news data. In the experiment without added perturbations, the gains are much smaller, and the main experiments show no improvement in average zero-shot accuracy.

Critical strengths
  1. The selection rule has a clear geometric interpretation. It considers how well each example’s gradient direction is represented by other members of its cluster. Using one ranking to decide both how much to remove from each cluster and which examples to remove gives the method a coherent design.
  2. The experiments examine both model performance and the data retained after selection. The paper reports perplexity, downstream accuracy, and source coverage, alongside comparisons with simple baselines and ablations of the main design choices. This gives a useful view of the method’s behavior and helps identify where its benefits are limited.
Critical weaknesses
  1. The methodological novelty appears limited. Previous methods such as DBP[1] and DiSF[2] already use cluster structure or feature geometry to guide pruning. Data2Delta mainly combines gradient sketches with ridge leverage and a global ranking. Although its feature ablation supports using gradients, the paper does not show what this ranking adds over other ways of selecting diverse examples from the same gradient representations.
  2. The initial gradient scores are not validated across training. The method uses gradients from one layer of the starting checkpoint and keeps the selected dataset fixed for roughly eight passes. Gradient relationships and sample usefulness can change as the model learns, but the paper does not examine whether the initial redundancy scores remain relevant at later checkpoints. This leaves the connection between the selection criterion and subsequent training value unclear.
  3. The results do not establish a meaningful advantage over simpler selection methods. Low-loss filtering achieves almost the same perplexity in the main experiment, and average zero-shot accuracy does not improve. In the separate original-only experiment, the PPL advantage over random selection is only 0.00050. Better source coverage on NEWS-COPY is measured without training on the selected data, so it does not establish an additional benefit for the resulting model.
  4. The deletion rule can remove almost all members of a diverse group. Scores are computed once and do not account for how redundancy changes as other examples are removed. The paper reports deleting 4,873 of 4,877 members from a cluster containing diverse original texts. Its justification for removing one example does not establish what happens when many examples are removed together, leaving the risk of losing useful training information unresolved.
  5. The overall efficiency benefit remains unclear. The reported 19% token saving comes from interpolating training curves and excludes selection costs. Gradient extraction is already slower than low-loss scoring, before clustering and ridge computation are added. Since low-loss filtering achieves almost the same perplexity and token savings, the paper does not show that Data2Delta makes better use of the total compute budget.

[1] Effective Pruning of Web-Scale Datasets Based on Complexity of Concept Clusters. ICLR 2024. [2] Combatting Dimensional Collapse in LLM Pre-Training Data via Submodular File Selection. ICLR 2025.

Questions for authors
  1. Does ridge leverage lead to better training results than cosine-based selection? Appendix H compares how well the two rules remove synthetic perturbations, but does not compare model performance after training on their selected data. Could you add this comparison using the same gradient features, clusters, retained dataset size, and training budget? This would clarify whether ridge leverage provides a useful improvement beyond the gradient representation itself.
  2. Do the initial gradient scores remain useful as training progresses? Could you recompute the gradient features and scores at one saved later checkpoint, keeping the cluster assignments unchanged? A short continuation comparison using the original versus updated selection, with the same training budget, would help test whether the fixed selection remains appropriate as the model’s learning needs change.
  3. Does removing most of a cluster discard useful training information? The C5 analysis shows that many diverse original texts are removed together. Could you examine whether retaining more of these examples improves model performance under the same data budget? This would help assess whether the current rule removes useful information along with redundant examples.
  4. Does Data2Delta achieve better performance under the same total compute budget? Could you compare it with low-loss filtering and random selection, accounting for the cost of both data selection and training? This would clarify whether its additional selection cost is justified by the final performance.
5 comments
AuthorsOpenReview ↗

Comment 1/3 — W1 and Q1: what the ranking adds beyond the gradient representation

Thank you for the concrete questions; we ran each of them as you specified. Existing evidence first.

What the paper isolates. Data2Delta has two ingredients: (i) a representation, normalized projected gradients; (ii) a selection rule, one closed-form within-cluster ridge-leverage score per row and one global sort, which by Proposition 1 orders single deletions by their decrease of ∑clog⁡det⁡(Mc+λI)\sum_c \log\det(M_c + \lambda I) and has cluster mean dλ(c)/ncd_\lambda(c)/n_c. The submission tests them separately: with the selector fixed, gradients beat dimension-matched BGE-M3 by 0.0131 PPL (Holm p=0.0015), worth 16.8% of training tokens (Tables 10–11); with allocation fixed, leverage-ranked membership beats random membership by 0.00168 PPL (Holm p=0.0023; Table 3).

Relation to DBP and DiSF. DBP (Abbas et al., 2024) computes a complexity score per embedding cluster (intra- × inter-cluster distance), turns it into per-cluster keep quotas through a temperature softmax and a bounded quadratic program, and then prunes inside each cluster by distance to the centroid. DiSF (Fan et al., 2025) greedily selects files so that the covariance spectrum of text-encoder features becomes uniform, within fixed-size batches. Data2Delta needs no temperature, quota or greedy pass: allocation and membership both follow from one score with an explicit log-det interpretation. We agree that the paper did not show what this rule adds over other rules on the same gradient features; the experiment below does.

Q1, as specified. Same 8,190-d gradient sketches, same residual clusters, same retained size (586,921), same 2.5B-token budget, seeds 300–302:

  • (a) gradient-cosine, i.e., the SemDeDup-style rule of Appendix H, now trained;
  • (b) exact greedy log-det: repeatedly delete the lowest current leverage with rank-one downdates, the exact greedy optimizer of the objective in Proposition 1;
  • (c) DBP on the same 512 gradient clusters: complexity = mean intra-cluster cosine distance × mean distance to the 20 nearest centroids, softmax keep-quotas (τ=0.1\tau=0.1, chosen from 0.1/0.5/1.0) via bounded projection, then delete the rows closest to each centroid;
  • (d) DiSF on 1,024-d gradient sketches: batches of 1,024 rows by the frozen shared rank, column z-scoring, greedily deleting the row whose removal most reduces the off-diagonal Frobenius energy of the feature covariance; paired with our existing 1,024-d Data2Delta run (Table 9).

Holm correction is applied within the 8,190-d family (a–c) and within the 1,024-d family (d) separately; we also report the sign consistency over all 30 saved checkpoints, because three paired endpoints have little power for differences of order 1e-3.

Result (Δ = Data2Delta − rule, 10⁻³ PPL, paired 95% interval):

Rule (features)PPLΔHolm pD2D lower atWord-shuffled deletedC5 deleted
Data2Delta (8,190-d)13.2234———23,0004,873
(a) gradient-cosine13.2254−1.95 [−2.96, −0.94]0.02930/3019,695656
(b) greedy log-det13.2257−2.29 [−3.16, −1.41]0.02330/3018,9631,947
(c) DBP13.2262−2.79 [−4.89, −0.69]0.02930/3020,4742,439
(d) DiSF (1,024-d; vs 1,024-d Data2Delta, 13.2251)13.2326−7.45 [−10.01, −4.88]0.00630/3011,646328

(PPL columns are rounded means; each Δ is the mean of the three paired per-seed differences.) Data2Delta is better than every rule in every seed and at every saved checkpoint, and all four differences survive Holm correction: 1.95–2.79×10⁻³ PPL against the 8,190-d rules, 7.45×10⁻³ against DiSF, and more corrupted copies removed than by any of them. Tables 10–11 measure what the representation contributes; this table measures what the rule contributes on top of it. We will add it to Section 4.

AuthorsOpenReview ↗

Comment 2/3 — W2/Q2 (validity across training) and W4/Q3 (cluster C5)

Q2, as specified. We take the saved checkpoint at update 1,192 (50% of training) of the Data2Delta run, recompute the 8,190-d sketches for all 624,822 candidates, keep the 512 cluster assignments, rescore, and continue to update 2,384 with (i) the original and (ii) the refreshed selection, three training seeds each. Both arms go through the same restart procedure, since restart effects are not assumed to be zero (App. A.4). The initial scores remain valid at mid-training. The refreshed scores have Spearman 0.996 with the original ones and 97.9% of the deletions are unchanged; the 804 rows that leave the deletion set are 298 originals, 267 word-shuffled and 239 template copies, and C5 is still emptied. The refreshed selection lowers PPL by 1.54×10⁻³ relative to the original one (95% interval [−4.39, +1.31], Holm p=0.29; lower in all three seeds and at 14 of 15 checkpoints). The restart itself is neutral (+0.15×10⁻³, p=0.79). One refresh is therefore a cheap option (2.0 GPU-h), not a requirement.

Existing evidence on the same question: the advantage of the one-shot selection does not fade over the eight passes. Relative to uniform selection it grows from −0.0096 PPL at 0.25B tokens to −0.0137 at 1.0B and −0.0154 at 2.5B (Figure 2a), and Data2Delta is ahead of shared-rank random at every saved checkpoint in both seed groups (Section 4.1; Appendix G).

Q3 and W4: what happened in C5. We agree that one-shot scores do not adapt as other rows are removed, and C5 is the clearest case. A new analysis of the saved scores explains it:

  • Cluster-mean leverage correlates −0.967 with cluster size. When ncn_c approaches dd, every row has a larger share of its energy in the span of the others, so even random directions score lower in larger clusters. Using the Marchenko–Pastur law we computed this baseline in closed form (checked by simulation to 1e-5): at C5's size (4,877) random directions have mean leverage 0.9767, only 0.0022 above the global cut (0.9745).
  • C5's internal geometry is that of an ordinary cluster: mean pairwise cosine 0.022 and a top direction carrying 2.5% of the energy, the same as clusters of 1,100–3,700 rows that lose almost nothing (0.018–0.021; 1.9–2.2%). Corruption-dominated clusters are clearly different (0.069–0.088; 7.7–9.5%).
  • So C5 was removed because of its size, not because its gradients are redundant. Ranking by leverage minus the size baseline is label-free; it still removes 22,689 of the 23,000 word-shuffled copies that the main rule removes, deletes 2,542 instead of 4,873 C5 rows, and removes no cluster entirely (main rule: four).

Experiments, all at the same budget: (i) C5 restored and its 4,873 deletions passed to the next-ranked rows, exactly as you asked; (ii) the calibrated ranking; (iii) exact greedy log-det from Comment 1, the adaptive version you suggest. Results (variant − main rule, 10⁻³ PPL, three seeds): (i) C5 restored −0.81 [−2.24, +0.62]; (ii) calibrated ranking −0.61 [−1.47, +0.25]; both are lower in all three seeds (25 and 27 of 30 checkpoints respectively; Holm p=0.18 for both). (iii) Greedy log-det deletes 1,947 C5 rows and empties no cluster, but is 2.29×10⁻³ worse than the main rule (Comment 1) because it spends 4,254 more deletions on original text. Both variants are sign-consistent in the direction you suspected but not significant, so we claim only that retaining C5 costs nothing: the calibrated ranking removes the wipe-out without labels and without a detectable loss of fit, whereas full greedy adaptation is not the right fix.

We will add the size analysis to Section 2 and the C5 experiments to Section 4.5, and offer the calibrated ranking in Algorithm 1. We thank you for this point; it led to a clearer understanding of the score.

AuthorsOpenReview ↗

Comment 3/3 — W5/Q4 (total compute) and W3 (simpler methods)

Q4: measured cost and equal-compute comparison. The measured end-to-end costs are tabulated below. We charge each method's selection cost by truncating its own training curve at 2.5B tokens minus that cost and compare at equal total compute (seeds 300–302):

  • Data2Delta vs uniform: −0.0107 PPL, all three seeds; about 12.5% net token savings.
  • Data2Delta vs low NLL: +0.0028 PPL, about 0.02% of PPL, in favor of low NLL on this pool.

We confirmed both by continuing the saved uniform and low-NLL runs from update 2,384 with their exact optimizer, data-order and RNG state by the equivalent extra updates (+165 and +108); a restored continuation reproduces an original run to 9.4×10⁻⁶ NLL. Uniform still trails Data2Delta by 11.25×10⁻³ PPL (p<0.001, all seeds), and low NLL leads by 2.54×10⁻³ (p=0.001, all seeds; 0.02% of PPL); both match the curve-based estimates. End-to-end selection cost on the 659,814-row pool (GPU-h, A800):

StageData2DeltaLow NLLBGE leverageUniform
Model pass (tokenization, model load, scoring)1.990.710.860
Feature I/O, k-means, ridge, sort, write selected set≤0.08 (93 s measured)<0.01≤0.080
Total (share of one 29.9 GPU-h training run)≤2.07 (6.9%)0.71 (2.4%)≤0.94 (3.2%)0

Tokenizing the retained set for training takes 8.5 CPU-minutes for every method, including uniform. Stopping backpropagation at the sketched layer gives bit-identical sketches at 1.66× the throughput, so the same selection costs about 1.28 GPU-h (4.3%).

Two considerations matter for the comparison with low NLL. First, low NLL's efficiency on this pool comes from the fact that its highest-loss rows are corrupted copies. On natural data the highest-loss rows are hard, informative text: on the original-only pool low NLL is worse than uniform (+0.0034 in all three replicates; exploratory) and is beaten by Data2Delta in all three replicates (−0.0039, raw p=0.021, Holm p=0.064). On the main pool, low NLL removes 13,383 of the 58,162 hardest-decile original chunks (23.0%), Data2Delta 1,421 (2.4%). Second, selection is paid once per pool and transfers across target sizes: the same original-only masks trained Pythia-410M, 1B and 1.4B (below), so the one-time 1.28–2.07 GPU-h is amortized over a model sweep.

W3. On the perturbed pool the two rules agree on most corrupted copies (21,948 word-shuffled copies deleted by both) and differ on originals: low NLL alone deletes 12,554 originals with mean probe NLL 4.42, Data2Delta alone deletes 8,124 originals with mean NLL 2.45 (4,826 of them in C5) plus 5,776 template copies that low NLL keeps. This explains equal PPL with different coverage. We agree the original-only gain over uniform (−0.0005 at 1B tokens) is small. At the matched 2.5B-token budget the ordering persists (three replicates, Data2Delta minus control): uniform −0.33×10⁻³ (n.s.), BGE leverage −1.36×10⁻³ and low NLL −3.34×10⁻³ (lower in all replicates and at all 18 checkpoints; Holm p=0.24 and 0.069). With the same masks on Pythia-410M and 1.4B, Data2Delta beats low NLL (−3.95 and −4.49×10⁻³, Holm p=0.011 and 0.067, every replicate) and matches uniform (−0.07 and −1.13×10⁻³, both n.s.). NEWS-COPY has 4,988 and 14,211 texts per split, too small for a meaningful pretraining run; it serves as a labelled test of selection behavior on natural duplicates, where Data2Delta loses 79 source families and low NLL 1,060.

Reviewer_8PUfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposed a deduplication method, Data2Delta, which adopted candidate-relative gradient geometry to select pretraining data. Based on a single checkpoint, Data2Delta first projected its gradients for each candidate, and then normalized their magnitudes and only retain the directions. Finally, it removed the examples with lowest within-cluster ridge leverage. This paper compared its Data2Delta with NLL-based methods and BGE-leverage method.

Critical strengths
  1. The problem and motivation is clear
  2. The investigation of related works is comprehensive.
Critical weaknesses

I think this work is too expensive and it is not suitable for the pretraining scenarios that it described. Furthermore, the experiments are not comprehensive and convincing, such as the data scale, ablations, and baseline comparisons. Details are described in questions as below.

Questions for authors
  1. Does the gradient of the first 256 tokens really represent the whole 512-token chunk?
  2. Why is the training sequence length set as 512? It seems to be far shorter than industrial settings. What if the sequence length were changed to 4k, 8k, or even 32k? Does this affect the method itself?
  3. How much does the checkpoint choice affect the results? The gradient of a given example, or of examples from the same category, seems to change over training. Why is a single checkpoint enough?
  4. It seems that this is a diversity-based selector, but why is it not compared with other diversity-based data selection methods?
  5. The candidate pool is only 0.3B tokens, which does not support the pretraining scenarios. Furthermore, the computational cost is expensive. Is this actually applicable to large-scale pretraining data selection?
  6. For pretraining, selected data is supposed to stay useful when models get larger. From the perspective of loss value, keeping only the easy, low-loss examples therefore tends to fail at a larger scale. But this paper does not analyze this phenomenon.
6 comments
AuthorsOpenReview ↗

Comment 1/3 — Cost, scale and baselines (Weaknesses, Q5, Q4)

Thank you for the review. We address cost and scale first, since they underlie several questions.

Measured cost (Q5). From the saved run metadata (A800 GPUs): gradient extraction for all 659,814 candidates took 1.99 GPU-h (about 94 texts/s per GPU, i.e., roughly 3 GPU-h per million 256-token windows), clustering and ridge scoring at most 0.08 GPU-h, while one 2.5B-token training run took 29.9 GPU-h. The whole selection is 6.9% of one training run, and extraction is embarrassingly parallel across shards. After charging this cost, Data2Delta still reaches better PPL than uniform selection at equal total compute (−0.0107 PPL, all three seeds), about 12.5% net token savings; actually continuing the uniform runs by the 165 updates this cost would buy leaves them 0.0109–0.0116 PPL behind. The selected set is computed once and transfers across target sizes: the same original-only masks trained Pythia-410M, 1B and 1.4B (Comment 3/3), so extraction is amortized over a model sweep rather than charged to one run.

Scale (Q5). The scoring step is linear in the number of candidates for fixed K and sketch dimension: within-cluster ridge scoring costs O(∑cncdqc+qc3)O(\sum_c n_c d q_c + q_c^3) with qc=min⁡(nc,d)q_c = \min(n_c, d) (Appendix F.2), and the final step is one sort. We timed clustering, scoring and sorting with the main settings on one A800-40GB GPU, using existing main-configuration sketches of a 9.76M-chunk pool (about 5B tokens) from the same Pythia stream: 0.6M rows 38 s, 2.4M rows 132 s, 9.76M rows 557 s (k-means 171 s, ridge 385 s, sort 1.5 s), with 14 GB peak GPU memory. Selection over a 5B-token pool therefore takes minutes on one GPU and grows linearly; extraction dominates at about 3 GPU-h per million rows, or 1.8 GPU-h when backpropagation stops at the sketched layer (bit-identical sketches). We position Data2Delta as a second-stage selector after cheap filters (MinHash in our pipeline), applied to the pool that deduplication leaves over budget, rather than as a first-pass filter over raw crawls.

Diversity baselines (Q4). We agree they were missing and have added six, on the same residual, budget and 2.5B-token protocol (three seeds each; Data2Delta minus baseline, 10⁻³ PPL, Holm-adjusted within feature family):

  • same gradient features: gradient-cosine (SemDeDup rule) −1.95 (p=0.029), exact greedy log-det −2.29 (p=0.023), DBP −2.79 (p=0.029), DiSF −7.45 (p=0.006, vs our 1,024-d run);
  • BGE-M3 features: SemDeDup −19.73, DBP −14.36 (Holm p=0.0003 and 0.0004 within the two BGE contrasts).

Data2Delta has the lower PPL in every seed and at all 30 saved checkpoints for every baseline. Neither text-embedding selector improves on uniform deletion: SemDeDup is 4.32×10⁻³ PPL worse than uniform and DBP only level with it (−1.05, n.s.). They spend 79–93% of the budget on original text and leave most corrupted copies in place. The submission already shows that, with the selector fixed, gradients beat dimension-matched BGE-M3 by 0.0131 PPL (Holm p=0.0015), worth 16.8% of training tokens (Tables 10–11).

AuthorsOpenReview ↗

Comment 2/3 — Q1 (256 tokens), Q2 (sequence length), Q3 (checkpoint)

Q1. The 256-token sketch is a cost-saving choice, and the paper tests its boundary: suffixes placed entirely after token 256 leave the deletion mask unchanged (Appendix D.1), whereas a template visible in the window raises residual deletion of unrelated originals from 1.75% to 30.97%. We recomputed the sketches over the full 512-token chunk, re-ran clustering and selection, and trained three seeds. The two selections remove largely the same data: 75.9% of the deletions are shared, and the 512-token selection removes 23,523 word-shuffled copies (256 tokens: 23,000). Its PPL is 0.63×10⁻³ lower (all three seeds; Holm p=0.16) at 1.9× the extraction cost (3.7 vs 2.0 GPU-h). The first 256 tokens therefore capture almost all of the signal, and the full window is a small, consistent upgrade when budget allows; it also no longer empties cluster C5 (93 of 4,877 rows deleted).

Q2. The sketch computation does not use the training context: each candidate is mapped to one sketch, and clustering and leverage depend only on the sketches, so the training sequence length does not enter the selection and only changes how retained documents are packed for training. This is a statement about the computation; we do not claim beyond it that the endpoint benefit transfers to longer-context training. For long documents, the sketch accumulates tokenwise outer products, so its cost grows linearly with the scored tokens, and long documents can be scored over sampled windows. Pythia's native context is 2,048 tokens, so 4k–32k contexts cannot be tested without context extension.

Q3. We test it directly: at the saved checkpoint at update 1,192 (50% of training) we recompute the sketches, keep the clusters, rescore, and continue to 2.5B tokens with the original and the refreshed selection under the same restart procedure. The scores barely move: the refreshed ranking has Spearman 0.996 with the original, and 97.9% of the deletions are unchanged. Continuing with the refreshed selection lowers PPL by a further 1.54×10⁻³ (all three seeds, 14 of 15 checkpoints; Holm p=0.29), while the restart itself is neutral (+0.15×10⁻³ vs the uninterrupted run). This is consistent with one checkpoint being adequate; the refresh has a lower mean in every seed but the difference is not significant, so we offer it as a cheap option (2.0 GPU-h) rather than a necessity. The existing curves already show that the one-shot selection keeps paying off during training: its advantage over uniform selection grows from −0.0096 PPL at 0.25B tokens to −0.0137 at 1.0B and −0.0154 at 2.5B (Figure 2a), and it is ahead of shared-rank random at every saved checkpoint in both seed groups (Section 4.1).

AuthorsOpenReview ↗

Comment 3/3 — Q6: does the selected data stay useful, and is this a low-loss filter?

The submitted experiments already show the phenomenon you describe, for loss filtering and not for Data2Delta.

Loss filtering fails on natural data; Data2Delta does not. On the original-only pool (no injected copies; Tables 12–13), low-NLL filtering has the worst PPL of the four selectors and is worse than uniform selection (+0.0034 in all three replicates; exploratory comparison), while Data2Delta has the best PPL and beats low NLL in all three replicates (−0.0039, raw p=0.021, Holm p=0.064). This matches prior observations that keeping only low-perplexity data is a poor pretraining strategy (Marion et al., 2023; Ankner et al., 2025).

Data2Delta keeps hard examples. It scores directions, not loss: row normalization removes the gradient norm, and its scores correlate only weakly with probe NLL (Spearman −0.21; Appendix M.3). Because its budget goes mostly to corrupted copies, it removes 1,421 of the 58,162 hardest-decile original chunks by probe NLL (2.4%), versus 3,497 for uniform selection (6.0%) and 13,383 for low NLL (23.0%). Low NLL's 13,383 original deletions are exactly that hardest decile; Data2Delta deletes 8,953 originals in total. On the perturbed pool its high-loss deletions are the word-shuffled copies (mean probe NLL about 6.0), which are noise rather than hard data. On natural news duplicates it loses 79 source families versus 1,060 for low NLL (Table 14).

Larger models. We trained Pythia-410M and Pythia-1.4B (step 5,000; all Pythia sizes share one data order, so the pool is unseen) for 1B tokens on the original-only selections made with the 1B probe, three replicates each (Data2Delta minus control, 10⁻³ PPL):

Targetvs uniform (10⁻³ PPL)vs low NLL (10⁻³ PPL)
Pythia-410M−0.07−3.95 (Holm p=0.011)
Pythia-1B (Table 13)−0.50−3.9
Pythia-1.4B−1.13−4.49 (Holm p=0.067)

Data2Delta beats low NLL in every replicate and at every checkpoint at all three sizes, while staying level with uniform selection at each size (all n.s.). The failure you anticipate appears for loss filtering, not for Data2Delta, and it does not shrink as the model grows; a mask computed once with a small probe transfers to larger targets.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Comment 1/3 — Summary of new evidence

We thank all reviewers for their careful reading. Several questions can be answered from the saved artifacts of the submitted study; we summarize those first.

  1. Gradient geometry is a better redundancy signal than text embeddings. With selector, dimension (1,024), clustering, λ\lambda and budget held fixed, replacing BGE-M3 with gradient sketches lowers endpoint PPL by 0.0131 (Holm p=0.0015; Table 10) and is worth 16.8% of the training tokens (Table 11). At 8,190 dimensions, gradients improve over cached BGE and over same-probe activations by 0.0153 and 0.0179 (Holm p=0.00018; Table 5).

  2. The gain survives full selection cost. From the saved run metadata, gradient extraction for the full 659,814-row pool took 1.99 GPU-h on A800s and clustering plus ridge scoring at most 0.08 GPU-h, versus 29.9 GPU-h for one 2.5B-token training run (6.9%). Charging this cost by truncating Data2Delta's training curve, Data2Delta still improves over uniform selection by 0.0107 PPL at equal total compute (all 3 seeds), about 12.5% net token savings on the perturbed pool. We confirmed this with real training: continuing the saved uniform runs by the 165 updates that Data2Delta's selection cost would buy leaves uniform 0.0109–0.0116 PPL behind in all three seeds, recovering only 27% of the gap. On this pool the same accounting favours low NLL, which leads by 0.0023–0.0027 PPL (p=0.001), i.e. 0.02% of the PPL value; Point 3 is where the two rules separate.

  3. On natural data, low-NLL filtering degrades fit and Data2Delta does not. On the original-only pool, Data2Delta has the lowest PPL of the four selectors and is ahead of low NLL in every replicate (−0.0039, raw p=0.021; Holm p=0.064, so this family does not clear correction at n=3). Low NLL is the only selector that degrades fit there, falling behind even uniform selection (+0.0034 in all three replicates; exploratory). The reason is that the highest-loss natural chunks are hard, informative text: on the main pool low NLL removes 23.0% of the hardest decile of original chunks, uniform 6.0%, and Data2Delta 2.4%.

  4. Fit and coverage together. At similar fit to loss filtering on the perturbed pool, Data2Delta loses 13× fewer natural source families on NEWS-COPY (79 vs 1,060 at 10% removal; Table 14).

  5. A mechanism and a fix for concentrated deletion (cluster C5). Cluster-mean leverage correlates −0.967 with cluster size. Using the Marchenko–Pastur law we derived the expected leverage of random directions for each cluster size; at C5's size (4,877) it is 0.9767, barely above the global cut (0.9745). C5's internal geometry (mean pairwise cosine 0.022; top direction 2.5% of energy) matches ordinary clusters that lose almost nothing, so its removal reflects cluster size rather than redundancy. Ranking by leverage minus this size baseline (label-free) still removes 22,689 of the 23,000 word-shuffled copies that the main rule removes, deletes 2,542 instead of 4,873 C5 rows, and removes no cluster entirely.

Additional runs. Unless noted, these use the same residual pool, budget, initialization and 2.5B-token protocol with seeds 300–302 and pre-registered Holm families; the original-only runs use replicates with seeds 500–502, and the target-transfer runs use 1B tokens.

  • Same gradient features and clusters: gradient-cosine (SemDeDup rule), exact greedy log-det, DiSF, DBP; text-feature SemDeDup and DBP (AkKg-Q1/W1, 8PUf-Q4, Y9qi-Q2).
  • Rescoring at the saved mid-training checkpoint with clusters fixed, then continuation (AkKg-Q2, 8PUf-Q3).
  • C5 restored under the same budget, and the size-calibrated ranking (AkKg-Q3/W4).
  • End-to-end timing and equal-compute continuation of uniform and low-NLL runs (AkKg-Q4, Y9qi-Q1).
  • Original-only pool at the matched 2.5B budget, and an oracle composition-matched control (Y9qi-Q3, AkKg-W3).
  • Full 512-token sketches (8PUf-Q1). All of these runs are now complete; their results are in Comment 3/3, and we will upload a revised PDF.
AuthorsOpenReview ↗

Comment 2/3 — Positioning and paper changes

Positioning. Data2Delta is a second-stage selector for the regime where deduplication leaves more data than the budget allows. Its central claim is that candidate-relative gradient geometry identifies redundancy that text-level signals miss: on the perturbed pool it concentrates deletions, without using any labels, on corrupted copies that survive MinHash (23,000 word-shuffled copies, versus 1,523 for BGE; Table 24), and on natural data it preserves hard examples and source coverage where loss filtering does not.

What changes in the paper.

  1. Results: the same-feature selector comparison, rescoring, C5 experiments, matched-budget natural-data results, and an end-to-end cost table.
  2. Method: a paragraph on the cluster-size effect of within-cluster leverage with its closed-form random baseline, and the calibrated ranking as an option in Algorithm 1.
  3. Abstract and discussion: the equal-compute result (Point 2), the natural-data result (Point 3), and an explicit statement that main-pool gains come from removing non-lexical corrupted copies.
  4. Related work: precise descriptions of DBP (complexity-based cluster quotas with within-cluster pruning; Abbas et al., 2024) and DiSF (greedy covariance-spectrum uniformization; Fan et al., 2025), and a sharper statement of how Data2Delta differs from spectral-diversity objectives in general: it decomposes a within-cluster log-determinant into closed-form single-deletion scores whose cluster mean is the effective dimension per member (Proposition 1), and is used for budgeted pruning after deduplication rather than for generation or global reweighting.

Scope. Our evidence is Pythia-1B continuation with about eight passes over each selected set. Downstream average zero-shot accuracy (Avg8) does not separate the selectors. Every method moves Avg8 by less than 0.6 points from initialization, and the gaps between selectors (0.1–0.2 points) sit inside the question-sampling standard error (about 0.44 points, unpaired). The intervals in App. A.3 are training-order intervals only and do not include that error. Task directions also disagree: Data2Delta is higher than all five controls on HellaSwag and lower on ARC-Challenge (Tables 7–8). Our claims concern fit, cost and coverage.

AuthorsOpenReview ↗

Comment 3/3 — Update: all additional runs are complete

All runs listed in Comment 1 are complete (72 training runs). A restored continuation reproduces an original run to 9.4×10⁻⁶ NLL, so the continuation experiments are faithful extensions of the saved runs. Differences are in 10⁻³ PPL over three seeds or replicates, Data2Delta minus control unless stated; p-values are paired t-tests on endpoint NLL, Holm-adjusted within each family.

  1. Diversity baselines (AkKg-Q1, 8PUf-Q4, Y9qi-Q2). On the same gradient features, Data2Delta beats gradient-cosine (−1.95), exact greedy log-det (−2.29), DBP (−2.79) and DiSF (−7.45, 1,024-d); all Holm p<0.03, and Data2Delta is ahead at all 30 saved checkpoints for every rule. BGE-based SemDeDup and DBP trail Data2Delta by 19.73 and 14.36×10⁻³ PPL; against uniform deletion SemDeDup is worse (+4.32) and DBP only level (−1.05, n.s.).
  2. Equal total compute (AkKg-Q4, Y9qi-Q1). Uniform runs continued by the 165 updates that Data2Delta's selection cost buys remain 11.25×10⁻³ PPL behind (all seeds, p<0.001). Low NLL continued by 108 updates leads by 2.54×10⁻³ PPL (p=0.001; 0.02% of the PPL value) on this pool, where its highest-loss rows happen to be corrupted copies. Stopping backpropagation at the sketched layer yields bit-identical sketches at 1.66× throughput, cutting Data2Delta's selection cost from 6.9% to about 4.3% of a training run.
  3. Validity across training (AkKg-Q2, 8PUf-Q3). Rescoring at the mid-training checkpoint changes 2.1% of the deletions (Spearman 0.996) and lowers PPL by a further 1.54 (all seeds; n.s.).
  4. Cluster C5 (AkKg-Q3/W4). Restoring C5 and the label-free size-calibrated ranking sit 0.81 and 0.61×10⁻³ PPL below the main rule in all seeds; neither difference is significant, so we claim only that retaining C5 costs nothing. Exact greedy log-det avoids emptying clusters but is 2.29×10⁻³ PPL worse.
  5. Natural data and larger targets (Y9qi-Q3, AkKg-W3, 8PUf-Q6). Original-only pool at the matched 2.5B budget: uniform −0.33 (Holm p=0.60), BGE leverage −1.36 (Holm p=0.24), low NLL −3.34 (all replicates and checkpoints; raw p=0.023, Holm p=0.069). Masks selected with the 1B probe and used to train Pythia-410M and 1.4B: low NLL −3.95 (Holm p=0.011) and −4.49 (Holm p=0.067), lower at every replicate and checkpoint; uniform −0.07 and −1.13, both n.s.
  6. What drives the main-pool gain (Y9qi-Q3). A label-using oracle that deletes Data2Delta's per-type counts at random reaches 95% of its gain over uniform; Data2Delta matches it without labels and is 0.74 better (p=0.041).
  7. Full 512-token sketches (8PUf-Q1). 75.9% of deletions are shared with the 256-token selection; PPL is 0.63 lower (all seeds; n.s.) at 1.9× the extraction cost.

Dear reviewers, thank you for taking part in the discussion. As a reminder, the author–reviewer discussion period ends on November 18 (AoE).

8PUf, thank you for following up. Please feel free to share any additional comments before the deadline.

AkKg, thank you for following up. Please feel free to share any additional comments before the deadline.

Y9qi, thank you for following up. Please feel free to share any additional comments before the deadline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.