- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Data2Delta, a pretraining data selection method based on gradient geometry. The method uses a fixed probe model to extract projected gradients for each training example, clusters the gradient representations, and removes samples with low ridge leverage scores. The authors also provide a mathematical explanation connecting leverage scores to the effective dimension of each cluster.
The method is evaluated through continued pretraining of Pythia-1B, with comparisons against random selection, loss-based filtering, and embedding-based methods. The experiments show improvements in held-out PPL over random selection, while achieving similar results to Low NLL filtering. The paper also includes ablation studies on representations and selection strategies, as well as experiments on source coverage and data selection without synthetic perturbations. However, the improvements do not translate into better average zero-shot performance.
Critical strengths
The method has a clear mathematical basis. The authors relate the average ridge leverage score within a cluster to its effective dimension, providing an explanation for the proposed selection strategy.
The paper includes several useful controlled experiments. The authors compare gradient and BGE representations with matched dimensions and test leverage-based selection against random selection while keeping the deletion count fixed within each cluster. The NEWS-COPY analysis is also useful, showing that Data2Delta preserves more source families than Low NLL, although BGE-leverage performs slightly better on this metric.
Critical weaknesses
My main concern is whether the performance gains justify the complexity of the method. Data2Delta achieves lower PPL than random selection, but performs almost the same as the much simpler Low NLL baseline, with no clear gains on downstream zero-shot tasks. The paper also lacks comparisons with stronger data selection methods, particularly other selection strategies using the same gradient features. Since the method requires additional gradient extraction, clustering, and leverage score computation, I think an end-to-end efficiency comparison is necessary to show whether these small improvements are worth the extra cost.
I am also concerned about how well the method works on natural pretraining data. The main PPL improvements come from experiments with artificially injected perturbations. Without these perturbations, the improvement over random selection becomes much smaller and is not statistically significant. The training budgets are different, so the two settings are not directly comparable, but this still leaves questions about generalization. The NEWS-COPY experiments only measure source coverage, without training a model on the selected data. As a result, it is still unclear whether the method provides meaningful training benefits on naturally duplicated data.
Questions for authors
Data2Delta has a higher selection cost than Low NLL, but their PPL results are almost identical. Is there any end-to-end runtime or compute comparison that includes gradient extraction, clustering, and leverage score computation? Does Data2Delta still offer an efficiency advantage over Low NLL after accounting for these costs?
Could the authors provide comparisons with stronger data selection baselines under the same dataset and training budget? The current comparisons are not sufficient to demonstrate the advantages of the proposed method over existing approaches.
The main PPL improvements are observed in the experiments with artificially injected perturbations, while the improvement over random selection is much smaller and not statistically significant in the original-only experiments. Is the improvement mainly driven by removing synthetic noise? Could the authors provide a comparison under matched training budgets to clarify this?
2 comments
Comment 2/2 — Q2 (stronger baselines) and W2/Q3 (natural data, matched budgets)
Q2: baselines on the same data and budget. All runs use the same residual, retained size (586,921), initialization, 2.5B-token budget and seeds 300–302:
- Same gradient sketches and clusters as Data2Delta: gradient-cosine (SemDeDup rule), exact greedy log-det (the exact greedy optimizer of the objective in Proposition 1), DBP, and DiSF (on 1,024-d sketches, paired with our 1,024-d Data2Delta run).
- Text-embedding diversity selectors on BGE-M3: SemDeDup and DBP.
Holm correction is applied within each family. Results (Data2Delta 13.2234, uniform 13.2389; Δ = Data2Delta − baseline, 10⁻³ PPL, paired 95% interval):
| Baseline (features) | PPL | Δ | Holm p |
|---|---|---|---|
| Gradient-cosine (8,190-d gradients) | 13.2254 | −1.95 [−2.96, −0.94] | 0.029 |
| Greedy log-det (8,190-d gradients) | 13.2257 | −2.29 [−3.16, −1.41] | 0.023 |
| DBP (8,190-d gradients) | 13.2262 | −2.79 [−4.89, −0.69] | 0.029 |
| DiSF (1,024-d gradients; vs 1,024-d Data2Delta) | 13.2326 | −7.45 [−10.01, −4.88] | 0.006 |
| SemDeDup (BGE-M3) | 13.2432 | −19.73 [−20.73, −18.72] | 0.0003 |
| DBP (BGE-M3) | 13.2378 | −14.36 [−15.61, −13.10] | 0.0004 |
Data2Delta is better than every baseline in every seed and at all 30 saved checkpoints. Neither text-embedding selector improves on uniform deletion (13.2389): SemDeDup is 4.32×10⁻³ PPL worse and DBP only level with it (−1.05, n.s.). They spend 79–93% of their deletions on original text and leave most corrupted copies in place. These complement the existing controls: dimension-matched BGE under the same selector (−0.0131 PPL for gradients, Holm p=0.0015) and random membership at fixed cluster budgets (−0.00168, Holm p=0.0023).
Q3: is the main-pool gain driven by the injected copies? Yes, and we now quantify it at matched budgets using the saved checkpoints at 1.0B tokens (the original-only budget):
| At 1.0B tokens | Data2Delta − uniform | Data2Delta − low NLL |
|---|---|---|
| Perturbed pool | −0.0137 [−0.0142, −0.0133] | −0.0002 [−0.0015, +0.0011] |
| Original-only pool | −0.0005 [−0.0025, +0.0016] | −0.0039 [−0.0064, −0.0014] |
(paired 95% intervals over three seeds/replicates). On the perturbed pool the deletions that carry the gain are indeed the corrupted copies, and those copies are the redundancy class the method targets: non-lexical duplicates that standard deduplication leaves behind. MinHash catches only 3,136 of the 29,898 word-shuffled copies, and BGE-based selection removes 1,523 of the survivors, whereas Data2Delta removes 23,000 of them using no labels at all (Table 24). On natural data, the relevant comparison is with loss filtering, which becomes harmful, while Data2Delta does not.
Two new runs make this concrete:
- Original-only pool at the matched 2.5B budget (continuing the saved runs, three replicates; Data2Delta minus control, 10⁻³ PPL): uniform −0.33 (n.s.), BGE leverage −1.36, low NLL −3.34 (lower in all replicates and at all 18 checkpoints; Holm p=0.24 and 0.069). The ordering of the 1B-token results persists at 2.5B, and it also holds when the same masks train Pythia-410M and 1.4B targets (vs low NLL −3.95 and −4.49, all replicates).
- Oracle composition control: deleting Data2Delta's numbers of originals, template and word-shuffled rows at random within each type, using the true labels, gives −14.67×10⁻³ vs uniform, i.e. 95% of Data2Delta's gain. Without any labels, Data2Delta matches this oracle and improves on it through its within-type choice (−0.74×10⁻³ [−1.40, −0.08], p=0.041; 25 of 30 checkpoints).
NEWS-COPY. With 4,988 and 14,211 texts per split it is too small for a meaningful pretraining run; it serves as a labelled test of selection behavior on natural duplicates, where Data2Delta loses 79 source families and low NLL 1,060 (Table 14).
Comment 1/2 — W1/Q1: end-to-end cost and efficiency
Thank you for the constructive review. Several of your questions can be answered from the saved artifacts of the submitted study; we report these first.
Measured selection cost. From the run metadata (A800 GPUs), gradient extraction for all 659,814 rows took 1.99 GPU-h, and clustering plus ridge scoring at most 0.08 GPU-h on one GPU (this bound also includes two other variants). Low-NLL scoring took 0.71 GPU-h, BGE-M3 encoding 0.86 GPU-h, and one 2.5B-token training run 29.9 GPU-h. Data2Delta's full selection therefore costs 6.9% of one training run, and low NLL 2.4%. End to end, including tokenization and I/O (GPU-h, A800):
Tokenizing the retained set for training (8.5 CPU-minutes) is the same for every method.
Equal total compute. Charging each method's selection cost by truncating its own training curve (seeds 300–302):
We confirmed both by actually training further: the saved uniform and low-NLL runs were continued from update 2,384 with their exact optimizer, data-order and RNG state (a restored continuation reproduces an original run to 9.4×10⁻⁶ NLL), by the 165 and 108 extra updates their cheaper selection buys. Uniform still trails Data2Delta by 0.0109–0.0116 PPL in all three seeds; the extra training recovers only 27% of the gap. Low NLL leads by 0.0023–0.0027 PPL (p=0.001, all three seeds; 0.02% of PPL), matching the curve-based estimate.
When the extra cost pays off. Low NLL's efficiency on the perturbed pool relies on its highest-loss rows being corrupted copies. On natural data this is no longer true: on the original-only pool low NLL is worse than uniform selection (+0.0034 in all three replicates; exploratory), because it removes hard, informative chunks (on the main pool it removes 23.0% of the hardest decile of original chunks, versus 6.0% for uniform and 2.4% for Data2Delta), and Data2Delta beats it in all three replicates (−0.0039, raw p=0.021, Holm p=0.064; Table 13). Selection is also computed once per pool and transfers across target sizes: the same original-only masks trained Pythia-410M, 1B and 1.4B (Comment 2/2), amortizing the one-time extraction over a model sweep. The extraction can also be made cheaper without changing the result: the sketch needs only the sketched layer's input and output gradient, so we stopped backpropagation at that layer and formed no other weight gradients. On 2,048 rows the sketches are bit-identical (maximum absolute difference 0) and throughput rises 1.66× (153 vs 92 rows/s per GPU), which puts the same selection at about 1.28 GPU-h (4.3% of a run) and its extra cost over low NLL at 1.9%.