- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Sequence Pruning (SP) for DNA language-model pretraining. At the end of each epoch, SP permanently removes every sequence whose training-time prediction accuracy (MLM or CLM, taken from the normal forward pass) is at or above a threshold τ_low. SP_x% means τ_low = x%. Savings are reported as DSI, the share of the baseline's sequence processing that is skipped (DSI = 50% means half is avoided). SP is tested in five settings: DNABERT-2 on a human and a multi-species corpus, NT-500M and HyenaDNA (1 kb and 16 kb) on the human genome. These runs last 2–99 epochs. Baselines are InfoBatch (loss-based soft pruning), BPAS (batch-level pruning), perplexity and EL2N criteria, and random pruning.
Savings: up to 96% DSI, e.g. Hyena SP_20% reaches GB 76.1 vs 77.0 unpruned, using 2.8 of 76.4 GPU-hours (Table 15). Gains: up to +1.2 MCC over full-data pretraining (NT-500M SP_50%, GUE 68.19 vs 66.98, Table 14). Controls: random pruning at matched DSI is worse (e.g. DB2-MSG GUE 68.1 vs 70.5, Table 2).
Critical strengths
A practical, transparently swept study. Five settings, two objectives, five benchmarks, and full threshold sweeps for SP and the baselines are all reported (App. E, Tables 12–15), including the variants that do not favor SP. A useful observation. In these many-epoch regimes, most sequence-epochs can be skipped at a modest cost. The paper's own multi-seed data put the cost of SP_20% at about 0.4–2 MCC in three of four settings (Tables 17–18). The method needs no reference model and no extra pass over the data.
Critical weaknesses
-
The method and the outcome are largely known; the analysis that could have contributed new knowledge is missing. Significance: efficiency is not the current bottleneck for DNA foundation models. DNA language models still capture biology poorly: randomly initialized models can match pretrained ones (Vishniakov et al., 2026, cited in the paper), and on Hyena pretraining adds only 3.7 MCC (73.3 vs. 77.0, Table 15). The open problem is how to make these models learn more biology, not how to reach the same representations more cheaply. Data pruning is therefore significant in this field mainly if it contributes to improving model capability, for example by redirecting compute toward sequences that still carry learning signal. The paper's design rules this out. The efficiency gains also fall where compute is not limiting: the evaluated runs cost 47 to 1,137 GPU-hours, while SP needs epoch boundaries and cannot act in single-pass pretraining (Section 5), the regime of the large-scale models the paper cites as its motivation. As framed, SP makes small DNA models cheaper to train without making them better. Missing analysis: the paper's new knowledge should therefore come from what SP reveals about DNA pretraining: what is pruned, and whether it is genuine genomic redundancy (e.g., repeats, low-complexity regions) or sequence the model has simply already learned. This should have been identified as part of the contribution, yet the paper leaves it as an open question. As it stands, the paper shows that a known technique saves compute in a new domain, not what DNA pretraining wastes compute on. Method precedents: the paper cites and compares against general and dynamic pruning methods (EL2N, perplexity pruning, InfoBatch, BPAS). It also cites Marion et al. (2023), who report that removing the easiest sequences can help web-text pretraining. However, the closest precedents for its central idea, a per-example learning-status signal computed during training to decide which examples can stop being trained on, are not cited. Instance-dependent Early Stopping (ICLR 2025) excludes "mastered" instances from backpropagation using one threshold. Example forgetting (Toneva et al., ICLR 2019) uses per-example correctness during training to identify examples that can be omitted. Relative to these, SP's contribution is adapting the idea to self-supervised sequence pretraining (token-level accuracy) with permanent removal at epoch boundaries. That is a modest step, and it should be positioned against them. This reduces Contribution 2 ("We introduce accuracy as a pruning criterion"). Domain precedent: pruning the pretraining corpora of biological foundation models has also been shown at scale, for RNA and protein models (Wu et al., AAAI 2026). The manuscript cites this work and distinguishes itself by pruning dynamically and focusing on DNA, but it weakens the contribution claiming the first to apply. Expected outcome: the stated motivation, that data pruning "has not been studied for DNA language-model pretraining" (abstract), explains why the work is possible, not why it is needed. Given these precedents and the many-epoch regime (28–99 epochs over a fixed corpus), it is expected that pruning learned sequences preserves performance.
-
The central claim, that the accuracy criterion matters, is untested. SP keeps the baseline epoch count, so its savings come from fewer training steps. No run trains on the full corpus for the same reduced step count. The 96%-DSI Hyena headline is equivalent to stopping the unpruned run after its first epoch: SP_10/20/30% all stop after 1.7k of 50k steps (1 − 1/28 = 96.43%, Table 15). The random control (RP_20%) removes data once before training, whereas SP sees the whole corpus first. The paper's own Table 6 shows that the removal schedule alone moves GUE by up to 2.3 points. So "the gains depend on which sequences are removed" is not yet supported.
Questions for authors
Clarifications and requests that could change my assessment:
- Can SP improve model capability? (Weakness 1) For DNA foundation models, the open problem is capability, not training cost. I would consider SP significant for this field only if it improves what a model can ultimately learn. I am not prescribing a protocol, but evidence addressing this concern should meet all of the following: Capability, not cost. The outcome is the final model's capability. No budget limits. Both SP and full-data training are trained until further training no longer improves them, under complete learning-rate schedules. Due to the selection itself. Any gain has to come from which sequences SP removes. Measured on what pretraining learned. The evaluations must be ones where fine-tuning cannot hide differences between pretrained models (e.g., zero-shot or frozen-representation tasks that are biologically meaningful). Parity on the current fine-tuned benchmarks is not enough (73.3 vs. 77.0 without pretraining table 15) Beyond noise and explanation. The result should come with an account of why selection helps
- What does pruning cost the model? (Weakness 1) High pretraining accuracy shows that a sequence is predictable by the current model, not that the model has learned the biology it contains. A low-capacity model may predict simple contexts from surface statistics while the knowledge that further exposure would provide is never learned. Pruning would then cap what the model can learn rather than remove redundancy. Several observations make this plausible: the same τ_low yields very different savings across models on the same corpus (Section 3.1), so "easy" is model-dependent; τ_low = 20% is below the ~25% chance level for next-nucleotide prediction, so SP_20% on Hyena prunes nearly every sequence after one epoch; on DB2-MSG it removes sequences near the corpus mean accuracy (0.23–0.27, Table 7); and accuracy was still rising for most sequences when they were pruned (Table 7). The downstream benchmarks cannot settle this, because fine-tuning recovers much of what pretraining misses (73.3 vs. 77.0 without pretraining, Table 15). Please (i) characterize the pruned sequences (repeat and low-complexity content, genomic region, and species for MSG), (ii) compare SP and unpruned models on zero-shot, likelihood-based evaluations that do not involve fine-tuning (e.g., variant effect prediction), and on functional elements split by whether they were pruned, and (iii) report whether pruned sequences were still improving when removed. Without evidence on what pruning costs, I lean toward rejection.
- Can you add two controls at the SP_20% and SP_50% budgets, in one DNABERT-2 setting and in Hyena: (a) an unpruned run stopped at the same step count, and (b) dynamic random removal that follows SP's per-epoch active-set size? These are cheap in the smaller settings (DB2-HRG 47 GPU-h, Hyena 76 GPU-h). If SP clearly beats both, the central claim would be supported; the paper's significance would still be limited by Weakness 1.
Suggestions (not part of my decision):
- "Improves over full-data pretraining" is within the paper's own noise. The +1.2 MCC headline compares two single seed-42 runs and is the best of many variants. Fine-tuning SDs reach 0.6 (Table 17), and three identical one-epoch Hyena checkpoints span 0.83 MCC (75.28–76.11, Table 15). Where multi-seed data exist, SP_20% is below Unpruned in three of four settings (e.g., DB2-HRG GUE 64.34 vs 66.29). If you cannot add pretraining seeds for both arms, please replace "improves" and "can outperform" with a quantified cost for SP_20%.
- Please state how the recommended τ_low values (50% and 20%) were chosen, and whether the test benchmarks were used to select them.
- Add BPAS_10/11 to Table 3.
- Figure 3: the A/B/C block labels sit at different heights, and the "Encoder Models" and "Decoder Models" callouts overlap the block borders. The introduction (about a quarter of the main text, longer than the method section) could be shortened, with Figures 1–2 moved to the method.