acceptodds
Under review as a conference paper at ICLR 2027

Pruning DNA Sequence

Abstract

Data pruning can substantially reduce training cost by removing examples that contribute little additional learning signal, but it has not been studied for DNA language-model pretraining. We introduce Sequence Pruning (SP), a dynamic pruning framework that removes genomic sequences as they become easy for the model to predict. SP uses per-sequence pretraining accuracy computed directly from the model's forward pass, requiring no separate scoring model or additional pass over the corpus. We evaluate SP across five pretraining settings spanning two corpora, three model families, and both MLM and CLM objectives, with downstream evaluation on five genomic benchmarks. SP reduces pretraining data processing by up to 96% while largely preserving downstream performance. At moderate savings, pruning improves performance by up to 1.2 MCC points over full-data pretraining; at the highest savings, pretraining uses only 4% of the baseline GPU hours with little performance loss. SP also outperforms the dynamic pruning methods InfoBatch and BPAS across most comparisons. Beyond accuracy, our framework supports alternative criteria such as perplexity and EL2N, although accuracy yields the strongest downstream performance at matched savings. These results show that genomic pretraining does not require repeatedly processing all available sequences and that dynamic sequence-level pruning can substantially improve its data and computational efficiency.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
data pruning, genomic language models, DNA pretraining, efficient pretraining, training efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3oJZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Sequence Pruning (SP) for DNA language-model pretraining. At the end of each epoch, SP permanently removes every sequence whose training-time prediction accuracy (MLM or CLM, taken from the normal forward pass) is at or above a threshold τ_low. SP_x% means τ_low = x%. Savings are reported as DSI, the share of the baseline's sequence processing that is skipped (DSI = 50% means half is avoided). SP is tested in five settings: DNABERT-2 on a human and a multi-species corpus, NT-500M and HyenaDNA (1 kb and 16 kb) on the human genome. These runs last 2–99 epochs. Baselines are InfoBatch (loss-based soft pruning), BPAS (batch-level pruning), perplexity and EL2N criteria, and random pruning.

Savings: up to 96% DSI, e.g. Hyena SP_20% reaches GB 76.1 vs 77.0 unpruned, using 2.8 of 76.4 GPU-hours (Table 15). Gains: up to +1.2 MCC over full-data pretraining (NT-500M SP_50%, GUE 68.19 vs 66.98, Table 14). Controls: random pruning at matched DSI is worse (e.g. DB2-MSG GUE 68.1 vs 70.5, Table 2).

Critical strengths

A practical, transparently swept study. Five settings, two objectives, five benchmarks, and full threshold sweeps for SP and the baselines are all reported (App. E, Tables 12–15), including the variants that do not favor SP. A useful observation. In these many-epoch regimes, most sequence-epochs can be skipped at a modest cost. The paper's own multi-seed data put the cost of SP_20% at about 0.4–2 MCC in three of four settings (Tables 17–18). The method needs no reference model and no extra pass over the data.

Critical weaknesses
  1. The method and the outcome are largely known; the analysis that could have contributed new knowledge is missing. Significance: efficiency is not the current bottleneck for DNA foundation models. DNA language models still capture biology poorly: randomly initialized models can match pretrained ones (Vishniakov et al., 2026, cited in the paper), and on Hyena pretraining adds only 3.7 MCC (73.3 vs. 77.0, Table 15). The open problem is how to make these models learn more biology, not how to reach the same representations more cheaply. Data pruning is therefore significant in this field mainly if it contributes to improving model capability, for example by redirecting compute toward sequences that still carry learning signal. The paper's design rules this out. The efficiency gains also fall where compute is not limiting: the evaluated runs cost 47 to 1,137 GPU-hours, while SP needs epoch boundaries and cannot act in single-pass pretraining (Section 5), the regime of the large-scale models the paper cites as its motivation. As framed, SP makes small DNA models cheaper to train without making them better. Missing analysis: the paper's new knowledge should therefore come from what SP reveals about DNA pretraining: what is pruned, and whether it is genuine genomic redundancy (e.g., repeats, low-complexity regions) or sequence the model has simply already learned. This should have been identified as part of the contribution, yet the paper leaves it as an open question. As it stands, the paper shows that a known technique saves compute in a new domain, not what DNA pretraining wastes compute on. Method precedents: the paper cites and compares against general and dynamic pruning methods (EL2N, perplexity pruning, InfoBatch, BPAS). It also cites Marion et al. (2023), who report that removing the easiest sequences can help web-text pretraining. However, the closest precedents for its central idea, a per-example learning-status signal computed during training to decide which examples can stop being trained on, are not cited. Instance-dependent Early Stopping (ICLR 2025) excludes "mastered" instances from backpropagation using one threshold. Example forgetting (Toneva et al., ICLR 2019) uses per-example correctness during training to identify examples that can be omitted. Relative to these, SP's contribution is adapting the idea to self-supervised sequence pretraining (token-level accuracy) with permanent removal at epoch boundaries. That is a modest step, and it should be positioned against them. This reduces Contribution 2 ("We introduce accuracy as a pruning criterion"). Domain precedent: pruning the pretraining corpora of biological foundation models has also been shown at scale, for RNA and protein models (Wu et al., AAAI 2026). The manuscript cites this work and distinguishes itself by pruning dynamically and focusing on DNA, but it weakens the contribution claiming the first to apply. Expected outcome: the stated motivation, that data pruning "has not been studied for DNA language-model pretraining" (abstract), explains why the work is possible, not why it is needed. Given these precedents and the many-epoch regime (28–99 epochs over a fixed corpus), it is expected that pruning learned sequences preserves performance.

  2. The central claim, that the accuracy criterion matters, is untested. SP keeps the baseline epoch count, so its savings come from fewer training steps. No run trains on the full corpus for the same reduced step count. The 96%-DSI Hyena headline is equivalent to stopping the unpruned run after its first epoch: SP_10/20/30% all stop after 1.7k of 50k steps (1 − 1/28 = 96.43%, Table 15). The random control (RP_20%) removes data once before training, whereas SP sees the whole corpus first. The paper's own Table 6 shows that the removal schedule alone moves GUE by up to 2.3 points. So "the gains depend on which sequences are removed" is not yet supported.

Questions for authors

Clarifications and requests that could change my assessment:

  1. Can SP improve model capability? (Weakness 1) For DNA foundation models, the open problem is capability, not training cost. I would consider SP significant for this field only if it improves what a model can ultimately learn. I am not prescribing a protocol, but evidence addressing this concern should meet all of the following: Capability, not cost. The outcome is the final model's capability. No budget limits. Both SP and full-data training are trained until further training no longer improves them, under complete learning-rate schedules. Due to the selection itself. Any gain has to come from which sequences SP removes. Measured on what pretraining learned. The evaluations must be ones where fine-tuning cannot hide differences between pretrained models (e.g., zero-shot or frozen-representation tasks that are biologically meaningful). Parity on the current fine-tuned benchmarks is not enough (73.3 vs. 77.0 without pretraining table 15) Beyond noise and explanation. The result should come with an account of why selection helps
  2. What does pruning cost the model? (Weakness 1) High pretraining accuracy shows that a sequence is predictable by the current model, not that the model has learned the biology it contains. A low-capacity model may predict simple contexts from surface statistics while the knowledge that further exposure would provide is never learned. Pruning would then cap what the model can learn rather than remove redundancy. Several observations make this plausible: the same τ_low yields very different savings across models on the same corpus (Section 3.1), so "easy" is model-dependent; τ_low = 20% is below the ~25% chance level for next-nucleotide prediction, so SP_20% on Hyena prunes nearly every sequence after one epoch; on DB2-MSG it removes sequences near the corpus mean accuracy (0.23–0.27, Table 7); and accuracy was still rising for most sequences when they were pruned (Table 7). The downstream benchmarks cannot settle this, because fine-tuning recovers much of what pretraining misses (73.3 vs. 77.0 without pretraining, Table 15). Please (i) characterize the pruned sequences (repeat and low-complexity content, genomic region, and species for MSG), (ii) compare SP and unpruned models on zero-shot, likelihood-based evaluations that do not involve fine-tuning (e.g., variant effect prediction), and on functional elements split by whether they were pruned, and (iii) report whether pruned sequences were still improving when removed. Without evidence on what pruning costs, I lean toward rejection.
  3. Can you add two controls at the SP_20% and SP_50% budgets, in one DNABERT-2 setting and in Hyena: (a) an unpruned run stopped at the same step count, and (b) dynamic random removal that follows SP's per-epoch active-set size? These are cheap in the smaller settings (DB2-HRG 47 GPU-h, Hyena 76 GPU-h). If SP clearly beats both, the central claim would be supported; the paper's significance would still be limited by Weakness 1.

Suggestions (not part of my decision):

  • "Improves over full-data pretraining" is within the paper's own noise. The +1.2 MCC headline compares two single seed-42 runs and is the best of many variants. Fine-tuning SDs reach 0.6 (Table 17), and three identical one-epoch Hyena checkpoints span 0.83 MCC (75.28–76.11, Table 15). Where multi-seed data exist, SP_20% is below Unpruned in three of four settings (e.g., DB2-HRG GUE 64.34 vs 66.29). If you cannot add pretraining seeds for both arms, please replace "improves" and "can outperform" with a quantified cost for SP_20%.
  • Please state how the recommended τ_low values (50% and 20%) were chosen, and whether the test benchmarks were used to select them.
  • Add BPAS_10/11 to Table 3.
  • Figure 3: the A/B/C block labels sit at different heights, and the "Encoder Models" and "Decoder Models" callouts overlap the block borders. The introduction (about a quarter of the main text, longer than the method section) could be shortened, with Figures 1–2 moved to the method.
Reviewer_nboVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Sequence Pruning (SP), a dynamic data pruning framework for DNA language model pretraining. SP uses per-sequence token prediction accuracy to identify and permanently remove well-learned sequences during training, without additional forward passes or scoring models. The method is evaluated across three model families, two pretraining objectives (MLM and CLM), and five pretraining configurations. Experiments show that SP can reduce pretraining computation while largely preserving downstream performance, outperforming several existing pruning baselines in many settings.

Critical strengths
  1. Practical computational efficiency. SP is lightweight and easy to integrate into existing pretraining pipelines, requiring no additional scoring model or forward passes.
  2. Extensive evaluation across models and tasks. The experiments cover multiple DNA foundation models, different context lengths, and both MLM and CLM objectives, demonstrating the applicability of the framework across different pretraining settings.
  3. Useful empirical observations. The paper investigates the stability of sequence difficulty across training epochs and explores how averaging windows and pruning schedules affect downstream performance.
Critical weaknesses
  1. Poor presentation. The writing lacks clarity, several figures are difficult to interpret (e.g., Figures 2 and 6), and the tables are poorly formatted (e.g., Tables 2 and 3).
  2. Limited methodological novelty. The proposed method essentially removes high-accuracy (easy) sequences during pretraining, following a widely established data pruning strategy. The main contribution appears to be applying this simple criterion to DNA language models, rather than introducing a fundamentally new pruning methodology.
  3. Insufficient DNA-specific motivation. The paper does not clearly explain why existing pruning methods are inadequate for DNA pretraining or identify the unique properties of genomic sequences that motivate the proposed criterion. The method does not explicitly exploit genomic characteristics such as repetitive regions, sequence redundancy, or functional diversity.
  4. Insufficient justification for accuracy as the pruning criterion. The paper demonstrates that per-sequence accuracy is relatively stable across epochs, but stable difficulty rankings do not necessarily imply that accuracy reflects sample importance. Although perplexity and EL2N are compared, the paper provides limited analysis explaining why accuracy should be preferred over alternative criteria such as loss, entropy, or gradient-based importance.
  5. Limited comparison with recent pruning methods. The experimental comparisons mainly focus on a few established baselines, such as InfoBatch and BPAS, without sufficiently covering more recent advances in data pruning. Stronger and more diverse baselines are needed to establish the effectiveness of SP.
  6. Limited evaluation across biological sequences. The proposed pruning strategy is not specific to DNA, yet its effectiveness on other biological sequences, such as RNA and proteins, remains unexplored. Evaluating these settings would help establish the generalizability of the method.
Questions for authors

See Weaknesses.

Reviewer_21bAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The main goal of this paper is to reduce training costs for DNA foundation model pretraining. The core idea is simple: sequences that become easy to predict (measured via accuracy of predicting the sequence) during training are dynamically removed from the training corpus. The authors evaluate their approach across two corpora, three model families, both MLM and CLM objectives, and five genomic benchmarks.

Critical strengths
  • A convincing finding that pruning can substantially reduce pretraining costs for genomic FMs while maintaining comparable downstream performance.

  • At moderate savings, pruning even improves performance by up to 1.2 MCC points over full-data pretraining. This is a very strong result if robust and statistically significant.

  • Extensive benchmark suite across different corpora, models, pretraining objectives, and downstream benchmarks.

  • I particularly like the examples in Figure 1 and learned a lot from them. I did not expect this effect to be so strong, especially for some sequences to have such high accuracy from so early on in training.

  • At the highest savings, the model pretrains using only 4% of the baseline GPU hours while maintaining comparable downstream performance. This is very impressive and might make it feasible to pretrain genomic foundation models on large cohorts of individual genomes and automatically focus on individual genetic variation.

Critical weaknesses
  • My main takeaway is that pruning helps substantially with pretraining genomic foundation models, but the specific approach does not seem to add much beyond what is already established in the literature. In many comparisons, it works about as well as existing baselines. Why is a DNA-specific method required? The approach is simple enough to serve as a reasonable baseline for future work, but I think the claims should be adjusted to make this clearer.

  • Overall, this is a convincing finding, but the analysis feels quite shallow. I would appreciate more results broken down by benchmark tasks: on which tasks does pruning help, where is it particularly effective, and where is it potentially harmful? More interpretation of these results would strengthen the paper.

  • The reported differences in downstream performance often seem small and could potentially be explained by statistical noise. I appreciate the authors' efforts here and understand that proper cross-validation, hyperparameter tuning, and repeated runs with different seeds are hard to do at this scale, but would make the results much more robust and able to distinguish statistical noise from meaningful contributions.

  • The claim that accuracy performs better than perplexity and EL2N is not convincing to me. The differences seem small and could be attributed purely to noise. I also do not find a convincing explanation of why accuracy should outperform these alternatives.

  • The authors propose an additional approach in which a target DSI curve determines how much data is pruned after each epoch. This seems like a negative result to me. The different schedules perform very similarly, potentially within statistical noise. I do not see a statistically meaningful benefit, so I am not convinced why this is included as an additional contribution.

Questions for authors

Main questions

  • How does the proposed method improve upon InfoBatch and BPAS? What is methodologically new, and why is a DNA-specific method required?

  • Are the reported performance improvements statistically significant? Could these differences be explained by run-to-run variability?

  • Is there any statistically meaningful evidence that accuracy outperforms perplexity and EL2N? Why would accuracy be a better pruning criterion?

  • Are there also results with matched compute budgets? Could models become stronger with dynamic pruning given a fixed compute budget?

  • Is there a statistically meaningful benefit from the proposed DSI scheduling strategy?

  • Are sequences that are pruned early in training potentially unlearned or forgotten later during training?

  • Is something missing from the title, "Pruning DNA Sequence"? Should this not be at the very least "Pruning DNA Sequences"?

Additional Comments and questions

  • Figure 3 mostly explains autoregressive and masked language modeling, which is not really the point of this paper. The figure could be simplified.

  • Why not pretrain and evaluate all foundation models on all corpora and benchmarks?

  • DSI is used as an abbreviation throughout the manuscript but never spelled out.

  • Why introduce τ_high if it is always fixed at 100% for the proposed method? This seems unnecessary.

  • The DSI scheduling approach is poorly explained. It took me some time to understand how exactly the empirical curves determine which samples are pruned and when.

  • The placement of Related Work after the experiments is unusual.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.