acceptodds
Under review as a conference paper at ICLR 2027

Do Genomic Language Models Inherit the Right Assumptions?

Abstract

Genomic language models largely inherit three choices from natural-language models: subword tokenization, learned input embeddings, and likelihood-based pretraining. We test whether these choices align with DNA using controlled comparisons that hold architecture, data, and nucleotide budget fixed, together with model-free diagnostics. Under DNABERT-2's published merge rules, BPE is mutation-stable: a substitution moves boundaries by a mean radius of bp, less than one mean token length. The larger mismatch is placement. At coding-sequence edges, boundaries occur at a length-matched null; the depletion replicates across eleven windows spanning six chromosomes and persists against a compositional control with the opposite predicted sign. Landmark stratification shows depletion at splice sites but enrichment at stop codons. Reverse-complement symmetry separates into segmentation, embedding, and contextual-representation levels. Five released embedding tables give ; only two have enough reverse-complement pairs for formal inference, and an architecturally equivariant model still has an unconstrained embedding table. Across four tokenizer arms, repeats receive – of total loss improvement versus – for the CDS-assigned class. In a controlled dose intervention, increasing repeat loss weight lowers retained splice-site signal in mean-pooled representations from to relative to a randomly initialised backbone across two seeds (blocked slope , CI ). Placebo arms make a batch-only explanation less consistent with the observed pattern. These results motivate evaluating inherited NLP design choices against genomic invariances and functional structure rather than assuming direct transfer.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
genomic language models, DNA foundation models, tokenization, reverse-complement symmetry, pretraining objectives, diagnostics, controlled intervention, representation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8uqGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper audits three design choices in genomic language models: subword tokenization, learned input embeddings, and likelihood-based pretraining. It distinguishes mutation-induced boundary movement from biological landmark placement, separates embedding correspondence from architectural reverse-complement equivariance, and attributes loss improvement to genomic annotation classes. It reports local BPE boundary changes and 63–83% of loss improvement accruing to repeats. A five-dose, two-seed repeat-weight intervention evaluates a frozen, mean-pooled splice readout with composition, random-initialization, and batch/update controls. The authors report better splice scores with less repeat-directed loss. The contribution is a set of empirical diagnostics and a controlled test of signal accessibility under a specified readout.

Critical strengths
  • The authors distinguish changes in token identity from movement of segmentation boundaries. Under the published BPE merge rules, they report a mean boundary influence radius of roughly 3 bp and different boundary preferences around splice sites and stop codons (Sections 3.1 and 4). This shows that local stability under mutation and boundary placement capture different properties of a tokenizer. Matching the reference for boundary density also makes the placement results easier to interpret.

  • The authors report both total and per-base loss improvement for each annotation class, showing that the concentration of improvement on repeats is not explained solely by their abundance (Section 6, Table 3). The intervention fixes the backbone and readout and includes batch/update controls (Section 8). This design helps assess what a downstream linear probe can recover from frozen features after changing the pretraining objective.

Critical weaknesses
  • The protocol specifies GUE reconstructed-splice, whose standard version has three classes ([1], Table 1, Appendix B.2). However, experiments/w2_gue_probe.py:65–70 counts only cases where both true and predicted labels are 0 or 1. On three-class inputs, this omits label-2 cases, including errors, depending on the predictions; sharing the function therefore does not guarantee valid comparisons. Please clarify the actual labels and evaluator used for the reported results, including any fixed binary recoding or different implementation. If this function was used on three-class inputs, the principal dose and reference/control comparisons need to be corrected using multiclass MCC.

  • The paper should position its contribution more precisely against earlier studies. GPN [2], which it already cites, and Mclaughlin and Lim [3] study repeat weighting, while Lindsey et al. [4] report tokenizer effects on splice tasks. This paper combines classwise loss accounting with a fixed probe on frozen features, a frozen randomly initialized reference, and batch/update controls. These are meaningful differences from variant-effect and fine-tuning evaluations. The diagnostic findings are useful, but the discussion should explain what this combination tells us about feature accessibility beyond the earlier results.

[1] Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R. V., and Liu, H. (2024). DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genomes. International Conference on Learning Representations (ICLR). arXiv:2306.15006v2. https://arxiv.org/abs/2306.15006v2.

[2] Benegas, G., Batra, S. S., and Song, Y. S. (2023). DNA language models are powerful predictors of genome-wide variant effects. Proceedings of the National Academy of Sciences, 120(44), e2311219120. https://doi.org/10.1073/pnas.2311219120.

[3] Mclaughlin, S. M., and Lim, D. A. (2026). Probing genomic language models: Nucleotide Generative Pretrained Transformer and the role of pretraining in learned representations. Briefings in Bioinformatics, 27(1), bbag011. https://doi.org/10.1093/bib/bbag011.

[4] Lindsey, L. M., Pershing, N. L., Habib, A., Dufault-Thompson, K., Stephens, W. Z., Blaschke, A. J., Jiang, X., and Sundar, H. (2025). The impact of tokenizer selection in genomic language models. Bioinformatics, 41(9), btaf456. https://doi.org/10.1093/bioinformatics/btaf456.

Questions for authors
  • Which label definition and MCC implementation were used for the main splice results? If the supplied binary function was used for the three-class task, how do the principal dose and control results change under an appropriate multiclass metric?

  • What additional scientific inference does the fixed-readout intervention provide beyond prior repeat-weighting studies?

Reviewer_upnzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper revisits the genomic language models through the lens of subword tokenization, embedding, and likelihood-based retraining. It shows coding-boundary statistics, embedding space RC-equivalency, and also find that the repeat-sequence losses are minimized the most.

Critical strengths
  1. The motivation for this paper is benign, as most papers in genomic language modeling are trying to figure out the right inductive biases for their design.
  2. This paper shows some basic statistics for different tokenizers.
Critical weaknesses
  1. Proposed measures are not linked to model behavior
  • The paper measures properties of the tokenizer (e.g., S, R, A_tao) and of the input embedding table (V(E)). It never shows how these properties connect to the downstream performance, making the effectiveness of these indicators unanswered. While these metrics are interesting statistics for the design purpose, whether a design is "right" or not should be judged by its effect on downstream tasks. For example, the paper's own tokenizer results show that with various A_tao and V_seg their ClinVar AUROCs are indistinguishable low.
  1. The model might be too small and weak
  • The pretrained representation scores below a randomly-initialized backbone (0.433 vs. 0.701). It is therefore unclear whether the conclusions transfer to larger GLMs, or to settings where pretrained models beat simple baselines.
  1. The takeaways from this paper is relatively limited
  • Many of the high-level observations are already known, for example RC-equivariant architectures (e.g., Caduceus), alternative tokenizers, and loss reweighting for repetitive positions in GPN-MSA. The paper should mention those works and position its contribution against these prior works.
  1. The experimental scope is limited:
  • This study involves only one trained backbone, trained on a single human dataset, evaluated on two downstream benchmarks. Also the performance is also close to chance, making the empirical evidence relatively weak.
  1. Others: The tokenizer measures are largely true by construction
  • For fixed-grid and single-base tokenizers, S = 1, R = 0 and A_τ = 1 should hold, by definition. The stability result concludes that BPE is stable under mutation, which is expected and is a non-finding.
  • The premise that token boundaries should align with annotation landmarks is never tested. A model can still represent a junction while the token cannot.
  1. Others: Reverse-complement analysis
  • Similar to the above, comparing rows of the embedding table ignores that the model can implement RC-consistent processing in later layers.
Questions for authors

Please answer the weaknesses above.

Reviewer_hXKSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper tests whether default NLP assumptions are appropriate for DNA language models, analyzing subword tokenization, learned input embeddings, and likelihood-based pretraining. The authors evaluate whether these choices align with DNA using controlled comparisons.

Critical strengths

The paper isolates the effects of individual design choices (tokenization, input embeddings, and training objective) on DNA language models by holding architecture, data, and nucleotide budget fixed, and adds composition and random-initialization baselines as controls. This controlled evaluation framework is a useful reference for future assessment of genomic language model design.

Critical weaknesses
  • Missing related work: Vishniakov et al. (ICLR 2026) and BEND (ICLR 2024) are neither cited nor discussed, although their findings overlap substantially with this paper’s (e.g., strong random-initialization baselines and the failure of zero-shot variant scoring).
  • The contribution is closer to a set of diagnostic tools.
  • The paper evaluates only the default NLP choices it critiques and does not compare against existing genomics-specific alternatives: learned tokenizers, RC-tied embeddings, or repeat-reweighted objectives.
  • Repeat down-weighting is already standard practice: it was introduced in GPN (Benegas et al., 2023, which the paper cites) and adopted in GPN-MSA and Evo 2 (weight 0.1). Yet Section8 presents a repeat-weight intervention without acknowledging this.
Questions for authors
  • Could the authors add comparisons with existing genomics-specific alternatives (e.g., learned tokenizers, RC-tied embeddings, or repeat-reweighted objectives)?
  • Repeat down-weighting is already standard practice: it was introduced in GPN (Benegas et al., 2023, which the paper cites) and adopted in GPN-MSA and Evo 2 (weight 0.1). How does the Section 8 intervention relate to this prior practice, and what does it add beyond it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.