acceptodds
Under review as a conference paper at ICLR 2027

Reconciling Coarse Tokenization with Single-Nucleotide Resolution in Genomic Language Modeling

Abstract

Genomic language models face a fundamental trade-off: coarse tokenization enables efficient long-context modeling, yet standard token-level objectives obscure single-nucleotide variation. We introduce Factorized Nucleotide Supervision (FNS), which derives position-specific nucleotide likelihoods from a distribution over -mers through exact marginalization. FNS retains the computational efficiency of non-overlapping -mers while providing dense nucleotide-level supervision and directly supporting variant scoring. We instantiate FNS in GENERator-v2, a family of autoregressive models for eukaryotic and prokaryotic genomes. Across training-free and supervised benchmarks, GENERator-v2 consistently improves over its predecessor, matches or exceeds substantially larger baselines, and offers up to faster inference. Beyond these model-level gains, our results challenge two common assumptions in GFM evaluation: supporting a longer context does not guarantee that a model benefits from the added sequence, and single-nucleotide tokenization is not required to learn nucleotide-scale biological structure.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
genomic foundation model, DNA tokenization, variant effect prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_y15KOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Factorised Nucleotide Supervision. It is a simple and straightforward idea to supervise causal genomic language models. It derives the marginal likelihood of each individual nucleotide from the output k-mer logits. The authors provide empirical results of GeneratorV2 on Sequence Recovery, Variant Effect Prediction, Embedding Analysis and Nucleotide Transformer Benchmark.

Critical strengths

Factorised Nucleotide Supervision provides nucleotide-level supervision on genomic language models with non-overlapping k-mers tokenisation. It is simple and effective.

Critical weaknesses
  1. The authors do not provide an FNS-only ablation. Several training choices described in the appendices lack controlled ablations as well.
  2. Section 5.4’s title is confusing. It is "SUPERVISED FINE-TUNING AND TRAINING-FREE EVALUATION" but Section 5.4 only presents supervised fine-tuning results.
Questions for authors
  1. Could you provide controlled ablations for FNS and the training choices you described in the appendices? What exactly changed in pretraining between GeneratorV1 and GeneratorV2?
  2. Could you report inference and training FLOPs under identical nucleotide input length of Evo1, Evo2 and GeneratorV2.
Reviewer_a8M7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a method called Factorized Nucleotide Supervision (FNS), which combines the computational efficiency of k-mer tokenization with nucleotide level supervision. The paper incorporates this method into GENERator-v2, a family of transformer based genomic language models using non-overlapping 6-mer tokenization. They evaluate the models on sequence recovery, variant-effect prediction, supervised genomic classification and show gains over other DNA language models.

Critical strengths

The proposed FNS method is a simple way of introducing nucleotide level supervision whilst retaining the benefits of coarser tokenization strategies, without requiring modifications to the architecture.

Critical weaknesses

I think that there are key critical weaknesses in the results and evaluation of the method that would need to be addressed:

  1. Limited isolation of the proposed method's contribution

Although FNS is the central methodological contribution, some of the main findings seem to reflect properties of the existing GENERator architecture rather than benefits introduced by FNS. For example, the inference speed advantages over Evo2, the long-context and representation findings. There is also no controlled comparison between a model that is trained with and without FNS. Whilst there are comparisons between GENERator and GENERator-v2, other aspects of training differ, such as separator-loss masking. I consider this a significant weakness because it is difficult to judge how much of the improvement actually comes from FNS, making it hard to assess the effectiveness and significance of the paper's main methodological contribution.

  1. Weak evaluation and results

The performance differences between GENERator and GENERator-v2 are generally small on ClinVar and the Nucleotide Transformer Benchmark, so I'm not convinced that FNS offers a meaningful practical improvement. For example, only one variant effect prediction benchmark is evaluated (ClinVar), where the difference between GENERator and GENERator-v2 is 0.921 vs 0.924. Evaluations on other variant effect predictions would be necessary to make the claim that FNS helps with this. On the sequence-recovery tasks, I am concerned about potential data leakage, as it is not clearly established if the sequences that are used are excluded from pretraining. Despite long-context modeling being a central motivation, the paper does not evaluate using long-range genomic prediction benchmarks[1, 2]. Furthermore, the observation that maximum supported context length does not imply effective context utilisation has already been investigated in prior work, eg [1, 2]. Consequently, the broader claims about long-context capabilities are not well supported.

[1] Cheng et al. (2025). DNALONGBENCH: A benchmark suite for long-range DNA prediction tasks. Nature Communications, 16, 10108.

[2]Trop et al The genomics long-range benchmark: Advancing DNA language models, 2025

Questions for authors

Can the authors provide a controlled comparison in which FNS is the only change?

How do the authors ensure that evaluation sequences do not overlap with pretraining data for the sequence recovery task?

Reviewer_USvROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Factorized Nucleotide Supervision (FNS), a training objective that derives nucleotide-level likelihoods from non-overlapping k-mer predictions through exact marginalization. The approach aims to combine efficient long-context genomic modeling with single-nucleotide supervision. The authors introduce GENERator-v2, a family of 1B and 3B parameter models trained using FNS and annotation-aware genomic pretraining. Evaluations cover sequence recovery, variant-effect prediction, representation analysis, and supervised genomic classification, with improvements over the preceding GENERator models and competitive results against larger baselines.

Critical strengths
  1. FNS provides a simple and computationally efficient way to introduce nucleotide-level supervision while preserving non-overlapping k-mer tokenization. The formulation requires no additional prediction heads or architectural modifications.

  2. The reported sequence-recovery and variant-effect prediction results are promising, and the inference-efficiency improvements demonstrate the practical benefits of coarse tokenization.

Critical weaknesses
  1. The paper does not provide a controlled ablation comparing identical models trained with and without FNS under matched data, compute, and optimization settings. Although the original GENERator serves as a baseline without FNS, GENERator-v2 also introduces changes in training and separator-token supervision. Consequently, the observed improvements cannot be confidently attributed to FNS alone. Furthermore, the motivation assumes that standard k-mer cross-entropy is insufficient for learning nucleotide-level similarities, but this assumption is not directly tested. A Transformer may learn relationships between k-mers through its shared representations without explicit nucleotide-level supervision.

  2. The paper introduces four foundation models but provides insufficient information about their architectures and pretraining configurations. Appendix F specifies parameter counts and data sources, but important details such as layer configurations, optimizer schedules, total training tokens, hardware requirements, and training curves are missing. The paper also lacks a comparison against alternative efficient tokenization approaches, particularly BPE, which is used in genomic language models such as GENA-LM and DNABERT-2.

  3. Section 5.1 claims that Evo2's sequence-recovery performance largely plateaus beyond approximately 1,024 bp, whereas GENERator continues to benefit from longer contexts. Figure S1 does not convincingly establish this distinction. Evo2-7B remains highly competitive and frequently achieves the best recovery accuracy across the reported settings. The comparison also involves different tokenization lengths and operating regimes. A more systematic analysis at matched nucleotide spans is necessary to support conclusions about relative context utilization.

Questions for authors
  1. Could the authors provide a controlled FNS ablation using the same architecture, training corpus, token budget, and optimization settings, changing only the supervision objective?

  2. Why is explicit nucleotide-level supervision necessary if the Transformer can potentially learn nucleotide similarity and Hamming-distance relationships between k-mers from standard cross-entropy training? Is there evidence that standard training fails to learn these relationships, or that FNS improves sample efficiency?

  3. Could the authors compare non-overlapping fixed-length k-mers against BPE tokenization, as used in GENA-LM and DNABERT-2? A comparison with an autoregressive BPE model under matched training and inference budgets would help establish the advantages of the proposed tokenization strategy.

  4. Could the authors clarify the complete training objective in the main text? Appendix A.3 provides the aggregation of nucleotide and special-token losses, but it would be useful to state explicitly whether standard joint k-mer cross-entropy is used alongside FNS and whether combining these objectives has been evaluated.

  5. Could the authors provide complete pretraining configurations, including architectural hyperparameters, dataset sizes, total training tokens, learning-rate schedules, training curves, and GPU compute?

  6. Could the authors substantiate the claimed Evo2 performance plateau using numerical context-length scaling results, preferably at matched nucleotide spans with uncertainty estimates? The current Figure S1 does not clearly support the broad conclusion drawn in Section 5.1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.