- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper audits three design choices in genomic language models: subword tokenization, learned input embeddings, and likelihood-based pretraining. It distinguishes mutation-induced boundary movement from biological landmark placement, separates embedding correspondence from architectural reverse-complement equivariance, and attributes loss improvement to genomic annotation classes. It reports local BPE boundary changes and 63–83% of loss improvement accruing to repeats. A five-dose, two-seed repeat-weight intervention evaluates a frozen, mean-pooled splice readout with composition, random-initialization, and batch/update controls. The authors report better splice scores with less repeat-directed loss. The contribution is a set of empirical diagnostics and a controlled test of signal accessibility under a specified readout.
Critical strengths
-
The authors distinguish changes in token identity from movement of segmentation boundaries. Under the published BPE merge rules, they report a mean boundary influence radius of roughly 3 bp and different boundary preferences around splice sites and stop codons (Sections 3.1 and 4). This shows that local stability under mutation and boundary placement capture different properties of a tokenizer. Matching the reference for boundary density also makes the placement results easier to interpret.
-
The authors report both total and per-base loss improvement for each annotation class, showing that the concentration of improvement on repeats is not explained solely by their abundance (Section 6, Table 3). The intervention fixes the backbone and readout and includes batch/update controls (Section 8). This design helps assess what a downstream linear probe can recover from frozen features after changing the pretraining objective.
Critical weaknesses
-
The protocol specifies GUE reconstructed-splice, whose standard version has three classes ([1], Table 1, Appendix B.2). However,
experiments/w2_gue_probe.py:65–70counts only cases where both true and predicted labels are 0 or 1. On three-class inputs, this omits label-2 cases, including errors, depending on the predictions; sharing the function therefore does not guarantee valid comparisons. Please clarify the actual labels and evaluator used for the reported results, including any fixed binary recoding or different implementation. If this function was used on three-class inputs, the principal dose and reference/control comparisons need to be corrected using multiclass MCC. -
The paper should position its contribution more precisely against earlier studies. GPN [2], which it already cites, and Mclaughlin and Lim [3] study repeat weighting, while Lindsey et al. [4] report tokenizer effects on splice tasks. This paper combines classwise loss accounting with a fixed probe on frozen features, a frozen randomly initialized reference, and batch/update controls. These are meaningful differences from variant-effect and fine-tuning evaluations. The diagnostic findings are useful, but the discussion should explain what this combination tells us about feature accessibility beyond the earlier results.
[1] Zhou, Z., Ji, Y., Li, W., Dutta, P., Davuluri, R. V., and Liu, H. (2024). DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genomes. International Conference on Learning Representations (ICLR). arXiv:2306.15006v2. https://arxiv.org/abs/2306.15006v2.
[2] Benegas, G., Batra, S. S., and Song, Y. S. (2023). DNA language models are powerful predictors of genome-wide variant effects. Proceedings of the National Academy of Sciences, 120(44), e2311219120. https://doi.org/10.1073/pnas.2311219120.
[3] Mclaughlin, S. M., and Lim, D. A. (2026). Probing genomic language models: Nucleotide Generative Pretrained Transformer and the role of pretraining in learned representations. Briefings in Bioinformatics, 27(1), bbag011. https://doi.org/10.1093/bib/bbag011.
[4] Lindsey, L. M., Pershing, N. L., Habib, A., Dufault-Thompson, K., Stephens, W. Z., Blaschke, A. J., Jiang, X., and Sundar, H. (2025). The impact of tokenizer selection in genomic language models. Bioinformatics, 41(9), btaf456. https://doi.org/10.1093/bioinformatics/btaf456.
Questions for authors
-
Which label definition and MCC implementation were used for the main splice results? If the supplied binary function was used for the three-class task, how do the principal dose and control results change under an appropriate multiclass metric?
-
What additional scientific inference does the fixed-readout intervention provide beyond prior repeat-weighting studies?