acceptodds
Under review as a conference paper at ICLR 2027

VANDAM: Viewing a nucleotide sequence with DNA molecular priors

Abstract

Contemporary Genomic Foundation Models (GFMs) rely on a DNA-as-a-string paradigm that employs masked token prediction objectives for pretraining. However, this abstraction does not explicitly models the biochemical, structural, and physical properties essential to biological function. Many molecular properties can be estimated from sequence using established biophysical models, so their utility lies not in providing an independent modality, but in introducing priors that training objectives can explicitly exploit. We introduce VANDAM, a framework that extends the training of GFMs with DNA molecular priors. In self-supervised training, VANDAM predicts regional molecular properties from pooled representations. When functional labels are available and can reward retaining molecular priors, local features are additionally injected at the input. VANDAM consistently improves downstream performance across four architecture families and nine held-out genomic tasks by complementing token-based objectives. Probing experiments further demonstrate that the use of molecular priors generalizes to other unseen molecular properties. Code and trained model checkpoints will be made publicly available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
genomic foundation models, DNA sequence modeling, representation learning, self-supervised learning, biophysical supervision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ZYdzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

VANDAM incorporates sequence-derived molecular priors into genomic foundation models through local feature injection (LFI) and regional property prediction (RPP). RPP complements the native token objective during continual pre-training, while LFI is additionally used under functional supervision. Across four backbones and nine downstream tasks, the approach improves most matched comparisons. Component ablations, non-biological controls, and probing of held-out molecular descriptors provide additional evidence for its effectiveness.

Critical strengths
  1. The evaluation is reasonably thorough. Matched continuation baselines across four backbones help distinguish the proposed objectives from simply training longer, while component and non-biological controls provide useful evidence beyond downstream gains alone.

  2. The distinction between supplying molecular features and explicitly supervising their prediction is informative. The PT/MT ablations suggest that their utility depends on the training setting, and the held-out descriptor probes provide additional evidence that RPP changes the learned representations.

Critical weaknesses
  1. The methodological novelty is incremental. UTR-LM already uses sequence-derived biophysical auxiliary targets, and MIXBend incorporates physicochemical features into DNA representation learning. The broader framing should more clearly distinguish these precedents from the specific contribution of regional supervision and its combination with LFI.

  2. The matched baselines establish improvements over the selected backbones, but leave the broader competitive position unclear. Comparison with suitable functionally supervised models, such as Enformer [1], SPACE [2], Borzoi [3], or AlphaGenome [4], on compatible tasks would strengthen the evaluation, with differences in supervision, context length, and data overlap made explicit.

  3. The explanation for the different roles of LFI and RPP in PT versus MT remains tentative. The central ablations use NTv3-8M, and masking removes approximately 47% of local feature vectors under MLM. This may also contribute to LFI's limited PT benefit; the alternative masking treatments in Appendix O are evaluated only in MT.

[1] Avsec et al. Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methods, 2021.

[2] Yang et al. SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model. ICML, 2025.

[3] Linder et al. Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation. Nature Genetics, 2025.

[4] Avsec et al. Advancing regulatory variant effect prediction with AlphaGenome. Nature, 2026.

Questions for authors

Does existing evidence distinguish the effect of functional supervision from feature sparsity caused by masking when explaining LFI's different benefits in PT and MT?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZYdz, part 1

Response to Question 1: Functional supervision versus masking-induced feature sparsity

We agree that the NTv3-8M ablation alone cannot distinguish the effect of functional supervision from masking-induced feature sparsity. To examine whether the same pattern occurs without this sparsity, we repeated the component and token-objective ablations with HyenaDNA-small. HyenaDNA uses causal next-token prediction, and its local descriptors are computed from right-aligned pentamers containing only visible bases. It therefore has no masking-induced feature removal, unlike the 47% zeroing of feature vectors under NTv3-8M’s MLM objective.

Component and token-objective ablations across PT and MT

ConfigurationNTv3-8M — PTNTv3-8M — MTHyenaDNA-small — PTHyenaDNA-small — MT
Original checkpoint59.559.559.359.3
Baseline-PT / Baseline-MT60.563.062.060.7
LFI only60.564.262.661.8
RPP only63.863.264.261.9
LFI + RPP63.465.164.164.4
RPP without token loss, masking retained60.260.2N/AN/A
RPP without token loss, no masking59.760.158.858.4

The HyenaDNA ablation reproduces the configuration ranking observed for NTv3-8M: RPP alone achieves the highest mean AUROC in PT, whereas LFI + RPP achieves the highest mean AUROC in MT. Crucially, this pattern persists in HyenaDNA without masking-induced feature sparsity. These results therefore support our interpretation that the relative benefits of LFI and RPP depend on the training regime, and that the pattern observed in NTv3-8M is not solely attributable to feature removal under MLM. Removing the native token objective also reduces performance in both regimes, supporting molecular supervision as a complement to token prediction.

Response to Weakness 1: Methodological novelty and prior work

We thank the reviewer for encouraging a sharper discussion of the methodological contribution. We agree that sequence-derived biophysical supervision and physicochemical feature integration have important precedents, and that our framing should distinguish these established ideas from the specific contributions of VANDAM.

Our contribution builds on these ideas. UTR-LM focuses on RNA, incorporating secondary-structure and sequence-level minimum-free-energy prediction for mRNA 5′ untranslated regions. MIXBend combines DNA sequence and physicochemical representations for DNA bendability prediction. VANDAM introduces tile-level prediction of multiple DNA molecular properties during continual training of existing genomic foundation models and evaluates transfer across diverse downstream tasks. We also investigate how this regional supervision combines with local feature injection under self-supervised and functionally supervised training regimes.

To make these distinctions explicit, we will add the following paragraph to Related Work:

“Biophysical supervision and physicochemical feature integration have established precedents in sequence representation learning. UTR-LM uses secondary-structure and minimum-free-energy prediction as auxiliary objectives for learning representations of mRNA 5′ untranslated regions, while MIXBend combines DNA sequence and physicochemical representations for DNA bendability prediction. Building on these ideas, VANDAM introduces regional property prediction (RPP), which provides tile-level supervision of multiple DNA molecular properties during continual training of genomic foundation models. We investigate its combination with local feature injection (LFI) across self-supervised and functionally supervised training regimes, with transfer evaluated across diverse downstream tasks. The specific contribution is this regional supervision strategy and the study of its interaction with LFI, rather than the general use of biophysical priors.”

AuthorsOpenReview ↗

Response to Reviewer ZYdz, part 2

Response to Weakness 2: Comparison with functionally supervised models

We thank the reviewer for this suggestion. We agree that comparison with functionally supervised genomic models provides important context beyond the matched-backbone comparisons. We have therefore expanded the evaluation to include Enformer, SPACE-large, Borzoi, and AlphaGenome, alongside DNABERT-2 and HyenaDNA-large with VANDAM-MT. The table reports the evaluated context lengths and separates tasks according to documented task or label overlap with the functionally supervised comparators.

We group tasks by documented overlap with at least one functionally supervised comparator. Task overlap includes supervision on the same prediction task, even with held-out evaluation examples.

The functionally supervised models achieve the highest scores on 11 of the 15 tasks, demonstrating stronger overall performance in this comparison. VANDAM-MT achieves the highest scores on regulatory variants (eQTL), ClinVar pathogenicity, ClinVar indel, and LOL-EVE causal eQTL. The specialist models also lead on several tasks without documented task or label overlap, so their advantage cannot be attributed solely to overlap. These results position VANDAM as a complementary strategy for improving existing genomic foundation models through molecular supervision, rather than as a replacement for functionally supervised models. Differences in supervision, context length, and training data mean that this comparison assesses broader competitive performance rather than isolating the effect of the VANDAM training objectives.

Benchmarking against functionally supervised models

Task or data overlap

ModelAlphaGenome all-foldsAlphaGenome all-foldsSPACE-largeEnformerBorzoiBorzoiDNABERT-2 VANDAM-MTHyenaDNA-large VANDAM-MT
Sequence length16k128k8k196,608 bp8k512k2.5k8k
TF binding91.2691.3588.8691.5090.9490.7582.6080.22
Histone marks75.6771.8573.4471.7273.1569.4572.6054.87
CAGE expression67.0022.8162.7869.7465.9711.8436.6032.37
Regulatory variants (eQTL)69.1166.1569.5566.0569.9265.1171.8070.25
AuthorsOpenReview ↗

Response to Reviewer ZYdz, part 3

No task or label overlap

ModelAlphaGenome all-foldsAlphaGenome all-foldsSPACE-largeEnformerBorzoiBorzoiDNABERT-2 VANDAM-MTHyenaDNA-large VANDAM-MT
Sequence length16k128k8k196,608 bp8k512k2.5k8k
ClinVar pathogenicity66.4265.1561.7761.1466.4361.6769.7076.49
OMIM84.4488.8576.9289.4184.3391.0166.4051.88
TraitGym Mendelian73.2173.9671.9273.3573.1671.7557.5053.86
BRCA167.4466.6757.5964.5261.6063.4256.2054.39
Gene-expression variants70.7270.9970.1770.2469.8269.9369.4065.47
ClinVar indel51.1947.9050.0748.2552.6846.8872.1074.42
LOL-EVE causal eQTL53.1253.5351.5553.2952.1553.4555.5054.09
SongLab ClinVar50.2451.1153.8352.1054.8553.7952.8053.69
TraitGym complex64.2862.8761.9261.8763.2162.0052.2049.18
Variant Benchmarks coding70.1269.9564.7464.8664.2664.6063.5063.92
Variant Benchmarks non-coding93.7893.9183.5586.4887.6386.9872.8066.88

Response to Weakness 3: LFI and RPP under PT and MT and alternative masking treatments

We clarify that the approximately 47% zeroing rate applies specifically to the single-nucleotide MLM backbones NTv3-8M and Caduceus, rather than to all backbones. In DNABERT-2, BPE tokens pool local descriptors across multiple nucleotides; with 12% of sequence tokens replaced by MASK tokens, only 13.2% of the pooled token-level feature vectors become entirely zero under the observed token-length distribution, on the multi-species dataset. HyenaDNA uses causal next-token prediction and does not remove local features through MLM masking. The potential effect of masking-induced feature sparsity must therefore be considered separately for each backbone.

Feature zeroing by tokenization and training objective

BackboneTokenizationTraining objectiveObserved zeroed feature vectors
NTv3-8MSingle nucleotideMLM47%
CaduceusSingle nucleotideMLM47%
DNABERT-2Byte-pair encoding (BPE)MLM13.2%
HyenaDNASingle nucleotideNTP0%

To address the restriction of Appendix O to MT, we extended the mask-imputation evaluation to PT. The tables below compare zeroing, uniform imputation, and conditional imputation across all 15 tasks for NTv3-8M and Caduceus.

AuthorsOpenReview ↗

Response to Reviewer ZYdz, part 4

Mask imputation - PT

TaskNTv3-8M — UniformNTv3-8M — Cond.NTv3-8M — Zeroing (VANDAM)Caduceus — UniformCaduceus — Cond.Caduceus — Zeroing (VANDAM)
ClinVar pathogenicity62.8359.2160.9163.6562.7262.34
OMIM52.7857.0255.2251.6355.2253.73
TraitGym Mendelian52.2955.5558.6854.2857.1657.77
BRCA151.5952.5052.6353.3553.8255.38
TF binding81.7981.8780.6679.1675.2775.60
Regulatory variants (eQTL)68.1968.4768.7066.7365.2065.22
Gene-expression variants63.7963.7462.6961.3859.2359.40
Histone marks58.9458.2767.4854.7952.4052.84
CAGE expression34.2832.1732.4226.6222.1622.06
ClinVar indel54.3352.5755.4355.2955.5555.44
LOL-EVE causal eQTL55.5554.8951.3453.1753.0352.12
SongLab ClinVar54.2954.4054.3353.4353.5053.29
TraitGym complex51.6351.5250.9450.7650.6750.64
Variant Benchmarks coding62.7161.3062.1957.3458.0057.61
Variant Benchmarks non-coding61.4561.7161.5059.3360.5560.59
Mean AUROC59.4459.5060.1958.1658.0258.00

Mask imputation - MT

TaskNTv3-8M — UniformNTv3-8M — Cond.NTv3-8M — Zeroing (VANDAM)Caduceus — UniformCaduceus — Cond.Caduceus — Zeroing (VANDAM)
ClinVar pathogenicity68.6370.8170.2051.7651.0551.40
OMIM58.1255.9557.4051.1151.5953.10
TraitGym Mendelian56.1054.3556.8053.0853.6355.60
BRCA159.3961.1654.6053.1349.9252.50
TF binding78.8678.6379.8062.4966.0766.40
Regulatory variants (eQTL)69.7970.1369.3062.1564.2963.00
Gene-expression variants70.7170.0363.1058.4158.3859.90
Histone marks70.9771.3969.3049.9450.0655.40
CAGE expression33.3533.5035.9012.6416.8514.60
ClinVar indel57.3860.3262.6047.0947.4449.60
LOL-EVE causal eQTL49.0451.1554.2055.7857.3751.40
SongLab ClinVar52.0953.3256.0051.9451.4850.20
TraitGym complex51.8650.9350.8050.4349.9749.60
Variant Benchmarks coding62.0560.2763.7051.9651.1753.10
Variant Benchmarks non-coding61.8962.2864.9057.5058.8959.00
Mean AUROC61.9262.1962.3454.0654.3855.01

In MT, zeroing achieves the highest score on 8/15 tasks for each backbone and the highest mean AUROC across the 14 AUROC tasks: 62.34 for NTv3-8M and 55.01 for Caduceus, compared with 61.92 and 54.06 for uniform imputation and 62.19 and 54.38 for conditional imputation, respectively. These results support retaining zeroing as the default in MT.

In PT, RPP alone retains the highest mean AUROC on both backbones: 60.71 for NTv3-8M and 59.67 for Caduceus, compared with 60.19/59.44/59.50 and 58.00/58.16/58.02 for LFI with zeroing/uniform/conditional imputation, respectively. Thus, alternative masking treatments do not reverse the preference for RPP alone in PT, supporting the conclusion that LFI’s limited PT benefit is not explained solely by masking-induced feature sparsity.

Reviewer_SoR1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VANDAM, a method for incorporating molecular priors into DNA language models. These molecular features can be calculated solely from the sequence, and cover biophysical attributes of the DNA sequence. It proposes 2 methods to do this, Local Feature Injection (LI) and Region Property Prediction (RPP). RPP introduces an extra objective to predict molecular properties, to preserve this information in the embeddings, whilst LFI adds these features to the token embeddings. The paper explores applying these in pretraining and multi task fine tuning settings, and assesses performance of this across multiple DNA language models and benchmarks.

Critical strengths

Incorporation of molecular attributes into the DNA sequence has been underexplored in genomic models, so this would be a potentially valuable contribution and new perspective that the field can build upon. The method presented seems to be broadly useful, it is quite simple, and can be applied generally across multiple classes of models.

I also appreciate the breadth of experimental evaluation in the paper. The paper has applied this to diverse DNA language models, eg showing utility over different models with various tokenization and architecture types, and the ablations are thorough. The results show generally consistent performance improvements across models and benchmarks.

Critical weaknesses

It is not clear the advantage of the framework over incorporating the features directly with the embeddings in downstream tasks. Since this would be a basic way to incorporate the molecular features, I think that demonstrating utility over this is necessary, for example by showing a baseline that incorporates these features.

The specific benchmarks that have been used are also not clearly explained. For example, the authors evaluate on nine tasks from GFMBench-API, with a substantial emphasis on variant-effect prediction (six of the nine tasks involve variant effects). It is unclear why these particular tasks were selected over other tasks available in GFMBench-API, and whether they are representative of the broader range of genomic prediction tasks. I consider this to be an important weakness, since the results may be influenced by the choice of tasks and limits claims about the generality of the method.

Questions for authors
  1. Could the authors provide justification for the selection of the nine evaluation tasks, particularly given the emphasis on variant-effect prediction tasks?
  2. Can the authors provide clear justification for the advantage of using this framework over incorporating these features into the downstream probes? For example, running a baseline which adds the features to the model embeddings on downstream benchmarks, to assess how much benefit the framework adds over just incorporating these features with the embeddings.
3 comments
AuthorsOpenReview ↗

Response to Reviewer SoR1, part 1

Response to Weakness 1 and Question 2: Comparison with downstream feature baselines

We thank the reviewer for this suggestion and agree that additional downstream feature baselines help assess the contribution of local feature injection (LFI) in the multi-task (MT) setting. We therefore compare VANDAM-MT with two baselines that incorporate the same molecular features during downstream evaluation.

The first baseline, the connector probe, attaches a freshly initialized feature-injection connector to the frozen pretrained backbone, without additional sequence pre-training. For each supervised downstream task, the connector and linear task head are trained jointly from scratch. The connector maps 14 z-scored DNAshapeR channels to 64 dimensions, concatenates them with token embeddings, and projects back to the backbone embedding width, matching the connector architecture and feature configuration of the corresponding VANDAM-MT run. This adds 2,256 trainable connector parameters for NTv3-8M and 83,136 for HyenaDNA-small beyond the task head.

For VANDAM-MT, the connector is learned during multi-task training and remains frozen alongside the backbone during downstream evaluation; only the linear task head is trained.

The second baseline, the direct-feature probe, removes the connector entirely and concatenates fixed DNAshapeR summaries with the frozen backbone representation immediately before the linear head. For variant/reference pairs, each allele is summarized by mean-pooling the five DNAshape positions centered on the variant; sequence-level tasks use mean pooling over valid positions; CAGE uses pooling within its fixed output bins. Only the linear head is trained, with no learned feature transformation before it.

We evaluate both baselines on NTv3-8M and HyenaDNA-small to cover both MLM and NTP training objectives. We restrict this comparison to the seven supervised downstream tasks because both baselines require task-specific training, which is unavailable under the evaluation protocol for the remaining zero-shot tasks.

Two downstream feature baselines compared with VANDAM-MT

TaskNTv3-8M — ConnectorNTv3-8M — FeaturesNTv3-8M — VANDAM-MTHyenaDNA-small — ConnectorHyenaDNA-small — FeaturesHyenaDNA-small — VANDAM-MT
Extra trained parameters2,25628083,136280
Regulatory variants (eQTL)65.9967.8869.1670.3768.7969.94
Gene-expression variants61.2563.0263.2369.8564.3064.99
Histone marks56.8654.9169.2863.4654.6669.07
TF binding81.1481.2679.8985.7979.5481.08
CAGE expression4.383.8335.8337.4426.9331.14
Variant Benchmarks coding57.2858.1763.6763.1357.2763.08
Variant Benchmarks non-coding59.5572.4164.8660.9675.8268.12
Mean over six AUROC tasks63.6866.2868.3568.9366.7369.38

For NTv3-8M, VANDAM-MT scores higher than the connector probe on 6/7 tasks and the direct-feature probe on 5/7 tasks. For HyenaDNA-small, it scores higher on 2/7 and 6/7 tasks, respectively. These comparisons assess the value of incorporating LFI during multi-task pre-training versus introducing molecular features only during downstream evaluation.

AuthorsOpenReview ↗

Response to Reviewer SoR1, part 2

Response to Weakness 2 and Question 1: Benchmark selection and task coverage

GFMBench-API contains 23 tasks, of which we used eight for multi-task training. Due to time and computational constraints, the original submission reported results on nine of the remaining 15 tasks. We have now extended the evaluation to the six previously unreported tasks, reporting both pre-training and multi-task results across all four backbone families. The revised study therefore covers all 23 tasks: eight used for multi-task training and 15 used for evaluation. This expanded evaluation addresses the concern that our conclusions may depend on the initial task subset.

Results on the six additional evaluation tasks

Pre-training (PT)

TaskNTv3-8M — BaselineNTv3-8M — VANDAMCaduceus — BaselineCaduceus — VANDAMDNABERT-2 — BaselineDNABERT-2 — VANDAMHyenaDNA-small — BaselineHyenaDNA-small — VANDAM
ClinVar indel57.355.762.4⁻57.460.171.3⁺67.468.9⁺
LOL-EVE causal eQTL52.852.054.354.851.256.452.553.5
Songlab ClinVar51.353.6⁺51.153.6⁺49.750.7⁺52.457.5⁺
TraitGym complex50.450.950.851.549.649.949.547.9
Variant Benchmarks coding63.364.960.660.757.961.0⁺61.463.6⁺
Variant Benchmarks non-coding60.462.3⁺58.060.4⁺59.162.0⁺61.161.6

Multi-task training (MT)

TaskNTv3-8M — BaselineNTv3-8M — VANDAMCaduceus — BaselineCaduceus — VANDAMDNABERT-2 — BaselineDNABERT-2 — VANDAMHyenaDNA-small — BaselineHyenaDNA-small — VANDAM
ClinVar indel61.762.653.3⁻49.660.172.1⁺53.470.0⁺
LOL-EVE causal eQTL55.054.251.551.450.655.554.955.3
Songlab ClinVar55.856.050.150.250.652.8⁺49.353.0⁺
TraitGym complex49.950.849.649.650.752.248.949.5
Variant Benchmarks coding63.363.756.5⁻53.158.763.5⁺58.263.1⁺
Variant Benchmarks non-coding61.264.9⁺56.759.0⁺60.872.8⁺60.268.1⁺

VANDAM remains competitive on the six additional tasks, improving over the matched baselines on 4/6 tasks in PT and 5/6 in MT for NTv3-8M, 5/6 and 3/6 for Caduceus, 6/6 and 6/6 for DNABERT-2, and 5/6 and 6/6 for HyenaDNA-small, respectively. In response to other reviewer comments, we also evaluated HyenaDNA-large and NTv3-650M: on these six additional tasks, VANDAM improves over the matched baselines on 5/6 tasks in PT and 4/6 in MT for HyenaDNA-large, and 3/6 in both PT and MT for NTv3-650M.

Thank you for the extra results, they address my main concerns. Given the breadth of results and the underexplored perspective of incorporating molecular attributes into the DNA sequence, I think that this is a useful contribution to the community and I have increased my score to reflect this.

Reviewer_cEhWOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that genomic foundation models (GFMs) should treat DNA not merely as a string of symbols but also as a molecule with computable physical properties, and it proposes the VANDAM framework, which injects molecular priors through two complementary routes. Local Feature Injection (LFI) concatenates 14 DNAshapeR local structural descriptors at the input and is used only in multitask training with functional labels (VANDAM-MT). Region Property Prediction (RPP) requires the tile-pooled representation to predict 6 regional properties (curvature, AT content, melting propensity, and G4/cruciform/Z-DNA propensities) and is used in both the self-supervised continued pretraining (VANDAM-PT) and multitask stages. Compute-matched controlled experiments are conducted on 4 heterogeneous backbones (NTv3-8M, Caduceus, DNABERT-2, and HyenaDNA) and evaluated on 9 held-out tasks (GFMBench-API).

Critical strengths
  1. What impressed me most about this paper is that it evaluates a broad range of tasks related to genomic regulation, rather than, as in many of the previous works I have read, only running simple tasks such as promoter classification, splice site classification, and EMP.

  2. Introducing molecular priors into genomic model pretraining is an interesting point.

Critical weaknesses
  1. The experimental sequence context length in this study is capped at 8,192 bp and the models size are a few small. Consequently, how the VANDAM framework performs when the spatial scale of the molecular priors, model size and the sequence context length grow substantially remains unverified.

  2. In masked language modeling (MLM) training, the pentamer features that touch a masked position are zeroed out to prevent information leakage, which turns roughly 47% of the internal local feature vectors into zeros. Although the authors attempt feature imputation in Appendix O, the improvement is strongly task specific, and a comprehensive cross-model test across all backbone networks has not been performed.

  3. The paper should additionally discuss the following limitation: the molecular priors adopted in this study are, in essence, deterministic static properties determined by sequence alone, and as such they cannot directly capture real cell-specific contexts, dynamic environmental conditions, or conformational variability.

Questions for authors

Please refer to Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer cEhW - Part 1

Response to Weakness 1: Model size and sequence-length scaling

We thank the reviewer for raising this point and agree that evaluating larger models and longer sequence contexts is important for assessing the scalability of VANDAM. In response to Reviewer 2’s comment on task coverage, we expanded the evaluation from the original nine tasks to 15 tasks, adding six previously unreported tasks. We evaluated larger models, HyenaDNA-large (6.6M parameters) and NTv3-650M (650M parameters), in both the pre-training (PT) and multi-task (MT) regimes across this expanded suite. For longer contexts, we focused on the PT regime: we trained VANDAM-PT with NTv3-8M at 32,768 bp (2¹⁵) and compared it to the compute matched training at the same sequence length. We also examined context extrapolation without further pre-training. Specifically, we evaluated the Baseline-PT and VANDAM-PT checkpoints on context lengths of up to 32,768 bp, although they had been pre-trained with a maximum context length of only 8,192 bp. The long-context table retains the seven tasks whose inputs can extend to 32,768 bp and for which both long-context comparisons are available.

Larger model results across all 15 evaluation tasks — PT

TaskHyenaDNA-large — Baseline-PTHyenaDNA-large — VANDAM-PTNTv3-650M — Baseline-PTNTv3-650M — VANDAM-PT
ClinVar pathogenicity69.0679.13⁺49.8274.93⁺
OMIM52.5054.6545.5857.68⁺
TraitGym Mendelian50.5152.7049.7057.57⁺
BRCA150.7258.67⁺48.7361.06⁺
TF binding77.1678.19⁺84.3585.37⁺
Regulatory variants (eQTL)68.1968.59⁺69.2168.87
Gene-expression variants63.1163.90⁺65.4066.28⁺
Histone marks53.0255.24⁺61.6263.70⁺
CAGE expression26.6230.30⁺21.4344.09⁺
ClinVar indel68.3073.63⁺48.9259.09⁺
LOL-EVE causal eQTL52.5553.7055.0453.21
Songlab ClinVar52.5756.88⁺50.6759.65⁺
TraitGym complex49.3748.9549.7651.21
Variant Benchmarks coding61.7662.4973.15⁻71.41
Variant Benchmarks non-coding61.0061.4168.31⁻64.72

Larger model results across all 15 evaluation tasks — MT

TaskHyenaDNA-large — Baseline-MTHyenaDNA-large — VANDAM-MTNTv3-650M — Baseline-MTNTv3-650M — VANDAM-MT
ClinVar pathogenicity74.4476.49⁺65.24⁻63.77
OMIM49.8251.8860.59⁻56.43
TraitGym Mendelian48.8953.86⁺55.9857.34
BRCA151.8554.3961.3159.83
TF binding76.5380.22⁺83.3284.54⁺
Regulatory variants (eQTL)69.7470.25⁺71.41⁻70.78
Gene-expression variants64.6765.47⁺71.3973.36⁺
Histone marks53.8954.87⁺65.91⁻64.93
CAGE expression31.0632.37⁺39.0241.41⁺
ClinVar indel74.8474.4256.9760.31⁺
LOL-EVE causal eQTL54.9454.0955.3152.96
Songlab ClinVar52.5253.69⁺54.12⁻53.14
TraitGym complex48.8149.1852.4952.30
Variant Benchmarks coding63.9063.9268.1869.65⁺
Variant Benchmarks non-coding64.7966.88⁺74.5782.41⁺
AuthorsOpenReview ↗

Response to Reviewer cEhW - Part 2

NTv3-8M at 32k sequence length

TaskTrained at 32k, evaluated at 32k — Baseline-PTTrained at 32k, evaluated at 32k — VANDAM-PTTrained at 8k, evaluated at 32k — Baseline-PTTrained at 8k, evaluated at 32k — VANDAM-PT
ClinVar VEP49.5563.34⁺51.4959.47⁺
Pathogenic OMIM45.0152.82⁺51.0458.40⁺
TraitGym Mendelian39.8855.92⁺49.4755.29⁺
BRCA152.0252.8248.3152.43⁺
Histone marks50.3856.30⁺55.6756.64
Causal eQTL63.8163.8364.7865.45
CAGE4.5333.55⁺24.3135.01⁺

Model-size scaling. VANDAM retains benefits on larger backbones, improving scores on 14/15 tasks for HyenaDNA-large PT and 13/15 for MT, and on 11/15 tasks for NTv3-650M PT and 7/15 for MT.

Sequence-length scaling. On the seven long-context tasks shown above, VANDAM-PT trained at 32,768 bp improves over Baseline-PT on all seven tasks, with five significant gains. Separately, the checkpoints trained at up to 8,192 bp demonstrate context extrapolation without additional pre-training: VANDAM-PT outperforms Baseline-PT on all seven tasks when evaluated at 32,768 bp. These results support benefits both from longer-context training and from evaluation beyond the training context on this task subset.

Response to Weakness 2: Masked-feature handling across backbones

We agree with the reviewer that the benefits of feature imputation are task dependent and that the original ablation did not sufficiently assess this issue across backbones. We also acknowledge that the wording in Appendix O incorrectly implied that approximately 47% of local feature vectors are zeroed out for all backbones. This applies specifically to MLM with single-nucleotide resolution, as used by NTv3-8M and Caduceus in our experiments.

The extent of feature zeroing depends both on the training objective and on the tokenization. Under DNABERT-2’s BPE tokenization, 12% of sequence tokens are replaced by MASK token, and each injected vector mean-pools nucleotide-level features across a variable-length BPE token. Using the observed DNABERT-2 token-length distribution on the multi-species dataset, 13.2% of the resulting pooled token-level feature vectors are entirely zero, substantially below the 47% observed at single-nucleotide resolution. For HyenaDNA, which uses next-token prediction (NTP), no feature vectors are zeroed out.

Feature zeroing by tokenization and training objective

BackboneTokenizationTraining objectiveObserved zeroed feature vectors
NTv3-8MSingle nucleotideMLM47%
CaduceusSingle nucleotideMLM47%
DNABERT-2Byte-pair encoding (BPE)MLM13.2%
HyenaDNASingle nucleotideNTP0%

We expand the feature-imputation ablation to include Caduceus alongside NTv3-8M. The revised ablation therefore covers both single-nucleotide MLM backbones affected by the 47% zeroing rate.

AuthorsOpenReview ↗

Response to Reviewer cEhW - Part 3

Masked-base handling ablation in VANDAM-MT

TaskNTv3-8M VANDAM-MT — UniformNTv3-8M VANDAM-MT — Cond.NTv3-8M VANDAM-MT — Zeroing (VANDAM)Caduceus VANDAM-MT — UniformCaduceus VANDAM-MT — Cond.Caduceus VANDAM-MT — Zeroing (VANDAM)
ClinVar pathogenicity68.6070.8070.2051.7651.0551.40
OMIM58.1055.9057.4051.1151.5953.10
TraitGym Mendelian56.1054.3056.8053.0853.6355.60
BRCA159.4061.2054.6053.1349.9252.50
TF binding78.9078.6079.8062.4966.0766.40
Regulatory variants (eQTL)69.8070.1069.3062.1564.2963.00
Gene-expression variants70.7070.0063.1058.4158.3859.90
Histone marks71.0071.4069.3049.9450.0655.40
CAGE expression33.3033.5035.9012.6416.8514.60
ClinVar indel57.3860.3262.6047.0947.4449.60
LOL-EVE causal eQTL49.0451.1554.2055.7857.3751.40
SongLab ClinVar52.0953.3256.0051.9451.4850.20
TraitGym complex51.8650.9350.8050.4349.9749.60
Variant Benchmarks coding62.0560.2763.7051.9651.1753.10
Variant Benchmarks non-coding61.8962.2864.9057.5058.8959.00
Mean AUROC (14 tasks)61.9262.1862.3454.0654.3855.01

Across both backbones, zeroing shows highest performance: it achieves the highest score on 8 of 15 tasks for NTv3-8M and 8 of 15 tasks for Caduceus. It also achieves the highest mean AUROC across the 14 AUROC tasks: 62.34 versus 61.92 for uniform and 62.18 for conditional imputation on NTv3-8M, and 55.01 versus 54.06 and 54.38, respectively, on Caduceus. These results support retaining zeroing as the standard masking treatment, with imputation offering task-specific benefits rather than a broadly preferable replacement.

Response to Weakness 3: Limitations of static molecular priors

We agree and will add the following paragraph to the Limitations section:

“The molecular priors used in VANDAM are deterministic, static functions of the input sequence. They therefore cannot directly represent cell-specific molecular states, responses to changing environmental conditions, or conformational variability in vivo. Although VANDAM priors do not capture these contextual factors, they provide a biologically motivated inductive bias that yields consistent improvements over matched baselines across diverse genomic prediction tasks. These gains demonstrate their utility despite this limitation, but do not replace context-specific experimental measurements. Incorporating such measurements is an important direction for extending VANDAM to context-dependent genomic prediction.”

Comments on the paper

1 comment

Dear Area Chair,

Thank you for overseeing the review of our submission. We appreciate the reviewers’ constructive feedback and have provided point-by-point responses supported by substantial additional experiments and clarifications of the contribution and its limitations. We summarize the main additions below.

  1. Expanded evaluation and scalability (Reviewers cEhW and SoR1). We expanded evaluation from nine to 15 tasks outside our eight-task multi-task training pool, addressing concerns about selective task coverage. On the six additional tasks, VANDAM improves the point estimate over matched baselines in 20/24 task–backbone comparisons in each training regime, supporting benefits beyond the original subset. We also evaluated HyenaDNA-large and NTv3-650M across all 15 tasks. VANDAM improves on 14/15 and 11/15 tasks, respectively, in pre-training (PT), and 13/15 and 7/15 in multi-task training (MT). At longer contexts, NTv3-8M trained at 32,768 bp improves over the evaluated baseline on all seven eligible tasks, with five significant gains. Checkpoints trained at 8,192 bp also outperform their comparison baselines on all seven tasks when evaluated at 32,768 bp without further pre-training. These experiments extend the evaluation to larger models and longer contexts; baseline definitions and full results are provided in our reviewer responses.
  2. Downstream feature baselines (Reviewer SoR1). We added two baselines using the same molecular features during downstream evaluation: a task-specific trainable feature-injection connector and direct feature concatenation. Across six AUROC tasks, VANDAM-MT achieves higher mean AUROC than both alternatives: 68.35 versus 63.68 and 66.28 for NTv3-8M, and 69.38 versus 68.93 and 66.73 for HyenaDNA-small. The connector baseline goes beyond linear probing by training a feature-injection module separately for each downstream task. VANDAM-MT instead freezes its previously learned connector and backbone and trains only a linear task head. These results support incorporating molecular priors during continual training rather than introducing them only at downstream evaluation.
  3. Masking and component ablations (Reviewers cEhW and ZYdz). We addressed whether masking-induced feature removal explains the different benefits of local feature injection (LFI) and regional property prediction (RPP). The approximately 47% feature-zeroing rate applies specifically to single-nucleotide MLM backbones (NTv3-8M and Caduceus), compared with 13.2% for BPE-tokenized DNABERT-2 and 0% for causal HyenaDNA. Expanded imputation experiments on NTv3-8M and Caduceus show that zeroing achieves the highest mean AUROC in MT, while RPP alone remains preferable in PT across the tested masking treatments. We also repeated the component and token-objective ablations on HyenaDNA-small, which has no masking-induced feature removal. The same configuration ranking persists: RPP alone performs best in PT and LFI + RPP in MT; removing the native token loss reduces performance in both. Thus, masking-induced feature sparsity alone cannot explain the observed regime dependence. These findings strengthen the interpretation that molecular supervision complements the native token objective and that the benefits of LFI and RPP depend on the training regime.
  4. Additional clarifications and comparisons (Reviewer ZYdz). We clarified the distinction between VANDAM and biophysical-prior methods already cited in our Related Work (UTR-LM which focuses on RNA analysis and MixBend which performs DNA bendability prediction), and have expanded this discussion in the revision: specifically, VANDAM introduces tile-level regional property prediction (RPP) across multiple DNA properties during continual foundation-model training and examines its interaction with local feature injection (LFI) across training regimes and diverse downstream tasks. Additionally, we added comparisons against functionally supervised models, positioning VANDAM as a complementary enhancement to foundation models that even outperforms specialist models on variant-effect tasks. The original biological-content controls and held-out molecular-descriptor probes complement the expanded evaluation: they support the importance of biological target content and improved access to molecular descriptors beyond those explicitly supervised during training.

Together, these additions address the principal reviewer concerns and make the scope of our claims more precise. The expanded evidence supports molecular supervision as a way to improve transferable genomic representations across architectures, with benefits that depend on the training regime and task. We respectfully ask that the submission be assessed in light of these additional results and clarifications.

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.