acceptodds
Under review as a conference paper at ICLR 2027

JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures

Abstract

Genomic Foundation Models (GFMs) typically rely on Masked Language Modeling (MLM) or Next-Token Prediction (NTP) to learn the "Laws of Nature". While effective at capturing local syntax, these generative paradigms prioritize token-level reconstruction over high-level functional context. We introduce JEPA-DNA, a model-agnostic continual training framework that integrates a Joint-Embedding Predictive Architecture (JEPA) with traditional generative objectives. By supervising global sequence embeddings in a latent space, JEPA-DNA forces models to predict the functional representations of masked genomic segments, shifting the learning signal from token recovery to semantic alignment. We evaluate JEPA-DNA on 17 diverse genomic benchmark tasks, demonstrating consistent gains in linear probing and zero-shot performance regardless of the underlying GFM architecture or generative objective, and establishing a new state-of-the-art for GFMs under the linear-probing protocol. Through extensive ablation studies, we further characterize the synergistic interplay between generative and latent objectives. Our code is provided in the supplementary material.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Joint-Embedding Predictive Architecture, JEPA, genomic representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xhCTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1(−1)
Presentation
3
Soundness
2
Read review
Paper summary

The paper adds a JEPA branch to the continual pre-training of existing genomic foundation models (GFMs). JEPA means the model predicts the embedding of masked spans, not the masked tokens. Here a predictor maps the masked-input embedding to the global embedding of the unmasked sequence, computed by an EMA copy of the encoder. The native MLM or next-token loss is kept. VICReg variance and covariance penalties prevent collapse, and a schedule grows the masked spans over training. The recipe is applied to DNABERT-2, NTv3 and HyenaDNA, using subsets of each model's own pre-training data.

Evaluation:

  • Frozen linear probe (encoder fixed, only a linear head trained) on 9 tasks: mean gains of +2.60%, +2.19% and +3.29% relative AUROC over the original checkpoints (Table 1). Native continual-training controls at matched steps and compute gain much less.
  • Zero-shot variant scoring by cosine distance between reference and variant embeddings (Tables 2, 9).
  • Factorial ablation on DNABERT-2 (Table 3).
Critical strengths

A cheap add-on that transfers across very different architectures, with careful controls. The same recipe is applied to a Transformer with MLM, a hybrid U-Net/Transformer, and a long-convolution model with next-token prediction. The paper includes step and compute matched native baselines, keeps data parity, and runs a factorial ablation. In the ablation, the latent objective adds about +3.0% on top of MLM + VICReg + masking (+3.24% to +6.22%, Table 3). Costs (Table 4) and regressions (e.g., HyenaDNA TF binding 8.60%) are reported openly. Improving off-the-shelf GFM embeddings cheaply is useful.

Critical weaknesses
  1. The probe may reward the readout's geometry rather than information. The probe reads one pooled vector per sequence: the CLS token for DNABERT-2 (Table 1), which MLM pretraining does not train to summarize the sequence. The method acts on this kind of vector directly. The VICReg variance and covariance terms are applied to the global vector h_glob (p.5), and the variance term carries the largest loss weight (25, p.23). The latent objective also predicts it. The probe is also brief and fixed across models: 3 epochs of AdamW at lr 3e-5 (p.24), with no feature standardization or per-model tuning described. It often does not fit the original checkpoints. Splice MCC is 0.000, −0.004 and −0.012 for the three originals (Table 5), and the DNABERT-2 probe reaches only 0.653 splice AUROC, versus 0.974 under full fine-tuning (Tables 1, 15). Under such a probe, a vector that is better spread and decorrelated can score higher without containing more information. The CLS concern does not apply to NTv3, which is read by mean pooling and still gains +2.19%, but the probe-conditioning concern does. The ablation is consistent with this partly happening: VICReg without latent prediction already gives +3.24% (Table 3). With a converged probe and a readout that does not depend on the CLS token, the gains may hold, shrink, or disappear. This determines my rating.

  2. For two of three backbones, the native controls may not be informative. For NTv3, the step- and compute-matched rows equal the original checkpoint on 6 of 9 tasks and lie within 0.001 on the rest (Table 1). The HyenaDNA controls move by at most 0.013. The controls' optimizer and learning rate are not reported. JEPA-DNA itself uses SGD with very small learning rates (NTv3 phase-2 lr 1e-6 to 1e-8, p.23). If the controls share this regime, their weights may hardly change. In that case, "not a by-product of additional optimization" is shown only for DNABERT-2. Even there, on some tasks the controls account for most of the gain over the original checkpoint. On GUE Promoter, compute-matched native training reaches 0.927 vs JEPA-DNA's 0.928, from 0.884; on LRB causal eQTL it reaches 0.705 vs 0.711, from 0.697 (Table 1). So the per-task gains over the original checkpoint overstate what is specific to the JEPA objective.

Questions for authors

Requests that could change my assessment:

  1. To test whether JEPA-DNA adds information, rather than changing the geometry or readout of the probed vector, can you compare the original checkpoint, the compute-matched native control and JEPA-DNA under two converged frozen probes on standardized features, with hyperparameters tuned on validation? First, a regularized linear probe in which each model uses its best layer and pooling (native readout, mean or max), selected on validation. Second, an attentive probe: a learned query that attends over the token embeddings of the selected layer, with learned key/value projections, followed by a linear classifier. This is the frozen protocol under which V-JEPA reports its feature-prediction gains (Bardes et al., 2024). Please report the paired difference against the compute-matched control, with 95% paired-bootstrap confidence intervals per task and for the mean across tasks. Neither probe needs additional pretraining. Could you also confirm whether the probed vector is the same h_glob that the VICReg and latent losses act on (CLS for DNABERT-2, last token for HyenaDNA, mean pool for NTv3)?

Minor Asks

  • In Table 15, fully fine-tuning the original DNABERT-2 on LRB causal eQTL gives 0.667, below its frozen-probe score (0.697, Table 1). This suggests the fine-tuning recipe may be unstable for the baseline, so the +8.51% gain should be interpreted with caution. A comparison against the compute-matched control would be more informative.
  • Report gains against the compute-matched native control, not only the original checkpoint. On DNABERT-2, the control accounts for most of the gain on some tasks: GUE Promoter 0.927 vs JEPA-DNA's 0.928 (from 0.884), and LRB causal eQTL 0.705 vs 0.711 (from 0.697) (Table 1). Please also state the controls' optimizer and learning-rate schedule; the paper says only that they "differ only in budget" (p.6).
  • Present the cosine-distance zero-shot results as exploratory. Cosine distance on pooled embeddings is sensitive to the same geometry changes. Zero-shot variant scoring by the likelihood ratio from the retained MLM/NTP head (as in GPN, Benegas et al., PNAS 2023; Evo 2, Brixi et al., Nature 2026) would give evidence independent of the pooled embedding.
  • The I-JEPA evaluation (Assran et al., CVPR 2023) used a tuned linear protocol (learning-rate and weight-decay sweeps, optional batch normalization before the linear head, best of two readouts). Adopting a comparable protocol would make the comparison more convincing.
  • Calibrate the claims about biology and range. The current evidence supports better-conditioned, more linearly accessible representations, which is useful in itself. It does not yet directly show "biologically relevant features" or "high-level biological context" (p.2, p.9). "Long-range interactions that govern genomic function" (p.1) is also not supported. Training windows are 512 tokens (≈2.5 kb) for DNABERT-2 and 8,192 bp for NTv3 and HyenaDNA, evaluation inputs are truncated to the same lengths (p.24), and Table 16 is presented as a failure-mode analysis "rather than as evidence of long-range reasoning." If the gains survive Question 1, the former framing is well supported. The latter claims would need evidence that does not read the pooled embedding and that spans beyond the training window.
5 comments

Response to Reviewer xhCT - part 1

We thank Reviewer xhCT for their feedback and valuable comments. We have revised the manuscript accordingly and address each question and concern raised below.

Hyperparameter-Tuned Linear Probing Analysis (Question 1 + Weakness 1 + Minor ask 4):

We confirm that the probed vector is the designated global representation, h_glob​, used by the VICReg constraints: the final-layer CLS embedding for DNABERT-2, last-token embedding for HyenaDNA, and mean-pooled embedding for NTv3.

As also described in our response to reviewer 9eSv, we have added a “Hyperparameter-Tuned Linear Probing” analysis in Appendix A.2 of the revised paper, with results summarized in Figure 2, to test whether our conclusions persist under independently tuned probes.

For each task and model in the DNABERT-2 comparison, we independently optimize the learning rate and weight decay using Optuna’s Bayesian TPE sampler, with 10 trials over log-uniform ranges of 10⁻⁵–10⁻² and 10⁻⁸–10⁻¹, respectively. We increase probe training from 3 to 10 epochs and select both the hyperparameter configuration and the best epoch by validation AUROC. We use the predefined validation splits for GUE tasks and hold out 10% of the training data for the remaining tasks. The held-out test set is evaluated only after selection.

We report full-test performance together with bootstrap medians and percentile 95% confidence intervals, computed from 1,000 paired bootstrap resamples using identical test-example indices for JEPA-DNA and its compute-matched control. The overall ranking trend persists under this validation-selected protocol: JEPA-DNA outperforms the compute-matched control on 6 of 9 supervised tasks, with four improvements remaining statistically significant at the 95% confidence level.

Regarding layer selection and aggregation, JEPA-DNA explicitly supervises the designated sequence-level representation to increase the information accessible through that representation. Our main evaluation therefore uses the same final-layer representation and aggregation operator during continual pretraining and downstream probing. Across backbones, these operators include CLS aggregation for DNABERT-2 (as in the original paper), mean pooling for NTv3, and last-token aggregation for HyenaDNA.

V-JEPA predicts continuous representations of masked spatiotemporal patches, whereas JEPA-DNA supervises a designated global sequence representation. This distinction motivates our primary evaluation of that representation, while alternative readouts test whether the benefits extend beyond it. Notably, V-JEPA 2.1 also introduces direct supervision at multiple intermediate encoder layers during pretraining, rather than using those layers only at evaluation. Exploring analogous intermediate-layer supervision in JEPA-DNA is a direction for future work.

Nevertheless, to address this question directly, we additionally evaluated the reported DNABERT-2 checkpoints with the layer and pooling operator selected independently for each model and task by validation performance, without additional pretraining. Under this protocol, JEPA-DNA significantly outperforms its compute-matched control on three tasks and significantly underperforms on one task. The mean paired AUROC difference across tasks is positive (towards JEPA-DNA), but its 95% paired-bootstrap confidence interval includes zero: [−0.00053, +0.00445]. Thus, some task-specific gains persist under validation-selected readouts, but the average advantage is not statistically significant under this protocol.

This indicates that some gains transfer to mean pooling and layers extractions, but the advantage is less consistent than when evaluating the last layer's CLS representation explicitly optimized during JEPA-DNA training. We have also updated the Future Work section of the revised paper to include evaluating JEPA-DNA with alternative pooling methods, and layers extraction during continual pre-training.

Response to Reviewer xhCT - part 2

Native controls may not be informative (Weakness 2):

We thank the reviewer for the opportunity to clarify this point. JEPA-DNA itself includes the same native token-prediction objective (e.g. MLM for DNABERT-2). To isolate the contribution of the JEPA components, the native controls retained this objective and the same optimization settings and training budget, while disabling only the JEPA-specific components.

Although the native controls showed limited changes on some downstream tasks, their self-supervised training was still impactful. For DNABERT-2, the step-matched native-MLM checkpoint had a global relative parameter drift of 4.54% from the original checkpoint, with a maximum tensor-wise drift of 22.4%. More importantly, on the same 1,000 sequences and fixed masks, perplexity decreased from 349.2 to 225.9 with 15% masking and from 474.3 to 291.4 with 30% masking (reductions of 35.3% and 38.6%, respectively). Thus, the native objective meaningfully updated the model even when this did not consistently improve every downstream task.

Compute-matched Full Fine-tuning (Minor ask 1):

We thank the reviewer for this suggestion, and we have added the compute-matched control to the full fine-tuning comparison in Table 17 of the revised manuscript. On LRB causal eQTL, this control achieves 0.6620 AUROC, compared with 0.7237 for JEPA-DNA, corresponding to a 9.32% relative improvement.

Gains and controls optimization details (Minor ask 2):

We thank the reviewer for highlighting the importance of the compute-matched comparison. The “Gain” columns in Tables 1 and 2 report relative AUROC improvements against the original checkpoint, step-matched control, and compute-matched control, respectively. We have additionally specified the controls’ optimizer and learning-rate schedules in the “Training & Implementation Details” subsection of Appendix A.3 in the revised manuscript.

Zero-shot evaluation (Minor ask 3):

We thank the reviewer for this observation. Our embedding-based zero-shot evaluation is intended to assess the global sequence representations that JEPA-DNA explicitly optimizes, which are central to this work. We therefore evaluate whether distances between these representations reflect functional differences without task-specific training, rather than evaluating the retained token-prediction head (as in Evo2 for example). To investigate whether the changes reflect differential sensitivity to functional effects rather than simply a general increase in embedding distances, we added a distribution-shift analysis in Appendix A.2, Table 15. For identical variants evaluated with the DNABERT-2 baseline and JEPA-DNA, we calculate the ratio of the median reference–variant cosine distance for putatively impactful variants to that for control variants. This ratio increases on seven of eight datasets, including from 1.884 to 2.773 for ClinVar pathogenicity, 1.099 to 1.378 for LRB pathogenic OMIM, and 1.156 to 1.390 for conserved versus non-conserved positions. These results suggest that the learned geometry exhibits greater relative sensitivity to biologically relevant variation, rather than merely increasing all distances uniformly.

Biology and range claims (Minor ask 5):

We thank the reviewer for this clarification. We have revised the manuscript to distinguish the demonstrated improvements in representation utility from claims about biological understanding and long-range reasoning. Specifically:

  • Page 1: We replaced “capture the high-level regulatory logic and long-range interactions that govern genomic function” with “capture sequence-level information in global representations.”
  • Page 2: We replaced “learns more linearly-separable and biologically relevant features” with “improve the linear accessibility of information relevant to genomic prediction tasks.”
  • Page 9: We replaced “JEPA-DNA captures high-level biological context more effectively than generative objectives alone, consistently boosting backbone performance across supervised and zero-shot tasks” with “JEPA-DNA improves backbone performance on the majority of evaluated supervised and zero-shot tasks.”

Thank you for the revision and the additional analyses. The repeated continual-pretraining runs, the tuned probes with paired bootstrap intervals, and the calibrated claims strengthen the paper. My two main points stand as follows.

PointStatus
W1: added information vs. readout geometryNot resolved. With validation-selected layer and pooling, the mean gain over the compute-matched control is not significant (CI [−0.0005, +0.0045]); with mean pooling, JEPA-DNA is better on 5 of 9 tasks. Pairwise cosine similarity falls from 0.40 to 0.15, confirming a large change in the optimized vector's geometry.
W2: native controls for NTv3 and HyenaDNANot resolved. Evidence that the controls meaningfully trained is shown for DNABERT-2 only.

The revised claim, improved linear accessibility of the designated global vector, is supported on DNABERT-2. My remaining concerns:

1. The gain should exceed simple baselines. Two comparisons against the compute-matched control, with paired bootstrap intervals, would show what JEPA adds over the same compute: (i) both models with the same post-hoc whitening (e.g., PCA or BERT-whitening), which controls for the geometry change; (ii) an attentive probe over the token embeddings, excluding the CLS token, as in V-JEPA's frozen evaluation (Bardes et al., 2024), which shows whether the gain reflects better token representations or information already present being routed into the optimized vector.

2. The gain should be substantial. Under validation-selected readouts the mean advantage is at most +0.0045 AUROC, and across the repeated runs clear gains appear on only four of nine tasks.

3. There is no evidence that the model learns functional or biological features. The supervised gains are measured on the optimized vector and do not hold on average under validation-selected readouts. The zero-shot evidence (variant-induced shifts, ClinVar separation) uses the same pooled vector, whose geometry changes substantially. As the response acknowledges, the objective does not directly supervise biological function. Yet the paper motivates JEPA as a way to learn abstract, functional features. Without evidence that the model itself learns them, better accessibility of a single vector does not require continual pretraining if a better readout or whitening provides it.

4. The paper should report the unfavorable results and align its claims. The validation-selected readout and mean-pooling results appear only in this discussion, while the appendix reports only the tuned probe on the optimized vector. The title ("Grounding") and parts of the text still describe functional grounding.

JEPA-style latent prediction with VICReg is established (I-JEPA, V-JEPA, LLM-JEPA, data2vec), and the demonstrated benefit here is a small, readout-specific gain in linear accessibility. I keep my rating at weak rejection. I lowered Informativeness to 1 because the revision withdraws the claim that the model learns functional or biological features. The remaining demonstrated contribution, improved linear accessibility of the optimized vector, is a known effect of JEPA- and VICReg-style objectives.

Reviewer_9eSvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

JEPA-DNA augments continual pre-training of genomic foundation models with prediction of an EMA teacher’s global sequence representation, alongside the original token-level objective and VICReg regularization. The framework supports three backbones with different architectures, tokenizations, and pooling strategies. Evaluation covers nine linear-probing and eight zero-shot tasks, with step-matched and compute-matched continual-training controls. Results show improvements on most supervised comparisons, while zero-shot benefits are less consistent. Extensive ablations examine the contributions of latent prediction, regularization, masking, predictor architecture, and optimization.

Critical strengths

1.The evaluation spans three substantially different backbones and includes both supervised probing and zero-shot variant scoring. The gains are not confined to one architecture: NTv3 improves on eight of nine supervised tasks, while DNABERT-2 shows meaningful improvements on transcription-factor binding and splice-site prediction. 2.The paper makes a serious effort to isolate the source of improvement. Step- and compute-matched continual-training baselines, EMA and masking controls, and the no-JEPA regularization variant help distinguish the proposed combination from simply training longer. The contrastive comparisons also include a variant initialized from the same DNABERT-2 backbone. 3.The supplementary experiments are useful and acknowledge trade-offs. They include full fine-tuning, loss-weight sensitivity, predictor-depth comparisons, and detailed regressions on individual tasks. Discarding the auxiliary branches at inference is a practical advantage.

Critical weaknesses

1.The evidence supports improved representations more directly than “functional grounding.” The main objective aligns a corrupted sequence’s global representation with that of the full sequence; it does not directly supervise biological function or require recovery of individual masked-span representations. Local composition or motif statistics could contribute to this alignment. The embedding analysis compares the context and EMA encoders rather than demonstrating an improvement over the original backbone under the same analysis. The length sweep is appropriately described as a stability test, but it does not substantiate the broader long-range reasoning motivation. 2.The linear-probing protocol needs stronger validation before supporting state-of-the-art claims. Every probe uses three epochs and a learning rate of 3 × 10⁻⁵, without reported convergence checks or validation-based selection of optimization settings. A common training recipe does not necessarily optimize representations with different scales and geometry equally well. Pooling sensitivity is also relevant because JEPA explicitly trains the global representation being evaluated. Table 6 repeats probe fitting over five seeds, but does not measure variation across independent continual-pretraining runs or uncertainty from the benchmark samples. 3.Some reporting details weaken the compute-matched comparison and the generality claims. In Table 4, 100,000 DNABERT-2 steps at 2.661 seconds imply approximately 73.9 GPU-hours, rather than the reported 128.9; comparable discrepancies occur for the other backbones. These could reflect additional overhead, but the accounting needs explanation because compute matching is a central control. Benefits are also uneven: HyenaDNA drops from 0.698 to 0.638 on supervised TF binding, and several zero-shot results remain near chance or deteriorate. The abstract’s wording about consistent gains should reflect these limitations.

Questions for authors

1.Do the main linear-probing rankings persist when probes are trained to convergence with validation-selected learning rates and regularization? For DNABERT-2, how do the results change with mean pooling instead of the CLS representation? 2.Can you provide repeated continual-pretraining runs for a representative backbone, together with paired uncertainty estimates on key evaluation tasks? Please distinguish this variability from the probe-initialization variability in Table 6

4 comments

Response to Reviewer 9eSv - part 1

We thank Reviewer 9eSv for their positive feedback and valuable comments. We have revised the manuscript accordingly (PDF file uploaded) and address each question and concern raised below.

Hyperparameter-Tuned Linear Probing Analysis (Question 1 + Weakness 2):

We thank the reviewer for highlighting that a shared optimization recipe does not necessarily provide an equally optimized probe for every representation. We have added a “Hyperparameter-Tuned Linear Probing” analysis in Appendix A.2 of the revised paper, with results summarized in Figure 2, to test whether our conclusions persist under independently tuned probes.

For each task and model in the DNABERT-2 comparison, we independently optimize the learning rate and weight decay using Optuna’s Bayesian TPE sampler, with 10 trials over log-uniform ranges of 10-5–10-2 and 10-8–10-1, respectively. We increase probe training from 3 to 10 epochs and select both the hyperparameter configuration and the best epoch by validation AUROC. We use the predefined validation splits for GUE tasks and hold out 10% of the training data for the remaining tasks. The held-out test set is evaluated only after selection. We report full-test performance together with bootstrap medians and percentile 95% confidence intervals, computed from 1,000 paired bootstrap resamples using identical test-example indices for JEPA-DNA and its compute-matched control.

The overall ranking trend persists under this validation-selected protocol: JEPA-DNA outperforms the compute-matched control on 6 of 9 supervised tasks (two task-level rankings reverse, one in favor of JEPA-DNA and the other in favor of the control), with four improvements remaining statistically significant at the 95% confidence level under paired bootstrap resampling. JEPA-DNA explicitly supervises a designated sequence-level representation, so our primary evaluations preserve training–evaluation consistency within each backbone: CLS for DNABERT-2, mean pooling for NTv3, and last-token aggregation for HyenaDNA. To address the reviewer’s question, we additionally evaluated mean-pooled embeddings from the reported DNABERT-2 checkpoints without retraining the CLS-based model. JEPA-DNA outperformed its control on 5 of 9 tasks, indicating partial but less consistent transfer to an aggregation operator not directly supervised during training. We also updated Future Work to include alternative pooling methods during continual pre-training.

Robustness to Continual-Training Seeds (Question 2 + Weakness 2):

We thank the reviewer for this distinction. Table 6 measures probe-fitting variability with fixed pretrained representations. We have now added three independent continual-pretraining runs for both JEPA-DNA and its compute-matched control using DNABERT-2. Table 16 in the revised Appendix, reports mean AUROC ± standard deviation across these runs, and included below for convenience. The overall trend persists: JEPA-DNA achieves higher mean performance on seven of nine supervised tasks and six of eight zero-shot tasks. Further details appear in “Robustness to Continual-Training Seeds” in Appendix A.2 and Table 16 of the revised paper.

Supervised Tasks

TaskCompute-matched controlJEPA-DNARelative gain (%)
GUE TF Binding0.822 ± 0.0230.841 ± 0.001+2.31
GUE Promoter0.927 ± 0.0000.930 ± 0.003+0.32
GUE Splice Site0.672 ± 0.0010.700 ± 0.013+4.17
VB Coding Pathogenicity0.606 ± 0.0010.609 ± 0.004+0.50
VB Expression Effect0.671 ± 0.0000.691 ± 0.002+2.98
VB Common vs. Rare0.503 ± 0.0010.503 ± 0.0010.00
VB meQTL0.608 ± 0.0020.643 ± 0.002+5.76
VB sQTL0.583 ± 0.0010.578 ± 0.002−0.86
LRB Causal eQTL0.703 ± 0.0020.704 ± 0.007+0.14

Zero-Shot Tasks

TaskCompute-matched controlJEPA-DNARelative gain (%)
BEND Disease Variant0.632 ± 0.0000.746 ± 0.039+18.04
BEND Expression Effect0.537 ± 0.0000.569 ± 0.005+5.96
BRCA10.502 ± 0.0000.515 ± 0.012+2.59
LOL-EVE Causal eQTL0.523 ± 0.0000.537 ± 0.007+2.68
LRB Pathogenic OMIM0.502 ± 0.0010.522 ± 0.038+3.98
Songlab ClinVar0.516 ± 0.0000.518 ± 0.010+0.39
TraitGym Complex0.514 ± 0.0000.502 ± 0.002−2.33
TraitGym Mendelian0.525 ± 0.0010.514 ± 0.022−2.10

Response to Reviewer 9eSv - part 2

JEPA-DNA “functional grounding.” statement (Weakness 1 part 1):

We thank the reviewer for distinguishing the training objective from the biological interpretation of the learned representations. We agree that the JEPA objective does not directly supervise biological function or require recovery of individual masked-span representations. We have therefore revised the abstract from “JEPA-DNA forces models to predict the functional representations of masked genomic segments, shifting the learning signal from token recovery to semantic alignment” to “JEPA-DNA learns to predict the representation of an unmasked sequence from its corrupted counterpart, extending the learning signal beyond token recovery toward sequence-level semantic alignment.”

In addition, following the suggestion of Reviewer zrV5 to strengthen the evidence for functional relevance beyond downstream predictive performance, we have added a variant-induced embedding-shift analysis. The results are presented below and in Appendix Table 15.

JEPA-DNA increases the median embedding-shift ratio on seven of the eight datasets. Notable gains include ClinVar pathogenicity (+47.19%), HG38 missense versus synonymous variants (+11.09%), and phastCons conservation (+20.24%). We use datasets available through GFMBench-API covering pathogenic versus benign variants and other functional distinctions. We additionally use a dataset with precomputed VEP annotations for the missense-versus-synonymous comparison and the UCSC hg38 100-way phastCons track for the conserved-versus-non-conserved comparison.

Further implementation details are provided in the “Distribution-Shift Analysis” subsection of Appendix A.2 in the revised manuscript. Specifically, we measure the angular distance between the embeddings of each reference sequence and its corresponding variant sequence. For each class, we sample 1,000 reference–variant pairs, yielding 1,000 distances per class. We then compute the ratio of the median embedding shift in the functionally impactful class to that in the control class. A higher ratio indicates a larger embedding shift for functionally impactful variants relative to controls.

DatasetLabels (control / impact)Data sourceHF originalJEPA-DNAGain JEPA/Original (%)
bend_variant_effects_expressionno effect / expression effecBEND expression-variant benchmark1.2011.211+0.83%
bend_variant_effects_diseasebenign / pathogenicClinVar1.8842.773+47.19%
traitgym_complexmatched control / putative causalUK Biobank fine-mapping1.0400.994-4.42%
traitgym_mendelianmatched control / causaOMIM causal variants1.1391.266+11.15%
brca1functional or intermediate / loss-of-functionFindlay et al. saturation genome editing1.0041.080+7.57%
lrb_variant_effect_pathogenic_omimbenign / pathogenicInstaDeepAI Genomics Long-Range Benchmark1.0991.378+25.39%
Songlab HG38 consequencesynonymous / missensesonglab/hg38-variant-consequences0.9291.032+11.09%
phastCons Conservationnon-conserved / conservedUCSC hg38 100-way phastCons track1.1561.390+20.24%

Length sweep stability test (Weakness 1 part 2) :

We agree that the sequence-length sweep evaluates stability across input lengths rather than directly demonstrating long-range reasoning. We therefore interpreted it specifically as a stability test, but also updated the “Limitations and Future Work” section in the revised paper to explicitly identify the evaluation of distal sequence dependencies and long-range reasoning as a direction for future work.

Latency and consistency clarifications (Weakness 3):

We thank the reviewer for identifying this ambiguity. The previously reported GPU-hour values were extracted from end-to-end training logs including variable non-training and shared-system overheads. Since these measurements can be confusing, we have removed them from Table 4 in the Appendix, which now reports controlled measurements only.

Importantly, for the purpose of compute-matched experiments, runtime was strictly determined using controlled measurements of GPU time per optimizer step under matched hardware, sequence length, and effective batch size with no connection to these noisy end-to-end runtimes. The number of control updates was then scaled according to the measured per-step ratio.

Regarding consistency clarifications, we have updated the abstract to clarify that the reported gains occur on the majority of linear-probing and zero-shot tasks, rather than uniformly across all tasks.

Reviewer_9eSvOpenReview ↗

Response to rebuttal -1

Thank you for the detailed response. The validation-tuned probing analysis and independent continual-pretraining runs address my main concerns about probe fairness and variance. The results show that the overall trend persists, while also presenting the task-level regressions appropriately.

I also appreciate the revised wording around functional grounding and long-range reasoning. The embedding-shift analysis provides useful additional evidence for functional sensitivity, although I agree with the more limited interpretation now adopted in the manuscript. Removing the ambiguous end-to-end GPU-hour values and clarifying the controlled compute-matching procedure resolves the timing inconsistency I identified.

Overall, the rebuttal and revised claims substantially increase my confidence in the work. I have no further questions and will take these clarifications into account in my final assessment.

Thank you for your thoughtful follow-up and for confirming that our validation-tuned probing analysis, independent continual-pretraining runs, calibrated framing, and clarified compute-matching procedure addressed all of your questions. We greatly appreciate your constructive feedback throughout the review process and your increased confidence in our work.

Reviewer_zrV5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a continual training framework for genomic foundation models, JEPA-DNA, which incorporates token-level genomic language modeling with JEPA-style latent-space prediction to help learning abstract, functional features invariant to high-dimensional sequence noise. Across fair benchmark settings and diverse genomic downstream tasks, JEPA-DNA shows supurior performance and adaptability to different architectures.

Critical strengths

I enjoy the organization and presentation of this work. The motivation is quite clear and well argued: token-level reconstruction over-emphasizes local genomic structures, and hence suffers from noises in high-resolution genomic sequences. Incorporating JEPA-style latent prediction is a natural way to extract high-level semantic representations for sequence-level function prediction.

The method is cleanly described and easy to follow. The framework is also model-agnostic across diverse training objectives (MLM/NTP), aggregation schemes (CLS/mean/last-token), and architectures.

The authors also conducted fair and thorough benchmark and ablation experiments to demonstrate JEPA-DNA's strength, including strict data parity with each backbone’s original corpus, factorial ablations, contrastive comparison with controlled variants, and multi-seed probing.

Critical weaknesses

The methodological novelty is relatively modest, since the JEPA-DNA framework is close to LLM-JEPA transferred to DNA sequences. However, I believe that transferring a mature technology from LLMs to genomic modeling with necessary engineering adaptations is, in itself, a valuable work.

I understand keeping the original pre-training corpora in continual training is a strength for attribution, but leaves the question whether the JEPA objective continues to benefit from additional or novel data, or whether its advantage is specific to re-exposure to seen data. A data-scaling curve (e.g., 25/50/100% of the continual corpus, and one setting with new data) would substantially strengthen the claim that JEPA-DNA is a general representational uplift rather than a fixed-budget regularizer.

Also, the central claim that latent prediction separates functional signal from sequence noise seemingly has not been tested directly. The paper does not examine how genomic variants change the global latent embedding: whether pathogenic/benign, missense/synonymous, or conserved/non-conserved variants induce systematically different displacements, or whether displacement direction aligns with the functional clusters in Fig. 2. The zero-shot results, which are the closest proxy (ref/variant cosine distance), mostly remain near 0.5, which if anything suggests the latent space does not yet distinguish important from unimportant variation well.

Besides, for variance/covariance regularization, it seems the authors choose to regularize a proxy (h_glob and hat{v}_glob come from a separate clean, unmasked forward pass) rather than the JEPA branch quantities hat{z}_glob, z_glob themselves. Could the authors explain the reason for doing this? Also, given the JEPA loss used is cosine (scale-invariant) instead of MSE in original VICReg, it seems to be possible that the model inflates the norm to satisfy the variance constraits without increasing directional diversity.

Questions for authors

Could you report embedding-shift distributions for pathogenic vs. benign, missense vs. synonymous variants (e.g., using ClinVar), for the original checkpoint and JEPA-DNA?

Could you report at which step did NTv3 and HyenaDNA trigger early stopping, and are the step-matched baselines aligned to that step?

4 comments

Response to Reviewer zrV5 - part 1

We thank Reviewer zrV5 for their positive feedback and valuable comments. We have revised the manuscript accordingly (PDF file uploaded) and addressed each question and concern raised below.

Embedding-shift distributions (Question 1 + Weakness 3):

We thank the reviewer for their suggestion and agree that an embedding-shift analysis can strengthen the evidence that latent prediction separates functional signal from sequence noise. We have therefore added this analysis to the revised manuscript. The results are presented below and in Appendix Table 15.

JEPA-DNA increases the median embedding-shift ratio on seven of the eight datasets. Notable gains include ClinVar pathogenicity (+47.19%), HG38 missense versus synonymous variants (+11.09%), and phastCons conservation (+20.24%). We use datasets available through GFMBench-API covering pathogenic versus benign variants and other functional distinctions. We additionally use a dataset with precomputed VEP annotations for the missense-versus-synonymous comparison and the UCSC hg38 100-way phastCons track for the conserved-versus-non-conserved comparison.

Further implementation details are provided in the “Distribution-Shift Analysis” subsection of Appendix A.2 in the revised manuscript. Specifically, we measure the angular distance between the embeddings of each reference sequence and its corresponding variant sequence. For each class, we sample 1,000 reference–variant pairs, yielding 1,000 distances per class. We then compute the ratio of the median embedding shift in the functionally impactful class to that in the control class. A higher ratio indicates a larger embedding shift for functionally impactful variants relative to controls.

DatasetLabels (control / impact)Data sourceHF originalJEPA-DNAGain JEPA/Original (%)
bend_variant_effects_expressionno effect / expression effecBEND expression-variant benchmark1.2011.211+0.83%
bend_variant_effects_diseasebenign / pathogenicClinVar1.8842.773+47.19%
traitgym_complexmatched control / putative causalUK Biobank fine-mapping1.0400.994-4.42%
traitgym_mendelianmatched control / causaOMIM causal variants1.1391.266+11.15%
brca1functional or intermediate / loss-of-functionFindlay et al. saturation genome editing1.0041.080+7.57%
lrb_variant_effect_pathogenic_omimbenign / pathogenicInstaDeepAI Genomics Long-Range Benchmark1.0991.378+25.39%
Songlab HG38 consequencesynonymous / missensesonglab/hg38-variant-consequences0.9291.032+11.09%
phastCons Conservationnon-conserved / conservedUCSC hg38 100-way phastCons track1.1561.390+20.24%

NTv3 and HyenaDNA early stopping (Question 2):

NTv3 early stopping occurred after 32,500 steps (4,160,000 samples), and for HyenaDNA after 45,000 steps (5,760,000 samples). Step-matched baselines aligned to these step-counts. We added these details to the revised paper in “Training & Implementation Details.” in section A.3 under the Appendix.

LLM-JEPA (Weakness 1):

We thank the reviewer for recognizing JEPA-DNA as “a valuable work”. We take this opportunity to clarify the architectural and conceptual differences between JEPA-DNA and LLM-JEPA. While both JEPA-DNA and LLM-JEPA apply JEPA principles to sequence-based inputs, they differ substantially in the source of their supervision. LLM-JEPA relies on externally supplied pairs of corresponding views, such as natural-language descriptions and code, for its JEPA objective. In contrast, JEPA-DNA is fully self-supervised: its context and target views are generated directly from the same DNA sequence, without externally supplied pairings or manual annotations. This distinction is important: JEPA-DNA enables JEPA training directly on unlabeled DNA sequences rather than requiring a dataset of semantically aligned examples. To the best of our knowledge, JEPA-DNA is the first to apply JEPA to sequence inputs in a fully self-supervised setting, identifying suitable global sequence representations for this purpose.

Response to Reviewer zrV5 - part 2

Exposure to unseen data (Weakness 2):

A central aim of our experiments is to isolate the contribution of the JEPA objective and its components. To this end, we retain the original model backbones, their native sequence-aggregation methods, and their original pre-training corpora, and compare against both the original checkpoints and step-matched and compute-matched continual-training baselines. This controlled design allows us to attribute improvements to the proposed training framework rather than to changes in data, architecture, or computational budget. We agree, however, that these experiments do not establish how the benefits of JEPA-DNA scale with the amount of continual-training data or generalize to previously unseen pre-training data. We plan to investigate varying corpus sizes and additional pre-training data, applying the same data conditions to JEPA-DNA and its matched baselines. Given the limited time available during the rebuttal period, we have identified these experiments as directions for future work in the revised manuscript.

Separate VicReg branch and directional diversity (Weakness 4):

We apply variance–covariance regularization to representations obtained from a separate, clean, unmasked forward pass to encourage diversity arising from differences between input sequences, rather than from stochastic perturbations introduced during training. In the main JEPA branch, stochastic masking and dropout can introduce variability in the representations independently of differences between the underlying sequences. Applying the regularizer directly to these representations could therefore allow the variance constraint to be satisfied partly through stochastic variability, without ensuring that distinct sequences produce distinct representations at inference time, when these perturbations are absent. We therefore apply the variance–covariance regularizer to the clean-path representations, without stochastic masking or dropout. This design directly targets the diversity of the representations used at inference and reduces the risk that training-time stochasticity obscures representation collapse.

We agree that the original VICReg formulation uses MSE for its invariance term. In our formulation, alignment is instead provided by the cosine-based JEPA prediction objective, consistent with GeneJEPA. The variance and covariance regularizers retain the VICReg-style formulation: a hinge penalty on per-dimension standard deviation and a squared off-diagonal covariance penalty, respectively. Thus, cosine distance is not used for either regularization term. Moreover, the variance hinge stops decreasing once its target standard deviation is reached, so further norm inflation does not improve that term. Nevertheless, these regularizers do not mathematically guarantee directional diversity. To address this concern empirically, we conducted a post-training analysis. using 1,000 randomly sampled DNA sequences. For each checkpoint, we computed the cosine similarities between the sequence embeddings for all 499,500 unique pairs. We evaluated the original DNABERT-2 checkpoint and JEPA-DNA checkpoints at 25,000, 50,000, 75,000, and 100,000 continual-training steps, with the final checkpoint corresponding to our reported model.

CheckpointMean pairwise cosine similarity
Original DNABERT-2 (step 0)0.4010
JEPA-DNA, step 25,0000.1651
JEPA-DNA, step 50,0000.1608
JEPA-DNA, step 75,0000.1571
JEPA-DNA, step 100,0000.1542

The mean pairwise cosine similarity decreases throughout training, indicating greater average angular separation between embeddings. Since cosine similarity is invariant to embedding magnitude, this decrease cannot be explained by norm inflation alone. These results therefore provide empirical evidence that JEPA-DNA increases directional diversity rather than merely increasing embedding norms. We added these conclusions to the revised paper in A.2 under the Appendix (“Directional Diversity During Continual Training”).

Thank you for your detailed response. In particular, the embedding shift results indicate that the introduction of JEPA-style latent space prediction helps to cature the functional signals: The model’s ability to distinguish between nonsense mutations and significant mutations is significantly enhanced compared to the original HF. Most of my concerns have been addressed, although the model’s scalability with respect to data has not been demonstrated (which could be crucial for a continual pretraining method). I will hence maintain my score as 3.

Thank you for your positive feedback and for recognizing how our embedding-shift analyses demonstrate JEPA-DNA’s enhanced sensitivity to functional mutations (such as distinguishing nonsense from significant variants).

Regarding data scalability and pretraining exposure: we agree that systematically disentangling the effects of continual-pretraining data volume, species composition, and overlap with base-model pretraining data is an important direction for scaling genomic foundation models. While our current study addresses generalization across data distributions by demonstrating consistent gains across diverse benchmarks spanning multiple species and regulatory/pathogenic tasks, we have expanded our Discussion/Future Work section (Section 5, in the updated revision) to outline a concrete experimental framework for scaling continual pretraining for future work, specifically curating held-out taxonomic clades and controlled corpus-size ablations to isolate data-scaling laws from prior pretraining exposure. Thank you again for your valuable insights and support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.