- Rating
- 2
- Confidence
- 3
- Informativeness
- 1(−1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper adds a JEPA branch to the continual pre-training of existing genomic foundation models (GFMs). JEPA means the model predicts the embedding of masked spans, not the masked tokens. Here a predictor maps the masked-input embedding to the global embedding of the unmasked sequence, computed by an EMA copy of the encoder. The native MLM or next-token loss is kept. VICReg variance and covariance penalties prevent collapse, and a schedule grows the masked spans over training. The recipe is applied to DNABERT-2, NTv3 and HyenaDNA, using subsets of each model's own pre-training data.
Evaluation:
- Frozen linear probe (encoder fixed, only a linear head trained) on 9 tasks: mean gains of +2.60%, +2.19% and +3.29% relative AUROC over the original checkpoints (Table 1). Native continual-training controls at matched steps and compute gain much less.
- Zero-shot variant scoring by cosine distance between reference and variant embeddings (Tables 2, 9).
- Factorial ablation on DNABERT-2 (Table 3).
Critical strengths
A cheap add-on that transfers across very different architectures, with careful controls. The same recipe is applied to a Transformer with MLM, a hybrid U-Net/Transformer, and a long-convolution model with next-token prediction. The paper includes step and compute matched native baselines, keeps data parity, and runs a factorial ablation. In the ablation, the latent objective adds about +3.0% on top of MLM + VICReg + masking (+3.24% to +6.22%, Table 3). Costs (Table 4) and regressions (e.g., HyenaDNA TF binding 8.60%) are reported openly. Improving off-the-shelf GFM embeddings cheaply is useful.
Critical weaknesses
-
The probe may reward the readout's geometry rather than information. The probe reads one pooled vector per sequence: the CLS token for DNABERT-2 (Table 1), which MLM pretraining does not train to summarize the sequence. The method acts on this kind of vector directly. The VICReg variance and covariance terms are applied to the global vector h_glob (p.5), and the variance term carries the largest loss weight (25, p.23). The latent objective also predicts it. The probe is also brief and fixed across models: 3 epochs of AdamW at lr 3e-5 (p.24), with no feature standardization or per-model tuning described. It often does not fit the original checkpoints. Splice MCC is 0.000, −0.004 and −0.012 for the three originals (Table 5), and the DNABERT-2 probe reaches only 0.653 splice AUROC, versus 0.974 under full fine-tuning (Tables 1, 15). Under such a probe, a vector that is better spread and decorrelated can score higher without containing more information. The CLS concern does not apply to NTv3, which is read by mean pooling and still gains +2.19%, but the probe-conditioning concern does. The ablation is consistent with this partly happening: VICReg without latent prediction already gives +3.24% (Table 3). With a converged probe and a readout that does not depend on the CLS token, the gains may hold, shrink, or disappear. This determines my rating.
-
For two of three backbones, the native controls may not be informative. For NTv3, the step- and compute-matched rows equal the original checkpoint on 6 of 9 tasks and lie within 0.001 on the rest (Table 1). The HyenaDNA controls move by at most 0.013. The controls' optimizer and learning rate are not reported. JEPA-DNA itself uses SGD with very small learning rates (NTv3 phase-2 lr 1e-6 to 1e-8, p.23). If the controls share this regime, their weights may hardly change. In that case, "not a by-product of additional optimization" is shown only for DNABERT-2. Even there, on some tasks the controls account for most of the gain over the original checkpoint. On GUE Promoter, compute-matched native training reaches 0.927 vs JEPA-DNA's 0.928, from 0.884; on LRB causal eQTL it reaches 0.705 vs 0.711, from 0.697 (Table 1). So the per-task gains over the original checkpoint overstate what is specific to the JEPA objective.
Questions for authors
Requests that could change my assessment:
- To test whether JEPA-DNA adds information, rather than changing the geometry or readout of the probed vector, can you compare the original checkpoint, the compute-matched native control and JEPA-DNA under two converged frozen probes on standardized features, with hyperparameters tuned on validation? First, a regularized linear probe in which each model uses its best layer and pooling (native readout, mean or max), selected on validation. Second, an attentive probe: a learned query that attends over the token embeddings of the selected layer, with learned key/value projections, followed by a linear classifier. This is the frozen protocol under which V-JEPA reports its feature-prediction gains (Bardes et al., 2024). Please report the paired difference against the compute-matched control, with 95% paired-bootstrap confidence intervals per task and for the mean across tasks. Neither probe needs additional pretraining. Could you also confirm whether the probed vector is the same h_glob that the VICReg and latent losses act on (CLS for DNABERT-2, last token for HyenaDNA, mean pool for NTv3)?
Minor Asks
- In Table 15, fully fine-tuning the original DNABERT-2 on LRB causal eQTL gives 0.667, below its frozen-probe score (0.697, Table 1). This suggests the fine-tuning recipe may be unstable for the baseline, so the +8.51% gain should be interpreted with caution. A comparison against the compute-matched control would be more informative.
- Report gains against the compute-matched native control, not only the original checkpoint. On DNABERT-2, the control accounts for most of the gain on some tasks: GUE Promoter 0.927 vs JEPA-DNA's 0.928 (from 0.884), and LRB causal eQTL 0.705 vs 0.711 (from 0.697) (Table 1). Please also state the controls' optimizer and learning-rate schedule; the paper says only that they "differ only in budget" (p.6).
- Present the cosine-distance zero-shot results as exploratory. Cosine distance on pooled embeddings is sensitive to the same geometry changes. Zero-shot variant scoring by the likelihood ratio from the retained MLM/NTP head (as in GPN, Benegas et al., PNAS 2023; Evo 2, Brixi et al., Nature 2026) would give evidence independent of the pooled embedding.
- The I-JEPA evaluation (Assran et al., CVPR 2023) used a tuned linear protocol (learning-rate and weight-decay sweeps, optional batch normalization before the linear head, best of two readouts). Adopting a comparable protocol would make the comparison more convincing.
- Calibrate the claims about biology and range. The current evidence supports better-conditioned, more linearly accessible representations, which is useful in itself. It does not yet directly show "biologically relevant features" or "high-level biological context" (p.2, p.9). "Long-range interactions that govern genomic function" (p.1) is also not supported. Training windows are 512 tokens (≈2.5 kb) for DNABERT-2 and 8,192 bp for NTv3 and HyenaDNA, evaluation inputs are truncated to the same lengths (p.24), and Table 16 is presented as a failure-mode analysis "rather than as evidence of long-range reasoning." If the gains survive Question 1, the former framing is well supported. The latter claims would need evidence that does not read the pooled embedding and that spans beyond the training window.
5 comments
Response to Reviewer xhCT - part 2
Native controls may not be informative (Weakness 2):
We thank the reviewer for the opportunity to clarify this point. JEPA-DNA itself includes the same native token-prediction objective (e.g. MLM for DNABERT-2). To isolate the contribution of the JEPA components, the native controls retained this objective and the same optimization settings and training budget, while disabling only the JEPA-specific components.
Although the native controls showed limited changes on some downstream tasks, their self-supervised training was still impactful. For DNABERT-2, the step-matched native-MLM checkpoint had a global relative parameter drift of 4.54% from the original checkpoint, with a maximum tensor-wise drift of 22.4%. More importantly, on the same 1,000 sequences and fixed masks, perplexity decreased from 349.2 to 225.9 with 15% masking and from 474.3 to 291.4 with 30% masking (reductions of 35.3% and 38.6%, respectively). Thus, the native objective meaningfully updated the model even when this did not consistently improve every downstream task.
Compute-matched Full Fine-tuning (Minor ask 1):
We thank the reviewer for this suggestion, and we have added the compute-matched control to the full fine-tuning comparison in Table 17 of the revised manuscript. On LRB causal eQTL, this control achieves 0.6620 AUROC, compared with 0.7237 for JEPA-DNA, corresponding to a 9.32% relative improvement.
Gains and controls optimization details (Minor ask 2):
We thank the reviewer for highlighting the importance of the compute-matched comparison. The “Gain” columns in Tables 1 and 2 report relative AUROC improvements against the original checkpoint, step-matched control, and compute-matched control, respectively. We have additionally specified the controls’ optimizer and learning-rate schedules in the “Training & Implementation Details” subsection of Appendix A.3 in the revised manuscript.
Zero-shot evaluation (Minor ask 3):
We thank the reviewer for this observation. Our embedding-based zero-shot evaluation is intended to assess the global sequence representations that JEPA-DNA explicitly optimizes, which are central to this work. We therefore evaluate whether distances between these representations reflect functional differences without task-specific training, rather than evaluating the retained token-prediction head (as in Evo2 for example). To investigate whether the changes reflect differential sensitivity to functional effects rather than simply a general increase in embedding distances, we added a distribution-shift analysis in Appendix A.2, Table 15. For identical variants evaluated with the DNABERT-2 baseline and JEPA-DNA, we calculate the ratio of the median reference–variant cosine distance for putatively impactful variants to that for control variants. This ratio increases on seven of eight datasets, including from 1.884 to 2.773 for ClinVar pathogenicity, 1.099 to 1.378 for LRB pathogenic OMIM, and 1.156 to 1.390 for conserved versus non-conserved positions. These results suggest that the learned geometry exhibits greater relative sensitivity to biologically relevant variation, rather than merely increasing all distances uniformly.
Biology and range claims (Minor ask 5):
We thank the reviewer for this clarification. We have revised the manuscript to distinguish the demonstrated improvements in representation utility from claims about biological understanding and long-range reasoning. Specifically:
- Page 1: We replaced “capture the high-level regulatory logic and long-range interactions that govern genomic function” with “capture sequence-level information in global representations.”
- Page 2: We replaced “learns more linearly-separable and biologically relevant features” with “improve the linear accessibility of information relevant to genomic prediction tasks.”
- Page 9: We replaced “JEPA-DNA captures high-level biological context more effectively than generative objectives alone, consistently boosting backbone performance across supervised and zero-shot tasks” with “JEPA-DNA improves backbone performance on the majority of evaluated supervised and zero-shot tasks.”
Thank you for the revision and the additional analyses. The repeated continual-pretraining runs, the tuned probes with paired bootstrap intervals, and the calibrated claims strengthen the paper. My two main points stand as follows.
| Point | Status |
|---|---|
| W1: added information vs. readout geometry | Not resolved. With validation-selected layer and pooling, the mean gain over the compute-matched control is not significant (CI [−0.0005, +0.0045]); with mean pooling, JEPA-DNA is better on 5 of 9 tasks. Pairwise cosine similarity falls from 0.40 to 0.15, confirming a large change in the optimized vector's geometry. |
| W2: native controls for NTv3 and HyenaDNA | Not resolved. Evidence that the controls meaningfully trained is shown for DNABERT-2 only. |
The revised claim, improved linear accessibility of the designated global vector, is supported on DNABERT-2. My remaining concerns:
1. The gain should exceed simple baselines. Two comparisons against the compute-matched control, with paired bootstrap intervals, would show what JEPA adds over the same compute: (i) both models with the same post-hoc whitening (e.g., PCA or BERT-whitening), which controls for the geometry change; (ii) an attentive probe over the token embeddings, excluding the CLS token, as in V-JEPA's frozen evaluation (Bardes et al., 2024), which shows whether the gain reflects better token representations or information already present being routed into the optimized vector.
2. The gain should be substantial. Under validation-selected readouts the mean advantage is at most +0.0045 AUROC, and across the repeated runs clear gains appear on only four of nine tasks.
3. There is no evidence that the model learns functional or biological features. The supervised gains are measured on the optimized vector and do not hold on average under validation-selected readouts. The zero-shot evidence (variant-induced shifts, ClinVar separation) uses the same pooled vector, whose geometry changes substantially. As the response acknowledges, the objective does not directly supervise biological function. Yet the paper motivates JEPA as a way to learn abstract, functional features. Without evidence that the model itself learns them, better accessibility of a single vector does not require continual pretraining if a better readout or whitening provides it.
4. The paper should report the unfavorable results and align its claims. The validation-selected readout and mean-pooling results appear only in this discussion, while the appendix reports only the tuned probe on the optimized vector. The title ("Grounding") and parts of the text still describe functional grounding.
JEPA-style latent prediction with VICReg is established (I-JEPA, V-JEPA, LLM-JEPA, data2vec), and the demonstrated benefit here is a small, readout-specific gain in linear accessibility. I keep my rating at weak rejection. I lowered Informativeness to 1 because the revision withdraws the claim that the model learns functional or biological features. The remaining demonstrated contribution, improved linear accessibility of the optimized vector, is a known effect of JEPA- and VICReg-style objectives.
Response to Reviewer xhCT - part 1
We thank Reviewer xhCT for their feedback and valuable comments. We have revised the manuscript accordingly and address each question and concern raised below.
Hyperparameter-Tuned Linear Probing Analysis (Question 1 + Weakness 1 + Minor ask 4):
We confirm that the probed vector is the designated global representation, h_glob, used by the VICReg constraints: the final-layer CLS embedding for DNABERT-2, last-token embedding for HyenaDNA, and mean-pooled embedding for NTv3.
As also described in our response to reviewer 9eSv, we have added a “Hyperparameter-Tuned Linear Probing” analysis in Appendix A.2 of the revised paper, with results summarized in Figure 2, to test whether our conclusions persist under independently tuned probes.
For each task and model in the DNABERT-2 comparison, we independently optimize the learning rate and weight decay using Optuna’s Bayesian TPE sampler, with 10 trials over log-uniform ranges of 10⁻⁵–10⁻² and 10⁻⁸–10⁻¹, respectively. We increase probe training from 3 to 10 epochs and select both the hyperparameter configuration and the best epoch by validation AUROC. We use the predefined validation splits for GUE tasks and hold out 10% of the training data for the remaining tasks. The held-out test set is evaluated only after selection.
We report full-test performance together with bootstrap medians and percentile 95% confidence intervals, computed from 1,000 paired bootstrap resamples using identical test-example indices for JEPA-DNA and its compute-matched control. The overall ranking trend persists under this validation-selected protocol: JEPA-DNA outperforms the compute-matched control on 6 of 9 supervised tasks, with four improvements remaining statistically significant at the 95% confidence level.
Regarding layer selection and aggregation, JEPA-DNA explicitly supervises the designated sequence-level representation to increase the information accessible through that representation. Our main evaluation therefore uses the same final-layer representation and aggregation operator during continual pretraining and downstream probing. Across backbones, these operators include CLS aggregation for DNABERT-2 (as in the original paper), mean pooling for NTv3, and last-token aggregation for HyenaDNA.
V-JEPA predicts continuous representations of masked spatiotemporal patches, whereas JEPA-DNA supervises a designated global sequence representation. This distinction motivates our primary evaluation of that representation, while alternative readouts test whether the benefits extend beyond it. Notably, V-JEPA 2.1 also introduces direct supervision at multiple intermediate encoder layers during pretraining, rather than using those layers only at evaluation. Exploring analogous intermediate-layer supervision in JEPA-DNA is a direction for future work.
Nevertheless, to address this question directly, we additionally evaluated the reported DNABERT-2 checkpoints with the layer and pooling operator selected independently for each model and task by validation performance, without additional pretraining. Under this protocol, JEPA-DNA significantly outperforms its compute-matched control on three tasks and significantly underperforms on one task. The mean paired AUROC difference across tasks is positive (towards JEPA-DNA), but its 95% paired-bootstrap confidence interval includes zero: [−0.00053, +0.00445]. Thus, some task-specific gains persist under validation-selected readouts, but the average advantage is not statistically significant under this protocol.
This indicates that some gains transfer to mean pooling and layers extractions, but the advantage is less consistent than when evaluating the last layer's CLS representation explicitly optimized during JEPA-DNA training. We have also updated the Future Work section of the revised paper to include evaluating JEPA-DNA with alternative pooling methods, and layers extraction during continual pre-training.