acceptodds
Under review as a conference paper at ICLR 2027

IsoAlign: Isoform-Aware DNA-Protein Alignment via Joint-Embedding Predictive Architectures

Abstract

Genomic foundation models (GFMs) have emerged as a powerful paradigm for learning DNA sequence representations, yet existing self-supervised pretraining strategies overlook alternative splicing and underperform compared to specialized models. To address this gap, we introduce IsoAlign, a multimodal framework that grounds genomic pretraining in isoform-aware DNA-protein alignment to learn biologically meaningful representations. IsoAlign aligns transcripts to their corresponding protein isoforms, capturing both coding sequences and broader regulatory contexts, using two Joint-Embedding Predictive Architectures (JEPA): DNA-Protein JEPA (DP-JEPA), which predicts protein representations from DNA using contrastive alignment, and Isoform-Masked JEPA (IM-JEPA), which models relationships among alternative isoforms of the same gene through masked protein prediction. We evaluate the learned DNA representations of these architectures across a multi-metric evaluation suite spanning tasks, including the prediction of pathogenicity, fitness, gene expression, and TF binding. Our results demonstrate that isoform-aware cross-modal alignment using JEPA significantly improves functional representations and downstream performance, establishing a promising direction for GFM pretraining.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
genomic foundation models, joint embedding predictive architecture, multimodal learning, alternative splicing, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1GPcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IsoAlign continues DNA-model pretraining with paired transcripts and protein isoforms. DP-JEPA aligns DNA-derived predictions with frozen ESM2 protein representations. IM-JEPA uses visible isoforms from the same gene to predict masked isoform representations. Evaluation covers mutational effects, splice-related variants, TF binding, tissue-specific expression, and DNA–protein retrieval. Parent-model and continued-pretraining comparisons show gains on several tasks, with DP-JEPA generally more consistent than IM-JEPA. Scalar variant distances are evaluated directly by AUPRC; vector probes use chromosome-held-out logistic regression. TFLink/GTEx prediction uses both DNA and protein features. The results support protein-informed pretraining, but do not yet isolate the isoform-specific contribution or establish regulatory understanding.

Critical strengths
  1. Protein-guided pretraining produces meaningful gains on some tasks. For example, NTv2 with DP-JEPA improves MFASS AUPRC under the L2 score from 0.04 to 0.17 (Table 1). Architecture-matched continued-pretraining controls help distinguish this benefit from simply training the DNA encoder longer.

  2. Several downstream splits account for dependence between examples. DMS examples are grouped by amino-acid mutation, and GTEx is split by gene. The combination of unsupervised variant scores and supervised tasks also gives a broader view of representation utility than retrieval alone.

Critical weaknesses
  1. The main missing comparison is protein-guided training with one canonical isoform per gene. Without it, the gains could come from protein alignment rather than alternative-isoform information. The “Canonical CDS” ablation changes the input format and does not answer this training-data question. This leaves the central isoform-aware contribution insufficiently supported.

  2. Pretraining and model-selection boundaries need clearer reporting. The final G2P gene/isoform counts, split unit, and downstream gene or family overlap are not sufficiently documented. Appendix E calls the retrieval data a “held-out validation set” without clarifying whether it differs from the early-stopping set. Section 3 reports the better of two alignment configurations without specifying the selection data. These gaps make generalization difficult to assess; they do not establish leakage.

  3. The sequence ablations do not justify the conclusion that IsoAlign learns “true regulatory syntax” (PDF p. 9). Each altered-input model is retrained. Successful retrieval from frameshifted sequences or UTRs could still rely on gene identity or homology. Figure 3 includes relevant isoform-swap and homolog-swap controls, but retraining on altered pairings does not test the original model's discrimination among same-gene isoforms. The evidence is stronger for sequence-dependent alignment than for regulatory understanding.

Questions for authors
  1. Is there a capacity- and budget-matched control trained with one canonical isoform per gene? Please distinguish this from the existing “Canonical CDS” input-format ablation.

  2. What are the final G2P gene/isoform counts, split units, and gene or protein-family overlaps with downstream benchmarks? Is retrieval evaluated separately from early stopping, and which data selected the better configuration in Section 3?

  3. For the original correctly paired model, how well does retrieval work when candidates are restricted to isoforms of the same gene? This could use existing retrieval scores and would differ from retraining with swapped pairings in Figure 3. It would help assess how much alignment goes beyond gene identification.

3 comments

We thank the reviewer for recognizing that our pretraining produces meaningful gains (e.g., MFASS L2 AUPRC 0.04→0.17), that architecture-matched controls separate this benefit from longer training, that our splits account for dependence between examples (GroupKFold for DMS, gene-level splits for GTEx), and that combining unsupervised variant scores with supervised tasks gives a broader view than retrieval alone.

[W1, Q1] Canonical-isoform-only control: The requested canonical-isoform-only training control is already included in our ablation analysis as “Canonical CDS” (Figure 3). We have clarified this in Section 5.4 of the manuscript. This configuration uses the same architecture, batch size, and training steps as DP-JEPA, but restricts training to one canonical isoform per gene instead of up to eight. Thus, the difference is in the training data rather than the input format. Multi-isoform training improves cross-gene MRR from 0.805 to 0.843 (+4.7%) and within-gene MRR from 0.823 to 0.884 (+7.4%). These results support the contribution of alternative-isoform information beyond protein-guided alignment alone.

[W2, Q2] Data splits and leakage across tasks: The G2P pretraining dataset contains 55,752 records covering 17,710 genes and 46,125 transcripts, split 80/10/10 at the gene level (14,168 / 1,771 / 1,771 genes). All isoforms of a gene stay in the same partition, and retrieval is evaluated only on held-out genes (Sec. 3, Appendix A.1). The partitions are not family-disjoint. However, the homolog-swap ablation (same protein family, different gene) reduces MRR from 0.843 to 0.326 (61.3% reduction), so retrieval does not rely only on protein-family identity (Sec. 5.4, Fig. 3).

Overlap with downstream benchmarks: DMS: TP53, CBS, PDE3A, and GDI1 appear in pretraining only as wild-type sequences, while evaluation uses unseen mutations (e.g., 7,448 TP53 variants). This follows standard VEP practice, where models are pretrained on reference sequences and evaluated on held-out variants (Riesselman et al., 2018; Frazer et al., 2021; Meier et al., 2021; Notin et al., 2023). Gene-level overlap is acknowledged in Sec. 3.

ClinVar and MFASS: We evaluate vector-based metrics using leave-one-chromosome-out (LOCO) cross-validation and scalar metrics using direct AUPRC (Sec. 5.2). Gene-level disjointness from G2P pretraining is not enforced, but evaluation variants are unseen during IsoAlign pretraining (Appendix A.3).

GTEx: We use an 80/10/10 gene-level split (15,671 / 1,958 / 1,958 genes), ensuring that transcripts from the same gene remain in the same partition (Sec. 5.3).

Checkpoint and configuration selection: We select checkpoints by validation loss and use retrieval metrics for monitoring only. This keeps model selection tied to the training objective instead of the retrieval benchmark we report. The “held-out validation set” in Appendix E is the validation split used for early stopping, and we have made this clear in the revised manuscript. We also chose the Section 3 configuration (architecture, embedding dimension, learning rate, temperature) by validation loss on the 1,771-gene validation split. Test genes were used only for a single final evaluation, after the configuration was fixed.

[W3, Q3] Sequence Ablation and Within-Gene Retrieval Performance: We agree that the retrained sequence ablations demonstrate which sequence features can support DNA–protein alignment, rather than establishing that the original model learns regulatory syntax. We have revised the Fig. 3 caption and Section 5.4 to state explicitly that these ablations are independently retrained models. To directly evaluate isoform discrimination without retraining, we restrict retrieval candidates to isoforms of the same gene using the original trained model and macro-average over genes. Within-gene retrieval achieves an MRR of 0.884 and R@1 of 78.3%, compared with 0.843 and 76.2% for cross-gene retrieval. Since gene identity is shared across all within-gene candidates, this evaluation provides evidence of isoform-level discrimination beyond gene identity alone. Furthermore, swapping protein isoforms within the same gene reduces within-gene MRR from 0.884 to 0.695 and R@1 from 78.3% to 46.8%, indicating sensitivity to the correct DNA–protein correspondence. We have revised the “true regulatory syntax” claim in the Conclusion to describe these findings more precisely as evidence of sequence-dependent alignment rather than demonstrated regulatory understanding.

[W4, Q4] Fixed scoring rule: We agree that a scoring rule fixed independently of downstream labels is important for practical deployment. We therefore summarize performance using cosine distance as a single fixed scoring rule across both datasets. Cosine distance provides a scale-invariant measure of representation changes between reference and alternate sequences, without requiring task-specific fitting or score selection. Under this rule, DP-JEPA (CLIP) improves NTv2-500M from 0.32 to 0.63 AUPRC on ClinVar Splicing and from 0.04 to 0.08 on MFASS. Similarly, for AlphaGenome, DP-JEPA (CLIP) improves performance from 0.61 to 0.63 on ClinVar and from 0.29 to 0.36 on MFASS. These results demonstrate improvements under a consistent scoring rule, without selecting the better metric separately for each dataset. We also acknowledge that performance depends on the scoring rule and model configuration, and have qualified our claims accordingly in Sec. 5.2.

[W5, Q5] We clarify the gradient flow and trainable components for each objective below:

ComponentDP-JEPAIM-JEPA
DNA encoderTrained (lr 1×10⁻⁵)Trained (lr 1×10⁻⁵)
Protein encoderTrained (lr 5×10⁻⁷)Frozen
TargetsStop-gradientDetached
Predictor4-layer Transformer, 8 heads, d=512Same
Projection head3-layer MLPNot used

In DP-JEPA, the protein pathway is optimized with a substantially lower learning rate than the DNA pathway (5e-7 vs. 1e-5), encouraging a relatively stable target representation, consistent with the asymmetric training strategy of VL-JEPA (Chen et al., 2026). We have clarified this in Sec. 4.1. The protein target representations are detached from the prediction loss, preventing gradients from this loss from updating the protein encoder or its projection head. We have corrected Eq. 4 (Sec. 4.2.1) in the revised manuscript to explicitly indicate this stop-gradient operation: zy=sg(y(pool(g(y)))). SigReg (Balestriero & LeCun, 2025) is used to discourage representation collapse (Sec. 4.2.2, Eq. 9). In IM-JEPA, the protein encoder is frozen, and masked isoform representations are predicted using an MSE objective Sec. 4.2.2, Eq. 8).

Thank you again for your thoughtful review. Following your suggestions, we have added within-gene retrieval on the original model and clarified that the canonical-only control differs only in its training data. We also documented the splits, the overlap with downstream benchmarks and model selection, and revised the “regulatory syntax” claim. As the discussion closes today, we wanted to check whether these responses address your concerns. If so, we would be grateful if you would consider updating your score.

Reviewer_cSt6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes IsoAlign, a cross-modal pretraining framework that aligns DNA representations with their corresponding protein isoforms to improve genomic representation learning. It introduces two architectures: DP-JEPA, which predicts protein representations from DNA sequences using contrastive alignment, and IM-JEPA, which models relationships among alternative isoforms through masked protein prediction. The framework is evaluated on several downstream tasks, including mutation fitness prediction, splicing variant effect prediction, transcription factor binding, and tissue-specific expression prediction. The results show improvements over several genomic foundation model baselines.

Critical strengths

The paper addresses a biologically meaningful problem by incorporating protein isoform information into genomic representation learning, offering a promising direction beyond conventional DNA-only pretraining. The proposed DP-JEPA and IM-JEPA architectures provide two complementary approaches to cross-modal alignment and modeling relationships among protein isoforms. The experimental evaluation covers a diverse range of biologically relevant downstream tasks and includes comparisons with multiple genomic and protein foundation models. Additionally, the architecture-matched baselines and sequence ablation experiments help investigate the effects of different pretraining objectives and sequence components.

Critical weaknesses
  1. The paper does not provide sufficient evidence that the model learns isoform-specific information. The current experiments do not sufficiently distinguish the benefits of protein-informed cross-modal pretraining from those of explicitly modeling multiple isoforms.
  2. The contribution of DP-JEPA relative to standard contrastive learning remains unclear. The architecture-matched baselines do not clearly establish whether the same Transformer Predictor, pooling operations, and gradient paths are used.
  3. The pretraining dataset statistics and data partitioning procedures are insufficiently documented. In particular, gene-level separation, sequence deduplication, and the independence of retrieval evaluation data from model selection remain unclear.
  4. The independent benefits of cross-modal pretraining in supervised tasks are not adequately isolated. The use of protein representations in the reported LoRA models is unclear, and strictly matched baselines controlling for protein inputs and fine-tuning are missing.
  5. Several methodological and reporting inconsistencies affect reproducibility, including the stop-gradient treatment of the protein projection head, incorrect compute totals in Table 9, inconsistent TFLink negative-sampling statistics, and ambiguous LOCO evaluation procedures.
  6. The experimental results do not fully support the claimed benefits of isoform-aware modeling. IM-JEPA substantially degrades AlphaGenome performance on splicing-related tasks, while the ablation results in Figure 3 provide limited evidence for learning isoform-specific regulatory information.
Questions for authors
  1. The authors should compare single-isoform training, independent multi-isoform alignment, and joint multi-isoform modeling under matched data and computational budgets.
  2. The authors should clarify whether the contrastive baselines use the same predictor, pooling strategy, and gradient paths as DP-JEPA, and provide controlled comparisons separating architectural improvements from those of the training objective.
  3. The authors should report detailed pretraining dataset statistics, gene-level splitting and deduplication procedures, and clarify whether model selection and retrieval evaluation use independent datasets.
  4. The authors should clarify whether protein representations are used in the final LoRA models and identify which IsoAlign architecture is reported in Tables 2 and 3. A matched baseline using the original DNA backbone, the same frozen ESM2 encoder, and identical fine-tuning settings should be included.
  5. The authors should clarify the gradient flow through the protein projection head, correct the compute totals in Table 9, reconcile the TFLink negative-sampling statistics, and specify how LOCO evaluation is applied to scalar VEP scores.
  6. The authors should investigate why IM-JEPA substantially degrades AlphaGenome performance and explain how the ablation results in Figure 3 support the claimed learning of isoform-specific regulatory information.
4 comments

We thank the reviewer for noting that IsoAlign addresses a biologically meaningful problem and offers a promising direction beyond DNA-only pretraining, with two complementary architectures (DP-JEPA, IM-JEPA), evaluation across diverse downstream tasks against multiple GFMs and PFMs, and architecture-matched baselines and sequence ablations.

[W1, Q1] Isoform Awareness: To isolate the contribution of multi-isoform training, we compare single-isoform (canonical-only) training against independent multi-isoform alignment (DP-JEPA), while keeping the architecture, training steps, batch size, and gene sets matched. The results are now included in Section 5.4 (Fig. 3) of the revised manuscript and summarized below:

ConfigurationCross-gene MRRWithin-gene MRR
Single-isoform (canonical_only)0.8050.823
Multi-isoform, independent (DP-JEPA)0.843 (+4.7%)0.884 (+7.4%)

The improvements in both cross-gene (+4.7%) and within-gene (+7.4%) retrieval indicate that incorporating multiple isoforms during training provides benefits beyond canonical-only alignment, even without explicit joint modeling in DP-JEPA. Importantly, within-gene retrieval directly evaluates discrimination among isoforms of the same gene, reducing reliance on gene-level identity. Furthermore, swapping isoforms within the same gene decreases within-gene MRR from 0.884 to 0.695 and R@1 from 78.3% to 46.8%, demonstrating sensitivity to the correct transcript–protein correspondence.

Joint modeling (IM-JEPA) further introduces an objective that predicts masked isoforms from DNA and sibling isoforms. While its downstream benefits are task-dependent, this objective explicitly models relationships among isoforms, providing a complementary approach to independent alignment.

[W2, Q2] JEPA model architectures: Both models use the same encoders (NTv2-500M, ESM2-150M), pooling, projection heads and protein pathway. The only difference is that DP-JEPA adds a 4-layer Transformer predictor on the DNA side (~4M parameters, <1% of the total). Following the JEPA framework (Chen et al. 2026), this predictor is deliberately introduced to predict protein representations from DNA embeddings in a shared latent space, rather than relying solely on direct alignment of the two modalities. Additionally, DP-JEPA uses a lower learning rate for the protein pathway than for the DNA pathway, encouraging a more stable prediction target, consistent with the asymmetric training strategies employed in JEPA-based approaches. Thus, these design choices are integral to our JEPA-inspired predictive formulation rather than independent architectural enhancements. We have added a citation to VL-JEPA (Chen et al., 2026) and clarified the motivation for the asymmetric architecture in Sec. 4.1 of the revised manuscript. The MLM and contrastive controls (Sec. 4.3) demonstrate the benefits of the proposed JEPA-inspired predictive formulation over conventional pretraining strategies, while maintaining shared encoder, pooling, and projection components.

[W3, Q3] Dataset statistics and evaluation protocol: The pretraining dataset contains 55,752 records covering 17,710 genes and 46,125 transcripts. We use an 80/10/10 gene-level split (14,168 / 1,771 / 1,771 genes) that keeps all isoforms of a gene in the same partition, so no gene appears in more than one partition (Sec. 3, Appendix A.1). Checkpoint selection and early stopping use validation loss only. Retrieval metrics are logged for monitoring and are not used for model selection, and the held-out test set is never used for training or selection (Appendix E.1). We do not deduplicate at the protein-family level, but the homolog-swap experiment (Figure 3, Section 5.4) reduces MRR by 61% (0.843 → 0.326), showing that the model relies on sequence-specific correspondence even among homologous proteins. We have added these statistics and the full split and model-selection protocol to Section 3 and Appendix A.1.

[W4, Q4] Supervised finetuning: Fine-tuning leverages DNA + frozen ESM2 embeddings concatenated before the head. Both apply LoRA (r=4, α=8) to the DNA encoder’s query/value projections as outlined in Appendix D.4. Table 2 includes DP-JEPA (CLIP) and IM-JEPA with NTv2, while Table 3 includes DP-JEPA (CLIP) with AlphaGenome and IM-JEPA with NTv2. We have clarified the configuration used in each row. The pretrained predictor is kept during fine-tuning. We added the matched baseline: NTv2-500M + frozen ESM2 with identical LoRA and no IsoAlign pretraining. On TFLink (Table 2), the baseline performs comparably to DP-JEPA, while IM-JEPA achieves modest improvements.

ModelAUROCAUPRC
NTv2 + ESM2 (no pretraining)0.935 ± 0.0010.928 ± 0.001
IsoAlign (NTv2, DP-JEPA)0.934 ± 0.0010.928 ± 0.001
IsoAlign (NTv2, IM-JEPA)0.936 ± 0.0010.930 ± 0.001

On GTEx (Table 3), IM-JEPA improves overall Spearman correlation from 0.650 to 0.668 and Pearson correlation from 0.647 to 0.667 over the matched baseline, supporting the benefit of isoform-aware pretraining for expression prediction.

ModelSpearman (ρ)Pearson (r)
AlphaGenome0.7170.732
NTv2 + ESM2 (no pretraining)0.6500.647
IsoAlign (NTv2, IM-JEPA)0.6680.667
IsoAlign (AlphaGenome, DP-JEPA)0.7250.738

[W5, Q5] Evaluation and compute statistics: We have revised the manuscript to clarify these points and correct the reported statistics. Specifically, the code detaches the full projected protein embedding, so neither the protein encoder nor its projection head receives gradient updates. We have corrected Eq. 4 to zy=sg(y(pool(g(y)))) to reflect this. The compute totals in Table 9 have been corrected to 105 runs, 841 wall-clock hours, and 3,368 GPU-hours. We have also revised Table 5 to clarify that the 100K TFLink negative samples consist of 35,380 within-family (35.4%) and 64,620 cross-family (64.6%) samples. Finally, we have clarified the LOCO evaluation protocol in Appendix B: vector-based scores ([CLS] differences and element-wise products) use leave-one-chromosome-out logistic regression, whereas scalar scores (cosine and L2 distances) are evaluated directly by AUPRC

[W6, Q6] IM-JEPA: We agree that IM-JEPA’s effect depends on the task and the DNA encoder, and we report this in Section 5.2. On NTv2, IM-JEPA improves every task we evaluate, raising DMS by +0.084 (comparable to DP-JEPA’s +0.086), splicing VEP ClinVar cosine-AUPRC from 0.32 to 0.35, and GTEx expression across all five tissues from 0.650 to 0.668 over the matched baseline. With AlphaGenome, IM-JEPA improves DMS but degrades VEP performance, as summarized below:

TaskAlphaGenome+ IM-JEPAΔ
DMS0.420.554+0.136
ClinVar splicing0.610.33−0.28
MFASS0.290.06−0.23

Several factors may explain these differences. First, predicting whole isoforms captures transcript-level structure, which helps protein fitness prediction but may not preserve the nucleotide-level detail that splice-variant scoring requires. This effect may be strongest for AlphaGenome, whose single-nucleotide resolution likely already encodes splice motifs, whereas NTv2’s 6-mer tokens have less such signal to lose. Training coverage also matters: the 46.2% of genes (8,178 / 17,710) with only one isoform are excluded from IM-JEPA training, and while four of the five DMS proteins are multi-isoform, ClinVar and MFASS include many single-isoform genes.

We have revised Figure 3 and Section 5.4 to analyze isoform-specific learning through cross-gene and within-gene retrieval ablations. IM-JEPA reaches a within-gene MRR of 0.88, compared with 0.82 for canonical-only training, so seeing multiple isoforms helps the model tell apart isoforms of the same gene. Swapping isoforms within a gene lowers MRR to 0.70, which shows the model depends on the correct transcript–protein pairing rather than gene identity alone. Retrieval also stays well above chance using only UTRs (MRR 0.75–0.78) or only splice junctions (0.55), suggesting the representations capture regulatory information beyond the coding sequence.

Thanks to the authors for providing further clarification. They have addressed some of my concerns. I have decided to keep my rating.

Thank you for your reply and for engaging with our responses. We are glad they addressed some of your concerns. To address your central question on isoform-specific learning, we added a matched canonical-only control and within-gene retrieval analyses. We also added a matched LoRA baseline, clarified the architectures, splits and model selection, and corrected the reporting issues you identified. Since the discussion closes today, we would welcome the chance to address any remaining concerns. If there is a specific point that still holds back your assessment, please let us know.

Reviewer_hvbiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

IsoAlign aligns a trainable DNA encoder (NTv2-500M or AlphaGenome) to a frozen ESM2-150M using transcript/protein isoform pairs from Broad G2P. Two objectives: DP-JEPA, a contrastive loss against stop-gradient ESM2 embeddings through a 4-layer predictor, and IM-JEPA, which jointly encodes all isoforms of a gene, masks half the protein slots and regresses them. Architecture-matched MLM and contrastive controls use the same backbone. Evaluation covers DMS fitness on 5 datasets, zero-shot VEP on ClinVar splice-region SNVs and MFASS, LoRA fine-tuning on TFLink and GTEx, and retrieval-MRR ablations.

Critical strengths
  1. Uniform scoring across 16 GFMs and 5 PFMs with explicit per-model handling (context windows, strand averaging, GroupKFold by amino-acid mutation to avoid codon-twin leakage), plus architecture-matched MLM and contrastive controls on the same backbone. This is the right way to isolate an objective, and the benchmark infrastructure has standalone value.
  2. DP-JEPA does lift the parent model on some zero-shot tasks: DMS median Spearman 0.46 to 0.557 for NTv2, +0.136 for IM-JEPA(AlphaGenome), ClinVar cosine-AUPRC 0.32 to 0.63.
Critical weaknesses
  1. Isoform-awareness is never tested, and the paper's evidence argues against it. IM-JEPA, the only isoform-specific component, is the weaker objective nearly everywhere: it drops AlphaGenome from 0.61/0.57 to 0.33/0.30 on ClinVar and 0.29/0.31 to 0.06/0.09 on MFASS. DP-JEPA is standard DNA-protein contrastive alignment; nothing in Eq. 5 distinguishes isoforms of one gene. No experiment tests isoform discrimination: no within-gene GTEx ranking, no gene-mean baseline, no canonical-isoform-only comparison. GTEx reports only overall correlation across 19,587 genes, which between-gene variance dominates. The isoform-swap ablation retains 79% of retrieval MRR when DNA is paired with the wrong isoform of the same gene, which reads as the model matching on gene identity, not the "subtle isoform-level differences" claimed in Section 5.4.
  2. The frame-shift ablation shows the opposite of what is claimed. Shifting the reading frame by 1 nt leaves the nucleotide sequence essentially unchanged while completely changing the protein. Retaining 0.78 MRR is therefore evidence that retrieval runs on sequence identity, not on codon dependencies or regulatory syntax. Codon shuffle, which preserves the protein and changes a third of nucleotides, hurts more (−19%), consistent with the same reading. No gene-level separation between pretraining and the retrieval validation set is reported, nor between pretraining and the evaluation genes (TP53, CBS, PDE3A, GDI1, ClinVar, GTEx).
  3. Gains are unattributable and metric-dependent. Tables 2 and 3 have no NTv2+ESM2 control fine-tuned with the same LoRA recipe but no IsoAlign pretraining, so 0.934 vs 0.892 AUROC cannot be credited to the method. The ClinVar headline holds for cosine only: under L2 the parent NTv2 is 0.73 and DP-JEPA drops to 0.67; under CLS/Mul the contrastive controls match or beat DP-JEPA. MFASS is at chance for nearly all models. Section 3 reports the best of 34 DP-JEPA and 48 contrastive runs across 3 losses and 4 metrics with no seeds or CIs. The obvious baseline, translate the CDS and embed with ESM2-150M, is absent, as are SpliceAI, Pangolin and Borzoi on the splicing tasks. No code release.
Questions for authors
  1. How is the G2P pretraining set split from the retrieval validation set and from the downstream evaluation genes? Is any split at gene or homology-family level?
  2. Please add NTv2-500M + frozen ESM2 with identical LoRA and no IsoAlign pretraining to Tables 2 and 3. Does AlphaGenome receive the TF protein sequence in Table 2? Which variant and how many seeds in Table 3?
  3. What is within-gene GTEx performance (Spearman across isoforms within each gene) against a gene-mean predictor and a canonical-isoform-only encoder?
  4. In the frame-shift ablation, is the nucleotide sequence identical apart from a 1 nt offset? If so, does 0.78 MRR not indicate sequence-identity matching? How do you read 79% retention under isoform swap as discrimination rather than its absence?
  5. Bootstrap CIs for Tables 1 and 6, seed variance for pretraining, and what validation signal selected the "best-performing" configurations. A translate-then-ESM2 baseline for DMS and coding ClinVar, and SpliceAI/Pangolin for the splicing tasks.
6 comments

We thank the reviewer for recognizing our rigorous benchmark of 16 GFMs and 5 PFMs with leakage-aware splits, our architecture-matched MLM and contrastive controls, the standalone value of the benchmark infrastructure, and DP-JEPA’s zero-shot gains over its parent model.

[W1, Q3, Q4] Isoform-awareness: We agree that our original phrasing overstated the isoform-awareness claim, and we have revised it in Section 1 so they reflect what the evidence supports. Although we cannot rule out that part of the retrieval signal is carried by gene-level cues, the following pieces of evidence support the contribution of isoform-aware training and that the model is not relying on gene identity alone:

  1. DP-JEPA samples in-batch negatives across all isoforms, covering both cross-gene pairs and 318,461 within-gene pairs (distinct isoforms of the same gene). These within-gene negatives force isoform-level discrimination, so the model does not minimize the loss using gene identity alone. IM-JEPA goes further and predicts masked isoforms from their visible siblings.

  2. In Section 5.4 (Fig. 3 and Appendix E), we compare against two controls (same architecture, training steps and batch size): a canonical-only model and an isoform-swap test. Relative to the canonical-only control, multi-isoform training raises within-gene MRR from 0.823 to 0.884 (+7.4%) and R@1 from 0.672 to 0.783 (+16.5%). Isoform swapping drops MRR to about 79% of baseline both across and within genes, and within-gene R@1 falls from 0.783 to 0.468 (−40%). A model relying only on gene identity would be unaffected by swapping in another isoform of the same gene, so this drop confirms that the model learns isoform-specific features.

  3. Within-gene GTEx evaluation. To assess isoform-specific expression variation beyond gene-level effects, we evaluated within-gene prediction performance on the eight genes containing more than three isoforms in our evaluation set. Given the limited number of eligible genes, we report complementary metrics rather than relying on a single measure:

ModelMean MSE ↓Mean-Gene Predictor MSE ↓Mean R² ↑Mean Spearman ρ ↑
Baseline0.13660.2308-0.64590.8125
Canonical-only0.09260.23080.08600.8125
DP-JEPA0.15410.23080.03200.9375
IM-JEPA0.16200.23080.11690.7500

DP-JEPA achieves the highest within-gene Spearman correlation (0.9375), suggesting that its learned representations preserve information relevant to relative isoform expression. In contrast, the canonical-only model achieves the lowest MSE (0.0926), while IM-JEPA obtains the highest mean R² (0.1169). The baseline's negative mean R² suggests limited ability to explain within-gene expression variation, despite its relatively high Spearman correlation. In contrast, DP-JEPA achieves positive mean R², reduces MSE relative to the gene-mean predictor, and shows strong isoform ranking across 7 of 8 genes. IM-JEPA achieves the highest mean R², while its lower mean Spearman correlation is largely driven by poor ranking on a single gene. Together, these results provide preliminary evidence that IsoAlign captures within-gene isoform-level variation, although the small evaluation set limits broader conclusions.

[W1] IM-JEPA: We note that IM-JEPA’s gains are task- and encoder-dependent. On NTv2, it consistently improves task performance across: DMS (+0.084 vs. DP-JEPA’s +0.086; Sec. 5.1, Fig. 2), splicing VEP (ClinVar cosine-AUPRC: 0.32 0.35; Table 1), and GTEx expression across all five tissues (: 0.650 0.668 over the matched baseline; Table 3). On AlphaGenome, it achieves our largest DMS gain (+0.136; Sec. 5.1), underperforming the parent model only on splicing VEP. We suspect this may be due to AlphaGenome’s single-nucleotide tokenization already capturing fine-grained splice motifs, compounded by excluding single-isoform genes (46.2%) from IM-JEPA training. We report this openly in Section 5.2 as a characterization of when multi-isoform objectives help.

[W2, Q4] Frame-shift ablation: Yes, the frame-shifted sequence is identical apart from a 1-nt offset, but we respectfully disagree that 0.783 MRR reflects sequence-identity matching. Retrieval is cross-modal (DNA-to-protein) on a gene-disjoint split (Sec. 3), meaning unseen DNA is mapped to unseen proteins with no nucleotide target to match. Because the retrained ablation pairs shifted DNA with its ground-truth isoform, every codon is preserved and only the frame start moves, so the control isolates reading-frame start sensitivity rather than codon use. A 1-nt frame-shift also preserves local order and contiguous motifs, retaining 0.783 MRR. This is consistent with the expected robustness of genomic foundation models such as NTv2 to small positional shifts, which largely preserve the underlying sequence content and local context. In contrast, randomizing base order by nucleotide shuffling collapses MRR to 0.113 (vs. 0.060 for random DNA). Conversely, synonymous codon shuffling keeps the protein identical while altering ~1/3 of nucleotides, and its larger −19% drop shows that alignment draws on nucleotide-level information not recoverable from the protein, which identity matching on unseen genes cannot explain. Accordingly, the frame-shift ablation reflects sensitivity to the reading-frame start rather than the absence of isoform awareness, which we address directly in Sec. 5.4 (Fig. 3; ablation definitions in Appendix E.2).

[W2, Q1] Data splits and leakage across tasks: G2P is split 80/10/10 at the gene level (14,168 / 1,771 / 1,771 genes), with all isoforms of a gene in the same partition and retrieval computed only on held-out genes (Sec. 3, Appendix A.1). The partitions are not family-disjoint. However, the homolog swap (same protein family, different gene) lowers MRR to 0.326 (38.7% retained), compared with 0.665 (78.9% retained) for the isoform swap, so the model distinguishes members of the same family (Sec. 5.4, Fig. 3). The DMS genes (TP53, CBS, PDE3A, GDI1) appear in pretraining only as wild-type sequences, and evaluation uses unseen mutations (e.g., 7,448 TP53 variants). This follows standard VEP practice, where models are pretrained on reference sequences that include all genes and are evaluated on held-out variants (Riesselman et al., 2018; Frazer et al., 2021; Meier et al., 2021; Notin et al., 2023). ClinVar is evaluated using LOCO cross-validation for vector-based metrics and direct AUPRC for scalar metrics (Sec. 5.2). Gene-level disjointness from G2P pretraining is not enforced for ClinVar or MFASS, but evaluation variants are unseen during IsoAlign pretraining (Appendix A.3). GTEx is split at the gene level (15,671 / 1,958 / 1,958 genes; Sec. 5.3). We have clarified the relevant data splits, pretraining overlap, and evaluation protocols in Sections 3, 5.2, 5.3, and Appendix A.

[W3, Q2] Supervised fine tuning methods and baselines: We have added an NTv2-500M + frozen ESM2-150M baseline without IsoAlign pretraining to Tables 2 and 3, using the same LoRA configuration (r=4, α=8, query/value projections), prediction head, training schedule, and data splits (Appendix C, D.4).

For TFLink (Table 2), the matched baseline achieves 0.935 AUROC and 0.928 AUPRC, comparable to DP-JEPA (0.934/0.928). IM-JEPA achieves the highest performance among these configurations (0.936/0.930), although the improvements are modest. These results suggest that the shared DNA–protein architecture explains much of the TFLink performance, while IM-JEPA provides a modest additional improvement.

ModelAUROCAUPRC
NTv2 + ESM2 (no pretraining)0.935 ± 0.0010.928 ± 0.001
IsoAlign (NTv2, DP-JEPA)0.934 ± 0.0010.928 ± 0.001
IsoAlign (NTv2, IM-JEPA)0.936 ± 0.0010.930 ± 0.001

On GTEx (Table 3), the matched NTv2-based IsoAlign (IM-JEPA) improves overall Spearman correlation from 0.650 to 0.668 and Pearson correlation from 0.647 to 0.667, with improvements across all five tissues. These results suggest that isoform-aware pretraining provides benefits beyond the shared DNA–protein fine-tuning architecture. We also clarify that our originally reported IsoAlign results used AlphaGenome as the DNA backbone with DP-JEPA (CLIP), which remains our best-performing configuration, achieving overall Spearman and Pearson correlations of 0.725 and 0.738, respectively.

ModelSpearman (ρ)Pearson (r)
AlphaGenome0.7170.732
NTv2 + ESM2 (no pretraining)0.6500.647
IsoAlign (NTv2, IM-JEPA)0.6680.667
IsoAlign (AlphaGenome, DP-JEPA)0.7250.738

All models in Table 2 receive the TF protein sequence. DNA-only models (AlphaGenome, BPNet, and ChromBPNet) use frozen ESM2-150M protein embeddings, which are mean-pooled and concatenated with DNA representations before prediction. Table 2 reports mean ± SD over four training seeds (42–45), while Table 3 uses a single seed due to its larger size (~445K records). We have clarified the IsoAlign configurations in both tables.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.