- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VenusGenome learns gene- and genome-level representations by reconstructing masked, frozen protein embeddings with a visible-only encoder. The framework combines high-ratio masking with contig-restricted attention, global tokens, relative-strand biases, and within-contig positional information. Across six genome-level phenotype tasks and five gene-level metrics, its frozen representations achieve the highest mean performance among the evaluated models on nine of eleven primary metrics; the reported ablations favor masked autoencoding and the genome-aware attention design on zero-shot PPI ranking.
Critical strengths
1.The paper is well organized and readable. 2. The full model yields competitive results on both organism-level phenotype and gene-function benchmarks.
Critical weaknesses
-
The contribution of the genome-aware architecture could be further clarified. For example, gLM (Hwang et al., 2024) reconstructs masked protein features from genomic context, BacFormer (Wiatrak et al., 2025) contextualizes ordered proteins across bacterial genomes, and MAE-LM (Meng et al., 2024) excludes masked inputs from the encoder. Against these precedents, explaining what contig-restricted attention and relative-strand biases add would make the architectural contribution easier to assess.
-
The experimental analysis could better establish which design choices drive the downstream gains. Most masking and attention ablations use STRINGDB PPI ranking, so their effects on phenotype and BGC prediction are less directly tested. Selected ablations on these tasks, with pretraining data and model size held fixed, and genomic-distance controls for PPI would strengthen the attribution of the reported improvements.
Reference:
- Hwang et al. (2024). Genomic language model predicts protein co-regulation and function. Nature Communications.
- Wiatrak et al. (2025). A contextualised protein language model reveals the functional syntax of bacterial evolution. bioRxiv.
- Meng et al. (2024). Representation Deficiency in Masked Language Modeling. ICLR.
Questions for authors
-
Relative to gLM’s masked protein-feature prediction and BacFormer’s genome-wide contextualization, what is the main architectural contribution of VenusGenome? In particular, how do the contig restrictions and relative-strand biases change the relationships the model can represent compared with their existing structural encodings? It would be useful to distinguish these genome-specific changes from the visible-only encoder–decoder arrangement established by MAE-LM.
-
Would the conclusions from the masking and attention ablations hold on representative phenotype and BGC tasks when pretraining data and model size are kept fixed? For the PPI results, which evidence channels enter the STRING target score, and do the gains persist after matching pairs by genomic distance or using experimental evidence alone? These results would make the connection between the pretraining rationale and the downstream functional benefits more convincing.
2 comments
[Q2, W2] With model scale and pretraining corpus held fixed (512M, 40k steps), genome-aware priors and 75% masking drive gains on gene-organization tasks (BGC, operon, PPI), and architectural PPI advantages persist across non-proximity channels and all 12 distance/strand strata.
Thank you for suggesting these controls to connect our pretraining rationale with downstream functional benefits. We address all three parts at fixed 512M scale and 40k steps:
(1) Cross-task ablations (fixed 512M scale, corpus, and probe protocol). We evaluated frozen 512M checkpoints on genome-disjoint BGC detection, species-disjoint binary phenotype, and zero-shot operon and PPI ranking (five probe seeds for supervised tasks):
| 512M / 40k configuration | BGC AUPRC | Phenotype AUROC | Operon AUPRC | PPI ρ |
|---|---|---|---|---|
| Full MAE-M75 | 0.6600 ± 0.0019 | 0.7893 ± 0.0023 | 0.3897 | 0.2930 |
| w/o contig attention | 0.5033 ± 0.0032 | 0.7903 ± 0.0040 | 0.3468 | 0.2029 |
| w/o global tokens | 0.6768 ± 0.0029 | 0.7890 ± 0.0054 | 0.3708 | 0.2645 |
| w/o strand bias | 0.6588 ± 0.0033 | 0.7944 ± 0.0028 | 0.3572 | 0.2609 |
| MAE-M15 | 0.6344 ± 0.0024 | 0.7919 ± 0.0038 | 0.3934 | 0.2819 |
| BERT-M15 | 0.6376 ± 0.0011 | 0.7896 ± 0.0035 | 0.3970 | 0.1920 |
This pattern directly links pretraining design to task structure: tasks governed by multi-gene spatial organization (BGCs, operons, and PPI) rely strongly on contig boundaries and strand biases (removing contig restriction drops BGC by 0.157 and operon by 0.043), 75% masking boosts BGC over 15% masking (+0.026), and MAE sharply improves zero-shot embedding geometry over BERT on PPI (0.2819 vs. 0.1920). Conversely, coarse genome-level binary phenotypes depend mainly on global gene-repertoire composition (mean-pooled across thousands of genes) and are therefore largely insensitive to local structural ablations.
(2) STRING channels and experimental-only feasibility. The main PPI target is STRING combined confidence across 8,525,221 within-contig pairs in 53 genomes (integrating neighborhood, fusion, co-occurrence, coexpression, experiments, curated databases, and text mining to measure functional-association ranking rather than physical binding). Experimental-only evaluation is infeasible here because only six pairs have nonzero experimental scores (and coexpression has none). Evaluating the five supported non-transferred channels independently (≥20 nonzero pairs per genome; shared sets per column) shows that the full model's gains extend beyond genomic neighborhood to fusion, co-occurrence, and curated database associations:
| 512M-M75 variant | Neigh. | Fusion | Cooccur. | Database | Text |
|---|---|---|---|---|---|
| Full | 0.509 | 0.242 | 0.289 | 0.209 | 0.159 |
| w/o contig attention | 0.420 | 0.160 | 0.233 | 0.137 | 0.122 |
| w/o global tokens | 0.485 | 0.218 | 0.281 | 0.198 | 0.164 |
| w/o strand bias | 0.494 | 0.196 | 0.262 | 0.191 | 0.155 |
| Eligible genomes | 53 | 52 | 53 | 50 | 7 |
| Included pairs | 664,849 | 90,912 | 4,106,451 | 50,199 | 3,693 |
(3) Genomic-distance and strand stratification. Stratifying the 8,525,221 combined-score pairs jointly by gene-index distance and strand orientation confirms that component gains persist within matched strata:
| Distance | Strand | Pairs | Genomes | Full | w/o contig | w/o global | w/o strand |
|---|---|---|---|---|---|---|---|
| 1–5 | Same | 384,829 | 53 | 0.540 | 0.445 | 0.525 | 0.486 |
| 1–5 | Opposite | 128,367 | 53 | 0.264 | 0.241 | 0.228 | 0.280 |
| 6–20 | Same | 249,379 | 53 | 0.283 | 0.183 | 0.255 | 0.230 |
| 6–20 | Opposite | 133,225 | 53 | 0.188 | 0.155 | 0.168 | 0.156 |
| 21–100 | Same | 386,110 | 53 | 0.126 | 0.067 | 0.085 | 0.096 |
| 21–100 | Opposite | 301,599 | 53 | 0.110 | 0.075 | 0.084 | 0.083 |
| 101–500 | Same | 949,407 | 52 | 0.097 | 0.047 | 0.065 | 0.074 |
| 101–500 | Opposite | 839,068 | 51 | 0.090 | 0.059 | 0.075 | 0.067 |
| 501–2,000 | Same | 1,591,146 | 35 | 0.100 | 0.046 | 0.066 | 0.073 |
| 501–2,000 | Opposite | 1,630,954 | 35 | 0.108 | 0.050 | 0.080 | 0.087 |
| >2,000 | Same | 925,967 | 13 | 0.105 | 0.077 | 0.093 | 0.080 |
| >2,000 | Opposite | 1,005,150 | 13 | 0.106 | 0.068 | 0.088 | 0.080 |
Full exceeds w/o contig and w/o global in all 12 strata and w/o strand in 11 of 12 (except distance 1–5, opposite strand). Although absolute correlations attenuate with distance and on short opposite-strand pairs (where co-transcribed operons are rarer), relative gains over all three ablations hold across every long-distance stratum (>100 to >2,000 genes), where local proximity cues are absent.
Manuscript change: Added cross-task ablations to Appendix B.5 (Table 12) and STRING channel/distance analyses to Appendix B.3 (Tables 10 and 11), and clarified the PPI target in Section 4.3.
Thank you for your positive assessment of the paper's clarity and competitive results, as well as your constructive questions on architectural positioning and gain attribution. We address your two questions below.
[Q1, W1] While visible-only continuous reconstruction builds on gLM, BacFormer, and MAE-LM, VenusGenome's genome-specific contribution is encoding assembly invariants directly into the attention graph via contig-restricted connectivity, global-token routing, and pairwise relative-strand biases.
Thank you for asking us to sharpen this distinction. We do not claim to originate continuous protein-feature reconstruction (gLM), genome-wide protein contextualization (BacFormer), or visible-only encoder–decoder masked autoencoding (MAE-LM). While the visible-only encoder–decoder layout is an adaptation of MAE-LM for compute efficiency and representation utilization, our genome-specific design changes which biological relationships can be represented without assembly-induced confounding:
Both encoder and decoder apply these genomic attention mechanisms with separate learned parameters, and our ablations (detailed in Q2) confirm that removing them substantially degrades local gene-organization tasks (e.g., BGC AUPRC drops from 0.660 to 0.503 and PPI from 0.293 to 0.203 when removing contig restriction).
Manuscript change: Updated Section 2 (Related Work) to explicitly credit gLM, BacFormer, and MAE-LM and contrast their encodings with VenusGenome's enforced contig-restricted connectivity, global-token routing, and pairwise relative-strand biases.