- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether biological language models are still learning more useful predictive information latently as they scale, even when zero-shot likelihood-based mutation/variant scoring (e.g., LLR) plateaus. They fit supervised probes to frozen embeddings from DNA, protein and RNA language models at different scales (ESM2, ProGen3, Evo2, and RiNALMo), and compare results on different tasks, in the different domains, primarily ProteinGym, FLAb2, NABench, growth and ADK. They check results in relation to mutation depth, fitness, regression - prediction beyond the training labels range, few shot learning, and transfer between tasks. They find that supervised probes generally outperforms zero-shot likelihood, and that larger models/scale often improve performance in several settings (moreso when using the supervised probes than in the 0 shot), although gains depend on model and domain. The authors conclude that these probes utilize scaling gains missed by likelihood scoring and can improve performance on predictions and assays/tasks.
Critical strengths
- Broad evaluation and comparisons across biological domains (dna/rna/prot), not just of models, is valuable and a useful reference point.
- Addresses a useful question of contrasting the approaches (and it’s saturation re scale) for variant scoring. Larger models saturating is known to practitioners, but is not necessarily established or citeable (Tables 6–7), and its non-intuitive to those from classic ml/nlp/dl (where bigger is typically better) .
- Useful generalization and transfer case studies: evaluating between tasks, predicting to out of distribution regression targets and the analysis on structure and sequence similar/dissimilar is interesting (although not that important to the overall paper I feel - but it is nice and inventive!) (e.g., Fig. 8).
Critical weaknesses
-
The novel contribution needs clearer support. The introduction already acknowledges that supervised prediction from pretrained embeddings generally outperforms zero-shot likelihood, and this is what we’d expect in most ml tasks. The LLR/0 shot approaches are originally justified by generalizability, lack of bias, and perf (dataset bias/task specificness or species specificness is a separate kettle of fish. I can live with it not being investigated here, but it is relevant for future work).he interesting potential contribution is showing that larger models contain additional useful information that likelihood scoring misses. However, the aggregate results do not consistently establish this distinction. ESM2 largely plateaus at larger sizes under both approaches, while ProGen3 and Evo2 improve with scale under both (Tables 6–7). I do not expect every model family or biological task to improve with scale (give or take pretraining size, diversity, model size, model architecture/embedding scale or pretraining tasks). My concern is what new knowledge/novel claims the paper establishes beyond prior work. E.g., (as a non-binding example) the authors could distinguish the benefit of adding supervision from the additional benefit of model scale (e.g. embedding size or benefits from more pretraining) that likelihood scoring fails to capture, but that’s not necessarily the right question. Also, the conclusion that scaling is restored by this is stronger than the evidence presented, if only due to it not being the case in all the domains or models (e.g., RNA figure , although it not applying in all tasks/domains is fine by me and does not specifically affect my acceptance score).
-
Hyperparameter and train/test splits need clarification re leakage. The paper sweeps layers, pooling windows, and normalization hyperparameters but does not clearly explain whether the final representation choices were selected independently of the test splits, doubly so given the many assays/tasks. Choosing configurations using test performance would bias the results in favour of the supervised approach. (I note that the best layers being final and 80% in those models is in alignment with other work, but them being searched for across many tasks and the other params is still a clear potential leak if there’s an overlap between train/test). It’s also unclear to me how train/test splits were done in the tasks, and if the authors did random splits, task defined splits (e.g. protein gym supervised splits), as random splits on many tasks is not enough given homologous families (that a supervised model especially can memorize, e..g close sequence homologs).
Questions for authors
- Main contribution and scaling Prior work and de facto best practices are that that supervised models from embeddings outperforms zero-shot likelihood/LLR in many cases and tasks (with the exceptions being the rule). What is your main new finding (about scaling) beyond this established result? As a non binding example (free to choose anything else), you could compare the per-assay changes in Spearman correlation across model sizes (give or take pretraining size) for the supervised probes and zero-shot, using the same assays and splits, and see In which cases do larger models improve supervised prediction when likelihood scoring does not? (and if it’s dependent on embedding size, model size, likelihood/perplexity, training size, memorization etc).
Tl;dr: Please clarify the main finding/claim/contribution here beyond “supervised learning does better than unsupervised/zero shot” - this is a must, and would move my score to accept.
- Hyperparameters. The paper sweeps layers, pooling windows, and normalization methods; but what was used to choose the configurations? Were these choices made using training/validation data only, independently of the final test folds (Across all assays)? Did you filter out (how? homology?) sequences appearing in any of the test sets, per domain and task/assay from this? If test scores informed configuration selection, please provide an independent evaluation of the affected results. E.g. naively choosing 80% or final layer by domain and mean pooling, defaults etc and comparing. Should report this. If its negligible params (e.g. just choosing one layer across all tasks and default pooling without tweaking), could also be enough to report on a few, while being clear about the leakage.
4 comments
Broken figure
Figure 4: your evo 2 subfigure is currently missing/broken in pdf
Reply
Thanks for the reply and followup. The new analyses help, and I'm leaning towards raising my score from 2 to 3.
The supervised baselines address my concern that the results might just reflect adding labels.
I still have two concerns:
-
Novelty. My original question wasn't only whether the gains come from supervision, but what's new compared to existing work (sorry if I wasn't clear enough). We know that supervised models often beat zero-shot, (e.g., ProGen3 and other works in dna and protein LLMs showed that scaling can help supervised prediction even when zero-shot gets worse or language modelling doesn't improve). Is the new finding that this also holds for simple probes on frozen representations, across different models and biological domains? If so, I think that's worth stating more clearly, rather than making broad claims about scaling laws in biology. I'd focus on making the actual contribution clearer.
-
Possible leakage. Thanks for clarifying the hyperparameter selection. Using PTEN's test folds to choose the layer/pooling settings still worries me, even if those settings were then fixed for other assays. I'd suggest checking that the main conclusions hold without the tuning assays. I don't see this as a dealbreaker, but it does weaken the evaluation, as leakage applies to hyperparam etc settings; (I wouldn't change the score about this, just any takeaways).
Thank you for the follow ups! Regarding novelty and possible leakage:
Novelty: We show that scaling biological sequence models yields increasingly informative representations of biological phenotypes across multiple modalities, architectures, and model families. Beyond per-assay prediction we analyze fitness landscape geometry (Fig. 5, pg 6), the near-orthogonality of phenotype and likelihood directions (Fig. 7 pg 9), and phenotype directions that transfer to held-out assays and sharpen with scale (Fig. 8, pg 10, Appendix pg 36):
Comparison with literature:
While ProGen3 notes that latents become more informative with scale, they use iterative reasoning preference optimization (IRPO), which updates all model weights. In contrast, we keep weights frozen across ESM2, ProGen3, Evo2, and RiNALMo, and examine transferable phenotype vectors across unseen proteins (Fig. 8, p. 10).
Li et al. similarly use ridge regressions on ESM-1b and CARP embeddings. They conclude that fitness prediction does not always scale, as seen through a few tasks (AAV, GB1) that do not show improvements with increasing ESM-1b sizes. However, their model size ladders stop below 1B, whereas our divergence from the one-hot ridge regression is clearest at ESM2 3B to 15B. Also, they use full-sequence mean pooling, which dilutes single-residue substitions. Our pooling sweep (Fig. 10) confirms whole-chain pooling minimizes the latent advantage. Re-running their GB1 and subcellular localization benchmarks on ESM2 (8M–650M) confirms this: per-token for multiple mutational depths and mutant-site pooling for single mutations outperforms whole-chain pooling and recovers scaling on GB1, even if the improvements are monotonic.
Lin et al. show structural (TM-score) and perplexity gains up to ESM2-15B. Since the emergence of structural features with scale is pretty well established in Li et al. and Lin et al.,’s papers, we examine scaling in the context of mutational fitness and general multi-assay feature vectors (thermostability, growth, activity, abundance; Fig. 8, p. 10). This resolves Hou et al. (Nat. Comput. Sci., 2026), who show zero-shot likelihoods peak at moderate confidence and degrade at larger scales.
Prior gLM probing studies evaluate regulatory or variant-effect tasks without multi-scale sweeps within a single family or without systematic pooling/normalization sweeps (e.g., Datta et al., 2026). The closest work, Benegas & Czech's MarinDNA (Open Athena, Aug 2026) sweeps MarinDNA and Evo2. This is just a blog post and does not examine measured, quantifiable phenotypes.
Possible Leakage: Removing the hyperparameter-tuned assay changes the overall mean Spearman by only 0.005, indicating hyperparameter selection does not meaningfully bias our results (Appendix A5.2, pg 21). Moreover, we fit probes for a single layer depth per model family across all sizes (final layer for ESM2, ProGen3, RiNALMo; ~80% depth for Evo2 models) rather than selecting the best layer per model size, which may artificially favor larger models. Using ~75–80% layer depth for Evo2 is also independently supported by prior Evo2 sparse autoencoder interpretability studies (Brixi et al. 2026).
We are in the process of rewriting to include more related work and clarify our specific contributions. We also added the missing Evo2 subfigure that was still running at the time of the prior Friday revision. Please let us know if there is anything else you are curious about, and thank you again for the feedback!
Thank you for the feedback and questions! Our main claim is that scaling language models trained on biological sequence data improves model representations, which in turn improves downstream tasks such as mutational effect prediction when conditioned on the phenotype of interest. We will continue to clean up the writing on scaling monotonicity for the camera-ready paper.
Scaling evidence: We added Chinchilla parametric fits of test loss as a function of N total parameters and D tokens (Sec. 2.4, Fig. 4; App. A.9, Figs. 23, 24). With higher dimensionality more features can be represented, which we see as features such as ADK kcat emerging in larger PLMs. On ADK kcat the fitted phenotype direction reaches rho=0.51 against 0.43 for the likelihood direction, at a cosine similarity of 0.35, so the two directions are distinct (Fig. 2; Sec. 2.2). Fits for Evo2 and ProGen3 are still in queue and not yet in the revision. We also add further controls by training a ridge regression on the one-hot encoded representations of the sequence variants (pLM and gLM independent) and the supervisory labels. We see that the ridge regressions fit on pLM and gLM representations generally outperform the one-hot encoded version, especially for the modulo and contiguous fits, and that the probe improvement is not just from inclusion of supervisory labels, but rather, improvement in model representations with scale.
Non-binding example: We include the ADK enzyme's catalytic activity rates across its 30 stratified splits (Secs. 2.1, 3.1), plus a new final figure on general phenotype directions with all ProteinGym assays grouped by similar measurement type (abundance, growth, etc.).
Noncoding RNA: The linear probe's median across assays dips slightly at the largest size (0.324, 0.344, 0.338 at 33.5M, 148M, 651M) while its mean increases monotonically (0.329, 0.331, 0.339; Table 6). We also found that for RiNALMo we had reused one ridge penalty (lambda) selection (leave-one-out GCV) across the random, modulo and contiguous folds, which made the Spearman correlation on the position-held-out folds worse. We keep lambda selection matched per scheme elsewhere (App. A.5.2, A.5.5; Figs. 14 to 16); we missed it for RiNALMo because we added the fold schemes later, as modulo and contiguous splits were not originally in NABench, unlike ProteinGym. We are redoing this now.
Hyperparameters: We took the settings that were best for PTEN growth on ProteinGym's published random, modulo and contiguous schemes and used them for all probes on the other assays (Sec. 2.1; App. A.5.1, Figs. 10 to 13; App. A.5.4, Tables 4, 5). We could have varied this per assay, fold and scheme, but our point is that scaling improves biological model representations, and we did not want to bias the result by optimizing the probe for every setting. Only lambda is chosen by nested cross-validation on inner folds matched to the outer scheme (Eq. 12; App. A.5.5). Changing the normalization does not drastically alter any of the 36 probe-versus-likelihood comparisons, and we include a naive default baseline (last layer, mean pooling, no sweep). The normalization sweep is now mentioned in the main text and described more fully in App. A.5.4; we proceed with z-scoring across the training fold for all probes.
Training/Testing Splits: We use each benchmark's published partitions throughout, where ProteinGym's modulo and contiguous schemes hold out entire residue positions (App. A.2). This is now clearer in the main text, and we report results by split scheme as well as pooled, noting that on the contiguous scheme the probe ties the likelihood at 650M. We also extended both protocols to NABench, which did not come with those folds. For homology across duplicate or related proteins, we recomputed our significance tests over single-linkage 80% identity clusters (199 assays to 164 clusters); the median margin shifts by at most 0.025 and all conclusions hold.
Dependence on embedding size, model size, likelihood and training size: We like this idea and added the analysis across the 186 common ESM2 assays (App. A.7; Fig. 19). Assay phenotype accounts for the largest share of variance in probe rho (R2=0.23 to 0.33), ahead of MSA depth (0.11), taxon (0.10) and assay size (0.04). After controlling for the other factors, MSA depth correlates significantly with the zero-shot likelihood's accuracy (partial rho of +0.14 to +0.38) but not with the probe's accuracy, so lower homology coverage hurts zero-shot likelihoods much more than conditioned probes.