- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces a new large scale protein language model. The model is autogressive, and one version can condition on homologs or other related sequences. The model is capable of generating functional proteins, as validated in the wetlab.
Critical strengths
This is a major release of a large scale, open source protein foundation model. The model development and testing is carefully done and carefully explained, so I expect the model will be widely used and an important asset for the research community.
Critical weaknesses
The FK-SMC algorithm does not appear to be rigorously justified - it is unclear what distribution it is sampling from or why. This makes the procedure difficult to understand and evaluate. The figures are tiny - unreadable in print.
Questions for authors
What is the justification for the FK-SMC algorithm: what is the distribution it is sampling from, even if approximate?
Protein language models often plateau and then even decrease in zero-shot mutation prediction performance as pretraining scales up, likely due to phylogenetic correlation in the data (see https://pascalnotin.substack.com/p/have-we-hit-the-scaling-wall-for for a review). But here performance steadily increases with scale. What do you think is the reason? Were there specific choices that went into the training that were intended to avoid this plateau? Or is it just because only a few different model sizes were trained, and so it’s hard to see where the peak might be?
What is the definition you are using of a “low complexity” sequence?
There are a variety of metrics for generation quality (site-wise KL, HMMER bit score), but none actually check distribution match with the data: it is possible to optimize these metrics without actually achieving a close match to the validation MSA. I would suggest using MMD or another distribution-matching metric.
2 comments
Thank you for the thorough response, this addresses all my questions and concerns.
We thank the reviewer for the positive assessment, for recognising the release as an asset for the research community, and for these thoughtful questions, which led us to the manuscript in several points.
W1/Q1. FK-SMC: justification and distribution. We apologise for the confusion. We chose the name expecting FK steering of diffusion models to give readers an intuition, but the procedure has no target measure, importance weights or resampling, so it does not sample from a target distribution in the SMC sense. In contrast, at each step, ProtGPT3-MSA generates N candidate continuations of the prompt, each made of K new sequences; we score each with the oracle, add the first sequence of the best one to the prompt, and repeat until the prompt holds 16 sequences (Algorithm 1). Its closest FK analogue is the limit in which resampling keeps only the best particle. The intuition is that ProtGPT3-MSA generates sequences that resemble its prompt (prompting with more stable variants yields more stable generations, Fig. 1g), so as the prompt fills with high-scoring sequences, the generations move towards high scores while staying within the family. Consistent with this, the final prompt makes the model more likely to generate variants measured as highly stable, and less likely to generate unstable ones (Fig. 3b). At matched compute, this steering procedure outperforms best-of-N on double and quadruple mutants (Fig. 3c–d). In the revised version we will drop the name FK-SMC and present the procedure as inference-time steering reminiscent of Feynman–Kac steering (Sec. 5, App. A.15); the results are unchanged.
W2. Figures. We apologise, this is true. We have enlarged the main-text figures and their fonts for the camera ready version. Thanks for noticing.
Q2. Scaling of zero-shot fitness prediction. We think the reviewer's last explanation is the main one. The gain in mean Spearman ρ from 1.3B to 10B (+0.02; +0.035 on substitutions, +0.007 on indels) is about a fifth of that from 112M to 1.3B (+0.12; Table 2), so our models are consistent with the plateau at 1-4B parameters described in the cited review. We hypothesize that two features may delay this plateau (while not avoiding it): compute-optimal training, in which each model sees its training sequences about once (Table 1), and 50–70% metagenomic GigaRef tokens, which the review suggests may help if they add diversity within the assayed families. In line with the review's emphasis on retrieval, homolog conditioning is the more efficient route: the 112M ProtGPT3-MSA matches ProtGPT3-10B on substitutions (ρ = 0.357 for both). We add this discussion to App. A.6.
Q3. Low complexity. We use SEG (Wootton & Federhen, 1993) as implemented in NCBI BLAST+ segmasker (Camacho et al., 2009) with default parameters. SEG scans the sequence with a short sliding window and masks stretches dominated by a few amino acids (e.g. ‘AAAAAA’) or short repeats (e.g. ‘KEKEKEKEK’). We call a sequence low-complexity when more than 25% of its residues are masked. We apologise for this oversight; the revised version will give the definition and these references in Sec. 3.3.
Q4. Distribution match. We agree, and computed the maximum mean discrepancy (MMD; Gretton et al., 2012) between each model's generations and the held-out sequences of each of the 25 evaluation families, with a Gaussian kernel on ESM-2 embeddings, equal numbers of generations per model, and the family's own training sequences as the natural reference (MMD² ≈ 0). The results reproduce the pattern of our other metrics. In families with < 500 sequences, ProtGPT3-MSA's generations are closer to the held-out distribution than those of SFT-112M (mean MMD² 0.038 vs 0.190; closer in 11/12 families, Wilcoxon p = 0.002) and SFT-10B (0.144; 10/12, p = 0.009), whereas in larger families the differences are narrower (p ≥ 0.38), and the gap differs between the two groups (p ≤ 0.02). We will add this analysis to App. A.10.
Please let us know if anything is not clear; we really appreciate your time and feedback!
References
Camacho, C., Coulouris, G., Avagyan, V., Ma, N., Papadopoulos, J., Bealer, K., & Madden, T. L. (2009). BLAST+: architecture and applications. BMC Bioinformatics, 10, 421.
Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., & Smola, A. (2012). A kernel two-sample test. JMLR, 13, 723–773.
Wootton, J. C., & Federhen, S. (1993). Statistics of local complexity in amino acid sequences and sequence databases. Computers & Chemistry, 17(2), 149–163.