acceptodds
Under review as a conference paper at ICLR 2027

ProtGPT3: An Open-source family of Promptable and Aligned Protein Language Models

Abstract

Generative protein language models (pLMs) enable exploration of vast sequence spaces, but controlling generation toward desired functional families remains challenging, particularly in low-data regimes. We introduce ProtGPT3, an open family of autoregressive pLMs from 112M to 10B parameters, comprising single-sequence and MSA-promptable models and, to our knowledge, the largest publicly released pLM trained exclusively with an autoregressive objective. We use ProtGPT3 to compare supervised fine-tuning against few-shot prompting with homologous sequences. Across 25 held-out protein families, prompting outperforms fine-tuning in the low-data regime and matches it as training-set size grows. In an extreme low-data case study on defluorinase enzymes, we generate enzymes by fine-tuning or prompting ProtGPT3 models with seven enzymes only, producing four new defluorinase with activity values in line with the natural ones. We further investigate two complementary ways to control generation. First, we apply direct preference optimization to align the models at a large scale, mitigating the well-established pre-training bias toward low-complexity sequences, while preserving sequence diversity. Second, we introduce a novel Feynman-Kac sequential Monte Carlo procedure that uses inference-time compute to steer generation toward sequence-level objectives without updating model parameters. We release model weights, configurations, training and inference code on Hugging Face ( https://huggingface.co/protgpt3 ) under a pseudonym.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein language models, protein design, enzyme, MSA

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9qEVOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a new large scale protein language model. The model is autogressive, and one version can condition on homologs or other related sequences. The model is capable of generating functional proteins, as validated in the wetlab.

Critical strengths

This is a major release of a large scale, open source protein foundation model. The model development and testing is carefully done and carefully explained, so I expect the model will be widely used and an important asset for the research community.

Critical weaknesses

The FK-SMC algorithm does not appear to be rigorously justified - it is unclear what distribution it is sampling from or why. This makes the procedure difficult to understand and evaluate. The figures are tiny - unreadable in print.

Questions for authors

What is the justification for the FK-SMC algorithm: what is the distribution it is sampling from, even if approximate?

Protein language models often plateau and then even decrease in zero-shot mutation prediction performance as pretraining scales up, likely due to phylogenetic correlation in the data (see https://pascalnotin.substack.com/p/have-we-hit-the-scaling-wall-for for a review). But here performance steadily increases with scale. What do you think is the reason? Were there specific choices that went into the training that were intended to avoid this plateau? Or is it just because only a few different model sizes were trained, and so it’s hard to see where the peak might be?

What is the definition you are using of a “low complexity” sequence?

There are a variety of metrics for generation quality (site-wise KL, HMMER bit score), but none actually check distribution match with the data: it is possible to optimize these metrics without actually achieving a close match to the validation MSA. I would suggest using MMD or another distribution-matching metric.

2 comments

We thank the reviewer for the positive assessment, for recognising the release as an asset for the research community, and for these thoughtful questions, which led us to the manuscript in several points.

W1/Q1. FK-SMC: justification and distribution. We apologise for the confusion. We chose the name expecting FK steering of diffusion models to give readers an intuition, but the procedure has no target measure, importance weights or resampling, so it does not sample from a target distribution in the SMC sense. In contrast, at each step, ProtGPT3-MSA generates N candidate continuations of the prompt, each made of K new sequences; we score each with the oracle, add the first sequence of the best one to the prompt, and repeat until the prompt holds 16 sequences (Algorithm 1). Its closest FK analogue is the limit in which resampling keeps only the best particle. The intuition is that ProtGPT3-MSA generates sequences that resemble its prompt (prompting with more stable variants yields more stable generations, Fig. 1g), so as the prompt fills with high-scoring sequences, the generations move towards high scores while staying within the family. Consistent with this, the final prompt makes the model more likely to generate variants measured as highly stable, and less likely to generate unstable ones (Fig. 3b). At matched compute, this steering procedure outperforms best-of-N on double and quadruple mutants (Fig. 3c–d). In the revised version we will drop the name FK-SMC and present the procedure as inference-time steering reminiscent of Feynman–Kac steering (Sec. 5, App. A.15); the results are unchanged.

W2. Figures. We apologise, this is true. We have enlarged the main-text figures and their fonts for the camera ready version. Thanks for noticing.

Q2. Scaling of zero-shot fitness prediction. We think the reviewer's last explanation is the main one. The gain in mean Spearman ρ from 1.3B to 10B (+0.02; +0.035 on substitutions, +0.007 on indels) is about a fifth of that from 112M to 1.3B (+0.12; Table 2), so our models are consistent with the plateau at 1-4B parameters described in the cited review. We hypothesize that two features may delay this plateau (while not avoiding it): compute-optimal training, in which each model sees its training sequences about once (Table 1), and 50–70% metagenomic GigaRef tokens, which the review suggests may help if they add diversity within the assayed families. In line with the review's emphasis on retrieval, homolog conditioning is the more efficient route: the 112M ProtGPT3-MSA matches ProtGPT3-10B on substitutions (ρ = 0.357 for both). We add this discussion to App. A.6.

Q3. Low complexity. We use SEG (Wootton & Federhen, 1993) as implemented in NCBI BLAST+ segmasker (Camacho et al., 2009) with default parameters. SEG scans the sequence with a short sliding window and masks stretches dominated by a few amino acids (e.g. ‘AAAAAA’) or short repeats (e.g. ‘KEKEKEKEK’). We call a sequence low-complexity when more than 25% of its residues are masked. We apologise for this oversight; the revised version will give the definition and these references in Sec. 3.3.

Q4. Distribution match. We agree, and computed the maximum mean discrepancy (MMD; Gretton et al., 2012) between each model's generations and the held-out sequences of each of the 25 evaluation families, with a Gaussian kernel on ESM-2 embeddings, equal numbers of generations per model, and the family's own training sequences as the natural reference (MMD² ≈ 0). The results reproduce the pattern of our other metrics. In families with < 500 sequences, ProtGPT3-MSA's generations are closer to the held-out distribution than those of SFT-112M (mean MMD² 0.038 vs 0.190; closer in 11/12 families, Wilcoxon p = 0.002) and SFT-10B (0.144; 10/12, p = 0.009), whereas in larger families the differences are narrower (p ≥ 0.38), and the gap differs between the two groups (p ≤ 0.02). We will add this analysis to App. A.10.

Please let us know if anything is not clear; we really appreciate your time and feedback!

References

Camacho, C., Coulouris, G., Avagyan, V., Ma, N., Papadopoulos, J., Bealer, K., & Madden, T. L. (2009). BLAST+: architecture and applications. BMC Bioinformatics, 10, 421.

Gretton, A., Borgwardt, K. M., Rasch, M. J., Schölkopf, B., & Smola, A. (2012). A kernel two-sample test. JMLR, 13, 723–773.

Wootton, J. C., & Federhen, S. (1993). Statistics of local complexity in amino acid sequences and sequence databases. Computers & Chemistry, 17(2), 149–163.

Thank you for the thorough response, this addresses all my questions and concerns.

Reviewer_EqneOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces autoregressive protein language models from 112M to 10B parameters, a 112M homolog-conditioned MSA model, and DPO-aligned variants. It compares homolog prompting with family-specific fine-tuning across 25 families, finding an advantage for prompting in shallow families. An experimental case study starting from seven known defluorinase sequences yields four active designs. Additional studies use preference optimization to reduce low-complexity generation and a trajectory-selection procedure called FK-SMC to improve oracle-predicted stability or fitness. The authors report releasing seven models and their training and evaluation code.

Critical strengths
  1. The model family spans several scales and includes both single-sequence and homolog-conditioned generation.

  2. The paper gives filtering and assay details and investigates why the SFT models produce few filter-passing designs.

Critical weaknesses
  1. The 50%-identity separation for single-sequence validation is described, but the construction and cross-dataset decontamination of the MSA validation families are less clear

  2. The paper does not specify a Feynman–Kac target measure, importance weights, or weighted particle resampling.

Questions for authors
  1. How were MSA training and validation families separated, and were homologs checked against both the MSA corpus and single-sequence pretraining data? Can you provide the split identifiers and the precise similarity/coverage rules?

  2. Were the 51 MSA designs sampled uniformly or selected through additional rankings?

2 comments

We thank the reviewer for the careful feedback, for recognising the manuscript's strengths, and for these precise questions, which led us to revise the manuscript in several places.

W1/Q1. MSA split and decontamination. We apologise for not describing this clearly. ProtGPT3-MSA was trained on OpenProteinSet's uniclust30_unfiltered MSAs (Ahdritz et al., 2023). We held out a random 5% of the MSAs (49 of the 980 files) for validation, the rest gave the ~8.5M training MSAs (App. A.4), and the 25 evaluation families were drawn from the held-out MSAs. Each MSA is built by HHblits around a different Uniclust30 cluster representative, and these representatives are < 30% identical to one another, so the queries of evaluation and training MSAs are distinct and distant. The split is per MSA, without homology filtering of member sequences. We have checked whether ProtGPT3-MSA's generations for the 25 held-out families are the product of memorisation: none of the 2,929 is a ≥ 99% copy of a held-out sequence.

Regarding the single-sequence pre-training data (UniRef90 + GigaRef), which ProtGPT3-MSA and SFT-112M both saw through their common starting checkpoint, we have checked whether there is any overlap. Of the 3,391 held-out MSA sequences, 60% match at ≥ 50% identity and 11.5% at ≥ 90%. This motivated a new experiment on 25 metagenomic families from NMPFamsDB (Baltoumas et al., 2023) with no detectable homology to ProtGPT3-MSA's training MSAs and little overlap with the pre-training data (23% of sequences at ≥ 50% identity). The SFT vs prompting comparison returned the same results we had reported: ProtGPT3-MSA again wins on all four metrics in families with < 500 sequences (fident +0.13, TM-score +0.11, HH-score +144, KL −0.88). In larger families. SFT-112M catches up and leads on TM-score and KL. However, we noticed in this case ProtGPT3-112M generated identical copies to the families it had been fine-tuned on (on 24% of its generations vs. 1.6% ProtGPT3-MSA), perhaps due to the MSAs being closer to the query.

The revised manuscript includes the new analysis and clearer descriptions of the splits.

W2. FK-SMC. The reviewer is right, we apologise for the misuse of the term. We chose the name expecting FK steering of diffusion models to give readers an intuition, but the procedure has no target measure, importance weights or resampling. At each step it samples N rollouts of K homologs from the current prompt, scores each by the mean objective over its K sequences, and appends the first sequence of the best rollout, until the prompt reaches 16 sequences. In the revision we drop the name FK-SMC and present the procedure as inference-time steering reminiscent of Feynman–Kac steering (Sec. 5, App. A.15); the results are unchanged.

Q2. The 51 defluorinase designs. Thank you for catching this inconsistency. All models' designs had to pass the same filters (85-95% identity to a known defluorinase, TM-score ≥ 0.5, hSASA ratio ≤ 1.7), and we tested every sequence from single-sequence models that would pass the filters (in total: 3 from ProtGPT3-1.3B, 7 from -10B, none from -112M). For ProtGPT3-MSA, 444 passed the filters, and we tested 10 selected by the additional ranking (App. A.14). Later, we picked 41 randomly from the rest. All three active ProtGPT3-MSA enzymes came from the random round (3/41 vs 0/10), so the ranking does not explain them. The revision now describes both rounds.

Please let us know if anything is not yet clear, we really appreciate your time and feedback.

References

Ahdritz, G., Bouatta, N., Kadyan, S., Jarosch, L., Berenberg, D., Fisk, I., Watkins, A., Ra, S., Bonneau, R., & AlQuraishi, M. (2023). OpenProteinSet: Training data for structural biology at scale. NeurIPS, 36, 4597–4609.

Baltoumas, F. A., Karatzas, E., Liu, S., Ovchinnikov, S., Sofianatos, Y., Chen, I.-M., Kyrpides, N. C., & Pavlopoulos, G. A. (2024). NMPFamsDB: a database of novel protein families from microbial metagenomes and metatranscriptomes. Nucleic Acids Research, 52(D1), D502–D512.

Pavlopoulos, G. A., Baltoumas, F. A., Liu, S., et al. (2023). Unraveling the functional dark matter through global metagenomics. Nature, 622, 594–602.

Thank you for the detailed responses and additional experiments. The clarification addresses most of my concerns. I will maintain my positive score.

Reviewer_rix7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper trains single-sequence protein language models across multiple sizes, and an MSA-conditioned protein language models. They empirically evaluate this on conditional protein generation tasks, and find the MSA-conditioned model better generates stable protein sequences. Futhermore, they find it does better than SFT on small datasets, and does well at inference-time steering. For their single-sequence models, they use DPO to improve sequence quality.

Critical strengths

The paper extensively evaluates their models, especially their MSA conditioned one. They find that it does better than SFT on small datasets, performs well with inference-time steering, and experimentally validate the model is able to generate enzymes from limited data samples.

Critical weaknesses

I'm not clear about what the paper's contribution is over/in relation to existing work. I wish the paper could have focused more on this point.

Is the claim that -- the MSA-conditioned pLM is better than that of existing ones (e.g. PoET)? I see that the authors benchmarked against PoET, but it wasn't clear to me that this model is fundamentally better. Perhaps PoET could have also successfully done the downstream MSA experiments. I also didn't get a clear reason (e.g. architectural, data, etc.) for why this model would be better.

Is the claim that -- MSA-conditioned pLMs are able to do all of these things? Here, I would need the paper to present a novel capability that previous papers didn't. It seems like the closest thing is the SFT vs few shot prompting comparison? However, this doesn't strike me as a particularly novel conclusion. I could have predicted in advance that few-shot prompting does better than SFT in low-N settings.

The FK-SMC/DPO, again, don't seem particularly conceptually novel to me.

Is the paper's contribution the released models themselves? Then, again, I would need to know what these models do that other (released) models don't do.

To recommend acceptance, I need to be able to extract a novel conceptual takeaway from the paper. I'm currently unclear on what that is.

Questions for authors

If the authors could explain 1) what their MSA-conditioned model/ single-sequence models do that is fundamentally better than the existing MSA/single-sequence/conditioning label-based models in this space and 2) why? a better architecture, training setup, larger scale, etc. -- then I would be inclined to change my recommendation.

Please see my points above. The model seems quite solid and the empirical results are impressive. However, to recommend acceptance, I need clear framing around conceptual novelty.

3 comments

We thank the reviewer for the time and careful read, and for recognising the paper's strengths.

We agree that we did not make the paper's contribution explicit enough, and we have revised the manuscript throughout as we articulate below. Our manuscript makes two conceptual contributions:

1. The SFT vs. few-shot prompting experiments highlight the need for a practical shift for pLM-based protein design. SFT is currently the main technique for generating new proteins with pLMs: of the 12 studies that experimentally validated proteins generated by autoregressive pLMs, 8 used SFT, and, to our knowledge, proteins generated by few-shot prompting had not been tested experimentally before this work. This practice may follow the early (2020–2022) NLP notion of pre-training a foundation model and then fine-tuning it for each task. It was not entirely obvious whether is the best strategy, in NLP, controlled comparisons find that fine-tuning often matches or beats few-shot prompting, even with ~100 labelled examples (Pecher et al., 2025; Mosbach et al., 2023). For us to provide a fair comparison, we needed a prompted and a fine-tuned model with the same architecture, size and starting checkpoint (ProtGPT3-MSA was initialised from ProtGPT3-112M). We could have not run the comparison with public models, PoET, for instance, can be prompted but has no single-sequence counterpart to fine-tune and models larger than 7B are not publicly available, preventing us to study the effect of scale appropriately, which is why we also trained ProtGPT3-10B. Our results show that prompting outperforms SFT in low-data regimes, and that SFT only catches up once it has hundreds of training sequences - while requiring considerably more compute - . We have further confirmed the low-data result on 25 new families with no detectable homology to ProtGPT3-MSA's training MSAs (response to Reviewer Eqne). This opens the door to engineering rare families with pLMs: for the first time we are beeing able to use autoregressive pLMs to engineer rare-family nucleases (unpublished) and (de)halogenases. We envision our strategy will also enable engeneering industrially-relevant families for other teams as well.

In addition, the comparison revealed two results that go against common assumptions:

(a) Parameter scale does not close the SFT-prompting gap. We expected an SFT model with ~90× more parameters to compensate for the evolutionary information the MSA model reads from the prompt. However, SFT-10B improves only marginally over SFT-112M (Fig. 2a-b, Fig. 10b) despite its lower validation loss (1.77 vs 1.93), and the prompted 112M MSA model is the best of the three on 18–19 of the manuscript's 25 families on sequence-, structure- and profile-level metrics (p ≤ 0.043).

(b) More data lets SFT catch up, at a cost. In deeper families (≥ 500 sequences), 15 random prompt sequences perform on par with SFT on most metrics, whereas SFT needs hundreds to thousands of training sequences and a training run per family (Table 7, Fig. 9).

2. The prompt can be used to steer properties beyond family membership (Sec. 3.4 and 5). Prompting ProtGPT3-MSA with variants from a given stability percentile yields generations of matching predicted stability, without simply replicating the prompt sequences (Spearman 0.85; Fig. 1g). To our knowledge, homolog-conditioned pLMs have mainly been evaluated for family fidelity, not for steering a property through the choice of prompt. Concurrent work posted after our submission (Brixi et al., 2026) reports the same for Evo2: a few same-function sequences in the prompt steer it towards new sequences with similar function, and feeding its own generations back into the prompt improves diversity. Our inference-time procedure (Sec. 5) is the protein counterpart, developed in parallel: generations are scored in silico and the best become the new prompt, bringing inference-time compute to autoregressive pLMs, which are usually sampled once. We do not claim novelty for Feynman–Kac steering itself, which has mostly targeted diffusion models, but to our knowledge evolving the context of a homolog-conditioned pLM for inference-time steering had not been explored.

Please let us know if anything is unclear; we really appreciate your time and feedback.

References

Pecher, B., Srba, I., & Bielikova, M. (2025). Comparing specialised small and general large language models on text classification: 100 labelled samples to achieve break-even performance. EMNLP 2025, 165–184.

Mosbach, M., Pimentel, T., Ravfogel, S., Klakow, D., & Elazar, Y. (2023). Few-shot fine-tuning vs. in-context learning: A fair comparison and evaluation. Findings of ACL 2023, 12284–12314.

Brixi, G., Hao, Y. A., Nori, D., Merchant, A., Widatalla, T., King, S., ... & Hie, B. (2026). Efficient biological design via genomic in-context learning. bioRxiv.

Reviewer_rix7OpenReview ↗

Reviewer Response

Thank you for the response.

I'm taking the main contribution of the paper to be the SFT vs few-shot comparison. The finding that few short prompting performs well in this setting until we have hundreds of labelled examples is a valuable one, and the experimental results (very rare to see in a paper) are quite strong. However, I see this central finding as more appropriate for a biology-focused journal than a ML conf venue.

With regards to the model itself, it seems like it has similar capabilities compared to PoET. As such, I don't regard the paper's fundamental contribution as modelling/algorithmic in nature.

Thus, I retain my score.

Thank you for engaging with our response, and for finding the central result valuable and the experimental results strong.

We understand the concern, and partly share it: we also used to perceive computer science conferences as mainly a venue for contributions that are algorithmic in nature. However, we have noticed that over the years ICLR and other ML conferences have increasingly welcomed applied work in biology. In our line of research, ProGen3 and compute-optimal scaling of protein language models were published at NeurIPS 2025 (spotlight) and NeurIPS 2024, respectively (Bhatnagar et al., 2025; Cheng et al., 2024), and, like ours, both focused on applying language modelling to proteins. In line with this, this year's ICLR call for papers welcomes work on the "integration with scientific and humanistic fields outside the historical domain of ‘core ML’", listing biology as a topic.

We believe this is useful to many groups; with so many research groups and large companies focusing on training large language models for proteins and DNA, these results can inform such choices.

Thank you again for taking the time to read our response and share your thoughts; we really appreciate it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.