- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes MuseDrift, a conditional discrete diffusion model for generating protein variants with a specified sequence identity to a wild-type (WT) protein. The authors construct a large training dataset of 38.2 million WT–homolog pairs and train a masked-token denoising model that combines WT sequence context, frozen ESM-2 representations, and identity-aware FiLM conditioning. During inference, the model iteratively generates variants while learning which positions to mutate and which amino acids to substitute.
Critical strengths
Conditioning protein variant generation directly on a target sequence identity is an interesting idea. The paper also provides extensive experiments across different identity levels, evaluating structural confidence, structural similarity, sequence diversity, evolutionary patterns, and predicted fitness and energy.
Critical weaknesses
-
The model is trained using MMseqs2 local-alignment identity but evaluated using full-length Hamming identity. These two definitions can differ substantially, especially for partial alignments and sequences containing insertions or deletions. Surprisingly, the model still achieves highly accurate identity control, but the paper does not explain why this transfer works so well or quantify the discrepancy between the two metrics in the training data. More importantly, I question whether Hamming identity alone is a sufficiently meaningful measure of evolutionary distance for protein engineering. It treats all amino acid substitutions equally, even though mutations at highly conserved or functionally important positions can have much larger effects than mutations at flexible or variable positions. Similarly, conservative and chemically disruptive substitutions contribute equally to the distance. Although the model may learn some of these preferences from evolutionary data, the identity condition itself does not capture them. The authors should better justify this choice and investigate whether more biologically informative measures could provide more meaningful control over protein variation.
-
The paper mainly evaluates generated variants using structural confidence, similarity to the WT structure, and computational fitness or energy scores. However, as the target identity decreases and more mutations are introduced, pLDDT and predicted fitness scores consistently become worse, while FoldX energy changes become less favorable. Although MuseDrift performs better than the baselines at the same identity levels, the results largely show that its mutations are less damaging, rather than that they provide any actual benefit over the WT. This raises a fundamental question about the practical value of the proposed method. If variants closer to the WT consistently achieve better scores, why not simply retain the original WT sequence? While generating diverse variants is a reasonable objective, the paper does not demonstrate how this diversity translates into useful improvements in protein engineering. For example, there is no evidence that the generated variants achieve better activity, stability, solubility, or other desirable properties than the WT.
Questions for authors
- Could the authors clarify why MMseqs2 local-alignment identity is used during training, while full-length Hamming identity is used during inference? How does this definition shift affect the learned conditioning? Moreover, the predicted structural confidence and fitness scores generally deteriorate as target identity decreases. Although MuseDrift performs better than the baselines, what practical benefits do the generated mutations offer over simply retaining the WT sequence? Could the authors provide evidence that the generated variants improve at least some desirable protein properties, even at the expense of others?
- Most of the evaluated baselines are relatively simple or based on older models. Could the authors consider comparisons with more recent methods specifically designed for protein evolution or reference-conditioned sequence editing, such as EvoFlows, PEINT, or DPLM-Evo?
3 comments
Comparisons with Newer Models (Part 2)
We added ESM3, ProteinMPNN, and PEINT. Official checkpoints for EvoFlows and DPLM-Evo are not publicly available.
5. ESM3 and ProteinMPNN. MuseDrift generates variants from a WT sequence and target identity, without a structure template. Including its frozen WT encoder, MuseDrift has 128.5M parameters (95M trainable), compared with ESM3's 1.4B.
We evaluated the first sample per WT and target on 429 WTs at T=1.0. MuseDrift uses its original sampler without enforcing exact counts. ESM3 and ProteinMPNN choose sites using their own 1 - p(WT residue), then exclude WT residues at those sites to meet an exact edit count. ESM3 receives only the WT sequence; ProteinMPNN also receives the ESMFold-predicted WT backbone.
Differences are MuseDrift minus baseline; brackets give 95% paired WT-bootstrap CIs (20,000 draws).
| Target | pLDDT vs ESM3 | TM-score vs ESM3 | pLDDT vs ProteinMPNN |
|---|---|---|---|
| 0.95 | +0.3 [+0.05, +0.6] | +0.013 [+0.006, +0.019] | 0.0 [-0.3, +0.2] |
| 0.75 | +1.2 [+0.5, +1.9] | +0.026 [+0.017, +0.036] | -0.3 [-0.8, +0.2] |
| 0.55 | 0.0 [-1.0, +1.0] | +0.032 [+0.020, +0.044] | -1.5 [-2.2, -0.8] |
| 0.30 | -3.7 [-4.9, -2.5] | +0.008 [-0.005, +0.021] | -1.6 [-2.8, -0.4] |
MuseDrift has higher TM-score than ESM3 at 0.95, 0.75, and 0.55, but lower pLDDT at 0.30. ProteinMPNN has higher pLDDT at 0.55 and 0.30; its mean TM-score is also higher there (differences -0.026/-0.035).
Sequence quality. With all three models using the same exact-edit decoder at T=1.0, MuseDrift has higher mean homolog-column support at all four targets (354 WTs). Mean differences across these targets are 0.0072-0.0216 versus ESM3 and 0.0135-0.0424 versus ProteinMPNN. Support counts substitutions observed in natural homologs and is related to our training objective.
ProSST-2048 (348 WTs) favors ESM3 at every target. Compared with ProteinMPNN, MuseDrift has higher mean scores at 0.95/0.75 and lower scores at 0.30; the difference at 0.55 is not significant. These are predicted mutation effects, not measured fitness.
6. PEINT. We ran the official code/checkpoint with default sampling on 429 WTs and the same ESMFold settings. We calibrated its evolutionary-time input to identity on 30 of the 438 CAMEO WTs, using identity only.
| Target | MuseDrift identity | PEINT identity | PEINT length differs | pLDDT vs PEINT | TM-score vs PEINT |
|---|---|---|---|---|---|
| 0.95 | 0.9500 | 0.906 | 48% | +0.67 [+0.27, +1.09] | +0.031 [+0.023, +0.040] |
| 0.75 | 0.7518 | 0.742 | 76% | +0.15 [-0.48, +0.80] | +0.029 [+0.018, +0.040] |
| 0.55 | 0.5515 | 0.537 | 89% | +0.17 [-0.65, +0.98] | +0.033 [+0.022, +0.044] |
| 0.30 | 0.3045 | 0.265 | 98% | +9.45 [+8.08, +10.87] | +0.131 [+0.115, +0.147] |
Identities are means; length changes are relative to WT. TM-score is WT-length normalized. At 0.95/0.75, one three-residue PEINT output per target could not be TM-scored (n=428); assigning zero leaves the conclusions unchanged. MuseDrift has higher TM-score throughout, but achieved identities and lengths differ, so this is not a strictly matched comparison.
Sampling affects the ranking. Greedy ESM3 gives pLDDT 84-85 across targets on 429 WTs, above MuseDrift. PEINT with nucleus sampling (p=0.90; 150 WTs) reaches identity 0.964 at target 0.95 and exceeds MuseDrift in pLDDT (-0.53 [-0.90, -0.16]). On 354 WTs, MuseDrift's exact-count decoder scores below default PEINT at 0.75 (78.7 vs 81.1) and 0.55 (72.1 vs 76.5).
We will add these comparisons and present MuseDrift as a sequence-based library generator, with quality depending on the target, metric, and sampler.
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to quantify the identity mismatch, validate identity control, and include stronger baselines. Some important issues remain:
-
Positioning MuseDrift as a library generator clarifies its intended use, but does not establish its practical advantage. The new analyses show nonrandom mutation preferences, but not improved functional properties or screening outcomes. More importantly, the predicted benefits over WT remain limited: no above-WT variants were found at identities of 0.75 or below in the original evaluation.
-
The new comparisons reveal mixed results across metrics and identity levels. Furthermore, MuseDrift uses its original sampler while ESM3 and ProteinMPNN enforce exact mutation counts, even though the authors' own ablations show that exact-count decoding can substantially reduce pLDDT. This raises concerns about whether the reported improvements reflect the model itself or differences in decoding constraints.
I am therefore inclined to maintain my rating.
Identity Control and Practical Use (Part 1)
Thank you for the review. We measured the identity mismatch, checked mutation-count control, and clarified the intended use.
1. Why different identity measures? Training uses the MMseqs2 identity labels from our homolog-pair dataset. Generation produces substitutions in a full WT sequence, so we use full-length Hamming identity to count the changes.
We checked 2.91M held-out pairs. We measured identity over the full WT, counting unaligned WT positions as mismatches. For gapped pairs, this is an alignment-based measure, not positional Hamming identity.
The labels are higher by 0.046 on average; the mean absolute difference is 0.053. The measures are strongly correlated, but correlation alone does not guarantee accurate control.
2. Does identity control work? We tested the original sampler on 438 CAMEO WTs, with 64 variants per WT per target.
This shows approximate control directly on generated sequences. We also tested an exact-count decoder on 354 WTs. It always meets the requested edit count, but lowers pLDDT at the lower identity targets.
CIs use paired WT bootstrap (20,000 draws). Exact-count decoding is optional; the main structural results use the original sampler.
3. What does identity mean biologically? It controls the number of substitutions. For example, identity 0.95 on a 300-residue protein means 15 substitutions. It does not measure how each change affects function. The model learns site and residue choices from homolog pairs.
At target 0.95, a position-matched comparison under the shared exact-edit decoder gives mean BLOSUM62 scores of +0.43 for MuseDrift, -0.41 for ESM3, and -0.48 for ProteinMPNN. MuseDrift's exact-count decoder places 3.8% of edits in conserved columns, versus 9.5% for random sites. These results show nonrandom mutation choices, not proven functional gains. Control by conservation-weighted distance remains untested.
4. Why generate variants instead of keeping WT? MuseDrift builds candidate libraries from a WT sequence and target identity, without needing a structure template. Researchers can choose how far to explore and then screen the variants for a desired property. This serves the library-generation step of directed evolution (Packer & Liu, 2015; Romero & Arnold, 2009).
We have not demonstrated improved experimental activity, stability, solubility, or screening success. At identity 0.95, ProSST-4096 predicts at least one variant above WT among 64 samples for 15.9% of WTs, versus 1.6-7.2% for the original baselines. At targets 0.75 and below, the original additive ProSST/VenusREM evaluation found no variants above WT (0/27,648 per method-target cell).
In a separate common-decoder evaluation at 0.95, the predicted above-WT fractions favor ESM3 (0.88%) and ProteinMPNN (0.81%) over MuseDrift (0.25%). These are computational predictions from different evaluation protocols; they do not establish experimental improvement.
References: Packer & Liu (2015), Nat Rev Genet 16:379-394; Romero & Arnold (2009), Nat Rev Mol Cell Biol 10:866-876.