- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Propose a self-supervised learning scheme for protein mutation effect prediction combining ProteinMPNN, pair state formation, and transition ratio target. Evaluated at ddG datasets and some DMS assays.
Critical strengths
- EvoEpi (additive) -> EvoEpi performance improvement (Table 1)
- ProteinMPNN redesign of sequence to augment the dataset beyond WT. Ablated in Appendix C.2. Showed large performance gap. However, not sure if this is original.
Critical weaknesses
-
Benchmark is limited to double and single mutants. Even though training is designed for double and single mutants via pair state sampling, this doesn't mean inference is limited to those. If the model can only be used for double mutants it significantly reduces the impact of the paper, and readers will also be curious if it's just not evaluated in the paper or it is evaluated and didn't perform well.
-
Table3: Why draw only 4 assays from ProteinGym and why only compare with weaker models? I don’t see any recent models that are higher up in ProteinGym leaderboard? I pulled up their numbers and they outperform EvoEpi with large margin.
- RSALOR is parameter-free. It is additive baseline computed from MSA frequencies. (EvoEpi uses MSA training-time too.) also the fact that additive baseline outperforms EvoEpi on double mutant set weakens epistasis learning argument.
- ESM3 doesn't use MSA inference-time, isn't limited to single and double mutants, yet outperforms EvoEpi. Of course ESM3 is at larger parameter size, but the paper should at least transparently state the tradeoff between performance and parameter size, rather than just excluding them and saying "EvoEpi achieves a mean Spearman correlation of.25, compared with.23 for the strongest baseline.".
And authors don't even have to inference these models themselves, they could have downloaded prediction files from ProteinGym github and computed the correlation.
| Assay | Double mutants | EvoEpi, reported | Best baseline, recomputed | Other strong baseline |
|---|---|---|---|---|
| ppluGFP2 | 15,992 | 0.18 | RSALOR: 0.421 | ESCOTT: 0.378 |
| amacGFP | 11,260 | 0.18 | RSALOR: 0.431 | ProSST-4096: 0.400 |
| His3 | 1,475 | 0.15 | ESM3: 0.301 | VenusREM: 0.277 |
| KRas abundance | 22,946 | 0.40 | ProGen2 XL: 0.472 | ProSST-20: 0.438 |
Questions for authors
- Have you benchmarked beyond double mutants?
- Is the idea of ProteinMPNN redesign of sequence to augment the dataset beyond WT is original? I know some models use this augmentation, like ESM3 or protein design models, but is this original in this context at least? If so, I think it worth clarifying in the paper.
4 comments
W2. Assay selection and baseline comparisons in Table 3.
Assay selection. We thank the reviewer for highlighting the need to clarify both the assay selection and the scope of our comparisons. Our initial selection followed ThermoMPNN-D’s curation of six assays containing more than 1,000 double mutants and endpoints that could serve as proxies for thermodynamic stability. This curation excluded assays overlapping with Megascale and required either an experimental crystal structure or a high-confidence AlphaFold model.
From these six assays, we excluded two fluorescence assays, avGFP and cgreGFP, and included two non-ProteinGym assays, IRED and FLAb, to broaden the representation of biological functions. We acknowledge that omitting this selection procedure from the manuscript made the evaluation difficult to assess and could give the impression of selective reporting.
To address this concern, we expanded the evaluation to all ProteinGym assays meeting our eligibility criteria: more than 1,000 double mutants, no overlap with Megascale, and either an available crystal structure or an AlphaFold prediction with mean pLDDT $\ge$90 and at least 90% of the evaluated region having pLDDT $\ge$70. These criteria yield 12 assays, all of which are included; no additional selection was made based on phenotype or model performance. The table below reports Spearman correlations for each assay and their mean across the complete eligible set.
| Method (#params) | avGFP | cgreGFP | ppluGFP2 | amacGFP | His3 | KRAS abundance | KRAS binding (K55) | AAV2 capsid | PAB1 | PSD95-PDZ3 | YAP1 WW | GRB2 C-SH3 | Mean (12 assays) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Mutate Everything (657.88M) | 0.22 | 0.15 | 0.04 | 0.13 | 0.19 | 0.27 | 0.34 | 0.11 | 0.44 | 0.55 | 0.37 | 0.55 | 0.28 |
| SPURS (662.54M) | 0.40 | 0.29 | 0.13 | 0.16 | 0.25 | 0.29 | 0.29 | 0.16 | 0.47 | 0.47 | 0.36 | 0.58 | 0.32 |
| ThermoMPNN-D (1.78M) | 0.39 | 0.23 | 0.03 | 0.15 | 0.24 | 0.31 | 0.26 | 0.19 | 0.45 | 0.47 | 0.36 | 0.61 | 0.31 |
| PottsMPNN-FT (1.71M) | 0.42 | 0.29 | 0.08 | 0.16 | 0.26 | 0.36 | 0.39 | 0.38 | 0.41 | 0.67 | 0.39 | 0.72 | 0.38 |
| MSA frequency (N/A) | 0.12 | 0.17 | 0.04 | 0.03 | 0.15 | 0.17 | 0.18 | 0.13 | 0.33 | 0.14 | 0.25 | 0.22 | 0.16 |
| EvoRank (19.33M) | 0.36 | 0.38 | 0.17 | 0.17 | 0.14 | 0.32 | 0.16 | 0.29 | 0.25 | 0.37 | 0.16 | 0.42 | 0.27 |
| ProteinMPNN (1.66M) | 0.25 | 0.20 | 0.10 | 0.11 | 0.17 | 0.31 | 0.24 | 0.18 | 0.19 | 0.29 | 0.20 | 0.51 | 0.23 |
| ProteinMPNN+ref (1.66M) | 0.40 | 0.30 | 0.16 | 0.16 | 0.22 | 0.25 | 0.23 | 0.29 | 0.35 | 0.52 | 0.42 | 0.58 | 0.32 |
| PottsMPNN (1.71M) | 0.42 | 0.32 | 0.12 | 0.15 | 0.23 | 0.45 | 0.40 | 0.42 | 0.44 | 0.68 | 0.38 | 0.72 | 0.39 |
| Caliby (3.93M) | 0.38 | 0.27 | 0.08 | 0.12 | 0.13 | 0.36 | 0.37 | 0.29 | 0.32 | 0.53 | 0.32 | 0.68 | 0.32 |
| ESM3 (1.43B) | 0.43 | 0.29 | 0.06 | 0.13 | 0.30 | 0.28 | 0.56 | 0.23 | 0.65 | 0.67 | 0.44 | 0.64 | 0.39 |
| RSALOR (N/A) | 0.49 | 0.56 | 0.42 | 0.43 | 0.20 | 0.24 | 0.55 | 0.59 | 0.64 | 0.66 | 0.52 | 0.62 | 0.49 |
| EvoEpi (2.07M) | 0.39 | 0.32 | 0.17 | 0.17 | 0.15 | 0.40 | 0.41 | 0.50 | 0.33 | 0.61 | 0.33 | 0.65 | 0.37 |
| EvoEpi (add) (2.07M) | 0.38 | 0.32 | 0.18 | 0.18 | 0.16 | 0.40 | 0.41 | 0.50 | 0.35 | 0.61 | 0.34 | 0.65 | 0.37 |
| EvoEpi+RSA (2.07M) | 0.43 | 0.38 | 0.25 | 0.26 | 0.19 | 0.37 | 0.40 | 0.53 | 0.39 | 0.64 | 0.39 | 0.66 | 0.41 |
Scope of the generalization claim. These results do not establish that EvoEpi outperforms competing methods across all phenotypes. RSALOR achieves the highest mean correlation in this expanded comparison, and EvoEpi achieves a mean of 0.37. We therefore do not interpret these assays as evidence of a consistent advantage from joint scoring.
Our main evidence for the benefit of joint scoring comes from the stability benchmarks, including the higher-order mutation evaluations. The additional phenotype assays assess whether the learned scores remain useful beyond thermodynamic stability. We will revise the discussion accordingly and make clear that the original statement comparing mean correlations of 0.25 and 0.23 applied only to the assays and baselines evaluated in Table 3, rather than to the strongest available ProteinGym methods.
Comparison with RSALOR. We thank the reviewer for suggesting RSALOR, an important baseline missing from our original evaluation. Its strong performance on the expanded ProteinGym panel demonstrates that a training-free additive method can outperform EvoEpi on these assays. We additionally evaluated RSALOR on the stability benchmarks to examine how this comparison varies across prediction tasks. RSA-based calibration, inspired by RSALOR, also complements our approach, as demonstrated by the EvoEpi+RSA results below.
Each entry below reports global / per-protein Spearman correlation.
| Method (#params) | Megascale-S | Megascale-D | PTMul-D | PTMul $\ge$3 | ThermoMutDB-S | ThermoMutDB-D | ThermoMutDB $\ge$3 |
|---|---|---|---|---|---|---|---|
| RSALOR (N/A) | 0.53 / 0.58 | 0.30 / 0.24 | 0.21 / 0.10 | -0.12 / 0.22 | 0.36 / 0.32 | 0.12 / 0.21 | -0.08 / 0.27 |
| ESM3 (1.43B) | 0.51 / 0.60 | 0.13 / 0.34 | 0.33 / 0.23 | 0.17 / 0.18 | 0.41 / 0.39 | 0.22 / 0.25 | 0.16 / 0.31 |
| EvoEpi (2.07M) | 0.60 / 0.66 | 0.42 / 0.55 | 0.62 / 0.51 | 0.47 / 0.41 | 0.54 / 0.42 | 0.50 / 0.50 | 0.53 / 0.40 |
| EvoEpi+RSA (2.07M) | 0.62 / 0.68 | 0.44 / 0.55 | 0.64 / 0.46 | 0.53 / 0.40 | 0.56 / 0.40 | 0.53 / 0.52 | 0.56 / 0.42 |
EvoEpi outperforms RSALOR under both metrics on all reported stability benchmarks. Together, the two evaluations show that the relative strengths of these methods depend on the phenotype: RSALOR performs strongly across the expanded ProteinGym panel, while EvoEpi performs better on the evaluated stability tasks. Incorporating RSA-based calibration into EvoEpi further improves global Spearman correlation on every reported stability benchmark, although its effect on mean per-protein correlation is mixed. These results suggest that RSA provides complementary information that can enhance EvoEpi’s predictions without changing its training procedure.
Model size and larger baselines. Our original self-supervised comparisons focused on models of comparable scale to ProteinMPNN (1.66M parameters), as noted in line 355, to reduce differences in model scale when comparing training objectives and scoring procedures. The supervised comparison also included larger models, such as Mutate Everything and SPURS, with approximately 660M parameters. Following the reviewer’s suggestion, we now include ESM3 (1.43B parameters) in both evaluations and report parameter counts alongside predictive performance to make the comparison transparent.
We agree that comparisons among similarly sized models do not replace comparisons with stronger, larger predictors. Across the 12 ProteinGym assays, ESM3 achieves a mean Spearman correlation of 0.39, compared with 0.37 for EvoEpi and 0.41 for EvoEpi+RSA, both with 2.07M parameters. EvoEpi also outperforms ESM3 under both metrics on all reported stability benchmarks. These results clarify how predictive performance varies across tasks for models of different sizes, without attributing performance differences to parameter count alone. We will incorporate these comparisons and parameter counts into the revised manuscript and explicitly scope our claims to the evaluated methods and benchmarks for greater transparency.
Q2. Originality of the sequence augmentation strategy.
We thank the reviewer for encouraging us to clarify this distinction. Using generative models to redesign protein sequences is not new,as illustrated by ESM3, and we do not claim to introduce synthetic sequence augmentation broadly.
To our knowledge, the novel contribution in this context is the integration of structure-conditioned sequence augmentation with pairwise evolutionary supervision for mutation-effect prediction. We will clarify this distinction and acknowledge related uses of inverse-folding-generated sequences, attributing novelty to this specific combination of augmentation and supervision rather than to generative augmentation in protein modeling broadly.
[1] Tsishyn, M., Hermans, P., Rooman, M. and Pucci, F., 2025. Residue conservation and solvent accessibility are (almost) all you need for predicting mutational effects in proteins. Bioinformatics, 41(6), p.btaf322.
We kindly ask the reviewer to review our rebuttal. If it addresses your concerns, we would greatly appreciate your acknowledgement and the potential reconsideration of the score. We also welcome any remaining questions or concerns and would be happy to address them during the discussion period.
W1,Q1. Scope of EvoEpi beyond double mutants.
We sincerely thank the reviewer for raising this concern. We agree that the original evaluation was limited to single and double mutations. To examine whether EvoEpi extends beyond this setting, we evaluated the same checkpoint, without additional training, on higher-order mutation subsets of PTMul and ThermoMutDB using the following inference strategy.
Scoring higher-order mutations. Retaining the notation from Section 3, let s′ denote the full mutant sequence, I=ℓ:sℓ′=sℓ the mutated positions, and k=∣I∣≥3. For each pair i<j in I, we construct the context
\begin{cases} s_\ell, & \ell \in \\{i,j\\}, \\\\ s'_\ell, & \ell \notin \\{i,j\\}. \end{cases}$$ Thus, the other $k-2$ substitutions are present in the sequence context, while all unmutated residues remain WT. We then average the scores across the $\binom{k}{2}$ mutated pairs: $$\frac{2}{k(k-1)} \sum_{\substack{i,j\in\mathcal{I}\\\\ i < j}} g_{ij}\\!\left((s'_i,s'_j)\mid\mathbf{s}^{(ij)},\mathbf{x}\right).$$ Each pair is therefore scored in the context of the remaining substitutions. When (k=2), this formulation reduces to the original double-mutation score. **RSA calibration.** Inspired by the training-free mutational effect prediction method RSALOR [1] mentioned in the review, we additionally evaluate calibration based on relative solvent accessibility (RSA). For residue $i$, we define $$c_i = 1 - \min\left(\frac{\mathrm{RSA}_i}{100},1\right),$$ and calibrate each pair score as $$\frac{c_i+c_j}{2} \cdot g_{ij}(\mathbf{v}\mid\mathbf{s},\mathbf{x}).$$ The calibrated pair scores are then aggregated using the same higher-order scoring rule. **Higher-order mutation benchmarks and results.** We evaluate this strategy on filtered subsets of PTMul and ThermoMutDB with the following mutation-order distributions: | Number of substitutions | PTMul $\ge$ 3 | ThermoMutDB $\ge$ 3 | |---|---:|---:| | 3 | 139 | 156 | | 4 | 64 | 90 | | 5 | 16 | 24 | | 6 | 10 | 14 | | 7 | 5 | 6 | | 8 | 1 | 2 | | 9 | 3 | 4 | | 10 | 1 | 1 | | 17 | 0 | 1 | Each entry below reports global / per-protein Spearman correlation; (n) and (P) denote the numbers of variants and proteins, respectively. | Method | PTMul \$\ge\$3 | ThermoMutDB \$\ge\$3 | |---|---:|---:| | Metadata (n,P) | (239, 23) | (298, 32) | | Mutate Everything | 0.41 / 0.32 | 0.27 / 0.05 | | SPURS | 0.15 / 0.40 | 0.36 / 0.22 | | ThermoMPNN-D (pair extension) | 0.23 / 0.18 | 0.37 / 0.25 | | PottsMPNN-FT | 0.18 / 0.23 | 0.38 / 0.26 | | MSA frequency | -0.17 / -0.11 | -0.17 / 0.08 | | EvoRank | 0.35 / 0.09 | 0.50 / 0.23 | | ProteinMPNN | 0.41 / 0.35 | 0.35 / 0.30 | | ProteinMPNN+ref | 0.43 / 0.38 | 0.39 / 0.30 | | PottsMPNN | 0.26 / 0.26 | 0.34 / 0.28 | | Caliby | 0.34 / 0.22 | 0.41 / 0.28 | | RSALOR | -0.12 / 0.22 | -0.08 / 0.27 | | EvoEpi | 0.47 / 0.41 | 0.53 / 0.40 | | EvoEpi+RSA | 0.53 / 0.40 | 0.56 / 0.42 | | EvoEpi (add) | 0.36 / 0.34 | 0.45 / 0.25 | | EvoEpi+RSA (add) | 0.42 / 0.33 | 0.48 / 0.32 | For variants with three or more substitutions ($\ge$ 3), the EvoEpi models achieve the highest global and per-protein Spearman correlations on both benchmarks. EvoEpi+RSA is the only model to reach at least 0.50 global and 0.40 per-protein Spearman on both datasets. The largest gains over the competing baselines occur on ThermoMutDB, where additive RSALOR achieves 0.27 per-protein Spearman and the strongest competing ML baselines reach 0.30. EvoEpi and EvoEpi+RSA reach 0.40 and 0.42, respectively, corresponding to **48% and 56% improvements over RSALOR** and **33% and 40% improvements over the strongest competing ML baselines**. The matched ablation shows how much the epistatic terms contribute to these gains. Relative to EvoEpi (add), EvoEpi improves per-protein Spearman from 0.34 to 0.41 on PTMul (**+0.07, approximately 21% improvement**) and from 0.25 to 0.40 on ThermoMutDB (**+0.15, 60% improvement**). These gains show that the epistatic terms EvoEpi learns through self-supervised training on double mutants generalize to ranking variants with three or more substitutions on both benchmarks. These results supports that learning epistatic pairwise scoring via self-supervised learning can translate to zero-shot prediction of higher-order mutation effects. Considering residue pairs in the context of the remaining substitutions allows the same trained model to extend beyond double mutants, consistent with the broader motivation for modeling pairwise interactions in Potts models. The inference procedure follows naturally from the existing joint-scoring formulation rather than introducing an unrelated ad-hoc scoring rule.