- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Suggest EP a pretraining transformer model based on electrostatic potential data for protein representations. They have better results on interaction centric tasks.
Critical strengths
EP is an efficient model, the results seem promising, and the electrostatic potential target is well motivated.
Critical weaknesses
- Because EP receives full atom coordinates and partial charges, whereas the baselines are sequence models with at most structure aware token vocabularies and no pretrained structure encoder is compared, the gains cannot be attributed to the electrostatic objective rather than to richer inputs.
- The paper attributes PLM performance partly to test set leakage, yet EP is pretrained on nearly the entire PDB, which likely includes the downstream test structures too.
Questions for authors
- How do representations from the same EP architecture and inputs perform when randomly initialized, and when pretrained with a non electrostatic self supervised objective?
- How does EP perform when partial charges and radii are removed from the input, so that any electrostatic information must be learned from atom identity and geometry alone?
6 comments
2. Pretraining overlap with downstream test sets
The paper attributes PLM performance partly to test set leakage, yet EP is pretrained on nearly the entire PDB, which likely includes the downstream test structures too.
Thank you for raising this concern. It is true that EP is pretrained on nearly the entire RCSB database and contains significant overlap. We have revised that claim in the paper and now include a section measuring sequence similarity between EP, SaProt, and ESM-2 pretraining datasets and the downstream tasks test splits using MMseqs2 search algorithm at 90%, 85%, and 80% residue identity with a minimum of. at 90% residue identity, overall similarity is 92.04%, 93.61%, and 70.82% for EP, ESM-2 and SaProt respectively. That is, we have less overlap than our main comparison, ESM, although we acknowledge that SaProt is better controlled. We will include this analysis in the updated paper.
Table 5: Sequence similarity between downstream test sets and pretraining databases for EP, SaProt, and ESM-2. Sequences are pooled across all downstream tasks
| Corpus | ≥90% identity | ≥85% identity | ≥80% identity |
|---|---|---|---|
| ESM-2 | 93.61% | 96.48% | 98.53% |
| SaProt | 70.82% | 76.13% | 81.14% |
| EP | 92.04% | 92.71% | 94.11% |
Moreover, in the section 4.1.3 "Pretraining Efficiency on a Shared Protein Corpus" experiment, we specifically controlled for test set leakage by pre-training EP, SaProt 35M, and ESM-2 8M architectures from scratch on the union of downstream training split PDBs obtained from the ProteinShake tasks, with any test or validation set PDBs removed. Under these controls, EP outperformed ESM-2 8M and SaProt 35M on interaction centric tasks at 24 hours of pretraining, where EP beats the second-best model, SaProt 35M, on binding-site prediction, LBA, and PPI. Moreover, the performance gap previously seen in enzyme class and gene ontology prediction shrank. More training time constraint results can be found in Appendix Section E Figure 4.
Matched-corpus pretraining downstream evaluation on ProteinShake tasks after 24 hours. Values are mean ± sample standard deviation over five probe seeds. Bold indicates the highest mean in each row.
| Task / metric | Electro-Prot | ESM-2 8M | SaProt 35M |
|---|---|---|---|
| Enzyme classification (accuracy ↑) | 0.644 ± 0.008 | 0.576 ± 0.011 | 0.642 ± 0.023 |
| Gene ontology (Fmax ↑) | 0.295 ± 0.003 | 0.307 ± 0.001 | 0.381 ± 0.002 |
| Structural classification (accuracy ↑) | 0.249 ± 0.016 | 0.403 ± 0.009 | 0.694 ± 0.002 |
| Protein family (accuracy ↑) | 0.250 ± 0.001 | 0.376 ± 0.011 | 0.562 ± 0.003 |
| Structure similarity (ρ ↑) | 0.536 ± 0.006 | 0.433 ± 0.011 | 0.578 ± 0.013 |
| Binding-site prediction (MCC ↑) | 0.281 ± <0.001 | 0.161 ± 0.001 | 0.240 ± <0.001 |
| Ligand-binding affinity (ρ ↑) | 0.349 ± 0.024 | 0.283 ± 0.054 | 0.322 ± 0.029 |
| PPI interface prediction (mean AUROC ↑) | 0.800 ± 0.003 | 0.604 ± 0.005 | 0.712 ± 0.002 |
3. Random initialization and alternative objectives
How do representations from the same EP architecture and inputs perform when randomly initialized, and when pretrained with a non electrostatic self supervised objective?
Thank you for suggesting these controls. We measured 3 randomly initialized untrained EP models and compared them with electrostatically pretrained EP. EP outperforms randomly initialized controls on 7/8 Proteinshake tasks, SKEMPI 2.0 mutation-induced ∆∆G predictions, and 4/5 PPB-Affinity ∆G predictions. We also included the previously mentioned residue reconstruction ablation experiment as an alternative objective for EP's architecture and inputs in section 1.
Table 6: Random-initialization control for Electro-Prot. Three independently initialized, untrained EP models are compared with electrostatically pretrained EP, using our linear probe evaluation protocol. Each entry reports the mean ± sample standard deviation across five seeds for each initialization.
| Task / metric | Randomly initialized EP 1 | Randomly initialized EP 2 | Randomly initialized EP 3 | Pretrained EP |
|---|---|---|---|---|
| ProteinShake | ||||
| Enzyme class (accuracy ↑) | .632 ± .013 | .629 ± .012 | .629 ± .010 | .632 ± .009 |
| GO (Fmax ↑) | .259 ± .002 | .275 ± .006 | .264 ± .005 | .282 ± .003 |
| Structural class (accuracy ↑) | .208 ± .007 | .205 ± .005 | .221 ± .004 | .295 ± .003 |
| Protein family (accuracy ↑) | .199 ± .004 | .229 ± .007 | .199 ± .007 | .249 ± .006 |
| Structure similarity (ρ ↑) | .456 ± .022 | .443 ± .011 | .451 ± .007 | .559 ± .005 |
| PPI (AUROC ↑) | .574 ± .003 | .600 ± .006 | .573 ± .004 | .812 ± .001 |
| Binding site (MCC ↑) | .242 ± .001 | .281 ± .0004 | .270 ± .001 | .323 ± .002 |
| Ligand affinity (ρ ↑) | .365 ± .019 | .363 ± .006 | .367 ± .012 | .350 ± .022 |
| SKEMPI 2.0: mutation-induced ΔΔG | ||||
| Pearson r ↑ | .288 ± .010 | .274 ± .012 | .279 ± .011 | .446 ± .017 |
| Spearman ρ ↑ | .296 ± .009 | .274 ± .009 | .303 ± .011 | .369 ± .009 |
| MAE ↓ | 1.512 ± .022 | 1.527 ± .017 | 1.509 ± .017 | 1.348 ± .010 |
| RMSE ↓ | 2.134 ± .047 | 2.126 ± .034 | 2.167 ± .038 | 1.889 ± .017 |
| PPB-Affinity: ΔG MAE (kcal/mol ↓) | ||||
| Overall | 2.225 ± .052 | 2.090 ± .039 | 1.952 ± .012 | 2.174 ± .040 |
| ATLAS | 1.120 ± .283 | 1.285 ± .281 | 1.463 ± .345 | 1.043 ± .164 |
| Affinity Benchmark v5.5 | 1.905 ± .176 | 1.702 ± .149 | 1.643 ± .279 | 2.378 ± .126 |
| PDBbind v2020 | 1.810 ± .022 | 1.824 ± .031 | 1.816 ± .007 | 1.840 ± .010 |
| SAbDab | 1.612 ± .099 | 1.577 ± .050 | 1.529 ± .020 | 1.442 ± .068 |
| SKEMPI v2.0 | 3.115 ± .191 | 2.749 ± .297 | 3.174 ± .271 | 2.487 ± .320 |
We thank the reviewer for recognizing the efficiency of Electro-Prot and the motivation for electrostatic pretraining. We address the concerns about controlled comparisons and pretraining overlap below.
1. Richer inputs and structure-based baselines
We agree that the effects of input information and pretraining objective should be distinguished. We now compare Electro-Prot to an alternative approach using the same network architecture and training set with a masked reconstruction loss. Compared to this purely structural training, Electro-Prot embeddings pretrained on electrostatics outperform reconstruction on 7/8 Proteinshake tasks, SKEMPI ∆∆G prediction, and 4/5 PPB-Affinity ∆G prediction datasets. The largest gains are observed in interaction centric tasks such as BS, PPI, PFam, and SKEMPI ΔΔG prediction. These results suggest that EP's performance on downstream tasks can be attributed to electrostatic supervision rather than the architecture's rich physical inputs like partial charges and radii The masked reconstruction objective ablation shows that electrostatic pretraining generally outperformed residue reconstruction on PPI, LBA, mutation induced ΔΔG predictions, with slightly lower overall ΔG predictions, which is driven by soley Affinity Benchmark v5.5. Showcasing that electrostatic predictions allows the models architecture to learn interaction centric tasks. This aligns with electrostatics roles in binding, recognition, and complex long-range interactions in the proteins. Electrostatic supervision also outperformed masked residue reconstruction on some evolutionary based tasks like GO and Pfam, although the benefits are much smaller, further aligning with our observed limitations of electrostatic representations contribution to those tasks.
Table 11: Electrostatic supervision versus masked-residue reconstruction on EP. Values are mean ± sample standard deviation over five probe seeds. Bold indicates the better mean in each row, and Δ = Reconstruction − EP.