- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Peppermint is a structure-conditioned encoder for interacting protein pairs. Initialized from ESM-C, it combines intra-chain self-attention, bidirectional cross-attention, local structural features, and inter-chain geometric biases. The same model accepts sequences with or without complex structures. Experiments show improved masked interface-residue recovery, with mixed gains on downstream interaction tasks.
Critical strengths
- A useful modeling goal. Learning partner-aware representations that accept optional complex structures is relevant to practical PPI analysis.
- A substantial implementation with a clear reconstruction benefit. Structural input improves interface recovery within the same checkpoint, and the paper reports several negative downstream results rather than claiming uniform gains.
Critical weaknesses
- The baseline coverage is incomplete. ProteinMPNN is included, but other relevant structure-conditioned models, such as ESM-IF1, MIF-ST, and PPIformer, are discussed without experimental comparisons under compatible protocols. Downstream experiments mainly compare general representations rather than task-specific interaction or mutation-effect models. The choice of ESM-C over other initializations is also insufficiently justified.
- The contribution of inter-chain geometry is unclear. In the small-model ablation, 3Di tokens reduce validation loss from 5.2049 to 5.1252, while adding cross-chain bias only reduces it to 5.1236. This does not establish the geometric modules’ value in the final model. A control that changes relative chain placement while preserving monomer structures would be more convincing.
- Genuine partner dependence is not directly tested. Masked residues may be recovered largely from their own chain. The paper lacks shuffled-partner controls and training-matched removal of cross-attention. Comparisons with ESM-C also confound the architecture with additional parameters, data, and training.
- Downstream benefits are inconsistent and narrowly evaluated. Interface recovery exceeds ProteinMPNN by 2.12 percentage points, but the difference is not significant and perplexity is worse. Structural input lowers COVID AUROC and worsens SKEMPI RMSE. Confidence filtering retains only 1.07% of Bernett and 22.46% of COVID rows, limiting broader conclusions. Complex-disjoint evaluation also needs clarification.
- Uncertainty and reproducibility remain incomplete. Recovery bootstrap intervals measure evaluation-sample variability rather than retraining variability. Downstream results lack consistent uncertainty estimates, and code is unavailable. This makes it difficult to assess the stability of small gains or verify the full training and evaluation pipeline.
- The writing needs clearer experimental takeaways. Frequent short paragraphs fragment the subsections, and results are often listed without a concise statement of what they establish. Figure 1 provides an overview, but the roles of the structural branches and their demonstrated contributions need clearer explanation.
Questions for authors
- How does Peppermint compare with ESM-IF1, MIF-ST, PPIformer, and relevant task-specific baselines under compatible inputs and evaluation protocols?
- Do gains persist with shuffled partners or perturbed relative chain poses? Can full-model ablations isolate cross-attention and inter-chain geometry?
- Are the downstream gains stable across seeds and complex-disjoint splits? Can code, checkpoints, and exact evaluation splits be provided?
5 comments
Structural input lowers COVID AUROC and worsens SKEMPI RMSE. Confidence filtering retains only 1.07% of Bernett and 22.46% of COVID rows, limiting broader conclusions.
Without confidence filtering, we retain all 1,052 COVID tests:
| Encoder/input | AUROC | AP |
|---|---|---|
| Peppermint, sequence | 0.678 | 0.531 |
| Peppermint, structure | 0.744 | 0.618 |
| MINT | 0.662 | 0.532 |
Structure improves AUROC by 0.065 (95% interval [0.019, 0.112]) over sequence input. Bernett keeps all 52,031 tests, using predicted complex structures for 48,217 and sequences for 3,814. AP rises from 0.545 to 0.563. The original 1.07% refers to the 2,933 Bernett rows passing our confidence thresholds across training, validation and test combined.
These tests use different training and test populations from the original filtered analysis, whose negative COVID and SKEMPI findings remain reported.
Added comparisons give PDBbind Spearman 0.306 versus MINT's 0.254 and TEIM contact AP 0.445 versus 0.459. Neither establishes a clear advantage over MINT.
Recovery bootstrap intervals measure evaluation-sample variability rather than retraining variability.
Are the downstream gains stable across seeds and complex-disjoint splits?
Recovery and COVID intervals measure test-set uncertainty with trained models fixed. The architecture and short geometry studies each use three training seeds to assess variation.
Earlier SKEMPI tests split individual mutation records. We now hold out groups of related sequences across 3,419 cases. Spearman is 0.097 for structural Peppermint and 0.073 for MINT, without a clear RMSE gain over the training mean. Transfer remains weak.
Can code, checkpoints, and exact evaluation splits be provided?
The code is included with our OpenReview submission and will be open sourced on GitHub after the review process.
The writing needs clearer experimental takeaways. Frequent short paragraphs fragment the subsections, and results are often listed without a concise statement of what they establish Figure 1 provides an overview, but the roles of the structural branches and their demonstrated contributions need clearer explanation.
We have grouped related results and stated more clearly their main takeaways. We distinguish local structural inputs, which describe each protein's shape, from geometric bias, which describes relative placement.
We would greatly appreciate if you could review our response above and share with us any comments or remaining concerns. Thank you for your consideration.
We added the requested baselines, partner and pose controls, training comparisons and complete-test evaluations. We also clarified the roles of structural inputs, data splits and reproducibility.
We compared the requested models on common recovery targets and PPB affinity tests:
Recovery uses identical masked positions on 249 complexes, with native input and decoding methods. PPB uses 340 common tests and the same fitting procedure.
Using the same trained model, autoregressive decoding fills one masked residue at a time, choosing the position with the lowest prediction uncertainty and recomputing predictions after each insertion.
IF1 has higher recovery. With simultaneous decoding, Peppermint's differences from MIF-ST and ProteinMPNN remain uncertain. ProteinMPNN has lower perplexity: 5.2 versus 5.7 for simultaneous Peppermint and 8.6 for autoregressive decoding.
The task-specific comparison uses supervised PPIformer on 170 measurements across four protein groups:
PPIformer has lower error averaged across groups.
Our initial project iterations used ESM-2, which is also the backbone used by MINT. We switched to ESM-C after observing better sequence recovery and downstream performance when starting training from ESM-C instead of ESM-2. Matched full-scale comparisons between the two initializations were beyond our available compute resources.
The trained model's recovery falls when we disrupt its partner or structural input:
Wrong partners and shuffled residues reduce recovery. Rotating one chain preserves both proteins' shapes but changes relative placement and reduces recovery. Moving both chains together has negligible effect.
The added comparison uses the same data and training budget, with about 18M parameters and three runs per design:
The paired model has the highest mean score. Removing cross-attention after training measures the model's reliance on it.
The original ablation shows that 3Di produces most of the loss reduction. A short training study finds a larger geometry effect on PDZ3 double-mutant ranking with 3Di (difference 0.045, adjusted 98.33% interval [0.016, 0.074]). RMSE rises from 0.287 to 0.367. Docking and interpolation effects remain uncertain.
During the rebuttal period, we prioritized controlled comparisons across three seeds and direct partner and pose interventions. These test how the model uses interaction information and how components affect short training runs. We were unable to dedicate the substantial compute resources needed for additional full training runs.