acceptodds
Under review as a conference paper at ICLR 2027

Peppermint: structure-conditioned representation learning for interacting protein pairs

Abstract

Protein-protein interactions depend on the sequences of both proteins and their spatial arrangement, motivating protein representations that incorporate molecular context. We introduce Peppermint, a structure-conditioned masked language model for interacting protein pairs. Peppermint maintains separate chain representations and alternates intra-chain self-attention with bidirectional cross-attention, preserving chain identity while allowing each residue to incorporate information from the other chain. The architecture assigns distinct roles to local conformation and interaction geometry: gated embeddings of discrete structural states and continuous backbone geometry enrich residue representations, while directional geometric biases guide attention between chains. Initialized from a pretrained sequence encoder, the model learns from paired sequences and protein complexes through masked amino-acid modeling and auxiliary structural prediction. Mixing examples with and without structure allows the same model to operate under either input condition. Supplying structure improves simultaneous recovery of masked interface residues by 9.3 percentage points relative to sequence-only input, with downstream benefits varying by task and metric. These results support incorporating complex geometry into paired protein representations while retaining sequence-only inference within the same encoder.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein language models, protein–protein interactions, representation learning, multimodal learning, masked language modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5f5YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Peppermint is a structure-conditioned encoder for interacting protein pairs. Initialized from ESM-C, it combines intra-chain self-attention, bidirectional cross-attention, local structural features, and inter-chain geometric biases. The same model accepts sequences with or without complex structures. Experiments show improved masked interface-residue recovery, with mixed gains on downstream interaction tasks.

Critical strengths
  1. A useful modeling goal. Learning partner-aware representations that accept optional complex structures is relevant to practical PPI analysis.
  2. A substantial implementation with a clear reconstruction benefit. Structural input improves interface recovery within the same checkpoint, and the paper reports several negative downstream results rather than claiming uniform gains.
Critical weaknesses
  1. The baseline coverage is incomplete. ProteinMPNN is included, but other relevant structure-conditioned models, such as ESM-IF1, MIF-ST, and PPIformer, are discussed without experimental comparisons under compatible protocols. Downstream experiments mainly compare general representations rather than task-specific interaction or mutation-effect models. The choice of ESM-C over other initializations is also insufficiently justified.
  2. The contribution of inter-chain geometry is unclear. In the small-model ablation, 3Di tokens reduce validation loss from 5.2049 to 5.1252, while adding cross-chain bias only reduces it to 5.1236. This does not establish the geometric modules’ value in the final model. A control that changes relative chain placement while preserving monomer structures would be more convincing.
  3. Genuine partner dependence is not directly tested. Masked residues may be recovered largely from their own chain. The paper lacks shuffled-partner controls and training-matched removal of cross-attention. Comparisons with ESM-C also confound the architecture with additional parameters, data, and training.
  4. Downstream benefits are inconsistent and narrowly evaluated. Interface recovery exceeds ProteinMPNN by 2.12 percentage points, but the difference is not significant and perplexity is worse. Structural input lowers COVID AUROC and worsens SKEMPI RMSE. Confidence filtering retains only 1.07% of Bernett and 22.46% of COVID rows, limiting broader conclusions. Complex-disjoint evaluation also needs clarification.
  5. Uncertainty and reproducibility remain incomplete. Recovery bootstrap intervals measure evaluation-sample variability rather than retraining variability. Downstream results lack consistent uncertainty estimates, and code is unavailable. This makes it difficult to assess the stability of small gains or verify the full training and evaluation pipeline.
  6. The writing needs clearer experimental takeaways. Frequent short paragraphs fragment the subsections, and results are often listed without a concise statement of what they establish. Figure 1 provides an overview, but the roles of the structural branches and their demonstrated contributions need clearer explanation.
Questions for authors
  1. How does Peppermint compare with ESM-IF1, MIF-ST, PPIformer, and relevant task-specific baselines under compatible inputs and evaluation protocols?
  2. Do gains persist with shuffled partners or perturbed relative chain poses? Can full-model ablations isolate cross-attention and inter-chain geometry?
  3. Are the downstream gains stable across seeds and complex-disjoint splits? Can code, checkpoints, and exact evaluation splits be provided?
5 comments

We added the requested baselines, partner and pose controls, training comparisons and complete-test evaluations. We also clarified the roles of structural inputs, data splits and reproducibility.

How does Peppermint compare with ESM-IF1, MIF-ST, PPIformer, and relevant task-specific baselines under compatible inputs and evaluation protocols?

We compared the requested models on common recovery targets and PPB affinity tests:

Encoder/inputRecovery, %PPB SpearmanPPB RMSE
Peppermint, sequence41.20.5031.480
Peppermint, structure50.50.4911.440
Peppermint, structure, autoregressive decoding51.7--
MINT40.50.5101.516
SaProt47.60.4571.537
MIF-ST50.80.4511.433
ESM-IF1, native55.80.4031.528
PPIformer SSL27.80.4021.522
ProteinMPNN48.4--

Recovery uses identical masked positions on 249 complexes, with native input and decoding methods. PPB uses 340 common tests and the same fitting procedure.

Using the same trained model, autoregressive decoding fills one masked residue at a time, choosing the position with the lowest prediction uncertainty and recomputing predictions after each insertion.

IF1 has higher recovery. With simultaneous decoding, Peppermint's differences from MIF-ST and ProteinMPNN remain uncertain. ProteinMPNN has lower perplexity: 5.2 versus 5.7 for simultaneous Peppermint and 8.6 for autoregressive decoding.

The task-specific comparison uses supervised PPIformer on 170 measurements across four protein groups:

ModelSpearmanPearsonRMSEMean group RMSE
Peppermint, structure0.6070.3814.1453.068
Supervised PPIformer0.4990.5503.8192.181

PPIformer has lower error averaged across groups.

The choice of ESM-C over other initializations is also insufficiently justified.

Our initial project iterations used ESM-2, which is also the backbone used by MINT. We switched to ESM-C after observing better sequence recovery and downstream performance when starting training from ESM-C instead of ESM-2. Matched full-scale comparisons between the two initializations were beyond our available compute resources.

Do gains persist with shuffled partners or perturbed relative chain poses?

The trained model's recovery falls when we disrupt its partner or structural input:

InterventionRecovery loss, percentage points
Wrong partner1.20
Shuffle visible partner residues0.54
Cross-attention updates removed1.28
Relative 90° rotation2.05
Geometric bias removed1.37
Both chains moved together0.01

Wrong partners and shuffled residues reduce recovery. Rotating one chain preserves both proteins' shapes but changes relative placement and reduces recovery. Moving both chains together has negligible effect.

Can full-model ablations isolate cross-attention and inter-chain geometry?

Comparisons with ESM-C also confound the architecture with additional parameters, data, and training.

The added comparison uses the same data and training budget, with about 18M parameters and three runs per design:

ArchitectureSKEMPI Spearman, mean ± SD
Paired0.448 ± 0.024
Independent chains0.413 ± 0.011
Concatenated sequences0.416 ± 0.006

The paired model has the highest mean score. Removing cross-attention after training measures the model's reliance on it.

The original ablation shows that 3Di produces most of the loss reduction. A short training study finds a larger geometry effect on PDZ3 double-mutant ranking with 3Di (difference 0.045, adjusted 98.33% interval [0.016, 0.074]). RMSE rises from 0.287 to 0.367. Docking and interpolation effects remain uncertain.

During the rebuttal period, we prioritized controlled comparisons across three seeds and direct partner and pose interventions. These test how the model uses interaction information and how components affect short training runs. We were unable to dedicate the substantial compute resources needed for additional full training runs.

Structural input lowers COVID AUROC and worsens SKEMPI RMSE. Confidence filtering retains only 1.07% of Bernett and 22.46% of COVID rows, limiting broader conclusions.

Without confidence filtering, we retain all 1,052 COVID tests:

Encoder/inputAUROCAP
Peppermint, sequence0.6780.531
Peppermint, structure0.7440.618
MINT0.6620.532

Structure improves AUROC by 0.065 (95% interval [0.019, 0.112]) over sequence input. Bernett keeps all 52,031 tests, using predicted complex structures for 48,217 and sequences for 3,814. AP rises from 0.545 to 0.563. The original 1.07% refers to the 2,933 Bernett rows passing our confidence thresholds across training, validation and test combined.

These tests use different training and test populations from the original filtered analysis, whose negative COVID and SKEMPI findings remain reported.

Added comparisons give PDBbind Spearman 0.306 versus MINT's 0.254 and TEIM contact AP 0.445 versus 0.459. Neither establishes a clear advantage over MINT.

Recovery bootstrap intervals measure evaluation-sample variability rather than retraining variability.

Are the downstream gains stable across seeds and complex-disjoint splits?

Recovery and COVID intervals measure test-set uncertainty with trained models fixed. The architecture and short geometry studies each use three training seeds to assess variation.

Earlier SKEMPI tests split individual mutation records. We now hold out groups of related sequences across 3,419 cases. Spearman is 0.097 for structural Peppermint and 0.073 for MINT, without a clear RMSE gain over the training mean. Transfer remains weak.

Can code, checkpoints, and exact evaluation splits be provided?

The code is included with our OpenReview submission and will be open sourced on GitHub after the review process.

The writing needs clearer experimental takeaways. Frequent short paragraphs fragment the subsections, and results are often listed without a concise statement of what they establish Figure 1 provides an overview, but the roles of the structural branches and their demonstrated contributions need clearer explanation.

We have grouped related results and stated more clearly their main takeaways. We distinguish local structural inputs, which describe each protein's shape, from geometric bias, which describes relative placement.

We would greatly appreciate if you could review our response above and share with us any comments or remaining concerns. Thank you for your consideration.

Reviewer_3LQPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds on MINT and extends ESM-C to structure-conditioned modeling of protein pairs. The proposed model combines cross-chain attention with geometric features to improve protein-complex representations. Structural inputs improve interface sequence recovery, while gains across downstream tasks and metrics are mixed.

Critical strengths
  • The paper is well written and easy to follow, with a clear description of the proposed method and its components.
Critical weaknesses
  • The claim of improved representations is not sufficiently supported. Sequence recovery primarily measures reconstruction performance, which does not necessarily translate into more useful representations. Downstream evaluation should therefore be prioritized. At a minimum, the evaluation should cover the full set of relevant downstream benchmarks used in MINT, including inter-chain contact prediction and absolute binding affinity prediction, with direct comparisons against MINT under matched evaluation protocols.
Questions for authors

Q1: Could you provide a more comprehensive downstream evaluation to address the limitations described above?

Q2: Is explicit cross-attention necessary, or could it introduce a modeling bottleneck compared with simple sequence concatenation?

Q3: Given the importance of scaling in representation learning, could you analyze how downstream performance changes with model scale?

Q4: Structural inputs provide additional information that can simplify prediction. Could you compare against strong structure-aware baselines, such as SaProt, to distinguish the benefits of the proposed architecture from those of the additional structural features?

Q5: The paper acknowledges that generalization to unfamiliar complex families remains unverified. Could you evaluate this setting? Demonstrating transfer beyond familiar interactions is important for establishing the model's practical utility.

4 comments

Thank you for emphasizing downstream usefulness. We expanded coverage of the MINT benchmarks and added controlled tests of architecture, model size and transfer.

Could you provide a more comprehensive downstream evaluation to address the limitations described above?

At a minimum, the evaluation should cover the full set of relevant downstream benchmarks used in MINT, including inter-chain contact prediction and absolute binding affinity prediction, with direct comparisons against MINT under matched evaluation protocols.

We now evaluate the main MINT benchmark families, including absolute affinity and inter-chain contacts:

Benchmark / metricMINTPeppermint sequenceStructure where available
Bernett / AP0.5270.5450.563
YeastPPI / accuracy0.6090.6380.671
HumanPPI / accuracy0.7760.7050.709
SKEMPI / Pearson0.6420.6400.609
PDBbind / Spearman0.2540.2450.306
MutationalPPI / AP0.6360.6260.584
FLAb: trastuzumab binding / r²0.3710.3370.362
FLAb: D44 binding / r²0.3530.4020.368
FLAb: G6 binding / r²0.3120.2910.288
FLAb: G6 expression / r²0.6560.6610.678
Desautels (0.5% training) / Spearman0.9270.9100.902
Desautels (20% training) / Spearman0.9720.9680.973
COVID cross-neutralization / AP0.5320.5310.618
TDC / AUROC0.5480.548-
PISTE: Random / AUROC0.6630.660-
PISTE: Reference TCR / AUROC0.6730.640-
PISTE: Unified peptide / AUROC0.6770.706-
TEIM contacts / macro AP0.4590.445-

AP means average precision, and FLAb uses squared Pearson correlation.

All encoders use the same cases and prediction procedure within each task, with their weights fixed. The affinity and contact results do not establish an advantage over MINT.

On all 1,052 COVID tests, AUROC is 0.678 for Peppermint sequence, 0.744 with structure and 0.662 for MINT. The gain over sequence input is 0.065 (95% interval [0.019, 0.112]), resampling antibody groups with trained models fixed. Both modes use the same Peppermint checkpoint with separately fitted predictors.

Structure also improves PDZ3 interpolation from 0.427 to 0.629 Spearman. In docking, adding Peppermint features to a geometry-based ridge predictor raises the number of targets whose correct pose ranks first from 1/35 to 7/35.

Is explicit cross-attention necessary, or could it introduce a modeling bottleneck compared with simple sequence concatenation?

With a 18M parameter model, we compared three designs using the same data and training budget, with three training runs each:

ArchitectureSKEMPI Spearman, mean ± SD
Paired0.448 ± 0.024
Independent chains0.413 ± 0.011
Concatenated sequences0.416 ± 0.006

The paired encoder has the highest mean score, supporting its competitiveness with concatenation at this scale.

Given the importance of scaling in representation learning, could you analyze how downstream performance changes with model scale?

We evaluated five sizes at the same estimated pretraining compute budget:

Parameters, MCOVID APSKEMPI Spearman
18.20.3590.449
28.60.4420.439
71.70.4650.490
183.10.4230.444
443.70.4810.535

Both scores improve from smallest to largest, but not at every intermediate size.

Could you compare against strong structure-aware baselines, such as SaProt, to distinguish the benefits of the proposed architecture from those of the additional structural features?

These models use the same 340 PPB affinity tests and fitting procedure:

Encoder/inputSpearmanRMSE
Peppermint, sequence0.5031.480
Peppermint, structure0.4911.440
MINT0.5101.516
SaProt0.4571.537
MIF-ST0.4511.433
ESM-IF10.4031.528
PPIformer SSL0.4021.522

Peppermint is competitive, with no clear overall winner within uncertainties. In recovery, native ESM-IF1 reaches 55.8% versus 50.5% for simultaneous Peppermint and 51.7% with autoregressive decoding, which fills one masked residue at a time starting with the least uncertain prediction. The architecture comparison above separately controls model size, data and training.

The paper acknowledges that generalization to unfamiliar complex families remains unverified. Could you evaluate this setting?

The added test keeps related sequences together when separating training and test data, dividing 3,419 mutation cases into 66 groups:

PredictorSpearmanRMSE
Peppermint, structure0.0972.224
MINT0.0732.267
Training mean-2.206

No encoder clearly improves RMSE over predicting the training mean.

We would greatly appreciate if you could review our response above and share with us any comments or remaining concerns. Thank you for your consideration.

I appreciate the authors’ thorough and transparent additional experiments, which largely address my concerns about evaluation coverage. However, the results presented so far, taken together, do not establish a consistent downstream advantage over MINT, which somewhat limits the demonstrated contribution of the proposed approach. I will reconsider my score based on these results.

We are pleased that the additional experiments address your concerns about evaluation coverage. We agree that downstream gains vary across tasks, and we have kept the mixed results explicit. We would nevertheless like to highlight where the additional experiments demonstrate practical benefits from the learned representations.

On the complete COVID test set, without confidence filtering, AUROC improves from 0.662 for MINT to 0.744 for Peppermint, and average precision from 0.532 to 0.618. Because some antibodies contribute multiple measurements, we estimate uncertainty by resampling antibodies and keeping each antibody’s measurements together. We compare both models on the same resampled examples, and the resulting 95% confidence intervals for both improvements exclude zero. On the complete Bernett test set, average precision also improves from 0.527 to 0.563.

The contribution also includes representations that distinguish different arrangements of the same two proteins. In our docking test, each protein retains its sequence and shape, while their relative position and orientation change. Adding Peppermint features to a geometry-based ridge predictor increases the number of targets for which the correct arrangement ranks first from 1/35 to 7/35. This provides a direct example of the representations helping with a task that depends on interaction geometry.

Together, these results demonstrate specific downstream gains and a useful structural capability within an encoder that also supports sequence-only input. We hope these findings help clarify the contribution when assessing the revised paper.

Reviewer_p1o1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Peppermint, a structure-conditioned protein language model for interacting protein pairs. The model integrates intra-chain self-attention, bidirectional cross-attention, and geometric features through discrete structural tokens, backbone embeddings, and attention biases. Experiments demonstrate improved masked interface residue recovery and competitive performance on several downstream tasks.

Critical strengths
  1. The proposed framework is well motivated and effectively combines sequence and structural information while supporting sequence-only inference.

  2. The paper presents substantial pretraining efforts and a reasonably comprehensive evaluation across multiple protein interaction tasks.

Critical weaknesses
  1. The methodological novelty is somewhat limited, as the proposed architecture primarily integrates existing techniques such as cross-attention, structural tokens, and geometric attention biases. The ablation results in Table D.1 suggest that most of the improvement comes from 3Di tokens, while cross-chain geometric biases provide relatively small additional gains.

  2. The downstream improvements are not consistently convincing. Although structural conditioning substantially improves masked interface residue recovery, its advantage over ProteinMPNN is not statistically significant, where COVID AUROC and SKEMPI RMSE even showing slight degradation when structural information is introduced.

  3. The evaluation of structural conditioning relies heavily on confidence-selected subsets, which may limit the generalizability of the conclusions. In particular, only 1.07% of the Bernett dataset is retained for structure-conditioned evaluation.

Questions for authors

See the weaknesses

2 comments

Thank you for the comments on novelty, structural inputs and downstream evaluation. We added direct tests of relative placement and complete COVID and Bernett evaluations.

The methodological novelty is somewhat limited, as the proposed architecture primarily integrates existing techniques such as cross-attention, structural tokens, and geometric attention biases.

Peppermint combines these components in a paired encoder with optional complex structures. Local inputs describe each protein's shape, while geometric bias describes relative placement. Our contribution is this integration and evidence of when structure helps.

The ablation results in Table D.1 suggest that most of the improvement comes from 3Di tokens, while cross-chain geometric biases provide relatively small additional gains.

We agree that 3Di gives the largest loss reduction in the original short ablation. The new controls test relative placement while preserving each protein's shape:

InterventionRecovery loss, percentage points
Move both chains together0.01
Rotate one chain by 90°2.05
Remove geometric bias1.37

Changing relative placement reduces recovery, while moving both proteins together has negligible effect.

A short study with three runs per configuration tests whether geometry helps more with 3Di. Entries give its effect with 3Di minus its effect without 3Di:

OutcomeDifference
Docking Success@1, percentage points0.00
PDZ3 interpolation Spearman0.031
PDZ3 double mutants across chains, Spearman0.045

For double mutants, the adjusted 98.33% interval is [0.016, 0.074]. With 3Di, ranking improves slightly in all three runs, while RMSE rises from 0.287 to 0.367.

The downstream improvements are not consistently convincing. Although structural conditioning substantially improves masked interface residue recovery, its advantage over ProteinMPNN is not statistically significant,where COVID AUROC and SKEMPI RMSE even showing slight degradation when structural information is introduced

On the complete COVID test, both AUROC and AP improve with structure:

Complete test / metricPeppermint sequenceStructure where availableMINT
COVID, 1,052, AUROC0.6780.7440.662
COVID, AP0.5310.6180.532
Bernett, 52,031, AUROC0.5630.5800.540
Bernett, AP0.5450.5630.527

The COVID AUROC gain is 0.065 (95% interval [0.019, 0.112]), resampling antibody groups with trained models fixed.

These COVID results use different training and test populations from the original filtered analysis. We retain the original COVID AUROC decrease and SKEMPI RMSE increase. With simultaneous decoding, the recovery advantage over ProteinMPNN remains uncertain. Autoregressive decoding raises recovery from 50.5% to 51.7% with the same trained model, filling one masked residue at a time, starting with the least uncertain prediction.

Structure also improves PDZ3 interpolation from 0.427 to 0.629 Spearman, close to ESM-C's 0.620, without improving prediction of double mutants across chains. For docking, adding Peppermint features to ridge regression raises Success@1 from 1/35 to 7/35. PPIformer also reaches 7/35, and the gain reverses with an MLP predictor.

The evaluation of structural conditioning relies heavily on confidence-selected subsets, which may limit the generalizability of the conclusions. In particular, only 1.07% of the Bernett dataset is retained for structure-conditioned evaluation.

The new evaluation retains every test example regardless of predicted-structure confidence. Peppermint receives predicted complex structures for all 1,052 COVID tests and 48,217 of the 52,031 Bernett tests, using sequence input for the remaining 3,814. The original 1.07% refers to the 2,933 Bernett rows passing our confidence thresholds across training, validation and test combined.

My question has been resolved, and I will increase my rating.

Comments on the paper

1 comment

We thank the reviewers for their feedback, which helped us strengthen the evaluation of Peppermint and clarify how it uses interaction information. We expanded coverage of the main MINT benchmark families, added structure-aware and task-specific baselines, and included binding-fitness and docking tests. We also evaluated complete test sets without confidence filtering and added stricter tests of transfer to less similar proteins. Controlled comparisons across three training seeds, together with tests that change the partner protein or its relative placement, help establish the contributions of the model’s components and interaction inputs. We added downstream scaling results and uncertainty estimates, and clarified the experimental descriptions to make the findings easier to assess.

We would greatly appreciate if the reviewers who haven't responded yet could assess our revised manuscript as well as rebuttal comments, and share with us any feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.