- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors propose a protein–protein binding-affinity predictor using frozen ESM-3 and MINT together as sequence encoders, with an optional structure branch using EGNN and invariant point attention. An interesting direction is instead of using a conventional scalar regression head, the model generates 16 symbolic experts whose arithmetic expressions operate on learned latent concept vectors. Predictions can be combined using a validation-selected consensus. The authors also use gradient-times-input to attribute an expert prediction to individual sequence or structure residues.
The model is trained on PDBbind complexes. Evaluation uses an DBbind-derived test set and a SKEMPI-derived set containing 26 wild-type and 140 mutant records from 26 base complexes. On Test Set 1, a best-observed sequence-only formula obtains MAE 1.2982 kcal/mol and PCC 0.7034, while the validation-selected top-7 consensus from the same checkpoint obtains 1.2990/0.6806. A separate best-observed formula obtains 0.9960/0.8493 on Test Set 2, although its validation-selected top-12 consensus has MAE 1.3910.
Critical strengths
Strengths of the paper are:
-
Supporting sequence-only and sequence-plus-structure inference with one checkpoint could be practically useful, especially when reliable complex structures are unavailable.
-
The paper documents PDBbind filtering, benchmark exclusion, chain assignment, and the four runtime exclusions. These are helpful for reproducibility.
-
Sections 2.4 and A.7 distinguish checkpoint scoring from final consensus construction and explain that expert ranking and are fixed before test inference.
-
Table 2 exposes a large robustness gap that should be resolved, even though its aggregation is not described clearly enough for direct comparison with Table 1.
-
The authors state that sequence-only inference cannot distinguish alternative binding arrangements, latent concepts are not predefined physical quantities, and residue scores are model-derived hypotheses rather than causal binding energies. The Test Set 2 peak is also explicitly distinguished from expected validation-selected performance.
-
The features such as exposing expert predictions, expressions, auxiliary distributions, and residue sensitivities could be useful for exploratory analysis given their reliability is validated.
Critical weaknesses
Following are the weaknesses
-
On Test Set 1, the highlighted individual formula obtains MAE/PCC 1.2982/0.7034, while the validation-selected top-7 consensus from the same checkpoint obtains 1.2990/0.6806 and still exceeds the strongest listed baseline. On Test Set 2, a different checkpoint formula achieves 0.9960/0.8493, but its validation-selected top-12 consensus has MAE 1.3910. Its not clear how the Table 1 checkpoint, configuration, seed, or individual formula was chosen from the full search. If test performance influenced any of these choices, selecting among 16 experts, multiple checkpoints, seeds, configurations, and modalities would create a large multiple-comparisons advantage.
-
The development/test separation removes identical PDB codes but does not cluster by sequence identity, protein family, interface similarity, or structural homology. Closely related complexes—or even the same protein paired in a related complex—may therefore occur in training and test. The paper should report nearest train–test sequence identity separately for both partners, family and interface overlap, and performance stratified by similarity.
-
Test Set 2 is SKEMPI which contains 166 observations but only 26 base complexes. The 140 mutant records are clustered and statistically non-independent within their wild-type parent complexes. Pooled PCC or SPCC across all 166 absolute affinities can be high because of between-complex differences, even if a model assigns nearly the same prediction to every mutant of a given parent. Consequently, the statement that SPCC above 0.83 shows that candidate variants are well ordered is not justified by the reported metric. It is also unclear how the sequence-plus-structure path represents mutants. Are mutated residue identities combined with wild-type coordinates, are mutant structures modeled, or are the same wild-type graph features reused? This must be specified, because the interpretation of “structure-conditioned” mutation prediction depends on it.
-
The formulas expose a sparse portion of the model’s computation, but don't provide a biochemical or mechanistic explanation. They operate on 32-dimensional learned concepts with expert-specific element-wise coefficients. Their vector-valued output is then reduced by an input-dependent softmax , itself produced by a neural function of the representation. For the single 1WR1 example in Appendix B, the 16 displayed experts reduce to three repeated syntactic families, with similar scales and biases near : L1–L2 share one template, L3–L8 another, and L9–L16 another. One example is not enough to establish globally low diversity, but it shows that symbolic diversity across samples and runs has not yet been demonstrated. Appendices A.5–A.7 indicate that formula syntax is conditioned on each input’s concept memory and generated at inference, so these are input-specific computational traces rather than a single learned global law of protein binding.
-
Across Table 2, supplying structure produces essentially no improvement and sometimes worsens both MAE and PCC. The consistency term explicitly trains structure-path predictions toward detached sequence-only predictions, which may discourage the structure branch from changing predictions in response to geometry. The stated motivation is instead that structure enables conformation- and interface-specific prediction, but no experiment demonstrates this behavior.
-
For every training sample, the regression objective directly supervises up to four experts selected using their error against that sample’s true target. At inference, however, experts are ranked globally on validation data and combined using a fixed top- consensus; there is no learned sample-specific router corresponding to the target-based training selection. The paper needs an ablation against uniform all-expert supervision, analysis of which samples each expert specializes on, and a comparison between the globally selected consensus and any label-free learned routing mechanism.
-
No confidence interval, bootstrap analysis, or paired significance test is reported. It is therefore unclear whether the difference is distinguishable from sampling variation. The quoted baselines also come from separate papers rather than a controlled retraining with the same data, split, labels, and preprocessing which make comparison harder.
Questions for authors
-
Does the “best-observed” Test Set 1 checkpoint mean best according to validation MAE or best after inspecting Test Set 1? How was its displayed individual formula selected?
-
What is the expected performance of the exact inference procedure a user would run on a new unlabeled protein pair? One consistent estimator should be identified and used throughout the paper.
-
Because a new train/validation split is generated for every seed, Table 2 combines initialization variance with split variance. Results on a fixed split across seeds, as well as variation across grouped splits, would be more informative.
-
How are mutant graph node features constructed for Test Set 2? In particular, are mutant residue identities inserted into graphs with wild-type coordinates and structural descriptors?