- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates whether general-purpose large language models (LLMs) can reason about protein mutation effects directly from three-dimensional structural information without specialised predictors. The authors introduce a partner-contrastive benchmark and evaluate multiple LLMs against physical prediction methods. Through structural interventions, evidence audits, and case studies, the study finds that LLMs can utilise relevant structural information and identify plausible molecular interactions, particularly for larger affinity improvements. However, their overall prediction accuracy remains modest, with significant challenges in predicting structural rearrangements and energetic consequences.
Critical strengths
The paper addresses an interesting and underexplored question about LLMs' ability to reason directly from three-dimensional protein structures. The proposed partner-contrastive benchmark provides a useful evaluation framework, while the comparisons with physical predictors and structural ablation experiments offer valuable insights into model behaviour. The mechanistic case studies further strengthen the analysis by highlighting both the capabilities and limitations of LLM-based structural reasoning.
Critical weaknesses
The overall prediction accuracy remains relatively low, raising concerns about the reliability and practical applicability of the evaluated LLMs. Although the structural ablation experiments demonstrate sensitivity to geometric information, they do not conclusively establish genuine physical reasoning capabilities. Additionally, the relatively small benchmark and uneven coverage of protein systems limit the generalisability of the findings. Stronger experimental validation and broader evaluations would help substantiate the paper's central claims.
Questions for authors
-
Given the relatively low prediction accuracy, how do the authors assess the practical usefulness of LLMs for structure-guided mutation design?
-
How can the authors further distinguish genuine physical reasoning from pattern recognition or reliance on previously learned structural knowledge?
-
How well do the findings generalise to unseen protein systems, given the limited benchmark size and uneven distribution of protein families?
-
Could additional experiments or stronger baselines help establish whether the observed structural sensitivity translates into reliable mutation-effect predictions?
5 comments
Response to Weaknesses: Part 1-2
To assess utility more directly, we evaluated candidate generation in six BPTI partner-selective design tasks involving binding to ChT and MT. Across sites T11, K15, R17, and V34, each task requires selecting one of 19 non-WT amino acids to strengthen binding to one partner while weakening binding to the other. Recommendations were evaluated against published DMS-derived binding-effect reference data withheld during inference. Success requires both reference effects to reach at least 0.20 kcal/mol in the requested direction; a reference-best recommendation matches the candidate ranked highest by the predefined selectivity score.
| Method | Successful recommendations | Reference-best recommendations |
|---|---|---|
| GLM-5.3 Flash, three rounds | 5/18 (27.78%) | 2/18 (11.11%) |
| FoldX5 | 2/6 (33.33%) | 1/6 (16.67%) |
| Rosetta flex ddG | 2/6 (33.33%) | 0/6 (0.00%) |
Across three rounds of six tasks each (18 attempts in total), GLM produced five successful recommendations,including two reference-best recommendations, and succeeds at least once on three of the six tasks. FoldX5 and Rosetta flex ddG each succeed on two of six tasks, with each method providing one final recommendation per task after aggregating five internal calculations. The design results demonstrate that the model can propose substitutions that satisfy a partner-selective objective against withheld experimental reference data. A practical use is therefore to generate candidates for subsequent screening together with explicit hypotheses about which contacts or interactions the substitutions would change. Those hypotheses provide concrete targets for structural inspection, focused calculations, or experimental testing. Part 2 below examines their physical support and its association with prediction correctness.
A further opportunity is to use computational or experimental feedback to refine these proposals. Models could suggest alternative interaction hypotheses, identify calculations that distinguish them, and revise candidate choices in light of the results. Existing physical tools encode accumulated biophysical knowledge and empirical observations through human-designed energy functions, structural representations, and modelling procedures. These formulations remain open to improvement. Beyond proposing individual mutations, LLMs could help discover structural descriptors, context-dependent scoring rules, or calculation strategies that capture effects overlooked by existing formulations. Iteratively testing and refining these ideas against computational and experimental evidence could contribute to a next generation of more accurate tools for mutation prediction and protein design.
Response to Weaknesses (continued): Part 2
2. Physical support
We expanded the evaluation to examine two questions: whether predictions depend on relevant structural information, and whether the resulting mutation-induced interaction hypotheses are independently supported.
The structural interventions address the first question. Mutation-site side-chain masking changes approximately 42% of group predictions, compared with approximately 10% after distant masking. Removing the partner reduces group accuracy even though enhancement predictions increase in all three models. Among pairs receiving different predicted labels, correct orientation falls from 96.43% to 59.38% for DeepSeek, from 77.78% to 56.76% for GLM, and from 92.59% to 32.14% for Qwen. These results indicate that relevant structural inputs affect the assignment of partner-dependent mutation effects.
To address the second question, we added paired WT/mutant MD comparisons covering 82 mutation–partner cases. The models provide structured responses separating WT structural observations, proposed mutation-induced interaction changes, the final prediction, and its explanation. We extract the declared interaction changes and compare them with changes in contact frequencies measured from the WT and mutant trajectories. These trajectory measurements are extracted independently of model outputs and experimental affinity labels.
The key result is that interaction hypotheses accompanying correct affinity predictions receive higher MD support than those accompanying incorrect predictions in all three models. This relationship is observed both across Dataset 1 and within the subset of 21 partner groups whose enhancing member improves affinity by at least twofold.
| Model | All groups: correct answers | All groups: incorrect answers | Larger-improvement groups: correct answers | Larger-improvement groups: incorrect answers |
|---|---|---|---|---|
| GLM-5.3 Flash | 42/82 (51.22%) | 22/63 (34.92%) | 28/46 (60.87%) | 9/18 (50.00%) |
| Qwen3.8 Flash | 48/112 (42.86%) | 25/78 (32.05%) | 32/59 (54.24%) | 9/24 (37.50%) |
| DeepSeek Flash | 49/113 (43.36%) | 40/110 (36.36%) | 31/61 (50.82%) | 15/39 (38.46%) |
Entries are MD-supported/scorable interaction events, grouped by the correctness of the answer containing the hypothesis.
Restricting the comparison to the larger-improvement subset therefore preserves the relationship between prediction correctness and hypothesis support. Overall hypothesis support is also higher in the larger- than smaller-improvement groups: 57.81% versus 33.33% for GLM, 49.40% versus 29.91% for Qwen, and 46.00% versus 34.96% for DeepSeek.
We further evaluated BBKI F64R hypotheses against existing experimental Arg64 structures withheld during inference. Of 18 extracted interaction events, nine are fully supported, three partially supported, and six unsupported. Supported hypotheses include an Arg64–Asp189 salt bridge and an Arg64–Gly219 backbone hydrogen bond absent from the supplied Phe64 structure. This provides additional evidence that some proposed mutation-induced contacts agree with experimentally observed mutant structures.
These results link prediction correctness to independently supported interaction changes.
3. Broader evaluation
To broaden coverage beyond the original paired benchmark, we added A300, comprising 300 unpaired interface-mutation cases from SKEMPI 2.0. We excluded overlapping records and structural mutations from the original benchmark and capped selection at two cases per mutated-protein name group. Of these cases, 290 are assigned to 139 protein-family labels, while ten remain unassigned. The overlap exclusions concern records and structural mutations; protein families may recur across evaluations.
Results of the added A300 evaluation (300 individual mutation cases):
| Method | Task accuracy |
|---|---|
| GLM-5.3 Flash | 78.00% |
| FoldX5 | 81.33% |
| PyRosetta flex ddG | 80.00% |
| Always predicting non-enhancement | 69.33% |
GLM-5.3 Flash also achieves 67.16% balanced accuracy and an MCC of 0.4393. Its enhancement precision is 78.26%, whereas recall is 39.13%: the model predicts enhancement in 46 of 300 cases, compared with 92 experimentally enhancing cases. These results extend the evaluation to a broader collection while also showing that the conservative prediction pattern persists. Appendix I provides the construction procedure, family distribution, and full results.
Response to Weaknesses: Part 1-1
Concern: The overall prediction accuracy remains relatively low, raising concerns about the reliability and practical applicability of the evaluated LLMs. Although the structural ablation experiments demonstrate sensitivity to geometric information, they do not conclusively establish genuine physical reasoning capabilities. Additionally, the relatively small benchmark and uneven coverage of protein systems limit the generalisability of the findings. Stronger experimental validation and broader evaluations would help substantiate the paper's central claims.
A: Thank you for raising these concerns. We have expanded the evaluation to address three questions: whether direct LLM structural judgments can contribute to mutation selection, whether their proposed interaction changes receive independent physical support, and whether the observed performance extends beyond the original benchmark.
1. Practical utility
For structure-guided mutation design, a useful capability is to connect the local molecular environment to a proposed substitution. Our results provide systematic evidence that general-purpose LLMs can use three-dimensional structural information to anticipate mutation-induced interaction changes and make partner-dependent affinity judgments, without specialized predictors supplying those judgments. Three observations support this potential:
First, prediction performance exceeds model-matched random expectations and is comparable to the tested physical predictors. Complete-group accuracy requires both partner-specific predictions to be correct. DeepSeek Flash, GLM-5.3 Flash, and Qwen3.8 Flash achieve 14.29%, 19.05%, and 9.52%, respectively, compared with frequency-matched independent expectations of 8.62%, 13.08%, and 6.63%. They also exceed conditional random expectations that preserve which pairs receive identical or different labels: 7.14%, 13.10%, and 4.76%, respectively (Appendix H).
On Dataset 1, GLM-5.3 Flash achieves 19.05% complete-group accuracy, compared with 16.67% for FoldX and 23.81% for Rosetta ddG. On the added A300 evaluation (300 unpaired interface-mutation cases), its task accuracy is 78.00%, compared with 81.33% for FoldX5 and 80.00% for five-model Rosetta flex ddG. The methods also exhibit different prediction patterns and partially different successful cases, suggesting potential complementarity.
Second, correct predictions are accompanied by better-supported mutation hypotheses. MD support for interaction events accompanying correct versus incorrect predictions is 51.22% versus 34.92% for GLM, 43.36% versus 36.36% for DeepSeek, and 42.86% versus 32.05% for Qwen. The same relationship holds within the larger-improvement subset for all three models. Thus, prediction correctness is associated with independently supported mutation-induced interaction changes (Part 2 below).
Third, larger affinity improvements are associated with stronger prediction and hypothesis-support results. For groups whose enhancing member improves affinity by at least twofold, complete-group accuracy is 33.33% for GLM-5.3 Flash and 23.81% for DeepSeek Flash, compared with 4.76% for each in the smaller-improvement subset; Qwen remains at 9.52% in both. Overall MD hypothesis support increases in all three models: 57.81% versus 33.33% for GLM, 46.00% versus 34.96% for DeepSeek, and 49.40% versus 29.91% for Qwen. These patterns are consistent with larger experimental effects being accompanied by more readily identifiable structural consequences.
Taken together, these observations support structure-informed predictive signal beyond random guessing: successful predictions are associated with physically supported hypotheses, and both prediction performance and hypothesis support vary with effect magnitude. Although accuracy remains modest, the models already perform comparably to the tested physical predictors while exhibiting partly different successes. This combination motivates their use in candidate generation and their potential contribution to improved mutation-prediction and design methods.