- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents StruPair, a structure-only classifier and benchmark for evaluating conformational changes between protein structure pairs. The authors introduce five classes describing different levels of structural rearrangement and construct a dataset from experimental structures, MD simulations, and computational designs. They also develop a hybrid classifier combining structural features with graph neural networks, achieving strong performance on the annotated evaluation set. The paper further explores the use of StruPair for selecting backbone pairs in multi-state protein design, showing improved structural recovery in a downstream design pipeline. Overall, I think the paper addresses a meaningful problem and provides a potentially useful benchmark. However, I have concerns about the validity of the proposed classification scheme, the reliability of the annotations, and whether the downstream experiments sufficiently demonstrate its practical value.
Critical strengths
-
The paper addresses a meaningful problem in protein structure evaluation. Conventional metrics such as RMSD and TM-score may not fully capture different patterns of conformational change. I think exploring more comprehensive structural representations is a valuable direction for multi-state protein design.
-
The paper makes an effort to construct a unified benchmark from different structural sources. Combining experimental structures, MD simulations, and computational designs provides a useful starting point for evaluating conformational changes.
Critical weaknesses
-
I am not fully convinced that the proposed five-class classification scheme is sufficiently validated as a benchmark for multi-state protein design. While these classes provide a reasonable description of structural differences, it remains unclear how they relate to the physical feasibility or functional relevance of multi-state proteins. I think a good benchmark should provide clearly defined evaluation criteria rather than relying mainly on descriptive categories. Moreover, the annotations are based on human consensus without retained individual labels, making their reliability difficult to assess. The evaluation set also contains no computationally designed structure pairs despite this being a major intended application. This is my primary concern with the paper.
-
The experimental results do not sufficiently establish the additional value of the proposed classifier. The current comparisons do not clearly isolate the contribution of the GNN, and the downstream experiment only compares StruPair-based selection against unfiltered candidates rather than simpler structural filtering methods such as RMSD or TM-score. Stronger ablations and baselines would help demonstrate the approach's practical benefit. In addition, the presentation could be improved, particularly the figure quality and readability. Although not strictly required, I think using vector graphics would make the figures clearer.
Questions for authors
-
Could the authors provide further evidence that the five structural classes are useful for evaluating multi-state protein designs, beyond describing geometric differences between structures?
-
Could the authors clarify the annotation procedure and provide additional evaluation on independently annotated or computationally designed structure pairs?
-
Could the authors provide more controlled ablations to isolate the contribution of the GNN from conventional structural features?
-
Could the authors compare StruPair-based candidate selection with simpler structural filtering methods and clarify how sequence-free backbone pairs are processed?