acceptodds
Withdrawn from ICLR 2027

StruPair: A structure-only classifier and benchmark for multi-state protein design

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Designing proteins with multiple functional states, including allosterically regulated switches, molecular sensors, and enzymes, requires generating physically plausible conformations that represent the distinct states needed to carry out the intended function. However, evaluating whether computationally generated structural states correspond to realistic and functionally meaningful conformational changes remains challenging, as commonly used structural similarity metrics do not fully capture the diverse modes and scales of conformational change observed in natural proteins. Here, we introduce StruPair, a structure-only framework for classifying and benchmarking protein conformational changes using structure pairs collected from experiments, molecular dynamics (MD) simulations, and computational design. We train a hybrid classifier that combines structural comparison features with graph neural network representations. The trained StruPair classifier assigns each structure pair to one of five ordered classes based on the extent of conformational change and the preservation or reorganization of secondary and tertiary structure. Using only paired backbone structures, without amino acid identities or sequence embeddings, allows StruPair to evaluate candidate pairs before sequence design or optimization..We provide the evaluation set and the trained classifier through a web interface and API.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein conformational changes, Protein structure comparison, Backbone-only representation, Multi-state protein design, Graph neural network

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zUTgOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents StruPair, a structure-only classifier and benchmark for evaluating conformational changes between protein structure pairs. The authors introduce five classes describing different levels of structural rearrangement and construct a dataset from experimental structures, MD simulations, and computational designs. They also develop a hybrid classifier combining structural features with graph neural networks, achieving strong performance on the annotated evaluation set. The paper further explores the use of StruPair for selecting backbone pairs in multi-state protein design, showing improved structural recovery in a downstream design pipeline. Overall, I think the paper addresses a meaningful problem and provides a potentially useful benchmark. However, I have concerns about the validity of the proposed classification scheme, the reliability of the annotations, and whether the downstream experiments sufficiently demonstrate its practical value.

Critical strengths
  1. The paper addresses a meaningful problem in protein structure evaluation. Conventional metrics such as RMSD and TM-score may not fully capture different patterns of conformational change. I think exploring more comprehensive structural representations is a valuable direction for multi-state protein design.

  2. The paper makes an effort to construct a unified benchmark from different structural sources. Combining experimental structures, MD simulations, and computational designs provides a useful starting point for evaluating conformational changes.

Critical weaknesses
  1. I am not fully convinced that the proposed five-class classification scheme is sufficiently validated as a benchmark for multi-state protein design. While these classes provide a reasonable description of structural differences, it remains unclear how they relate to the physical feasibility or functional relevance of multi-state proteins. I think a good benchmark should provide clearly defined evaluation criteria rather than relying mainly on descriptive categories. Moreover, the annotations are based on human consensus without retained individual labels, making their reliability difficult to assess. The evaluation set also contains no computationally designed structure pairs despite this being a major intended application. This is my primary concern with the paper.

  2. The experimental results do not sufficiently establish the additional value of the proposed classifier. The current comparisons do not clearly isolate the contribution of the GNN, and the downstream experiment only compares StruPair-based selection against unfiltered candidates rather than simpler structural filtering methods such as RMSD or TM-score. Stronger ablations and baselines would help demonstrate the approach's practical benefit. In addition, the presentation could be improved, particularly the figure quality and readability. Although not strictly required, I think using vector graphics would make the figures clearer.

Questions for authors
  1. Could the authors provide further evidence that the five structural classes are useful for evaluating multi-state protein designs, beyond describing geometric differences between structures?

  2. Could the authors clarify the annotation procedure and provide additional evaluation on independently annotated or computationally designed structure pairs?

  3. Could the authors provide more controlled ablations to isolate the contribution of the GNN from conventional structural features?

  4. Could the authors compare StruPair-based candidate selection with simpler structural filtering methods and clarify how sequence-free backbone pairs are processed?

Reviewer_9FoVOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces StruPair a framework for classifying conformational changes between pairs of protein backbones to evaluate candidates in multistate design. The authors construct a dataset of structure pairs and train a gnn classifier to predict 5 classes of structural rearrangement.

Critical strengths

I think the paper addresses a relevant problem in multistate protein design -- evaluating whether two generated protein backbones represent a desired conformational change. The means of doing this is not as justified and could use more work.

Critical weaknesses

The main weakness is the utility of the classifier (is it that much more helpful than filtering? are the 5 categories biologically grounded?). The methodological novelty is also limited and the downstream results lack comparisons to give the reader confidence that such a model is necessary.

Questions for authors

Unclear motivation for proposed classification task

Before training new models, we as a field should properly motivate the need of a task. Here my main concern is that it is still unclear how useful these five structural classes actually are for multistate protein design. The authors show that their model can reproduce human judgements about the extend and type of conformational change, but this does not tell us whether a single sequence can adopt both structures/or whether a seqeuence can act as a switch between conformations. A large structural rearrangement is not always undesirable, nor is a coherent domain movement necessarily favorable. I would like to see stronger justification as to why these classes are meaningful grounded in biological insight.

limited methodological novelty

The proposed model is a supervised classifier combining a relatively standard gnn with existing structural comparsion features. I appreciate the effor to construct an annotated dataset, but the method itself seems fairly limited and the utility is not properly exposed. Much of the task also seems to involve learning structural properties already captured by metrics like rmsd, tmscore, contact preservation, etc. So, why use the classifier when u can use the myriad metrics already used for (multistate) protein design.

insufficient downstream validation

Authors do not compare their selection strategy against simpler alternatives, such as filtering candidates (switchcraft criterion). This seems like a particularly important missing baseline. Moreover, only 2/100 selected pairs pass the stricter confidence and there is no experimental validation of switching behavior (though not required for an ml conference), but with this the evaluation seems quite inconclusive. I think stronger downstream evidence is needed to establish that the proposed classifier provides meaningful value.

Reviewer_9o6TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StruPair introduces a structure-only benchmark and classifier for categorizing protein conformational changes into five ordered classes. It distinguishes movement of preserved structural units from increasingly extensive structural reorganization. The authors assemble 108,629 structure pairs after filtering and deduplication, including a 575-pair evaluation set. A hybrid model combining a pair-graph neural network with global geometric features achieves a reported macro-F1 of 0.9635, compared with 0.8754 and 0.9103 for conventional-feature classifiers. In a computational design study, StruPair selection increases recovery of both target states by AlphaFold 3 from 0/100 to 10/100 candidates, falling to 2/100 when requiring high confidence for both predictions. These results assess endpoint classification and computational recovery, rather than experimental switching.

Critical strengths
  1. Practical sequence-free screening: StruPair distinguishes coherent structural motion from deformation using backbone geometry, enabling screening before sequence design and improving reported AF3 dual-state recovery from 0/100 to 10/100.
  2. Diverse data and safeguards against leakage: The dataset combines experimental, simulated, and designed structures, using related-sample grouping and sequence-based exclusion to reduce overlap between training and evaluation sets.
  3. Strong classification performance: The hybrid classifier achieves a macro-F1 of 0.9635, outperforming conventional-feature baselines with all errors confined to adjacent classes.
Critical weaknesses
  1. The benchmark's annotation reliability and reproducibility are insufficiently established. The paper describes human annotation and review of a corpus containing 108,629 retained pairs but does not explain whether every pair received individual inspection or if some labels involved automated assignment, propagation, or batch review. Individual annotator decisions were not retained. The reported k≈0.75 depends on an unverifiable assumption about their marginal distributions. Consequently, the high classification score establishes agreement with the final consensus labels, while independent reproducibility of the class boundaries remains unclear. This concern is compounded by unavailable evaluation artifacts in the linked repository, despite Section 8 describing them as released, and by incomplete documentation of the hybrid model's training and evaluation protocol. These gaps are consequential because the annotated benchmark is a central contribution.
  2. The experiments do not establish how much the proposed classifier adds beyond simpler geometric screening. Annotators consulted structural metrics that overlap with the classifier’s inputs. This makes a controlled comparison against a compact metric-based classifier particularly important. The conventional feature sets in Table 3 omit contact features included in the hybrid. No ablation using the same global features isolates the graph branch's contribution. The downstream study likewise compares StruPair selection only against unfiltered candidates. Without the unfiltered arm's class distribution and matched comparisons against conventional classifiers or simple geometric filters, the 0/100 versus 10/100 result does not establish a distinctive benefit from StruPair's learned representation or taxonomy.
Questions for authors
  1. Did every retained pair receive individual inspection by both annotators? Please distinguish direct manual labeling, automated or propagated labels, and batch review by data source. Which metrics and previous labels were visible to each annotator, and how was the 16% reconciliation rate recorded given that individual decisions were not retained? Is there an independently recorded, blinded assessment of label agreement?
  2. How does the hybrid compare with a classifier using the same seven global features without the graph branch? Does StruPair outperform conventional-feature selection or simple geometric filters under matched candidate pools and downstream budgets?
  3. What was the class distribution of the unfiltered design candidates?
  4. Could the authors confirm whether the hybrid classifier in Section 5.1 used the same data splits as the Section 4.2 baselines, and clarify how its checkpoint was selected?
  5. Regarding lines 939–940 in Appendix C.1, which samples were used to estimate the preprocessing statistics applied in the reported 575-pair evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.