- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates limitations of self-consistency as an evaluation metric for protein design models, showing that comparison with a single refolded structure can reject plausible, flexible conformations. It formalizes structural plausibility through sequence-conditioned equilibrium ensembles, distinguishes flexibility failures from folding-model failures, and proposes ensemble self-consistency, which accepts a structure when it matches at least one predicted conformation. Validation uses ATLAS molecular dynamics trajectories, longer MEGASim simulations, experimentally characterized de novo proteins, Rosetta decoys, and generations from three protein design models. Using SimpleFold, increasing ensemble size from one to 16 predictions improves ATLAS native-structure recall from approximately 74% to 82%. The method also accepts additional generated structures and preserves high specificity on the evaluated decoy benchmark.
Critical strengths
The paper identifies a consequential limitation of widely used evaluation practices. The ATLAS analysis shows that self-consistency rejection increases with MD-derived flexibility, while the tested distributional metrics show comparatively weak separation between low- and high-flexibility subsets. This matters because self-consistency guides model selection and generation filtering: its association with flexibility could favor rigid proteins without that bias being clearly reflected in other evaluation scores. Identifying this risk provides a valuable contribution to evaluating conformational diversity, independently of the proposed ensemble-based remedy.
Critical weaknesses
-
The failure analysis does not distinguish successful rescues from reclassification. Under Section 5’s definitions, adding an MD-incompatible prediction can change a still-rejected protein from a flexibility failure to a folder failure. The aggregate reduction in flexibility failures therefore does not establish how many initially affected proteins were successfully rescued. Appendix I’s longer-trajectory analysis does not resolve this ensemble-size effect. Per-protein transitions are needed to substantiate the central claim that moderate ensemble sizes nearly eliminate flexibility-related rejections. This limits the interpretation of the failure decomposition, while leaving the measured recall improvements intact.
-
Specificity evidence is limited for the flexible proteins motivating the method. The 935 Rosetta decoys cover only ten targets, described as rigid, with tight predicted ensembles. These results and Appendix H’s threshold-relaxation control support high specificity on the tested benchmark, but leave false-positive behavior for broader conformational ensembles unresolved. Since accepting a match to any prediction can admit erroneous outlier matches, specificity testing on flexible targets is consequential to judging the proposed metric’s reliability. Broader target coverage and uncertainty estimates at the protein level would strengthen the justification for adopting it as a general plausibility filter.
Questions for authors
-
Among proteins classified as flexibility failures at , what proportions become successful matches, remain flexibility failures, or become folder failures at ?
-
Can you compare SF-1 and SF-16/32 on flexible proteins with broad predicted ensembles, including uncertainty estimates at the protein level, to determine whether the low false-positive rate extends beyond the current ten-target rigid benchmark?
-
Does the association between self-consistency rejection and flexibility persist when flexibility is measured using post-equilibration RMSF or pairwise MD distances, with protein length controlled?
-
Can you provide evidence independent of SimpleFold that the additional generations accepted by SF-16 are physically plausible and broaden the flexibility distribution of the accepted set, given that SimpleFold supplies both the acceptance criterion and the flexibility estimate?
4 comments
Expanded specificity evaluation, including flexible targets, and per-target analysis (W2, Q2)
We have now expanded the specificity evaluation to 3DRobot [1], applying the same preprocessing and ATLAS-based filtering used for Rosetta decoy dataset (Appendix F). This yields 4,716 decoys across 19 additional targets, including three flexible targets with ATLAS-derived mean RMSFs of 2.35, 2.39, and 3.37 Å, placing them in the top 18.5%, 18.1%, and 9.9% of our ATLAS native protein cohort, respectively.
At the fixed RMSD threshold of 2 Å, increasing the ensemble size from (N=1) to (N=32) increases false positives from 2 to only 13 out of 4,716 decoys, corresponding to an FPR increase from 0.042% to 0.276% (0.233 percentage points). 13 false positives at (N=32) arise from five targets with mean MD RMSFs of 0.91, 0.93, 1.10, 1.25, and 2.00 Å. None arise from the three highest RMSF targets highlighted above. Together with the single additional false positive observed on Rosetta, these results extend the evidence for low false-positive rates beyond the original ten-target benchmark, including to more flexible proteins.
To quantify uncertainty at the protein level, we repeated SimpleFold sampling 10 times for each target, generating an ensemble of (N=32) predictions in each replicate, and computed the mean and standard deviation of each target’s false-positive rate (FPR). 19 of the 29 targets had zero observed false positives in all 10 replicates, and the mean FPR was below 1.2% for every target. For the remaining 10 targets, the FPRs (mean ± SD, in %) were 1.176 ± 0.452, 1.154 ± 0.000, 1.083 ± 0.657, 0.449 ± 0.449, 0.442 ± 0.296, 0.377 ± 0.000, 0.313 ± 0.503, 0.137 ± 0.221, 0.042 ± 0.133, and 0.039 ± 0.124. Pooling decoys across all targets, the overall FPR was 0.227% ± 0.036% (mean ± SD across 10 replicates; 95% confidence interval for the mean: 0.201–0.253%). Giving each protein equal weight yielded a mean FPR of 0.180%, indicating that the low overall FPR was not driven solely by targets contributing more decoys. If the reviewer meant a different notion of uncertainty, we would be happy to run additional analyses and follow up.
[1] Haiyou Deng, Ya Jia, Yang Zhang. 3DRobot: Automated Generation of Diverse and Well-packed Protein Structure Decoys. Boinformatics, 32: 378-87 (2016).
Thank you for the detailed response and the additional experiments. I appreciate the effort to address the concerns raised in my review.
The response and additional analyses have satisfactorily addressed most of my concerns and strengthened my assessment of the paper.
For clarity and reproducibility, I encourage the authors to incorporate the expanded specificity results into the manuscript and clarify that the reported uncertainty intervals reflect repeated SimpleFold sampling on a fixed set of targets. I also encourage the authors to distinguish successful recovery from reductions in the flexibility-failure category and retain their stated qualification that the MD analysis does not establish the plausibility of every predicted conformation.
Overall, I find the rebuttal convincing. The additional evidence has substantially strengthened my assessment, and I am inclined to update my recommendation to weak acceptance.
Thank you for your follow up and positive reassessment. We will ensure that the revised manuscript includes the expanded specificity results and clearly states that the scope of the added analyses.
We thank the reviewer for recognizing the contribution of our work and the constructive feedback. We ran four analyses that directly address the reviewer’s concerns and questions. The changes are reflected in the manuscript. We also provide updated Appendix J, K pages in this anonymous link for your convenience: https://anonymous.4open.science/r/rebuttal-pdfs-B5F6/appendix_JK.pdf
Is flexibility failures rescued or reclassified as folder failure? (W1/Q1)
Thank you for highlighting this distinction. We tracked the 92 proteins classified as flexibility failures under SimpleFold at (N=1) (6.1% of the ATLAS cohort) to distinguish successful recovery from reclassification. Percentages below are relative to these 92 proteins. (The values are the same for N=16 and 32 because flexibility failures remain the same for two, in this cohort.)
Of the 79 ATLAS proteins leaving the flexibility failure category, 60 (75.9%) are successfully recovered. The remaining 19 are reclassified because additional predictions lack support from the MD reference ensemble. They remain rejected and do not contribute to recall gains. This diagnostic label is relative to the available MD ensemble, whose finite coverage is discussed in Appendix I.
We included this transition analysis in the main text, and revised the phrase “nearly eliminates” to “largely eliminates”:
These updates are reflected in the manuscript, text color green.
Same diagnostic, but by MD post-equilibration RMSF / MD mean pairwise RMSD (Q3)
Thank you for the suggestion. Using MD-derived post-equilibration RMSF or mean pairwise RMSD yields the same conclusions: self-consistency failures are enriched among more flexible proteins, and ensemble self-consistency improves recall. Both conclusions also persist when we stratify the analysis by protein length. We report these analyses in Appendix J and point them in the main text.
Are the additional accepted designs by SF-16 more flexible, measured by something independent of SimpleFold? (Q4)
To address this concern, we performed MD simulations to independently assess the flexibility of the additional accepted designs. We added the following paragraph to the revised manuscript, with simulation details and per-design RMSF profiles in Appendix K:
Agreement with MD supports the plausibility of the predicted fluctuations and lends support to the SimpleFold conformations exhibiting them, although it does not establish the plausibility of every predicted conformation. Direct experimental verification of specific conformations would require wet-lab validation, which is beyond the scope of this paper.