acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Self-Consistency in Protein Generative Models

Abstract

Self-consistency is the standard metric for evaluating structural plausibility in protein generative models: a generated sequence–structure pair is accepted if an independent folding model refolds the sequence to the generated structure within a fixed RMSD threshold. However, self-consistency rejects 44% of native sequence–structure pairs, a gap previously overlooked as a folding model limitation. In this paper, we connect self-consistency to the biophysical ideal of structural plausibility–sufficiently low free energy in that protein's equilibrium ensemble–which makes two failure modes visible: *flexibility failures*, where a broad ensemble cannot be summarized by a single refold, and *folder failures* . We use ATLAS molecular dynamics trajectories to empirically analyze native structure rejections through this lens, and find that both failure modes are enriched among flexible proteins. We propose ensemble self-consistency using conformational ensemble predictors, improving native protein recall across the flexibility spectrum and rescuing co-design generations previously rejected, without compromising specificity on a Rosetta decoy dataset. Since flexibility is often central to protein function, a metric that penalizes it misdirects generative model development; ensemble self-consistency offers a more faithful framework, and our formalization and diagnostics let it evolve as structure prediction models improve.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein generative models, designability, self-consistency, evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Pi28OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates limitations of self-consistency as an evaluation metric for protein design models, showing that comparison with a single refolded structure can reject plausible, flexible conformations. It formalizes structural plausibility through sequence-conditioned equilibrium ensembles, distinguishes flexibility failures from folding-model failures, and proposes ensemble self-consistency, which accepts a structure when it matches at least one predicted conformation. Validation uses ATLAS molecular dynamics trajectories, longer MEGASim simulations, experimentally characterized de novo proteins, Rosetta decoys, and generations from three protein design models. Using SimpleFold, increasing ensemble size from one to 16 predictions improves ATLAS native-structure recall from approximately 74% to 82%. The method also accepts additional generated structures and preserves high specificity on the evaluated decoy benchmark.

Critical strengths

The paper identifies a consequential limitation of widely used evaluation practices. The ATLAS analysis shows that self-consistency rejection increases with MD-derived flexibility, while the tested distributional metrics show comparatively weak separation between low- and high-flexibility subsets. This matters because self-consistency guides model selection and generation filtering: its association with flexibility could favor rigid proteins without that bias being clearly reflected in other evaluation scores. Identifying this risk provides a valuable contribution to evaluating conformational diversity, independently of the proposed ensemble-based remedy.

Critical weaknesses
  1. The failure analysis does not distinguish successful rescues from reclassification. Under Section 5’s definitions, adding an MD-incompatible prediction can change a still-rejected protein from a flexibility failure to a folder failure. The aggregate reduction in flexibility failures therefore does not establish how many initially affected proteins were successfully rescued. Appendix I’s longer-trajectory analysis does not resolve this ensemble-size effect. Per-protein transitions are needed to substantiate the central claim that moderate ensemble sizes nearly eliminate flexibility-related rejections. This limits the interpretation of the failure decomposition, while leaving the measured recall improvements intact.

  2. Specificity evidence is limited for the flexible proteins motivating the method. The 935 Rosetta decoys cover only ten targets, described as rigid, with tight predicted ensembles. These results and Appendix H’s threshold-relaxation control support high specificity on the tested benchmark, but leave false-positive behavior for broader conformational ensembles unresolved. Since accepting a match to any prediction can admit erroneous outlier matches, specificity testing on flexible targets is consequential to judging the proposed metric’s reliability. Broader target coverage and uncertainty estimates at the protein level would strengthen the justification for adopting it as a general plausibility filter.

Questions for authors
  1. Among proteins classified as flexibility failures at N=1N=1, what proportions become successful matches, remain flexibility failures, or become folder failures at N=16/32N=16/32?

  2. Can you compare SF-1 and SF-16/32 on flexible proteins with broad predicted ensembles, including uncertainty estimates at the protein level, to determine whether the low false-positive rate extends beyond the current ten-target rigid benchmark?

  3. Does the association between self-consistency rejection and flexibility persist when flexibility is measured using post-equilibration RMSF or pairwise MD distances, with protein length controlled?

  4. Can you provide evidence independent of SimpleFold that the additional generations accepted by SF-16 are physically plausible and broaden the flexibility distribution of the accepted set, given that SimpleFold supplies both the acceptance criterion and the flexibility estimate?

4 comments

We thank the reviewer for recognizing the contribution of our work and the constructive feedback. We ran four analyses that directly address the reviewer’s concerns and questions. The changes are reflected in the manuscript. We also provide updated Appendix J, K pages in this anonymous link for your convenience: https://anonymous.4open.science/r/rebuttal-pdfs-B5F6/appendix_JK.pdf

Is flexibility failures rescued or reclassified as folder failure? (W1/Q1)

Thank you for highlighting this distinction. We tracked the 92 proteins classified as flexibility failures under SimpleFold at (N=1) (6.1% of the ATLAS cohort) to distinguish successful recovery from reclassification. Percentages below are relative to these 92 proteins. (The values are the same for N=16 and 32 because flexibility failures remain the same for two, in this cohort.)

Ensemble sizeSuccessfully acceptedRemain flexibility failuresReclassified as folder failures
(N=16)60 (65.2%)13 (14.1%)19 (20.7%)
(N=32)60 (65.2%)13 (14.1%)19 (20.7%)

Of the 79 ATLAS proteins leaving the flexibility failure category, 60 (75.9%) are successfully recovered. The remaining 19 are reclassified because additional predictions lack support from the MD reference ensemble. They remain rejected and do not contribute to recall gains. This diagnostic label is relative to the available MD ensemble, whose finite coverage is discussed in Appendix I.

We included this transition analysis in the main text, and revised the phrase “nearly eliminates” to “largely eliminates”:

Increasing the ensemble size to N=16N=16 substantially reduces SimpleFold’s flexibility failures. Of the 79 proteins leaving this category, 60 (75.9%) are successfully recovered, while the remaining 19 remain rejected and are reclassified as folder failures because additional predictions lack support from the MD reference ensemble. These results suggest that a moderate number of samples is often sufficient to recover a conformation close to the experimental reference that is missed by a single prediction.

These updates are reflected in the manuscript, text color green.

Same diagnostic, but by MD post-equilibration RMSF / MD mean pairwise RMSD (Q3)

Thank you for the suggestion. Using MD-derived post-equilibration RMSF or mean pairwise RMSD yields the same conclusions: self-consistency failures are enriched among more flexible proteins, and ensemble self-consistency improves recall. Both conclusions also persist when we stratify the analysis by protein length. We report these analyses in Appendix J and point them in the main text.

Are the additional accepted designs by SF-16 more flexible, measured by something independent of SimpleFold? (Q4)

To address this concern, we performed MD simulations to independently assess the flexibility of the additional accepted designs. We added the following paragraph to the revised manuscript, with simulation details and per-design RMSF profiles in Appendix K:

To independently assess the flexibility of designs additionally accepted by SF-16, we performed 100ns MD simulations initialized from 40 randomly selected La-Proteina designs of length 100 (Appendix K): 32 accepted by SF-1 and eight accepted by SF-16 but rejected by SF-1. The SF-16 rescues exhibited higher mean MD-derived RMSF than the SF-1 accepted designs (0.95 versus 0.84 Å; Mann-Whitney UU test, p=0.012p=0.012), supporting the recovery of more flexible designs in this subset. This difference was more pronounced when RMSF was averaged over the top 20% most mobile residues (1.93 versus 1.60 Å; Mann-Whitney UU test, p=0.025p=0.025). SimpleFold also captured the locations of fluctuations along the sequence: the median per-design Pearson correlation between MD and SimpleFold per-residue RMSF profiles was r=0.73r=0.73. However, SimpleFold underestimated fluctuation amplitudes, consistent with our ATLAS observations (Appendix D). Per-design RMSF profiles are provided in Appendix K.

Agreement with MD supports the plausibility of the predicted fluctuations and lends support to the SimpleFold conformations exhibiting them, although it does not establish the plausibility of every predicted conformation. Direct experimental verification of specific conformations would require wet-lab validation, which is beyond the scope of this paper.

Expanded specificity evaluation, including flexible targets, and per-target analysis (W2, Q2)

We have now expanded the specificity evaluation to 3DRobot [1], applying the same preprocessing and ATLAS-based filtering used for Rosetta decoy dataset (Appendix F). This yields 4,716 decoys across 19 additional targets, including three flexible targets with ATLAS-derived mean RMSFs of 2.35, 2.39, and 3.37 Å, placing them in the top 18.5%, 18.1%, and 9.9% of our ATLAS native protein cohort, respectively.

At the fixed RMSD threshold of 2 Å, increasing the ensemble size from (N=1) to (N=32) increases false positives from 2 to only 13 out of 4,716 decoys, corresponding to an FPR increase from 0.042% to 0.276% (0.233 percentage points). 13 false positives at (N=32) arise from five targets with mean MD RMSFs of 0.91, 0.93, 1.10, 1.25, and 2.00 Å. None arise from the three highest RMSF targets highlighted above. Together with the single additional false positive observed on Rosetta, these results extend the evidence for low false-positive rates beyond the original ten-target benchmark, including to more flexible proteins.

To quantify uncertainty at the protein level, we repeated SimpleFold sampling 10 times for each target, generating an ensemble of (N=32) predictions in each replicate, and computed the mean and standard deviation of each target’s false-positive rate (FPR). 19 of the 29 targets had zero observed false positives in all 10 replicates, and the mean FPR was below 1.2% for every target. For the remaining 10 targets, the FPRs (mean ± SD, in %) were 1.176 ± 0.452, 1.154 ± 0.000, 1.083 ± 0.657, 0.449 ± 0.449, 0.442 ± 0.296, 0.377 ± 0.000, 0.313 ± 0.503, 0.137 ± 0.221, 0.042 ± 0.133, and 0.039 ± 0.124. Pooling decoys across all targets, the overall FPR was 0.227% ± 0.036% (mean ± SD across 10 replicates; 95% confidence interval for the mean: 0.201–0.253%). Giving each protein equal weight yielded a mean FPR of 0.180%, indicating that the low overall FPR was not driven solely by targets contributing more decoys. If the reviewer meant a different notion of uncertainty, we would be happy to run additional analyses and follow up.

[1] Haiyou Deng, Ya Jia, Yang Zhang. 3DRobot: Automated Generation of Diverse and Well-packed Protein Structure Decoys. Boinformatics, 32: 378-87 (2016).

Thank you for the detailed response and the additional experiments. I appreciate the effort to address the concerns raised in my review.

The response and additional analyses have satisfactorily addressed most of my concerns and strengthened my assessment of the paper.

For clarity and reproducibility, I encourage the authors to incorporate the expanded specificity results into the manuscript and clarify that the reported uncertainty intervals reflect repeated SimpleFold sampling on a fixed set of targets. I also encourage the authors to distinguish successful recovery from reductions in the flexibility-failure category and retain their stated qualification that the MD analysis does not establish the plausibility of every predicted conformation.

Overall, I find the rebuttal convincing. The additional evidence has substantially strengthened my assessment, and I am inclined to update my recommendation to weak acceptance.

Thank you for your follow up and positive reassessment. We will ensure that the revised manuscript includes the expanded specificity results and clearly states that the scope of the added analyses.

Reviewer_Bv5tOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines the limitations of self-consistency as a metric for evaluating structural plausibility in protein sequence–structure co-design. Motivated by the observation that ESMFold rejects 44% of 1,520 native sequence–structure pairs, the authors formalize structural plausibility in terms of equilibrium conformational ensembles and use molecular dynamics (MD) simulations to distinguish flexibility-related failures from folding-model errors. They further propose ensemble self-consistency, which accepts a candidate structure if at least one sampled refold falls within a specified RMSD threshold. Evaluations on native proteins, experimentally characterized de novo designs, generated proteins, and Rosetta decoys, together with robustness analyses of RMSD thresholds and MD trajectory lengths, demonstrate improved recall, particularly for flexible conformations, while maintaining comparative specificity on the evaluated decoy set.

Critical strengths
  1. Natural sequence-structure pairs provide an intuitive positive control for identifying false rejections; furthermore, validating the approach using experimentally characterized de novo designed proteins—which extend beyond the benchmarks established by natural proteins—offers valuable insights.
  2. SimpleFold’s recall rate for natural proteins increases from approximately 74% (at N=1N=1) to 82% (at N=16N=16). The paper also provides threshold-based recall comparisons and a robustness analysis using longer molecular dynamics (MD) trajectories; these results are quite impressive.
Critical weaknesses
  1. Targets that remain rejected are classified as "flexible failures" only if all their corresponding predictions are supported by the MD ensemble; if even one prediction lacks such support, the target is reclassified as a collapsed predictor failure. Consequently, a decrease in the proportion of flexible failures—prior to distinguishing between successful recovery and label switching—does not constitute proof that these targets have truly been recovered.
  2. Even if an MD trajectory relaxes into a conformational region characterized by minimal internal fluctuations but significant deviation from the initial structure, the resulting average RMSD (relative to the experimental structure) may remain high. This makes it difficult to ascertain to what extent the observed rejection—or deviation—is genuinely attributable to conformational flexibility.
  3. For newly generated candidate structures, both the acceptance decision and the flexibility estimate are derived from SimpleFold. Therefore, it is challenging to determine whether an increase in the variability of predicted structures—observed alongside changes in candidate acceptance rates—reflects authentic conformational properties, or merely uncertainty or bias.
Questions for authors
  1. For targets classified as "flexible failures" when N=1N=1, what proportions are successfully accepted, remain classified as flexible failures, or are reclassified solely as "folding predictor failures" when N=16/32N=16/32?
  2. How should the ensemble size and the RMSD threshold be interpreted or calibrated to align with the proposed "balanced probability" criterion? For flexible targets with large conformational ensembles, does the advantage in specificity still hold, and is it possible to assess statistical uncertainty on a per-protein-target basis?
  3. If targets are grouped based on MD-derived RMSF (centered on the mean position of the ensemble) or pairwise distances between MD conformations—rather than by their displacement relative to the initial PDB structure—does the observed relationship between the rejection rate and flexibility still hold?
5 comments

We thank the reviewer for recognizing the insights our diagnostic provide and the constructive feedback. We ran four analyses that directly address the reviewer’s concerns and questions. The changes are reflected in the manuscript. We also provide updated Appendix J, K pages in this anonymous link for your convenience: https://anonymous.4open.science/r/rebuttal-pdfs-B5F6/appendix_JK.pdf

Is flexibility failures rescued or reclassified as folder failure? (W1/Q1)

Thank you for highlighting this distinction. We tracked the 92 proteins classified as flexibility failures under SimpleFold at (N=1) (6.1% of the ATLAS cohort) to distinguish successful recovery from reclassification. Percentages below are relative to these 92 proteins. (The values are the same for N=16 and 32 because flexibility failures remain the same for two, in this cohort.)

Ensemble sizeSuccessfully acceptedRemain flexibility failuresReclassified as folder failures
(N=16)60 (65.2%)13 (14.1%)19 (20.7%)
(N=32)60 (65.2%)13 (14.1%)19 (20.7%)

Of the 79 ATLAS proteins leaving the flexibility failure category, 60 (75.9%) are successfully recovered. The remaining 19 are reclassified because additional predictions lack support from the MD reference ensemble. They remain rejected and do not contribute to recall gains. This diagnostic label is relative to the available MD ensemble, whose finite coverage is discussed in Appendix I.

We included this transition analysis in the main text, and revised the phrase “nearly eliminates” to “largely eliminates”:

Increasing the ensemble size to N=16N=16 substantially reduces SimpleFold’s flexibility failures. Of the 79 proteins leaving this category, 60 (75.9%) are successfully recovered, while the remaining 19 remain rejected and are reclassified as folder failures because additional predictions lack support from the MD reference ensemble. These results suggest that a moderate number of samples is often sufficient to recover a conformation close to the experimental reference that is missed by a single prediction.

These updates are reflected in the manuscript, text color green.

Same diagnostic, but by MD RMSF / MD mean pairwise RMSD (W2/Q3)

Thank you for the suggestion. Using MD-derived RMSF or mean pairwise RMSD yields the same conclusions: self-consistency failures are enriched among more flexible proteins, and ensemble self-consistency improves recall. Both conclusions also persist when we stratify the analysis by protein length. We report these analyses in Appendix J and point them in the main text.

Are the additional accepted designs by SF-16 more flexible, measured by something independent of SimpleFold? (W3)

To address this concern, we performed MD simulations to independently assess the flexibility of the additional accepted designs. We added the following paragraph to the revised manuscript, with simulation details and per-design RMSF profiles in Appendix K:

To independently assess the flexibility of designs additionally accepted by SF-16, we performed 100ns MD simulations initialized from 40 randomly selected La-Proteina designs of length 100 (Appendix K): 32 accepted by SF-1 and eight accepted by SF-16 but rejected by SF-1. The SF-16 rescues exhibited higher mean MD-derived RMSF than the SF-1 accepted designs (0.95 versus 0.84 Å; Mann-Whitney UU test, p=0.012p=0.012), supporting the recovery of more flexible designs in this subset. This difference was more pronounced when RMSF was averaged over the top 20% most mobile residues (1.93 versus 1.60 Å; Mann-Whitney UU test, p=0.025p=0.025). SimpleFold also captured the locations of fluctuations along the sequence: the median per-design Pearson correlation between MD and SimpleFold per-residue RMSF profiles was r=0.73r=0.73. However, SimpleFold underestimated fluctuation amplitudes, consistent with our ATLAS observations (Appendix D). Per-design RMSF profiles are provided in Appendix K.

Agreement with MD supports the plausibility of the predicted fluctuations and lends support to the SimpleFold conformations exhibiting them, although it does not establish the plausibility of every predicted conformation. Direct experimental verification of specific conformations would require wet-lab validation, which is beyond the scope of this paper.

Expanded specificity evaluation, including flexible targets, and per-target analysis (Q2)

We have now expanded the specificity evaluation to 3DRobot [1], applying the same preprocessing and ATLAS-based filtering used for Rosetta decoy dataset (Appendix F). This yields 4,716 decoys across 19 additional targets, including three flexible targets with ATLAS-derived mean RMSFs of 2.35, 2.39, and 3.37 Å, placing them in the top 18.5%, 18.1%, and 9.9% of our ATLAS native protein cohort, respectively.

At the fixed RMSD threshold of 2 Å, increasing the ensemble size from (N=1) to (N=32) increases false positives from 2 to only 13 out of 4,716 decoys, corresponding to an FPR increase from 0.042% to 0.276% (0.233 percentage points). 13 false positives at (N=32) arise from five targets with mean MD RMSFs of 0.91, 0.93, 1.10, 1.25, and 2.00 Å. None arise from the three highest RMSF targets highlighted above. Together with the single additional false positive observed on Rosetta, these results extend the evidence for low false-positive rates beyond the original ten-target benchmark, including to more flexible proteins.

To quantify uncertainty at the protein level, we repeated SimpleFold sampling 10 times for each target, generating an ensemble of (N=32) predictions in each replicate, and computed the mean and standard deviation of each target’s false-positive rate (FPR). 19 of the 29 targets had zero observed false positives in all 10 replicates, and the mean FPR was below 1.2% for every target. For the remaining 10 targets, the FPRs (mean ± SD, in %) were 1.176 ± 0.452, 1.154 ± 0.000, 1.083 ± 0.657, 0.449 ± 0.449, 0.442 ± 0.296, 0.377 ± 0.000, 0.313 ± 0.503, 0.137 ± 0.221, 0.042 ± 0.133, and 0.039 ± 0.124. Pooling decoys across all targets, the overall FPR was 0.227% ± 0.036% (mean ± SD across 10 replicates; 95% confidence interval for the mean: 0.201–0.253%). Giving each protein equal weight yielded a mean FPR of 0.180%, indicating that the low overall FPR was not driven solely by targets contributing more decoys. If the reviewer meant a different notion of uncertainty, we would be happy to run additional analyses and follow up.

Finally, we would appreciate clarification of what is meant by the “balanced probability” criterion (Q2), as we do not introduce a criterion by that name in the manuscript. We interpret this question as asking how the parameters NN and τ\tau relate to our ideal probability-based plausibility criterion. Since the true equilibrium ensemble is unknown, neither parameter can be directly calibrated to some free energy threshold. We chose τ=2\tau=2Å following conventional self-consistency that is used in the field, and select N=16N=16 based on the observed saturation in recall gains. We additionally evaluate sensitivity to τ\tau and the corresponding recall-specificity tradeoff (Appendix H and updated Fig. 7).

[1] Haiyou Deng, Ya Jia, Yang Zhang. 3DRobot: Automated Generation of Diverse and Well-packed Protein Structure Decoys. Boinformatics, 32: 378-87 (2016).

We kindly ask the reviewer to review our rebuttal. If it addresses your concerns, we would greatly appreciate your consideration of an updated score. We also welcome any remaining questions or concerns and would be happy to address them during the discussion period.

Reviewer_5jESOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper exposes a problem in a current metric for protein generative models. This metric accepts a generated structure only if a folding model such as ESMFold refolds its sequence to within 2 ångström RMSD. However this metric rejects 44% of native protein structures, and the authors show these rejections concentrate on flexible proteins. The authors then propose a new criteria that works on the ensemble of samples: a structure is accepted if its Cα RMSD to at least one of N samples from an ensemble predictor (e.g., SimpleFold) is below 2 ångström RMSD. The authors then evaluate the recall on a variety of proteins and Specificity on Rosetta decoys.

Critical strengths
  • Identifies an important, overlooked limitation of a standard metric:T he paper shows that canonical self-consistency rejects 44% of native PDB structures, and that these rejections concentrate on flexible proteins (Fig. 1, Sec. 4)
  • Proposes a simple, alternative with a potentially better recall-specificity tradeoff: Ensemble self-consistency increases native recall (74% → 82% with SimpleFold, N = 16) with almost no additional false positives on the decoys (1 → 2 of 935).
Critical weaknesses

The specificity evaluation is not strict enough to support the recall gains.

A more permissive criterion trivially increases recall, so the recall gains (Sec. 5.1) need to be backed by a meaningful specificity test. The authors report specificity on a Rosetta decoy set (Sec. 5.5), but most decoys are far from the 2 Å threshold (Fig. 10). As a result, both criteria produce almost no false positives (1 vs 2 of 935), and the test cannot show whether the ensemble becomes too permissive.

The recall-specificity tradeoff is only partially reported in the appendix.

A full tradeoff curve (e.g., ROC across τ for N = 1 and N = 16/32) would show whether the ensemble gives a better tradeoff, or only a different operating point. This would make the paper much stronger. Because it is hard to obtain a meaningful specificity on the Rosetta decoys at the 2 Å threshold, one could compute it at higher thresholds. This is partially documented in the appendix: SimpleFold-1 requires τ = 3.46 Å to match the recall of SimpleFold-16 at τ = 2 Å (App. H). This is a highly relevant comparison to show the clear benefit of the proposed model, but it is buried in the appendix.

I would highly recommend sweeping the recall and specificity over different threshold values and reporting the different ROC curves for different N values. This would clearly show whether the proposed model performs better, which seems to be the case from the value in the appendix.

Questions for authors

I would ask the authors to show the ROC curve, swept over different threshold values, for N = 1 and, e.g., N = 32. I think it would really strengthen the paper (see Weaknesses 1-2). This would show whether ensemble self-consistency gives a better recall-specificity tradeoff, or only a more permissive operating point. And given the numbers in Appendix H seems the model has a better recall-specificity tradeoff.

2 comments
AuthorsOpenReview ↗

Thank you for your constructive feedback.

Thank you for recognizing the value of our work and for your constructive feedback. We understand your main concern to be whether the specificity evaluation is sufficiently stringent to support the recall gains. Also, we agree that the recall-specificity tradeoff, which directly addresses the reviewer’s concerns, was not sufficiently emphasized in the original manuscript.

Following your suggestion, we have added ROC curves for SF-1 and SF-32, sweeping the RMSD threshold from 0.5 to 6.0 Å, to the main paper (page 10, Figure 7, https://anonymous.4open.science/r/rebuttal-pdfs-B5F6/updated_page10.pdf; Manuscript file is updated as well.). These curves show that:

  • Rosetta decoy test is sensitive to increased permissiveness: simply relaxing the RMSD threshold substantially increases false positives,
  • SF-32 dominates SF-1 across the evaluated range. To match SF-32’s 83.1% native protein recall at a 2 Å threshold, SF-1 requires a substantially higher threshold of 3.6 Å, resulting in 152 false positives compared with only 2 for SF-32 at 2 Å. (We now use SF-32, not SF-16, for consistency with the rest of the section.) Thus ensemble self-consistency is materially different from simply making the definition of acceptance more permissive.

Thank you again for this suggestion, which helped us strengthen the paper!

Thank you again for your helpful feedback. As the discussion period is ending soon, we would appreciate your assessment of the added ROC analysis, which shows that SF-32 achieves a substantially better recall-specificity tradeoff than SF-1. If this addresses your concerns, we would be grateful if you would consider increasing your score. We are happy to address any outstanding questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.