acceptodds
Under review as a conference paper at ICLR 2027

Correcting the Dropout-LayerNorm Expectation Gap Improves Protein Structure Models

Abstract

Although , here we show that is not equal to . Accordingly, the pattern of a Dropout layer followed by a LayerNorm, which is common to many AlphaFold2-based protein structure predictors, produces a systematic bias at evaluation time that can hamper performance. To address this, we derive a closed-form, first-order correction for this gap, which we call a Dropout-LayerNorm Correction (DLC). DLC empirically matches the performance boost of large Monte Carlo dropout ensembles. We evaluate its effect across nine protein structure models (ESMFold, OpenFold, ABB3, FlashABB, Ibex, NbForge, Genie1, Genie2, Genie3) on both paired and single-chain antibody structures as well as one protein-ligand docking model (QuickBind). The correction is computationally negligible and improves accuracy in all ten models tested (), with a modest but consistent improvement in ESMFold and OpenFold and a substantial improvement in antibody-specific models. This work identifies the mathematical consequence of chaining together Dropout and LayerNorm and provides a free, principled adjustment to improve the evaluation performance of many pretrained models.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
dropout, normalization, bias, protein, antibody, structure prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ra1KOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Dropout–LayerNorm Correction (DLC), a closed-form, inference-only hook that does not require retraining. It corrects the systematic overshoot, relative to the training-time expectation, that appears at evaluation when Dropout is immediately followed by LayerNorm in AlphaFold2-style structure modules. On antibody and nanobody data, the authors show that it improves the accuracy of a range of structure-prediction models.

Critical strengths

The finding is both concrete and useful. A Dropout layer immediately followed by LayerNorm leaves a systematic gap between the training-time expectation and the evaluation output, and the authors’ closed-form correction matches the Monte Carlo mean on ABodyBuilder3. On antibody data the same inference-only hook improves structure accuracy, most clearly for ABB3, with no retraining and negligible cost. This is a practical guide for inference in later structure models that keep the Dropout-then-LayerNorm pattern.

Critical weaknesses
  1. The experiments do not support the broad claim that DLC “improves protein structure models.” Every reported structure-prediction result is on antibodies or nanobodies. There is no CASP, CAMEO, or other general monomer benchmark, so the effect on ordinary monomers is unknown. Most of the tested models are antibody-specific. The general folders, ESMFold and OpenFold, are also evaluated only on these immunoglobulin sets, where the gains are about 0.2%–0.8%. Current mainstream predictors such as AlphaFold3 and Boltz use residual updates rather than Dropout immediately before LayerNorm, so DLC may not apply to them.

  2. The comparisons are incomplete. The headline antibody numbers use 34 structures, whereas ABodyBuilder3’s own test set has 100. Genie1/2/3 are general backbone generators, but they are compared only by denoising loss on 100 nanobodies, with no measure of structural accuracy.

I like this simple but effective idea, and I would raise my score if these concerns were addressed.

Questions for authors
  1. Does DLC improve accuracy on every structure? For the models that currently report only mean RMSD, especially ABB3, FlashABB, and Ibex, could you show the per-structure comparison with and without DLC?
  2. Could you add results on general monomers, for example CASP or CAMEO? If not, could the claim be narrowed from protein structure models to antibody and nanobody models?
  3. Could you report ABB3 on its full 100-structure test set, and evaluate Genie1/2/3 by structural quality rather than denoising loss alone?
3 comments

Thank you for your review of our work. We have performed new experiments/analysis and addressed your questions below. We hope you will agree that these experiments have increased the robustness of the manuscript and are happy to answer any questions you may still have.

Broad claim unsupported: no CASP/CAMEO, general-monomer effect unknown

"The experiments do not support the broad claim that DLC 'improves protein structure models.' Every reported structure-prediction result is on antibodies or nanobodies. There is no CASP, CAMEO, or other general monomer benchmark, so the effect on ordinary monomers is unknown. Most of the tested models are antibody-specific. The general folders, ESMFold and OpenFold, are also evaluated only on these immunoglobulin sets, where the gains are about 0.2%–0.8%. Current mainstream predictors such as AlphaFold3 and Boltz use residual updates rather than Dropout immediately before LayerNorm, so DLC may not apply to them."

To addres this, we ran ESMFold, OpenFold, and AlphaFold2 (new) on CASP14, CASP15, and CAMEO (144 targets from an archived window chosen to approximate ESMFold's own CAMEO benchmarking methodology). Here are the domain-level paired t-tests throughout:

ModelCASP14 (n=38)CASP15 (n=45)CAMEO (n=144)
ESMFold+1.02%, 29/38, p=0.003+0.32%, 28/45, p=0.243+0.60%, 105/144, p=0.0038
OpenFold+0.43%, 30/38, p=0.023-0.31%, 24/45, p=0.182+0.51%, 96/144, p=0.0041
AlphaFold2+0.43%, 30/38, p=0.053+0.38%, 32/45, p=0.184+0.25%, 99/144, p=0.218

DLC improves performance in 8/9 cases and 4/4 which have p<0.05. This provides a computationally neglible, calibration-free improvement to three very widely used models.

Added to the manuscript as a new main-text Section "General-monomer evaluation: CASP14, CASP15, and CAMEO".

Additionally, the insights from this work may explain why models like AlphaFold3 remove dropout. It is possible that reintroducing the Dropout+LayerNorm pattern under the DLC correction could improve generalization in future models.

Narrow the claim from "protein structure models" to antibody/nanobody?

Question: "Could you add results on general monomers, for example CASP or CAMEO? If not, could the claim be narrowed from protein structure models to antibody and nanobody models?"

In addition to the experiments above, we also ran Genie 1, 2, and 3 on CASP14, CASP15, and CAMEO.

modelCASP14 loss (n=38)CASP15 loss (n=45)CAMEO loss (n=144)nanobody-100 loss (n=100)
Genie1+2.88%, 38/38, p=1.2e-34+2.71%, 45/45, p=1.5e-44+2.70%, 144/144, p=1.5e-141+3.12%, 100/100, p=1.4e-91
Genie2+2.71%, 38/38, p=1.1e-27+2.54%, 45/45, p=7.0e-33+2.46%, 144/144, p=7.7e-92+2.90%, 100/100, p=1.2e-67
Genie3+1.54%, 38/38, p=1.9e-23+1.52%, 45/45, p=2.1e-24+1.43%, 144/144, p=2.2e-70+1.39%, 99/100, p=1.8e-37

We saw that the effect held up significanly in all cases. The largest structure prediction improvements were found for FlashABB and ABB3 which are both antibody-specific, but we believe that this is because they are domain-specific rather than a particular idiosyncrasy of antibodies.

Report ABB3 on its full 100-structure test set

"The comparisons are incomplete. The headline antibody numbers use 34 structures, whereas ABodyBuilder3's own test set has 100. Genie1/2/3 are general backbone generators, but they are compared only by denoising loss on 100 nanobodies, with no measure of structural accuracy."

Question: "Could you report ABB3 on its full 100-structure test set, and evaluate Genie1/2/3 by structural quality rather than denoising loss alone?"

We had originally restricted the analysis to the set of 34 used in the Ibex manuscript which limits training leakage between methods. Here, we have expanded the analysis to the full test set of 100 used by ABB3 and FlashABB. We have also included the fraction of improved structures and associated p values.

ModelWhole-Fv Impr. (p)CDR-H3 Impr. (p)
ABB3+12.90% (99/100, p≈0)+4.67% (80/100, p=1.4e-13)
FlashABB+10.60% (97/100, p≈0)+2.39% (61/100, p=0.0011)

Added to the manuscript in the same new appendix as above ("ABB3, FlashABB, and Ibex on ABB3/FlashABB's own held-out 100-structure test set").

Evaluate Genie1/2/3 by structural quality, not just denoising loss

"Genie1/2/3 are general backbone generators, but they are compared only by denoising loss on 100 nanobodies, with no measure of structural accuracy." / "...evaluate Genie1/2/3 by structural quality rather than denoising loss alone?"

We built a full ProteinMPNN+ESMFold+TM-score designability pipeline for all three where we generate 30 backbones total (10 each at lengths 50, 100, and 150), with 8 ProteinMPNN sequences sampled per backbone. Each structure generated shares the seed between DLC off and on for both Genie and ProteinMPNN.

modelmean scTM Δ (on−off)scTM improvedpaired t-test p[email protected] off→on
Genie1-0.000214/300.99283.3%→90.0% (n.s.)
Genie2-0.042110/300.03086.7%→76.7%
Genie3+0.006114/300.502100%→100% (ceiling)

We see no difference in Genie1, a small improvement in Genie3, and a somewhat larger degradation of Genie2. Despite our efforts to reduce noise by sharing seeds, it is difficult to tell if this effect exceeds the noise threshold. However, as shown above, DLC improved the loss for all models on all sets. It is unclear what is driving the gap between loss and designability, but we expect that DLC's ability to improve the underlying model at inference time can be effectively leveraged by future models.

Added to the manuscript in the same new Appendix "Genie designability" as above.

Does DLC improve accuracy on every structure? Per-structure comparison for ABB3/FlashABB/Ibex?

"Does DLC improve accuracy on every structure? For the models that currently report only mean RMSD, especially ABB3, FlashABB, and Ibex, could you show the per-structure comparison with and without DLC?"

We have added the fraction of improved structures throughout, and added a new per-structure RMSD-shift figure (Figure 6) for ABB3, FlashABB, and Ibex on the 100-structure held-out set, in the same style as the existing ESMFold/OpenFold per-structure shift figure (Figure 7). Briefly, whole antibody structures and Genie loss predictions improve almost uniformly, while CDRH3 and general monomers improve at varying levels. When running Ibex on the full ABB3 test set, we see that the apparent CDRH3 degradation was a statistical artefact which is resolved under the larger test set.

Ibex variant34-structure held-out (CDR-H3)full 100-structure (CDR-H3)
1 member (member 0)+2.22%, 21/34, p=0.0615+3.86%, 69/100, p≈0
8-member ensemble (medoid)-1.37%, 18/34, p=0.52+2.40%, 68/100, p=0.023

Added to the manuscript in the same new appendix as above.

Thank you for the authors' efforts in addressing my concerns. The experiments on CASP and CAMEO substantially strengthen the claims of this work. I think this work is interesting and meaningful, and I have raised my score to 3.

I also have a question out of curiosity: Is the Dropout-then-LayerNorm design commonly used in other fields, such as computer vision and NLP? If so, could DLC be applied to these domains as well?

Reviewer_W76LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper targets a specific engineering problem in the AlphaFold series model where the combination of the LayerNorm and the Dropout leads to variance drift during inference. The derivation is clear and the experiments is satisfactory. I think this trick is useful for the BioInfo community, but seems not satisafactory to ICLR.

Critical strengths

This engineering trick is highly practical for AlphaFold. Given the widespread demand for AlphaFold within the field of bioinformatics, such a simple yet useful technique will be a valuable contribution to the community.

Critical weaknesses
  1. This issue only appears in older AlphaFold2-style designs without residual connections. Modern models like AlphaFold3 already fix this by setting dropout to 0 or adding residual links. Does this method have any real practical value for future architectures? For example, the gain on ESMFold and OpenFold is small (<0.8%).

  2. The test sets are limited. It would be better for the authors to show standard protein structure evaluation, like CASP.

Questions for authors

See weakness

1 comment

Thank you for your review of our work. We have performed new experiments/analysis and addressed your questions below. We hope you will agree that these experiments have increased the robustness of the manuscript and are happy to answer any questions you may still have.

Does this matter for modern architectures?

"This issue only appears in older AlphaFold2-style designs without residual connections. Modern models like AlphaFold3 already fix this by setting dropout to 0 or adding residual links. Does this method have any real practical value for future architectures? For example, the gain on ESMFold and OpenFold is small (<0.8%)."

In this work, we introduce the underlying effect of the inference-time expectation shift induced by Dropout layers followed by LayerNorms. We see this as a useful new result in statistical machine learning which motivates the myriad strategies for avoiding the pattern, such as those introduced in AlphaFold3-style models and modern Transformers. It is likely that many researchers have experimented with Dropouts preceding LayerNorms and found that they had mysteriously poor performance at inference time. This work explains why, and provides an effective strategy to fix it, which may unlock developments in future models by reintroducing the pattern.

Additionally, many modern methods are still based on AlphaFold2-style architectures, such as ABB3, FlashABB, Ibex, and Genie. This work materially improves those models and enables the use of these architectures to create efficient, accurate, domain-specific structure predictors.

Need standard protein structure evaluation (CASP)

"The test sets are limited. It would be better for the authors to show standard protein structure evaluation, like CASP."

To addres this, we ran ESMFold, OpenFold, and AlphaFold2 (new) on CASP14, CASP15, and CAMEO (144 targets from an archived window chosen to approximate ESMFold's own CAMEO benchmarking methodology). Here are the domain-level paired t-tests throughout:

ModelCASP14 (n=38)CASP15 (n=45)CAMEO (n=144)
ESMFold+1.02%, 29/38, p=0.003+0.32%, 28/45, p=0.243+0.60%, 105/144, p=0.0038
OpenFold+0.43%, 30/38, p=0.023-0.31%, 24/45, p=0.182+0.51%, 96/144, p=0.0041
AlphaFold2+0.43%, 30/38, p=0.053+0.38%, 32/45, p=0.184+0.25%, 99/144, p=0.218

DLC improves performance in 8/9 cases and 4/4 which have p<0.05. This provides a computationally neglible, calibration-free improvement to three very widely used models.

Added to the manuscript as a new main-text Section "General-monomer evaluation: CASP14, CASP15, and CAMEO".

Reviewer_nYjqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the expectation mismatch introduced by applying LayerNorm after inverted dropout. The authors derive a bias-preserving rescaling of normalized activations and apply it during inference without updating pretrained weights. Experiments cover ten models, evaluating antibody and nanobody structure prediction, confidence estimates, diffusion noise-prediction losses, and protein–ligand docking. The largest improvements in final structure prediction occur for ABB3 and FlashABB; benefits for general-purpose predictors are smaller, while evidence for generation and docking is more limited.

Critical strengths
  1. The correction is inexpensive and supported by a concrete mechanism. Connecting dropout-induced normalization changes to subsequent coordinate updates is practically meaningful. On ABB3, the first-block translation-length ratio relative to the Monte Carlo reference decreases from 1.056 to 1.001, directly supporting the correction’s effect on the identified local discrepancy.

  2. The strongest results concern final predicted structures. Without retraining, whole-Fv RMSD decreases from 1.176 to 1.019 Å for ABB3 and from 1.156 to 1.029 Å for FlashABB. These improvements are meaningful for a small inference-time modification, although their robustness requires broader validation.

Critical weaknesses
  1. The incremental value over existing inference-gap and calibration ideas is insufficiently established. Dropout inference gaps and normalization-related variance mismatch have been studied previously [1–3]. These methods are not directly interchangeable with the proposed LayerNorm patch, but the authors still need to distinguish its specific advance more clearly. In particular, the paper does not establish the benefit of the input-dependent correction over constant q\sqrt q scaling or global/per-layer scaling selected on an independent calibration set.

  2. The approximation’s validity conditions are insufficiently specified. Appendix A attributes neglected effects to terms vanishing with channel dimension, but approximating the expectation of a normalized random variable using expected statistics requires additional concentration assumptions. For a vector with one nonzero channel, unit affine scale, zero bias, and negligible numerical epsilon, the dropout-on mean is approximately q$\,$\mathrm{LN}(x), while the proposed correction factor approaches ($\sqrt q$) as dimension increases. Thus, large dimension alone does not ensure accuracy; the authors should specify the required activation-distribution conditions and assess approximation errors.

  3. The BatchNorm extension appears to apply the correction in the wrong direction. Under Appendix B.2’s stated objective, a zero-mean input with variance σ2\sigma^2 has dropout-trained running variance σ2/q\sigma^2/q, so the original inference output is already attenuated by q\sqrt q. Applying the multiplicative correction from Eq. (5) shrinks it further, whereas recovering dropout-free normalization requires the inverse factor. This concerns an auxiliary extension and does not directly invalidate the main LayerNorm experiments, but it requires correction or clarification.

  4. Independent validation of the largest structural gains is limited. The headline ABB3 and FlashABB results use a 34-structure subset. The nanobody evaluation overlaps with training data for some models, and improvement on an additional Ibex training-set subset does not establish that its held-out CDRH3 regression is merely noise. Larger independent evaluations and paired uncertainty estimates would strengthen the main practical claim.

  5. The broader application claims exceed the demonstrated outcomes. Genie1/2/3 are evaluated only on noise-prediction loss, without assessing generated-structure quality or designability. On the full QuickBind benchmark, 240/427 cases improve, but the mean RMSD reduction from 9.384 to 9.343 Å is not statistically significant under the reported test. Subsets selected using the uncorrected prediction’s RMSD against the ground truth are useful diagnostics, but do not establish an overall docking benefit or a deployable criterion for applying the correction.

References

[1] Xuezhe Ma, Yingkai Gao, Zhiting Hu, Yaoliang Yu, Yuntian Deng, and Eduard Hovy. Dropout with Expectation-linear Regularization. ICLR 2017. Paper

[2] Dan Hendrycks and Kevin Gimpel. Adjusting for Dropout Variance in Batch Normalization and Weight Initialization. arXiv:1607.02488, 2016. Paper

[3] Xiang Li, Shuo Chen, Xiaolin Hu, and Jian Yang. Understanding the Disharmony Between Dropout and Batch Normalization by Variance Shift. CVPR 2019. Paper

Questions for authors
  1. Using the same pretrained checkpoints, how does the full correction compare with constant q\sqrt q scaling and independently calibrated global or per-layer scaling? Does the input-dependent formula provide a consistent additional benefit?

  2. What assumptions justify the concentration approximation? How accurate is it on actual model activations across correction sites? Please also clarify the direction of the BatchNorm correction in Appendix B.2.

  3. Does Figure 4 average samples locally at each Dropout–LayerNorm site, as its axis and caption indicate, or average final coordinates from complete stochastic forward passes, as the surrounding text describes? These procedures differ; which supports the claim of matching MC dropout ensembles?

  4. Can the authors report paired confidence intervals, improvement fractions, and regression magnitudes on a larger, independent structural test set? For the broader application claims, do the gains persist in actual Genie generation and standard docking success metrics?

3 comments

Thank you for your review of our work. We have performed new experiments/analysis and addressed your questions below. We hope you will agree that these experiments have increased the robustness of the manuscript and are happy to answer any questions you may still have.

Incremental value over constant/calibrated scaling

"The incremental value over existing inference-gap and calibration ideas is insufficiently established. Dropout inference gaps and normalization-related variance mismatch have been studied previously [1–3]. These methods are not directly interchangeable with the proposed LayerNorm patch, but the authors still need to distinguish its specific advance more clearly. In particular, the paper does not establish the benefit of the input-dependent correction over constant (c) scaling or global/per-layer scaling selected on an independent calibration set."

Question: "Using the same pretrained checkpoints, how does the full correction compare with constant (c) scaling and independently calibrated global or per-layer scaling? Does the input-dependent formula provide a consistent additional benefit?"

We experimented with calibrating a single global constant c on an independent valid split (10 train/valid-direction splits), compared to the theory formula on held-out test. We also bootstrapped stability (2000 random 50-structure subsets of the 100-structure test set).

mean held-out whole-Fv RMSDimprovement % (bootstrap mean, 95% CI)improved (full 100-set)
theory correction1.047612.94% [11.31, 14.60]99/100, p=3.5e-40
independently-calibrated constant1.047712.98% [11.38, 14.62]99/100, p=5.1e-42

The benefit of our theory-derived correction is very close to the calibrated constant. The calibrated c ranged 0.950–0.955 across splits which is close to the theory's parameter-free √q=0.949 under a 0 mean. Our method is unique compared to [1-3] in that it requires no calibration or retraining.

Added to the manuscript as Appendix "Comparison with an independently-calibrated constant correction" (Table "Theory-derived correction vs. an independently-calibrated constant scale factor").

Concentration approximation validity

"The approximation's validity conditions are insufficiently specified. Appendix A attributes neglected effects to terms vanishing with channel dimension, but approximating the expectation of a normalized random variable using expected statistics requires additional concentration assumptions. For a vector with one nonzero channel, unit affine scale, zero bias, and negligible numerical epsilon, the dropout-on mean is approximately (...), while the proposed correction factor approaches (...) as dimension increases. Thus, large dimension alone does not ensure accuracy; the authors should specify the required activation-distribution conditions and assess approximation errors."

Question: "What assumptions justify the concentration approximation? How accurate is it on actual model activations across correction sites?"

To show this empirically, we measured concentration directly on real FlashABB activations (8 headline structures, 132k positions, N=5000 masks/site). CV(V) across masks (the concentration operand) had a mean of 6.1% and a max of 14.6%. Closed-form vs. true large-N MC relative error had (mean 0.50%, p95 0.66%, max 1.76%). Transition sites (higher CV, 7-12%) show modestly higher error than IPA sites (flat 5.85% CV) consistent with, but only partially explained by, a standard delta-method second-order term (measured correlation r≈0.38 between CV(V)² and error; fitted slope ~3x the pure-denominator prediction, meaning most residual error comes from the other dropped term.

We separately checked whether a literal small-N Monte Carlo would do better. We ran N=100 real masks per site which gives 3.28% error (pure sampling noise) vs. the closed form's 0.50% (small fixed bias). The closed form beats real MC at this N because sampling variance dominates a bias this small and the concentration approximation holds reasonably well.

BatchNorm Appendix B.2 direction

"The BatchNorm extension appears to apply the correction in the wrong direction. Under Appendix B.2's stated objective, a zero-mean input with variance (...) has dropout-trained running variance (...), so the original inference output is already attenuated by (...). Applying the multiplicative correction from Eq. (5) shrinks it further, whereas recovering dropout-free normalization requires the inverse factor. This concerns an auxiliary extension and does not directly invalidate the main LayerNorm experiments, but it requires correction or clarification."

Question: "Please also clarify the direction of the BatchNorm correction in Appendix B.2."

We thank the reviewer for noticing this and have verified that it should be 1/c. We have corrected the manuscript to reflect this.

Fixed in the manuscript, Appendix B.2 (BatchNorm).

Independent validation of the 34-structure headline result

"Independent validation of the largest structural gains is limited. The headline ABB3 and FlashABB results use a 34-structure subset. The nanobody evaluation overlaps with training data for some models, and improvement on an additional Ibex training-set subset does not establish that its held-out CDRH3 regression is merely noise. Larger independent evaluations and paired uncertainty estimates would strengthen the main practical claim."

Question: "Can the authors report paired confidence intervals, improvement fractions, and regression magnitudes on a larger, independent structural test set?"

We had originally restricted the analysis to the set of 34 used in the Ibex manuscript which limits training leakage between methods. Here, we have expanded the analysis to the full test set of 100 used by ABB3 and FlashABB. We have also included the fraction of improved structures and associated p values.

ModelConditionWhole-FvImpr. (n improved, p)CDR-H3Impr. (n improved, p)
ABB3Raw1.2028—2.8349—
+DLC1.0476+12.90% (99/100, p≈0)2.7026+4.67% (80/100, p=1.4e-13)
FlashABBRaw1.2096—2.8594—
+DLC1.0813+10.60% (97/100, p≈0)2.7910+2.39% (61/100, p=0.0011)

Both models on both metrics were significant at the full 100-structure scale, including FlashABB's CDR-H3 effect, which was not significant on the smaller 34-subset (p=0.327) but is here (p=0.0011) with the greater statistical power of this test.

Added to the manuscript as Appendix "ABB3, FlashABB, and Ibex on ABB3/FlashABB's own held-out 100-structure test set" (Table "Uncorrected vs. corrected RMSDs on the full 100-structure held-out test set").

Ibex held-out CDR-H3 regression: noise or real?

"...improvement on an additional Ibex training-set subset does not establish that its held-out CDRH3 regression is merely noise." (same paragraph as quoted above)

We also expanded the Ibex results to the expanded set of 100. We have not verified whether Ibex contains training overlap with this set, but here we only measure DLC's improvement. Any memorization-related accuracy gains would apply equally to both off and on and shouldn't bias the difference.

Ibex variant34-structure held-out (CDR-H3)full 100-structure (CDR-H3)
1 member (member 0)+2.22%, 21/34, p=0.0615+3.86%, 69/100, p≈0
8-member ensemble (medoid)-1.37%, 18/34, p=0.52+2.40%, 68/100, p=0.023

The apparent 34-subset regression (-1.37%, not significant) flips to a real, significant improvement at the larger sample size (+2.40%, p=0.023) — the small-sample result was noise pointing the wrong direction, not a real regression that got masked by one lucky subset. Member 0's effect also strengthens with more data (+2.22%→+3.86%). Whole-Fv on the 100-set, ensemble: +2.56%, 71/100, p=7.0e-5.

Added to the manuscript in the same new appendix as above (Table "Ibex CDR-H3 improvement under the correction, 34-structure vs. full 100-structure").

Genie1/2/3 only evaluated on loss, no designability

"The broader application claims exceed the demonstrated outcomes. Genie1/2/3 are evaluated only on noise-prediction loss, without assessing generated-structure quality or designability. On the full QuickBind benchmark, 240/427 cases improve, but the mean RMSD reduction from 9.384 to 9.343 Å is not statistically significant under the reported test. Subsets selected using the uncorrected prediction's RMSD against the ground truth are useful diagnostics, but do not establish an overall docking benefit or a deployable criterion for applying the correction."

Question: "For the broader application claims, do the gains persist in actual Genie generation and standard docking success metrics?"

We built a full ProteinMPNN+ESMFold+TM-score designability pipeline for all three where we generate 30 backbones total (10 each at lengths 50, 100, and 150), with 8 ProteinMPNN sequences sampled per backbone. Each structure generated shares the seed between DLC off and on for both Genie and ProteinMPNN.

modelmean scTM Δ (on−off)scTM improvedpaired t-test p[email protected] off→on
Genie1-0.000214/300.99283.3%→90.0% (n.s.)
Genie2-0.042110/300.03086.7%→76.7%
Genie3+0.006114/300.502100%→100% (ceiling)

We see no difference in Genie1, a small improvement in Genie3, and a somewhat larger degradation of Genie2. Despite our efforts to reduce noise by sharing seeds, it is difficult to tell if this effect exceeds the noise threshold.

Added to the manuscript as a new Appendix "Genie designability" (Table "Genie1/2/3 designability under the correction").

However, we also reran Genie on CASP14, CASP15, and set of CAMEO structures used by ESMFold.

modelCASP14 loss (n=38)CASP15 loss (n=45)CAMEO loss (n=144)nanobody-100 loss (n=100)
Genie1+2.88%, 38/38, p=1.2e-34+2.71%, 45/45, p=1.5e-44+2.70%, 144/144, p=1.5e-141+3.12%, 100/100, p=1.4e-91
Genie2+2.71%, 38/38, p=1.1e-27+2.54%, 45/45, p=7.0e-33+2.46%, 144/144, p=7.7e-92+2.90%, 100/100, p=1.2e-67
Genie3+1.54%, 38/38, p=1.9e-23+1.52%, 45/45, p=2.1e-24+1.43%, 144/144, p=2.2e-70+1.39%, 99/100, p=1.8e-37

We found that DLC improved the loss for all models on all sets. It is unclear what is driving the gap between loss and designability, but we expect that DLC's ability to improve the underlying model at inference time can be effectively leveraged by future models.

Added to the manuscript as a new main-text Section "General-monomer evaluation: CASP14, CASP15, and CAMEO", folded into the same table as the ESMFold/OpenFold/AlphaFold2 result below.

QuickBind docking benefit not established

"On the full QuickBind benchmark, 240/427 cases improve, but the mean RMSD reduction from 9.384 to 9.343 Å is not statistically significant under the reported test. Subsets selected using the uncorrected prediction's RMSD against the ground truth are useful diagnostics, but do not establish an overall docking benefit or a deployable criterion for applying the correction." (same paragraph as quoted above)

The point of DLC is to allow the model to more accurately realize its internal theory of the final pose. Part of the difficulty here is that QuickBind performs relatively poorly on PoseBusters. If the underlying model is wrong about the structure, then DLC will not improve performance which degrades statistical power. By limiting our analysis to only those poses which the model was approximately correct about, we can see that DLC indeed produces better structures. Also note that we are specifically sampling the positive tails of the distribution, which, under a neutral effect, would appear to make predictions worse, whereas DLC still appears to make them better.

Figure 4: local site-level MC or full stochastic forward passes?

"Does Figure 4 average samples locally at each Dropout–LayerNorm site, as its axis and caption indicate, or average final coordinates from complete stochastic forward passes, as the surrounding text describes? These procedures differ; which supports the claim of matching MC dropout ensembles?"

This was an interesting point, we originally ran MC only locally on each site. End-to-end MC actually produces slightly better structures (on the order of 0.01-0.04 Å RMSD) and have added a note about this to the manuscript. Creating a method which can track the knock-on effects of variance changes through the whole model remains an interesting area of future research.

Added to the manuscript as a new sentence at the end of Section "Theory matches the true Monte Carlo expectation."

Comments on the paper

1 comment
Area_Chair_9zD9OpenReview ↗

Authors: please respond to your reviewers

To the paper authors: Please acknowledge and respond to the thoughtful replies by reviewers. They have raised some going points about strengths and weaknesses of your paper, as well as questions that merit replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.