- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose Dropout–LayerNorm Correction (DLC), a closed-form, inference-only hook that does not require retraining. It corrects the systematic overshoot, relative to the training-time expectation, that appears at evaluation when Dropout is immediately followed by LayerNorm in AlphaFold2-style structure modules. On antibody and nanobody data, the authors show that it improves the accuracy of a range of structure-prediction models.
Critical strengths
The finding is both concrete and useful. A Dropout layer immediately followed by LayerNorm leaves a systematic gap between the training-time expectation and the evaluation output, and the authors’ closed-form correction matches the Monte Carlo mean on ABodyBuilder3. On antibody data the same inference-only hook improves structure accuracy, most clearly for ABB3, with no retraining and negligible cost. This is a practical guide for inference in later structure models that keep the Dropout-then-LayerNorm pattern.
Critical weaknesses
-
The experiments do not support the broad claim that DLC “improves protein structure models.” Every reported structure-prediction result is on antibodies or nanobodies. There is no CASP, CAMEO, or other general monomer benchmark, so the effect on ordinary monomers is unknown. Most of the tested models are antibody-specific. The general folders, ESMFold and OpenFold, are also evaluated only on these immunoglobulin sets, where the gains are about 0.2%–0.8%. Current mainstream predictors such as AlphaFold3 and Boltz use residual updates rather than Dropout immediately before LayerNorm, so DLC may not apply to them.
-
The comparisons are incomplete. The headline antibody numbers use 34 structures, whereas ABodyBuilder3’s own test set has 100. Genie1/2/3 are general backbone generators, but they are compared only by denoising loss on 100 nanobodies, with no measure of structural accuracy.
I like this simple but effective idea, and I would raise my score if these concerns were addressed.
Questions for authors
- Does DLC improve accuracy on every structure? For the models that currently report only mean RMSD, especially ABB3, FlashABB, and Ibex, could you show the per-structure comparison with and without DLC?
- Could you add results on general monomers, for example CASP or CAMEO? If not, could the claim be narrowed from protein structure models to antibody and nanobody models?
- Could you report ABB3 on its full 100-structure test set, and evaluate Genie1/2/3 by structural quality rather than denoising loss alone?
3 comments
Evaluate Genie1/2/3 by structural quality, not just denoising loss
"Genie1/2/3 are general backbone generators, but they are compared only by denoising loss on 100 nanobodies, with no measure of structural accuracy." / "...evaluate Genie1/2/3 by structural quality rather than denoising loss alone?"
We built a full ProteinMPNN+ESMFold+TM-score designability pipeline for all three where we generate 30 backbones total (10 each at lengths 50, 100, and 150), with 8 ProteinMPNN sequences sampled per backbone. Each structure generated shares the seed between DLC off and on for both Genie and ProteinMPNN.
| model | mean scTM Δ (on−off) | scTM improved | paired t-test p | [email protected] off→on |
|---|---|---|---|---|
| Genie1 | -0.0002 | 14/30 | 0.992 | 83.3%→90.0% (n.s.) |
| Genie2 | -0.0421 | 10/30 | 0.030 | 86.7%→76.7% |
| Genie3 | +0.0061 | 14/30 | 0.502 | 100%→100% (ceiling) |
We see no difference in Genie1, a small improvement in Genie3, and a somewhat larger degradation of Genie2. Despite our efforts to reduce noise by sharing seeds, it is difficult to tell if this effect exceeds the noise threshold. However, as shown above, DLC improved the loss for all models on all sets. It is unclear what is driving the gap between loss and designability, but we expect that DLC's ability to improve the underlying model at inference time can be effectively leveraged by future models.
Added to the manuscript in the same new Appendix "Genie designability" as above.
Does DLC improve accuracy on every structure? Per-structure comparison for ABB3/FlashABB/Ibex?
"Does DLC improve accuracy on every structure? For the models that currently report only mean RMSD, especially ABB3, FlashABB, and Ibex, could you show the per-structure comparison with and without DLC?"
We have added the fraction of improved structures throughout, and added a new per-structure RMSD-shift figure (Figure 6) for ABB3, FlashABB, and Ibex on the 100-structure held-out set, in the same style as the existing ESMFold/OpenFold per-structure shift figure (Figure 7). Briefly, whole antibody structures and Genie loss predictions improve almost uniformly, while CDRH3 and general monomers improve at varying levels. When running Ibex on the full ABB3 test set, we see that the apparent CDRH3 degradation was a statistical artefact which is resolved under the larger test set.
| Ibex variant | 34-structure held-out (CDR-H3) | full 100-structure (CDR-H3) |
|---|---|---|
| 1 member (member 0) | +2.22%, 21/34, p=0.0615 | +3.86%, 69/100, p≈0 |
| 8-member ensemble (medoid) | -1.37%, 18/34, p=0.52 | +2.40%, 68/100, p=0.023 |
Added to the manuscript in the same new appendix as above.
Thank you for the authors' efforts in addressing my concerns. The experiments on CASP and CAMEO substantially strengthen the claims of this work. I think this work is interesting and meaningful, and I have raised my score to 3.
I also have a question out of curiosity: Is the Dropout-then-LayerNorm design commonly used in other fields, such as computer vision and NLP? If so, could DLC be applied to these domains as well?
Thank you for your review of our work. We have performed new experiments/analysis and addressed your questions below. We hope you will agree that these experiments have increased the robustness of the manuscript and are happy to answer any questions you may still have.
Broad claim unsupported: no CASP/CAMEO, general-monomer effect unknown
To addres this, we ran ESMFold, OpenFold, and AlphaFold2 (new) on CASP14, CASP15, and CAMEO (144 targets from an archived window chosen to approximate ESMFold's own CAMEO benchmarking methodology). Here are the domain-level paired t-tests throughout:
DLC improves performance in 8/9 cases and 4/4 which have p<0.05. This provides a computationally neglible, calibration-free improvement to three very widely used models.
Added to the manuscript as a new main-text Section "General-monomer evaluation: CASP14, CASP15, and CAMEO".
Additionally, the insights from this work may explain why models like AlphaFold3 remove dropout. It is possible that reintroducing the Dropout+LayerNorm pattern under the DLC correction could improve generalization in future models.
Narrow the claim from "protein structure models" to antibody/nanobody?
In addition to the experiments above, we also ran Genie 1, 2, and 3 on CASP14, CASP15, and CAMEO.
We saw that the effect held up significanly in all cases. The largest structure prediction improvements were found for FlashABB and ABB3 which are both antibody-specific, but we believe that this is because they are domain-specific rather than a particular idiosyncrasy of antibodies.
Report ABB3 on its full 100-structure test set
We had originally restricted the analysis to the set of 34 used in the Ibex manuscript which limits training leakage between methods. Here, we have expanded the analysis to the full test set of 100 used by ABB3 and FlashABB. We have also included the fraction of improved structures and associated p values.
Added to the manuscript in the same new appendix as above ("ABB3, FlashABB, and Ibex on ABB3/FlashABB's own held-out 100-structure test set").