- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents H3Fold, a conditional diffusion framework for generating antibody CDR-H3 Cα conformations. It combines loop-specific representations from H3BERTa with VH sequence representations from Ab-RoBERTa. A four-stage curriculum transfers geometric knowledge from general protein loops to immune loops and finally CDR-H3 structures. The method also introduces a stem-grounded canonical coordinate frame for loop placement. The evaluation includes architectural and training ablations, sequence-clustered internal splits, and an external temporal holdout. The results indicate promising inference efficiency, while revealing a substantial gap between internal and external performance and between oracle and practical candidate selection.
Critical strengths
-
A well-motivated approach to efficient loop modeling. The combination of loop-local and VH-level sequence conditioning, progressive training, and a stem-grounded coordinate frame is technically reasonable. Focusing computation on CDR-H3 is relevant to iterative antibody engineering, where repeated full-structure prediction can be costly. The framework therefore has plausible practical value even without achieving the best single-structure accuracy.
-
Substantial empirical investigation and transparent failure analysis. The ablations provide useful evidence about the importance of pretraining, immune-loop data, and sequence conditioning. The external holdout and the investigation of the internal–external performance gap are particularly valuable. Reporting unresolved failures, including the difficulty of cysteine-containing loops and the unexplained residual performance gap, makes the evaluation more informative than an internal benchmark alone.
Critical weaknesses
-
The central accuracy–efficiency advantage is insufficiently established against directly relevant alternatives. The benchmark primarily compares H3Fold with full-antibody structure predictors, whereas H3Fold generates only a CDR-H3 Cα trace. Although Appendix H uses the same GPU, the methods perform different amounts of work and produce different outputs. A dedicated loop predictor such as ABlooper, discussed in the related work, would provide a more direct comparison with conditioning requirements explicitly accounted for. This matters because practical candidate selection remains less accurate than several evaluated predictors: on the common 69-case holdout, Table 4 reports median Kabsch RMSDs of 2.95 Å for a random H3Fold sample and 2.82 Å for centroid-based selection, compared with 2.19 Å for ESMFold. The 2.31 Å oracle result shows the potential benefit of selecting a better candidate from the generated ensemble, but this selection requires access to the reference structure. A controlled accuracy–runtime comparison is needed to establish the proposed method’s practical advantage.
-
The evaluation provides limited evidence for the additional value of the generated ensembles. The internal validation and test sets retain one reference structure per unique CDR-H3 sequence, and the main metrics primarily assess agreement with that reference. Best-of-five RMSD measures the accuracy of the best sampled candidate relative to a reference structure, but it does not establish recovery of distinct plausible conformations for a given input. The aggregate pseudo-dihedral analysis offers some geometric evidence, yet sequence-conditioned coverage, diversity versus accuracy, and structural validity in the surrounding framework remain insufficiently characterized. These evaluations are consequential because ensemble generation is a central motivation of the paper and could justify the method despite its weaker practical single-candidate accuracy.
Questions for authors
-
Direct comparison and practical advantage: Can the authors provide a comparison with a dedicated loop-modeling method such as ABlooper, explicitly accounting for differences in required inputs? An accuracy–runtime comparison under matched time or sampling budgets would help determine whether H3Fold offers a meaningful advantage for the intended workflow.
-
Evidence for ensemble utility: What evidence shows that the generated samples cover distinct plausible conformations for the same input, beyond improving RMSD to one reference? Where suitable experimental conformations are available, a coverage analysis would be useful. Otherwise, can the authors provide a joint assessment of diversity, reference accuracy, and framework compatibility as the sampling budget increases?
-
Inference-time clarification: Section 3.3 reports approximately 3.5 seconds for five candidates, whereas Table 11 reports 0.70 seconds per Fv and states that five samples were generated per loop. Does 0.70 seconds cover one candidate or all five? Please clarify the batching strategy, whether runtime is normalized per candidate or per input Fv, and which preprocessing steps are included. For placing generated loops into an antibody framework, which stem/backbone coordinates are assumed available, how are they obtained in the intended inference workflow, and is this cost included in the reported runtime?
-
Consistency of random and oracle statistics: In Table 10, some subgroups have a higher oracle median than random-sample median; for example, the holdout Q1 subgroup reports 3.382 Å versus 3.138 Å. Is the random sample drawn from the same five-candidate pool used for oracle selection? If so, these values appear inconsistent with the best-of-five definition. If different draws or aggregation procedures are used, please explain the protocol and its implications for the gap analysis.
W1 — Accuracy-efficiency comparison
Three loop-local baselines are added with conditioning requirements made explicit: H3Fold, ABlooper, Ibex and ABodyBuilder3 all require a framework or stem coordinates; IgFold, ESMFold, ColabFold, Chai-1 and tFold predict the full Vᴴ from sequence. The benchmark table is split into these two blocks.
Ibex 1.187 Å (best-of-8: 0.846, n=68), ABodyBuilder3 1.893 Å (n=68), ABlooper 2.671 Å (oracle 2.125, n=45). All three exceed our random sample; Ibex and ABodyBuilder3 exceed our oracle. Leakage is disclosed as described under Reviewer 2, W2. The SOTA accuracy claim is withdrawn. The paper is reframed around cheap, diverse, stem-grounded Cα ensemble generation at 0.85 s per candidate.
Paired Wilcoxon over the 69 shared holdout loops: our random sample vs ColabFold Δ=+0.747 Å (p=6.3×10⁻⁶); vs ESMFold Δ=+0.756 Å (p=1.8×10⁻⁴); vs tFold Δ=+0.659 Å (p=4.3×10⁻⁵). Oracle vs oracle: ColabFold still ahead (1.803 vs 2.307 Å, p=6.6×10⁻⁴).
W2 — Ensemble utility
A coverage analysis is not feasible with the available data, as described under Reviewer 1, Q3. No calibrated-ensemble claim is made.
The gap between oracle and reference-free selection as budget increases is characterised. Oracle falls from 2.265 to 1.894 Å from N=5 to N=25 on the holdout. Centroid moves from 2.917 to 2.609 Å. Three rankers were tested: centroid closes 49 % of the rand-to-oracle margin on validation but only 34 % on the holdout; a geometry-violation score closes 26 % on validation and costs 22 % on the holdout; mean-Cα ensemble selection closes 28 %/27 %. These results characterise the current limit on reference-free selection rather than making a positive ensemble claim.
Q1 — Direct comparison with loop-local methods
ABlooper, ABodyBuilder3 and Ibex are added with explicit conditioning requirements and leakage disclosure (see W1 above). The honest summary: at equal or greater compute cost, Ibex reaches 1.19 Å single-sample against our 2.95 Å random. H3Fold's advantage is in sample speed per candidate and stem-grounded placement, not in accuracy.
Q2 — Evidence for ensemble utility
No coverage evidence exists in our data for the reason described under W2. The joint assessment is delivered as (1) the N=25 accuracy–runtime curve under oracle and centroid selection and (2) the placement analysis giving terminus error, span error and framework clash rate in both crystal and predicted Fv contexts.
Q3 — Inference-time scope
The 0.70 s figure was per candidate; five candidates cost 3.5 s. Re-measured unbatched on A40: 0.851 s at N=1, 4.168 s/Fv at N=5. The H3BERTa and Ab-RoBERTa passes cost 19 ms (0.5 % of the N=5 total) and are included. DDIM sampling alone is 4.122 s. Table 11 now carries explicit per-candidate and per-Fv columns with PLM-included flags.
Stem coordinates are supplied from the framework (IMGT 104/118 Cα). Their cost is not in Table 11. The placement analysis quantifies what using a predicted framework costs: IgFold 26.1 % of termini beyond 1 Å; ABodyBuilder3 13.2 %; crystal framework 7.7 %.
Q4 — Oracle-random inversion
Unfortunately, the oracle column in Table 10 was DirectSel throughout. DirectSel is the sample with the lowest Direct RMSD (no superposition), not argmin Kabsch, and it can exceed a random draw. Under the true argmin-Kabsch oracle, holdout Q1 gives rand 3.138 Å, oracle 3.048 Å. The inversion disappears. Two further inversions found in the same audit are corrected (see Reviewer 2, W1). The evaluation pipeline now enforces oracle ≤ centroid ≤ rand per loop before writing results.
The L12–15 "model-level failure" open problem in §4.3.1 and the Conclusion rested on a 2.5 % DirectSel gain. The true argmin-Kabsch gain is 26 % (2.193 to 1.624 Å), comparable to the test set. The open-problem claim is withdrawn.