- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper analyzes the problem of diffusion-generated structures and provides a predictive model capable of measuring whether a generated structure is suitable.
The authors investigate the meaningful characteristics in the structure selection process, relating both to the chemistry of the represented proteins and to the sampling procedure.
The work introduces AQA-Score, which performs this assessment for antibody-antigen interacting structures, providing a means to rank candidate structures with respect to an experimentally validated reference structure. The paper also introduces AF3-AQA, a benchmark that measures the effectiveness of the proposed algorithms, providing a fair baseline for comparison. The data are collected from SAbDab, filtered, and clustered into antibody and antigen clusters.
By sampling trajectories for each test target, the authors independently generate pools of 100 and 1,000 AF3 structures, retaining confidence scores and contact probabilities. Experimental structures provide reference labels for evaluation rather than inputs to the scoring model.
AQA-Score is introduced as a predictive model that combines residue-pair compatibility with structural evidence from each candidate. The first module encodes the chains and is pretrained to predict interfaces, while the second module predicts residue-pair compatibility. Once trained, the models work jointly to assess candidate interfaces. The model also predicts contact recovery and the two structural-error components used in DockQ, combining them into a final ranking score. These components are supervised during training via a selection-regret loss that focuses on prioritizing the best structure in each training pool.
The model is evaluated against baseline metrics on the test split of AF3-AQA across both the 100- and 1,000-structure splits. The results show marginal improvements across the reported metrics.
The authors also analyze the results by grouping data according to the quality of the available structures, observing larger gains where data quality is higher and negligible improvements otherwise. Finally, the paper reports a comparison of the components included in AQA-Score; while all components prove useful, the relative contribution or optimal proportion of each is not further examined.
Critical strengths
Overall, the paper is clear, addresses a relevant problem with few existing solutions, and presents an interesting approach. The dataset itself represents a valuable contribution that could be adopted in the future for studying antibody-antigen interactions. The experimental section is thorough, including training losses alongside their individual components. Additionally, providing candidate pool sizes of varied cardinalities (100 vs. 1.000) allows for evaluating performance scalability. In the conclusion, the authors thoughtfully analyze limitations, offer observations on the method, and suggest promising future directions.
Critical weaknesses
The methodology lacks clear overall organization and formal notation, both in the score definition and in the dataset construction. The way biological domain knowledge is introduced in certain sections makes comprehension difficult. I suggest the authors add a preliminary section to clarify notation and introduce the necessary biological concepts.
The authors should evaluate additional benchmark models on their proposed dataset, as there is currently limited evidence regarding its broader applicability. A more in-depth characterization of the dataset is also needed to provide deeper insights into the data itself. Furthermore, the performance gains obtained across the evaluated cases are not consistently substantial. This highlights a lack of compelling evidence for the method's effectiveness, exacerbated by limited baseline comparisons and an absence of analysis explaining the underlying results.
Questions for authors
a. Could you provide a better characterization of the AF3-AQA dataset, including more insights into the generation process?
b. Could you clarify the rationale behind adding the two additional loss components to the model?
c. What prevents the inclusion of additional baselines or data in the comparison table?
d. I find it a really critic point the lack of insights on the generalization on unseen antigen families. If possible I would integrate them.
6 comments
Response to Reviewer 4vsH (2/3)
W2 & Q(c). Baseline Coverage and Additional Comparisons
Takeaway: We added pDockQ, a supervised fusion baseline, and a retrained external ranker. These experiments expand the comparison beyond the original confidence metrics and learned models, while AQA-Score retains the highest Top-1 DockQ point estimate.
AF3-AQA focuses on ranking AF3 antibody–antigen candidates. We therefore prioritized methods directly relevant to this setting and applicable to the same candidate pools.
The original comparison included iPTM, ipSAE, PAE and contact probability scores, Rosetta energy and geometry signals, and three learned rankers: DeepRank-Ab, ABAG-Rank, and ARID-sf.
Following the reviewer's suggestion, we added:
- pDockQ, as an additional protein complex quality estimator.
- Logistic Fusion (C+E+G), combining confidence, energy, and geometry features.
- Retrained ABAG-Rank, using AF3-AQA supervised training data to examine the effect of training distribution.
Table R1. Additional baselines on AF3-AQA-100 (391 targets).
| Method | DockQ@1 | DockQ@5 |
|---|---|---|
| iPTM | 0.3449 | 0.3798 |
| ipSAE | 0.3302 | 0.3846 |
| pDockQ | 0.3170 | 0.3767 |
| Logistic Fusion (C+E+G) | 0.3464 | 0.3993 |
| ABAG-Rank (original) | 0.2850 | 0.3540 |
| ABAG-Rank (retrained) | 0.3277 | 0.3847 |
| DeepRank-Ab | 0.3331 | 0.4045 |
| AQA-Score | 0.3631 | 0.4023 |
The fusion baseline reaches 0.3464/0.3993, showing that combining existing signals already provides competitive performance. AQA-Score achieves higher Top-1 quality, although the Top-5 difference is small.
Retraining ABAG-Rank improves DockQ@1/5 from 0.2850/0.3540 to 0.3277/0.3847, indicating that differences in training data are important when comparing external models.
We also evaluated pDockQ on the unseen antigen subset, where it achieves 0.3231/0.3716, compared with 0.3516/0.3921 for AQA-Score.
The updated comparison covers native confidence scores, complex quality estimators, physical and geometric signals, simple supervised fusion, and specialized learned rankers. We do not claim to exhaust every available scoring function, and will clarify the selection criteria and training protocols in the revision.
W2. Modest and Uneven Performance Improvements
Takeaway: The largest improvement occurs in the Medium oracle quality stratum, where AQA-Score improves DockQ@1 by 0.0560 over iPTM. This group accounts for only 28.9% of the targets but contributes 88.6% of the overall Top-1 gain.
AF3-AQA evaluates candidate selection rather than generation. If no acceptable candidate exists, ranking cannot recover one. Conversely, confidence scores already perform well on some targets with high quality candidates, leaving less room for improvement.
Our stratified analysis shows the largest gains in the Medium group (113 targets):
| Method | DockQ@1 | DockQ@5 |
|---|---|---|
| iPTM | 0.429 | 0.484 |
| ipSAE | 0.417 | 0.497 |
| DeepRank-Ab | 0.446 | 0.550 |
| AQA-Score | 0.485 | 0.539 |
| Oracle | 0.673 | 0.673 |
Compared with iPTM, the improvements are +0.0560/+0.0549 in DockQ@1/5. Both remain positive under target and antigen cluster bootstrap:
| Metric | Target 95% CI | Antigen cluster 95% CI |
|---|---|---|
| DockQ@1 gain | [0.0237, 0.0908] | [0.0053, 0.0970] |
| DockQ@5 gain | [0.0228, 0.0902] | [0.0079, 0.0920] |
The Medium group contributes 88.6% of the overall Top-1 gain and 70.7% of the Top-5 gain. In contrast, in the High group, iPTM already achieves DockQ@1 of 0.774, compared with 0.775 for AQA-Score.
We will retain the full 391 target evaluation as the primary result and use the stratified analysis to explain why the improvements vary across targets.
Response to Reviewer 4vsH (3/3)
Q(b). Motivation for the Additional Loss Components
Takeaway: Predicting the three DockQ components separately provides structural supervision that a single scalar target cannot capture. The selection loss addresses a different objective: choosing the best candidate within each pool, rather than minimizing average prediction error alone.
The final score follows the standard DockQ decomposition:
where the three terms represent contact recovery, transformed interface RMSD, and transformed antibody ligand RMSD.
However, regression on the final scalar and regression on individual components are not equivalent. With scalar supervision, an underestimated contact term can be canceled by an overestimated geometry term, producing a seemingly accurate final DockQ despite inaccurate component predictions.
We therefore supervise the components separately and use local contact and residue error losses to provide more specific structural signals.
Quality prediction and candidate selection are also different objectives. A model may predict the quality of most candidates accurately but still rank a worse pose above the best available structure.
We address this using pool wise selection regret:
where is the predicted selection probability and is the oracle DockQ within the candidate pool.
Loss weights balance the different numerical and gradient scales of component regression, local supervision, and selection regret. They do not change the definition of DockQ.
The existing ablations support these choices:
| Variant | DockQ@1 | DockQ@5 |
|---|---|---|
| Full AQA-Score | 0.3631 | 0.4023 |
| Aggregate score supervision | 0.3509 | 0.3978 |
| Without selection loss | 0.3475 | 0.3924 |
Replacing component regression with aggregate score supervision reduces DockQ@1 to 0.3509, while removing selection regret reduces it to 0.3475.
These results support the two training objectives within our current setting. We will clarify their respective roles and mathematical definitions in the manuscript.
Q(d). Generalization to Unseen Antigens
Takeaway: AQA-Score remains competitive on 163 targets with antigen sequence clusters unseen during supervised training, achieving the highest Top-1 DockQ point estimate among the evaluated methods.
Among 391 test targets, 163 (41.7%) have antigen sequence clusters absent from the audited supervised training set.
Table R2. Performance on unseen antigen sequence clusters (163 targets).
| Method | DockQ@1 | DockQ@5 | [email protected]/1 | [email protected]/5 |
|---|---|---|---|---|
| iPTM | 0.3401 | 0.3736 | 50.3% | 52.8% |
| ipSAE | 0.3284 | 0.3729 | 49.1% | 51.5% |
| pDockQ | 0.3231 | 0.3716 | 46.0% | 50.9% |
| CP-H3 max | 0.3337 | 0.3734 | 47.2% | 50.3% |
| ABAG-Rank | 0.2877 | 0.3455 | 40.5% | 47.2% |
| ARID-sf | 0.2087 | 0.3041 | 29.4% | 39.9% |
| DeepRank-Ab | 0.3387 | 0.4031 | 49.7% | 55.2% |
| AQA-Score | 0.3516 | 0.3921 | 50.9% | 53.4% |
AQA-Score improves DockQ@1/5 over iPTM by 0.0115/0.0185, and over ipSAE by 0.0232/0.0192. It achieves the highest Top-1 point estimate, while DeepRank-Ab remains stronger at Top-5.
These results indicate competitive performance on antigen sequence clusters unseen during supervised training. We do not interpret the positive differences as statistically established improvements without paired confidence intervals.
We also distinguish unseen sequence clusters from broader antigen families. The present evaluation does not guarantee complete independence from homologous proteins in the pretraining corpus, and we will state this limitation explicitly.
We appreciate the reviewer's constructive feedback on presentation, evaluation coverage, and generalization. We will incorporate these clarifications and additional results into the revised manuscript. We hope that the new evidence addresses the reviewer's main concerns, and we would appreciate it if the reviewer could reconsider their assessment in light of these results.
Response to Reviewer 4vsH (1/3)
We thank the reviewer for the careful assessment of AF3-AQA and for recognizing the relevance of the problem, the value of the benchmark, and the breadth of our experimental analysis. We appreciate the suggestions concerning presentation, benchmark characterization, baseline coverage, and generalization. We address each concern below.
W1. Method Organization and Notation
Takeaway: We agree that the presentation can be improved, particularly the distinction between the evaluation metric, predicted quality components, and training objectives. We will revise the organization and add a short preliminary section.
The current manuscript introduces several concepts together, including antibody–antigen quality assessment, pretrained binding representations, candidate assessment, and structural supervision. We recognize that this may make the method harder to follow for readers outside structural biology.
We will clarify three aspects:
These changes do not alter the method but should make its assumptions and information flow easier to understand.
Q(a). Benchmark Characterization and Data Generation
Takeaway: AF3-AQA uses two independently sampled candidate pools to separate the quality made available by AF3 sampling from the ability of ranking methods to select good candidates. Increasing the pool size substantially improves oracle quality, but the improvement in selected quality is much smaller.
We constructed AF3-AQA from 20,943 SAbDab antibody–antigen entries. After filtering for experimental structure quality, chain completeness, CDR integrity, antigen length, and interface contacts, we retained 3,316 valid antibody–antigen pairs.
We then applied MMseqs2 clustering and a temporal cutoff of September 30, 2021, with partitioning at the antibody–antigen cluster pair level. The resulting public test set contains 391 targets.
For each test target, we independently generated two AF3 candidate pools:
The two pools use disjoint random seeds. AF3-AQA-100 is therefore not a subset of AF3-AQA-1000.
We further characterize the pools using four strata defined by the best available DockQ:
Increasing the number of candidates from 100 to 1,000 improves mean oracle DockQ from 0.5015 to 0.6015. In comparison, iPTM DockQ@1 increases only from 0.3449 to 0.3881.
This difference shows that additional sampling makes better structures available, but existing selectors do not fully recover the resulting improvement.
We will make these statistics and the sampling protocol more prominent in the revised manuscript, together with a clearer explanation of candidate availability, oracle quality, and selection difficulty.