acceptodds
Under review as a conference paper at ICLR 2027

AF3-AQA: Assessing and Selecting Antibody–Antigen Structures from AlphaFold3 Ensembles

Abstract

Accurate antibody–antigen complex structures are important for structure-based antibody discovery and design. Although AlphaFold3 (AF3) can generate diverse candidate structures through stochastic sampling, these candidates often differ substantially in binding pose and structural accuracy, making reliable candidate selection a critical challenge. We study this problem by systematically analyzing scoring signals and developing a learned model to assess candidate quality. To support this study, we introduce AF3-AQA (AlphaFold3 Antibody–Antigen Quality Assessment), comprising 430,100 predicted structures for 391 experimentally resolved targets, with independently sampled pools of 100 and 1,000 candidates per target. Under a unified evaluation protocol, we compare AF3 confidence scores and external rankers, and investigate how different factors contribute to candidate selection. Our analyses show that regional restriction has feature-dependent benefits, aggregation can substantially change selection performance, and strong scalar scores can still fail to distinguish different candidate poses. We further introduce AQA-Score, which transfers single-chain interface-site and residue-pair compatibility representations into candidate-specific assessment with additional structure and energy features. On AF3-AQA-100, AQA-Score improves DockQ@1/5 from 0.345/0.380 for iPTM to 0.363/0.402. Together, these results characterize the strengths and limitations of current selection signals and establish a learned approach for improving antibody–antigen structure selection.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
AlphaFold3, Antibody–antigen complexes, Structure ranking, Quality assessment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4vsHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper analyzes the problem of diffusion-generated structures and provides a predictive model capable of measuring whether a generated structure is suitable.

The authors investigate the meaningful characteristics in the structure selection process, relating both to the chemistry of the represented proteins and to the sampling procedure.

The work introduces AQA-Score, which performs this assessment for antibody-antigen interacting structures, providing a means to rank candidate structures with respect to an experimentally validated reference structure. The paper also introduces AF3-AQA, a benchmark that measures the effectiveness of the proposed algorithms, providing a fair baseline for comparison. The data are collected from SAbDab, filtered, and clustered into antibody and antigen clusters.

By sampling trajectories for each test target, the authors independently generate pools of 100 and 1,000 AF3 structures, retaining confidence scores and contact probabilities. Experimental structures provide reference labels for evaluation rather than inputs to the scoring model.

AQA-Score is introduced as a predictive model that combines residue-pair compatibility with structural evidence from each candidate. The first module encodes the chains and is pretrained to predict interfaces, while the second module predicts residue-pair compatibility. Once trained, the models work jointly to assess candidate interfaces. The model also predicts contact recovery and the two structural-error components used in DockQ, combining them into a final ranking score. These components are supervised during training via a selection-regret loss that focuses on prioritizing the best structure in each training pool.

The model is evaluated against baseline metrics on the test split of AF3-AQA across both the 100- and 1,000-structure splits. The results show marginal improvements across the reported metrics.

The authors also analyze the results by grouping data according to the quality of the available structures, observing larger gains where data quality is higher and negligible improvements otherwise. Finally, the paper reports a comparison of the components included in AQA-Score; while all components prove useful, the relative contribution or optimal proportion of each is not further examined.

Critical strengths

Overall, the paper is clear, addresses a relevant problem with few existing solutions, and presents an interesting approach. The dataset itself represents a valuable contribution that could be adopted in the future for studying antibody-antigen interactions. The experimental section is thorough, including training losses alongside their individual components. Additionally, providing candidate pool sizes of varied cardinalities (100 vs. 1.000) allows for evaluating performance scalability. In the conclusion, the authors thoughtfully analyze limitations, offer observations on the method, and suggest promising future directions.

Critical weaknesses

The methodology lacks clear overall organization and formal notation, both in the score definition and in the dataset construction. The way biological domain knowledge is introduced in certain sections makes comprehension difficult. I suggest the authors add a preliminary section to clarify notation and introduce the necessary biological concepts.

The authors should evaluate additional benchmark models on their proposed dataset, as there is currently limited evidence regarding its broader applicability. A more in-depth characterization of the dataset is also needed to provide deeper insights into the data itself. Furthermore, the performance gains obtained across the evaluated cases are not consistently substantial. This highlights a lack of compelling evidence for the method's effectiveness, exacerbated by limited baseline comparisons and an absence of analysis explaining the underlying results.

Questions for authors

a. Could you provide a better characterization of the AF3-AQA dataset, including more insights into the generation process?

b. Could you clarify the rationale behind adding the two additional loss components to the model?

c. What prevents the inclusion of additional baselines or data in the comparison table?

d. I find it a really critic point the lack of insights on the generalization on unseen antigen families. If possible I would integrate them.

6 comments

Response to Reviewer 4vsH (1/3)

We thank the reviewer for the careful assessment of AF3-AQA and for recognizing the relevance of the problem, the value of the benchmark, and the breadth of our experimental analysis. We appreciate the suggestions concerning presentation, benchmark characterization, baseline coverage, and generalization. We address each concern below.

W1. Method Organization and Notation

Takeaway: We agree that the presentation can be improved, particularly the distinction between the evaluation metric, predicted quality components, and training objectives. We will revise the organization and add a short preliminary section.

The current manuscript introduces several concepts together, including antibody–antigen quality assessment, pretrained binding representations, candidate assessment, and structural supervision. We recognize that this may make the method harder to follow for readers outside structural biology.

We will clarify three aspects:

  • Task definition: AF3-AQA evaluates the selection of accurate structures from an existing AF3 candidate pool, rather than predicting new binding poses or binding affinity.
  • Quality definition: We use antigen fixed AbAg-DockQ, treating the antigen as the receptor and the antibody as the ligand. We will distinguish the reference DockQ, its three components, and the score predicted by AQA-Score.
  • Model organization: We will present the method in the order of binding representations (Modules A/B), candidate structural evidence, quality prediction, and training objectives, with consistent notation throughout.

These changes do not alter the method but should make its assumptions and information flow easier to understand.

Q(a). Benchmark Characterization and Data Generation

Takeaway: AF3-AQA uses two independently sampled candidate pools to separate the quality made available by AF3 sampling from the ability of ranking methods to select good candidates. Increasing the pool size substantially improves oracle quality, but the improvement in selected quality is much smaller.

We constructed AF3-AQA from 20,943 SAbDab antibody–antigen entries. After filtering for experimental structure quality, chain completeness, CDR integrity, antigen length, and interface contacts, we retained 3,316 valid antibody–antigen pairs.

We then applied MMseqs2 clustering and a temporal cutoff of September 30, 2021, with partitioning at the antibody–antigen cluster pair level. The resulting public test set contains 391 targets.

For each test target, we independently generated two AF3 candidate pools:

DatasetSeeds × SamplesCandidates per targetTotal
AF3-AQA-10020 × 510039,100
AF3-AQA-1000200 × 51,000391,000

The two pools use disjoint random seeds. AF3-AQA-100 is therefore not a subset of AF3-AQA-1000.

We further characterize the pools using four strata defined by the best available DockQ:

Oracle quality stratumPool 100Pool 1000
Below acceptable11766
Acceptable6565
Medium113123
High96137

Increasing the number of candidates from 100 to 1,000 improves mean oracle DockQ from 0.5015 to 0.6015. In comparison, iPTM DockQ@1 increases only from 0.3449 to 0.3881.

This difference shows that additional sampling makes better structures available, but existing selectors do not fully recover the resulting improvement.

We will make these statistics and the sampling protocol more prominent in the revised manuscript, together with a clearer explanation of candidate availability, oracle quality, and selection difficulty.

Response to Reviewer 4vsH (2/3)

W2 & Q(c). Baseline Coverage and Additional Comparisons

Takeaway: We added pDockQ, a supervised fusion baseline, and a retrained external ranker. These experiments expand the comparison beyond the original confidence metrics and learned models, while AQA-Score retains the highest Top-1 DockQ point estimate.

AF3-AQA focuses on ranking AF3 antibody–antigen candidates. We therefore prioritized methods directly relevant to this setting and applicable to the same candidate pools.

The original comparison included iPTM, ipSAE, PAE and contact probability scores, Rosetta energy and geometry signals, and three learned rankers: DeepRank-Ab, ABAG-Rank, and ARID-sf.

Following the reviewer's suggestion, we added:

  • pDockQ, as an additional protein complex quality estimator.
  • Logistic Fusion (C+E+G), combining confidence, energy, and geometry features.
  • Retrained ABAG-Rank, using AF3-AQA supervised training data to examine the effect of training distribution.

Table R1. Additional baselines on AF3-AQA-100 (391 targets).

MethodDockQ@1DockQ@5
iPTM0.34490.3798
ipSAE0.33020.3846
pDockQ0.31700.3767
Logistic Fusion (C+E+G)0.34640.3993
ABAG-Rank (original)0.28500.3540
ABAG-Rank (retrained)0.32770.3847
DeepRank-Ab0.33310.4045
AQA-Score0.36310.4023

The fusion baseline reaches 0.3464/0.3993, showing that combining existing signals already provides competitive performance. AQA-Score achieves higher Top-1 quality, although the Top-5 difference is small.

Retraining ABAG-Rank improves DockQ@1/5 from 0.2850/0.3540 to 0.3277/0.3847, indicating that differences in training data are important when comparing external models.

We also evaluated pDockQ on the unseen antigen subset, where it achieves 0.3231/0.3716, compared with 0.3516/0.3921 for AQA-Score.

The updated comparison covers native confidence scores, complex quality estimators, physical and geometric signals, simple supervised fusion, and specialized learned rankers. We do not claim to exhaust every available scoring function, and will clarify the selection criteria and training protocols in the revision.

W2. Modest and Uneven Performance Improvements

Takeaway: The largest improvement occurs in the Medium oracle quality stratum, where AQA-Score improves DockQ@1 by 0.0560 over iPTM. This group accounts for only 28.9% of the targets but contributes 88.6% of the overall Top-1 gain.

AF3-AQA evaluates candidate selection rather than generation. If no acceptable candidate exists, ranking cannot recover one. Conversely, confidence scores already perform well on some targets with high quality candidates, leaving less room for improvement.

Our stratified analysis shows the largest gains in the Medium group (113 targets):

MethodDockQ@1DockQ@5
iPTM0.4290.484
ipSAE0.4170.497
DeepRank-Ab0.4460.550
AQA-Score0.4850.539
Oracle0.6730.673

Compared with iPTM, the improvements are +0.0560/+0.0549 in DockQ@1/5. Both remain positive under target and antigen cluster bootstrap:

MetricTarget 95% CIAntigen cluster 95% CI
DockQ@1 gain[0.0237, 0.0908][0.0053, 0.0970]
DockQ@5 gain[0.0228, 0.0902][0.0079, 0.0920]

The Medium group contributes 88.6% of the overall Top-1 gain and 70.7% of the Top-5 gain. In contrast, in the High group, iPTM already achieves DockQ@1 of 0.774, compared with 0.775 for AQA-Score.

We will retain the full 391 target evaluation as the primary result and use the stratified analysis to explain why the improvements vary across targets.

Response to Reviewer 4vsH (3/3)

Q(b). Motivation for the Additional Loss Components

Takeaway: Predicting the three DockQ components separately provides structural supervision that a single scalar target cannot capture. The selection loss addresses a different objective: choosing the best candidate within each pool, rather than minimizing average prediction error alone.

The final score follows the standard DockQ decomposition:

s=F^+I^+L^3,s=\frac{\hat F+\hat I+\hat L}{3},

where the three terms represent contact recovery, transformed interface RMSD, and transformed antibody ligand RMSD.

However, regression on the final scalar and regression on individual components are not equivalent. With scalar supervision, an underestimated contact term can be canceled by an overestimated geometry term, producing a seemingly accurate final DockQ despite inaccurate component predictions.

We therefore supervise the components separately and use local contact and residue error losses to provide more specific structural signals.

Quality prediction and candidate selection are also different objectives. A model may predict the quality of most candidates accurately but still rank a worse pose above the best available structure.

We address this using pool wise selection regret:

Lselect,c=∑ipci(Dc∗−Dci),\mathcal L_{\mathrm{select},c} =\sum_i p_{ci}(D_c^*-D_{ci}),

where pcip_{ci} is the predicted selection probability and Dc∗D_c^* is the oracle DockQ within the candidate pool.

Loss weights balance the different numerical and gradient scales of component regression, local supervision, and selection regret. They do not change the definition of DockQ.

The existing ablations support these choices:

VariantDockQ@1DockQ@5
Full AQA-Score0.36310.4023
Aggregate score supervision0.35090.3978
Without selection loss0.34750.3924

Replacing component regression with aggregate score supervision reduces DockQ@1 to 0.3509, while removing selection regret reduces it to 0.3475.

These results support the two training objectives within our current setting. We will clarify their respective roles and mathematical definitions in the manuscript.

Q(d). Generalization to Unseen Antigens

Takeaway: AQA-Score remains competitive on 163 targets with antigen sequence clusters unseen during supervised training, achieving the highest Top-1 DockQ point estimate among the evaluated methods.

Among 391 test targets, 163 (41.7%) have antigen sequence clusters absent from the audited supervised training set.

Table R2. Performance on unseen antigen sequence clusters (163 targets).

MethodDockQ@1DockQ@5[email protected]/1[email protected]/5
iPTM0.34010.373650.3%52.8%
ipSAE0.32840.372949.1%51.5%
pDockQ0.32310.371646.0%50.9%
CP-H3 max0.33370.373447.2%50.3%
ABAG-Rank0.28770.345540.5%47.2%
ARID-sf0.20870.304129.4%39.9%
DeepRank-Ab0.33870.403149.7%55.2%
AQA-Score0.35160.392150.9%53.4%

AQA-Score improves DockQ@1/5 over iPTM by 0.0115/0.0185, and over ipSAE by 0.0232/0.0192. It achieves the highest Top-1 point estimate, while DeepRank-Ab remains stronger at Top-5.

These results indicate competitive performance on antigen sequence clusters unseen during supervised training. We do not interpret the positive differences as statistically established improvements without paired confidence intervals.

We also distinguish unseen sequence clusters from broader antigen families. The present evaluation does not guarantee complete independence from homologous proteins in the pretraining corpus, and we will state this limitation explicitly.

We appreciate the reviewer's constructive feedback on presentation, evaluation coverage, and generalization. We will incorporate these clarifications and additional results into the revised manuscript. We hope that the new evidence addresses the reviewer's main concerns, and we would appreciate it if the reviewer could reconsider their assessment in light of these results.

Reviewer_XaBuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the practical problem of selecting accurate antibody-antigen (Ab-Ag) complex structures from AlphaFold3 (AF3) stochastically sampled ensembles. The authors introduce a large-scale benchmark, AF3-AQA, consisting of 2,216 pre-cutoff training/validation targets and 391 post-cutoff test targets with independently sampled pools of 100 (AF3-AQA-100) and 1,000 (AF3-AQA-1000) candidates. Using this benchmark, the authors systematically analyze native AF3 confidence metrics, Rosetta energy terms, and geometric heuristics across different antibody region masks and aggregation rules. Furthermore, they propose AQA-Score, a learned ranking model that transfers frozen pretrained single-chain site (Module A) and cross-chain pair compatibility (Module B) representations into candidate assessment. AQA-Score constructs a sparse pair graph combining formed contacts and unformed pair hypotheses, predicts the three DockQ components with local structural supervision, and optimizes a pool-wise selection regret objective.

Critical strengths

Comprehensive benchmark and systematic evaluation. The paper introduces a large-scale benchmark of AF3-generated antibody–antigen structures and provides a systematic analysis of confidence, geometric, and energy-based scoring signals. The experiments cover different aggregation strategies, candidate pool sizes, and structural quality levels, offering valuable insights into the limitations of existing structure selection methods.

Well-motivated framework and thorough empirical analysis. AQA-Score provides a reasonable approach to integrating pretrained binding representations with candidate-specific structural evidence. The paper includes comparisons with multiple baselines, meaningful input and loss ablations, difficulty-stratified evaluations, and statistical uncertainty analysis. The authors also carefully acknowledge the limitations of their findings, making the conclusions generally well-supported and appropriately scoped.

Critical weaknesses

Missing Ablations on Core Methodological Claims: The paper attributes the effectiveness of AQA-Score to transferring pretrained binding priors (Modules A & B) and modeling both formed contacts and unformed hypotheses. However, Table 5 only ablates AF3 confidence inputs, local Rosetta energy inputs, and loss terms. As the authors acknowledge in Section 5.4 and Appendix E.3, there are no ablations isolating the impact of the pretrained representations or the unformed hypothesis pool. Without these, it is unclear whether the complex pretraining and dual-pool message passing are actually driving the gains over a simpler MLP/GNN trained solely on AF3 confidence and Rosetta features.

Unfair / Incomplete Comparison with External Learned Rankers: External rankers (DeepRank-Ab, ABAG-Rank, ARID-sf) are evaluated out-of-the-box without retraining on the AF3-AQA training split. Since they were trained on different decoy distributions (e.g., docking decoys or AF2), their poor performance may be conflated with domain shift.

Questions for authors

Can you isolate the contributions of pretrained binding representations and unformed-pair hypotheses, ideally with repeated training seeds?

While the external learned baselines are evaluated on the same AF3-AQA candidates, they retain their original training histories. Could the authors clarify how differences in training data distributions and protocols may affect the comparison with AQA-Score?

4 comments

Response to Reviewer XaBu (1/2)

We thank the reviewer for the positive assessment of AF3-AQA, particularly the recognition of our benchmark design, systematic signal analysis, and the motivation behind AQA-Score. Your comments identified two important gaps in our original evaluation: the contributions of our core model components and the fairness of comparisons with external rankers.

We conducted additional experiments to address both concerns, as summarized in our General Response.

W1 & Q1. Contributions of Binding Representations and Unformed Pair Hypotheses

Takeaway: Removing Modules A/B or unformed pair hypotheses reduces ranking performance. The effect is particularly pronounced in the Medium group, where removing unformed pairs lowers DockQ@1 from 0.4850 to 0.4305.

We agree that the original Table 5 mainly examined confidence features, Rosetta energy, and training objectives, without directly evaluating the components emphasized in our method.

1. Contributions of Modules A and B.

Module A learns single chain interface representations, while Module B uses cross attention to learn residue pair compatibility. We trained additional variants removing Module A, Module B, or both.

Table R1. Module ablations on AF3-AQA-100 (391 targets).

ModelDockQ@1DockQ@5[email protected]/1[email protected]/5
Full AQA-Score0.36310.402351.9%54.7%
Without Module A0.35370.394750.6%52.9%
Without Module B0.34640.399349.4%54.0%
Without Modules A+B0.33500.389449.6%52.2%

Removing Module A, Module B, or both reduces DockQ@1 by 0.0094, 0.0167, and 0.0281, respectively. The larger reduction when removing Module B is consistent with the value of modeling compatibility between antibody and antigen residues.

These ablations evaluate the contribution of the corresponding modules within our architecture. They do not completely isolate the effect of pretraining initialization, since removing a module can also change downstream representations and pair proposals.

2. Contribution of unformed pair hypotheses.

We additionally trained a Formed only variant that retains contacts realized in the candidate structure but excludes unformed pair hypotheses from subsequent message passing and pooling.

Table R2. Formed only ablation.

ModelOverall D@1Overall D@5Medium D@1Medium D@5
Full AQA-Score0.36310.40230.48500.5390
Formed only0.34320.39000.43050.5062
Difference+0.0199+0.0123+0.0545+0.0328

Removing unformed hypotheses lowers overall DockQ@1/5 by 0.0199/0.0123. In the Medium group (113 targets), DockQ@1 falls from 0.4850 to 0.4305, close to iPTM's 0.429.

This supports our motivation for considering potential interactions proposed by Module B or AF3 contact probabilities, even when they are not realized in the current candidate pose. Such information can help assess interfaces beyond the contacts already formed.

The ablation also changes the set of pairs participating in message passing, so it measures the contribution of the full modeling pathway rather than isolating each information source independently.

3. Comparison with simple signal fusion.

We also trained a Logistic Fusion baseline combining confidence, energy, and geometry features (C+E+G). It achieves DockQ@1/5 of 0.3464/0.3993, compared with 0.3631/0.4023 for AQA-Score.

The full model achieves higher Top-1 quality, although Top-5 performance is similar. This provides a direct comparison against simple signal fusion, but does not rule out stronger MLP or GNN alternatives.

Regarding repeated training seeds, the results reported here do not establish variability across independent training runs. We therefore interpret the ablations as observed component effects rather than claiming statistical significance across training seeds.

Response to Reviewer XaBu (2/2)

W2 & Q2. Fairness of External Ranker Comparisons

Takeaway: Retraining ABAG-Rank on AF3-AQA substantially improves its performance, confirming that training data compatibility matters. AQA-Score nevertheless retains higher average DockQ in this comparison.

We agree that evaluating external rankers on the same AF3 candidates does not fully control for their different training histories. DeepRank-Ab, ABAG-Rank, and ARID-sf were originally trained using different structural data and decoy distributions. Their results with released weights should therefore be interpreted as transfer performance, rather than a strictly matched comparison of model architectures.

1. Retraining ABAG-Rank on AF3-AQA.

To examine the effect of training distribution, we retrained ABAG-Rank using the AF3-AQA supervised training data and evaluated it on the same 391 test targets.

Table R3. ABAG-Rank retraining on AF3-AQA-100.

MethodDockQ@1DockQ@5[email protected]/1[email protected]/5
ABAG-Rank (original)0.28500.354041.7%49.4%
ABAG-Rank (retrained)0.32770.384746.3%51.7%
AQA-Score0.36310.402351.9%54.7%

Retraining improves ABAG-Rank's DockQ@1 from 0.2850 to 0.3277 (+0.0427) and DockQ@5 from 0.3540 to 0.3847 (+0.0307).

This substantial improvement is consistent with the reviewer's concern that differences in training distributions can affect the evaluation of external rankers.

After retraining, ABAG-Rank still obtains lower DockQ@1/5 than AQA-Score, with differences of 0.0354/0.0176. Thus, the original performance gap cannot be explained solely by comparing AQA-Score against ABAG-Rank's released weights.

2. Training protocols of other external rankers.

DeepRank-Ab learns quality assessment from antibody–antigen interface graphs, with training data based on docking decoys. ARID-sf combines structural, physical, and protein language model features under a different training and preprocessing protocol.

We evaluated both methods using their released weights but did not retrain them on AF3-AQA. Their results remain useful for understanding how existing scoring methods transfer to AF3 ensembles, but they should not be treated as fully controlled architecture comparisons.

We will make this distinction explicit in the revised manuscript, separating evaluations using original weights from those using the AF3-AQA training data. We will also clarify the differences in training distributions and protocols.

We appreciate the reviewer's constructive suggestions. The additional experiments have helped us better understand the contributions of our core components and the impact of training data differences on external rankers. We will incorporate these results and clarifications into the revised manuscript.

Dear Reviewer XaBu,

We understand this is a very busy period, so we just wanted to send a brief note as the author-reviewer discussion window wraps up. Have our responses and updates satisfactorily resolved the points you raised? If anything remains unclear, please let us know. We would gladly address it before the deadline.

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts and consider my original concerns largely addressed. I will maintain my positive assessment of the paper and keep my current rating.

Reviewer_dKNoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper deals with a practical scenario: AlphaFold3 produces many candidate antibody–antigen complexes for the same input, and a practitioner has to pick one, but the candidates differ substantially in binding pose and interface accuracy. To this end, the authors provide AF3-AQA, a benchmark of 430,100 predicted structures over 391 temporally held-out targets in two independently sampled pools, together with an analysis of how region choice and aggregation affect confidence, energy, and geometric scores. Furthermore, the authors propose AQA-Score, which transfers frozen single-chain interface-site and residue-pair compatibility representations into candidate-specific assessment, distinguishing contacts the candidate realizes from proposed pairings it leaves unformed, and trains with a pool-wise selection objective. From experiments on this benchmark, the authors report that regional restriction helps only for some signals, that strong scalar scores still leave distinct poses tied, and that AQA-Score improves first-choice quality over AF3 confidence.

Critical strengths
  1. The paper addresses a practical scenario — AF3 returns many candidates and one has to be chosen — and builds a benchmark for it with care: the split is enforced at the cluster-pair level with a temporal cutoff, two independently sampled pools are released for the same targets, and uncertainty is reported with both target and antigen-family resampling.

  2. The analysis sweeps regions, aggregation rules, and energy definitions under a common protocol, and reports which choices make a given signal a better or worse selector — findings a practitioner can act on directly when deciding how to rank AF3 candidates.

Critical weaknesses
  1. The split is enforced at the cluster-pair level, so an antibody cluster or an antigen cluster may appear in both partitions even though no exact pair does, and the paper does not report how often this happens. This matters because AQA-Score’s pretrained modules learn interface sites and residue-pair compatibility, so prior exposure to a test target’s antigen surface or CDR pattern would plausibly transfer, and the antigen-family bootstrap addresses dependence within the test set rather than leakage across the split. The pretraining corpus is a further gap the authors acknowledge in Appendix A.1: Modules A and B are trained on 170,629 general protein-complex records whose stated cutoff was not revalidated, and no homology map against the AF3-AQA test set is available. For a paper whose benchmark contribution rests on a clean temporal split, reporting the overlap fractions and the performance on the non-overlapping subset would be the natural check.

  2. While the authors are candid about the limits of their method — the overall Top-1 gain is reported as uncertain under antigen-family resampling, and DeepRank-Ab retains a higher Top-5 point estimate — a benchmark and a signal analysis alone would be a thin contribution for this venue, so the method has to carry part of the weight, and I do not think it does. The gain is small against what the paper’s own analysis already provides: on AF3-AQA-1000, PAE H1–H3 mean alone reaches 0.396/0.437 against AQA-Score’s 0.4004/0.4518, and no baseline combines the three signal families the analysis characterizes, although a linear fit or a small gradient-boosted model over them would need no new data. The ablations do not close this gap either, since all four vary input channels or loss terms rather than the two components the paper claims as novel — the pretrained pair prior and the formed/unformed construction — which the authors note in Appendix E.3. As it stands, the reported improvement cannot be attributed to the contributions the method is built around.

Questions for authors
  1. The training resource is released with the benchmark, but only AQA-Score is trained on it. Could any of the external rankers be retrained on the same split, or if not, which ones are not retrainable?

  2. Module A is pretrained for interface-site prediction, which is what existing epitope and interface predictors do. How does that module perform on its own as a selector — for instance, scoring each candidate by how well its interface overlaps the predicted sites?

5 comments

Response to Reviewer dKNo (1/3)

We thank the reviewer for the careful assessment of AF3-AQA and for recognizing the value of our benchmark and systematic signal analysis. Your comments identified important gaps in our evaluation, particularly regarding data overlap and the contributions of our model components. We conducted additional analyses and experiments to address these concerns.

W1. Data Split and Pretraining Overlap

Takeaway: Our benchmark evaluates generalization to unseen antibody–antigen pairs, rather than requiring both proteins to be unseen. We found no repeated cluster pairs between the audited supervised training targets and the test set. We also quantified single chain overlap and pretraining homology, which were not sufficiently documented in the original submission.

1. Supervised training overlap.

Our split follows a pair level nonredundancy criterion, which has precedent in protein interaction prediction. For example, Partner-Aware Prediction of Interacting Residues in Protein-Protein Complexes from Sequence Data (PLOS ONE, 2011) considers sequence similarity on both interacting partners when controlling complex redundancy.

The biological motivation is that a shared antigen does not uniquely determine the antibody binding interface or pose. Different antibodies can recognize different epitopes or adopt different orientations against the same antigen. Thus, sequence similarity on one side does not necessarily imply exposure to the correct interaction in a test complex.

We agree, however, that individual chain overlap may still provide transferable interface knowledge. We therefore audited 868 supervised training targets against all 391 test targets.

Table R1. Sequence cluster overlap with audited supervised training targets.

Test target categoryCountFraction
Neither cluster seen15639.9%
Antibody cluster only71.8%
Antigen cluster only22557.5%
Both clusters seen, pair unseen30.8%
Cluster pair seen00.0%

Antibody clusters use 90% sequence identity over concatenated CDRs, while antigen clusters use 70% identity, both with 80% coverage.

In particular, 381/391 (97.4%) test targets contain unseen antibody sequence clusters, and 156/391 (39.9%) have neither cluster seen during the audited supervised training.

We also evaluated 163 targets with unseen antigen sequence clusters. AQA-Score achieves DockQ@1/5 of 0.3516/0.3921, compared with 0.3401/0.3736 for iPTM. For the subset where both clusters are unseen, the available results do not establish a clear advantage over iPTM.

The audit covers 868 actual supervised training targets. We will clarify how these relate to the larger training resource described in the manuscript.

2. Pretraining corpus.

The original 170,629 records refer to the source training corpus. After filtering, Modules A/B were pretrained on 46,496 complexes and 68,126 chain pairs.

We additionally searched the pretraining corpus for sequences homologous to the test antigens. At ≥90% sequence identity and ≥80% query and subject coverage, 246/391 (62.9%) test antigens have homologous pretraining chains.

Antigen homology alone does not establish that the model encountered the corresponding antibody, interaction pair, or correct binding pose. Modules A/B learn general interface site and residue pair representations rather than test candidate DockQ labels. Nevertheless, homologous antigens may provide transferable interface information, and our current audit cannot exclude all paired complex overlap.

Temporal provenance also remains incomplete: 46,415 of 46,496 filtered records lack reliably matched dates, while available dates are primarily deposition rather than release dates. We therefore cannot independently verify the stated temporal cutoff for the entire pretraining corpus.

We will report these findings explicitly and distinguish supervised pair independence from possible information transfer through pretraining.

Response to Reviewer dKNo (2/3)

W2. Performance Gains and Methodological Contributions

Takeaway: We agree that the overall improvement alone does not establish the value of AQA-Score's architecture. Our additional experiments show that simple signal fusion does not fully recover its Top-1 performance, while removing Modules A/B or unformed pair hypotheses reduces selection quality, particularly in the Medium group.

1. Understanding the modest overall improvement.

The reviewer correctly notes that on AF3-AQA-1000, PAE H1–H3 mean achieves DockQ@1/5 of 0.396/0.437, close to AQA-Score's 0.4004/0.4518. We agree that this difference alone does not justify a more complex ranking model.

Our oracle quality analysis provides additional context. AF3-AQA evaluates candidate selection rather than generation. Ranking cannot recover an acceptable structure when none exists in the pool, while existing confidence scores already perform well for some targets.

The largest gains occur in the Medium oracle quality group (113 targets), where AQA-Score improves DockQ@1/5 from 0.429/0.484 to 0.485/0.539 over iPTM. These targets represent only 28.9% of the test set but contribute 88.6% of the overall Top-1 improvement.

The improvements of 0.0560/0.0549 have positive antigen cluster bootstrap 95% CIs of [0.0053, 0.0970] and [0.0079, 0.0920].

We retain evaluation on all 391 targets as our primary result. The Medium analysis explains where selection improvements occur, rather than replacing the full benchmark.

2. Simple signal fusion.

Following the reviewer's suggestion, we trained a Logistic Fusion baseline combining confidence, energy, and geometry (C+E+G) features.

It achieves DockQ@1/5 of 0.3464/0.3993 on AF3-AQA-100, compared with 0.3631/0.4023 for AQA-Score.

The Top-1 difference is +0.0167, while Top-5 performance is similar. This provides a direct comparison against simple supervised signal fusion, although it does not rule out stronger nonlinear fusion models.

3. Ablations of the proposed components.

We agree that the original Table 5 only examined input features and training objectives, without directly testing the components emphasized in the method.

We therefore trained additional variants removing Module A, Module B, both modules, and unformed pair hypotheses.

Table R2. Additional component ablations on AF3-AQA-100.

VariantDockQ@1DockQ@5
Full AQA-Score0.36310.4023
Without Module A0.35370.3947
Without Module B0.34640.3993
Without Modules A+B0.33500.3894
Formed only0.34320.3900

Removing Module A, Module B, or both reduces DockQ@1 by 0.0094, 0.0167, and 0.0281, respectively.

For the second methodological claim, we trained a Formed only variant that excludes unformed hypotheses from message passing and pooling. Its overall DockQ@1/5 decreases to 0.3432/0.3900.

The difference is particularly clear in the Medium group: removing unformed hypotheses reduces DockQ@1 from 0.4850 to 0.4305, close to iPTM's 0.429.

These results support the use of binding representations and proposed interactions beyond the contacts realized in the current candidate.

We note that module removal can also affect downstream representations and pair proposals, while Formed only changes the graph. Thus, these experiments measure the contribution of the corresponding modeling pathways rather than completely isolating pretraining initialization or every aspect of graph construction.

We will update the manuscript to include these direct ablations and clarify the scope of the conclusions.

Response to Reviewer dKNo (3/3)

Q1. Can External Rankers Be Retrained on the Same Split?

Takeaway: We successfully retrained ABAG-Rank using AF3-AQA supervised data. Its performance improves substantially, confirming the importance of training distribution, but remains below AQA-Score in the reported DockQ metrics.

The original submission evaluated DeepRank-Ab, ABAG-Rank, and ARID-sf using their released weights. These results primarily measure transfer to AF3-generated candidates rather than a comparison under identical training conditions.

We have now retrained ABAG-Rank on the AF3-AQA supervised training data and evaluated it on the same 391 test targets.

Table R3. ABAG-Rank retraining on AF3-AQA-100.

MethodDockQ@1DockQ@5[email protected]/1[email protected]/5
ABAG-Rank (original)0.28500.354041.7%49.4%
ABAG-Rank (retrained)0.32770.384746.3%51.7%
AQA-Score0.36310.402351.9%54.7%

Retraining improves ABAG-Rank by 0.0427/0.0307 in DockQ@1/5, showing that the original training distribution matters. AQA-Score remains higher by 0.0354/0.0176 in this comparison.

DeepRank-Ab and ARID-sf were not retrained in this round. Their published weights remain useful for evaluating transfer performance, but we agree that they should not be treated as strictly matched training comparisons.

We will distinguish original and retrained models in the revised comparison and make their different training histories explicit.

Q2. How Does Module A Perform as a Standalone Selector?

Takeaway: Interface site prediction alone is insufficient for reliable AF3 candidate ranking. Module A standalone achieves DockQ@1/5 of 0.1764/0.2631, substantially below the full AQA-Score model.

Following the reviewer's suggestion, we constructed a selector using Module A's predicted interface sites to score their agreement with the interfaces formed by AF3 candidates.

Table R4. Module A standalone ranking on AF3-AQA-100.

MethodDockQ@1DockQ@5[email protected]/1[email protected]/5
Module A standalone0.17640.263124.6%35.8%
iPTM0.34490.379849.6%52.9%
AQA-Score0.36310.402351.9%54.7%

Module A alone performs substantially worse than both iPTM and the full model.

This is consistent with the distinction between predicting which residues may participate in an interface and assessing the quality of a specific binding pose. The latter requires information about the relative placement of the chains, residue pair compatibility, formed contacts, and the candidate's geometry, confidence, and energy.

Module A therefore serves as a source of binding representations within AQA-Score, rather than a complete structure quality estimator.

We thank the reviewer again for the constructive feedback. We hope that the additional experiments and clarifications address the main concerns raised in the review, and we would appreciate it if the reviewer could reconsider their assessment in light of this new evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.