acceptodds
Under review as a conference paper at ICLR 2027

How Much Do Antibody Benchmarks Depend on Related Training Examples? A Cross-Benchmark Audit

Abstract

Machine-learning antibody benchmarks can retain training examples related to antibodies in the test set, obscuring how much reported performance depends on that support. We audit four predictive benchmarks while keeping each test set fixed. Ordinary training O uses the original training pool, family-filtered training F removes examples related to fixed test antibodies under a benchmark-specific rule, and size-matched blind control S removes the same quantity without using family relationships, labels, or outcomes. In AVIDa, O–F is positive in all 160 robustness analyses, and median O–S is approximately zero despite no exact mature-VHH recurrence. In AbDesign, scaffold-averaged O–F is 0.142 with a 95% bootstrap interval of [0.038, 0.259]. AlphaSeq shows large O–F values for AbLang2 and ESM-2, while MutAb shows smaller and more variable differences. The median auxiliary distribution-matched control remains closer to O than F in all four predictive benchmarks. In AbDesign, the same direction persists under IMGT CDRH3/CDRL3 and global-alignment relations fixed before evaluation that alter realized family memberships. Between ESM-2 and AbLang2, the preferred model remains unchanged across O, F, and S in three benchmarks and changes only in AlphaSeq. Antibody benchmarks therefore differ substantially in their dependence on related training support.

Primary area:
datasets and benchmarks
Keywords:
antibody machine learning, antibody benchmarks, benchmark evaluation, generalization, train-test similarity, sequence similarity, family-aware splitting

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_ak5NOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper explores the train-test split of several public antibody datasets by varying the removal of training data that are seemingly related to the test sets (F). The main conclusion is that if data points that are related to test sets are removed, the model performance drops across different antibody datasets.

Critical strengths
  1. Auditing several "benchmark" datasets in antibody worlds, thus pinpointing that current benchmark datasets may have overestimated the antibody model's performance
Critical weaknesses

Literally, I learned nothing except that the training sets of common public datasets contain data points related to the test sets. It reads like a workshop paper that needs more improvement in investigation and research questions.

Questions for authors

None

1 comment

Thank you for the feedback. We agree that removing closely related training examples can be expected to lower benchmark scores. The revised investigation asks how much of that drop persists after controlling for alternative explanations, and whether it changes which antibody candidates a predictor prioritizes.

Does the gap survive matching training-set composition?

We hold the test examples and labels fixed. Family filtering removes training examples from sequence families represented in the test set. On AVIDa, we compare it with a different deletion that matches the number of training observations, distinct antibodies, and positive and negative observations in each of 20 training assay contexts. Both conditions keep or remove all observations of an antibody together.

With these quantities held fixed, median matched-control gains over family filtering are +0.1944 AUROC for AbLang2 Ridge, +0.1264 for ESM-2 Ridge, and +0.0608 for the neural pair classifier. All three gains are positive on all ten splits and for all 50 deterministically selected controls. The matched controls keep a direct training neighbor at 90% identity or higher for 57.9–62.1% of test rows, compared with zero under filtering. A separate comparison that also preserves the full distribution of retained antibody counts per family gives median gains of +0.0307 and +0.0128 AUROC for the two Ridge models.

These comparisons show that the number of antibodies and their measured assay composition are insufficient to explain the gap. Which examples remain available matters under these controls. A published binding predictor, DeepInterAware, shows the same direction on two splits from the same AVIDa collection. See Sections 4.1–4.2 and 4.4.

Does the gap survive when the test scaffold remains represented?

In the original AbDesign comparison, family filtering withheld the entire test scaffold. In the new comparison, both training conditions retain all 14 evaluated scaffolds. On 234 fixed test variants from seven antigens, the compared deletions exchange matched sequence groups within scaffolds while matching training counts, chain lengths, and recorded assay and context counts.

The matched control improves Spearman rank correlation by +0.1032 for AbLang2 and +0.1336 for ESM-2. The average gap remains positive for all five evaluated predictors after omitting any one antigen. This shows that performance can depend on the local training examples available for a scaffold that the model already sees in training. The exchanged groups also correspond to mutation sites, so the comparison measures local sequence and site-specific support together. See Section 5, Table 5, and Figure 4.

Does the difference affect which candidates are prioritized?

We refit the same neural classifier architecture under each training condition and rank the same held-out observations. At an identical top-5% screening budget within each assay context, matched-control training retrieves a median of 97.5 more measured binder observations per split than family-filtered training. The median paired precision gain is 19.1 percentage points. The retrieval gain holds on all ten splits and for all 50 controls. This is a retrospective comparison on one AVIDa collection. See Section 4.3 and Figure 3.

Building on similarity-aware evaluations such as BASE and SPECTRA, the revision connects the benchmark score to the training support available in the intended use case. We recommend stating which experimental groups the task assumes are represented, checking which related examples the filter actually removes, and reporting the fixed-budget retrieval difference alongside AUROC when the model is used to prioritize candidates. Figure 1 now shows both training interventions before the results.

Reviewer_v22zOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits whether antibody benchmark scores depend on supervised training examples related to a fixed test set. It compares ordinary training (O), family-filtered training (F), and size-matched blind removal (S) across AVIDa, AbDesign, AlphaSeq, and MutAb-15,, with Ginkgo as a structural boundary case. Large O–F differences appear in AVIDa and AbDesign while S remains near O. AlphaSeq shows a large retrospective difference across three lineages; MutAb has smaller, heterogeneous, statistically unresolved effects. The reported preferred model among AbLang2 and ESM-2 changes only for AlphaSeq under F.

Critical strengths
  1. Holding each test set fixed gives the O/F/S comparison a clear interpretation so it is clear that any score change comes from the training set. The blind control removes the same amount of data, addressing the most immediate training-size explanation.

  2. The additional distribution-matched control explains antigen context and label prevalence, prior exposure, replicate availability, or mutated chain as applicable. Its scores keep closer to O than F in the settings tested.

  3. The analysis includes VHH binding, mutation ranking, interaction ranking, and affinity-change ranking. The different metrics are kept separate, and Ginkgo is correctly treated as a structural boundary instead of a predictive null result.

Critical weaknesses
  1. In AbDesign, AlphaSeq, and MutAb, primary family components closely follow scaffold, source lineage, or parent system. Removing a test-related component therefore also removes training from the corresponding experimental system; the result is not an isolated effect of sequence proximity.

  2. The matched controls cover selected features of the training distribution, but other differences remain. This matters most for AVIDa, where F removes 47.1% of audited training membership; altered diversity or family composition beyond matched context and prevalence could contribute to the drop.

  3. The distribution-matched control uses one learner and one setting per benchmark. Its result cannot stand in for a check across the full AVIDa grid or the model-selection comparison.

  4. AbDesign has 14 scaffold units; AlphaSeq is retrospective with only three lineages; MutAb comparisons are nonsignificant across 12 parents. A broad statement about antibody benchmarks rests most heavily on AVIDa and the narrower AbDesign result.

  5. The presentation could be significantly improved, including the tables, figures and the content layout.

Questions for authors
  1. Is there a setting within a scaffold or lineage where both related and less-related training examples are available? That comparison could help separate sequence similarity from removing the whole group.

  2. How do the results change as the family threshold or nearest retained training similarity varies, especially for AVIDa and AbDesign?

  3. Could you show the unit-level S–F differences for the distribution-matched controls? The aggregate median hides substantial variation in MutAb.

1 comment

Thank you for the questions. We keep the test examples and labels fixed. Family filtering removes training examples from sequence families represented in the test set. A matched control uses a different deletion under the stated constraints. See Tables 1–2 and Figure 1.

Within-scaffold comparison

The original filters remove all training support from the test scaffold, lineage, or parent system in 28/28 AbDesign, 3/3 AlphaSeq, and 12/12 MutAb-15 panels, respectively. The new AbDesign comparison keeps all 14 scaffolds represented. Across two complementary fixed panels with 234 variants from seven antigens, disjoint deletions exchange related and matched components within scaffolds. Counts, chain lengths, recorded assay and context counts, and component structure are matched. All 234 test variants retain a direct neighbor at ≥99%\geq99\% identity under the control versus zero under filtering.

Matched versus filtered Spearman rank correlation is 0.1600 versus 0.0568 for AbLang2 (+0.1032) and 0.2071 versus 0.0735 for ESM-2 (+0.1336). Both favor the control in five of seven antigen blocks. All five evaluable predictors retain a positive aggregate gap after omitting any antigen. The exchanged components also correspond to mutation sites. See Section 5 and Figure 4.

AVIDa controls and threshold sensitivity

Across ten fixed splits, joint controls match observations, distinct single-domain antibodies (VHHs), and positive and negative counts in each of 20 assay contexts. The median matched-minus-filtered AUROC gains are +0.1944, +0.1264, and +0.0608 for AbLang2 Ridge, ESM-2 Ridge, and the neural classifier. All three favor the control on ten splits and all 50 selected controls. The joint controls minimize overlap with the filtered deletion. Direct-neighbor coverage at ≥90%\geq90\% identity is 57.9–62.1% under matching and 0% under filtering. A swap preserving the retained family-size distribution exchanges 7.04–9.22% of filtered-out rows. Its median AUROC gains are +0.0307 and +0.0128, positive on 10/10 and 9/10 splits. The gaps describe different deletions. See Sections 4.1–4.2.

We refit the same AVIDa training sets at higher optimization accuracy. OO uses the original pool, FF applies family filtering, and SS matches the number of training examples in FF without using family information.

IdentityMedian O−FO-F AUROCPositive
80%+0.1045438/40
85%+0.1068840/40
90%+0.1115040/40
95%+0.0795438/40

156/160 supersedes the original 160/160. Separately, all 40/40 primary blind-control minus filtered gaps stay positive.

In AbDesign, the original paired-chain identity filter gives identical deletions at 85%, 90%, and 95% in all 28 evaluations. At 80% it changes four deletions. Two alternative definitions change the memberships and yield positive O−FO-F and S−FS-F in every evaluable unit. Global alignment at 80% gives median gaps +0.063 and +0.062 (34/34 units). CDR3 identity at 90% gives +0.071 and +0.070 (38/38 units). CDR3 filtering at 95% removes zero rows. See Appendix G, Table S10.

Requested unit-level effects

SstratS_{\mathrm{strat}} is the median score of controls that also match benchmark-specific training distributions. Scores are AUROC for AVIDa split 37037 and Spearman elsewhere. AbDesign uses 3-mer and the other datasets use AbLang2 Ridge. Eight of 12 MutAb-15 parent Sstrat−FS_{\mathrm{strat}}-F gaps are positive. The preregistered parent-level tests remain nonsignificant after Holm correction. Appendix G, Table S9 reports benchmark aggregates. All 35 requested unit-level contrasts are below.

BenchmarkUnitS−FS-FSstrat−FS_{\mathrm{strat}}-F
AVIDaIL-6_C72A+0.2647+0.2243
AVIDaIL-6_D99A+0.3488+0.3166
AVIDaIL-6_E108A+0.3858+0.3459
AVIDaIL-6_L126A+0.3545+0.3076
AVIDaIL-6_P93A+0.3832+0.3420
AVIDaIL-6_T48A+0.2695+0.2289
AbDesign1BJ1+0.3729+0.3281
AbDesign1BVK−0.0741−0.0745
AbDesign2YSS−0.1253−0.1156
AbDesign5F9W+0.0858+0.0642
AbDesign5GGS+0.4148+0.4234
AbDesign5GGV+0.1467+0.1181
AbDesign6MFP−0.0080−0.0149
AbDesign6NMS+0.6585+0.6290
AbDesign6NMU+0.0868+0.0403
AbDesign6RP8+0.0857+0.0845
AbDesign6XKR−0.0534−0.0535
AbDesign7BEJ+0.0616+0.0798
AbDesign7JMO+0.2656+0.2618
AbDesign7KF0+0.0587+0.0494
AlphaSeqAAYL49/50 shared seed+0.3854+0.3852
AlphaSeqAAYL51_SEED+0.4274+0.4311
AlphaSeqAAYL52_SEED+0.2326+0.2317
MutAb-15068eff2b+0.5512+0.5512
MutAb-15164d0ac7−0.4667−0.4121
MutAb-1521cf36c6−0.4571−0.4286
MutAb-1528d9310f−0.0952−0.1190
MutAb-153631dac9−0.0304+0.0607
MutAb-153b9e103c−0.6000−0.7000
MutAb-1545143632+0.1429+0.1978
MutAb-1545c15e03+1.2000+1.1500
MutAb-155a11e128+0.1203+0.0556
MutAb-1580037866+0.2181+0.1966
MutAb-1591f1b1d1+0.2347+0.3292
MutAb-15e306bc8c+0.3520+0.2420
Reviewer_RLiHOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper audits the dependence of antibody benchmarks on related training examples. The authors compare ordinary training (O), family-filtered training (F), and size-matched random removal (S) across four predictive benchmarks, with additional distribution-matched controls. Substantial performance drops are observed after family filtering in AVIDa, AbDesign, and AlphaSeq, while random removal has much smaller effects. However, model selection between AbLang2 and ESM-2 remains unchanged in three of the four benchmarks.

Critical strengths
  1. The paper addresses a relevant issue in biological ML evaluation, especially in antibody prediction, where related sequences across training and test sets can obscure generalization.
  2. The evaluation spans several antibody datasets, with size-matched and distribution-matched controls to investigate alternative explanations.
  3. The authors distinguish absolute performance changes from model-selection sensitivity and acknowledge several limitations of their analysis.
Critical weaknesses
  1. The core methodology closely resembles existing similarity-aware evaluations, particularly BASE and SPECTRA. The primary contribution is applying these ideas to additional antibody benchmarks, but the central findingthat performance decreases when related training examples are removed is largely expected. The paper offers limited new methodological or biological insight beyond quantifying this effect across datasets.

  2. The family definitions closely track experimental groupings: AbDesign has 14 components across 14 scaffolds, while AlphaSeq and MutAb similarly align with source lineages and parent systems. Consequently, family filtering often resembles removing support from entire experimental groups. The observed performance drops may therefore reflect the difficulty of cross-family generalization rather than a problematic dependency in the original benchmarks. Both within-family and cross-family prediction are legitimate settings, and the paper does not clearly distinguish their implications.

  3. The common model comparison relies on frozen AbLang2 and ESM-2 representations with relatively simple downstream learners, limiting the relevance to modern antibody predictors. Some benchmarks also have weak original predictive performance (e.g., AbDesign Spearman ρ = 0.150), while MutAb shows no statistically significant differences. More importantly, model selection remains unchanged in three of four benchmarks, leaving the practical implications of the findings unclear.

  4. In AVIDa, family filtering removes observations associated with related VHHs, whereas the random control matches the number of removed rows rather than necessarily matching unique antibody sequences or family diversity. This may confound relatedness with differences in training-set composition. The distribution-matched analyses are useful but cover only selected configurations and do not fully resolve this concern.

  5. The manuscript is unnecessarily difficult to follow given the relatively simple experimental framework. Important information about family structure and experimental interventions is buried in the appendix, while the main text spends a considerable amount of space on procedural details using opaque terminology (e.g., "frozen evaluation cells" and "evaluable model-seed units"). The paper would benefit from a clearer focus on the biological meaning of its interventions and results.

Questions for authors
  1. How can the effect of sequence relatedness be separated from removing entire scaffolds, source lineages, or parent antibody systems?

  2. For AVIDa, would the conclusions hold under random-removal controls matched by unique VHHs or family diversity rather than observation count?

  3. Beyond BASE and SPECTRA, what new conclusions or actionable recommendations for antibody benchmark design does this study establish?

  4. Would the observed effects persist with high-performing antibody-antigen structure prediction models such as Protenix-V2 and OpenDDE?

1 comment

Thank you for the detailed review. We keep the test examples and labels fixed. Family filtering removes training examples from sequence families represented in the test set. A matched control uses a different deletion under the stated constraints.

Does the AVIDa gap survive matching VHHs and family diversity?

Yes. We sample 20 controls per split, removing all observations of a single-domain antibody (VHH) together while matching observation and antibody counts. All six predictors favor the control on all ten splits. The median gains are +0.2387 and +0.1535 AUROC for AbLang2 and ESM-2 Ridge (Appendix B.3).

The joint controls also match positive and negative counts within each of 20 training assay contexts. We choose five deterministic deletions per split to minimize overlap with filtering, without test outcomes or model scores. They share 5.55–10.97% of the filtered deletion and retain direct neighbors at ≥90%\geq90\% identity for 57.9–62.1% of test rows, versus 0% after filtering.

PredictorMedian AUROC gain over filteringPositive splitsPositive controls
AbLang2 Ridge+0.194410/1050/50
ESM-2 Ridge+0.126410/1050/50
Neural pair classifier+0.060810/1050/50

These gaps describe the selected deletions. Estimating an average would require a sampling design. A separate bundle swap preserves the full distribution of retained VHH counts per family. Median gaps are +0.0307 and +0.0128 AUROC for AbLang2 and ESM-2, positive on 10/10 and 9/10 splits. See Sections 4.1–4.2 and Appendix B.

Can we vary support within a represented experimental group?

The original filters remove all training examples from the test scaffold, lineage, or parent system in 28/28 AbDesign, 3/3 AlphaSeq, and 12/12 MutAb-15 panels, respectively. The new AbDesign experiment keeps all 14/14 scaffolds represented. Two complementary fixed panels contain 234 variants across seven antigens. The two deletions remove different sequence groups within scaffolds and match training counts, chain lengths, assay and context counts, and group structure. Every test variant retains a direct neighbor at ≥99%\geq99\% identity under the control, compared with zero after related removal.

Matched versus filtered Spearman rank correlation is 0.1600 versus 0.0568 for AbLang2 (+0.1032) and 0.2071 versus 0.0735 for ESM-2 (+0.1336). All five evaluable predictors retain positive aggregate gaps after omitting any one antigen. The original 0.150 baseline used 3-mer features and withheld the scaffold. The new contrast varies local sequence and mutation-site support. See Section 5, Table 5, and Appendix G.

Does the effect extend to stronger predictors?

We refit DeepInterAware for 30 epochs on two splits from the same AVIDa collection, with one matched control per split. Each split uses identical eligible test rows across conditions. Matched and filtered pools agree on rows, VHHs, and all 40 assay-by-label counts.

SplitFull training AUROCFilteredMatchedMatched minus filtered
370380.97680.69490.8800+0.1851
370390.97060.56330.9199+0.3566

All six contexts favor the control on both splits. Protenix-v2 weights are proprietary. OpenDDE provides structure-prediction checkpoints, but input validation alone does not establish a binding result. Running a frozen structure predictor on the same test pairs would not measure sensitivity to supervised training membership. A comparable test requires refitting a validated binding predictor on each training pool, as we did with DeepInterAware on two AVIDa splits. Extension to Protenix-v2 or OpenDDE-based binding models remains untested. See Section 4.4 and Appendix C.

What is new and practically relevant?

BASE compares similarity-based and size-matched random removal on fixed tests. SPECTRA varies train–test overlap. We measure the gap under tighter matching and verify whether a deletion removes local support or an entire group. We recommend reporting the intended prediction setting, deletion overlap, retained neighbors, model rankings, and fixed-budget retrieval.

The neural classifier reaches 0.9806 median AUROC under full training. At the same top-5% budget in each test assay, matched-control training retrieves a median of 97.5 more measured binder observations per split (range +46 to +149), a median paired precision gain of 19.1 percentage points. All ten splits and all 50 controls favor matched training. The splits reuse one source collection. The retrospective retrieval result matters even when model preference stays fixed, as it did in three of four original comparisons. See Sections 2, 4.3, and 7 and Figure 3.

Section 3 defines sequence families before the results. Tables 1–2 explain training conditions and evaluation units. Figure 1 shows which neighbors are removed and which scaffolds stay represented.

Reviewer_8adxOpenReview ↗
Rating
3(−1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Antibody machine-learning benchmarks are often used to compare models performance . The authors point out that having no exact duplicates between training and test sets does not mean the two set are independent. Antibodies are connected to each other through sequence similarity, shared parent backgrounds and so on. The authors design a single fixed-test audit and apply it on five different antibody benchmarks. The test set stays completely fixed and only the training set changes, giving three conditions: O, the original training pool; F, it removes training examples having relationships with the test examples; and S, it removes the same number of examples but without using family relationships, labels or outcomes. The authors compute the relevant differences, which separate sensitivity to the choice of removed examples from sensitivity to the loss of training data.The authors also add a distribution-matched control to check the effect of distribution shift, switch to two alternative relatedness definitions on AbDesign to check model-selection sensitivity on its own. The conclusion is that antibody benchmarks differ a lot in how much they depend on related training support.

Critical strengths
  1. Setting out a clear experimental design for the question. It examines what happens when related training examples are removed, and reports the numbers in a way that lets readers compare the conditions directly.
  2. The component structure of each dataset is disclosed clearly. For Ginkgo GDPa1, there is no training examples are removed, the authors state explicitly that this is a structural boundary.
Critical weaknesses

1.As the authors noted, in AbDesign, AlphaSeq and MutAb-15 the primary component structure is closely tied to scaffold, source-lineage, and parent-system organisation. This has a substantial effect on training examples family filtering selects for removal, and may remove more data than is genuinely related. This treatment could also affect model performance in its own right. The observation that removing training data from the same group as the test set lowers scores is a fairly familiar point. 2.The evidence across benchmarks does not seem to provide fully strong support. The one preregistered confirmatory benchmark shows no significant effect.

Questions for authors
  1. The authors use single-linkage to organise the relationships between training examples. Does this over-connect examples and produce overly aggregated components? For tens of thousands of entities, it seems unlikely that every pair reaches 0.90 identity. In Table 5, 71,830 entities form only 3 components; removing one of these removes a substantial fraction of the data, so does this affect benchmark performance more than intended?
  2. Could the pretraining data behind the benchmark models also affect performance, how could its influence be separated from the effect being measured?
1 comment

Thank you for raising the AlphaSeq single-linkage, pretraining, and strength-of-evidence questions. We keep the test examples and labels fixed. Family filtering removes training examples from sequence families represented in the test set. A matched control uses a different deletion under the stated constraints.

Why does AlphaSeq have only three large components?

Under the paired identity rule, the 71,830 entities fall into three ordered heavy/light chain-length blocks corresponding to the three source lineages. Different ordered lengths prevent edges across blocks. Within each block, the Hamming distances to a verified reference aa satisfy dH(x,y)≤dH(x,a)+dH(a,y)d_H(x,y)\leq d_H(x,a)+d_H(a,y). At the primary threshold this upper bound falls within the allowed mismatch count for each chain, certifying that each lineage block is a complete graph at 90% identity.

We also checked the actual training deletions directly. An example qualifies for direct-only removal if it has a test neighbor at ≥90%\geq90\% identity in both corresponding chains. Every removed instance qualifies.

Fixed test lineageComponent-filter removalsDirect-only removals
AAYL5220,55220,552
AAYL49/50 shared lineage26,10426,104
AAYL5110,81010,810
Total57,46657,466

The removed identifiers agree exactly in all three panels, with zero chain-only removals. The same equality holds at 80%, 85%, 90%, and 95%. Direct-only removal therefore reproduces the original Spearman drops of approximately 0.352 for AbLang2 and 0.397 for ESM-2. The deletion also equals withholding the corresponding lineage. The scores describe retrospective dependence on same-lineage training support. See Section 6 and Appendix E, Table S6.

How does pretraining enter the comparison?

Within a pretrained-model comparison, the encoder is frozen and identical across training conditions. The intervention changes supervised benchmark examples. This measures dependence on supervised support conditional on the given encoder. To quantify the contribution of pretraining exposure itself, one could hold the downstream intervention fixed and compare otherwise comparable encoders trained on corpora with and without verified exposure to the test families. That exposure is unresolved for the specific released encoders.

The effect also appears with predictors that use no pretrained representations. AVIDa controls remove all observations of a single-domain antibody (VHH) together and match family filtering on both observation and antibody counts. Median AUROC gains over family filtering for composition and models based on 2-, 3-, and 4-residue sequence fragments are +0.0629, +0.0893, +0.1115, and +0.1096, respectively. All ten splits favor the control for each model, and all 800 individual controls outperform their corresponding filtered fits. On the separate scaffold-preserving AbDesign cohort, the composition, 2-mer, and 3-mer gaps are +0.0246, +0.0307, and +0.0453 Spearman. See Section 7 and Appendix B.3, Table S2.

How much of the evidence survives a comparison within a represented group?

The original family filters remove all training examples from the test scaffold in 28/28 AbDesign half-panels, from the test lineage in 3/3 AlphaSeq panels, and from the test parent system in 12/12 MutAb panels. We therefore added a separate AbDesign experiment that retains all 14 evaluated scaffolds under both training conditions. On 234 fixed variants across seven antigens, the matched and filtered Spearman rank correlations are 0.1600 versus 0.0568 for AbLang2 and 0.2071 versus 0.0735 for ESM-2. The gaps are +0.1032 and +0.1336 Spearman, respectively. Both aggregates stay positive after any one antigen is omitted. The comparison changes local component and mutation-site support within represented scaffolds. The preregistered MutAb-15 parent-level comparisons remain nonsignificant after Holm correction. The AVIDa and AbDesign follow-ups add exploratory evidence. See Section 5, Table 5, Figure 4, and Appendix G.

Reviewer_YNqBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how antibody benchmark performance depends on related training examples. With the test set fixed, the authors compare ordinary training, family-filtered training, and size-matched random removal across four predictive benchmarks. Additional analyses examine distribution-matched controls, alternative family definitions, and model ordering.

The fixed-test design is useful for distinguishing sensitivity to training-set size from sensitivity to the examples removed. The authors also make appropriate distinctions between statistical evidence across benchmarks. In particular, the AVIDa configurations are treated as dependent evaluations, AlphaSeq is identified as retrospective, and Ginkgo is reported only as a structural boundary case. The observation that substantial score changes need not alter model ordering is also worth reporting.

Critical strengths

1.The authors distinguish the strength of evidence across benchmarks and use scaffold-level statistical units where appropriate.

  1. The separate analysis of absolute performance and model ordering provides a useful perspective on benchmark sensitivity.
Critical weaknesses

1.In several benchmarks, family components closely follow scaffolds, lineages, or parent systems. Filtering therefore removes support from entire experimental groups, making it difficult to separate sequence relatedness from other group-specific effects.

Questions for authors
  1. Can related and less-related training examples be compared within the same scaffold or lineage?

  2. Do the AVIDa results persist when random removal is performed at the VHH level rather than the observation level?

1 comment

Thank you for asking for a comparison within a represented scaffold and for an AVIDa control that removes whole antibodies. We keep the test examples and labels fixed. Family filtering removes training examples from sequence families represented in the test set. A matched control uses a different deletion under the stated constraints.

Can we vary related support within a scaffold?

Yes. In all 28 original AbDesign half-panels, family filtering removed all training examples from the test scaffold. In the new comparison, both training conditions retain examples from all 14 evaluated scaffolds. We reserve two complementary test panels totaling 234 variants from seven antigens outside the common training pool. We form 19 matched pairs of sequence groups within scaffolds. For one panel, we delete the remaining training examples from its related components or from their matched partners. For the other panel, we reverse the roles. The two deletions are disjoint and match training counts, scaffold coverage, chain lengths, recorded assay and context counts, and component structure. All 234 test variants retain a direct training neighbor at ≥99%\geq99\% identity under the control, and the related deletion removes that support.

PredictorFull training SpearmanMatched controlFamily filteredMatched minus filtered
AbLang2 Ridge0.16420.16000.0568+0.1032
ESM-2 Ridge0.22200.20710.0735+0.1336

We average the two directions within each scaffold and weight all 14 scaffolds equally. Both predictors favor the control in five of seven antigen blocks. Their aggregate gaps remain positive after every single-antigen omission, with ranges of +0.0397 to +0.1343 and +0.0948 to +0.1728, respectively. The paired components in this design correspond to different mutation sites, so the contrast also changes site-specific support. The new AbDesign experiment supplies the within-scaffold comparison. The corresponding original AlphaSeq and MutAb-15 filters coincide with lineage and parent-system withholding in 3/3 and 12/12 panels, respectively. See Section 5, Table 5, and Appendices D and G.

Does the AVIDa result persist when removal operates on whole VHHs?

Yes. We remove all observations of a single-domain antibody (VHH) together and match both the number of observations and distinct antibodies. The count-matched whole-VHH comparison uses 20 controls per split. We set quotas by observations per VHH to meet both exact budgets, then use seeded hash ordering to select VHHs within each quota. Selection uses neither family assignments nor labels. All six predictors favor this control on all ten primary splits.

PredictorMedian paired whole-VHH control − filtered AUROC
Amino-acid composition+0.0629
2-mer+0.0893
3-mer+0.1115
4-mer+0.1096
AbLang2 Ridge+0.2387
ESM-2 Ridge+0.1535

A separate joint control also exactly matches the positive and negative counts within every training assay context. We construct these controls deterministically to minimize deletion overlap subject to the matching quotas. Under joint matching, the controls retain direct training neighbors at ≥90%\geq90\% identity for 57.9–62.1% of test rows, compared with 0% after family filtering. On the same fixed tests, the median paired advantages are +0.1944, +0.1264, and +0.0608 AUROC for AbLang2 Ridge, ESM-2 Ridge, and the neural pair classifier, respectively. Each predictor favors the control on all ten splits and in all 50 deterministic control realizations. The ten splits measure consistency within the AVIDa source collection. See Section 4.1, Table 3, and Appendix B.3, Table S2.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Brief follow-up on the revised manuscript and rebuttal

We thank the reviewers for their detailed evaluations. As the discussion period approaches its end, we would appreciate any further feedback on whether our revised experiments address the principal concerns raised in the initial reviews.

The revision adds three particularly important results:

  1. Matched AVIDa controls: We now match observations, distinct VHHs, and assay-by-label counts. The matched-control advantage persists across ten splits for all three principal predictors.
  2. Within-scaffold AbDesign evaluation: Both training conditions retain all 14 evaluated scaffolds, with positive aggregate performance differences on 234 held-out variants.
  3. Practical and architectural relevance: A neural classifier retrospectively retrieves a median of 97.5 additional measured binder observations at the same top-5% screening budget. DeepInterAware also shows positive matched-control advantages on two AVIDa splits.

We have addressed additional reviewer-specific questions in our individual responses.

We hope these additional experiments and clarifications help address the concerns raised in the initial reviews, and we would appreciate reviewers reconsidering their assessments in light of the revised evidence. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.