- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper explores the train-test split of several public antibody datasets by varying the removal of training data that are seemingly related to the test sets (F). The main conclusion is that if data points that are related to test sets are removed, the model performance drops across different antibody datasets.
Critical strengths
- Auditing several "benchmark" datasets in antibody worlds, thus pinpointing that current benchmark datasets may have overestimated the antibody model's performance
Critical weaknesses
Literally, I learned nothing except that the training sets of common public datasets contain data points related to the test sets. It reads like a workshop paper that needs more improvement in investigation and research questions.
Questions for authors
None
Thank you for the feedback. We agree that removing closely related training examples can be expected to lower benchmark scores. The revised investigation asks how much of that drop persists after controlling for alternative explanations, and whether it changes which antibody candidates a predictor prioritizes.
Does the gap survive matching training-set composition?
We hold the test examples and labels fixed. Family filtering removes training examples from sequence families represented in the test set. On AVIDa, we compare it with a different deletion that matches the number of training observations, distinct antibodies, and positive and negative observations in each of 20 training assay contexts. Both conditions keep or remove all observations of an antibody together.
With these quantities held fixed, median matched-control gains over family filtering are +0.1944 AUROC for AbLang2 Ridge, +0.1264 for ESM-2 Ridge, and +0.0608 for the neural pair classifier. All three gains are positive on all ten splits and for all 50 deterministically selected controls. The matched controls keep a direct training neighbor at 90% identity or higher for 57.9–62.1% of test rows, compared with zero under filtering. A separate comparison that also preserves the full distribution of retained antibody counts per family gives median gains of +0.0307 and +0.0128 AUROC for the two Ridge models.
These comparisons show that the number of antibodies and their measured assay composition are insufficient to explain the gap. Which examples remain available matters under these controls. A published binding predictor, DeepInterAware, shows the same direction on two splits from the same AVIDa collection. See Sections 4.1–4.2 and 4.4.
Does the gap survive when the test scaffold remains represented?
In the original AbDesign comparison, family filtering withheld the entire test scaffold. In the new comparison, both training conditions retain all 14 evaluated scaffolds. On 234 fixed test variants from seven antigens, the compared deletions exchange matched sequence groups within scaffolds while matching training counts, chain lengths, and recorded assay and context counts.
The matched control improves Spearman rank correlation by +0.1032 for AbLang2 and +0.1336 for ESM-2. The average gap remains positive for all five evaluated predictors after omitting any one antigen. This shows that performance can depend on the local training examples available for a scaffold that the model already sees in training. The exchanged groups also correspond to mutation sites, so the comparison measures local sequence and site-specific support together. See Section 5, Table 5, and Figure 4.
Does the difference affect which candidates are prioritized?
We refit the same neural classifier architecture under each training condition and rank the same held-out observations. At an identical top-5% screening budget within each assay context, matched-control training retrieves a median of 97.5 more measured binder observations per split than family-filtered training. The median paired precision gain is 19.1 percentage points. The retrieval gain holds on all ten splits and for all 50 controls. This is a retrospective comparison on one AVIDa collection. See Section 4.3 and Figure 3.
Building on similarity-aware evaluations such as BASE and SPECTRA, the revision connects the benchmark score to the training support available in the intended use case. We recommend stating which experimental groups the task assumes are represented, checking which related examples the filter actually removes, and reporting the fixed-budget retrieval difference alongside AUROC when the model is used to prioritize candidates. Figure 1 now shows both training interventions before the results.