- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces MDLAS, a framework for constructing antibody–antigen structural benchmarks that controls multiple forms of data redundancy while attempting to maintain distributional balance. The authors construct splits based on antibody sequence, antigen identity, epitope identity, and paired heavy/light-chain similarity. They evaluate the resulting benchmark using four existing antibody design models.
Critical strengths
The authors address an important problem in evaluating antibody design models. I appreciate the systematic characterization of biological redundancy in existing structural datasets, particularly the observation that more than half the dataset becomes connected through overlapping similarity relationships. The framework is carefully implemented, and the authors provide extensive analyses of the resulting splits.
Critical weaknesses
My primary concern is the assumption that eliminating overlap across multiple biological dimensions necessarily produces a better benchmark. The appropriate splitting strategy depends on the intended design objective. For example, designing new antibodies against previously characterized antigens is an important application where leveraging information from related antibody–antigen interactions is desirable. Antigen-disjoint and epitope-disjoint evaluations test fundamentally different generalization capabilities, and I would view these as complementary rather than treating all overlap as undesirable leakage.
Additionally, the distributional balancing provides only modest improvements over random assignment, and the downstream model evaluations largely reproduce previously reported rankings. Consequently, the practical advantages over existing benchmarks remain unclear.
Questions for authors
- How do the authors envision MDLAS supporting different antibody design objectives, particularly applications where leveraging previously observed antibody–antigen interactions is desirable?
- Can the authors demonstrate that MDLAS provides more reliable estimates of generalization performance than existing benchmark splits?
- Could a stronger optimization procedure substantially improve distributional balance compared with the current greedy approach?
Overlap reframed as objective-dependent, balance headroom quantified, and the generalization-estimate question answered directly.
Thank you for the careful review. We uploaded a revised PDF. Replies follow your points and questions.
W1 / Q1 (objectives and overlap). We agree. Overlap along a relation is leakage for a task that must generalize across it, and may be the setting of interest for another, for example designing against a previously characterized antigen. We reworded the abstract and introduction so that MDLAS is a conservative, leakage-aware protocol with an explicit, auditable relation set, not "the correct" split. A leave-one-relation-out ablation makes the trade-off measurable. The graph is a union-find over per-relation edge sets, so a different objective means a different set. Removing the antigen edge (epitope edges retained, so this is in effect an epitope-disjoint split) gives 3,814 components and a largest component of 4.2% instead of 51.6%, and a weighted balance of 0.0188 instead of 0.0525, at the price of 67.0% of test entries sharing an antigen cluster with training. Antigen-disjoint and epitope-disjoint evaluations are therefore complementary, as you say. We report this variant as a measurement; we have not trained baselines on it or released it as a benchmark split.
W2 (modest balance; rankings reproduce prior work). On balance, we agree and now report exactly how modest. The released split is better than 74 of 100 random feasible assignments under the weighted objective (63 of 100 unweighted), and the greedy fill closes at most about 11% (weighted) or 5% (unweighted) of the gap to what a swap search finds. Across 21 seeds the greedy mean is 0.36 random-draw standard deviations below the random mean. We deleted the sentence claiming the fill captures most of the achievable gain. On rankings, that they match earlier reports is the intended result: the claim is that the ranking survives a stricter split. We tested this directly: on test entries with no close training neighbour, rankings equal the full-test rankings on AAR, CAAR, RMSD, TM-score, DockQ, Fnat, iRMSD, Epitope-F1 and both Chimera scores. The practical advantage we claim is narrower than the original text suggested: the split removes a measurable inflation channel (near entries score 0.12 to 0.18 AAR higher) and makes residual similarity auditable. We do not claim a head-to-head advantage over other benchmarks.
Q2 (more reliable generalization estimates). No, not in this revision, and the paper no longer implies it. We have not compared against existing splits for reliability of generalization estimates. What we show is narrower: (i) residual near-duplicates inflate sequence-level scores (AAR higher by 0.12 to 0.18 on near entries), so the split controls an identified inflation channel; (ii) rankings on far entries only equal the full-test rankings on all primary metrics; (iii) all four baselines are retrained on a second tier (90%) split and the paper reports how results change. The right test of reliability is prospective: evaluate the trained models on SAbDab entries released after our 28 May 2026 snapshot. That needs new feature extraction and inference on the retained checkpoints, and we state it as future work.
Q3 (stronger optimization). Yes, and we now report it. A swap-based local search started from the released split lowers the weighted balance score from 0.0524 to 0.0208 (unweighted 0.0455 to 0.0184). Five restarts agree (0.0208 to 0.0210) and no random feasible draw approaches it (best 0.0418). Components stay indivisible, so Criterion 1 is unaffected. We did not adopt the searched split as the benchmark because it would invalidate the baseline results and the baselines cannot be retrained within the discussion window. We report the headroom, say that the greedy fill leaves substantial room, and leave a refined split to the next release. The ablation also shows that the dominant limit is the antigen-disjointness constraint: with the antigen edge removed, the same greedy fill reaches a weighted 0.0188.