acceptodds
Under review as a conference paper at ICLR 2027

MDLAS: Multi-Dimension Leakage-Aware Splits for Antibody-Antigen Structural Benchmarks

Abstract

Existing antibody-antigen structural benchmarks often treat leakage control primarily as a sequence-level property. We argue that this is insufficient: a split can prevent direct sequence overlap while retaining redundancy through shared antigens, epitopes, or paired antibody sequences, and can still exhibit substantial distributional imbalance. We present MDLAS (Multi-Dimension Leakage-Aware Split), a framework for constructing antibody-antigen structural benchmark splits that jointly controls cross-split redundancy and marginal distributional consistency. MDLAS builds a multi-relation graph over five redundancy signals spanning CDR-H3 identity (exact duplication and sequence clustering), antigen identity (shared antigen chains), epitope identity (shared binding footprint), and paired variable-domain similarity (co-occurring heavy/light chain pairs), assigns connected components indivisibly across train, validation, and test, and then optimizes the remaining feasible assignments across seven metadata axes. On a 15,557-entry protein/peptide-antigen working population, MDLAS produces an 80/10/10 split (12,445/1,556/1,556 entries) at the primary 70% CDR-H3 identity tier; independent auditing confirms zero cross-split overlap for the modeled redundancy relations, while distributional consistency is measured directly from the resulting assignments. This leakage control involves a substantial trade-off: because component assignment is indivisible, an antigen-dominated component containing 8,029 entries (51.6% of the working population) would overwhelm the balance of whichever split received it. Together with low-resolution-only components, it therefore cannot be assigned to the validation or test split. We further evaluate four CDR-H3 design baselines (MEAN, DiffAb, RefineGNN, and AgForce) under MDLAS. The models exhibit distinct performance profiles, and a binding-accuracy ranking reported on a smaller, differently constructed benchmark does not transfer to MDLAS, demonstrating that relative model comparisons can depend on benchmark construction. These results motivate treating leakage structure, distributional composition, and residual within-test redundancy as distinct properties when designing antibody structural benchmarks.

Primary area:
datasets and benchmarks
Keywords:
antibody design, antibody-antigen structure prediction, benchmark construction, data leakage, evaluation methodology, CDR-H3 design, structural biology, generative models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_txTVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MDLAS, a framework for constructing antibody–antigen structural benchmarks that controls multiple forms of data redundancy while attempting to maintain distributional balance. The authors construct splits based on antibody sequence, antigen identity, epitope identity, and paired heavy/light-chain similarity. They evaluate the resulting benchmark using four existing antibody design models.

Critical strengths

The authors address an important problem in evaluating antibody design models. I appreciate the systematic characterization of biological redundancy in existing structural datasets, particularly the observation that more than half the dataset becomes connected through overlapping similarity relationships. The framework is carefully implemented, and the authors provide extensive analyses of the resulting splits.

Critical weaknesses

My primary concern is the assumption that eliminating overlap across multiple biological dimensions necessarily produces a better benchmark. The appropriate splitting strategy depends on the intended design objective. For example, designing new antibodies against previously characterized antigens is an important application where leveraging information from related antibody–antigen interactions is desirable. Antigen-disjoint and epitope-disjoint evaluations test fundamentally different generalization capabilities, and I would view these as complementary rather than treating all overlap as undesirable leakage.

Additionally, the distributional balancing provides only modest improvements over random assignment, and the downstream model evaluations largely reproduce previously reported rankings. Consequently, the practical advantages over existing benchmarks remain unclear.

Questions for authors
  1. How do the authors envision MDLAS supporting different antibody design objectives, particularly applications where leveraging previously observed antibody–antigen interactions is desirable?
  2. Can the authors demonstrate that MDLAS provides more reliable estimates of generalization performance than existing benchmark splits?
  3. Could a stronger optimization procedure substantially improve distributional balance compared with the current greedy approach?
1 comment
AuthorsOpenReview ↗

Overlap reframed as objective-dependent, balance headroom quantified, and the generalization-estimate question answered directly.

Thank you for the careful review. We uploaded a revised PDF. Replies follow your points and questions.

W1 / Q1 (objectives and overlap). We agree. Overlap along a relation is leakage for a task that must generalize across it, and may be the setting of interest for another, for example designing against a previously characterized antigen. We reworded the abstract and introduction so that MDLAS is a conservative, leakage-aware protocol with an explicit, auditable relation set, not "the correct" split. A leave-one-relation-out ablation makes the trade-off measurable. The graph is a union-find over per-relation edge sets, so a different objective means a different set. Removing the antigen edge (epitope edges retained, so this is in effect an epitope-disjoint split) gives 3,814 components and a largest component of 4.2% instead of 51.6%, and a weighted balance of 0.0188 instead of 0.0525, at the price of 67.0% of test entries sharing an antigen cluster with training. Antigen-disjoint and epitope-disjoint evaluations are therefore complementary, as you say. We report this variant as a measurement; we have not trained baselines on it or released it as a benchmark split.

W2 (modest balance; rankings reproduce prior work). On balance, we agree and now report exactly how modest. The released split is better than 74 of 100 random feasible assignments under the weighted objective (63 of 100 unweighted), and the greedy fill closes at most about 11% (weighted) or 5% (unweighted) of the gap to what a swap search finds. Across 21 seeds the greedy mean is 0.36 random-draw standard deviations below the random mean. We deleted the sentence claiming the fill captures most of the achievable gain. On rankings, that they match earlier reports is the intended result: the claim is that the ranking survives a stricter split. We tested this directly: on test entries with no close training neighbour, rankings equal the full-test rankings on AAR, CAAR, RMSD, TM-score, DockQ, Fnat, iRMSD, Epitope-F1 and both Chimera scores. The practical advantage we claim is narrower than the original text suggested: the split removes a measurable inflation channel (near entries score 0.12 to 0.18 AAR higher) and makes residual similarity auditable. We do not claim a head-to-head advantage over other benchmarks.

Q2 (more reliable generalization estimates). No, not in this revision, and the paper no longer implies it. We have not compared against existing splits for reliability of generalization estimates. What we show is narrower: (i) residual near-duplicates inflate sequence-level scores (AAR higher by 0.12 to 0.18 on near entries), so the split controls an identified inflation channel; (ii) rankings on far entries only equal the full-test rankings on all primary metrics; (iii) all four baselines are retrained on a second tier (90%) split and the paper reports how results change. The right test of reliability is prospective: evaluate the trained models on SAbDab entries released after our 28 May 2026 snapshot. That needs new feature extraction and inference on the retained checkpoints, and we state it as future work.

Q3 (stronger optimization). Yes, and we now report it. A swap-based local search started from the released split lowers the weighted balance score from 0.0524 to 0.0208 (unweighted 0.0455 to 0.0184). Five restarts agree (0.0208 to 0.0210) and no random feasible draw approaches it (best 0.0418). Components stay indivisible, so Criterion 1 is unaffected. We did not adopt the searched split as the benchmark because it would invalidate the baseline results and the baselines cannot be retrained within the discussion window. We report the headroom, say that the greedy fill leaves substantial room, and leave a refined split to the next release. The ablation also shows that the dominant limit is the antigen-disjointness constraint: with the antigen edge removed, the same greedy fill reaches a weighted 0.0188.

Reviewer_rPibOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a new data splitting method MDLAS for antibody-antigen structural benchmarks. It builds one multi-relation graph over five redundancy signals and assigns connected components of that graph as indivisible units, to achieve zero cross-split overlap for all five modeled redundancy relations simultaneously. Evaluation on four baselines under the new split of the existing SAbDab database has been conducted.

Critical strengths
  1. The main splitting criteria about leakage-freedom and distributional consistency have been demonstrated in detail.
  2. The evaluation of the baselines under MDLAS has been conducted to reveal their performance profiles.
Critical weaknesses
  1. The paper falls within the area of datasets and benchmarks. It has limited contributions to this kind of area as it does not introduce a new dataset but only proposes a data splitting method.
  2. The criterion about distributional consistency is unconvincing. As in real-world cases, the distribution of the test set is usually unknown and can be different in different downstream tasks. In addition, the full working population is hard to determine.
  3. The paper has limited analysis regarding the insights or benefits that the new splitting provides, for example, its ability to better simulate real-world testing scenarios, or the fact that success on a test set created via this method often translates to a higher likelihood of passing laboratory validation.
Questions for authors
  1. Limited contribution.
  2. The criterion of distributional consistency is not well justified.
  3. Limited analysis of the benefits of the proposed method.
2 comments
AuthorsOpenReview ↗

Contribution clarified, distributional-consistency criterion reframed as a control, and new measurements of what the split changes.

Thank you for the review. We uploaded a revised PDF. Our replies follow your three points.

W1 / Q1 (contribution). We agree the paper should state plainly what is and is not new. The structures come from SAbDab and we claim no new experimental data. The contribution is a split-construction protocol and a benchmark resource built with it: (i) a 15,557-entry working population (19 known-bad entries excluded); (ii) a leakage graph over five redundancy relations; (iii) fixed train, validation and test assignments at four CDR-H3 identity tiers with zero cross-split overlap on all five relations; (iv) baseline results for four co-design models on the 70% tier. We rewrote the abstract and the contribution list to describe the paper this way and removed wording that could read as a new dataset. We also added a measurement of what the protocol changes compared with splits that control fewer relations. Rebuilding the split without the antigen edge leaves 67.0% of test entries sharing an antigen cluster with training; without CDR-H3 clustering, 6.9% share a CDR-H3 cluster; without the paired-Fv edge, 6.9% share a paired-Fv cluster.

W2 / Q2 (distributional consistency). We accept that, as worded, the criterion read like an independent measure of quality. It is a control: its purpose is to check that the leakage constraints do not push validation and test into a different population from training. We reworded it as a split-induced-shift control, reported alongside the Jensen-Shannon values, and state that low divergence does not by itself show the test set is representative of any deployment distribution. The reference is the working population, defined by explicit filters (protein antigen, known-bad exclusions) and stated as such; it is a convention, not a claim about the world. We also tempered the claim that the assignment is balanced: under the weighted objective the released split is better than 74 of 100 random feasible assignments, and a swap search finds much lower values (0.0208 against 0.0524), so we no longer present balance as a headline result.

W3 / Q3 (benefits and real-world relevance). We agree there is no wet-lab validation and the paper does not claim any. Whether better in-silico generalization estimates carry over to binding or function needs experimental work outside a benchmark-construction paper; we now state this as a limitation in the main text. What we can offer is a measurement of how conclusions depend on the split. Residual near-duplicates inflate sequence-level scores: on the 79 test entries with a ≥70% CDR-H3 neighbour in training, AAR is higher by 0.12 to 0.18 and RMSD lower by 0.25 to 0.74 Å for all four baselines (intervals exclude zero), while model rankings on far entries only match the full-test rankings on all primary metrics. The ablation shows that a split controlling CDR-H3 and paired-Fv similarity but not antigen identity leaves 67.0% of test entries with an antigen-cluster sibling in training. The paper also retrains all four baselines on the 90% tier split to show how results change with the split.

After reviewing the authors' response, I find that it does not directly address my key concerns and instead focuses on points that are not central to the issues I raised. Therefore, I maintain my original rating.

Reviewer_oU5VOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

MDLAS constructs train, validation and test splits for antibody-antigen complex entries. It assigns whole components of an antibody-antigen relation graph while optimizing seven marginal metadata distributions. Four CDR-H3 sequence-structure design models illustrate the benchmark, with additional similarity-tier and component-level analyses.

Critical strengths
  • The graph combines antibody and antigen relations without similarity-based deduplication. The authors explain why epitope edges are redundant after antigen edges.
  • Structural extraction documents missing atoms, low-confidence structures, chain errors and epitope alignment across differently numbered antigens.
  • The evaluation includes second-tier retraining and component-level means with bootstrap intervals, addressing dependence among related test entries.
Critical weaknesses
  • The authors scope their guarantee to specific identifiers and report 1,134 test-training CDR-H3 pairs at >70% identity. Their impact on scores remains unclear: this includes entries excluded from baseline training, and analogous antigen/paired-Fv pairwise audits are not reported.
  • The prior-work contrast needs qualification: CHIMERA-Bench v3 explicitly reports matching three covariates, and DataSAIL supports stratified similarity control for two entity types. The benefit of MDLAS's specific relation construction remains unquantified under matched constraints, as the authors acknowledge.
  • Under identical hard constraints, unweighted mean JS improves and outperforms 63 of 100 random assignments. This does not establish the claim that the heuristic captures most achievable improvement; the weighted objective needs the same comparison.
Questions for authors
  • Against the data actually used for training, what fraction of evaluation entries has a >=70%-identity CDR-H3 neighbor, and how do scores vary with this proximity?
  • Do rankings survive standard DockQ and loop errors after alignment on antigen or framework atoms, excluding the generated loop?
  • Which versioned baseline configurations and sampling settings were inherited or changed, and how variable are results across independent training runs?
2 comments
AuthorsOpenReview ↗

New measurements: nearest-neighbour audit with score effects, weighted balance comparison, relation ablation, baseline configurations.

Thank you for the specific questions. We uploaded a revised PDF; each point states what we measured and changed.

W1 / Q1 (proximity audit and effect on scores). You are right that counting pairs does not show their effect. For every test entry we computed the identity of its nearest training neighbour for CDR-H3, antigen and paired Fv (MMseqs2), against both the full 12,445-entry training set and the 11,152 full-confidence entries the baselines trained on. Test entries (of 1,556) with a close neighbour: CDR-H3 ≥70%: 81 (5.2%) against full training and 79 against baseline training, so the mismatch you noted changes two entries; antigen ≥70%: 8 (0.5%) and 7, a lower bound because our 80% coverage rule excludes short antigens contained in longer ones; paired Fv ≥90%: 26 (1.7%) in both. Effect on scores (1,514 test entries scored by all four baselines, 79 of them near): near entries score higher on sequence-level metrics for all four models, with AAR higher by 0.12 to 0.18 and RMSD lower by 0.25 to 0.74 Å, and component-level bootstrap intervals excluding zero. Because near entries are 5.2% of the test set, they move each model's mean by only about 0.006 to 0.009 AAR and 0.01 to 0.04 Å RMSD. On interface metrics (DockQ, Fnat, Epitope-F1, Chimera-B) all intervals include zero, but with 79 near entries they are wide, so we claim no absence of effect. Rankings on far entries only equal the full-test rankings on all primary metrics; they differ on three secondary property metrics, each between adjacent models. These are associations, not causal estimates, and we apply no multiplicity correction over the roughly 30 metrics screened. The main text and the pairwise-audit appendix report this.

W2 (prior-work contrast). Agreed. The paper now states that CHIMERA-Bench v3 reports matching three covariates and that DataSAIL supports stratified similarity control for two entity types, and it no longer implies MDLAS is the first to control more than one axis. We have not run a matched-constraint head-to-head against DataSAIL or GraphPart, and the limitations say so. We added a leave-one-relation-out ablation (the all-five control reproduces the released split exactly). Without the antigen edge, 67.0% of test entries share an antigen cluster with training; without CDR-H3 clustering, 6.9%; without the paired-Fv edge, 6.9%. Exact-duplicate and epitope edges are not independently binding (identical components and split when removed) because CDR-H3 and antigen clustering already imply them, so we keep them as leakage checks. This is not a comparison against other tools.

W3 (balance claim). You are right and the sentence was wrong; we deleted it. For the weighted objective Stage B optimizes, the released split scores 0.05245 against a random-feasible mean of 0.05618: better than 74 of 100 draws. A swap-based local search from the released split reaches 0.0208 weighted (0.0184 unweighted), which no random draw approaches. The greedy fill therefore closes at most about 10.6% of the gap between random assignment and what we can find (about 5% unweighted). Across 21 seeds the greedy mean is only 0.36 random-draw standard deviations below the random mean. The paper now says the fill is deterministic and auditable but leaves headroom. The ablation shows where the limit comes from: with the antigen edge removed, the same greedy fill reaches a weighted 0.0188.

Q2 (alignment and DockQ). The saved structures are CA coordinates of the generated CDR-H3 loop, not full complexes. MEAN and AgForce predict in the native antigen frame and RefineGNN in its own, so a no-fit loop RMSD is invalid for it. Because full complexes were not retained, we cannot re-score DockQ with antigen- or framework-anchored alignment from the existing outputs; that needs regeneration from the retained checkpoints, which we did not do, so we do not claim rankings survive it. The paper states that each interface metric uses its own optimal superposition, that no metric measures absolute loop placement in an independently fixed frame, and that our interface values depart from the DockQ convention, and lists this as a limitation.

Q3 (configurations and variability). A new appendix table gives each baseline's architecture, optimizer, budget, best and final checkpoint, parameter count and seed. All four were trained and run on MDLAS with their authors' provided parameters; none was re-tuned. Three stopped early (patience 10 on validation) and MEAN reached its 20-epoch cap. AgForce uses pretrained ESM-2 (650M) embeddings, which the table discloses; its authors report their own ablation of this component. We ran one training run per model; the cluster bootstrap covers test-set sampling, not training-seed variance, and the paper says so.

Presentation. We added an introduction roadmap and moved interrupting construction details to the appendix.

I thank the authors for their response which addresses my leakage concern and improved the discussion of prior work. I retain concerns with the limitations in structural evaluation and clear demonstration of benefits to the balancing procedure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.