acceptodds
Under review as a conference paper at ICLR 2027

EpiBench: Can LLMs Understand Epitopes for Antibody Drug Discovery?

Abstract

Epitopes determine where antibodies bind antigens and shape downstream therapeutic properties such as functional blockade and escape resistance, making epitope understanding central to antibody drug discovery. Although large language models (LLMs) have shown strong biomedical reasoning ability, it remains unclear whether they can infer epitope information directly from antigen and antibody sequences. Existing epitope resources typically focus on isolated prediction tasks or rely on specialized structural settings, while general protein benchmarks do not evaluate epitope-centered decisions across the antibody development workflow. To address this gap, we introduce EpiBench, a closed-book, sequence-based, and automatically scorable benchmark for evaluating epitope reasoning in LLMs. EpiBench contains 1,609 curated samples grounded in structural antibody–antigen contacts, curated functional B-cell assays, and deep mutational scanning escape measurements. It covers five connected tasks: targetable region discovery, antibody-conditioned epitope identification, epitope binning, functional epitope assessment, and antibody escape assessment, with controlled sampling to reduce shortcut-based evaluation artifacts. We evaluate nine general-purpose LLMs and analyze their behavior through task-specific baselines, antigen length stratification, explicit-reasoning comparison, and failure-mode inspection. The results show that current LLMs capture partial epitope-related signals but remain limited in antibody-specific sequence grounding, long-context residue localization, and biologically grounded reasoning. Therefore, EpiBench provides a diagnostic testbed for measuring and improving sequence-aware biomedical LLMs toward reliable LLM-assisted antibody discovery.

Primary area:
datasets and benchmarks
Keywords:
LLMs, Antibody Drug Discovery, Epitope Prediction, Antibody–Antigen Interaction, Protein Sequence Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_hDhPOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EpiBench evaluates whether general-purpose LLMs can infer epitope-related information from antigen and antibody sequences. It contains 1609 examples spanning targetable region discovery, antibody-conditioned epitope identification, epitope binding, functional epitope assessment, and escape assessment. Labels are derived from structural contacts, curated functional assays, and deep mutational scanning measurements. Nine LLMs are evaluated through closed-book, automatically scored prompts, with additional analyses of antigen length, explicit reasoning, and failure patterns.

Critical strengths
  1. The benchmark brings several relevant antibody-discovery decisions into a clear, unified evaluation framework grounded in existing experimental resources.
  2. The construction includes useful shortcut controls, particularly antibody-similarity matching for binning and mutation-matched escape/non-escape examples.
Critical weaknesses
  1. The validity of a unique answer in functional epitope assessment needs further verification. Task 4 uses functionally supported epitopes as correct answers, while distractors can include binding-only epitopes or synthetic windows. These distractors are not necessarily functionally negative, making it unclear whether the task measures functional understanding or differences in available experimental evidence. Have you audited candidate sets for alternative functional support and ambiguous answers? Reporting this audit and the proportions of different distractor types would strengthen confidence in the task's interpretation.
  2. The evaluation metrics and controls are insufficiently specified to support some central conclusions. The exact RegionRecall@50 calculation and the conversion of ranked residue lists into AUROC scores are unclear. Furthermore, the fixed 50-residue prediction budget complicates interpreting the decline with antigen lengths as an LLM-specific localization limitation. Can the authors specify these scoring rules and provide length-stratified random or simple heuristic baselines? These details are important for distinguishing model limitations from effects of the evaluation design.
Questions for authors

Please see the questions included in Critical Weakness 1 and 2.

5 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for the constructive comments. We appreciate your recognition of EpiBench's unified framework and sampling controls. Your suggestions on functional-label validity, scoring definitions and length-matched references help us clarify the evaluation. We address each point below.


Weakness 1: The validity of a unique answer in functional epitope assessment needs further verification, and the candidate sets should be audited for alternative functional support

Thank you for highlighting candidate-set validity. Binding-only evidence and untested windows leave the requested functional outcome unknown. Table R1 gives the construction categories; negative tags require assay-level verification.

We audited the 19 items with three negative-tagged distractors against individual IEDB records and available primary-source evidence, checking candidate identity, measured endpoint and assay context. Four items from three antigens met the record-level criteria for one positive and three negative outcomes for a selected endpoint under comparable conditions. Eligibility was fixed before re-scoring saved predictions against the original labels. Other items did not qualify or remain unresolved.

These four items provide a descriptive sensitivity analysis, with substantial model variation. They cannot validate the full 340-item task or establish stable rankings. Negative outcomes apply only to the selected endpoint and conditions; untested functionality remains unknown.

Appendix D.2 will document criteria, per-item decisions, sources and re-scoring implementation. Appendix B.2.4 will clarify functional scope, conflicting evidence and uncertain or multiple-supported answers. The revised release will flag affected items and unresolved evidence.

Table R1: Provenance of all 1,020 Task 4 distractors.

Distractor categorynshare
Binding-only peptide57756.6%
Property-matched synthetic window30129.5%
Functional-negative (construction tag)14213.9%

Table R2: Saved-prediction accuracy on the original 19 tagged items and the 4 record-matched items (%).

ModelOriginal 19 accuracyMatched correct/nMatched accuracy
Gemini-3-Flash42.13/475.0
GPT-5.521.12/450.0
Qwen3.7-Plus21.10/40.0
GLM-5.215.80/40.0
DeepSeek-V4-Pro10.51/425.0
Kimi-K2.626.31/425.0
GPT-4o26.30/40.0
Grok 4.2021.11/425.0
Qwen3-235B31.62/450.0
Mean across nine models24.0—27.8

Weakness 2: The evaluation metrics and controls are insufficiently specified, and the fixed prediction budget complicates the interpretation of the length trend

Thank you for requesting these scoring details and references. For each antigen, let S contain valid positions after ordered de-duplication and truncation to 50, with reference patches {P_j}:

RegR@50 = (1/M) sum_j 1[|S intersect P_j| / |P_j| >= 0.2].

A patch is hit at 20% coverage; scores are averaged over antigens. For AUROC, rank i among n <= 50 positions receives score n-i+1; unlisted positions receive zero. Positives are the union of observed epitope residues; other positions are benchmark negatives. Ties receive half credit, followed by averaging over antigens.

Random RegR@50 uses 400 uniform draws of min(50,L) positions per antigen (seed 0) under the same scorer, giving 40.8 overall. Table R3 compares one fixed model on the same 295 antigens.

Random performance also declines with length. Section 4.2 will interpret model scores relative to this reference; raw decline alone does not establish an LLM-specific localization limitation. These length bins do not jointly control region size.

The mean ideal top-50 AUROC ceiling is 95.2, so the budget does not force chance-level scores. Appendix D.1 will provide scoring definitions, references, labels and implementations.

Table R3: Task 1 RegR@50 (0–100), using Figure 4's original bins. Length is in residues. Middle bins include the lower boundary and exclude the upper boundary. Differences are percentage points, calculated before rounding.

Antigen lengthnGemini-3-FlashRandom-50Difference
<2007885.790.4-4.7
200–40010758.639.4+19.2
400–6005939.810.4+29.4
600–8002830.34.0+26.3
>8002327.61.6+26.0

Table R4: Task 1 references (0–100). Propensity scales use seven-residue smoothing before top-50 selection. Random AUROC is theoretical; random RegR@50 is Monte Carlo estimated.

MethodRegR@50AUROC
Uniform random 50 positions40.850.0
Kyte-Doolittle hydrophilicity47.351.8
Kolaskar-Tongaonkar antigenicity31.447.7
N-terminal 50 residues32.748.8
C-terminal 50 residues29.548.0
Gemini-3-Flash56.954.6

Thank you for the clarification and additional analyses, which adequately address my concerns about the scoring definitions and length-dependent evaluation. Regarding Task 4, reframing this task around documented functional evidence would improve its validity. Please revise the interpretation and report whether the main conclusion holds when the Task 4 interpretation changes. With these revisions, I will raise my score to 3.

AuthorsOpenReview ↗

Thank You and Follow-up

Thank you for your follow-up. We are glad that the clarifications and additional analyses addressed your concerns about the scoring definitions and length-dependent evaluation. We also appreciate your suggestion to frame Task 4 around documented functional evidence. We will revise the Task 4 description, Table 2, and the discussion to clarify that the task asks models to identify the candidate with documented functional support in our curated assay records. For candidates with only binding evidence or no functional testing, the available records leave functional activity undetermined. Across the nine evaluated LLMs, Task 4 accuracy ranged from 23.8% to 35.6%, compared with a 25% chance baseline. These scores describe how often the models selected the candidate supported by the curated records.

The main finding of partial and uneven performance under the sequence-only protocol still holds when we consider Tasks 1, 2, 3, and 5 alone. On Task 1, Gemini-3-Flash achieved 56.9 RegR@50 versus 40.8 for the random top-50 baseline on the same antigens. Its residue-level AUROC was 54.6 versus 50.0 at chance, indicating stronger performance at locating broad regions than at distinguishing individual residues. The best accuracies on Tasks 2 and 5 were 33.6% and 54.7%, compared with chance levels of 25% and 50%, respectively. Task 3 showed a larger gain, reaching 67.3% against a 50% chance baseline. We will make clear in the revised discussion how these four tasks support the main finding.

Reviewer_x69nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EpiBench, a closed-book, sequence-based benchmark for evaluating LLMs on epitope-related decisions in antibody discovery. It contains 1,609 examples across five tasks: targetable region discovery, antibody-conditioned epitope identification, epitope binning, functional epitope assessment, and antibody escape assessment. The examples are derived from AsEP, SAbDab, IEDB, UniProt-linked antigen sequences, and deep mutational scanning measurements. The construction removes direct identifiers and uses task-specific sampling controls to reduce potential shortcuts. The authors evaluate nine general-purpose LLMs, compare them with specialized methods where applicable, and analyze antigen length, explicit reasoning, and incorrect reasoning traces. The intended contribution is a standardized diagnostic resource for studying LLM sequence-grounded epitope understanding.

Critical strengths
  1. Important question and useful task coverage. Epitope localization, antibody specificity, competition, functional activity, and escape are relevant to antibody discovery. Bringing these questions into a common, automatically scored interface could make a useful contribution beyond evaluating biomedical factual knowledge. The paper is clearly written and easy to follow.
  2. Experimentally grounded sources and thoughtful construction controls. The benchmark draws on structural contacts, curated assays, and escape measurements. Same-antigen distractors, antibody-similarity matching for binning, and mutation-matched escape/non-escape pairs are sensible attempts to make the tasks more informative. These design choices are a substantive strength, although their effectiveness needs further validation.
  3. The LLM evaluation lacks biomedical domain-adapted models. Although specialized epitope predictors are included for some tasks, all nine evaluated LLMs are general-purpose models. Evaluating a biomedical LLM, such as MedGemma, under the same protocol would help establish whether biomedical adaptation improves performance on these tasks. A comparison with its general-purpose counterpart would be particularly informative. Without this comparison, the conclusions remain limited to general-purpose LLMs and provide little evidence about the benefits or limitations of biomedical adaptation for epitope understanding.
Critical weaknesses
  1. The validity of the functional-assessment labels is not sufficiently established. Appendix B.2.4 allows Task 4 distractors to be functionally negative, binding-only, or synthetic peptides. Binding-only evidence does not establish that a peptide lacks the requested functional activity, and an untested synthetic region is not a verified negative. Consequently, a biologically plausible prediction could be marked incorrect because functional evidence has been reported for only one candidate. The example in Appendix C.2 also groups several different functional outcomes together, while the construction aggregates assay evidence by its strongest type. It is unclear how conflicting assay contexts or multiple functional candidates are handled. This matters because functional assessment is a central distinction from conventional epitope prediction. The single-answer labels need stronger validation to establish that the task measures biological functionality rather than differences in annotation availability.
  2. The evaluation is insufficiently specified and controlled to support its diagnostic conclusions. I could not find a mathematical definition of RegionRecall@50, its region-hit criterion and aggregation rule, or the construction of full-sequence scores for AUROC from a list of at most 50 predicted positions. Table 2 provides no random RegR@50 reference. With a fixed prediction budget, performance can depend strongly on antigen length and region size, so the length analysis does not isolate loss of sequence localization ability. The specialized baselines are also underspecified: Section 4.1 names “EpiPred-derived” methods for Tasks 2–3 without explaining their adaptation or inputs. The original EpiPred uses structural information, making this distinction consequential to the comparison. Finally, targeted input ablations are needed to establish whether above-chance results use antibody-specific information. These omissions limit reproducibility and the interpretation of benchmark scores.
Questions for authors
  1. For Task 4, how many distractors are measured functional negatives, binding-only peptides, or synthetic windows? How are assay context, conflicting records, and multiple functional-positive candidates handled? Could you report an audited subset with comparable functional testing and an unambiguous answer, or accommodate uncertain/multiple answers?

  2. What is the exact definition of RegR@50, including region matching and aggregation? How are AUROC scores assigned to all residues? Please report a random 50-position baseline stratified by length and region size to establish how much of the observed length trend exceeds the baseline trend.

  3. What exactly are the EpiPred-derived baselines for Tasks 2–3? Do they use experimental structures, predicted structures, or sequences only? How are their predictions converted into benchmark answers, and what checks were made for overlap with their training data?

  4. Do the findings persist under antibody-removal/swap controls for Tasks 2 and 5, candidate-only controls for Tasks 2 and 4, and a CDR-similarity-only baseline for Task 3? These would clarify whether the scores measure the intended conditioning rather than residual shortcuts.

  5. Could you provide the evaluation implementation, exact model/API identifiers and evaluation dates, decoding and reasoning settings, parsing/failure rules, and per-item predictions? Uncertainty estimates should also account for shared antigens, antibodies, and matched mutation pairs where applicable.

6 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for the constructive review. We appreciate your recognition of EpiBench's task coverage and construction controls. We also value your suggestions on functional labels, scoring, input controls, reproducibility and biomedical model comparisons. We address each point below.


Weakness 1: The validity of the functional-assessment labels is not sufficiently established

Thank you for raising this concern about label validity. Binding-only evidence and untested candidates leave the requested functional outcome unknown. Table R1 reports construction provenance; functional-negative tags require assay-level verification.

We audited the 19 items with three negative-tagged distractors against IEDB records and available primary-source evidence. Four items from three antigens met the record-level criteria for one positive and three negative outcomes for a selected endpoint under comparable conditions. Eligibility was fixed before re-scoring saved predictions against the original labels.

Within this audit, conflicts are assessed by endpoint, study, antibody preparation, host and assay context. Outcomes from different functions or conditions are not pooled into universal activity labels. Candidate sets without a unique supported answer for the selected endpoint are not retained in the sensitivity subset.

With only four items and three antigen groups, this is a descriptive sensitivity analysis. It cannot validate the full 340-item task, establish stable model rankings, or attribute low scores solely to functional reasoning limitations. Negative outcomes apply only to the selected endpoint and conditions.

Appendix D.2 will document criteria, per-item decisions and sources. Appendix B.2.4 will clarify evidence aggregation, conflicting or multiple-supported answers, and functional scope. The revised release will flag affected items and unresolved evidence.

Table R1: Provenance of all 1,020 Task 4 distractors.

Distractor categorynshare
Binding-only peptide57756.6%
Property-matched synthetic window30129.5%
Functional-negative (construction tag)14213.9%

Table R2: Saved-prediction accuracy on the original 19 tagged items and the 4 record-matched items (%).

ModelOriginal 19 accuracyMatched correct/nMatched accuracy
Gemini-3-Flash42.13/475.0
GPT-5.521.12/450.0
Qwen3.7-Plus21.10/40.0
GLM-5.215.80/40.0
DeepSeek-V4-Pro10.51/425.0
Kimi-K2.626.31/425.0
GPT-4o26.30/40.0
Grok 4.2021.11/425.0
Qwen3-235B31.62/450.0
Mean across nine models24.0—27.8

Weakness 2: The evaluation is insufficiently specified and controlled to support its diagnostic conclusions

Thank you for highlighting these evaluation details. For each antigen, let S contain the valid predicted positions after ordered de-duplication and truncation to 50, and let {P_1,...,P_M} be the reference patches:

RegR@50 = (1/M) sum_j 1[|S intersect P_j| / |P_j| >= 0.2].

A patch is hit at 20% coverage; scores are averaged over antigens. For AUROC, rank i in a list of n <= 50 positions receives score n-i+1; unlisted positions receive zero. Positives are the union of observed epitope residues, with other positions treated as benchmark negatives. Ties receive half credit, followed by averaging over antigens.

Random RegR@50 uses 400 uniform draws of min(50,L) positions per antigen (seed 0), scored identically, giving 40.8 overall. Table R3 compares one fixed model on the same 295 antigens.

Random scores also decline with length, so Section 4.2 will compare model performance against this reference. Separate median-patch-size bins (<18, 18–22, ≥23 residues) give Gemini 54.4/56.8/60.3 versus random 44.7/36.7/39.6. These strata do not jointly control length and patch size.

Random-ranking AUROC has expectation 50.0. Ideal budgeted predictions attain min(1, 0.5 + 25/|U|), for union epitope U; the verified mean ceiling is 95.2. Thus, the budget does not force AUROC near chance.

Re-scoring BepiPred-3.0 top-50 predictions on the current 295 antigens gives 60.4 RegR@50 and 58.0 ranked-list AUROC under the same budget and scorer. Questions 3–4 address reference provenance and completed versus outstanding input controls. Appendix D.1 will give the scoring definitions, random protocol and implementations.

Table R3: Task 1 RegR@50 (0–100), using Figure 4's original bins. Length is in residues. Middle bins include the lower boundary and exclude the upper boundary. Differences are percentage points, calculated before rounding.

Antigen lengthnGemini-3-FlashRandom-50Difference
<2007885.790.4-4.7
200–40010758.639.4+19.2
400–6005939.810.4+29.4
600–8002830.34.0+26.3
>8002327.61.6+26.0

AuthorsOpenReview ↗

Author Response (Part 2)


Question 1: Distractor proportions, assay context, conflicting records, multiple positives, and an audited subset

Thank you for requesting these assay-level clarifications. Table R1 in our response to Weakness 1 gives distractor provenance; Table R2 reports saved-prediction results on the original 19 items and the four matched items from three antigens. The audit requires one positive and three negative outcomes for a selected endpoint under comparable conditions. Conflicting or unresolved evidence is not treated as verified inactivity. The four-item analysis remains descriptive and cannot validate the full task. Appendix D.2 will document criteria and supporting sources.


Question 2: Exact RegR@50, AUROC assignment, and random references by length and region size

Thank you for requesting these scoring details and references. Our response to Weakness 2 gives the formulas and rank-to-score assignment. Table R3 there reports the length-matched random reference; the text also gives separate patch-size strata. Appendix D.1 will include the definitions and sampling protocol, with labels and scorer provided for reproduction.


Question 3: EpiPred-derived inputs, adaptations, conversion to answers, and training overlap

Thank you for requesting clarification of these baselines. Task 2 uses EpiPred with experimental antibody and antigen structures. Up to three predicted patches are combined, mapped to sequence positions and scored by fractional candidate overlap; zero-overlap cases use a deterministic guess. Archived choices matched by candidate identity give 135/336 = 40.2%. Eighteen chain-matching failures are excluded (336/354 coverage); counting them as incorrect gives 38.1% on 354.

Task 3 is epitope transfer + Jaccard. It selects footprints of the same antibody or CDR-H3 nearest neighbors from records outside the test antigen's cluster (95% identity, 80% coverage), projects them through sequence alignment, then ranks pairwise Jaccard scores using the known 130/130 class balance. The archived result is 184/260 = 70.8%; this does not involve EpiPred inference.

These are structure-input and database-assisted references, respectively, with additional information beyond closed-book LLM prompts. We will retain the verified scores with these method names, inputs and coverage clarified in the revision. We have re-scored saved predictions; original inference has not been rerun, and overlap with EpiPred's original training data remains unverified.


Question 4: Antibody-removal/swap, candidate-only, and CDR-similarity controls

Thank you for suggesting these input controls. We have completed the references below. Frozen ESM-2 probes use supervised logistic regression with five-fold cross-validation grouped by exact antigen sequence for Tasks 2 and 4 and by matched mutation for Task 5. Task 3 uses sequence-similarity rules without fitting a classifier.

Task 2's Tier B result identifies residual candidate-only signal. Task 3 similarity rules rank all 260 pairs and assign the top 130 to the same-bin class using the known 130/130 balance; their scores depend on this class-balance information. Task 5 holds out mutations but allows antibodies to recur across folds, so the antibody-only score indicates antibody-associated label signal without establishing mutation-specific reasoning.

The requested same-LLM antibody-removal and swap experiments remain outstanding. These reference diagnostics do not establish the evaluated LLMs' dependence on antigen or antibody inputs. We will add paired input ablations using the same models, prompt settings and scorer, and distinguish those results from supervised feature references.

Table R4: Completed reference diagnostics (0–100). Tasks 2 and 4 report accuracy on the original task sets; Tasks 3 and 5 report balanced accuracy. Chance is 25 for Tasks 2/4 and 50 for Tasks 3/5.

TaskReference inputs or ruleScore
T2Candidate embeddings and position/size features, no antibody28.8; real-distractor Tier A 24.8; synthetic-distractor Tier B 39.0
T3CDR sequence-similarity rules53.8–63.8
T3Full-VH similarity (SequenceMatcher)72.3
T4Candidate peptide embeddings only27.1
T5Local WT/mutant windows and embedding difference only50.0
T5Antibody VH, VL and CDR-H3 only73.6

AuthorsOpenReview ↗

Author Response (Part 3)

Question 5: Implementation, exact model identifiers and dates, settings, failure rules, predictions and dependence-aware uncertainty

Thank you for requesting these reproducibility details. We will release evaluation and scoring code, per-item predictions and per-run parameter manifests. Archived API identifiers include gemini-3-flash-preview, grok-4.20-0309-non-reasoning and qwen3-235b-a22b-instruct-2507. The revision will document recorded decoding/reasoning settings, mark unspecified fields, and provide evaluation dates after verification. Available raw outputs, parsed answers, reasoning, finish reasons and token/latency metadata will accompany the release.

De-duplication prefers successful records and the latest success. Answers are re-parsed from raw outputs using final JSON answers and task-specific fallbacks. Task 1 removes out-of-range positions, de-duplicates in order and retains at most 50. Returned but unparseable discrete answers count as incorrect; API-error records are excluded from the main success-only scores.

For Gemini on Task 5, 164 correct among 355 successful records gives 46.2%; counting five API-error records as incorrect gives 45.6% over 360 attempts. We will report denominators, API errors, truncations and parsing failures explicitly, and align confidence intervals with the declared failure policy.

Existing 4,000-replicate bootstrap diagnostics group Tasks 2–4 by exact antigen sequence and Task 5 by matched mutation. For Task 3, Gemini's interval widens from [47.4, 59.4] to [39.4, 67.3]. Shared antibodies across groups and related antigens remain dependencies not fully captured by this grouping. Four-item Task 4 sensitivity results remain descriptive.


Reviewer_DK1XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors argue that existing biomedical evaluations do not adequately test whether general-purpose LLMs can infer epitope information from antigen and antibody sequences or use it in antibody discovery. To this end, they propose EpiBench, a benchmark of 1,609 examples spanning antigen-level region discovery, antibody-conditioned epitope identification, epitope binning, functional assessment, and antibody-specific escape prediction. Specifically, structural records from AsEP and SAbDab supply the first three tasks, while IEDB assays and published DMS measurements supply functional and escape labels. The construction removes identifying metadata and controls answer distributions, antibody similarity, and mutation identity to reduce shortcuts. Through zero-shot evaluation of nine LLMs, specialized references for Tasks 1–3, and analyses of antigen length, explicit reasoning, and incorrect explanations, the authors report limited and uneven performance. They position the benchmark as a diagnostic resource for studying deficiencies in sequence-grounded epitope reasoning.

Critical strengths
  1. EpiBench brings five epitope-related decisions into a common evaluation framework built from structural and experimental evidence. Its standardized inputs and automatic scoring could reduce the effort required to evaluate future models and make results easier to compare across studies. The task-specific controls, particularly pairing opposite escape outcomes for the same mutation, give subsequent work a useful starting point for assessing antibody-dependent predictions.
Critical weaknesses
  1. The empirical finding is largely what one would expect going in. General-purpose LLMs receive raw amino-acid sequences with no structural input, and epitope recognition is a spatial property of the antibody–antigen interface, so the result that they perform near random on antibody-conditioned identification and functional assessment confirms a prior rather than revising one. The paper’s own specialized references make the same point more directly: BepiPred-3.0 and the EpiPred-derived baselines lead on Tasks 1–3, which is the expected ordering between models trained on this data and models that were not. A benchmark can still be valuable when the headline result is unsurprising, but then the contribution rests on the resource, and I do not think the analyses here — length stratification, the CoT comparison, and the failure-mode taxonomy — add enough beyond confirming that expectation to carry acceptance on their own.

  2. Task 4 does not consistently distinguish missing functional evidence from experimentally established functional inactivity. Appendix B.2.4 permits binding-only peptides and synthetic windows to serve as distractors alongside experimentally negative peptides, although the former categories need not be functionally inactive. This leaves the task partly dependent on which candidates have documented evidence and weakens the interpretation of low scores as deficient functional reasoning.

Questions for authors
  1. Beyond BepiPred-3.0 and the EpiPred-derived baselines, were protein language model representations considered as a reference — for example, ESM2 embeddings with a light task-specific head, which takes the same sequence-only input the benchmark provides? This would help separate whether the reported difficulty is specific to general-purpose LLMs or persists in models pretrained on protein sequences, and it would be especially informative for Tasks 4 and 5, where no specialized reference is currently available.

  2. What fraction of Task 4 distractors are experimentally verified functional negatives, binding-only peptides, and synthetic windows? Do the findings persist on a subset where the candidate outcomes have been experimentally established under comparable assay conditions? This would materially affect whether I interpret the low scores as a limitation in functional prediction or a consequence of incomplete experimental annotations.

4 comments
AuthorsOpenReview ↗

Author Response (Part 1)

Thank you for the constructive review. We appreciate your recognition of EpiBench's unified framework and construction controls, and your suggestions on the benchmark contribution, functional evidence and sequence-based references. We address each point below.


Weakness 1: What EpiBench shows beyond the expected result

Thank you for this comment. EpiBench provides a reusable evaluation layer over AsEP/SAbDab contacts, IEDB assays and DMS measurements. We convert them into sequence prompts, candidate comparisons and automatic scoring, so researchers can evaluate one model across five decisions without rebuilding a separate evaluation for each source.

The shared protocol covers region discovery, antibody-conditioned identification, footprint binning, functional assessment and escape assessment. Developers can check whether localization improvements accompany stronger antibody conditioning or escape prediction, while tasks retain distinct samples and metrics. Task 3 matches pairs within CDR-similarity strata; Task 5 pairs opposite outcomes for the same mutation.

On Task 1, Gemini-3-Flash obtains 56.9 RegR@50 against random 40.8, with AUROC 54.6 against 50.0. This indicates coarse region prioritization with limited residue-level discrimination. BepiPred-3.0 remains stronger at 60.4 RegR@50 (Table R1).

For Task 2, EpiPred uses experimental structures (40.2%, covering 336/354 items). Task 3's 70.8% comes from epitope transfer + Jaccard, using external structural footprints and the known class balance. These provide context with additional information, rather than input-matched LLM comparisons.

Under the 50-position budget, random RegR@50 falls from 90.4 for antigens shorter than 200 residues to 1.6 above 800. Raw decline alone cannot establish an LLM-specific long-context limitation. Section 4.2 will compare performance against references using Figure 4's original bins.

We will foreground the resource contribution and calibrated comparisons, with CoT and the trace taxonomy as supporting analyses. Construction metadata, reference implementations and per-item predictions will accompany the release, with input and training settings stated.

Table R1: Task 1 references (0–100; top-50 budget). RegR@50 requires 20% patch coverage and averages over antigens. AUROC assigns descending ranks and zero elsewhere. Random RegR@50 uses 400 draws per antigen; random AUROC is theoretical. Heuristics use seven-residue smoothing. BepiPred was re-scored on the 295 current antigens. Scores are point estimates.

MethodRegR@50AUROC
Uniform random 50 positions40.850.0
Kolaskar-Tongaonkar antigenicity31.447.7
Kyte-Doolittle hydrophilicity47.351.8
ESMBind, sequence-based binding-site predictor46.151.1
Gemini-3-Flash, zero-shot56.954.6
BepiPred-3.0, top-50 predictions60.458.0
AuthorsOpenReview ↗

Author Response (Part 2)

Weakness 2: Whether Task 4 measures functional reasoning or annotation availability

Thank you for highlighting this distinction. Binding-only evidence and untested windows leave the requested functional outcome unknown. Table R2 reports construction categories; the audit below checks comparable measured outcomes.

An assay-record audit of the 19 items with three negative-tagged distractors retained four items from three antigens, each with one positive and three negative outcomes for its selected endpoint under comparable conditions. We checked IEDB records and available primary-source evidence, fixing eligibility before re-scoring saved predictions against the original labels. The remaining items were excluded from this sensitivity subset or remain unresolved.

With only four items and three antigen groups, this is a descriptive sensitivity analysis. It cannot validate the full 340-item task, establish stable model rankings, or show that low aggregate scores are solely due to functional reasoning limitations. Negative outcomes apply only to the tested endpoint and context; untested functionality remains unknown.

We will document the audit criteria, per-item decisions and source identifiers in Appendix D.2, clarify the functional evidence categories and their scope in Appendix B.2.4, and flag affected items in the revised release.

Table R2: Provenance of all 1,020 Task 4 distractors. Functional-negative tags are construction metadata, subject to assay-level verification.

Distractor categorynshare
Binding-only peptide57756.6%
Property-matched synthetic window30129.5%
Functional-negative (construction tag)14213.9%

Table R3: Re-scoring saved predictions on the original 19 tagged items and the 4 record-matched items. Accuracy is in percent.

ModelOriginal 19 accuracyMatched correct/nMatched accuracy
Gemini-3-Flash42.13/475.0
GPT-5.521.12/450.0
Qwen3.7-Plus21.10/40.0
GLM-5.215.80/40.0
DeepSeek-V4-Pro10.51/425.0
Kimi-K2.626.31/425.0
GPT-4o26.30/40.0
Grok 4.2021.11/425.0
Qwen3-235B31.62/450.0
Mean across nine models24.0—27.8

Question 1: Were protein language model representations considered as a reference...

Thank you for suggesting protein language model references. We have evaluated frozen, mean-pooled esm2_t12_35M_UR50D embeddings with logistic-regression heads on Tasks 2, 4 and 5, alongside the existing fine-tuned binding-site predictor ESMBind for Task 1. The probes use grouped five-fold cross-validation, with all predictions out-of-fold. Heads are trained on other folds' task labels, whereas the LLMs are evaluated zero-shot.

Input coverage differs from the LLM prompts. Task 2 does not embed the full antigen or VL. Task 4 omits the full antigen and functional condition. Task 5 uses local mutation windows and VH/CDR-H3, omitting VL and the rest of the supplied antigen region. These comparisons do not isolate the effect of model pretraining.

Task 5 holds out mutations but allows antibodies to recur across folds. A separate antibody-only probe using VH, VL and CDR-H3 reaches 73.6% under the same grouping. Its antibody inputs differ from the combined probe, and the result identifies antibody-associated label signal. The 71.9% score therefore does not establish mutation-specific escape reasoning or performance on unseen antibodies.

Appendix D.4 will document feature construction, grouping and training settings, and provide the reference implementations.

Table R4: Sequence-based references (0–100). Task 1 reports RegR@50; the other tasks report accuracy on the original task sets. Task 4's audited subset is reported separately in Table R3.

TaskReference inputsReference scoreBest evaluated LLM, zero-shotEvaluation grouping
T1ESMBind: antigen sequence46.156.9Existing model; 295 antigens
T2Candidate embeddings and position/size features; VH, CDR-H329.433.6Exact antigen sequence
T4Candidate peptide embeddings only27.135.6Exact antigen sequence
T5Local WT/mutant windows, embedding difference; VH, CDR-H371.954.7Matched mutation

Question 2: What fraction of Task 4 distractors are experimentally verified functional negatives...

Thank you for requesting this evidence breakdown. Table R2 gives provenance proportions. Table R3 reports four record-matched items from three antigens, with a nine-model mean of 27.8%. This small, endpoint-specific analysis cannot establish that full-task conclusions persist. Untested functionality remains unknown. Appendix D.2 will document eligibility and sources.

Dear Reviewer DK1X,

Thank you again for your careful review and constructive feedback, which helped us substantially improve the paper and conduct several additional analyses.

We are writing to kindly follow up and ask whether you have had an opportunity to review our response. Should any concerns or questions remain, we would be grateful for the opportunity to clarify them further.

If our response and additional results have addressed your concerns, we would sincerely appreciate your reconsideration of the assessment.

Thank you for your time and consideration.

Best regards, The Authors

Thank you for the detailed response and additional experiments. My concern regarding Weakness 1 has been clearly addressed. However, my concern regarding Task 4 (Weakness 2) remains. As the authors report, only 13.9% of the distractors are tagged as functional negatives, while the majority have unknown functional outcomes. Moreover, the sensitivity analysis includes only four matched items, which is insufficient to determine whether the conclusions hold when functional outcomes are experimentally verified. Therefore, I will maintain my current score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Program Chairs and Senior Area Chairs,

Thank you for your time and consideration. We also thank the reviewers for their careful evaluations and constructive feedback. Reviewer hDhP raised their overall rating from 2 to 3 after discussing the scoring definitions, length analysis, and revised interpretation of Task 4.

During the rebuttal, we specified the scoring rules and added random baselines evaluated on the same antigens. These comparisons clarify what EpiBench can diagnose. For example, Gemini-3-Flash achieved 56.9 RegR@50 versus 40.8 for random predictions, while its residue-level AUROC was 54.6 versus 50.0 at chance. The two metrics reveal a gap between prioritizing broad epitope regions and discriminating individual residues.

We audited 19 Task 4 items with three distractors tagged as functional negatives. Four met the assay comparability criteria. We will frame Task 4 as identifying candidates with documented functional support, since activity remains unknown for candidates with only binding evidence or no functional testing. The finding of partial and uneven performance remains supported by Tasks 1, 2, 3, and 5.

We also added sequence-based references, clarified the inputs and coverage of the specialized baselines, and evaluated MedGemma 1.5-4B-IT and Gemma 3-4B-IT on all 1,609 items under matched settings. Their relative performance varied across tasks.

We will incorporate these analyses and clarifications into the revised manuscript and release the evaluation code, reference implementations, and per-item predictions.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.