- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
EpiBench evaluates whether general-purpose LLMs can infer epitope-related information from antigen and antibody sequences. It contains 1609 examples spanning targetable region discovery, antibody-conditioned epitope identification, epitope binding, functional epitope assessment, and escape assessment. Labels are derived from structural contacts, curated functional assays, and deep mutational scanning measurements. Nine LLMs are evaluated through closed-book, automatically scored prompts, with additional analyses of antigen length, explicit reasoning, and failure patterns.
Critical strengths
- The benchmark brings several relevant antibody-discovery decisions into a clear, unified evaluation framework grounded in existing experimental resources.
- The construction includes useful shortcut controls, particularly antibody-similarity matching for binning and mutation-matched escape/non-escape examples.
Critical weaknesses
- The validity of a unique answer in functional epitope assessment needs further verification. Task 4 uses functionally supported epitopes as correct answers, while distractors can include binding-only epitopes or synthetic windows. These distractors are not necessarily functionally negative, making it unclear whether the task measures functional understanding or differences in available experimental evidence. Have you audited candidate sets for alternative functional support and ambiguous answers? Reporting this audit and the proportions of different distractor types would strengthen confidence in the task's interpretation.
- The evaluation metrics and controls are insufficiently specified to support some central conclusions. The exact RegionRecall@50 calculation and the conversion of ranked residue lists into AUROC scores are unclear. Furthermore, the fixed 50-residue prediction budget complicates interpreting the decline with antigen lengths as an LLM-specific localization limitation. Can the authors specify these scoring rules and provide length-stratified random or simple heuristic baselines? These details are important for distinguishing model limitations from effects of the evaluation design.
Questions for authors
Please see the questions included in Critical Weakness 1 and 2.
5 comments
Thank you for the clarification and additional analyses, which adequately address my concerns about the scoring definitions and length-dependent evaluation. Regarding Task 4, reframing this task around documented functional evidence would improve its validity. Please revise the interpretation and report whether the main conclusion holds when the Task 4 interpretation changes. With these revisions, I will raise my score to 3.
Thank You and Follow-up
Thank you for your follow-up. We are glad that the clarifications and additional analyses addressed your concerns about the scoring definitions and length-dependent evaluation. We also appreciate your suggestion to frame Task 4 around documented functional evidence. We will revise the Task 4 description, Table 2, and the discussion to clarify that the task asks models to identify the candidate with documented functional support in our curated assay records. For candidates with only binding evidence or no functional testing, the available records leave functional activity undetermined. Across the nine evaluated LLMs, Task 4 accuracy ranged from 23.8% to 35.6%, compared with a 25% chance baseline. These scores describe how often the models selected the candidate supported by the curated records.
The main finding of partial and uneven performance under the sequence-only protocol still holds when we consider Tasks 1, 2, 3, and 5 alone. On Task 1, Gemini-3-Flash achieved 56.9 RegR@50 versus 40.8 for the random top-50 baseline on the same antigens. Its residue-level AUROC was 54.6 versus 50.0 at chance, indicating stronger performance at locating broad regions than at distinguishing individual residues. The best accuracies on Tasks 2 and 5 were 33.6% and 54.7%, compared with chance levels of 25% and 50%, respectively. Task 3 showed a larger gain, reaching 67.3% against a 50% chance baseline. We will make clear in the revised discussion how these four tasks support the main finding.
Author Response (Part 1)
Thank you for the constructive comments. We appreciate your recognition of EpiBench's unified framework and sampling controls. Your suggestions on functional-label validity, scoring definitions and length-matched references help us clarify the evaluation. We address each point below.
Weakness 1: The validity of a unique answer in functional epitope assessment needs further verification, and the candidate sets should be audited for alternative functional support
Thank you for highlighting candidate-set validity. Binding-only evidence and untested windows leave the requested functional outcome unknown. Table R1 gives the construction categories; negative tags require assay-level verification.
We audited the 19 items with three negative-tagged distractors against individual IEDB records and available primary-source evidence, checking candidate identity, measured endpoint and assay context. Four items from three antigens met the record-level criteria for one positive and three negative outcomes for a selected endpoint under comparable conditions. Eligibility was fixed before re-scoring saved predictions against the original labels. Other items did not qualify or remain unresolved.
These four items provide a descriptive sensitivity analysis, with substantial model variation. They cannot validate the full 340-item task or establish stable rankings. Negative outcomes apply only to the selected endpoint and conditions; untested functionality remains unknown.
Appendix D.2 will document criteria, per-item decisions, sources and re-scoring implementation. Appendix B.2.4 will clarify functional scope, conflicting evidence and uncertain or multiple-supported answers. The revised release will flag affected items and unresolved evidence.
Table R1: Provenance of all 1,020 Task 4 distractors.
Table R2: Saved-prediction accuracy on the original 19 tagged items and the 4 record-matched items (%).
Weakness 2: The evaluation metrics and controls are insufficiently specified, and the fixed prediction budget complicates the interpretation of the length trend
Thank you for requesting these scoring details and references. For each antigen, let
Scontain valid positions after ordered de-duplication and truncation to 50, with reference patches{P_j}:RegR@50 = (1/M) sum_j 1[|S intersect P_j| / |P_j| >= 0.2].A patch is hit at 20% coverage; scores are averaged over antigens. For AUROC, rank
iamongn <= 50positions receives scoren-i+1; unlisted positions receive zero. Positives are the union of observed epitope residues; other positions are benchmark negatives. Ties receive half credit, followed by averaging over antigens.Random RegR@50 uses 400 uniform draws of
min(50,L)positions per antigen (seed 0) under the same scorer, giving 40.8 overall. Table R3 compares one fixed model on the same 295 antigens.Random performance also declines with length. Section 4.2 will interpret model scores relative to this reference; raw decline alone does not establish an LLM-specific localization limitation. These length bins do not jointly control region size.
The mean ideal top-50 AUROC ceiling is 95.2, so the budget does not force chance-level scores. Appendix D.1 will provide scoring definitions, references, labels and implementations.
Table R3: Task 1 RegR@50 (0–100), using Figure 4's original bins. Length is in residues. Middle bins include the lower boundary and exclude the upper boundary. Differences are percentage points, calculated before rounding.
Table R4: Task 1 references (0–100). Propensity scales use seven-residue smoothing before top-50 selection. Random AUROC is theoretical; random RegR@50 is Monte Carlo estimated.