acceptodds
Under review as a conference paper at ICLR 2027

ProteinFitnessBench: Learning to Extrapolate Protein Fitness Across Mutation Orders

Abstract

Protein engineering often requires selecting a small number of higher-order variants from models trained mainly on wild-type and low-order mutants. We introduce ProteinFitnessBench, a controlled framework that separates representation, readout, objective, and evaluation, and reports fixed-budget true-top-K recovery (TTK@K) alongside global ranking metrics. Across five seeds and four protein-fitness landscapes, a tested ΔESM-WHT readout package improves global ranking relative to a matched ΔESM-MLP comparator, but does not yield a consistent TTK@10 gain. This exposes a key evaluation mismatch: better global ranking does not necessarily produce better elite retrieval under a finite assay budget. Supporting GB1 analyses under sparse supervision and altered label resolution reinforce this distinction. ProteinFitnessBench provides a controlled way to test whether improvements in protein-fitness models translate into experimentally useful candidate selection.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein fitness prediction, protein engineering, deep mutational scanning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_2t9GOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces ProteinFitnessBench, a controlled framework for mutation-order extrapolation in protein fitness prediction that separates representation, readout, objective, and evaluation. It reports fixed-budget TTK@K alongside global ranking metrics such as Spearman correlation and NDCG. Using frozen ESM-2 feats, the authors compare a ESM-WHT readout package against a ESM-MLP comparator, along with contextual controls, across four protein-fitness landscapes and five seeds. The main finding is that ESM-WHT improves global ranking over ESM-MLP but does not yield a consistent TTK@10 gain, which the authors describe as a ranking–retrieval misalignment. Supporting GB1 analyses under sparse supervision and altered label resolution, together with decision-alignment interventions such as elite-focused weighting, pair-difference supervision, and observed four-point supervision, further examine the distinction between global ranking and elite retrieval under a finite assay budget.

Critical strengths

The paper explores a relevant and practically important problem: elite retrieval under a fixed assay budget and mutation-order extrapolation. It explicitly separates global ranking from TTK@K in evaluation, and its negative result that improved global ranking does not consistently yield better elite retrieval motivates more decision-aware evaluation in protein fitness prediction.

Critical weaknesses
  1. The paper's framing is misleading. The title and abstract suggest a benchmark contribution, but the paper does not introduce a new dataset, a standard evaluation protocol, a public leaderboard, or external model submissions. It only runs internal comparisons on existing datasets. It is also not a method paper: there is no new module, no new training mechanism, no new objective, and no new algorithm. The approach freezes ESM-2 features and swaps readout heads. This is an empirical exercise, not a research contribution. For ICLR, this is far below the expected bar.

  2. The novelty and depth are insufficient. Every component used in this paper is pre-existing. ESM-2 features are frozen and taken off the shelf. The MLP head, mutation-order embedding, WHT, weighted squared error, margin hinge, and LambdaRank are all existing techniques. The authors simply arrange and combine these components into different readout packages and then compare them. There is no new module, no new training mechanism, no new objective, and no new learning algorithm. The WHT is not introduced as a novel contribution, and the authors themselves admit it is used as structured feature mixing rather than a new epistasis estimator. This is not a methodological innovation; it is a combinatorial exercise. The paper also lacks depth: it does not analyze why global ranking and fixed-budget retrieval become misaligned, does not probe the ESM features themselves, and does not study the mechanisms of different readouts. A stronger paper would use such questions as motivation and then propose targeted solutions. Instead, this paper spends extensive space documenting a shallow empirical observation.

  3. The experimental design is repetitive and lacks research value. Since the method is merely a combination of existing components, the experiments are naturally limited to comparing these combinations. The paper presents many tables, numbers, seeds, and conditions, but the core conclusion is essentially one point: good global ranking does not imply good fixed-budget elite retrieval. This conclusion is already well known in information retrieval and ranking. The supporting analyses are underpowered, including a single-seed sparse-supervision stress test without a matched non-WHT arm, validation-only interventions, and a four-point auxiliary that fails its pre-specified promotion gate. The main comparison is also confounded and not parameter-matched, so the claimed effect cannot be attributed to WHT. Despite the long experimental narrative, the work does not provide a deeper mechanistic understanding, a new decision-aware objective, or a robust empirical law.

Questions for authors

Seed the weakness above.

Reviewer_fYwUOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces ProteinFitnessBench, a framework for evaluating protein fitness prediction models, with an emphasis on that better global ranking does not necessarily produce better elite retrieval under limited experimental budgets. The authors compare Walsh–Hadamard-based readout and MLP baselines, and show that improvements in NDCG and Spearman correlation do not necessarily lead to better top-K retrieval.

Critical strengths

The manuscript addresses a practical problem in protein engineering by emphasizing the importance of evaluating models under limited experimental budgets. One strength is its experimental reporting of mixed and negative results, without overstating the benefits of the WHT-based model.

Critical weaknesses
  1. The logic of this paper is really difficult to follow, especially the Introduction and the Method part. After gathering info from everywhere, the claimed main contribution of this paper seems not an improved protein fitness predictor, but rather a benchmarking framework intended to distinguish global ranking performance from fixed-budget elite retrieval. However, I believe this claim is already well established in this field, e.g., ProteinGym. The manuscript should articulate what new methodological or scientific insights ProteinFitnessBench provides beyond illustrating this known distinction in a protein fitness prediction setting.

  2. ProteinGym and FLIP2 already provide extensive evaluations for protein fitness prediction. Furthermore, TTK@K is equivalent to standard top-K precision/recall under the stated definition. The authors should clearly describe which previously unavailable evaluation capabilities ProteinFitnessBench introduces and why these constitute a substantive contribution.

Questions for authors

the role of the WHT-based readout remains unclear. The experimental results are not architecturally matched and do not isolate the effect of the WHT-transform. If the WHT readout serves only as a case study illustrating disagreement between evaluation metrics, is this particular comparison is central to the paper? And whether the observation persists across a more broader range of predictors?

Reviewer_HBFhOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes ProteinFitnessBench, a new protein ML benchmark based on the FLIP2 dataset. The main contribution of the paper is to primarily report top-K recovery (TTK @ K) and can disentangle the effect of the design protocols (e.g., readout heads). Experimental results show that those design protocols behave differently in TTK @ K and Spearman ranking.

Critical strengths

S1. Thorough comparison in TTK @ K vs Spearman ranking *The paper claims that TTK @ K is more closely related to realistic applications than Spearman ranking, and compares that TTK @ K and Spearman ranking can behave differently in various experimental settings and design choices. *However, I do not consider this contribution itself strong, as different metrics having different trends are kind of expected (especially Spearman ranking vs top-k).

Critical weaknesses

W1. Weak contribution on the benchmark. *The benchmark uses the FLIP2 dataset and the only difference is the metric and its protocol. *However, TTK @ K is already reported in prior works too, and it can’t be considered as a novel contribution. *The evaluation protocol from the benchmark is more like a design choice of the individual methods, rather than a feature that is newly enabled by this benchmark. W2. Serious concerns in the writing *Overall, the paper is not organized well and really hard to read. *For example, the introduction does not make the claim clearly; for example, promoting TTK @ K metric is vaguely splitted across 2nd - 4th paragraph, and it is not even included in the main proposal “To disentangle representation, readout, training objective, distribution shift, and decision quality,” *Moreover, experimental results do not explain what each experiment tries to show and how it reinforces the paper’s claim or the contribution. It is deferred in the discussion section, but still the section reiterates the results, rather than explaining how it is related to the contribution / novelty of the proposed benchmark. *I believe that the paper requires a substantial amount of revision on the writing.

Questions for authors

TeX is supported Q1. Connection between the experimental result and the contribution of the benchmark

  • Can you clarify how those experimental results support the main claim of the paper or the contribution of the benchmark?
  • If the experimental result clearly supports the novelty and contribution of the benchmark, then I’m willing to raise the score (I cannot see those connections in the current version of the paper).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.