- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces ProteinFitnessBench, a controlled framework for mutation-order extrapolation in protein fitness prediction that separates representation, readout, objective, and evaluation. It reports fixed-budget TTK@K alongside global ranking metrics such as Spearman correlation and NDCG. Using frozen ESM-2 feats, the authors compare a ESM-WHT readout package against a ESM-MLP comparator, along with contextual controls, across four protein-fitness landscapes and five seeds. The main finding is that ESM-WHT improves global ranking over ESM-MLP but does not yield a consistent TTK@10 gain, which the authors describe as a ranking–retrieval misalignment. Supporting GB1 analyses under sparse supervision and altered label resolution, together with decision-alignment interventions such as elite-focused weighting, pair-difference supervision, and observed four-point supervision, further examine the distinction between global ranking and elite retrieval under a finite assay budget.
Critical strengths
The paper explores a relevant and practically important problem: elite retrieval under a fixed assay budget and mutation-order extrapolation. It explicitly separates global ranking from TTK@K in evaluation, and its negative result that improved global ranking does not consistently yield better elite retrieval motivates more decision-aware evaluation in protein fitness prediction.
Critical weaknesses
-
The paper's framing is misleading. The title and abstract suggest a benchmark contribution, but the paper does not introduce a new dataset, a standard evaluation protocol, a public leaderboard, or external model submissions. It only runs internal comparisons on existing datasets. It is also not a method paper: there is no new module, no new training mechanism, no new objective, and no new algorithm. The approach freezes ESM-2 features and swaps readout heads. This is an empirical exercise, not a research contribution. For ICLR, this is far below the expected bar.
-
The novelty and depth are insufficient. Every component used in this paper is pre-existing. ESM-2 features are frozen and taken off the shelf. The MLP head, mutation-order embedding, WHT, weighted squared error, margin hinge, and LambdaRank are all existing techniques. The authors simply arrange and combine these components into different readout packages and then compare them. There is no new module, no new training mechanism, no new objective, and no new learning algorithm. The WHT is not introduced as a novel contribution, and the authors themselves admit it is used as structured feature mixing rather than a new epistasis estimator. This is not a methodological innovation; it is a combinatorial exercise. The paper also lacks depth: it does not analyze why global ranking and fixed-budget retrieval become misaligned, does not probe the ESM features themselves, and does not study the mechanisms of different readouts. A stronger paper would use such questions as motivation and then propose targeted solutions. Instead, this paper spends extensive space documenting a shallow empirical observation.
-
The experimental design is repetitive and lacks research value. Since the method is merely a combination of existing components, the experiments are naturally limited to comparing these combinations. The paper presents many tables, numbers, seeds, and conditions, but the core conclusion is essentially one point: good global ranking does not imply good fixed-budget elite retrieval. This conclusion is already well known in information retrieval and ranking. The supporting analyses are underpowered, including a single-seed sparse-supervision stress test without a matched non-WHT arm, validation-only interventions, and a four-point auxiliary that fails its pre-specified promotion gate. The main comparison is also confounded and not parameter-matched, so the claimed effect cannot be attributed to WHT. Despite the long experimental narrative, the work does not provide a deeper mechanistic understanding, a new decision-aware objective, or a robust empirical law.
Questions for authors
Seed the weakness above.