- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
In this submission, the authors study how the performance of active-search methods in protein-variant screening depends on underlying components, particularly initial candidate rankings, feedback-update mechanisms, and acquisition strategies. The authors conduct their experiments on assays from ProteinGym and Domainone.
Critical strengths
I find the biggest strength of this paper to be its central theme: studying how different components of the search procedure influence the performance of active-search methods is in principle relevant.
Critical weaknesses
W1. Presentation quality
The central weakness of this submission is that it's very poorly written, to the extent that large portions of the text give the impression of being AI-generated. Many consecutive passages lack consistency and logical structure, the paper is poorly organized with specific parts scattered across the entire manuscript (for example, parts of the experimental setup are introduced as early as in the very second sentence of the introduction), and written using dense and convoluted language. Unfortunately, in my view, this significantly undermines the accessibility of this submission and, consequently, limits its relevance for the community.
W2. Limited contribution and not sufficient evaluation
Having said that the central theme of the submission is relevant, I do not find the contributions particularly novel given how the authors have approached the problem. Specifically, among the evaluated factors: holding the initial ranking fixed is a relatively standard experimental intervention, as well as comparisons between acquisition functions. Regarding the latter, the authors mostly restrict their comparison to greedy acquisition and ENS, omitting also very commonly employed acquisition functions, such as BALD or EPIG. The investigation of ENS itself also appears limited, as the authors consider a fixed one-observation lookahead without systematically studying the effect of the lookahead horizon. This leaves the feedback-update mechanism as the only component that I consider a substantial contribution, which, in my view, is insufficient given the aforementioned weaknesses.
Questions for authors
Q1. Could the authors provide the comparison with BALD and EPIG?
Q2. Fitness landscapes
Could the authors elaborate more on the underlying fitness landscapes? In particular, statistics of the number of substitutions, whether the mutational sites are fixed, what's the coverage of the variant space if the mutational sites are fixed, etc