- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
First of all, could the authors clarify what does appendix D mean? Please do explain the hash numbers given in the appendix. I am really shocked by the 6 lines of random values that appeared at the bottom of the paper.
This paper investigates how to allocate limited experimental measurements for identifying low-stability proteins. The authors propose PRIB Failure-Aware, an active learning strategy that combines ensemble disagreement with predicted low stability through rank-based aggregation. The method uses frozen ESM-2 representations and a bootstrap Ridge regression ensemble, which is updated after each round of acquiring experimental stability labels. The authors evaluate the approach through a retrospective closed-loop experiment on the MGnify Stability dataset. PRIB achieves substantially higher low-stability recall than random and uncertainty sampling across different measurement budgets. Overall, I think the paper addresses a practically relevant problem and presents encouraging results. However, I have significant concerns about the methodological novelty, the lack of important baselines, and whether the experiments sufficiently support the claimed contributions. I think the paper should be clearly rejected, even desk-rejected.
Critical strengths
-
The paper addresses a meaningful problem in protein engineering. Experimental measurements are often expensive, and prioritizing which protein candidates to measure is an important research problem. I think exploring failure-aware acquisition strategies is a reasonable direction.
-
The experimental setup is relatively clear and controlled. The authors use the same protein representations, regression architecture, and initial labeled sets across different acquisition strategies. The retrospective evaluation also separates model predictions from subsequently revealed experimental labels.
Critical weaknesses
-
The main methodological contribution is not sufficiently supported by the experiments. The proposed acquisition function combines existing uncertainty and prediction-based signals, but the evaluation only compares against random and generic uncertainty sampling. Since the primary metric directly rewards discovering low-stability proteins, I think a failure-relevance-only baseline is essential to establish the benefit of combining these signals. Moreover, the paper does not compare against a static acquisition strategy, making it unclear whether sequential model updating provides any additional value. I am not fully convinced that the current experiments adequately demonstrate the proposed method's contributions.
-
The experimental evaluation and overall presentation are relatively limited for ICLR. The experiments rely on a single retrospective dataset and fixed candidate pool, without sufficient evidence of generalization or improved reliability assessment on unseen proteins. I also think the method design is relatively simple, while the related work discussion, experimental analysis, and figure presentation could be substantially improved. Overall, the paper would benefit from stronger experimental validation and a clearer presentation of its technical contributions.
Questions for authors
-
Could the authors compare PRIB with a failure-relevance-only acquisition strategy? In particular, does incorporating uncertainty provide a meaningful improvement over simply selecting proteins with the lowest predicted stability?
-
Could the authors compare the proposed sequential acquisition with a static strategy that does not update the model after each measurement round? I think this would help clarify the contribution of closed-loop learning.
-
Could the authors provide comparisons with stronger target-aware acquisition methods and further investigate the contribution of the uncertainty component?
-
Could the authors evaluate whether PRIB improves failure detection on held-out proteins and whether the results generalize across different candidate pools or protein families?