acceptodds
Under review as a conference paper at ICLR 2027

From Plausibility to Reliability: Failure-Aware Closed-Loop Learning for Protein Design

Abstract

Protein design is increasingly able to produce large sets of computationally plausible candidates, while physical measurements remain the scarce resource that establishes reliability. We study folding stability as a controlled reliability endpoint and formulate measurement selection as a sequential learning problem under sparse physical feedback. We test whether an acquisition objective that combines model disagreement with predicted low stability can improve discovery of low-stability candidates under a fixed budget. PRIB Failure-Aware implements this objective as a scale-free rank aggregation of bootstrap-ensemble disagreement and failure relevance. In a frozen retrospective replay over 30,000 MGnify Stability sequences, each acquisition round reveals a batch of previously hidden experiment-derived targets and refits a three-member Ridge ensemble on frozen ESM-2 representations. Across five algorithmic seeds and cumulative budgets from 512 to 8,192 labels, PRIB achieves higher low-stability recall than Random and generic uncertainty sampling at every checkpoint. At 8,192 labels, recall is , compared with for Random and for Uncertainty; at 4,096 labels, PRIB already exceeds both baselines at 8,192 labels. On this fixed pool, the direction holds in all 25 seed-by-budget comparisons against each baseline. A target-provenance audit fixes the released convention before production replay, and predictive test evaluation is reserved for the final 8,192-label checkpoint. These results show that the acquisition objective can materially change which low-stability risks are resolved within a finite measurement budget and motivate experimental allocation as a first-class component of protein reliability workflows.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
protein design, protein reliability, active learning, closed-loop learning, experimental design, protein stability, uncertainty quantification, evidence allocation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_PCWaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

First of all, could the authors clarify what does appendix D mean? Please do explain the hash numbers given in the appendix. I am really shocked by the 6 lines of random values that appeared at the bottom of the paper.

This paper investigates how to allocate limited experimental measurements for identifying low-stability proteins. The authors propose PRIB Failure-Aware, an active learning strategy that combines ensemble disagreement with predicted low stability through rank-based aggregation. The method uses frozen ESM-2 representations and a bootstrap Ridge regression ensemble, which is updated after each round of acquiring experimental stability labels. The authors evaluate the approach through a retrospective closed-loop experiment on the MGnify Stability dataset. PRIB achieves substantially higher low-stability recall than random and uncertainty sampling across different measurement budgets. Overall, I think the paper addresses a practically relevant problem and presents encouraging results. However, I have significant concerns about the methodological novelty, the lack of important baselines, and whether the experiments sufficiently support the claimed contributions. I think the paper should be clearly rejected, even desk-rejected.

Critical strengths
  1. The paper addresses a meaningful problem in protein engineering. Experimental measurements are often expensive, and prioritizing which protein candidates to measure is an important research problem. I think exploring failure-aware acquisition strategies is a reasonable direction.

  2. The experimental setup is relatively clear and controlled. The authors use the same protein representations, regression architecture, and initial labeled sets across different acquisition strategies. The retrospective evaluation also separates model predictions from subsequently revealed experimental labels.

Critical weaknesses
  1. The main methodological contribution is not sufficiently supported by the experiments. The proposed acquisition function combines existing uncertainty and prediction-based signals, but the evaluation only compares against random and generic uncertainty sampling. Since the primary metric directly rewards discovering low-stability proteins, I think a failure-relevance-only baseline is essential to establish the benefit of combining these signals. Moreover, the paper does not compare against a static acquisition strategy, making it unclear whether sequential model updating provides any additional value. I am not fully convinced that the current experiments adequately demonstrate the proposed method's contributions.

  2. The experimental evaluation and overall presentation are relatively limited for ICLR. The experiments rely on a single retrospective dataset and fixed candidate pool, without sufficient evidence of generalization or improved reliability assessment on unseen proteins. I also think the method design is relatively simple, while the related work discussion, experimental analysis, and figure presentation could be substantially improved. Overall, the paper would benefit from stronger experimental validation and a clearer presentation of its technical contributions.

Questions for authors
  1. Could the authors compare PRIB with a failure-relevance-only acquisition strategy? In particular, does incorporating uncertainty provide a meaningful improvement over simply selecting proteins with the lowest predicted stability?

  2. Could the authors compare the proposed sequential acquisition with a static strategy that does not update the model after each measurement round? I think this would help clarify the contribution of closed-loop learning.

  3. Could the authors provide comparisons with stronger target-aware acquisition methods and further investigate the contribution of the uncertainty component?

  4. Could the authors evaluate whether PRIB improves failure detection on held-out proteins and whether the results generalize across different candidate pools or protein families?

Reviewer_Do7zOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes an acquisition function to find the least stable protein among a pre-defined pool of proteins. The surrogate is an ensemble of 3 ridge regression models on ESM2 embeddings. Uncertainty estimates are given by the standard deviation of the 3 ridge regression models predictions. The acquisition function ranks the predictions and the uncertainties from the surrogate. The ranks are normalized and then added to provide a combined ranking of the candidates. The top ranking candidates are selected. The method is compared to (1) random sampling and (2) selecting the candidates with the largest uncertainty estimates. They show that their method finds less stable variants than the two baselines.

Critical strengths

The paper propose an acquisition function to address a relevant problem of searching for better proteins.

Critical weaknesses

There is a lack of ablations: For example it would be nice to see a comparison to an acquisition function just using R_t(F_t;x). Also a comparison to other standard acquisition functions such as EI or UCB would help.

Lack of novelty: The paper proposes an acquisition function using rankings of predictions and uncertainties.

Questions for authors

Did you compare to some of the ablations I mentioned above?

Why did you optimize for negative and not positive stability?

Equation 1 seem to have an error?

Do you have any theoretical or intuitive arguments for why/when the proposed acquisition function should be better than standard acquisition functions?

Reviewer_cVmKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper formulates an active-learning framework for a protein folding stability problem. The method (PRIB Failure-Aware) proposed by the authors selects a new batch of candidate protein sequences to be evaluated using the oracle (MGnify Stability dataset). This process of selection and evaluation is done sequentially until the oracle query budget is reached. The authors compare the method to two benchmarks: Random and Uncertainty, and find that their method proposes sequences with higher recall of low-stability proteins.

Critical strengths
  1. The application of protein design and protein stability is very interesting, and it's an important problem to tackle.
  2. Active learning is a good framework to use for the sequential select->make->test cycle (or design->make->test).
Critical weaknesses
  1. It's very unclear what the novel contributions of the paper are. The budgeted evidence allocation in Section 3.1 is very similar to a standard active learning application in protein design or drug discovery. For example, [1] is a review paper on AL for drug discovery, and the sequential design presented as novel by the authors is very similar to existing works. The other novel contribution claimed is the acquisition function of PRIB. However, the authors only compare their proposed acquisition function to 2 baselines: Random and Uncertainty, and they omit many commonly used acquisition functions like Upper Confidence Bound (UCB), Probability of Improvement (PI) or Expected Improvement (EI). UCB also balances between the μt\mu_t and UtU_t (using the same notation as the methods section), and I would expect it to perform similarly to the acquisition proposed in the paper.

  2. The experimental evaluation should be more thorough. The current experiments are missing baselines (mentioned above). The experiments also report mainly the recall value of the low-stability proteins. Precision should also be reported. Additionally, while PRIB has higher recall than the 2 baselines, the Spearman correlation is comparable to the baselines (and actually worse than baselines on the validation set, which seems surprising). The proposed method could also be evaluated on other tasks too; while the paper focuses on protein stability, the method is somewhat task agnostic so it would be great to expand the evaluation to other evaluation tasks.

[1] Wang, L., Zhou, Z., Yang, X., Shi, S., Zeng, X., & Cao, D. (2024). The present state and challenges of active learning in drug discovery. Drug Discovery Today, 29(6), 103985. https://doi.org/10.1016/j.drudis.2024.103985

Questions for authors
  1. Could you report your experimental results with the additional baselines of UCB, PI and EI?
  2. Could you also report the precision values in the experiments (not just recall)?
  3. Could you clearly present your novel contributions (better contextualised in the field of active learning and existing acquisition functions)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.