acceptodds
Withdrawn from ICLR 2027

Beyond the Initial Ranking: How Feedback Shapes Active Search

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Active search aims to discover as many targets as possible within a limited measurement budget. Initial ranking quality captures only part of this task: observations change future choices, and planning acts through those changes. We separate initial predictions, feedback updates, and acquisition rules to evaluate their distinct roles in discovery. In a coherent probability model, we characterize when observing a fixed query has strictly positive value for a subsequent batch selection: no single continuation set is optimal across all possible outcomes. An exact correlated-group model expresses this value through dependence strength, selection margins, and the remaining capacity to exchange candidates. We then investigate these distinctions in retrospective protein-variant searches on Domainome and ProteinGym, with hits defined by within-pool top-5% fitness. In nested comparisons holding initial predictions and greedy acquisition fixed, adding sequence-position feedback improves discovery by approximately 0.4–1.0 hits at budgets of 12 and 24 measurements. A separate common-prior evaluation extends positive feedback gains to ridge and Gaussian-process updates on both corpora. Under its matched selected configurations, however, neither position-planning gains nor update-by-acquisition interactions pass multiplicity correction. Terminal-set comparisons distinguish changing query order, exchanging candidates, and improving yield. Together, these comparisons ask which relations carry useful feedback, whether candidate exchanges improve yield, and when planning adds value. Ranking quality, feedback utility, and planning value are distinct evaluation targets: a useful update can improve discovery without guaranteeing an additional benefit from planning.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Active Search, Sequential Experimental Design, Structured Priors, Nonmyopic Planning, Deep Mutational Scanning, Protein Variant Discovery

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_agvkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In this submission, the authors study how the performance of active-search methods in protein-variant screening depends on underlying components, particularly initial candidate rankings, feedback-update mechanisms, and acquisition strategies. The authors conduct their experiments on assays from ProteinGym and Domainone.

Critical strengths

I find the biggest strength of this paper to be its central theme: studying how different components of the search procedure influence the performance of active-search methods is in principle relevant.

Critical weaknesses

W1. Presentation quality

The central weakness of this submission is that it's very poorly written, to the extent that large portions of the text give the impression of being AI-generated. Many consecutive passages lack consistency and logical structure, the paper is poorly organized with specific parts scattered across the entire manuscript (for example, parts of the experimental setup are introduced as early as in the very second sentence of the introduction), and written using dense and convoluted language. Unfortunately, in my view, this significantly undermines the accessibility of this submission and, consequently, limits its relevance for the community.

W2. Limited contribution and not sufficient evaluation

Having said that the central theme of the submission is relevant, I do not find the contributions particularly novel given how the authors have approached the problem. Specifically, among the evaluated factors: holding the initial ranking fixed is a relatively standard experimental intervention, as well as comparisons between acquisition functions. Regarding the latter, the authors mostly restrict their comparison to greedy acquisition and ENS, omitting also very commonly employed acquisition functions, such as BALD or EPIG. The investigation of ENS itself also appears limited, as the authors consider a fixed one-observation lookahead without systematically studying the effect of the lookahead horizon. This leaves the feedback-update mechanism as the only component that I consider a substantial contribution, which, in my view, is insufficient given the aforementioned weaknesses.

Questions for authors

Q1. Could the authors provide the comparison with BALD and EPIG?

Q2. Fitness landscapes

Could the authors elaborate more on the underlying fitness landscapes? In particular, statistics of the number of substitutions, whether the mutational sites are fixed, what's the coverage of the variant space if the mutational sites are fixed, etc

Reviewer_jxDwOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors present an explainability framework for active search that aims to discover targets from a larger pool of candidates using a limited number of evaluation queries. The paper identifies 3 axes of evaluation for active search methods: quality of initial ranking, feedback updates and acquisition (also referred to as planning utility in the paper). The main-text experimental analysis is presented on the Domainome and ProteinGym datasets using greedy and ENS (Efficient Nonmyopic Active Search) active search methods.

Critical strengths
  1. The figures are visually quite nice.
  2. Explainability tools for active search are an interesting and useful research direction.
Critical weaknesses
  1. The paper lacks clarity and is overall difficult to follow. My main critique is that even after carefully reading the paper, it is difficult for me to summarise its main contributions. It is unclear in the paper what the existing methods are and what novel contributions are proposed by the authors. Additionally, in Sections 3 and 4, the mathematical notation is not precise. Here are a couple of examples:
  • line 117: rir_i is introduced for the first time here but never defined
  • line 132: it's unclear to me what position hh is in the context of the theory presented so far
  • line 149: introduces Tm(v;S)T_m(v;S), what is vv? It's never defined
  • Equation 4: uses r(⋅)r(\cdot) but everywhere else it's ri(⋅)r_i(\cdot), is ii subscript missing?
  • line 169: $ U$ is defined as the candidates remaining after the query, however, Section 3, line 116 already defined UU as a feedback update. Which one is it? On prose clarity, here is also an example from Section 8, line 473: "These results show why a good initial ranking, useful feedback and added lookahead value should be evaluated as distinct properties of a search system." But in the earlier sentence, it says: "matched planning tests remain inconclusive". Planning and lookahead are referring to the same thing, so first the paper states that planning tests are inconclusive and next that the results show that planning tests should be evaluated as a distinct property. However, the results don't support that, as they are inconclusive. Here are a few resources the authors might find helpful to improve the clarity of the manuscript: https://sebastianfarquhar.com/on-research/2024/11/04/how_to_write_ml_papers/, https://jsteinhardt.stat.berkeley.edu/blog/advice-for-authors, https://www.jakobfoerster.com/how-to-ml-paper.
  1. Main conclusions are unclear. I would like to see a clear statement about what the proposed framework adds. What are the main takeaways from the presented experiments? Section 8 talks about conclusions but as a reader I'm still unsure what should I take away from this work, should we change anything about the current active search methods? How should we be using these 3 evaluation criteria that the authors define?
Questions for authors
  1. In your own words, what is novel about your paper?
  2. What would you like the reader to take away from your paper?
  3. How would you summarise the conclusions from your experiments?
Reviewer_iDJ1OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies active search, which aims to find as many targets (i.e., candidates with a positive label) as possible within a fixed budget by querying candidates one at a time and observing each label before choosing the next candidate. The paper distinguishes three components: (i) initial estimates of how likely each candidate is to be a target, (ii) updates to these estimates after observing labels, and (iii) the rule for choosing the next candidate. The main experiments replay measurements from the Domainome and ProteinGym datasets: each search uses 12 or 24 queries among one protein's single amino acid variants to find hits, defined as the top 5% by measured performance (fitness) within each candidate pool. The results show that adding feedback from variants at the same sequence position helps find more targets on average than using feature similarity alone.

Critical strengths
  • The paper provides empirical evidence that position-based feedback can improve discovery across different update methods. The main comparison uses the same initial estimates and selection rule, helping isolate the benefit of the added feedback (Table 1). The benefit also appears with ridge regression and Gaussian process updates (Table 2).
Critical weaknesses
  • The writing makes the paper’s main argument difficult to follow across both the main text and the appendix. Although Section 5.1 and Table 8 summarize the comparisons, following Sections 5.4, 6, and 7 still requires repeated checks of which protocol supports each conclusion. Parts of the appendix also mix the final analysis with earlier attempts and corrections (Appendices A.2 and F.6). Appendix D mentions a ranking-stability certificate and a proposed gain law without defining them, while Appendix H reports diagnostic results before explaining the model’s training and evaluation split and decision rule. This organization requires the reader to reconstruct the final experimental setup and how the results support the claimed contribution.
  • I find the additional insight provided by the evaluation framework limited beyond its controlled empirical findings. The authors explicitly present the theory as a conceptual account, and the experiments do not establish whether its conditions explain or predict the observed gains (Section 4 and Appendix F.6). Although this limitation is acknowledged, I remain unconvinced that the supporting analyses provide a sufficiently substantial advance in understanding when and why feedback improves discovery.
Questions for authors
  • No further questions

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.