- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper formulates protein binder design as a compute-allocation problem, with the objective of finding structurally distinct in silico hits under a limited budget. The authors separate the discovery of structural clusters from the discovery of hits within those clusters. CGA estimates both processes online and uses them to decide when to generate more candidates, when to evaluate existing candidates, and which clusters to prioritize. This framework incorporates information accumulated throughout the campaign into subsequent decisions.
Critical strengths
What I find most valuable is the paper’s reconsideration of the optimization objective. A candidate’s value depends on what the campaign has already discovered. Once a cluster has produced a hit, another hit from the same cluster adds nothing to the objective. Maximizing sample-level hit probability alone does not fully capture this diminishing marginal value. This explains why a higher hit rate need not produce more unique hits. Incorporating this distinction directly into the decision objective identifies a meaningful way to improve a campaign: keep the generator unchanged and improve how compute is allocated.
The analysis of failure information is also insightful. Equation 5 relates the reduction in next-hit probability to the remaining uncertainty about cluster quality. If all clusters have the same hit probability, previous failures provide no reason to lower the priority of a particular cluster. When cluster qualities differ, repeated failures gradually change our assessment of a structural region. This analysis clarifies when moving away from an unsuccessful region is justified. It also connects the allocation policy to the statistical properties of a particular generator–target combination.
Critical weaknesses
My main reservation concerns the assumption that a cluster’s accessibility to the generator is independent of its quality. Under Appendix B.2, how frequently the generator visits a cluster provides no information about that cluster’s probability of producing a hit. Consequently, all unevaluated clusters receive the same initial quality prediction, regardless of how many candidates they contain.
This assumption can materially affect the decisions. If the generator’s preference for a structural region carries information about quality, CGA discards information that should influence evaluation priorities. For example, frequently generated and rarely generated clusters receive the same predicted hit prospects before any evaluation. Two aspects need to be distinguished here. Separating structural discovery from hit discovery is useful; fitting them as independent processes further restricts the phenomena the model can explain. I would like the authors to clarify the scope of this approximation.
The algorithmic contribution could also be stated more clearly. With a shared quality prior and equal evaluation costs, the evaluation order reduces to least-failures-first regardless of the particular prior. The fitted probability distributions mainly affect when to generate more candidates and the prediction of final yield. I would therefore like a fuller explanation of the generation–evaluation switching rule: how do the probability of discovering a new cluster, the hit prospects of existing candidates, and the costs of both operations jointly determine this choice? This deserves more attention than the connection to classical multi-armed bandit indices and would better bring out the method’s distinctive contribution.
Questions for authors
Why is cluster generation frequency treated as uninformative about cluster quality? Is this a simplifying approximation, or do the authors consider it a property of these generators? If it is a modeling approximation, please clarify which explanatory conclusions depend on it. A relationship between accessibility and quality would affect both evaluation priorities and the interpretation of the fitted scaling model.
6 comments
| Diversity | Cluster quality | Overall hit rate | Why the policy favors G or E | cG/cE break-even ratio |
|---|---|---|---|---|
| High | Heterogeneous | Low | G has the strongest advantage. Successful clusters hit early and retire, while repeated failures lower the prospects of the clusters still available. These uneven outcomes suggest that fresh clusters can be substantially better than the remaining pool. Higher discovery probability makes those fresh opportunities easier to reach, allowing generation to justify a larger cost. | 0.3602 |
| Low | Heterogeneous | Low | Same as 1, fresh clusters can offer substantially better hit prospects. However, most generations revisit existing clusters, so generation must be cheaper to justify the search. | 0.2269 |
| High | Homogeneous | Low | E is preferred unless generation is inexpensive. Repeated failures across every cluster lower expectations for fresh clusters too. A higher discovery probability makes fresh opportunities easier to reach, but their quality advantage must still justify the extra generation cost. | 0.0983 |
| High | Heterogeneous | High | G offers better prospects, but E still has useful candidates. High diversity makes fresh clusters more accessible, while the high overall hit rate supports promising fresh clusters. However, available clusters with low failure counts still offer useful candidates. Generation must be sufficiently cheap to justify searching for better prospects instead of evaluating those candidates directly. | 0.0672 |
| Low | Homogeneous | Low | Same as 3, but discovery is even less likely. | 0.0547 |
| Low | Heterogeneous | High | Same as 4, but discovery is even less likely. | 0.0334 |
| Low/High | Homogeneous | High | E wins at any positive generation cost. Observed successes support high hit prospects, and an available candidate already matches the fitted prospects of a fresh draw. Generation adds acquisition cost without improving hit prospects, regardless of diversity. Free generation only ties direct evaluation. | 0 |
Our algorithmic contribution is an adaptive allocation rule that combines estimated cluster discovery and quality with generation and evaluation costs to decide which operation to perform next. The rule responds to changing opportunities throughout the campaign and empirically improves unique hit yield under a fixed compute budget across a range of generator–target pairs.
I thank the authors for the detailed response and the additional analyses.
On the independence assumption. The quartile-level enrichment ratios are helpful and I appreciate the honesty in characterizing this as a simplifying approximation. However, my concern is less about whether the marginal correlation is strong and more about whether CGA's decisions are sensitive to even mild violations. The enrichment ratios show that the relationship varies in both direction and strength across campaigns, which means the approximation is better for some campaigns than others. A brief experiment showing how CGA's yield changes when a small positive or negative correlation between accessibility and quality is introduced would go a long way toward bounding the practical impact of this assumption. Without such a check, I cannot fully assess whether the method's improvements come partly from regimes where the assumption holds well and are offset in regimes where it does not.
On the G/E switching rule. The break-even cost ratio table and interactive visualization are genuinely clarifying. This analysis makes the algorithmic contribution much more concrete and I encourage the authors to incorporate it into the camera-ready version. I note that the table also reveals that generation is never preferred over evaluation when cluster quality is homogeneous and the hit rate is high (break-even ratio of 0). This is a useful boundary condition that deserves explicit discussion, as it helps practitioners understand when CGA offers the most value over simpler strategies.
Overall, this rebuttal strengthens my understanding of the method and partially addresses my concerns. The G/E analysis is a meaningful addition. The independence assumption remains an open limitation whose practical impact is not yet bounded. I maintain my score of weak accept.
Thank you for recognizing the value of our objective and perspective. We also appreciate your questions about the assumption and the generation-evaluation rule, which motivated additional analyses and helped us clarify our algorithmic contribution.
Why is cluster generation frequency treated as uninformative about cluster quality?
We treat independence as a simplifying approximation motivated by our observations from naive campaigns. Larger clusters show little enrichment overall, and the relationship between cluster size and quality varies across campaigns, as shown in this plot*. Within each campaign, we grouped clusters by size quartile and measured hit enrichment relative to the campaign’s overall hit rate. Thin gray lines show individual campaigns, while the thick colored line shows the mean enrichment with equal weight assigned to each campaign.
The largest size quartile has mean hit enrichment of 0.94×, 1.07×, and 1.03× for Complexa, Genie 3, and RFDiffusion3, respectively. But because the direction and strength of enrichment vary across generator-target pairs, the relationship has to be learned rather than using one fixed rule. We therefore chose not to model this dependence in the current formulation to avoid additional complexity. Studying this relationship across a larger target set, and understanding why it differs between generators and campaigns, is an interesting future direction that could inform the development of both better allocation policies and generators.
*As in Fig. 3, the comparison is restricted to generator-target pairs with at least 20 observed hits.
Fuller explanation of the G-E rule: how do the probability of discovering a new cluster, the hit prospects of existing candidates, and the costs of both operations jointly determine this choice?
Thank you for this constructive feedback. Generation and evaluation indices calculate expected additional unique hits per unit compute. Evaluating uses a candidate already available, whereas generating incurs an additional cost to obtain a potentially more promising candidate. This tradeoff is ultimately a function of the property of generator-target pair characterizing the campaign (e.g. Is this generator diverse? overall high hit rate?) and generation-to-evaluation cost ratio, cG/cE.
For each campaign state, there will be a break-even cost ratio at which the generation index equals the best available evaluation index. Generation is preferred below this ratio. A larger break-even ratio therefore means that generation remains worthwhile even at a higher cost.
We created an interactive visualization (Please download here) covering combinations characterizing campaigns: high or low diversity, heterogeneous or homogeneous cluster quality, and high or low overall hit rates. A slider varies cG/cE while holding the selected campaign state fixed, showing how the indices and preferred action change. The table below summarizes these examples, ordered by their break-even ratios. The thresholds and ranking are specific to the illustrated states in the visualization. We will include the visualization in the project page when we release the paper.
(continued in next response)