- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Instead of running an expensive protein-folding model to completion, predict the final folding quality from its intermediate representations. Use those predictions to make antibody design and screening faster. They made 2 types of critics, condifence and structure cyclecritic: a lightweight surrogate of the original folding model's final quality assessment. 2 applications using these critics are shown: one for scree ning and one for hallucination based design
Critical strengths
- Empirical motivation is quite clear. and the overall structure is simple, and efficient
- Paper treats recycling as an adaptive computation problem, which was not done in past.
- Results are promising, with boosts clear
Critical weaknesses
- methodological novelty: main novelty is adaptive early stopping, rather than a fundamentally new confidence estimation method. also, the architecture uses deepset, and lacks novelty in the architecture itself.
- Hallucination improvements are partly explained by increased sampling For Mosaic, CycleCritic allows approximately 1.6× more designs under the same computational budget. Naturally, generating more candidates increases the chance of finding high-scoring designs and also acknowledges in Appendix H
- Figure 3 does not establish a universal property of AF3-like models.Figure 3 shows that Protenix-v2 confidence remains relatively stable even after substantial coordinate perturbations.The authors interpret this as evidence that confidence can be predicted from trunk representations.However, there is another interpretation:The original confidence head may be insufficiently sensitive to structural errors.If CycleCritic learns to imitate these confidence scores, it inherits their limitations.This is particularly concerning for antibody–antigen complexes, where confidently predicted but incorrect interfaces are a known problem.Also, the paper reports that ESMFold2 behaves differently: its confidence predictions are more dependent on coordinates.
- Early stopping sacrifices some promising candidates. Some of these may be interesting difficult-to-predict binders.The method is particularly vulnerable if its early predictions are systematically pessimistic about unusual but valid binding modes.
- It is unclear whether the approach generalizes to:Other models, such as OpenDDE or Boltz, completely novel target familie, other complex types, such as protein–peptide interactions. Or, cases where the original folding model confidently predicts incorrect interfaces.
- If 1-2 recycles is enough(early trunk representations) why didn't the authors include this setting for comparison?
Questions for authors
I think this paper has promising results so far, but needs some more clear experiments and supporting evidence
- Methodological novelty
The main methodological contribution appears to be adaptive early stopping using a learned confidence predictor, rather than a fundamentally new confidence estimation method. Given existing approaches that extract confidence information directly from trunk representations (e.g., distogram-based proxies), could the authors clarify what is fundamentally new about CycleCritic beyond applying a lightweight predictor for early stopping?
- Comparison with simpler fixed-recycling baselines
The screening experiments compare CycleCritic against Protenix-v2 with 4 and 10 recycles. However, Figure 2 suggests that early recycling already contains substantial information about final confidence. Could the authors include additional baselines using only 1 or 2 fixed recycles? A simple two-stage approach—performing 1–2 recycles for initial screening and fully refolding only promising candidates—would also be informative. These comparisons would help establish whether the improvements come from learned adaptive computation rather than simply reducing the recycling budget.
- Hallucination improvements versus increased sampling
For Mosaic, CycleCritic enables approximately 1.6× more designs under the same computational budget. Consequently, improvements in top-50 design quality may largely result from increased sampling rather than improved per-design quality. Although the authors partially address this in Appendix H (Figure H.1), could they further quantify the improvements attributable to the CycleCritic objective itself? In particular, an ablation isolating CycleCritic from the replacement of PAE-based losses with distogram-based losses would help clarify its contribution.
- Reliability of confidence prediction and coordinate sensitivity
Figure 3 shows that Protenix-v2 confidence scores remain highly correlated with their original values even after substantial coordinate perturbations. While the authors interpret this as evidence that confidence is largely determined by trunk representations, an alternative explanation is that the confidence head is insufficiently sensitive to structural errors. Moreover, ESMFold2 exhibits different, more coordinate-dependent behavior. Could the authors investigate whether CycleCritic inherits these limitations, particularly for confidently predicted but incorrect antibody–antigen interfaces? Evaluation against experimentally determined structures using metrics such as DockQ would help clarify this issue. If time is available, can you expand this to OpenDDE as well?
- False negatives introduced by early stopping
The proposed gating strategy retains 97.3% of highly reproducible designs (≥5/9 successful refolds), but only 89.2% of candidates that pass at least one reference refold. This raises concerns that early stopping may preferentially discard challenging candidates, including unusual but potentially valid binding modes. Could the authors analyze the characteristics of these false negatives? In particular, are they enriched among difficult targets, uncommon epitopes, or structurally diverse binding modes? Understanding this would help establish when adaptive early stopping might be detrimental.
- Generalization across models, targets, and complex types
The experiments primarily focus on Protenix-v2 and ESMFold2, with downstream evaluation on seven antigens. How well does CycleCritic generalize to other AF3-like architectures, such as Boltz or OpenDDE, and to previously unseen antigen families? Furthermore, could the approach extend to other interaction types, such as protein–peptide complexes? It would be useful to understand whether the critics require model-specific retraining and how their prediction accuracy changes under these distribution shifts.
5 comments
Questions
[Q1] The main methodological contribution appears to be adaptive early stopping using a learned confidence predictor, rather than a fundamentally new confidence estimation method. Given existing approaches that extract confidence information directly from trunk representations (e.g., distogram-based proxies), could the authors clarify what is fundamentally new about CycleCritic beyond applying a lightweight predictor for early stopping?
Distogram-based proxies are mainly used in the literature to avoid the diffusion rollout. The CycleCritic goes beyond this by also dynamically allocating the recycling budget. Compute costs are heavily dominated by the trunk. As such, CycleCritic can cut the main computational cost well beyond the diffusion rollout while the latter provides the ceiling for distogram proxy methods.
Applying diffusion only for promising candidates further ensures high fidelity confidence and structure prediction, resulting in being faster and higher precision than simply using distogram-based proxies, like previous work.
For hallucination losses, we show that CycleCritic outperforms distogram-based proxies, see Appendix F.4.2. Using the iptm-CC for hallucination yields over 50% higher increase in performance than the iptm proxy (+.0343 vs +.0229). Adding the signal of the ipsae-CC leads to a 90% increase(+.0435), showing the versatility of the approach.
[W6] If 1-2 recycles is enough(early trunk representations) why didn't the authors include this setting for comparison?
[Q2] The screening experiments compare CycleCritic against Protenix-v2 with 4 and 10 recycles. However, Figure 2 suggests that early recycling already contains substantial information about final confidence. Could the authors include additional baselines using only 1 or 2 fixed recycles?
We have added comparison to these baselines in the revised manuscript, see Appendix N and Figures N.1 & N.2. Briefly summarised, for 1&2 recycles the compute cost is dominated by diffusion and the performance drops significantly. This is why we used Protenix-mini, which optimised throughput while losing as little performance as possible. Protenix-mini is significantly faster than ptx1 and still has better agreement with ptx10.
Increasing the aggressiveness of the CycleCritic on the other hand allows the adaptive compute allocation to keep up with protenix-mini in throughput while being significantly closer to protenix10.
A simple two-stage approach—performing 1–2 recycles for initial screening and fully refolding only promising candidates—would also be informative. These comparisons would help establish whether the improvements come from learned adaptive computation rather than simply reducing the recycling budget.
As Figure N.1 shows, doing 1-2 cycles and diffusion on all samples for preselection is quite costly, so the proposed approach would definitely be slower than our deployed ptx10&CC. If we used a lightweight predictor to filter out promising candidates, then this would be an edge case of our approach. This approach then would not take advantage of the better signal at later stages and definitely perform worse. Note that in our experiments, based on preliminary experiments (compare e.g. Figure E.6), we only enabled the sCC-gates after the first recycle.
[Q3] For Mosaic, CycleCritic enables approximately 1.6× more designs under the same computational budget. Consequently, improvements in top-50 design quality may largely result from increased sampling rather than improved per-design quality. Although the authors partially address this in Appendix H (Figure H.1), could they further quantify the improvements attributable to the CycleCritic objective itself? In particular, an ablation isolating CycleCritic from the replacement of PAE-based losses with distogram-based losses would help clarify its contribution.
We have added population statistics in Appendix G (Tables G.2 and G.4) of the revised manuscript. In summary, when comparing the population statistics across the 50th, 25th, 10th, and 1st percentiles, we see that for Mosaic, CC is higher in 59% of cases (33/56) with a net average of +0.006. Here we are able to produce significantly more samples, while the generated designs are not worse, even slightly better. We have done a more extensive ablation study for ESMFold2 in Section F.4, but did not study the parameters for Mosaic in detail. While the experiments with Mosaic show that the CycleCritic and distogram based proxies can be drop in replacements, the parameters used are defaults and could most likely be optimized on both sides. Given the time-frame of the rebuttal period we are not able to provide these experiments in a fair fashion to the baseline, but would be happy to provide them for the camera ready version. Given that for ESMFOLD, the CycleCritic showed to be beneficial for hallucination-based design, we expect the same for Mosaic.
[Q4] Figure 3 shows that Protenix-v2 confidence scores remain highly correlated with their original values even after substantial coordinate perturbations. While the authors interpret this as evidence that confidence is largely determined by trunk representations, an alternative explanation is that the confidence head is insufficiently sensitive to structural errors. Moreover, ESMFold2 exhibits different, more coordinate-dependent behavior. Could the authors investigate whether CycleCritic inherits these limitations, particularly for confidently predicted but incorrect antibody–antigen interfaces? Evaluation against experimentally determined structures using metrics such as DockQ would help clarify this issue. If time is available, can you expand this to OpenDDE as well?
We agree that the limited sensitivity of Protenix-v2 confidence scores to large coordinate perturbations admits the alternative interpretation proposed by the reviewer. However, this interpretation is not mutually exclusive with our central observation that confidence is strongly determined by trunk representations. One can view the coordinate output of a co-folding model as a stochastic geometric realisation of the constraints encoded in the pair representation. The CC learns the expected confidence of the final structure from a given trunk state. Our ESMFold2 results demonstrate that confidence can be reliably predicted from trunk representations alone, despite the differing coordinate-dependence of its confidence head.
We agree that CycleCritic may inherit systematic biases in the original confidence scores, including overconfidence in incorrect antibody–antigen interfaces. CycleCritic is not intended to replace an experimentally calibrated structure-quality estimator. This is a separate modelling problem and beyond the scope of this work. However, for the screening application, this limitation is mitigated by the design of our method as the CycleCritic only rejects candidates early. Retained candidates still undergo full structure prediction and must satisfy the original confidence and structural self-consistency filters. Consequently, CycleCritic cannot independently accept an incorrect interface that the full model would reject. Its principal additional failure mode can only be premature rejection of otherwise acceptable candidates. Table E.2 shows that CycleCritic-gated screening achieves 0.96 precision compared with 0.95 for full Protenix-v2, at comparable recall (0.51 versus 0.52), while reducing computational cost.
Regarding DockQ evaluation with experimentally-determined structures, Figure A.1 in the manuscript provides evidence that intermediate recycling outputs contain information predictive of final DockQ for both Protenix-v2 and ESMFold2. Does the reviewer have a different experiment in mind?
In Appendix I (Figures I.1–I.2; see also Figure 3), we verified that early-cycle trunk representations in OpenDDE abag strongly predict final ipSAE (). While we are unable to run the coordinate-perturbation sensitivity experiment of Figure A.2 on OpenDDE in the rebuttal timeframe, we will be glad to include it in the camera-ready manuscript.
We thank the reviewer for their valuable feedback.
To address the reviewers’ comments, we have conducted extensive new experiments and added the following to the revised manuscript:
In the following, we will reply to the specific comments by the reviewer:
Weaknesses
We indeed do not claim major architectural advances nor do we propose a new confidence metric. Our contribution is to provide an algorithmic mechanism to evaluate the need for incremental recycles. This allows dynamic allocation of inference compute. One may loosely think about this as a form of reasoning in structure prediction models although the analogy is imperfect.
We agree that the improvements for mosaic are mostly due to the computational efficiency gains provided by CycleCritic (see Appendix G, Tables G.2 and G.4, in the revised manuscript). The presented experiments with Mosaic show that dropping the entire diffusion and confidence head and replacing them with CycleCritic and distogram-based proxies does not lead to a drop in performance while leading to a speed up of 60%. While speedups typically incur a tradeoff, Appendix G (Tables G.2 and G.4) shows that the performance is, on average, slightly better.
We agree with this valuable suggestion.
In the revised manuscript, we have added experiments that extend the CycleCritic to OpenDDE, Mol1, Boltz-1, and Boltz-2. Please see Appendices I and J in the updated manuscript. Briefly summarised, we find that for all models, single latent dimensions of the pair representation, correlate surprisingly well with the model’s own final ipSAE (Figure I.1). Boltz-2 is the weakest (ρ = 0.61), while ESMFold2, OpenDDE, Mol-1 and Protenix-v2 all reach ρ > 0.89. Although the designed VHHs and the natural SAbDab complexes show a slight shift in distribution, a single pooled dimension, an extremely simple potential CycleCritic, predicts both domains well. Tracking the Spearman correlation over recycling iterations (Figure I.2) shows that this signal emerges early during recycling. In Appendix I.1 (Table I.1) of the revised manuscript, we actually deploy the Analytic CycleCritic (see Section 4.1) to Mol-1 and find that with the thresholds fitted on Protenix-v2 it operates nearly as well.
Furthermore, we have added experiments demonstrating that CycleCritic directly extends to minibinder design in Appendix J (Table J.1 and Figures J.1–J.3). Interestingly, this works without any additional minibinder-specific training.