acceptodds
Under review as a conference paper at ICLR 2027

Is Retrieval All You Need? Assessment and Emergence of Novelty in Protein Structure Generation

Abstract

Protein backbone generation models are often credited with exploring novel fold space based solely on low full-chain similarity to known proteins, yet this cannot distinguish a genuinely new fold from a novel assembly of known structural units. To quantify the extent of this recombination, we introduce the Domain Retrieval Rate (DRR), defined as the fraction of generated backbones for which any constituent domain matches a known domain in CATH S40. Applied to eight backbone generation models spanning diffusion and flow-matching paradigms, DRR reveals that the vast majority of outputs contain domains already present in known fold space, even when their full-chain structures appear novel. The gap between domain-level and full-chain retrievability provides a direct measure of how much apparent novelty is combinatorial rather than structural. To calibrate what retrieval alone can achieve, we propose RetFold, a zero-training baseline that constructs backbones by retrieving CATH domains and refining inter-domain connections through geometry-based helix-linker optimization. RetFold attains designability and full-chain novelty on par with the strongest generators while running over two orders of magnitude faster on CPU alone. By saturating these metrics without learning, RetFold shows they cannot distinguish learned generation from retrieval and recombination. Together, DRR and RetFold establish that full-chain novelty is insufficient evidence of fold-level innovation, and future evaluation must decompose novelty across scales and calibrate against retrieval-based lower bounds.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Protein Backbone Generation, Structural Novelty Assessment, Domain Retrieval, Combinatorial Novelty

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zGnLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the novelty evaluation of protein structure generation, and questions whether low full-chain similarity really means novel folds. The authors propose DRR to evaluate domain-level retrieval, and RetFold, a training-free method that assembles retrieved protein domains. They also compare different TM-score definitions and show that the novelty results can change a lot depending on the evaluation protocol.

Critical strengths
  1. The problem is interesting and worth studying. I agree that full-chain structural novelty does not necessarily mean a new fold, and this can be an issue in evaluating protein generation models.

  2. RetFold is a nice and simple baseline to demonstrate the issue. The results clearly show that even structures assembled from known domains can get low full-chain retrieval scores. I think this is a useful observation.

  3. The paper has quite extensive experiments, covering different generation models, retrieval metrics, and evaluation settings. I also appreciate that the authors investigate the limitations of their own proposed metric.

Critical weaknesses
  1. The novelty of this work is somewhat limited. The idea that protein structures can be composed of known domains is not new. The proposed DRR is also mainly a combination of existing domain segmentation and structural retrieval tools. I think the most meaningful finding is the problem of full-chain qTM when searching against a domain database. But I am not sure how general this issue is for existing protein generation evaluations. It would be better to clarify which existing evaluation protocols actually have this problem.

  2. I am not convinced that DRR can well support the main motivation. DRR only requires one domain to match, and the aligned-length TM-score can easily find local structural similarities. As shown in the paper, even after excluding the same topology, the retrieval rate is still 90.04% at alnTM ≥ 0.5. The results also drop a lot when using stricter criteria. So I don't think the high DRR numbers can tell much about how many known folds are actually reused. A more direct per-domain evaluation, with proper coverage constraints, would be more convincing.

  3. The calibration and the conclusions about generated structures are still not sufficient. RetFold directly uses unmodified CATH domains, and searches against the same database, so this positive control seems quite easy. More challenging positive examples, such as different structures from the same known topology, would be useful. Also, in Table 3, RetFold achieves 100% retrieval at tTM ≥ 0.7, while the six unconditional generators only get 0–4.4%. This actually suggests a large difference between RetFold and these generators. However, the paper does not further investigate whether the unmatched structures are really novel folds or just distorted known folds. I think this is an important missing analysis.

  4. The designability evaluation is not fully convincing. The paper mainly uses AF3 pLDDT to evaluate generated structures. However, high pLDDT does not mean that the predicted structure matches the originally generated backbone. I think self-consistency metrics, such as scTM or scRMSD, are needed. It would also be useful to re-evaluate the novelty results on structures that pass such checks. Otherwise, low retrieval rates might simply come from unrealistic generated structures.

  5. There seem to be some inconsistencies in the experimental results. For example, in Table 6, some retrieval rates at qTM ≥ 0.7 are higher than those at qTM ≥ 0.5 (e.g., FrameDiff and ProtPardelle), which should not happen for the same set of scores. The timing numbers in Table 9 also do not seem consistent with the stated wall-time calculation. These results should be double-checked.

Questions for authors
  1. How general is the issue identified in this paper? Can the authors clarify which existing protein generation works actually use full-chain qTM against a domain-level database like CATH S40? Would the same issue still be significant if using a full-chain structure database?

  2. How should we interpret the high DRR results? Given the 90.04% retrieval rate in the topology-exclusion control, I am not sure how much the original DRR results can tell about known fold reuse. Can the authors provide a more direct domain-level evaluation, for example with reciprocal coverage, to better support the conclusions?

  3. Can the authors further investigate the results in Table 3? RetFold achieves 100% retrieval at tTM ≥ 0.7, while the six unconditional generators only achieve 0–4.4%. This seems to suggest quite different behaviors between RetFold and learned generators. Can the authors explain this gap, and whether the unmatched structures are novel folds or just distorted known structures? It would also be useful to check this on structures passing backbone self-consistency evaluation, rather than only AF3 pLDDT.

  4. Could the authors double-check some experimental results? In Table 6, the qTM ≥ 0.7 retrieval rates are higher than qTM ≥ 0.5 for several models, which seems incorrect. Also, the timing numbers in Table 9 do not seem consistent with the wall-time calculation described in the paper. Could the authors clarify these?

Reviewer_wh9AOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

in this submission, authors bring attention to an important question - how novel are de novo designs? The work argues that when the domain level resolution is used, the novelty drops. They also introduce a simple model that retrieves domains and adds linkers as a zero shot generator using retrieval which is able to perform well in full domain metrics. The work can bring the attention of the community to potential flaw of a commonly used metric.

Critical strengths

Relevant topic to the community. Meaningful contribution with a cheap calibration tool that reveals the flaw of a widely used metric. Hope the presence of this work in the conference will inspire others to approach critically to other common metrics used in generative modeling of biomolecules.

Critical weaknesses

The main weakness of the paper is that its message ends with the uncovered flaw, without much practical recipe forward. It would have been a positive contribution if the authors came up with multiple metrics that are more helpful (e.g. what type of novelty is really useful? how should we measure it? there are applications where a known fold is needed but affinity needs to be optimized etc. How does the designability-novelty front of the commonly-used models change in the light of this work? More insight would be helpful. Some of this can perhaps already be inferred in the designability related analysis in the supplementary.

In the same spirit it would have been useful to point at various positive ways to integrate RetFold learnings into making the generative modeling better.

Separately, this reviewer had a hard time to read the text due to the flow of LLM generated text. it would be helpful if at least the key parts of the paper were to be touched up for readability and flow.

Questions for authors

Please see comments in the weaknesses section.

Reviewer_BZLjOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how protein-backbone novelty estimates depend on structural granularity, TM-score normalization and the reference database. It introduces an any-domain retrieval statistic, compares eight generators, and constructs RetFold by assembling known CATH domains. Positive and topology-exclusion controls show that low query-normalized full-chain retrieval can miss known components, while aligned-length domain retrieval can be overly permissive. The paper consequently argues for calibrated, protocol-specific novelty reporting rather than interpreting either statistic as direct evidence of new folds.

Critical strengths
  1. The known-composition control gives the metric critique a concrete basis. RetFold contains complete reference domains by construction, yet its full-chain retrieval is only 20.0% under query-normalized TM, versus 100.0% under reference-normalized TM at the tested thresholds (Section 4.2 and Appendix G.2). Together with the length ceiling in Eq. (2), this shows why a low full-chain score against a domain database is insufficient evidence that the components are new.

  2. The paper tests the limitations of its own domain statistic. Table 2 reports 90.04% retrieval under aligned-length TM at 0.5 after excluding the query's topology. The any/all-domain, coverage, threshold, segmentation and quality-conditioned analyses help distinguish local alignability from complete-domain reuse. The low RetFold linker confidence is also reported explicitly, appropriately limiting its role to a calibration baseline rather than a validated protein-design system.

Critical weaknesses
  1. Table 6 violates the monotonicity required by its stated threshold comparison. For the same AF3-passed cohort and FC qTM score, ProtPardelle rises from 0.0% retrieval at 0.5 to 6.4% at 0.7; FrameDiff rises from 1.4% to 13.0% (p. 15). A stricter threshold must retrieve a subset when scores and the cohort are fixed. Appendix G.1's explanation of different alignments across normalization choices does not explain an increase within the same score column. Since this table supports the separation of granularity and normalization effects, it needs correction or a precise explanation of a changing protocol.

  2. The calibration does not support the categorical interpretation of the domain–full-chain gap. Section 4.1 correctly says that the topology-exclusion control differs from generated backbones and that comparing its retrieval rate with Delta is descriptive. The takeaway on p. 8 nevertheless attributes the gap to normalization conventions rather than structures. Comparing Delta with a domain-query false-positive rate does not establish that conclusion. Similarly, the 5.38% reference-normalized false-positive rate is measured on domains, not on the generated multi-domain chains in Table 3. The results support protocol sensitivity; a calibrated estimate of chain-level reuse would need a matched control or narrower claims.

Questions for authors
  1. Can you regenerate Table 6 from one fixed per-query hit table and explain the threshold increases for ProtPardelle and FrameDiff? Please state whether the correction changes the main normalization comparisons or any reported generator ordering.

  2. What evidence separates the contribution of structural composition from normalization to Delta on the generated-chain population? If the available controls cannot separate them, can the main takeaways be restricted to the demonstrated protocol dependence, with the domain-query false-positive rates kept distinct from chain-level claims?

Comments on the paper

1 comment

Dear reviewers,

Please be aware that little time remains for the author-reviewer discussion. As the deadline is approaching, we kindly request that you refrain from asking for additional experiments. Please summarize your final position concisely.

Sincerely, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.