- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the novelty evaluation of protein structure generation, and questions whether low full-chain similarity really means novel folds. The authors propose DRR to evaluate domain-level retrieval, and RetFold, a training-free method that assembles retrieved protein domains. They also compare different TM-score definitions and show that the novelty results can change a lot depending on the evaluation protocol.
Critical strengths
-
The problem is interesting and worth studying. I agree that full-chain structural novelty does not necessarily mean a new fold, and this can be an issue in evaluating protein generation models.
-
RetFold is a nice and simple baseline to demonstrate the issue. The results clearly show that even structures assembled from known domains can get low full-chain retrieval scores. I think this is a useful observation.
-
The paper has quite extensive experiments, covering different generation models, retrieval metrics, and evaluation settings. I also appreciate that the authors investigate the limitations of their own proposed metric.
Critical weaknesses
-
The novelty of this work is somewhat limited. The idea that protein structures can be composed of known domains is not new. The proposed DRR is also mainly a combination of existing domain segmentation and structural retrieval tools. I think the most meaningful finding is the problem of full-chain qTM when searching against a domain database. But I am not sure how general this issue is for existing protein generation evaluations. It would be better to clarify which existing evaluation protocols actually have this problem.
-
I am not convinced that DRR can well support the main motivation. DRR only requires one domain to match, and the aligned-length TM-score can easily find local structural similarities. As shown in the paper, even after excluding the same topology, the retrieval rate is still 90.04% at alnTM ≥ 0.5. The results also drop a lot when using stricter criteria. So I don't think the high DRR numbers can tell much about how many known folds are actually reused. A more direct per-domain evaluation, with proper coverage constraints, would be more convincing.
-
The calibration and the conclusions about generated structures are still not sufficient. RetFold directly uses unmodified CATH domains, and searches against the same database, so this positive control seems quite easy. More challenging positive examples, such as different structures from the same known topology, would be useful. Also, in Table 3, RetFold achieves 100% retrieval at tTM ≥ 0.7, while the six unconditional generators only get 0–4.4%. This actually suggests a large difference between RetFold and these generators. However, the paper does not further investigate whether the unmatched structures are really novel folds or just distorted known folds. I think this is an important missing analysis.
-
The designability evaluation is not fully convincing. The paper mainly uses AF3 pLDDT to evaluate generated structures. However, high pLDDT does not mean that the predicted structure matches the originally generated backbone. I think self-consistency metrics, such as scTM or scRMSD, are needed. It would also be useful to re-evaluate the novelty results on structures that pass such checks. Otherwise, low retrieval rates might simply come from unrealistic generated structures.
-
There seem to be some inconsistencies in the experimental results. For example, in Table 6, some retrieval rates at qTM ≥ 0.7 are higher than those at qTM ≥ 0.5 (e.g., FrameDiff and ProtPardelle), which should not happen for the same set of scores. The timing numbers in Table 9 also do not seem consistent with the stated wall-time calculation. These results should be double-checked.
Questions for authors
-
How general is the issue identified in this paper? Can the authors clarify which existing protein generation works actually use full-chain qTM against a domain-level database like CATH S40? Would the same issue still be significant if using a full-chain structure database?
-
How should we interpret the high DRR results? Given the 90.04% retrieval rate in the topology-exclusion control, I am not sure how much the original DRR results can tell about known fold reuse. Can the authors provide a more direct domain-level evaluation, for example with reciprocal coverage, to better support the conclusions?
-
Can the authors further investigate the results in Table 3? RetFold achieves 100% retrieval at tTM ≥ 0.7, while the six unconditional generators only achieve 0–4.4%. This seems to suggest quite different behaviors between RetFold and learned generators. Can the authors explain this gap, and whether the unmatched structures are novel folds or just distorted known structures? It would also be useful to check this on structures passing backbone self-consistency evaluation, rather than only AF3 pLDDT.
-
Could the authors double-check some experimental results? In Table 6, the qTM ≥ 0.7 retrieval rates are higher than qTM ≥ 0.5 for several models, which seems incorrect. Also, the timing numbers in Table 9 do not seem consistent with the wall-time calculation described in the paper. Could the authors clarify these?
Dear reviewers,
Please be aware that little time remains for the author-reviewer discussion. As the deadline is approaching, we kindly request that you refrain from asking for additional experiments. Please summarize your final position concisely.
Sincerely, AC