- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors present an evaluation benchmark for cyclic peptide structure prediction. The dataset is carefully curated to prevent data leakage. The metrics they select cover both accurate structure and chemical validity. However, the overall value of this work is limited because the whole pipeline is too trivial and is easy to carry out which provide limited insights. It is a standard data curation and folding model benchmark pipeline that has already been widely adopted in this field.
Critical strengths
- Compared to other methods, this paper has more recent datapoints from PDB database
- 4 folding models are evaluated under the same setting
Critical weaknesses
- Limited novelty and value to the community. This paper only presents a trivial evaluation set curation pipeline which is already a standardized procedure that is familiar with all structure prediction researchers.
- Limited insights are extracted from the benchmarked results.
- The second part, “PRETZEL-TWIST” is just adjusting input formats for different baseline models.
Questions for authors
Why this work uses RMSD instead of lDDT in the metrics, which is more common for structure prediction and more robust to local variance.
5 comments
Continued Rebuttal for Reviewer eh3Y
On the insights extracted from the benchmark
To our knowledge this is the first formal benchmark of co-folding engines on macrocyclic peptides, and it yields findings not available before it. Chemical validity is decorrelated from pose accuracy: AlphaFold3 is the weakest engine on RMSD yet best on bond-length validity at 8.1%, while Boltz-2 and Protenix-v2 sit at opposite ends of the accuracy ranking, so ranking models by pose accuracy carries essentially no information about whether they produce chemically correct molecules. Protenix-v2’s advantage is mechanistically localized rather than general: it leads on dihedrals and ring closure while ranking last on bond lengths. Difficulty is driven by input representation through an interaction effect, with targets combining a CCD modification and a bonded ligand succeeding at roughly half the rate of plain single-chain targets for every engine. And the C–N amide is the bond type reproduced within a narrow range, indicating these models have learned the one bond abundant in protein training data and generalize to none of the synthetic chemistries that define this modality. Overall, these results establish how difficult co-folding cyclic peptides remains and point to the specific directions co-folding engines should build on: geometric guidance extended to bond lengths, handling of jointly non-canonical and covalently bonded inputs, and coverage of synthetic bond chemistries beyond the backbone amide.
lDDT and RMSD
The two metrics answer different questions and are not directly comparable. lDDT is superposition-free: it enumerates all interatomic distance pairs within a cutoff in the reference, checks how many are reproduced in the prediction within a set of tolerance bands, and reports the average fraction passed, so it measures whether local environments are locally right. RMSD is superposition-dependent: it places the prediction in a common frame - in our case by rigid-body alignment on the reference binding pocket, following PXMeter - and reports the average atomic displacement, so it measures whether the molecule is in the right place in the right orientation and conformation. For macrocycles the latter is the quantity of interest, for two reasons. First, a ring can retain most of its contacts while being placed incorrectly. A macrocycle is far more rotationally symmetric in its presentation than a linear peptide, so rotating the ring about its own axis in the pocket, or shifting its register, still recovers a large share of the native contact set because similar side chains end up directed at similar receptor residues; the same holds for a ring that is locally in the right neighborhood but incorrectly closed. A contact-agreement score places these mid-to-high, whereas pocket-aligned RMSD is sensitive to which atom is where rather than merely to what is near what. Second, a distance-agreement score blends placement and local geometry into a single number, which is precisely what PRETZEL separates: pose accuracy is carried by pocket-aligned RMSD and local chemical correctness by PRETZEL-SALT’s bond-distance, angle, dihedral and ring-closure checks, measured directly against reference intervals. Since the contacting region can still be of interest, we also compute iRMSD as a supplementary metric. Since the reviewer is interested in lDDT-PLI, which is the more suitable variant for our case, we have computed it on all predictions:
-
AlphaFold3: Mean 0.354, Median 0.201, ≥0.5: 37.5%, ≥0.8: 18.9%
-
HighFold3: Mean 0.364, Median 0.234, ≥0.5: 37.2%, ≥0.8: 21.1%
-
Boltz-2: Mean 0.346, Median 0.172, ≥0.5: 36.5%, ≥0.8: 18.9%
-
Protenix-v2: Mean 0.406, Median 0.271, ≥0.5: 44.6%, ≥0.8: 27.4%
These results are consistent with the rankings produced by our RMSD evaluation, and the absolute values confirm the paper’s central finding that current engines struggle on macrocycles regardless of which pose metric is used.
Continued Rebuttal for Reviewer eh3Y
We therefore respectfully submit that no such pipeline exists for cyclic peptides to our knowledge, and that most work in this area has proceeded through the critical issue of data leakage and unorganized, hand-curated evaluation. If the reviewer is aware of an established pipeline that addresses these issues, we would genuinely welcome the pointer and will cite and compare against it in revision. Otherwise, we ask that the contribution be assessed on the novelty set out above. Thank you for your time.
======================================================================
[1] Morehead, Alex, et al. “Deep learning for protein–ligand docking: are we there yet?” arXiv preprint arXiv:2405.14108 (2024).
[2] Buttenschoen, Martin, Garrett M. Morris, and Charlotte M. Deane. “PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences.” Chemical Science 15.9 (2024): 3130–3139.
[3] Škrinjar, Peter, et al. “Evaluating generalization in protein–ligand cofolding methods.” Nature Structural & Molecular Biology (2026): 1–13.
[4] Cao, Sen, et al. “Accurate structure prediction of cyclic peptides containing unnatural amino acids using HighFold3.” Briefings in Bioinformatics 26.5 (2025): bbaf488.
[5] Xie, Xuezhi, et al. “CyclicBoltz1, fast and accurately predicting structures of cyclic peptides and complexes containing non-canonical amino acids using AlphaFold 3 Framework.” BioRxiv (2025): 2025-02.
[6] Zhao, Huifeng, et al. “Comprehensive evaluation of 10 docking programs on a diverse set of protein–cyclic peptide complexes.” Journal of Chemical Information and Modeling 64.6 (2024): 2112–2124.
Rebuttal for Reviewer eh3Y
Dear Reviewer,
We would like to respectfully refute the claim that this is a standard data curation and folding model benchmark pipeline already widely adopted in this field. The small-molecule field does have established pipelines, including PoseBench [1] for evaluation against docking baselines, and PoseBusters [2] and Runs N’ Poses [3] for evaluation purposes. We agree that this area has become standard practice for small molecules, but no equivalent has been established for cyclic peptides. As illustrated in the paper, HighFold3 [4] and CyclicBoltz1 [5] both use the same 63 monomer samples from the AfCycDesign dataset, all of which carry a data leakage issue, since the AlphaFold3 and Boltz-1 frameworks they build upon were trained on structures from that period. Beyond that, these are NMR structures composed exclusively of canonical residues. HighFold3’s complex evaluations use only 17 PDB entries, whose filters are not clearly illustrated and which are hand-curated. CPSet [6] was built to test docking of physics-based engines, and all of its data predates the cutoff date. We therefore do not have a clear, well-made dataset and evaluation methodology for cyclic peptides. We will walk through the novelty of each component of PRETZEL below, along with an lDDT/RMSD comparison.
Novelty of PRETZEL-SET
PRETZEL-SET provides a well-made pipeline for extracting cyclic peptides suitable for testing in co-folding engines. Unlike prior work, we created filters specific to cyclic peptides, and for the 285 entries in the current test set we inspected each one individually to confirm that it is a valid cyclic peptide. CPSet does not consider ring chemistry during curation and attends only to canonical versus non-canonical content, in addition to its leakage issue, while Riptides has no macrocycle stratification and no test of ring closure. PRETZEL-SET provides a clear filtering procedure together with macrocycle stratification that will be useful for future analysis.
Novelty of PRETZEL-TWIST
PRETZEL-TWIST addresses a failure mode that is easy to overlook: a configuration can be written in an incorrect form that still yields a confident, well-formed prediction of the wrong molecule. For example, Boltz-2’s “cyclic” flag closes a ring between the first and last residue of one contiguous polymer entity, which is correct only for a head-to-tail amide spanning exactly that entity. A disulfide-closed ring, an interior disulfide, a staple, or a peptide split across entities is accepted by the same flag and folded as something else, with nothing in the output announcing the substitution. The hazard differs per engine — Protenix-v2 needs an explicit bond record, AlphaFold3 discards polymer–polymer bonds before inference, HighFold3 uses positional-encoding flags — so hand-written configurations are wrong invisibly, and part of what published comparisons measure is the configuration procedure rather than the model. PRETZEL-TWIST detects from deposited coordinates, not from sequence, where the peptide must split across entities, whether each non-canonical residue is a CCD modification or a bonded ligand entity, every inter-residue bond typed at both endpoints as implicit or explicit, and the chemistry of each closure. Only then does a per-engine builder choose how to express it. Because the manifest is engine-independent and the builders are the sole engine-aware component, every model receives the same molecule. This gives a fair benchmarking ground on which each engine can be evaluated.
Novelty of PRETZEL-SALT
PRETZEL-SALT provides evaluation metrics specific to cyclic peptides. For accuracy we report RMSD, iRMSD and C$\alpha$-RMSD, alongside the percentage of predictions falling within 2 Å and 5 Å, following the convention in the literature for distinguishing accurate cyclic peptide predictions. The greater novelty lies in the chemical evaluation. PoseBusters and Runs N’ Poses are both built for small molecules, whose atom counts and molecular weights are far below those of cyclic peptides, so their criteria do not transfer directly and PB-Valid cannot be used properly here. We derive bond-length and bond-angle reference intervals for the bond types that actually occur in cyclic peptides, and we add cyclic-peptide-specific checks such as dihedrals and ring closure, which have no small-molecule analogue. To our knowledge, no existing cyclic peptide evaluation assesses chemical validity at this level.