acceptodds
Under review as a conference paper at ICLR 2027

PRETZEL: A Benchmark for Cyclic Peptide Structure Prediction

Abstract

Macrocyclic peptides have drawn growing attention as a therapeutic modality, and co-folding models are increasingly applied to predicting their structures. Yet published evaluations rely on small, loosely curated target sets that differ from paper to paper, so reported results are not comparable, and most report accuracy through pose RMSD alone, without checking whether the predicted molecule is chemically correct. We introduce PRETZEL (Prediction of Ring-Enclosed Topology: Zero-shot Evaluation of cycLic peptides), a benchmark of 285 curated cyclic peptide–protein complexes spanning three cyclization chemistries, all post-dating the training cutoff of every model evaluated. PRETZEL comprises three components: PRETZEL-SET, an evaluation set annotated by cyclization chemistry and chemical novelty; PRETZEL-TWIST, a specification layer that compiles a single canonical macrocycle description into each engine's native input format and verifies the returned structure; and PRETZEL-SALT, an evaluation suite pairing standard pose metrics with chemical validity checks for bond distance, bond angles, dihedrals, and ring closure. Using these, we evaluate four engines: AlphaFold3, Boltz-2, Protenix-v2, and HighFold3. Protenix-v2 achieves the best median RMSD at 9.05 Å, with 19.3% of predictions under 2 Å and 37.5% under 5 Å. Performance varies by cyclization chemistry: models handle head-to-tail macrocycles relatively well but degrade on stapled targets. Chemical validity is even weaker, with at most 8.1% of predictions having a fully valid set of inter-residue bonds, and no engine closes every macrocycle in more than 80.7% of its predictions, which indicates that pose accuracy alone is an insufficient criterion for these models. These failures share a cause: co-folding models place macrocycles without representing them, treating cyclization and non-canonical chemistry as annotations layered onto a residue sequence rather than as first-class structure. PRETZEL is released with all inputs, predictions, and evaluation code.

Primary area:
datasets and benchmarks
Keywords:
Cyclic Peptide, Co-folding Models, Drug Discovery, Dataset

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eh3YOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors present an evaluation benchmark for cyclic peptide structure prediction. The dataset is carefully curated to prevent data leakage. The metrics they select cover both accurate structure and chemical validity. However, the overall value of this work is limited because the whole pipeline is too trivial and is easy to carry out which provide limited insights. It is a standard data curation and folding model benchmark pipeline that has already been widely adopted in this field.

Critical strengths
  • Compared to other methods, this paper has more recent datapoints from PDB database
  • 4 folding models are evaluated under the same setting
Critical weaknesses
  • Limited novelty and value to the community. This paper only presents a trivial evaluation set curation pipeline which is already a standardized procedure that is familiar with all structure prediction researchers.
  • Limited insights are extracted from the benchmarked results.
  • The second part, “PRETZEL-TWIST” is just adjusting input formats for different baseline models.
Questions for authors

Why this work uses RMSD instead of lDDT in the metrics, which is more common for structure prediction and more robust to local variance.

5 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer eh3Y

Dear Reviewer,

We would like to respectfully refute the claim that this is a standard data curation and folding model benchmark pipeline already widely adopted in this field. The small-molecule field does have established pipelines, including PoseBench [1] for evaluation against docking baselines, and PoseBusters [2] and Runs N’ Poses [3] for evaluation purposes. We agree that this area has become standard practice for small molecules, but no equivalent has been established for cyclic peptides. As illustrated in the paper, HighFold3 [4] and CyclicBoltz1 [5] both use the same 63 monomer samples from the AfCycDesign dataset, all of which carry a data leakage issue, since the AlphaFold3 and Boltz-1 frameworks they build upon were trained on structures from that period. Beyond that, these are NMR structures composed exclusively of canonical residues. HighFold3’s complex evaluations use only 17 PDB entries, whose filters are not clearly illustrated and which are hand-curated. CPSet [6] was built to test docking of physics-based engines, and all of its data predates the cutoff date. We therefore do not have a clear, well-made dataset and evaluation methodology for cyclic peptides. We will walk through the novelty of each component of PRETZEL below, along with an lDDT/RMSD comparison.

Novelty of PRETZEL-SET

PRETZEL-SET provides a well-made pipeline for extracting cyclic peptides suitable for testing in co-folding engines. Unlike prior work, we created filters specific to cyclic peptides, and for the 285 entries in the current test set we inspected each one individually to confirm that it is a valid cyclic peptide. CPSet does not consider ring chemistry during curation and attends only to canonical versus non-canonical content, in addition to its leakage issue, while Riptides has no macrocycle stratification and no test of ring closure. PRETZEL-SET provides a clear filtering procedure together with macrocycle stratification that will be useful for future analysis.

Novelty of PRETZEL-TWIST

PRETZEL-TWIST addresses a failure mode that is easy to overlook: a configuration can be written in an incorrect form that still yields a confident, well-formed prediction of the wrong molecule. For example, Boltz-2’s “cyclic” flag closes a ring between the first and last residue of one contiguous polymer entity, which is correct only for a head-to-tail amide spanning exactly that entity. A disulfide-closed ring, an interior disulfide, a staple, or a peptide split across entities is accepted by the same flag and folded as something else, with nothing in the output announcing the substitution. The hazard differs per engine — Protenix-v2 needs an explicit bond record, AlphaFold3 discards polymer–polymer bonds before inference, HighFold3 uses positional-encoding flags — so hand-written configurations are wrong invisibly, and part of what published comparisons measure is the configuration procedure rather than the model. PRETZEL-TWIST detects from deposited coordinates, not from sequence, where the peptide must split across entities, whether each non-canonical residue is a CCD modification or a bonded ligand entity, every inter-residue bond typed at both endpoints as implicit or explicit, and the chemistry of each closure. Only then does a per-engine builder choose how to express it. Because the manifest is engine-independent and the builders are the sole engine-aware component, every model receives the same molecule. This gives a fair benchmarking ground on which each engine can be evaluated.

Novelty of PRETZEL-SALT

PRETZEL-SALT provides evaluation metrics specific to cyclic peptides. For accuracy we report RMSD, iRMSD and C$\alpha$-RMSD, alongside the percentage of predictions falling within 2 Å and 5 Å, following the convention in the literature for distinguishing accurate cyclic peptide predictions. The greater novelty lies in the chemical evaluation. PoseBusters and Runs N’ Poses are both built for small molecules, whose atom counts and molecular weights are far below those of cyclic peptides, so their criteria do not transfer directly and PB-Valid cannot be used properly here. We derive bond-length and bond-angle reference intervals for the bond types that actually occur in cyclic peptides, and we add cyclic-peptide-specific checks such as dihedrals and ring closure, which have no small-molecule analogue. To our knowledge, no existing cyclic peptide evaluation assesses chemical validity at this level.

AuthorsOpenReview ↗

Continued Rebuttal for Reviewer eh3Y

On the insights extracted from the benchmark

To our knowledge this is the first formal benchmark of co-folding engines on macrocyclic peptides, and it yields findings not available before it. Chemical validity is decorrelated from pose accuracy: AlphaFold3 is the weakest engine on RMSD yet best on bond-length validity at 8.1%, while Boltz-2 and Protenix-v2 sit at opposite ends of the accuracy ranking, so ranking models by pose accuracy carries essentially no information about whether they produce chemically correct molecules. Protenix-v2’s advantage is mechanistically localized rather than general: it leads on dihedrals and ring closure while ranking last on bond lengths. Difficulty is driven by input representation through an interaction effect, with targets combining a CCD modification and a bonded ligand succeeding at roughly half the rate of plain single-chain targets for every engine. And the C–N amide is the bond type reproduced within a narrow range, indicating these models have learned the one bond abundant in protein training data and generalize to none of the synthetic chemistries that define this modality. Overall, these results establish how difficult co-folding cyclic peptides remains and point to the specific directions co-folding engines should build on: geometric guidance extended to bond lengths, handling of jointly non-canonical and covalently bonded inputs, and coverage of synthetic bond chemistries beyond the backbone amide.

lDDT and RMSD

The two metrics answer different questions and are not directly comparable. lDDT is superposition-free: it enumerates all interatomic distance pairs within a cutoff in the reference, checks how many are reproduced in the prediction within a set of tolerance bands, and reports the average fraction passed, so it measures whether local environments are locally right. RMSD is superposition-dependent: it places the prediction in a common frame - in our case by rigid-body alignment on the reference binding pocket, following PXMeter - and reports the average atomic displacement, so it measures whether the molecule is in the right place in the right orientation and conformation. For macrocycles the latter is the quantity of interest, for two reasons. First, a ring can retain most of its contacts while being placed incorrectly. A macrocycle is far more rotationally symmetric in its presentation than a linear peptide, so rotating the ring about its own axis in the pocket, or shifting its register, still recovers a large share of the native contact set because similar side chains end up directed at similar receptor residues; the same holds for a ring that is locally in the right neighborhood but incorrectly closed. A contact-agreement score places these mid-to-high, whereas pocket-aligned RMSD is sensitive to which atom is where rather than merely to what is near what. Second, a distance-agreement score blends placement and local geometry into a single number, which is precisely what PRETZEL separates: pose accuracy is carried by pocket-aligned RMSD and local chemical correctness by PRETZEL-SALT’s bond-distance, angle, dihedral and ring-closure checks, measured directly against reference intervals. Since the contacting region can still be of interest, we also compute iRMSD as a supplementary metric. Since the reviewer is interested in lDDT-PLI, which is the more suitable variant for our case, we have computed it on all predictions:

  • AlphaFold3: Mean 0.354, Median 0.201, ≥0.5: 37.5%, ≥0.8: 18.9%

  • HighFold3: Mean 0.364, Median 0.234, ≥0.5: 37.2%, ≥0.8: 21.1%

  • Boltz-2: Mean 0.346, Median 0.172, ≥0.5: 36.5%, ≥0.8: 18.9%

  • Protenix-v2: Mean 0.406, Median 0.271, ≥0.5: 44.6%, ≥0.8: 27.4%

These results are consistent with the rankings produced by our RMSD evaluation, and the absolute values confirm the paper’s central finding that current engines struggle on macrocycles regardless of which pose metric is used.

AuthorsOpenReview ↗

Continued Rebuttal for Reviewer eh3Y

We therefore respectfully submit that no such pipeline exists for cyclic peptides to our knowledge, and that most work in this area has proceeded through the critical issue of data leakage and unorganized, hand-curated evaluation. If the reviewer is aware of an established pipeline that addresses these issues, we would genuinely welcome the pointer and will cite and compare against it in revision. Otherwise, we ask that the contribution be assessed on the novelty set out above. Thank you for your time.

======================================================================

[1] Morehead, Alex, et al. “Deep learning for protein–ligand docking: are we there yet?” arXiv preprint arXiv:2405.14108 (2024).

[2] Buttenschoen, Martin, Garrett M. Morris, and Charlotte M. Deane. “PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences.” Chemical Science 15.9 (2024): 3130–3139.

[3] Škrinjar, Peter, et al. “Evaluating generalization in protein–ligand cofolding methods.” Nature Structural & Molecular Biology (2026): 1–13.

[4] Cao, Sen, et al. “Accurate structure prediction of cyclic peptides containing unnatural amino acids using HighFold3.” Briefings in Bioinformatics 26.5 (2025): bbaf488.

[5] Xie, Xuezhi, et al. “CyclicBoltz1, fast and accurately predicting structures of cyclic peptides and complexes containing non-canonical amino acids using AlphaFold 3 Framework.” BioRxiv (2025): 2025-02.

[6] Zhao, Huifeng, et al. “Comprehensive evaluation of 10 docking programs on a diverse set of protein–cyclic peptide complexes.” Journal of Chemical Information and Modeling 64.6 (2024): 2112–2124.

Reviewer_TcJ9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PRETZEL, a unified benchmarking framework for cyclic peptide–protein complex structure prediction that integrates dataset construction, model input standardization, and chemical validity evaluation. Instead of developing a new structure prediction model, the authors introduce a benchmarking pipeline consisting of PRETZEL-SET, which contains 285 curated cyclic peptides, PRETZEL-TWIST, which converts chemical descriptions into model-specific input configurations, and PRETZEL-SALT, which evaluates both structural accuracy and chemical validity. These components are used to evaluate structural predictions through RMSD, bond lengths, bond angles, dihedrals, and ring closure across four existing models. The paper positions PRETZEL as a standardized benchmark for cyclic peptide–protein complex structure prediction and demonstrates the limitations of existing models in handling complex macrocycles, particularly that structural accuracy alone does not guarantee chemical validity.

Critical strengths
  1. The paper presents a unified benchmarking framework for cyclic peptide–protein complex structure prediction that goes beyond evaluating structural accuracy alone.

  2. The idea of systematically evaluating predictions using multiple complementary aspects (structural accuracy, chemical geometry, and ring closure) is conceptually clear and practically valuable.

  3. The framework is end-to-end, reasonably engineered, and evaluated across four representative structure prediction models, revealing limitations in predicting complex macrocycles that are often overlooked by conventional evaluation metrics.

Critical weaknesses
  1. The overall novelty and research contributions of PRETZEL are relatively limited. The three components mainly integrate existing data curation, input format conversion, and structural evaluation techniques, while related benchmarks such as CPSet and PepPCBench, as well as chemical validity assessment tools such as PoseBusters, have already explored similar problems. The paper does not clearly establish its substantive advances over these works or provide sufficiently novel scientific insights.

  2. The paper attributes prediction failures primarily to inadequate representations of cyclic peptide chemistry, yet the evaluated models differ in training data, architectures, and chemical input specifications. There is no systematic controlled analysis to disentangle these factors or account for differences in peptide length, non-canonical residue composition, and structural complexity across cyclization types.

  3. The experimental evaluation is limited to four models that largely follow similar all-atom structure prediction frameworks. The paper does not sufficiently evaluate other representative cyclic peptide prediction methods, such as CyclicBoltz1, NCPepFold, and HighFold2, even on compatible subsets, making it difficult to assess how comprehensively the benchmark reflects existing approaches.

  4. The paper presents chemical validity assessment as a central contribution, yet the robustness of its evaluation criteria is insufficiently examined. The strict all-bonds-must-pass criterion makes structure level validity rates sensitive to peptide length and bond count, while systematic per-bond validity analysis and threshold sensitivity studies are lacking. Moreover, the chirality check does not enforce stereochemical sign consistency, potentially overlooking important stereochemical errors.

Questions for authors
  1. How does the benchmark address potential training data leakage? Would the main findings remain consistent after excluding the 32 peptides with fingerprint-identical matches in the pre-cutoff PDB data?

  2. Can structural refinement improve chemical validity? Would energy minimization or chemistry-aware post-processing substantially improve chemical validity without compromising pose accuracy?

  3. What are the advantages of PRETZEL-SALT over existing evaluation tools? How does it differ from approaches such as PoseBusters, and what additional evaluation capabilities does it provide?

  4. Can the authors evaluate additional cyclic peptide structure prediction methods? Including methods such as CyclicBoltz1, NCPepFold, and HighFold2, at least on applicable subsets, would provide a more comprehensive comparison.

7 comments
AuthorsOpenReview ↗

Comment for Reviewer TcJ9

Dear Reviewer,

Thank you for your thorough review. We would like to address some of the concerns and questions that you had.

Novelty and research contributions of PRETZEL

We would like to illustrate novelty of PRETZEL across each components, and state how they contribute to the field.

  • PRETZEL-SET

Before turning to the quantitative comparison, we would like to emphasize the importance of the training cutoff. Many evaluations of co-folding models do not account for it and continue to use sets such as CPSet [1], where every target falls inside the models’ training windows. CPSet was assembled to evaluate ten physics-based docking programs, for which training cutoffs are irrelevant. When the same targets are reused to score learned co-folding models, however, the result measures recall together with prediction. This effect has been observed directly: Boltz-1 appears to recall the bound conformation of PDB 3ava, approximately cyclizing the peptide even with no target protein present [2]. The authors attribute this to memorization, because the behavior does not transfer to peptides outside the training set. This is in line with the low overall metrics we report, which are obtained on targets that no evaluated model could have seen.

To show that PRETZEL is meaningful beyond its temporal property, we ran a head-to-head comparison of PRETZEL and CPSet through a single pipeline applied identically to both. We note that PepPCBench [3] is about 97% linear peptides and so offers little macrocycle coverage to compare against. Riptides targets general peptides rather than macrocycles specifically; PRETZEL includes every Riptides entry that meets our curation criteria and covers a broader range of chemistries.

PRETZEL also has more chemical variation than CPSet. PRETZEL has 91.5 effective receptors from 285 chains, against 84.3 from 484 for CPSet, and has a median of 15 residues and 1,704 Da, against 11 residues and 1,212 Da for CPSet. PRETZEL also expands the non-canonical residue vocabulary by 62%.

  • PRETZEL-TWIST

PRETZEL-TWIST addresses a failure mode that is easy to overlook: a configuration can be written in an incorrect form that still yields a confident, well-formed prediction of the wrong molecule. For example, Boltz-2’s “cyclic” flag closes a ring between the first and last residue of one contiguous polymer entity, which is correct only for a head-to-tail amide spanning exactly that entity. A disulfide-closed ring, an interior disulfide, a staple, or a peptide split across entities is accepted by the same flag and folded as something else, with nothing in the output announcing the substitution. The hazard differs per engine - Protenix-v2 needs an explicit bond record, AlphaFold3 discards polymer–polymer bonds before inference, HighFold3 uses positional-encoding flags - so hand-written configurations are wrong invisibly, and part of what published comparisons measure is the configuration procedure rather than the model. PRETZEL-TWIST detects from deposited coordinates, not from sequence, where the peptide must split across entities, whether each non-canonical residue is a CCD modification or a bonded ligand entity, every inter-residue bond typed at both endpoints as implicit or explicit, and the chemistry of each closure. Only then does a per-engine builder choose how to express it. Because the manifest is engine-independent and the builders are the sole engine-aware component, every model receives the same molecule. This gives a fair benchmarking ground on which each engine can be evaluated.

  • PRETZEL-SALT

PRETZEL-SALT provides evaluation metrics specific to cyclic peptides. For accuracy we report RMSD, iRMSD and Cα-RMSD, alongside the percentage of predictions falling within 2 Å and 5 Å, following the convention in the literature for distinguishing accurate cyclic peptide predictions. The greater novelty lies in the chemical evaluation. PoseBusters [4] and Runs N’ Poses are both built for small molecules (PoseBusters only address 900 Da while our dataset has 92% about 900 Da), whose atom counts and molecular weights are far below those of cyclic peptides, so their criteria do not transfer directly and PB-Valid cannot be used properly here. We derive bond-length and bond-angle reference intervals for the bond types that actually occur in cyclic peptides, and we add cyclic-peptide-specific checks such as dihedrals and ring closure, which have no small-molecule analogue. To our knowledge, no existing cyclic peptide evaluation assesses chemical validity at this level.

AuthorsOpenReview ↗

Continued comment for Reviewer TcJ9

Additional model evaluation

We have performed additional model evaluation in NCPepFold. Note that HighFold2 is earlier version of HighFold3, and CyclicBoltz-1 is exactly same as cyclic flag of Boltz-1. When we ran NCPepFold, it was the worst amongst all cofolding models. Reasoning is clear: NCPepFold cannot naturally deal with disulfide cyclization and per-chain cyclization due to its intrinsic nature. NCPepFold got median RMSD of 17 Å, 1.1% below 2 Å, and 5.6% below 5 Å. Chemical validity is also suboptimal, with none passing bond angle, only 1 passing bond length, 4.9% passing dihedrals and 3.2% passing ring closure. Therefore, the overall evaluation trend does not change after including NCPepFold.

Data leakage address

PRETZEL-SET’s curation is purely temporal: we keep only entries released after 2023-06-01, and we deliberately did not filter on chemical similarity to pre-cutoff molecules, as stated in Section 3. A benchmark cannot guarantee a leakage-free evaluation for models whose training sets are not disclosed, since those sets differ beyond the shared cutoff date. We did, however, measure the overlap per entry and release it in the supplementary data: novelty_all.tsv gives every PRETZEL-SET chain’s ECFP4/MinMax similarity to its closest pre-cutoff match, so any evaluation can re-slice the set against whatever training window applies to the model under test. The 32 chains in question sit at the extreme of that distribution, at similarity 1.0.

Following the suggestion, we reran all four engines on the remaining 253 chains.

Table 1. Results on the benchmark set (N = 253 for all models). Best value per RMSD column in bold.

ModelTraining cutoffMedian RMSD (Å) ↓Mean RMSD (Å) ↓< 2 Å (%) ↑< 5 Å (%) ↑Bond distanceBond anglesDihedralRing closure
AlphaFold32021-09-3011.0916.1416.628.99.113.442.368.4
HighFold32021-09-3010.3415.7017.829.68.714.245.169.2
Boltz-22023-06-0111.0617.0010.730.44.77.137.259.7
Protenix-v22021-09-309.2415.2116.234.84.316.250.682.2

The main findings hold. All four engines still struggle, and more so: the median rises for every engine and none exceeds 18% under 2 Å. Chemical validity remains far weaker than pose accuracy. Difficulty still tracks cyclization chemistry and configuration family, with head-to-tail the most tractable class and stapled the hardest. Protenix-v2 keeps the best median and mean RMSD and the highest 5 Å success rate. At 2 Å, HighFold3, AlphaFold3 and Protenix-v2 again all outperform Boltz-2, and HighFold3 is highest at 45 of 253, against 42 and 41.

We continue to report the 285-chain set as primary, since it is an unbiased sample of the post-cutoff period, and the released per-chain scores let any reader reproduce the 253-chain analysis above.

AuthorsOpenReview ↗

Continued comment for Reviewer TcJ9

Structural refinement

We thank the reviewer for raising this as it is the natural next question. Prior work establishes that physics-based and physics-augmented methods produce more physically valid poses than purely learned ones [5], and recent engines move such corrections inside the sampler rather than applying them afterwards, as with Boltz-1x’s steering potential. Those results concern small molecules, however, whose internal geometry is fixed by construction, so they speak to bond and angle geometry and not to ring closure or stereochemistry - the two categories this question turns on for macrocycles. We therefore addressed it directly on our own predictions. Rather than run a single minimizer, whose outcome would depend on the force field and restraint choices made, we bound what any pose-preserving refinement could achieve: PRETZEL-SALT records the magnitude of every violation, so we forgive each one whose deviation falls within a given tolerance at zero cost in pose and recompute the per-chain verdict, which yields an upper bound that a real minimizer lands at or below. Tolerance is not free, since repairing a 30° angle error displaces the terminal atom of a 1.5 Å bond by roughly 0.8 Å, so we report the trade-off rather than a single number. Values below are the percentage of chains passing all four validity categories simultaneously, which is stricter than the per-category rates in the paper, given as AlphaFold3 / Boltz-2 / HighFold3 / Protenix-v2:

  • Published, no refinement: 5.6 / 2.1 / 6.0 / 2.8

  • Repairable within 0.10 Å / 10° (≲0.26 Å displacement): 6.3 / 2.5 / 8.1 / 6.3

  • Repairable within 0.30 Å / 30° (~0.8 Å displacement): 18.9 / 13.3 / 22.1 / 30.5

  • Repairable at any magnitude (unbounded): 50.5 / 55.8 / 55.1 / 72.3

The gain is marginal at tolerances that genuinely preserve the pose, and two ceilings survive any tolerance whatsoever. Forgiving every bond, angle and torsion violation at unbounded magnitude still caps validity at 50–72%, because 55–119 chains per model contain a broken macrocycle bond; granting perfect ring closure as well, stereochemistry alone caps it at 72–90%, since minimization moves atoms downhill in energy and cannot invert a stereocentre. The violations are also not refinement-scale to begin with - the median failing bond deviates by 0.12–0.25 Å and the median failing angle by 17–23°. Nor do these defects appear to be an artefact of incorrect placement: among chains each model already places within 2 Å ligand RMSD, validity is still only 9.3–12.7%, against 0.4–5.2% for the rest. We would therefore read refinement as a complement to cyclization-aware modelling rather than a substitute for it.

We agree that the empirical counterpart to this bound is worth having, and we intend to pursue it as future work: applying restrained energy minimization and chemistry-aware post-processing to the full prediction set, re-scoring with PRETZEL-SALT, and reporting the per-category improvement alongside the change in pose accuracy. Because PRETZEL-TWIST already supplies the intended cyclization chemistry for every target, it provides what such a pipeline would need in order to attempt ring closure rather than merely relax an open ring. We are grateful to the reviewer for prompting this analysis.

Systematic controlled analysis

We thank the reviewer for this criticism, which identified a real gap. Our comparisons across cyclization classes and configuration families were descriptive and did not adjust for covariates that differ between groups. We have now fitted a mixed-effects model of log RMSD on all 1,140 predictions (285 peptides × 4 engines), adjusting for peptide length, non-canonical residue count, and structural complexity, and tested engine-specific effects with exact within-peptide F tests. Full details are in Appendix Q.

  1. Difficulty is shared across engines. Peptide identity explains 80.5% of the variance in log RMSD, while engine identity explains 0.26%. The engines agree on which peptides are hard (pairwise Spearman ρ = 0.78–0.93). The engine × configuration-family interaction is not significant (p = 0.32), so the consistent ordering in Figure 4 is now supported by a formal test.

  2. The super-additivity claim is not supported, and we have removed it. The interaction between inline modifications and bonded ligands is not significant in any specification (p = 0.36–0.88). That family remains the hardest in the benchmark, but the data are consistent with two additive penalties.

  3. Covalent cross-link count is the main driver. Each additional cross-link raises RMSD by 46% (p = 0.0008). Once it is included, the stapled versus head-to-tail gap falls from +88% (p = 0.002) to +42% (p = 0.11). We therefore now describe cyclization class as a marker of covalent constraint burden, and no longer claim that it accounts for a substantial share of the variation. The effect of non-canonical residue count also does not survive adjustment (p = 0.27).

Reviewer_jZtkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PRETZEL, a benchmark framework for protein–cyclic peptide complex structure prediction. It comprises PRETZEL-SET, containing 285 cyclic peptide targets released after the stated training cutoffs of the evaluated models; PRETZEL-TWIST, which translates a shared molecular specification into engine-specific inputs while documenting complex chemical components and covalent connections; and PRETZEL-SALT, which evaluates both pose accuracy and inter-residue geometric validity. The authors evaluate AlphaFold3, HighFold3, Boltz-2, and Protenix-v2 with five generated samples per target, scoring each model’s highest-confidence prediction. Results show limited pose accuracy and chemical validity, particularly for stapled peptides and targets combining inline modifications with covalently attached ligands. Additional analyses examine junction-specific bond-length errors and the relationships between chemical validity, pose accuracy, and model confidence.

Critical strengths
  1. A unified and auditable evaluation workflow for chemically complex cyclic peptides. Non-canonical residues, synthetic linkers, and cross-entity covalent bonds create opportunities for opaque differences in model inputs. PRETZEL-TWIST documents this chemistry and engine-specific interface limitations through a shared molecular specification and dedicated input builders. Combined with the curated temporal evaluation set, this workflow improves reproducibility and helps distinguish input-configuration issues from prediction failures. This is the paper’s most convincing contribution.

  2. Useful analysis of failure modes beyond pose accuracy. Beyond RMSD, the paper examines inter-residue geometric validity, errors at canonical and non-canonical residue junctions, and the relationship between confidence and chemical validity. Checking connections defined in the reference avoids overlooking a required bond merely because it is broken in a prediction. These analyses are practically informative and demonstrate why pose accuracy or model confidence alone is insufficient to establish structural reliability.

Critical weaknesses
  1. The incremental contribution over existing datasets and evaluation tools lacks sufficient quantitative comparison. Table 1 includes CPSet and Riptides, but primarily compares target counts and temporal cutoffs. CPSet contains 493 protein–cyclic peptide complexes with substantial non-canonical peptide coverage, yet the paper does not quantify what PRETZEL adds in cyclization chemistry, chemical composition, or receptor/binding-pocket diversity. The broad characterization of existing resources as insufficiently large and diverse is therefore not adequately supported. Furthermore, the public Riptides pipeline already includes PoseBusters checks, making a blanket characterization of prior evaluations as pose-only inaccurate. SALT’s additional coverage and practical detection value also require comparison on the same predictions. For a benchmark paper, this evidence directly affects the assessment of its contribution and necessity.

  2. The explanations concerning input representation and model failure mechanisms exceed the evidence provided. Figure 4 compares different molecular subsets rather than alternative representations of the same molecules. These subsets may also differ in cyclization chemistry, molecular size, non-canonical residue count, and chemical novelty. The results establish an association between configuration category and accuracy, but do not isolate a causal effect of representation or a specific interaction between inline modifications and bonded ligands. The abstract and conclusion similarly attribute failures to how models represent macrocycles without controls that isolate this factor. Although chemical novelty and pocket diversity are characterized, they are not sufficiently connected to prediction performance, limiting the interpretation of generalization and failure mechanisms.

Questions for authors
  1. Clarification and validation of the dataset contribution: Could the authors quantitatively compare PRETZEL and CPSet in terms of cyclization types, non-canonical residue composition, peptide lengths, and receptor/binding-pocket diversity, including similarity or overlap at the peptide-chemistry and receptor levels? Beyond providing structures released after the evaluated models’ training cutoffs, what chemical or biological coverage does PRETZEL add?

  2. Clarification and validation of the evaluation contribution: Riptides already includes PoseBusters checks. What specific capabilities does SALT add relative to this pipeline? Could the authors report agreement and disagreement on the same predictions and provide examples of consequential errors additionally detected by SALT, establishing its incremental diagnostic value?

  3. Validation of the central interpretation: Is there a representation-controlled comparison that preserves molecular chemistry, or an analysis controlling for molecular size, cyclization chemistry, non-canonical residue count, and chemical novelty, supporting the interpretation of representation and interaction effects in Section 6.2.3? If not, should these conclusions be narrowed to observational associations across configuration categories?

3 comments
AuthorsOpenReview ↗

Comment for Reviewer jZtk

Dear Reviewer,

Thank you for your thoughtful and constructive feedback. We address each question below.

Clarification and validation of the dataset contribution

Before turning to the quantitative comparison, we would like to emphasize the importance of the training cutoff. Many evaluations of co-folding models do not account for it and continue to use sets such as CPSet, where every target falls inside the models’ training windows. CPSet was assembled to evaluate ten physics-based docking programs, for which training cutoffs are irrelevant. When the same targets are reused to score learned co-folding models, however, the result measures recall together with prediction. This effect has been observed directly: Boltz-1 appears to recall the bound conformation of PDB 3ava, approximately cyclizing the peptide even with no target protein present [1]. The authors attribute this to memorization, because the behavior does not transfer to peptides outside the training set. This is in line with the low overall metrics we report, which are obtained on targets that no evaluated model could have seen.

To show that PRETZEL is meaningful beyond its temporal property, we ran a head-to-head comparison of PRETZEL and CPSet [2] through a single pipeline applied identically to both. We note that PepPCBench [3] is about 97% linear peptides and so offers little macrocycle coverage to compare against. Riptides targets general peptides rather than macrocycles specifically; PRETZEL includes every Riptides entry that meets our curation criteria and covers a broader range of chemistries.

We compared the two sets on cyclization type, non-canonical residue composition, peptide length, and receptor and pocket diversity, together with direct overlap at the chemistry and receptor levels:

  • Overlap: The two sets share no PDB entries (0 of 280). Only 13.3% of PRETZEL chains match a CPSet chain in both pocket and peptide chemistry, so for 86.7% the combination does not occur in CPSet at all.

  • Size: PRETZEL has a median of 15 residues and 1,704 Da, against 11 residues and 1,212 Da for CPSet. It contains 85 chains above 2,000 Da against 25, so roughly a quarter of PRETZEL lies beyond CPSet’s entire mass range.

  • Non-canonical residues: PRETZEL expands the non-canonical residue vocabulary by 62%. Of its 244 residue types, 157 never occur in CPSet, affecting 47.7% of chains.

  • Receptors: 79.8% of UniProt accessions and 73.9% of Pfam families in PRETZEL are absent from CPSet, and the functional composition differs (13% peptidase against 27%).

  • Redundancy: PRETZEL has 91.5 effective receptors from 285 chains, against 84.3 from 484 for CPSet, and a median within-set nearest-neighbour Tanimoto of 0.719 against 0.899. At matched sample size, PRETZEL exceeds CPSet on peptide, receptor, and pocket cluster counts.

PRETZEL’s contribution is therefore not only temporal. On every axis the reviewer raises, it reaches chemistry and receptor space that CPSet does not, with substantially less internal redundancy.

Clarification and validation of the evaluation contribution

We would like to clarify the difference between PRETZEL-SALT and the use of PoseBusters in Riptides. Riptides, which is currently available as a technical report, applies PoseBusters as a filtering step and does not use it as an evaluation of chemical validity. To our knowledge, no existing cyclic peptide benchmark reports chemical validity as an evaluation metric.

More fundamentally, PoseBusters [4] is not designed for peptides. Its authors describe it as a tool for small-molecule docking poses, developed on drug-like ligands below roughly 900 Da, whereas 92% of PRETZEL peptides exceed 900 Da. Its energy check relies on force-field-relaxed conformer ensembles, which are not reliable for molecules of this size. PoseBusters' geometry checks apply to the bonds present in the predicted molecule, with no knowledge of the connections the reference requires.

PRETZEL-SALT is built for cyclic peptides. It checks every connection defined in the reference, uses reference intervals derived for the bond types that occur in cyclic peptides, and adds dihedral and ring-closure checks that have no small-molecule counterpart. The 9HGC_B AlphaFold3 prediction illustrates the consequence: the head-to-tail bond (GLY1 N – ASP15 C) is visibly broken, PRETZEL-SALT marks it as a failure, and PoseBusters marks it as a pass.

Because PoseBusters is being applied outside its stated domain, we do not consider aggregate agreement statistics between the two tools meaningful, and we instead highlight this representative failure case.

AuthorsOpenReview ↗

Continued comment for Reviewer jZtk

Validation of the central interpretation

Thank you for this suggestion. We have added a controlled experiment in Appendix O. We restricted the evaluation to a single configuration category, non-canonical single-chain peptides (N = 13), so that all targets share the same input configuration, and re-evaluated all four models on it (Table 12, Figure 10).

The results are consistent with the full benchmark. HighFold3 and Protenix-v2 remain slightly ahead on pose accuracy (median RMSD of 2.23 Å and 46.2% success at 2 Å, respectively). Successes are concentrated in head-to-tail cyclized peptides and failures in purely disulfide-bonded peptides, the same pattern seen across all 285 targets. All models are more accurate on this easier subset, but the overall trend does not change.

If the reviewer would like us to validate an additional category, we are happy to run that experiment and include it in the manuscript. We have additionally added statistical analysis in Appendix Q.


[1] Damjanovic, Jovan, and Rodrigo Ochoa. “Benchmarking AI-driven protocols for cyclic peptide conformer prediction and ranking.” RSC Medicinal Chemistry (2026).

[2] Zhai, Silong, et al. "Peppcbench is a comprehensive benchmarking framework for protein–peptide complex structure prediction." Journal of Chemical Information and Modeling 65.16 (2025): 8497-8513.

[3] Zhao, Huifeng, et al. "Comprehensive evaluation of 10 docking programs on a diverse set of protein–cyclic peptide complexes." Journal of Chemical Information and Modeling 64.6 (2024): 2112-2124

[4] Buttenschoen, Martin, Garrett M. Morris, and Charlotte M. Deane. "PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences." Chemical science 15.9 (2024): 3130-3139.

AuthorsOpenReview ↗

Response for Reviewer jZtk

Dear Reviewer jZtk,

Thank you again for your review.

We would greatly appreciate any feedback on our responses and would be happy to address any additional questions you may have.

We will remain responsive for the remainder of the rebuttal period.

Thank you.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

Dear AC and reviewers,

Thank you for your time and consideration. We summarize below what we did during the rebuttal period. Changes appear in the main text and Appendices O, P, and Q.

Summary of the work

PRETZEL is a benchmark framework for protein–cyclic peptide complex structure prediction, combining a curated target set (PRETZEL-SET), a shared molecular specification translated into engine-specific inputs (PRETZEL-TWIST), and an evaluation of both pose accuracy and chemical validity (PRETZEL-SALT). We evaluate AlphaFold3, HighFold3, Boltz-2, and Protenix-v2.

To our knowledge this is the first formal dedicated dataset and benchmark of co-folding models on macrocyclic peptides. Existing evaluations reuse targets inside the models' training windows, so they measure recall together with prediction. On 285 unseen targets all four engines struggle: only 2.1–6.0% of structures pass every validity criterion, and pose accuracy says little about chemical validity.

Novelty of each component

PRETZEL-SET. Every target postdates 2023-06-01, the latest cutoff among the evaluated engines. Curation criteria are cyclic-peptide-specific, all 285 entries were manually verified, and the set is stratified by macrocycle chemistry. The contribution is not only temporal: for 86.7% of chains the pocket–peptide chemistry combination does not occur in CPSet, peptides are larger (median 1,704 against 1,212 Da), and the non-canonical residue vocabulary grows by 62%.

PRETZEL-TWIST. An incorrect configuration can yield a confident, well-formed prediction of the wrong molecule: Boltz-2's cyclic flag is correct only for head-to-tail amides, yet accepts disulfide-closed and stapled rings silently. The hazard differs per engine, so hand-written configurations are wrong invisibly and published comparisons partly measure the configuration procedure. TWIST detects entity splits, bonded ligands, and every inter-residue bond from deposited coordinates rather than sequence. The manifest is engine-independent, so every model receives the same molecule.

PRETZEL-SALT. Small-molecule tools such as PoseBusters do not transfer to peptides of this size. SALT checks every connection the reference defines against intervals derived for the bond types occurring in cyclic peptides, and adds dihedral and ring-closure checks which are specific for cyclic peptides.

Reviewer response

  • Reviewer jZtk

Dataset contribution. A single pipeline compared PRETZEL and CPSet: no shared PDB entries, 157 of 244 residue types absent from CPSet, 79.8% of UniProt accessions absent, and lower internal redundancy.

Evaluation contribution. PoseBusters was developed on ligands below roughly 900 Da; 92% of our peptides exceed that. For 9HGC_B, SALT fails the broken head-to-tail bond that PoseBusters passes.

Central interpretation. Re-evaluating all four models within a single configuration category reproduces the full-benchmark trends (Appendix O), and we added a covariate-adjusted analysis (Appendix Q).

  • Reviewer TcJ9

Leakage. Rerunning all engines without the 32 fingerprint-identical chains leaves the findings unchanged on the remaining 253.

Controlled analysis. A mixed-effects model on 1,140 predictions attributes 80.5% of log-RMSD variance to peptide identity and 0.26% to engine identity. Covalent cross-link count is the main driver; we removed the unsupported super-additivity claim.

Refinement. Bounding any pose-preserving refinement shows marginal gains at tight tolerances, with broken macrocycle bonds capping validity at 50–72% even under unbounded repair.

Validity criteria. We added a per-bond forest plot (Appendix P) and a separate stereochemical sign criterion. AlphaFold3 and HighFold3 invert 44–48% of D centres, while all-criteria rates are unchanged.

Additional models. NCPepFold is the weakest engine on every metric (median RMSD 17 Å).

Following this exchange, TcJ9 confirmed that their concerns were fully addressed.

  • Reviewer eh3Y

Novelty. No equivalent pipeline exists for cyclic peptides; prior evaluations rely on small, hand-curated sets inside the models' training windows.

RMSD versus lDDT. We explain why pocket-aligned RMSD suits macrocycles and computed lDDT-PLI on all predictions; the rankings are consistent.

Training data. Three of the four engines share a 2021-09-30 cutoff, and Boltz-2's roughly 20 extra months do not make it lead, so training data does not explain the ranking. Engine-specific effects are claimed only where significant.

Cyclic Peptide Cofolding Models HighFold2 precedes HighFold3, and CyclicBoltz1's cyclic offset is evaluated through Boltz-2's flag. NCPepFold, now run on all 285 targets, does not change the conclusion.


We thank the reviewers again for the constructive feedback, which improved both the analysis and the claims we make from it. We hope our responses resolve any concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.