acceptodds
Under review as a conference paper at ICLR 2027

CatGeoBench: Mechanism-Guided Catalytic Geometry for Enzyme–Ligand Co-Folding

Abstract

Existing benchmarks for protein–ligand structure prediction and docking rely on crystal structures and evaluate binding geometry and physical plausibility, but do not assess whether predicted enzyme complexes satisfy the geometric prerequisites of catalysis. We present CatGeoBench, a benchmark for catalytic geometric compatibility based on mechanisms from M-CSA. It includes a core set built directly from M-CSA and an extended set that covers related enzymes with alternative substrates. For each mechanism we build electron-flow chains and extract reacting atom pairs as the intermolecular bond-forming atom pairs collected up to the first intermolecular structure-changing electron-flow event. CatGeoBench scores predictions with mechanism-derived distance criteria based on scaled van der Waals radii of these pairs, rather than by comparison to crystal ligand poses. The core set comprises 239 entries with 320 reacting atom pairs, and the extended set comprises 123 entries with 161 pairs. Each entry provides a curated enzyme functional unit, validated substrate and cofactor inputs, explicit atom mappings for downstream structure evaluation, and the associated distance thresholds. We evaluate representative co-folding models under a unified protocol. Models achieve high success under the distance-only criterion on the core set but degrade on the extended set, where core performance does not reliably indicate success on related enzymes with alternative substrates.

Primary area:
datasets and benchmarks
Keywords:
enzyme benchmark, co-folding, catalytic geometry, reaction mechanism

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ET2pOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a binding pose prediction benchmark for enzyme systems. Since enzyme–substrate complexes are often not static, experimentally determined crystal structures are limited. Rather than evaluating how well methods reproduce entire binding poses, the proposed benchmark evaluates how they capture interactions between pre-defined reacting atom pairs by measuring the distances between them. The authors first validate the distance criterion -- 1.4 x (r1 + r2) --against resolved crystal structures. Then, co-folding models are evaluated on enzyme–ligand systems with known reaction centers.

Critical strengths

The benchmark addresses the scarcity of experimentally resolved enzyme–substrate complexes by focusing on the proximity of reacting atom pairs rather than the reproduction of complete binding poses. This reaction-centered evaluation provides a targeted way to assess predicted enzyme–ligand structures. The authors also validate the proposed distance criterion against experimentally resolved crystal structures before applying it to co-folding predictions.

Critical weaknesses
  1. As the authors acknowledge, the current criteria consider distances between reacting atoms but omit their relative orientation. Molecular orientation is an important determinant of chemical reactivity: even when reacting atoms are sufficiently close, an unfavorable orientation may prevent the complex from satisfying the geometric requirements for reaching the transition state. Consequently, distance alone provides an incomplete assessment of whether a predicted binding pose is compatible with catalysis. I recommend incorporating reaction-specific angular criteria, at least for representative reaction types, and reporting their effects on pass rates and model rankings. The angular criteria implemented in PLIP [1] for non-covalent interactions could provide a useful implementation reference.
  2. The evaluation is limited to co-folding models. Because the benchmark assesses the geometric plausibility of predicted enzyme–ligand complexes, physics-based docking methods would provide important baselines. Moreover, Boltz2ESI [2] reports that EZSpecificity [3], which uses AF2 and Vina-based rigid docking, performs competitively with the co-folding-based Boltz2ESI approach. Including physics-based docking baselines alongside pocket-conditioned co-folding models would therefore strengthen the comparative evaluation.

References

  1. https://github.com/pharmai/plip
  2. Cheng, Xiwei, et al. "Boltz2ESI: Accurate Enzyme Specificity Prediction with Co-folding Foundation Model." ICLR 2026 GEM
  3. Cui, Haiyang, et al. "Enzyme specificity prediction using cross-attention graph neural networks." Nature 647.8090 (2025): 639-647.
Questions for authors

The main issues that could affect my assessment are already outlined in the Critical Weaknesses section. I am willing to raise the score based on the author's response.

3 comments

We thank the reviewer for the constructive comments. We have added the corresponding analyses and experiments to the revision (new appendix material is marked in blue), as summarized below.

Weakness 1

We agree that NAC geometry depends on orientation as well as distance. We therefore classified reacting atom pairs in the core set by reaction type (Appendix B.5, Figure 14a). The three most frequent classes are proton transfer (55%), SN2-like substitution (20%), and nucleophilic addition (17%). Since hydrogen coordinates are often uncertain, we do not impose angular constraints on proton-transfer pairs. For SN2-like substitution and nucleophilic addition, we introduce angular criteria based on their expected reaction geometries, using broader ranges of 150°–180° and 90°–120°, respectively, to accommodate deviations in single predicted structures. As shown in Appendix B.5, Table 7, these additional constraints lower absolute success rates, but model rankings remain largely stable. The angle distributions of distance-passing Protenix-v2 Top-1 predictions also peak near the expected geometries (Appendix B.5, Figure 14b,c).

Weakness 2

We agree that including a classical docking baseline strengthens the evaluation. We therefore evaluated AutoDock Vina on 75 core-set entries with a single substrate and no metal-dependent catalytic centers, for which Vina is applicable. We used the top-ranked Protenix-v2 protein structure as the receptor and centered the docking box on the predicted substrate position. Success was assessed using the same reacting-atom distance criterion as for the co-folding models. As shown in Appendix B.6, Table 8, co-folding models generally perform better without the PB-Valid filter. With this filter, Vina becomes competitive at Top-1, while Boltz-1x achieves the strongest Top-5 performance.

We thank the reviewer again for the careful reading and helpful suggestions.

Thank you for your response and additional experiments. I'll increase my score to 3.

Thank you for your positive response. We appreciate your time and thoughtful feedback!

Reviewer_U6EDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CatGeoBench asks whether co-folding models produce enzyme–substrate–cofactor complexes that satisfy the geometric requirements of catalysis, rather than whether they reproduce a crystallographic ligand pose. From M-CSA's curly-arrow annotations the authors build electron-flow chains for the first mechanistic step, stop at the first intermolecular structure-changing event, and keep the intermolecular bond-forming atom pairs; each pair is scored by whether its distance satisfies d≤1.4(ri+rj)d \le 1.4(r_i + r_j), with rr taken from van der Waals radii, so the mechanism determines which atoms are measured while the cutoff depends only on their element types. Filtering reduces 1003 M-CSA entries to a core set of 239, with 320 reacting pairs. A separately constructed extended set of 123 entries with 161 pairs is drawn from related enzymes acting on alternative substrates. The cutoff is checked against 69 pairs measurable in crystal structures, and eight co-folding models are then evaluated under one protocol at pair and complex level, with and without an additional PoseBusters (PB-Valid) requirement. Two findings are reported: success is 67–80% under the distance criterion alone but falls to 29–50% once PB-Valid is also required, with the model ranking changing; and every model scores lower on the extended set than on the matched core subset.

Critical strengths

1. Mechanism annotations repurposed as geometric evaluation targets.

M-CSA's curly-arrow annotations have been mined programmatically before (e.g. EzMechanism, which converts them into reaction rules to propose mechanisms), but using them to derive which interatomic distances a predicted structure must satisfy is, as far as I am aware, new. This is the one component of the paper that could not have come from a docking dataset — though the contribution is limited to selecting which atoms are measured, not how close they must be.

2. Eight co-folding models evaluated under one protocol, with physical validity reordering them.

Requiring PoseBusters validity alongside the distance criterion changes the ranking, across ~4×10⁴ predictions. This is directly actionable for the community, and the reordering is driven by PoseBusters, an external and already-validated check.

Critical weaknesses

1. The 1.4 coefficient is not derived from anything in the paper, and is never tested.

§2 defines the near-attack conformation, which the paper gives as its rationale, as geometries in which "reacting atoms approach van der Waals contact" — that is, a coefficient of 1.0. The authors' own crystal measurements, read off Figure 3(a), average close to that. The criterion is nonetheless set at 1.4, with NAC's angular component dropped (limitation iii). The only reason given (§3.3) is that single structures rather than MD ensembles are evaluated, which motivates being permissive but does not fix a value. Neither the theory the paper invokes nor the data it reports points to 1.4.

That leaves the coefficient to be justified empirically, and it is not. At 1.4 the bound admits every reacting-atom distance the paper reports, so nothing it reports measuring is rejected by it; whether a structure incapable of catalysing the reaction would also pass is never tested, and no decoy poses, shuffled ligands or randomly placed substrates appear anywhere. §4.3 examines positives only and so cannot settle it. Nor is the criterion compared against ligand RMSD: §1 justifies avoiding pose comparison on the grounds that a reactive ligand pose is often unavailable, but §3.6 states that most core entries do come with a substrate, cofactor or close-analogue complex — so for most of this benchmark's own entries a reference pose exists, and the comparison could have been made.

This matters because every number the paper's own criterion produces is defined by that coefficient, and none of the three experiments that could have disconfirmed it — a negative control, a sweep of the coefficient, a comparison against a docking metric — is reported. The central claim, that the benchmark measures catalytic geometry rather than binding pose, is neither derived nor tested.

2. The generalisation finding is reported without any basis for judging it, and is confounded.

The extended-set comparison is restricted to the 43 core entries that have extended enzymes (Appendix Figure 7), each weighted equally (§4.5). Read off Figure 5(a), the entire spread across all eight models there is roughly 4.5 points, against a granularity in which one entry changing completely moves the mean by about 2.3. The differences are small and three seeds were run, so a standard deviation or a significance test could have been reported; neither appears anywhere in the paper, and a reader has no way to judge whether the differences it highlights are distinguishable from sampling variation.

Attribution is a separate problem. The same §3.6 that notes most core entries come with a substrate, cofactor or close-analogue complex also states that the extended set is built to include "cases without experimental holo structures" — two things differ between the sets, and the gap is credited entirely to the substrate. The paper cites PLINDER for being leakage-aware but reports no training-set overlap, temporal-cutoff or novel-sequence analysis of its own. So one of the paper's two headline claims rests on an effect it cannot measure and an attribution it cannot support; the other, the collapse under PoseBusters, comes from an external check rather than from the criterion this paper contributes.

Questions for authors
  1. Please compare the model ranking produced by CatGeoBench against a ligand-RMSD ranking, on the subset of core entries that come with a holo complex.

  2. Please report pass rates under negative controls — decoy poses, shuffled ligands, randomly placed substrates — and how the ranking in Table 1 changes as the coefficient is varied between 1.0 and 1.6.

  3. In §4.3, how many reacting atom pairs were excluded because their median distance exceeded the threshold, as the released evaluation code appears to do? Separately: do the two panels of Figure 3 show the same set of pairs? Converting the per-category distributions in (a) into the normalised units used in (b) does not appear to reproduce the distribution shown there.

  4. Of the 239 core entries, how many reference structures predate the training-data cutoff of each evaluated model? Please repeat Table 1 and Figure 5(a) on the subset that post-dates those cutoffs.

  5. Weighting the per-stratum rates in Figure 4(a) by the pair counts in Figure 6(b) does not reproduce the Pair@1 values of Table 1, and all four models deviate in the same direction — what subset and what averaging over seeds does each use? Also, PB-Valid appears to leave the residue–residue stratum unchanged: is it vacuously satisfied for pairs that have no associated ligand?

6 comments

We thank the reviewer for reading the paper so carefully and for the concrete suggestions, including checking the evaluation code. That feedback helped us catch several problems in the text. We have added experiments and analyses in the revision (new appendix material is marked in blue). We go through the questions first, then address the weaknesses.

Q1. Please compare the model ranking produced by CatGeoBench against a ligand-RMSD ranking, on the subset of core entries that come with a holo complex.

The sentence in §3.6 stating that “most include a substrate, cofactor, or close analogue complex” could be misleading. We have revised the wording to clarify that suitable crystal structures often contain related ligands rather than the exact substrates or cofactors specified by the mechanisms. As shown by the ligand-similarity analysis in Appendix B.1, a substantial fraction of evaluation molecules have low maximum similarity to small molecules found in related crystal structures. Consistent with this limitation, only a small number of pairs support a meaningful ligand-RMSD calculation (Appendix B.3). Of the 69 crystal-validated reacting atom pairs, 29 involve a ligand, but only 9 have a crystal ligand with a fingerprint similarity of 1.0. Six of these are stereoisomers, leaving only 3 pairs for which conventional ligand RMSD can be calculated. This subset is too small to support a meaningful comparison of model rankings.

More broadly, the absence of the exact reactive ligand in enzyme crystal structures makes ligand RMSD difficult to use as a systematic evaluation metric. CatGeoBench was designed to address this limitation: its geometric criteria are derived from catalytic mechanism annotations and can be evaluated without requiring a complete holo complex containing the exact substrate or cofactor.

Q2. Please report pass rates under negative controls — decoy poses, shuffled ligands, randomly placed substrates — and how the ranking in Table 1 changes as the coefficient is varied between 1.0 and 1.6.

Appendix B.2 reports geometric negative controls. We applied random rigid transformations to the 29 ligand-containing pairs in the crystal validation set and to the 277 ligand-containing pairs from Protenix-v2 on the core set. Pass counts are high without perturbation and drop sharply afterward, so the distance criterion is sensitive to geometric disruption. Appendix B.4 sweeps the coefficient from 1.0 to 1.6. Absolute success rates rise with the coefficient. Rankings stay broadly stable over the range we looked at, especially once PB-Valid is applied.

Q3. In §4.3, how many reacting atom pairs were excluded because their median distance exceeded the threshold, as the released evaluation code appears to do? Separately: do the two panels of Figure 3 show the same set of pairs? Converting the per-category distributions in (a) into the normalised units used in (b) does not appear to reproduce the distribution shown there.

In the crystal validation, 69 pairs meet the threshold and 1 exceeds it (entry 451, PDBID:1EC9): the catalytic HIS sits on a flexible loop, and the distance in the crystal is 5.02 Å, above 1.4(ri+rj)=4.621.4(r_i+r_j)=4.62 Å.

The two panels of Figure 3 are meant to show the same set of pairs. Thank you for catching the error: the normalisation scale in the original Figure 3b was wrong, and we have corrected it. With the fix, all validated pairs fall inside the 1.4(ri+rj)1.4(r_i+r_j) threshold, and the distribution is more consistent with using that value as a working cutoff.

Q4. Of the 239 core entries, how many reference structures predate the training-data cutoff of each evaluated model? Please repeat Table 1 and Figure 5(a) on the subset that post-dates those cutoffs.

All 239 core-set reference structures were released in 2018 or earlier and therefore predate the training-data cutoffs of all evaluated models (Appendix A.4). Consequently, there is no post-cutoff subset on which to repeat Table 1 and Figure 5(a). We note, however, that temporal overlap does not necessarily imply the same form of complex-level leakage found in conventional docking benchmarks. CatGeoBench evaluates mechanism-derived reacting-atom geometry rather than recovery of a crystal ligand pose, and many entries lack a corresponding substrate-bound crystal structure altogether. Thus, the fact that a reference structure predates a model's training cutoff does not establish that the specific catalytic geometry evaluated by CatGeoBench was present in its training data.

To further examine potential ligand-level overlap, Appendix B.1 compares the evaluation substrates and cofactors with small molecules in related crystal structures. A substantial fraction of entries show low maximum ligand similarity (Figure 10). We further examine the relationship between ligand similarity and geometric accuracy on 75 single-substrate, metal-free core entries (Figure 11). For most models, lower ligand similarity is associated with larger normalized reacting-atom distances.

Q5. Weighting the per-stratum rates in Figure 4(a) by the pair counts in Figure 6(b) does not reproduce the Pair@1 values of Table 1, and all four models deviate in the same direction — what subset and what averaging over seeds does each use? Also, PB-Valid appears to leave the residue–residue stratum unchanged: is it vacuously satisfied for pairs that have no associated ligand?

Thank you for checking this carefully. We found an inconsistency from multiple data iterations: the plots were still using an older stratum count. The scripts and figures have been corrected. On PB-Valid: the checks focus on ligand geometry and ligand–protein contacts. Residue–residue pairs do not involve a ligand, so your reading is correct, PB-Valid adds essentially no extra constraint in that stratum.

Weaknesses

Weakness 1 concerns the 1.4 coefficient. In Q2 we spell out the reasons we chose it (tolerance for single-structure evaluation, behaviour on the crystal validation set) and point to the negative controls and threshold sweep in Appendices B.2 and B.4.

Weakness 2 concerns the basis for the extended-set claims and attribution. We agree that, given the number of annotated mechanisms and the way the extended set is built, the scale cannot match typical docking benchmarks. The drop from core to extended is seen across models, and we agree that the performance drop cannot be attributed to substrate differences alone, as both enzyme and substrate distributions differ between the core and extended sets. We have clarified this point in the revised main text, emphasizing that the observed differences reflect the combined effects of these changes. On training-data overlap and ligand-level leakage, please see our reply to Q4 and the analyses in Appendices A.4 and B.1.

Thank you again for the close reading, including the code and figure checks. Your comments led us to fix errors, add experiments, and make the design choices clearer. With these revisions, we hope the work is more rigorous and the claims are more carefully stated.

Thank you for the corrections and the new appendices. Most of my questions and concerns are resolved.

Three things remain:

  • [W1 / Q2] B.2 does not enough to answer Q2 asked for: a ligand displaced 5–10 Å has left the pocket, so it shows the criterion catches gross misplacement, not that it separates a reactive pose from a merely bound one. The central claim — that this measures catalytic geometry rather than binding pose — is still without direct evidence.

  • [Table 8] The conclusion attributes the PB-Valid drop to enzyme–substrate–cofactor systems being hard to satisfy on both counts at once. But on those same systems PB-Valid costs Vina only 4.95 points on Pair@1, against 15.8 to 32.7 for the co-folding models — and Vina's conformers are valid by construction, since it only rotates torsions of an already-valid 3D structure. The drop looks like a property of how co-folding models generate ligands rather than of the enzyme systems.

  • [W2] With 43 entries weighted equally, one entry moves the mean by 2.3 points and the whole spread in Figure 5(a) is about 4.5. That the extended set is harder, I accept. But "Protenix-v2 drops markedly, Boltz-1x least" is a ranking claim worth about two entries — could you give a bootstrap interval over the 43 entries?

Reviewer_CheJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Overall, this paper introduces CatGeoBench, a mechanism-guided benchmark for evaluating catalytic geometry in enzyme–ligand co-folding models. By extracting reacting atom pairs from catalytic mechanisms and electron-flow information in M-CSA, the authors establish distance-based geometric criteria for evaluating predicted complexes. The dataset is systematically constructed and includes evaluations of several representative co-folding models, along with an extended dataset designed to assess generalization. However, several concerns remain regarding the validity of the evaluation metric, experimental validation, and generalization analysis.

Critical strengths
  1. The research problem is meaningful. Existing protein–ligand structure prediction benchmarks primarily focus on binding poses and physical plausibility. This work further considers mechanism-specific catalytic geometry, providing a potentially valuable direction for evaluating enzyme–ligand complexes.

  2. The dataset construction is systematic. The authors extract reacting atom pairs from electron-flow annotations in M-CSA and carefully curate catalytic residues, substrates, cofactors, and atom mappings, resulting in a reusable benchmark resource.

Critical weaknesses
  • The core evaluation metric is overly simplified. The benchmark primarily relies on distances between reacting atoms to assess catalytic geometric compatibility, while overlooking important factors such as attack angles, hydrogen-bond networks, proton transfer, and conformational dynamics. Satisfying a distance constraint does not necessarily imply that a predicted conformation is mechanistically plausible, raising concerns about the discriminative capability of the proposed metric.

  • The distance threshold lacks sufficient validation. The authors adopt (1.4(r_i+r_j)) as a universal distance threshold without providing a systematic sensitivity analysis. Although all 69 validated reacting atom pairs from crystal structures satisfy this criterion, this only demonstrates consistency with a subset of experimental structures rather than its ability to distinguish correct from incorrect catalytic conformations.

  • The generalization conclusions from the extended set are not sufficiently supported. The extended dataset is constructed by matching EC annotations, catalytic residues, active-site geometries, and reaction centers, resulting in substantial similarities to the core set. The authors do not sufficiently investigate whether the observed performance degradation arises from substrate complexity, protein structural differences, or distribution shifts, making it difficult to attribute the results specifically to limitations in model generalization.

  • The experimental analysis and comparisons remain insufficient. The evaluation mainly focuses on general-purpose co-folding models, without including more targeted catalytic geometry assessment methods or simple geometry-optimization baselines. Furthermore, the analysis of different reaction types and failure modes is limited, and confidence intervals or statistical significance tests are not provided to establish the reliability of the reported performance differences.

  • The distinctive value of the benchmark is not sufficiently demonstrated. Although the paper introduces a mechanism-guided evaluation framework, it remains unclear whether the proposed metric can identify catalytically incorrect structures that existing geometric or physical validity metrics fail to detect. The lack of validation against independent evidence of catalytic function further limits the demonstrated significance of the benchmark.

Questions for authors
  1. How do the authors justify using reacting-atom distances as the primary criterion for catalytic geometric compatibility, given that catalytic reactions also depend on attack angles, hydrogen-bond networks, and conformational dynamics? Can the proposed metric reliably distinguish mechanistically plausible conformations from incorrect ones?

  2. What motivates the choice of 1.4(ri+rj)1.4(r_i+r_j) as a universal distance threshold? Can the authors provide a sensitivity analysis to demonstrate whether the evaluation results and model rankings remain consistent under different thresholds?

  3. How do the authors ensure that the performance degradation on the extended set reflects limitations in model generalization rather than differences in substrate complexity, protein structures, or data distributions between the core and extended sets?

  4. Why are more targeted catalytic geometry assessment methods or geometry-optimization baselines not included? Can the authors provide additional analyses across reaction types and failure modes, together with statistical significance tests, to strengthen the experimental conclusions?

  5. Can the authors demonstrate that CatGeoBench identifies catalytically incorrect structures that existing geometric or physical validity metrics fail to detect? Is there any independent evidence linking the proposed benchmark scores to actual catalytic functionality?

2 comments

We thank the reviewer for the careful reading and constructive comments. In response, we have added several experiments and analyses to the revised manuscript (new appendix material is marked in blue). The reviewer’s weaknesses and questions correspond closely, and we address them in order below.

Q1. How do the authors justify using reacting-atom distances as the primary criterion for catalytic geometric compatibility, given that catalytic reactions also depend on attack angles, hydrogen-bond networks, and conformational dynamics? Can the proposed metric reliably distinguish mechanistically plausible conformations from incorrect ones?

We agree that enzymatic catalysis depends on multiple factors. CatGeoBench is grounded in near-attack conformation (NAC) theory, in which reacting-atom distances, together with angles for specific reaction types, are among the geometric features most directly associated with catalytic function in enzymology and MD studies. Accordingly, we define a distance criterion based on scaled van der Waals radii and additionally apply angular constraints (Appendix B.5). The criterion is intended to assess catalytic geometric compatibility, rather than serve as a sufficient condition for catalysis.

As discussed in the main text, structures exceeding the threshold are less likely to represent physically plausible pre-reaction arrangements, whereas satisfying the criterion does not guarantee catalytic activity. The negative-control experiments in Appendix B.2 show that random rigid-body perturbations of ligands substantially reduce the success rates, indicating that the metric can detect geometric disruption.

Q2. What motivates the choice of 1.4(ri+rj)1.4(r_i+r_j) as a universal distance threshold? Can the authors provide a sensitivity analysis to demonstrate whether the evaluation results and model rankings remain consistent under different thresholds?

There are two main reasons for this choice. First, we evaluate individual predicted structures rather than MD ensembles, so we use a relatively permissive threshold to accommodate structural uncertainty while retaining a mechanistic basis. Second, all reacting-atom pairs in the crystal validation set satisfy this threshold, and the normalized-distance distribution in Figure 3b also supports this choice.

Appendix B.4 reports a sensitivity analysis in which the coefficient is varied from 1.0 to 1.6. As expected, absolute success rates increase with the coefficient, while model rankings remain largely stable across the tested range, particularly with the PB-Valid constraint.

Q3. How do the authors ensure that the performance degradation on the extended set reflects limitations in model generalization rather than differences in substrate complexity, protein structures, or data distributions between the core and extended sets?

All evaluated models show some degree of performance degradation on the extended set (Figure 5a). We agree that this comparison alone cannot attribute the decline to any single factor, such as substrate complexity, structural differences, or distribution shifts. These differences are inherent to the design of the extended set, which aims to evaluate model performance on related enzymes and alternative substrates beyond the core set. Thus, the observed decline reflects the overall challenge of generalizing across these distinct enzyme–substrate systems, although we cannot isolate the contribution of each factor.

Q4. Why are more targeted catalytic geometry assessment methods or geometry-optimization baselines not included? Can the authors provide additional analyses across reaction types and failure modes, together with statistical significance tests, to strengthen the experimental conclusions?

We appreciate the reviewer’s suggestions. We have added geometric negative controls based on random rigid-body perturbations (Appendix B.2) and a docking baseline using AutoDock Vina (Appendix B.6). Figure 4a in the main text reports success rates stratified by reacting-pair type, while Figure 4b shows the distribution of normalized distances. This analysis indicates that many geometric failures involve moderate deviations from the threshold rather than extreme distances from the catalytic site.

Q5. Can the authors demonstrate that CatGeoBench identifies catalytically incorrect structures that existing geometric or physical validity metrics fail to detect? Is there any independent evidence linking the proposed benchmark scores to actual catalytic functionality?

Ligand RMSD is often difficult to use systematically for enzyme systems because crystal structures containing the cognate reactive ligand pose are frequently unavailable. As shown in Appendix B.3, RMSD could be calculated for only three reacting-atom pairs in the core set. The ligand-similarity analysis in Appendix B.1 further shows that a substantial fraction of evaluation entries have low maximum similarity between their substrates/cofactors and small molecules in related crystal structures, limiting the availability of suitable holo structures as systematic references. By deriving its geometric criteria from mechanistic annotations and NAC theory, CatGeoBench does not require crystal structures containing the complete substrate and cofactor set, thereby addressing this limitation.

The connection between our benchmark and catalytic function is grounded in existing NAC studies, which have repeatedly shown that the fraction of conformations satisfying specific distance and angle criteria is associated with catalytic activity. CatGeoBench extends this idea into a computationally tractable evaluation of individual static predictions.

We thank the reviewer again for the careful reading and constructive feedback, which has helped us improve the manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.