- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CQFiner for protein binding-site detection. It combines local residue-to-query message passing, near-site supervision, a ViSNet coarse stage, and an EGNN refinement stage with fixed protein features. The read-only protein context allows the query count and sampling distribution to change at inference. Experiments show large gains on HOLO4K and COACH420, with further structural-robustness evaluation on AlphaFold-predicted structures, APOBind, and LIGYSIS.
Critical strengths
-
Strong overall performance. CQFiner achieves substantial improvements on the main benchmarks, with gains in both binding-site localization and residue-mask prediction. These results provide strong empirical support for the proposed framework.
-
Good generalization ability across different structure types. The model maintains competitive performance on AlphaFold-predicted and apo structures without dataset-specific tuning. This broadens its practical relevance beyond experimental holo structures.
Critical weaknesses
-
The spatial-resolution argument is too broad. The paper does not sufficiently distinguish spatially sampled queries from learned detection queries, such as those used in UniSite. For spatial queries, adding more points can improve geometric coverage. For DETR-like queries, however, the query count controls the capacity to represent candidate pockets and does not directly correspond to spatial resolution. The motivation should therefore be scoped more carefully.
-
Improved generalization is not clearly separated from stronger in-domain performance. CQFiner obtains higher absolute scores on most OOD benchmarks, but these results may largely reflect its stronger in-domain performance. UniSite also handles structural changes well and sometimes overtakes CQFiner: on HOLO4K, AP₀.₅ changes from 0.591 to 0.592 for UniSite when moving to AlphaFold structures, compared with a drop from 0.640 to 0.575 for CQFiner. The current evidence therefore supports strong performance across structure types, but does not clearly establish better generalization due to the proposed architecture. Since this work largely inherits UniSite’s data organization and task formulation, which differ substantially from those used in earlier approaches, it remains unclear how much of the observed generalization comes from the proposed architecture versus the inherited dataset design and training protocol.
Questions for authors
-
Please clarify the scope of the spatial-resolution argument. In particular, how does it apply to learned DETR-like queries, such as those in UniSite, whose number does not directly correspond to spatial sampling density?
-
What evidence shows that CQFiner generalizes better than UniSite, rather than mainly retaining its stronger in-domain performance? Table 2 suggests that UniSite shows smaller performance drops or even gains under some structural changes. Please discuss how these results affect the generalization claim.
-
How much of the performance on OOD benchmarks can be attributed to the proposed architecture versus the data organization and task formulation inherited from UniSite?
3 comments
Reply of W2/Q2
W2 / Q2: Is better OOD performance merely a continuation of higher ID accuracy?
Q: CQFiner's higher absolute OOD accuracy may mainly come from its higher ID accuracy. In Table 2, UniSite exhibits smaller performance drops under some structural changes, or even slight improvements. What evidence can distinguish higher baseline accuracy from stronger generalization ability?
A: We thank the reviewer for the constructive comments. First, we would like to clarify that CQFiner itself was not specifically designed for OOD settings, and our OOD evaluation is primarily intended to demonstrate CQFiner's accuracy. However, we did find that CQFiner has certain OOD advantages, mainly including generalization across interaction patterns and protein lengths.
Protein-length scaling: Unlike VNEGNN and UniSite, where each query interacts with the entire protein, queries in CQFiner interact only locally, which means that even when protein length increases, the messages received by each query in CQFiner remain essentially unchanged. In contrast, VNEGNN and UniSite experience distribution shifts and performance loss because the number of residues interacting with each query increases. This allows CQFiner to scale to greater protein lengths at test time while maintaining better accuracy. To illustrate this, we sort samples by the number of input residues and use the longest 10% as the OOD test set, splitting the remaining samples into training and ID validation sets at a ratio of 8∶1, for an overall ratio of 8∶1∶1. We first train UniSite and then select CQFiner weights based only on ID validation performance, such that its in-distribution validation AP₀.₅ is lower than UniSite's. After fixing the weights, we evaluate on the test set consisting of the longest 10% of proteins. The results are quite striking. Even when CQFiner and UniSite have comparable [email protected] on the ID dataset (proteins of lengths comparable to those in the training set), CQFiner has an advantage of approximately 10 percentage points on the OOD test set, demonstrating CQFiner's length scalability.
Table R2. Comparison of protein-length OOD generalization with approximately matched ID validation accuracy.
| Model | ID validation AP₀.₅ (%) | Long-protein OOD AP₀.₅ (%) |
|---|---|---|
| UniSite | 44.03 | 11.90 |
| CQFiner | 42.50 | 22.94 |
Interaction-pattern generalization: The full analysis shows that CQFiner's advantage over UniSite is more pronounced for pockets with lower similarity to the training set. We use PLIP (PLIP 2021: expanding the scope of the protein–ligand interaction profiler to DNA and RNA) to extract protein–ligand interactions, take the 12,653 sites from all 8,348 training proteins as the reference, compute each evaluation site's maximum similarity to training pockets, and then compare the two models' prediction quality on that site. All three evaluation sets show a consistent trend. For example, in the group with similarities of 0.25–0.50, CQFiner's IoU@5 advantages are 8.23, 10.31, and 8.75 percentage points, respectively; when similarity reaches above 0.75, the two models perform similarly. This shows that CQFiner's prediction advantage also extends to pockets whose interaction patterns are less similar to those in the training set. We believe this may be because CQFiner's GNN-based backbone models interactions more effectively than DETR's Transformer.
Table R3. Pocket-interaction similarity to the complete training set and CQFiner's relative prediction advantage.
| Evaluation set | Similarity to the training set | Number of sites | CQFiner IoU@5 (%) | UniSite IoU@5 (%) | Difference (percentage points) |
|---|---|---|---|---|---|
| UniSite validation set | [0.00, 0.25) | 60 | 42.31 | 34.73 | +7.58 |
| UniSite validation set | [0.25, 0.50) | 330 | 55.91 | 47.68 | +8.23 |
| UniSite validation set | [0.50, 0.75) | 989 | 68.22 | 62.69 | +5.53 |
| UniSite validation set | [0.75, 1.00] | 526 | 76.29 | 74.99 | +1.30 |
| COACH420 | [0.00, 0.25) | 10 | 55.45 | 38.98 | +16.48 |
| COACH420 | [0.25, 0.50) | 71 | 61.84 | 51.53 | +10.31 |
| COACH420 | [0.50, 0.75) | 104 | 70.07 | 62.55 | +7.52 |
| COACH420 | [0.75, 1.00] | 28 | 73.91 | 75.79 | −1.87 |
| HOLO4K-sc | [0.00, 0.25) | 35 | 51.17 | 45.58 | +5.59 |
| HOLO4K-sc | [0.25, 0.50) | 289 | 59.26 | 50.51 | +8.75 |
| HOLO4K-sc | [0.50, 0.75) | 941 | 68.21 | 66.74 | +1.47 |
| HOLO4K-sc | [0.75, 1.00] | 352 | 75.97 | 77.11 | −1.14 |
Reply of Q3
Q3: How much of the OOD performance comes from CQFiner's architecture, and how much from UniSite's data organization?
Q: CQFiner inherits UniSite's data organization and task formulation, which differ substantially from those of earlier methods. How can the contributions of model architecture and the training-data scheme to OOD performance be distinguished?
A: We thank the reviewer for the constructive comments. We have added a control in which CQFiner is trained on the scPDB data used by VN-EGNN (scPDB contains 4,499 training complexes, while UniSite-DS contains 8,348 training proteins). Overall, we find that the training-data scheme affects different test sets differently. CQFiner trained on scPDB achieves AP₀.₅ and localization performance on COACH420 comparable to those obtained with UniSite-DS training, but performance declines on HOLO4K and the UniSite validation set. For the other OOD datasets, the UniSite-DS-trained model performs better on AF-HOLO4K-sc and CryptoBench, while the scPDB-trained model performs slightly better on APObind. Overall, the model trained on scPDB performed comparably to the model trained on UniSite-DS on the out-of-distribution (OOD) datasets, but substantially worse on the UniSite-DS validation set. These results suggest that UniSite-DS differs markedly from existing public benchmarks in how its data are organized. One possible explanation is that UniSite maps multiple binding sites onto a single structure, which may introduce structural discrepancies and misalignment of ligand coordinates. Owing to these differences in data organization, evaluation on these OOD datasets may disadvantage models trained on UniSite-DS.
Table R4. Internal validation and external test results for CQFiner trained on UniSite-DS or per-complex scPDB. Each cell lists the results of UniSite/scPDB training. The internal validation results of the UniSite-DS model retain the corresponding validation records from the main table in the paper.
| Evaluation dataset | Proteins | AP₀.₃ (%) | AP₀.₅ (%) | DCC (%) | DCA (%) |
|---|---|---|---|---|---|
| UniSite validation set | 2,053 | 66.42 / 50.68 | 51.67 / 35.51 | 61.45 / 48.63 | 74.59 / 62.92 |
| COACH420 | 287 | 77.75 / 72.39 | 56.65 / 56.22 | 64.29 / 63.99 | 82.74 / 82.74 |
| HOLO4K-sc | 1,633 | 75.84 / 69.54 | 59.36 / 52.86 | 72.66 / 65.17 | 85.35 / 77.57 |
| AF-COACH420 | 254 | 71.94 / 69.53 | 47.02 / 48.29 | 63.12 / 60.80 | 82.39 / 75.75 |
| AF-HOLO4K-sc | 1,337 | 74.31 / 68.35 | 52.52 / 46.80 | 69.31 / 64.58 | 83.51 / 77.87 |
| APObind holo | 10,650 | 53.31 / 56.89 | 22.00 / 23.13 | 54.30 / 57.67 | 71.76 / 72.57 |
| APObind apo | 10,650 | 49.47 / 51.15 | 14.80 / 15.42 | 47.66 / 49.49 | 65.69 / 65.83 |
| LIGYSIS | 2,775 | 26.27 / 19.24 | 14.51 / 10.36 | 21.13 / 17.34 | 32.78 / 28.13 |
| CryptoBench | 214 | 35.28 / 30.54 | 16.22 / 13.30 | 37.85 / 33.64 | 52.34 / 49.07 |
Reply of W1/Q1
The reviewer acknowledged CQFiner's performance advantage on experimental holo structures and its competitiveness on predicted and APO structures. However, the reviewer questioned the scope of the spatial-resolution argument and the sources of OOD performance (including UniSite's data organization and CQFiner's architecture). We first revised the main text to delimit the spatial-resolution argument for DETR-like query-based methods such as UniSite. Using UniSite, we then showed how CQFiner's components/designs can improve these models, with locality-preserving architecture and supervision tasks being key. Next, we analyzed CQFiner's OOD generalization across protein lengths and interaction patterns. Finally, we discussed UniSite's distinctive data organization, which appears to exhibit a clear distribution shift relative to earlier datasets.
W1 / Q1: How should the spatial-resolution argument apply to learned-query methods such as UniSite?
Q: The number of spatially sampled queries affects geometric coverage, whereas the number of learned queries in DETR-like methods such as UniSite mainly controls candidate-pocket capacity and does not directly correspond to spatial sampling density. Please clarify the scope of the spatial-resolution argument and how this idea can be applied to learned queries.
A: We thank the reviewer for the constructive comments. We will explicitly distinguish spatial queries with physical coordinates from purely learned queries. Spatial resolution directly corresponds to the former. For the latter, although CQFiner's approach cannot be applied directly, its design principles are easy to incorporate because each query still predicts a single site: each query primarily makes a local prediction, and more queries ensure greater spatial coverage and accuracy. To achieve this, we added spatial anchors to UniSite's DETR-style architecture, giving query initialization and mask prediction explicit spatial references. Experiments show that this version outperforms standard UniSite with the same number of queries, and increasing inference queries further improves pocket recall, demonstrating that this design idea can also be combined with DETR-like architectures.
Original UniSite learns a separate embedding for each query, so more queries mainly increase candidate capacity. We initialized queries with FPS-sampled residue features plus a shared learnable embedding, added CQFiner's distance gate to the final query–residue dot product, and supervised each query to predict its neighboring site's residue mask; these designs essentially fully reuse CQFiner's design (see Methods in the main text). Each query thus corresponds to a residue coordinate, primarily focuses on surrounding residues, and predicts the corresponding mask. Therefore, more queries essentially increase spatial coverage.
We compare this version with standard UniSite on the validation set below.
Bold and italicized values indicate the best and second-best results within the same comparison group, respectively, with ties marked at the precision shown in the table.
Table R1. Prediction performance and inference-query scaling of standard UniSite and spatially anchored UniSite on the UniSite-DS validation set.
With the same 32 queries, the spatially anchored version improves AP₀.₃ by 3.17 percentage points, AP₀.₅ by 3.21 percentage points, and recall by 4.97 percentage points over standard UniSite. Increasing the number of queries to 200 with fixed weights raises recall from 55.11% to 58.08%, a further improvement of 2.97 percentage points and 7.94 percentage points above standard UniSite; when only the top five candidates are retained, recall is also 5.78 percentage points higher than standard UniSite. With 400 queries, recall reaches 58.57%. However, overall AP does not improve, possibly because of generalization problems introduced by multiple queries (the queries are not independent because of self-attention). These results show the following, although the DETR-like query design itself is unrelated to spatial resolution. Because each query still represents a site, it is easy to incorporate CQFiner's approach and improve performance by designing query features and supervision targets.
Accordingly, we have revised the paper to restrict the resolution argument to spatial resolution and the corresponding specific query types. These approaches combining learnable DETR-like queries with CQFiner's design have been added to the appendix.