- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CRYOCUE uses predicted non-protein density to improve protein model building from cryo-EM maps. An anchor-supervised detector represents five hetero classes and supplies multiscale features to a backbone-localization network. Sparse predicted hetero candidates subsequently condition structure refinement through class information, confidence, and geometry relative to evolving protein frames. The method improves structural agreement, coverage, and connectivity over the reported baselines,
Critical strengths
- The scientific motivation is specific and well supported. The preliminary analysis shows why uniformly suppressing candidates near hetero components can help some cases and damage others.
- The design connects the representation to both stages of reconstruction.** Anchor targets provide a consistent localization task across disparate component shapes; coarse attention and fine-scale gates expose hetero context during backbone localization; frame-relative conditioning makes that context responsive to structure updates.
- The performance is good on benchmarks.
- The data protocol may be instructive to other cryo-em methods.
Critical weaknesses
-
The most important issue is that the ablations do not fully isolate hetero semantics from capacity and auxiliary supervision. The additional detector is supervised on protein backbone and side-chain classes as well as hetero classes. It is hard to tell whether the structure prediction network use the protein feature or use the hetero feature. Maybe the detector provide additional protein signals and do not use hetro features at all. A matched auxiliary stream trained only on protein/background labels, or a collapsed-hetero-class control, would test whether the benefit requires the proposed semantic representation.
-
External-baseline comparability is not sufficiently documented. The paper specifies its own large training inventory and staged optimization, but does not provide comparably detailed baseline configurations, training data, checkpoint provenance, tuning budgets, or overlap checks.
-
As Table 16 reports, the different classes of hetero have largely different accuracies. F1 values are 76.04% for nucleic acid, 22.77% for glycan, 26.85% for nucleotide/cofactor, and 14.07% for lipid/detergent. Partial detections may still provide useful context, as the authors explain, but aggregate reconstruction gains do not show that all component classes benefit. A per-class dropout experiment may be valuable.
-
Anchor supervision brings large improvement, from 74.7 to 84.6. But 74.4 is even lower than protein-only baseline, maybe because anchor change how we positive region number or ratio. So this need be claimed whether the improvement is brought from the anchor data curation or real anchor. Maybe author should clarify the supervision region data curation in details.
Questions for authors
Is it necessary to distinguish between different classes of hetero as the final task is for protein structure. Will merging all classes of hetero into one imparing the final performance?
7 comments
Response to W2: Baseline Comparability and Reproducibility
W2: External-baseline comparability is not sufficiently documented.
W2.1: The paper specifies its own large training inventory and staged optimization, but does not provide comparably detailed baseline configurations, training data, checkpoint provenance, tuning budgets, or overlap checks.
A: We understand the reviewer's concern regarding baseline configurations, checkpoint provenance, tuning budgets, and potential training–test overlap. All external baselines were evaluated using their official pretrained checkpoints and recommended inference procedures, without retraining or additional hyperparameter tuning; we also examined potential direct training–test overlap using available documentation. Specifically:
(1) Baselines use official pretrained checkpoints without additional training or tuning. We evaluated EMProt, E3-CryoFold, ModelAngelo, EModelX, and CryoBoltz using their publicly released pretrained checkpoints and the inference procedures provided in their official GitHub repositories and README instructions. All methods were evaluated on the same 300-map test set using their respective supported inputs and a consistent evaluation protocol. No baseline was retrained, fine-tuned, or subjected to additional hyperparameter searches. Table B summarizes the experimental configurations.
(2) Training–test overlap checks are based on available inventories and documentation. We examined the available training-data information and checkpoint documentation for each baseline. For EMProt, E3-CryoFold, and EModelX, comparisons against their available training inventories identified no direct PDB/EMDB entry overlap with our test set. For ModelAngelo, the documented training-data cutoff (April 2022) precedes the release of our test structures. For CryoBoltz, we additionally checked for direct PDB entry overlap with the Boltz-1 structural training data, which uses a September 30, 2021 cutoff.
Table B. External baseline configurations and evaluation protocol.
| Method | Checkpoint source | Inference configuration | Additional training or tuning |
|---|---|---|---|
| EMProt | Official pretrained weights | Official pipeline; three de novo recycling rounds and sequence-guided assembly | None |
| E3-CryoFold | Official pretrained weights | Official Chroma-based backbone generation and sequence-alignment pipeline | None |
| ModelAngelo | Official pretrained weights (v1.0.18) | Official Cα/GNN inference and sequence-guided model building | None |
| EModelX | Official pretrained weights | Official backbone prediction, sequence alignment, and structure completion | None |
| CryoBoltz | Official pretrained weights | Official GitHub/README inference procedure | None |
Response to W3: Class-wise Hetero Contributions
W3: As Table 16 reports, the different classes of hetero have largely different accuracies.
W3.1: F1 values are 76.04% for nucleic acid, 22.77% for glycan, 26.85% for nucleotide/cofactor, and 14.07% for lipid/detergent. Partial detections may still provide useful context, as the authors explain, but aggregate reconstruction gains do not show that all component classes benefit.
A: We understand the reviewer's concern that the large variation in hetero detection accuracy makes it unclear whether all five component classes contribute to protein reconstruction. Our per-class evaluation and newly added dropout experiments provide evidence that all five hetero classes offer useful reconstruction cues, including classes with relatively low detection F1 scores. Specifically:
Class-wise reconstruction improvements extend to all five hetero classes. Although detector F1 varies considerably across hetero classes, CryoCue consistently improves local Cα localization and protein–hetero contact recovery over EMProt for all five classes, as shown in Table C. This indicates that imperfect hetero detection can still provide useful structural context.
Table C. Class-specific detection and reconstruction performance.
| Hetero class | Detector F1 (%) | EMProt Cα F1 (%) | CryoCue Cα F1 (%) | EMProt Contact F1 (%) | CryoCue Contact F1 (%) |
|---|---|---|---|---|---|
| Nucleic acid | 76.04 | 85.937 | 89.902 | 75.836 | 78.621 |
| Glycan | 22.77 | 80.49 | 85.427 | 75.243 | 78.426 |
| Nucleotide/cofactor | 26.85 | 86.475 | 90.335 | 79.252 | 81.734 |
| Other ligand | 49.1 | 95.799 | 98.485 | 83.017 | 85.074 |
| Lipid/detergent | 14.07 | 93.209 | 96.355 | 79.686 | 82.55 |
W3.2: A per-class dropout experiment may be valuable.
A: Removing the conditioning cues of any individual hetero class degrades reconstruction performance across all six reported metrics. We further remove the conditioning cues from each hetero class individually while keeping the trained model unchanged. As shown in Table D, removing any class consistently degrades all six reconstruction metrics. For example, removing nucleic-acid cues reduces TM-score from 0.846 to 0.835, while removing lipid/detergent cues reduces it to 0.843. These results provide additional evidence that conditioning cues from each hetero class contribute to reconstruction, rather than the improvements being driven solely by classes with high detection accuracy.
Table D. Effect of removing individual hetero-class cues during inference.
| Method | TM-score ↑ | lDDT ↑ | Coverage (%) ↑ | RMSD (Å) ↓ | Q-score ↑ | Break rate (%) ↓ |
|---|---|---|---|---|---|---|
| w/o nucleic-acid cues | 0.835 | 0.776 | 83.91 | 0.584 | 0.588 | 0.84 |
| w/o glycan cues | 0.842 | 0.784 | 84.61 | 0.576 | 0.591 | 0.75 |
| w/o nucleotide/cofactor cues | 0.84 | 0.782 | 84.43 | 0.578 | 0.59 | 0.78 |
| w/o other-ligand cues | 0.839 | 0.781 | 84.28 | 0.58 | 0.589 | 0.8 |
| w/o lipid/detergent cues | 0.843 | 0.785 | 84.73 | 0.575 | 0.592 | 0.74 |
| CryoCue (Full) | 0.846 | 0.789 | 85.02 | 0.571 | 0.593 | 0.69 |
Response to W1: Hetero Semantics and Model Capacity
Response to Reviewer cNhy
We thank the reviewer for the thoughtful and constructive review. We are pleased that the reviewer recognizes the well-supported scientific motivation, hetero-aware conditioning across both reconstruction stages, strong experimental performance, and the potential value of our data processing protocol. Below, we address each concern and provide further clarification of our experimental setup and methodological design.
Responses to Critical Weaknesses
W1: The most important issue is that the ablations do not fully isolate hetero semantics from capacity and auxiliary supervision.
W1.1: The additional detector is supervised on protein backbone and side-chain classes as well as hetero classes. It is hard to tell whether the structure prediction network use the protein feature or use the hetero feature. Maybe the detector provide additional protein signals and do not use hetro features at all.
A: We understand the reviewer's concern that CryoCue's improvements may arise from additional model capacity or auxiliary protein supervision rather than hetero-specific representations. Our architecture-matched Protein-only and Collapsed-hetero ablations provide evidence that both hetero-label supervision and class-specific representations contribute to the improvement beyond matched model capacity and protein supervision. Specifically:
(1) Matched model capacity and protein supervision do not explain the full performance gain. The Protein-only variant retains the same detector and reconstruction architecture, including auxiliary protein supervision, but removes hetero-class supervision. CryoCue improves TM-score from 0.783 to 0.846, lDDT from 0.712 to 0.789, and Coverage from 77.73% to 85.02%. Since the architecture and auxiliary protein supervision are matched, these gains cannot be attributed to additional model capacity or protein supervision alone.
(2) Preserving hetero-class distinctions improves reconstruction beyond identifying hetero regions. We further introduce a Collapsed-hetero control that merges all five hetero classes into a single category while retaining the same architecture and protein supervision. As shown in Table A, Collapsed-hetero already achieves a TM-score of 0.833, outperforming the Protein-only variant (0.783). Importantly, distinguishing the five hetero classes further improves TM-score from 0.833 to 0.846, lDDT from 0.773 to 0.789, and Coverage from 83.82% to 85.02%. This indicates that CryoCue benefits not only from identifying hetero regions but also from their class-specific representations.
W1.2: A matched auxiliary stream trained only on protein/background labels, or a collapsed-hetero-class control, would test whether the benefit requires the proposed semantic representation.
A: Both suggested controls are included: the architecture-matched Protein-only control isolates the additional benefit of hetero-label supervision, while the Collapsed-hetero control demonstrates the benefit of retaining separate hetero classes. As shown in Table A and detailed in W1.1, the controls achieve TM-scores of 0.783 and 0.833, respectively, compared with 0.846 for full CryoCue.
Table A. Ablation of hetero supervision and semantic granularity.