- The primary residue-level metric is pooled across proteins
According to Appendix D, for residue-level tasks S is the total number of residues across all proteins. The released fine-tuning code likewise concatenates predictions and targets from all validation proteins and computes Pearson and Spearman once per fold. This pooled estimand gives long proteins disproportionate influence and mixes between-protein and within-protein variability. It also does not directly measure whether the predicted flexibility pattern is correct within each protein. In contrast, RMSF-net computes one correlation per protein and reports the mean over 335 proteins. The paper does not make this protocol distinction explicit, even though it compares CryoProt with RMSF-net in the same table.
- Per-protein evaluation shows a large performance gap and unreproduced results
I evaluated the public RMSF annotations using the released CryoProt fine-tuned checkpoints and compared them with the per-protein RMSF-net results reported in the authors' supporting workbook. Of the 335 published proteins, 292 had sufficient finite annotations to compute correlations. Using an ensemble of the five released CryoProt folds, the mean per-protein Pearson was 0.354 and the mean per-protein Spearman was 0.412. The workbook reports a mean per-protein RMSF-net Pearson of 0.746 over all 335 proteins. On the 291 proteins represented in both evaluations, RMSF-net's mean per-protein Pearson was 0.760 versus 0.354 for CryoProt, a mean gap of 0.407. The two methods' per-protein correlation values had almost no association (Pearson correlation approximately 0.007).
PDB Metric CryoProt RMSF-net
3J9D RMSE 0.963 0.376
Pearson 0.419 0.840
Spearman 0.481 0.815
3J9E RMSE 2.088 0.710
Pearson 0.088 0.892
Spearman 0.238 0.787
5I68 RMSE 1.214 0.752
Pearson 0.512 0.837
Spearman 0.723 0.768
CryoProt's Appendix I reports RMSE values of 0.527, 0.519, and 0.583 for these same three examples, whereas the released-checkpoint pipeline produces 0.963, 2.088, and 1.214. None of the five CryoProt fold checkpoints brings 3J9E close to 0.519.
Because the official CryoProt inference entry point is missing, I treated the numerical reproduction as a best-effort reconstruction using the released model definition, preprocessing, and checkpoints. As a sanity check, I rendered the three structures with the same PDB, residue set, view, and color scale. The reproduced CryoProt flexibility patterns qualitatively resembled those in Appendix I, reducing concern that the reimplementation is grossly incorrect. Nevertheless, both the original Appendix I visualization and the reproduced CryoProt visualization appeared visibly less consistent with the ground-truth high- and low-flexibility regions than the RMSF-net visualization. The qualitative comparison corroborates the quantitative gap.
These results suggest that the current flexibility model has not been adequately optimized or calibrated, and that the flexibility task has not been evaluated in a way that supports the claimed advantage over RMSF-net.
3. The documented inference entry point is missing
The README instructs users to run python -m Case.predict, but no Case directory or corresponding module is released, so the documented command fails. Because this is a required inference entry point rather than omitted training data, the repository does not support the paper's reproducibility claim. The missing prediction code should be provided.
Rebuttal by Authors
Thanks for your thoughtful and encouraging review. We will incorporate this discussion into the revised manuscript.
Response: Although some improvements are within the standard deviation of competing baselines, CryoProt achieves higher mean performance than the baselines across multiple downstream tasks. More importantly, our primary contribution is not merely achieving substantial numerical improvements over existing methods, but rather introducing a protein pretraining framework that leverages experimental cryo-EM density maps for downstream property prediction. In particular, CryoProt explicitly models cross-box interactions, addressing the limitation of existing approaches that process local density boxes independently and overlook their spatial dependencies. Our experimental results demonstrate the effectiveness of this framework, with consistently competitive performance and improvements in mean metrics across downstream tasks. We believe these contributions extend beyond numerical gains and provide a new perspective on learning protein representations from experimental cryo-EM density maps.
Response: Since CryoProt is a protein pretraining framework, we compare its model size with representative pretrained models rather than domain-specific models. As shown in Table R1, CryoProt contains 178M parameters, approximately 3.7× fewer than ESM-1b, and has a model size comparable to ESM2-150M and OmegaFold. CryoProt completes pretraining in 26 hours on two RTX 5090 GPUs, and the resulting pretrained representations can be reused across multiple downstream tasks without retraining from scratch.
For the pretrained baselines, we directly use their officially released weights, as retraining these large-scale models from scratch would require substantial computational resources. Moreover, their official implementations do not provide validated pretraining configurations for RTX 5090 GPUs, making a reliable comparison under identical hardware conditions difficult. Therefore, we avoid estimating their pretraining times or claiming superior pretraining efficiency without direct evidence. Nevertheless, CryoProt's moderate model size and reusable pretrained representations support efficient adaptation to downstream tasks. As further demonstrated in Q2, CryoProt requires only 5 hours of fine-tuning for protein flexibility prediction, compared with 7–20 hours for domain-specific models, highlighting its advantage in task-specific training efficiency.
Table R1. Model Size and Training Time Comparison.
Response: CryoProt is more efficient for downstream task adaptation than training domain-specific models from scratch. Taking protein flexibility prediction as an example, CryoProt requires only 5 hours of fine-tuning, compared with 7–20 hours for competing models, achieving up to 4× faster training, as shown in Table R2.
Table R2. Training Time Comparison for Protein Flexibility Prediction.
Response: We have addressed this question in Response W1.