- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2(+1)
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents STORM, a lightweight, structure-free model for protein–ligand binding affinity prediction. It combines 15 small MLPs that use different molecular fingerprints, descriptors, and protein features, including target IDs and functional annotations. The models learn both individual activity values and activity differences between compounds in the same assay. Their predictions are averaged and then smoothed using molecular similarity. STORM is trained on ChEMBL and BindingDB data and evaluated on four affinity benchmarks. The authors report performance close to Boltz-2 at much lower computational cost, along with gains over several structure-free models. They also show that fine-tuning on small amounts of target-specific data can improve predictions for new targets.
Critical strengths
- This paper addresses a useful practical question about the cost of affinity prediction. STORM uses standard molecular features, protein annotations, and small MLPs without requiring 3D structures or pretrained language-model embeddings. The reported results suggest that this simple approach can achieve useful accuracy at a low computational cost.
Critical weaknesses
- The method has limited novelty, and the source of its gains is unclear. STORM combines existing molecular representations and protein annotations using small MLPs, weighted ensembling, and similarity-based smoothing. This may be useful, but the paper does not clearly show a new way to transfer affinity information across targets. The ensemble comparison changes both the diversity of ligand representations and the use of protein information at the same time. It therefore cannot show how much protein information contributes to the improvement. A ligand-only ensemble with the same molecular representations and model count would help answer this question. Without this comparison, the paper provides limited insight beyond showing that a particular combination of existing methods performs well.
- The comparisons do not provide enough support for the main performance claims. The comparison with Boltz-2 uses results from previous studies rather than running the model under the same evaluation setup. For ChEMBL35, the authors calculate an overall score by assuming that the original report uses compound-weighted correlations. It is therefore unclear whether the two models’ scores were calculated in the same way. At least one structure-based affinity predictor should be directly tested on the same data using the same evaluation procedure.
Questions for authors
- Can you confirm whether the Boltz-2 ChEMBL35 results use the same compound-weighted correlation as STORM? To support the main performance claim, can you directly evaluate Boltz-2, or another structure-based affinity predictor, on the same test data using the same evaluation procedure?
- What explains AttentionDTA’s low FEP+4 correlation and large standard error (r = 0.062 0.218)? Please provide the correlations for the four individual targets and explain how the training settings and final checkpoint were selected. What checks were used to confirm that the model was properly trained?
- How much of STORM’s improvement comes from protein information rather than the use of different ligand representations? Can you compare STORM with a ligand-only ensemble using the same molecular representations, model count, loss, and smoothing? This would help determine whether the proposed transfer across targets adds value beyond a diverse ligand-only ensemble.
6 comments
Response to Reviewer uWrZ, Part 2
2. Evaluating significance of accuracy compared to other methods.
We directly test Boltz-2 on our same evaluation procedure. We ran Boltz-2 using all of the exact hyperparameters detailed in v2.2.1 [3], with the exception of toggling on affinity molecular weight correction. We noticed an increase in compound-weighted Pearson r, so we kept the stronger baseline. We scored all local models on the same available rows using the compound-weighted within-assay Pearson correlation described in Section 4.3. The Boltz-2 paper specifies compound weighting for its own affinity evaluations [1].
| Model | ChEMBL35 Time Split | OpenFE | FEP+4 | CASP16 |
|---|---|---|---|---|
| STORM | 0.355 ± 0.007 | 0.576 ± 0.024 | 0.705 ± 0.053 | 0.671 ± 0.045 |
| Boltz-2 (local) | 0.303 ± 0.007 | 0.570 ± 0.024 | 0.675 ± 0.062 | 0.640 ± 0.052 |
| Boltz-2 (published references) | ≈0.341 [2] | 0.620 [1] | 0.660 [1] | 0.650 [1] |
| Structure-free DTA | 0.238 ± 0.008 | 0.328 ± 0.030 | 0.441 ± 0.092 | 0.380 ± 0.075 |
| AttentionDTA | 0.157 ± 0.008 | 0.244 ± 0.031 | 0.062 ± 0.089 | 0.580 ± 0.049 |
| PSICHIC | 0.142 ± 0.008 | 0.381 ± 0.029 | 0.518 ± 0.072 | 0.565 ± 0.054 |
| BALM | 0.210 ± 0.008 | 0.328 ± 0.031 | 0.481 ± 0.086 | 0.387 ± 0.066 |
| Mattsson-Walters | 0.230 ± 0.008 | 0.264 ± 0.032 | 0.638 ± 0.073 | 0.252 ± 0.085 |
Due to computational cost, we present results for Boltz-2 obtained for a single seed (2023). We show comparisons to all other methods in a matched seed setting, comparing to predictions from a seed 2023 run on STORM. All models were scored on the same within-assay correlations and the +/- values are one within-assay bootstrap standard error. We note that our results are lower than the publicly reported Boltz-2 numbers, but fall within the publicly reported error bar ranges. All settings were defaults except the molecular-weight correction, which substantially improved Boltz-2 performance.
On the ChEMBL 35 Time Split, STORM scored 0.0521 higher than our evaluation of Boltz-2 (unadjusted paired 95% bootstrap-t CI [+0.0359, +0.0691], Holm-adjusted ). The difference persisted on the 426 assays with complete Boltz-2 coverage (0.3583 versus 0.3008). The other three Boltz-2 comparisons were inconclusive after correction. STORM had higher Pearson point estimates in all 20 comparisons with the retrained structure-free baselines. We used 50,000 paired within-assay bootstrap replicates and applied Holm correction across 72 tests spanning three metrics, four benchmarks, and six methods [4].
We will add the matched comparison to Section 4.3 and report the full paired tests and inference settings in the appendix.
[1] Passaro et al., Boltz-2, Section 5
[2] Isomorphic Labs, IsoDDE technical report, Figure 8
[3] Boltz-2 v2.2.1 prediction options and molecular-weight correction implementation
[4] Holm, A Simple Sequentially Rejective Multiple Test Procedure
3. Compouned-weighted correlations for Boltz-2 on ChEMBL 35 versus STORM.
Yes, we do confirm that Boltz-2 uses the same compound-weighted Pearson correlation as STORM for the results reported by the Boltz-2 paper (the CASP16, FEP+4, and OpenFE reported Boltz-2 results). Please see Section 5 ("Evaluation"), where the Boltz-2 manuscript states that: “For the affinity measurements, all cross-assay averages are weighted by the number of compounds in the assay" of [1]. The approximately 0.341 ChEMBL 35 Time Split reference in our submission comes from aggregating the per-class Boltz-2 scores in IsoDDE Figure 8. The figure leaves the within-class weighting unspecified, so our reconstruction assumes compound weighting, as described in submitted Appendix E [2].
In our direct comparison in Point (2), in our evaluation of Boltz-2, we calculated Pearson correlation separately within each assay and averaged the assay correlations using the number of evaluated compounds as weights, also following the same procedure as [1]. STORM and Boltz-2 were scored on the same evaluation data with the same assay groupings and weights. We confirm all of our evaluations of Boltz-2 on all benchmarks are scored with an identical procedure.
Response to Reviewer uWrZ, Part 3
4. AttentionDTA's low FEP+4 correlation and large standard error.
Here are the individual target results for AttentionDTA on FEP+4.
| FEP+4 target | Compounds | Pearson correlation |
|---|---|---|
| CDK2 | 16 | −0.785 |
| JNK1 | 21 | 0.003 |
| p38 | 34 | 0.259 |
| TYK2 | 16 | 0.567 |
| Overall | 87 | 0.062 ± 0.089 |
The negative CDK2 correlation contributes substantially to the low overall score. As a diagnostic, the compound-weighted correlation across the other three targets is 0.253. Resampling the four target assays gives an SE of 0.215, close to the 0.218 in the submission. Resampling compounds within the four fixed assays, as in Point (2), gives the SE of 0.089 shown here.
We mirror the same hyperparameters as specified in the AttentionDTA code (https://github.com/zhaoqichang/AttentionDTA_TCBB/blob/main/AttentionDTA_main.py). We use the same max epochs, learning rate, learning rate schedule, patience, loss, model size, and stopping criterion. We trained the model with a maximum of 500 epochs and patience set to 50. Training stopped at epoch 86. We monitored the training loss and saw it decrease from 5 to 0.3 over the course of training. All 87 FEP+4 compounds received noncollapsed, nonconstant predictions. The large spread in standard error and low performance is because CDK2 has a very poor Pearson correlation, dragging down the overall performance on FEP+4.
Response to Reviewer uWrZ, Part 1
We thank the reviewer for their feedback. We appreciate that reviewer recognizes the practical importance of our work and the need for faster methods of binding affinity predictions. We hope to address each of their concerns and questions below:
1. Methodological novelty and contributions of protein input and ligand input diversity.
We agree that molecular fingerprints, MLPs, pairwise losses, and ensembling are established techniques. Our contribution is empirical. As we discuss in Point (2), STORM achieves competitive affinity prediction with Boltz-2 at a substantially lower computational cost, while exceeding the performance of previously reported structure-free methods. We provide further results to help quantify how much the protein inputs and ligand representation diversity we apply in STORM contribute to its performance.
We complete two additional ablation comparisons. We first compare STORM to a matched ligand-only STORM ensemble designed to preserve the ligand representations and model count while removing protein features. We then compare the diverse ligand-only ensemble with 15 independently trained fits that use the same RDKit path-count fingerprint representation, the highest-scoring standalone STORM component by mean Pearson correlation across OpenFE and ChEMBL35, as described in Appendix D. For all ablations below we use the same training loss, training data, 15 MLPs, 18 votes, and the same assay graph smoothing procedure.
Scores are means of compound-weighted within-assay Pearson correlation +/- one bootstrap standard error, across three seeds. We used 50,000 within-assay compound resamples, sharing each draw across all seeds and ablation conditions and averaging the three seed-level scores within each replicate. Protein-input gain is STORM minus matched ligand-only STORM. Ligand-diversity gain is matched ligand-only STORM minus the repeated ligand-only ensemble. The paired bootstrap-t intervals are unadjusted. Bold values are significant under Holm correction across 16 ablation contrasts.
Both gains are significant on ChEMBL35 (adjusted p=0.00064 each). On OpenFE, the protein-input gains are significant (adjusted p=0.00616), while the ligand-diversity gain is inconclusive (adjusted p=0.127). Both contrasts are inconclusive on FEP+4 and CASP16, which feature substantially fewer protein targets. On OpenFE, protein inputs improved within-assay Pearson correlation by 0.0464 even on protein targets that were held out under the protocol detailed in Section 4.1. As detailed in Point (2), STORM also achieves significantly higher Pearson point estimates than the retrained structure-free baselines in all 20 comparisons.
We will summarize these results in Section 4.3, document the statistical procedure in Appendix D, and clarify the empirical contribution in the Introduction and Conclusion.