acceptodds
Under review as a conference paper at ICLR 2027

STORM: A Lightweight Structure-Free Model for Binding Affinity Prediction

Abstract

Accurately predicting a small molecule’s binding affinity to a protein, a critical task in drug discovery, remains challenging. Physics-based approaches like free-energy perturbation (FEP) have very high computational costs and require expert setup. Recent machine learning–based cofolding models have reported accuracy approaching that of FEP—and at least one of them, Boltz-2, has already been widely adopted for affinity prediction—but they remain expensive for large computational screens and even more expensive for retraining on newly collected experimental data. To address these challenges, we introduce STORM: Similarity Transfer from Observed Reference Molecules. STORM is a fast, simple machine learning model that ensembles many small model fusions of diverse protein and molecule representations for binding affinity prediction. Trained on the same data, STORM matches or exceeds the accuracy of Boltz-2 on multiple evaluation datasets, while delivering a speed improvement of 6 orders of magnitude for both inference and training. Our results also suggest opportunities to substantially improve affinity prediction accuracy by integrating historical experimental affinity data with structural and biophysical information.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
binding affinity prediction, drug-target affinity, binding free energy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uWrZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper presents STORM, a lightweight, structure-free model for protein–ligand binding affinity prediction. It combines 15 small MLPs that use different molecular fingerprints, descriptors, and protein features, including target IDs and functional annotations. The models learn both individual activity values and activity differences between compounds in the same assay. Their predictions are averaged and then smoothed using molecular similarity. STORM is trained on ChEMBL and BindingDB data and evaluated on four affinity benchmarks. The authors report performance close to Boltz-2 at much lower computational cost, along with gains over several structure-free models. They also show that fine-tuning on small amounts of target-specific data can improve predictions for new targets.

Critical strengths
  • This paper addresses a useful practical question about the cost of affinity prediction. STORM uses standard molecular features, protein annotations, and small MLPs without requiring 3D structures or pretrained language-model embeddings. The reported results suggest that this simple approach can achieve useful accuracy at a low computational cost.
Critical weaknesses
  • The method has limited novelty, and the source of its gains is unclear. STORM combines existing molecular representations and protein annotations using small MLPs, weighted ensembling, and similarity-based smoothing. This may be useful, but the paper does not clearly show a new way to transfer affinity information across targets. The ensemble comparison changes both the diversity of ligand representations and the use of protein information at the same time. It therefore cannot show how much protein information contributes to the improvement. A ligand-only ensemble with the same molecular representations and model count would help answer this question. Without this comparison, the paper provides limited insight beyond showing that a particular combination of existing methods performs well.
  • The comparisons do not provide enough support for the main performance claims. The comparison with Boltz-2 uses results from previous studies rather than running the model under the same evaluation setup. For ChEMBL35, the authors calculate an overall score by assuming that the original report uses compound-weighted correlations. It is therefore unclear whether the two models’ scores were calculated in the same way. At least one structure-based affinity predictor should be directly tested on the same data using the same evaluation procedure.
Questions for authors
  • Can you confirm whether the Boltz-2 ChEMBL35 results use the same compound-weighted correlation as STORM? To support the main performance claim, can you directly evaluate Boltz-2, or another structure-based affinity predictor, on the same test data using the same evaluation procedure?
  • What explains AttentionDTA’s low FEP+4 correlation and large standard error (r = 0.062 ±\pm 0.218)? Please provide the correlations for the four individual targets and explain how the training settings and final checkpoint were selected. What checks were used to confirm that the model was properly trained?
  • How much of STORM’s improvement comes from protein information rather than the use of different ligand representations? Can you compare STORM with a ligand-only ensemble using the same molecular representations, model count, loss, and smoothing? This would help determine whether the proposed transfer across targets adds value beyond a diverse ligand-only ensemble.
6 comments
AuthorsOpenReview ↗

Response to Reviewer uWrZ, Part 1

We thank the reviewer for their feedback. We appreciate that reviewer recognizes the practical importance of our work and the need for faster methods of binding affinity predictions. We hope to address each of their concerns and questions below:

1. Methodological novelty and contributions of protein input and ligand input diversity.

We agree that molecular fingerprints, MLPs, pairwise losses, and ensembling are established techniques. Our contribution is empirical. As we discuss in Point (2), STORM achieves competitive affinity prediction with Boltz-2 at a substantially lower computational cost, while exceeding the performance of previously reported structure-free methods. We provide further results to help quantify how much the protein inputs and ligand representation diversity we apply in STORM contribute to its performance.

We complete two additional ablation comparisons. We first compare STORM to a matched ligand-only STORM ensemble designed to preserve the ligand representations and model count while removing protein features. We then compare the diverse ligand-only ensemble with 15 independently trained fits that use the same RDKit path-count fingerprint representation, the highest-scoring standalone STORM component by mean Pearson correlation across OpenFE and ChEMBL35, as described in Appendix D. For all ablations below we use the same training loss, training data, 15 MLPs, 18 votes, and the same assay graph smoothing procedure.

BenchmarkSTORMMatched ligand-only STORMRDKit path-count ensembleProtein-input gain [95% CI]Ligand-diversity gain [95% CI]
ChEMBL 35 Time Split0.3530 ± 0.00690.2910 ± 0.00700.2499 ± 0.0072+0.0619 [+0.0554, +0.0707]+0.0411 [+0.0327, +0.0508]
OpenFE0.5813 ± 0.02280.5349 ± 0.02400.5011 ± 0.0238+0.0464 [+0.0191, +0.0658]+0.0338 [+0.0083, +0.0755]
FEP+40.6941 ± 0.05380.6727 ± 0.05510.6445 ± 0.0543+0.0215 [−0.0144, +0.0575]+0.0282 [−0.0169, +0.0781]
CASP160.6678 ± 0.04460.6515 ± 0.04550.6537 ± 0.0458+0.0163 [−0.0105, +0.0449]−0.0022 [−0.0464, +0.0436]

Scores are means of compound-weighted within-assay Pearson correlation +/- one bootstrap standard error, across three seeds. We used 50,000 within-assay compound resamples, sharing each draw across all seeds and ablation conditions and averaging the three seed-level scores within each replicate. Protein-input gain is STORM minus matched ligand-only STORM. Ligand-diversity gain is matched ligand-only STORM minus the repeated ligand-only ensemble. The paired bootstrap-t intervals are unadjusted. Bold values are significant under Holm correction across 16 ablation contrasts.

Both gains are significant on ChEMBL35 (adjusted p=0.00064p = 0.00064 each). On OpenFE, the protein-input gains are significant (adjusted p=0.00616p = 0.00616), while the ligand-diversity gain is inconclusive (adjusted p=0.127p = 0.127). Both contrasts are inconclusive on FEP+4 and CASP16, which feature substantially fewer protein targets. On OpenFE, protein inputs improved within-assay Pearson correlation by 0.0464 even on protein targets that were held out under the protocol detailed in Section 4.1. As detailed in Point (2), STORM also achieves significantly higher Pearson point estimates than the retrained structure-free baselines in all 20 comparisons.

We will summarize these results in Section 4.3, document the statistical procedure in Appendix D, and clarify the empirical contribution in the Introduction and Conclusion.

AuthorsOpenReview ↗

Response to Reviewer uWrZ, Part 2

2. Evaluating significance of accuracy compared to other methods.

We directly test Boltz-2 on our same evaluation procedure. We ran Boltz-2 using all of the exact hyperparameters detailed in v2.2.1 [3], with the exception of toggling on affinity molecular weight correction. We noticed an increase in compound-weighted Pearson r, so we kept the stronger baseline. We scored all local models on the same available rows using the compound-weighted within-assay Pearson correlation described in Section 4.3. The Boltz-2 paper specifies compound weighting for its own affinity evaluations [1].

ModelChEMBL35 Time SplitOpenFEFEP+4CASP16
STORM0.355 ± 0.0070.576 ± 0.0240.705 ± 0.0530.671 ± 0.045
Boltz-2 (local)0.303 ± 0.0070.570 ± 0.0240.675 ± 0.0620.640 ± 0.052
Boltz-2 (published references)≈0.341 [2]0.620 [1]0.660 [1]0.650 [1]
Structure-free DTA0.238 ± 0.0080.328 ± 0.0300.441 ± 0.0920.380 ± 0.075
AttentionDTA0.157 ± 0.0080.244 ± 0.0310.062 ± 0.0890.580 ± 0.049
PSICHIC0.142 ± 0.0080.381 ± 0.0290.518 ± 0.0720.565 ± 0.054
BALM0.210 ± 0.0080.328 ± 0.0310.481 ± 0.0860.387 ± 0.066
Mattsson-Walters0.230 ± 0.0080.264 ± 0.0320.638 ± 0.0730.252 ± 0.085

Due to computational cost, we present results for Boltz-2 obtained for a single seed (2023). We show comparisons to all other methods in a matched seed setting, comparing to predictions from a seed 2023 run on STORM. All models were scored on the same within-assay correlations and the +/- values are one within-assay bootstrap standard error. We note that our results are lower than the publicly reported Boltz-2 numbers, but fall within the publicly reported error bar ranges. All settings were defaults except the molecular-weight correction, which substantially improved Boltz-2 performance.

On the ChEMBL 35 Time Split, STORM scored 0.0521 higher than our evaluation of Boltz-2 (unadjusted paired 95% bootstrap-t CI [+0.0359, +0.0691], Holm-adjusted p=0.00288p = 0.00288). The difference persisted on the 426 assays with complete Boltz-2 coverage (0.3583 versus 0.3008). The other three Boltz-2 comparisons were inconclusive after correction. STORM had higher Pearson point estimates in all 20 comparisons with the retrained structure-free baselines. We used 50,000 paired within-assay bootstrap replicates and applied Holm correction across 72 tests spanning three metrics, four benchmarks, and six methods [4].

We will add the matched comparison to Section 4.3 and report the full paired tests and inference settings in the appendix.

[1] Passaro et al., Boltz-2, Section 5

[2] Isomorphic Labs, IsoDDE technical report, Figure 8

[3] Boltz-2 v2.2.1 prediction options and molecular-weight correction implementation

[4] Holm, A Simple Sequentially Rejective Multiple Test Procedure

3. Compouned-weighted correlations for Boltz-2 on ChEMBL 35 versus STORM.

Yes, we do confirm that Boltz-2 uses the same compound-weighted Pearson correlation as STORM for the results reported by the Boltz-2 paper (the CASP16, FEP+4, and OpenFE reported Boltz-2 results). Please see Section 5 ("Evaluation"), where the Boltz-2 manuscript states that: “For the affinity measurements, all cross-assay averages are weighted by the number of compounds in the assay" of [1]. The approximately 0.341 ChEMBL 35 Time Split reference in our submission comes from aggregating the per-class Boltz-2 scores in IsoDDE Figure 8. The figure leaves the within-class weighting unspecified, so our reconstruction assumes compound weighting, as described in submitted Appendix E [2].

In our direct comparison in Point (2), in our evaluation of Boltz-2, we calculated Pearson correlation separately within each assay and averaged the assay correlations using the number of evaluated compounds as weights, also following the same procedure as [1]. STORM and Boltz-2 were scored on the same evaluation data with the same assay groupings and weights. We confirm all of our evaluations of Boltz-2 on all benchmarks are scored with an identical procedure.

AuthorsOpenReview ↗

Response to Reviewer uWrZ, Part 3

4. AttentionDTA's low FEP+4 correlation and large standard error.

Here are the individual target results for AttentionDTA on FEP+4.

FEP+4 targetCompoundsPearson correlation
CDK216−0.785
JNK1210.003
p38340.259
TYK2160.567
Overall870.062 ± 0.089

The negative CDK2 correlation contributes substantially to the low overall score. As a diagnostic, the compound-weighted correlation across the other three targets is 0.253. Resampling the four target assays gives an SE of 0.215, close to the 0.218 in the submission. Resampling compounds within the four fixed assays, as in Point (2), gives the SE of 0.089 shown here.

We mirror the same hyperparameters as specified in the AttentionDTA code (https://github.com/zhaoqichang/AttentionDTA_TCBB/blob/main/AttentionDTA_main.py). We use the same max epochs, learning rate, learning rate schedule, patience, loss, model size, and stopping criterion. We trained the model with a maximum of 500 epochs and patience set to 50. Training stopped at epoch 86. We monitored the training loss and saw it decrease from 5 to 0.3 over the course of training. All 87 FEP+4 compounds received noncollapsed, nonconstant predictions. The large spread in standard error and low performance is because CDK2 has a very poor Pearson correlation, dragging down the overall performance on FEP+4.

Reviewer_kkoFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces STORM, a machine learning model formed of an ensemble of 15 MLPs, for predicting the binding affinity between a small molecule and a protein that does not compute the cofolded 3D structure, making it faster at both training and inference than methods like Boltz-2. The authors show performance that is competitive with that of Boltz-2, with higher speed and lower computational cost.

Critical strengths

The problem that the authors aim to solve is interesting and important. Binding affinity prediction does remain an unsolved problem that is crucial to drug discovery, and speed is important for applications like virtual screening.

Every structure-free baseline was retrained on the author's own corpus. This is a significant effort that strengthens the paper.

The model is indeed extremely lightweight for both training and inference.

The figures are good, and the results are well laid out.

Reproducibility is appreciated, and the authors do provide everything needed to reproduce their results

Critical weaknesses

Overstating results

Some of the results are heavily overstated. For example, “On CASP16, FEP+4, and ChEMBL35, STORM also outperforms Boltz-2” isn't supported by Table 4, which shows no statistically significant improvement over Boltz-2. Ironically, a more modest claim about matching their performance with a much lighter and faster model would have been significant enough. The idea of outperforming Boltz-2 is repeated in other parts of the paper, and never supported by the results.

No protein sequence

I found this very puzzling, but if I understand correctly, the protein sequence never enters the model. As I understand, for proteins not on the training data, the model will basically use sequence similarity to find the most similar target. It seems like this would be an issue especially as we go to out of distribution targets. More interestingly, it is puzzling that the performance matches that of Boltz-2, which does use sequence, though this might be largely explained by my next point

Data Leakage

The data used for evaluation contains very serious leakage from the training data. This was already an issue for Boltz-2. Both CASP-16 targets were seen during training. For ChEMBL, a time split is likely to introduce huge leakage as people try new compounds against similar or the same targets. For OpenFE / FEP+4, only very similar targets are removed (over 0.9 via mmseqs). This should be much more stringent, and also very similar ligands should also be removed. It is hard to interpret the results seriously while such concerns exist.

Inaccuracies

The text makes several unfounded or incorrect statements. For example, it says ISODDE uses a similar trunk + affinity prediction architecture as Boltz-2. This is not stated anywhere in the ISODDE report, so if true, it is information that is not in the public domain. They also say ISODDE does not disclose the composition of their training data, but their report does specify a 2023 cutoff and removing targets by sequence similarity, similarly to what this paper does, as well as some other cutoffs.

Writing

Not a major issue, but the writing is too informal for the venue (things like "makes it highly accessible for researchers everywhere", “extremely fast”). Also, claims like “on a laptop in minutes” are not proven (experiments are only disclosed on an L4) and it's again very informal and unscientific. There are other typos, such as [CITE?] in the introduction, ICLR 2026 on a header, "relase", “course grained,”

Questions for authors

Can you report results on all four benchmarks stratified by the maximum Tanimoto similarity of each test ligand to ligands in the training set for that target?

For the OpenFE and FEP+4 benchmarks, what is the distribution of identity similarities to their nearest neighbours in the training set after the 0.9 mmseqs removal? And could you provide numbers for different subsets based on similarity (e.g. similarity between 0.8 and 0.9, then separately similarity between 0.7 and 0.8, etc.)

When you say fine-tuning can be done "on a laptop in minutes" could you be more specific?

9 comments
AuthorsOpenReview ↗

Response to Reviewer kkoF, Part 1

We thank the reviewer for recognizing the importance of fast affinity prediction for virtual screening, the effort to retrain the structure-free baselines on a common corpus, and the reproducibility of our evaluation. We address their concerns and questions below.

1. Significance of accuracy improvements relative to Boltz-2.

We agree that Table 4 does not establish STORM outperforms the published Boltz-2 results.

The reviewer is correct that when comparing the error bars shown in Table 4, STORM is not clearly a statistically significant improvement when compared to publicly reported Boltz-2 Pearson r correlation values. We have updated the language in our draft to more clearly reflect this.

However, we have also now completed follow up analysis by running Boltz-2 on all of our benchmark assays that we evaluated STORM on. In these evaluations, we see that STORM's performance exceeds Boltz-2's performance when evaluated with a paired 95% confidence interval on the ChEMBL 35 Time Split. On the other three benchmarks, STORM and Boltz-2's performance are within error. In these evaluations, Boltz-2 was run mirroring all the settings for inference for the Boltz-2 v2.2.1 release (https://github.com/jwohlwend/boltz/blob/v2.2.1/docs/prediction.md#options), with ColabFold MSAs. However, we set affinity molecular weight correction to True, as we found that toggle improved Pearson r by 0.012 to 0.055. We thus include the stronger baseline, and we have also updated the results in our draft to match. We show 95% bootstrap-t confidence intervals; significance is assessed using Holm-adjusted p < 0.05. These analyses hold the fitted models, assay set, and saved graph-smoothed predictions fixed, and use 50,000 paired within-assay resamples. Bold indicates statistical significance at Holm-adjusted p < 0.05 across the same 72 comparisons reported in our response to Reviewer U2k4. STORM values do not precisely line up with the numbers reported in the main text table since we use a single seed 2023 for the bootstrap-t intervals, and use the same seed 2023 for Boltz-2.

BenchmarkSTORMPublished Boltz-2 referenceLocal Boltz-2STORM − local Boltz-2 [paired 95% CI]
ChEMBL35 Time Split0.35490.34090.3028+0.0521 [+0.0359, +0.0691]
OpenFE0.57620.6200.5696+0.0066 [−0.0516, +0.0560]
FEP+40.70480.6600.6753+0.0295 [−0.0655, +0.1386]
CASP160.67060.6500.6403+0.0302 [−0.0528, +0.1101]

These scores are compound-weighted within-assay Pearson r, and bold results are significant.

Independent of whether our model matches or exceeds Boltz-2's publicly reported performance, we also agree with the reviewer that providing comparable accuracy to Boltz-2 with a much simpler and faster model is a significant contribution. We hope to highlight this point more clearly in our draft, and in our final camera ready version of our manuscript.

AuthorsOpenReview ↗

Response to Reviewer kkoF, Part 2

2. No protein sequence.

We hope to clarify the reviewer's confusion on how STORM represents proteins and uses a provided protein sequence. STORM uses a provided input protein sequence to obtain InterPro terms, GO terms, and to assign a learned ID representation.

For all input protein sequences, regardless of whether the protein was or was not observed during training, STORM uses the sequence to query the official InterPro database (https://www.ebi.ac.uk/interpro/) for its matching InterPro terms and GO terms. By default, for training and evaluation proteins the canonical UniProt accession sequence and its corresponding InterPro and GO terms are used, though in principle the official InterPro database or the InterProScan software (https://pmc.ncbi.nlm.nih.gov/articles/PMC13221978/) could be used to get matching InterPro and GO terms from any input sequence. This process does not involve a sequence similarity search to the nearest training target.

However, sequence similarity is sometimes used for assigning learned ID representations to new protein targets that were not observed during training. Targets that are present during training each get a unique learned ID representation. When a new protein target is evaluated, if its UniProt accession was not present in the training data, a full sequence similarity search is conducted against all training protein targets, and the learned ID representation from the protein target with the highest joint sequence identity and coverage is assigned as the input representation for the new protein target.

We agree with the reviewer that our result that this simple set of target representations is enough to match Boltz-2 performance and exceed the performance of previous structure-free methods is surprising. We interpret our results as suggesting that although InterPro terms and GO terms are very broad characterizations of targets, they encode strong expert knowledge guided biological priors on the functional behavior of various targets that is connected to but distinct from sequence identity or similarity based relationships alone. For example, kinases may be evolutionarily quite diverse, but will still feature similar InterPro and GO term annotations, and in practice their binding pockets often share similar attributes due to convergent evolution. We also think that this result provides an important direction for future research, since it suggests that current cofolding or structure-based models may not be fully using the rich sequence information or structural information they receive for binding affinity prediction.

To show the importance of protein information, we have run an ablation comparing STORM to a ligand-only model. To construct the ligand only model, we take all of the STORM MLPs, and remove their target representations, but train them as independent MLPs (with the ligand part of their representations kept), and finally ensemble them. Voting and smoothing still stay the same. We then calculated a paired t-test, with Holm correction, bootstrapped 50,000 replicates to obtain error bars, and report 95% confidence intervals.

BenchmarkSTORM (mean ± bootstrap SE)Ligand-only control (mean ± bootstrap SE)STORM − ligand-only [paired 95% CI]
ChEMBL35 Time Split0.3530 ± 0.00690.2910 ± 0.0070+0.0619 [+0.0536, +0.0682]
OpenFE0.5813 ± 0.02280.5349 ± 0.0239+0.0464 [+0.0280, +0.0725]
FEP+40.6941 ± 0.05440.6727 ± 0.0561+0.0215 [−0.0133, +0.0581]
CASP160.6678 ± 0.04480.6515 ± 0.0458+0.0163 [−0.0116, +0.0439]

On ChEMBL35 and OpenFE, the largest and most diverse benchmarks, we find that the protein sequence is indeed useful and enables a statistically significant gain. On the smaller FEP+4 and CASP16 benchmarks, there still is a gain, albeit smaller, and the confidence interval does span to zero. We believe these results show that our protein representations are important to STORM's performance.

We have updated our draft Appendix to further clarify these points, and will update our final camera ready manuscript with additional discussion on our target representations.

AuthorsOpenReview ↗

Response to Reviewer kkoF, Part 3.1

3. Data leakage.

We appreciate the reviewer's concerns around leakage from the training data. Demonstrating generalization to new protein targets and to new small molecules is an open challenge in the field of binding affinity prediction, but we note that standard benchmarks in the field do not provide optimal settings for understanding either target or small molecule generalization. As per the reviewer's request, we provide additional analysis tables that examine how STORM's performance changes across different levels of small molecule similarity to the training dataset and target protein similarity to the training dataset. In order to provide a direct comparison to how a structure-based method also generalizes, we complete a matching analysis of the results from our evaluation of Boltz-2, conducted with the publicly reported defaults for running Boltz-2, version 2.2.1 (https://github.com/jwohlwend/boltz/blob/v2.2.1/docs/prediction.md#options), with ColabFold MSAs. However, we set --affinity_mw_correction = True, as we found that toggle improved Pearson r by 0.012 to 0.055. We thus include the stronger baseline. We also emphasize that the similarity being measured here is similarity to the STORM curation of binding affinity training data. While we attempt to match Boltz-2's training data curation, we note that Boltz-2 has seen additional sources of training data that might increase the potential similarity of evaluation compounds or targets to its training corpus.

First, we report trends in STORM and Boltz-2 accuracy when considering the maximum similarity of each evaluation molecule to any training molecules at the same target.

BenchmarkModel<0.400.40–<0.600.60–<0.90≥0.90No same-target training ligands
ChEMBL35 Time SplitSTORM0.269 ± 0.0140.298 ± 0.0140.386 ± 0.0140.146 ± 0.1080.353 ± 0.021
ChEMBL35 Time SplitBoltz-2 (MW corrected)0.264 ± 0.0130.224 ± 0.0140.331 ± 0.0150.372 ± 0.1060.358 ± 0.023
OpenFESTORM————0.576 ± 0.024
OpenFEBoltz-2 (MW corrected)————0.570 ± 0.024
FEP+4STORM————0.705 ± 0.053
FEP+4Boltz-2 (MW corrected)————0.675 ± 0.062
CASP16STORM0.598 ± 0.0850.565 ± 0.0940.799 ± 0.088——
CASP16Boltz-2 (MW corrected)0.507 ± 0.0960.506 ± 0.0850.681 ± 0.155——

Number of assays by same-target ligand similarity bin

Benchmark<0.400.40–<0.600.60–<0.90≥0.90No same-target training ligands
ChEMBL35 Time Split2272512121856
OpenFE000058
FEP+400004
CASP1622100

Then, we report trends in STORM and Boltz-2 accuracy when considering the maximum similarity of each evaluation molecule to any training molecules at any target.

BenchmarkModel<0.400.40–<0.600.60–<0.90≥0.90
ChEMBL35 Time SplitSTORM0.168 ± 0.0270.288 ± 0.0110.348 ± 0.0110.450 ± 0.024
ChEMBL35 Time SplitBoltz-2 (MW corrected)0.177 ± 0.0270.257 ± 0.0110.283 ± 0.0130.396 ± 0.025
OpenFESTORM0.460 ± 0.1000.474 ± 0.0610.599 ± 0.0380.748 ± 0.049
OpenFEBoltz-2 (MW corrected)0.347 ± 0.0910.500 ± 0.0590.599 ± 0.0400.689 ± 0.046
FEP+4STORM——0.727 ± 0.0610.837 ± 0.093
FEP+4Boltz-2 (MW corrected)——0.659 ± 0.0890.881 ± 0.050
CASP16STORM0.526 ± 0.2260.588 ± 0.0790.773 ± 0.0620.671 ± 0.253
CASP16Boltz-2 (MW corrected)0.625 ± 0.1500.550 ± 0.0830.721 ± 0.1030.625 ± 0.220

Number of scorable assays by ligand similarity bin

Benchmark<0.400.40–<0.600.60–<0.90≥0.90
ChEMBL35 Time Split8734629695
OpenFE10213212
FEP+40033
CASP162221

Similarity thresholds are defined as the Tanimoto similarity of the Morgan fingerprint (ECFP4), radius 2, 2048 bits. Note that for some datasets specific similarity ranges may not have any data.

Overall, We can see that STORM and Boltz-2 perform comparably across the different similarity thresholds, in both cases.

Reviewer_U2k4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents STORM, a lightweight, structure-free model for protein-ligand binding affinity prediction. STORM combines multiple protein and molecular representations through an ensemble of 15 small MLPs, together with pairwise loss and graph smoothing. The method achieves competitive performance with Boltz-2 across four benchmarks while substantially reducing computational costs.

Critical strengths

The method is computationally efficient and has practical value for large-scale binding affinity prediction. The experiments cover multiple benchmarks and include comparisons with several existing methods. The ablation studies support the benefit of combining diverse representations, and the fine-tuning experiments demonstrate potential for adaptation to new targets.

Critical weaknesses
  1. Limited methodological novelty. The main components, including molecular fingerprints, MLPs, pairwise loss, and ensemble learning, are established techniques. The contribution primarily lies in combining existing methods rather than introducing a substantially new methodology.

  2. Insufficient evaluation of binding affinity prediction accuracy. The experiments rely almost exclusively on within-assay Pearson correlation, without reporting absolute prediction errors such as MAE or RMSE. Moreover, the improvements over Boltz-2 are relatively small, and the different evaluation procedures make it difficult to establish a clear accuracy advantage.

  3. Unclear contribution of graph smoothing. The graph smoothing step uses predictions from other compounds in the same evaluation assay, making individual predictions dependent on the test set composition. However, its contribution to the reported performance is not independently evaluated.

Questions for authors
  1. Could the authors provide MAE or RMSE results and clarify whether the improvements over Boltz-2 are statistically significant?

  2. Could the authors report results without graph smoothing and discuss its implications for predicting individual compounds independently?

4 comments
AuthorsOpenReview ↗

Response to Reviewer U2k4, Part 1

We thank the reviewer for recognizing STORM’s computational efficiency and practical value. We also appreciate the reviewer's support of our representation ablations and target-specific fine-tuning results. We hope to address the reviewer’s questions about absolute prediction error, the comparison with Boltz-2, and graph smoothing below with Points (1), (2), and (3).

1. Methodological novelty and contribution.

We agree that molecular fingerprints, MLPs, pairwise losses, and ensemble learning are established techniques. Our key contribution is the empirical finding that STORM's recipe of ensembling small MLPs using complementary protein and molecular representations can achieve competitive performance with Boltz-2 at substantially lower computational cost. To the best of our knowledge, no previous structure-free methods have been shown to achieve performance competitive with Boltz-2.

Under matched training data, STORM achieves significantly higher Pearson correlations than every evaluated structure-free baseline on all four benchmarks. In the paired analysis described in point (2), these differences are statistically significant after Holm multiple hypothesis testing corrections in 18 of the 20 comparisons. As we described in the Related Work, the PSICHIC, BALM, Structure-Free DTA, and Attention DTA manuscripts all report high accuracy in their evaluations and describe a variety of novel architectures, losses, and protein and small molecule representations. In contrast, STORM's simplicity and high accuracy demonstrate a new baseline for future research work that has not yet previously been described. Furthermore, as illustrated in point (2), STORM even achieves significantly higher accuracy than Boltz-2 on the ChEMBL 35 Time Split, the benchmark with the most protein targets and small molecules.

STORM's performance is also a product of the specific design choices around how to incorporate simple protein and ligand representations, ensembling, and additional techniques. In Appendix D (“Diverse representations versus repeated fits of one model”), we compare the diverse STORM ensemble with an ensemble of 15 independent fits of the target-blind RDKit path model, the strongest individual component on average across OpenFE and the ChEMBL 35 Time Split benchmarks. With the same voting and graph smoothing applied to both ensembles, STORM's mean compound-weighted within-assay Pearson r is 0.5814 on OpenFE and 0.3529 on ChEMBL35, compared with 0.5011 and 0.2499 for the repeated-fit ensemble. This comparison supports the contribution of representation diversity beyond averaging repeated fits of the strongest individual component. These results further support STORM as a practical baseline for structure-free binding affinity prediction.

In our revised manuscript, we will summarize the Appendix D comparison in the main results and clarify the empirical contribution in the Introduction and Conclusion. For our updated draft, we've incorporated additional ablations and discussions into the Appendix. Together, the benchmark comparisons and ablation support STORM as a practical structure-free baseline for binding affinity prediction.

AuthorsOpenReview ↗

Response to Reviewer, U2k4, Part 2

2. Evaluation of binding affinity prediction accuracy.

Including MAE and RMSE as Additional Metrics

We agree that MAE and RMSE provide complementary information and have added both metrics to the revised manuscript. To evaluate Boltz-2 using the same records and scoring procedures as STORM, we ran Boltz-2 on all four benchmarks using inference seed 2023 and compared its predictions with the STORM ensemble trained with seed 2023. While our main reported STORM results include evaluation across three seeds, due to computational costs, we could only complete a single seed Boltz-2 evaluation. The STORM–Boltz-2 comparisons in this section show those individual seed runs specifically. the main manuscript’s three-fit STORM averages and the smoothing ablation are reported separately.

Our Boltz-2 evaluation used ColabFold MSA Server generated multiple sequence alignments (MSAs). We followed the documented Boltz-2 v2.2.1 default inference settings (https://github.com/jwohlwend/boltz/blob/v2.2.1/docs/prediction.md#options), with one exception: we enabled --affinity_mw_correction, which is disabled by default. Enabling this option increased Boltz-2’s compound-weighted Pearson correlation by 0.012–0.055 across the four benchmarks. We therefore use the --affinity_mw_correction = True predictions throughout these comparisons to provide the stronger Boltz-2 baseline.

All absolute errors were computed on a common pActivity scale. For OpenFE, FEP+4, and CASP16, we converted the experimental free-energy labels to pActivity using �0��0� at 298.15 K. For the ChEMBL 35 Time Split benchmark we calculate pActivity values from log-concentration labels. Full details are documented in the Appendix.

MAE (pActivity units)

ModelChEMBL 35 Time SplitOpenFEFEP+4CASP16
STORM0.860 ± 0.0041.044 ± 0.0190.948 ± 0.0560.874 ± 0.039
Boltz-21.078 ± 0.0050.856 ± 0.0170.554 ± 0.0420.781 ± 0.051
Structure-free DTA0.967 ± 0.0051.396 ± 0.0231.233 ± 0.0691.029 ± 0.051
AttentionDTA0.904 ± 0.0051.124 ± 0.0210.872 ± 0.0660.831 ± 0.042
PSICHIC0.972 ± 0.0051.121 ± 0.0200.705 ± 0.0510.915 ± 0.044
BALM0.906 ± 0.0051.226 ± 0.0210.919 ± 0.0591.038 ± 0.049
Mattsson-Walters0.942 ± 0.0051.241 ± 0.0211.227 ± 0.0551.632 ± 0.083

RMSE (pActivity units)

ModelChEMBL 35 Time SplitOpenFEFEP+4CASP16
STORM1.066 ± 0.0051.292 ± 0.0231.174 ± 0.0611.011 ± 0.039
Boltz-21.355 ± 0.0061.108 ± 0.0240.702 ± 0.0520.986 ± 0.059
Structure-free DTA1.232 ± 0.0061.710 ± 0.0261.446 ± 0.0741.207 ± 0.054
AttentionDTA1.152 ± 0.0051.401 ± 0.0261.076 ± 0.0730.986 ± 0.044
PSICHIC1.196 ± 0.0051.372 ± 0.0250.859 ± 0.0531.061 ± 0.044
BALM1.149 ± 0.0051.493 ± 0.0261.096 ± 0.0641.225 ± 0.049
Mattsson-Walters1.219 ± 0.0061.526 ± 0.0281.394 ± 0.0551.907 ± 0.077

All models were scored on the same compound measurements. Values are point estimates ± one standard error from paired bootstrap resamples of measurements within assays. Full details are provided in the Appendix.

Evaluating Statistical Significance

To address the reviewer’s question about statistical significance, we performed two-sided paired bootstrap tests for differences in Pearson correlation, MAE, and RMSE. For Pearson correlation, we computed compouned-weighted within-assay correlations. We follow the same procedure implemented in the Boltz-2 manuscript, as reported in Section 5 ("Evaluation") (https://jeremywohlwend.com/assets/boltz2.pdf#page=9). For MAE and RMSE, we computed the errors directly. Full details are available in the Appendix. To control for multiple hypothesis testing, we applied Holm correction jointly across the four benchmarks, three metrics, and six methods. Our table below reports STORM vs Boltz-2 for brevity. Full comparisons with the structure-free baselines are provided in the Appendix.

AuthorsOpenReview ↗

Response to Reviewer U2k4, Part 3

2. Evaluation of binding affinity prediction accuracy, continued

BenchmarkMetricSTORM − Boltz-2 [95% CI]Holm-adjusted pSignificant advantage
ChEMBL 35 Time SplitPearson r+0.0521 [+0.0359, +0.0691]0.00288STORM
ChEMBL 35 Time SplitMAE−0.2179 [−0.2288, −0.2070]0.00288STORM
ChEMBL 35 Time SplitRMSE−0.2891 [−0.3010, −0.2772]0.00288STORM
OpenFEPearson r+0.0066 [−0.0516, +0.0560]1.000None
OpenFEMAE+0.1877 [+0.1403, +0.2347]0.00288Boltz-2
OpenFERMSE+0.1835 [+0.1327, +0.2348]0.00288Boltz-2
FEP+4Pearson r+0.0295 [−0.0655, +0.1386]1.000None
FEP+4MAE+0.3936 [+0.2317, +0.5405]0.00288Boltz-2
FEP+4RMSE+0.4722 [+0.2851, +0.6402]0.00288Boltz-2
CASP16Pearson r+0.0302 [−0.0528, +0.1101]1.000None
CASP16MAE+0.0927 [−0.0070, +0.1904]0.682None
CASP16RMSE+0.0245 [−0.0900, +0.1353]1.000None

Confidence intervals are 95% intervals; significance is assessed using Holm-adjusted p < 0.05. These analyses hold the fitted models, assay set, and saved graph-smoothed predictions fixed.

STORM achieves significantly higher Pearson correlation and lower MAE and RMSE than our Boltz-2 evaluation on ChEMBL35. On OpenFE and FEP+4, Boltz-2 achieves lower MAE and RMSE with statistical significance, while we detect no significant difference in Pearson correlation. On CASP16, we detect no significant difference on any of the three metrics. Against the structure-free baselines, STORM achieves significantly lower MAE and RMSE on ChEMBL 35 Time Split and OpenFE, while the absolute error results are more mixed on FEP+4 and CASP16. Against all structure-free baselines, STORM also achieves significantly higher Pearson correlation. We emphasize within-assay Pearson correlation as our primary metric because it is invariant to batch effects, systematic shifts, and differences in experimental setup. Ross et al (2023), the paper introducing the FEP+4 and OpenFE benchmark, states:

"We are not concerned with the difference in the absolute binding free energies as these can include systematic errors that are ultimately irrelevant for determining which ligands bind stronger than the others. Instead, we evaluated how well pairs of assays agreed in terms of relative binding free energies and rank ordering of the compounds." Therefore, we report compound-weighted Pearson r as our primary metric for comparison. We retain the MAE and RMSE metrics, and add additional tables into the Appendix.

Our reproduced Boltz-2 correlations overlap with the published uncertainty ranges on OpenFE, FEP+4, and CASP16. The ChEMBL 35 Time Split estimate remains lower, and we have not established the cause of that remaining discrepancy.

3. Contribution of graph smoothing.

To quantify how much graph smoothing contributed to STORM's performance, we evaluated three independent fits of the 15-member STORM ensemble with and without graph smoothing.

STORM Graph Smoothing Ablation.

BenchmarkWithout smoothingWith smoothingChange in Pearson correlation
ChEMBL35 Time Split0.3491 ± 0.00700.3530 ± 0.0069+0.0038
OpenFE0.5708 ± 0.02330.5813 ± 0.0228+0.0105
FEP+40.6832 ± 0.05680.6941 ± 0.0538+0.0109
CASP160.6616 ± 0.04500.6678 ± 0.0446+0.0062

Scores are compound-weighted within-assay Pearson correlations averaged across three independently trained ensembles, +/- one within-assay bootstrap standard error.

Graph smoothing resulted in increases to Pearson correlations on all four benchmarks. These gains are statistically significant for our ChEMBL 35 Time Split and the OpenFE benchmarks, where the unadjusted paired 95% bootstrap intervals excluded zero on the ChEMBL 35 Time Split and OpenFE. The bootstrap intervals included zero on FEP+4 and CASP16.

In Section 3.5, graph smoothing moves each ensemble prediction toward a similarity-weighted average of nearby compounds’ predictions. Graph smoothing guides predictions towards the prior expectation that similar molecules are likely to have more similar binding affinities at the same protein targets than less similar compounds. It uses only molecular fingerprints and ensemble predictions. In our benchmarks, each assay defines a query group. Adding or removing a compound can change the neighbors used for smoothing and therefore change another compound’s score. This local similarity assumption can also attenuate activity cliffs, where chemically similar compounds have substantially different measured affinities.

For applications that requires independent scores, users can disable smoothing. A query containing one compound or a compound with no eligible similar neighbors receives its original score. We have added this ablation and its paired bootstrap analysis to Appendix D and clarify these settings in Section 3.5.

AuthorsOpenReview ↗

Follow Up Comment to Reviewer U2k4

Hello Reviewer U2k4, we hope this message finds you well. As we approach the final days of the discussion phase, which concludes on November 18th AoE, we kindly request your feedback on our responses. If you have further questions or need additional clarifications, please do not hesitate to reach out. We greatly appreciate your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank all three reviewers for their feedback. We appreciate the reviewers' recognition of STORM's computational efficiency, simplicity, target-specific finetuning results, and overall value as a practical choice for obtaining accurate binding affinity predictions with low computational cost. We also appreciate the reviewers' recognition of our efforts to enable fair comparisons through retraining all structure-free baselines on a common training corpus, and the reproducibility of our evaluations.

We provide a summary of our responses to the reviewer's questions and concerns below, in addition to our updates to our draft manuscript.

1. Methodological contribution and novelty.

STORM combines various different techniques, including those established in other areas of machine learning. However, our main contribution is the empirical finding that a small ensemble of MLPs using complementary protein and ligand representations can achieve within-assay affinity correlations competitive with Boltz-2 at substantially lower computational cost. STORM also achieves higher Pearson r on all four benchmarks than the structure-free baselines retrained on the same corpus. We have revised the manuscript to emphasize STORM’s value as a practical structure-free baseline.

2. Statistical significance of STORM's accuracy compared to other methods.

We revised the claims based on the published Boltz-2 scores and reran Boltz-2 v2.2.1 on all four benchmarks, scoring both methods on the same assays with the same procedure. In our ChEMBL35 evaluation, STORM significantly outperforms Boltz-2 and the structure-free models, using paired 95% confidence intervals. The confidence intervals included zero on CASP16, OpenFE, and FEP+4. Our Boltz-2 rerun scores were below the published Boltz-2 scores, and enabling affinity molecular weight correction improved them on all four benchmarks. The updated comparison tables in our replies to kkoF and uWrZ show these results in greater detail.

3. Choice of accuracy metrics for benchmarks.

We retain compound-weighted Pearson r as the primary metric because compound prioritization depends on how well predicted activities track measured activities within a series. This metric is insensitive to a constant offset within an assay, which can affect absolute metrics such as MAE. Our reply to Reviewer U2k4 adds MAE and RMSE on a common pActivity scale. STORM has lower absolute errors than Boltz-2 on ChEMBL35, while Boltz-2 has lower errors on the other three benchmarks.

4. Ablations for STORM's protein and ligand representation.

We trained a matched 15 member ligand only ensemble to evaluate the importance of the protein representations. Removing protein features lowered Pearson r on all four benchmarks, with the clearest evidence on ChEMBL35 and OpenFE. On ChEMBL35, the diverse ligand-only ensemble also improved on repeated fits of the best ligand only representation. Together, these comparisons support contributions from both protein information and ligand representation diversity.

5. Ablations for contribution of graph smoothing to STORM's performance.

The ablation study in our reply to Reviewer U2k4 shows statistically significant gains of 0.004–0.011 in ChEMBL35 and OpenFE Pearson r when enabling graph smoothing. Smoothing uses neighboring compounds’ predicted scores, so a compound’s score can depend on which other compounds are queried. Users can disable smoothing to score compounds independently.

6. Training-set overlap and generalization.

The ChEMBL35 time split allows related compounds and targets to recur, and both CASP16 targets fall in the >= 90% training sequence identity bin. OpenFE and FEP+4 use a 90 percent target sequence identity filter. In our reply to Reviewer kkoF, we report STORM and Boltz-2 performance stratified by ligand similarity and target sequence identity. For both methods, the bins measure similarity to STORM’s training corpus. Boltz-2 draws on additional training sources, so its exposure to similar ligands and targets may differ. In the individual reviewer responses, we have clarified how these additional benchmarks relate to generalization.

7. STORM training on laptop.

We corrected the runtime claim. On a 2024 MacBook Pro with an Apple M4 Pro and 24 GB of unified memory, training the full STORM ensemble took approximately 1 hour, 20 minutes using PyTorch’s MPS backend and 2 hours, 5 minutes using only the CPU.

8. Manuscript corrections and clarity.

We corrected the description of the IsoDDE and its training setup, clarified how STORM uses protein sequences to obtain annotations and assign learned target IDs, and addressed the wording issues raised by the reviewers. Our reply to Reviewer uWrZ also reports the per target AttentionDTA results on FEP+4 and describes its training setup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.