- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Instead of directly predicting the EC50 (regression task), this paper casts it to a classification problem by generating the classification label by comparing the sampled concentrations against the experimentally measured EC50. In the inference time, the predicted EC50 can be obtained by fitting a logistic curve on the predicted probability w.r.t many sampled concentrations. The paper evaluated on unseen compounds, proteins, compound cluster, and protein cluster for both the classification task performance and regression task performance (MCC).
Critical strengths
It casted a regression problem to a classification problem such that it can perform both activity classification and potency prediction and obtained better performance in comparison to 4 potency regression baseline and 5 activity classification baselines and ablation studies clarify what does not drive the performance gains.
Critical weaknesses
• The paper evaluates the regression performance (RMSLE) on the intersection of the correctly predicted active pairs of ASMI-DR and the DTA baselines. I assume the reason for this constraint comparison is that ASMI-DR can only estimate EC50 when its predicted response curve crosses certain probabilities, so pairs predicted as inactive have no potency value. However, this excludes false negatives from RMSLE and likely favours easier pairs. Additionally, this is a limitation of the method such that the other DTA baselines do not have. • As the benefit of the approach is that it can perform both classification and regression task, comparing against approaches such as Nesso-1 that can also perform both activity classification and affinity prediction would be helpful. Additionally, the proposed method seems to perform similar or worse than HyperAttnDTI on all the four generalization tasks.
Questions for authors
• Sensitivity analysis between the performance and the sampling procedure. There are a few sampling decisions: number of sampled classification pairs, number and range of the sampled concentrations, sampled decisions when the concentration is close to the ground truth EC50. Understanding the sensitivity of these parameters to the performance is critical to justify the robustness of the model. • Figure 4 is a bit confusing. What do the error bars around the baseline methods represent? Evaluating performance across multiple folds produces a standard deviation, but selecting different thresholds also changes the number of interactions among the correctly predicted active pairs for ASMI-DR and the DTA baselines, which could produce a standard deviation as well. A clarification would be helpful.
5 comments
Part 2
W2: Comparison with Nesso-1 and activity classification performance
We thank the reviewer for suggesting Nesso-1 and have added it to our experiments (Figure 3 and Tables J2–J4).
Similarly to Boltz-2, Nesso-1 predicts protein-ligand binding likelihood and binding affinity, whereas our primary objective is to predict functional receptor activation and its potency. This distinction is particularly important for GPCRs, where ligand binding does not necessarily induce receptor activation. A binder may instead act as an inverse agonist or produce no measurable functional response.
Consistent with this distinction, Nesso-1 achieves very low activity recall (0.033–0.075 across the evaluated settings), identifying only a small fraction of experimentally active pairs. Consequently, its reported potency errors are evaluated on active pairs identified by other models, most of which Nesso-1 itself classifies as inactive. ASMI-DR achieves lower RMSLE in four of five scenarios, with statistically significant improvements in two. The exception is Sequence-Cluster, where the difference is not statistically significant. These results highlight the difficulty of directly transferring binding affinity predictions to functional GPCR activation.
Regarding HyperAttentionDTI, we agree that ASMI-DR does not consistently outperform dedicated activity-classification models. However, our principal contribution is the substantial improvement in EC50 estimation while retaining competitive activity prediction, rather than establishing a new state of the art in activity classification alone, which we now emphasise more in the revised version.
In the i.i.d. setting, ASMI-DR achieves MCC 0.665 compared with 0.656 for HyperAttentionDTI. The differences are also small in three of the four OOD settings: Sequence-Single (0.481 vs. 0.480), Sequence-Cluster (0.218 vs. 0.232), and Molecule-Single (0.593 vs. 0.608). We acknowledge a larger difference in Molecule-Cluster (0.398 vs. 0.482), where HyperAttentionDTI performs better.
Unlike HyperAttentionDTI, however, ASMI-DR simultaneously provides an EC50 estimate for predicted active pairs, substantially outperforming the dedicated regression baselines in EC50 estimation. We therefore view the competitive classification performance, combined with the substantial improvements in EC50 prediction, as the main practical advantage of the proposed framework.
Part 3
Q1: Sensitivity to concentration sampling and hyperparameters
We thank the reviewer for highlighting the importance of sensitivity to the sampling procedure. The original Appendix F already included several relevant ablations, and we have now made these analyses more explicit in Section 6.3 in the revised text.
We evaluated the sensitivity to the main sampling choices as follows:
- Transition margins (Table F1): Varying over yields stable performance, with i.i.d. RMSLE between 0.708 and 0.755 and MCC between 0.652 and 0.673.
- Soft-label generation (Table F2): We compare uniformly sampled soft labels with linear interpolation across the transition region. Both retain substantially better EC50 estimation than the DTA regression baselines.
- Concentration sampling (Table F3): We compare uniform sampling with a strategy that preferentially samples concentrations near the measured EC50. Both yield comparable performance, indicating that the improvement does not depend on concentrating training samples around the transition region.
- Sample weighting (Table F4): We additionally compare dynamic concentration-dependent class weights with standard class-imbalance weights, finding that the formulation remains effective under both strategies.
Regarding the number of concentrations used at training, this is adapted by the best checkpoint selection. During training, one concentration is sampled per protein-molecule pair in each epoch, with a new concentration drawn in subsequent epochs. Thus, the number of sampled concentrations per pair over the full training is given by the index of the best training epoch.
As for the number of concentrations used at inference, Appendix G evaluates this trade-off explicitly. Reducing the number of queried concentrations from 62 to only 7 changes RMSLE by less than 0.015, substantially reducing inference cost without materially affecting performance.
Finally, the training concentration range is not a free hyperparameter, and it is determined by the concentrations tested in the underlying experimental assays.
Taken together, these experiments demonstrate that the improvements in EC50 estimation are robust to the main choices introduced by the concentration-conditional formulation, supporting our conclusion that the gain primarily originates from the learning formulation rather than a particular sampling configuration.
Conclusion
We hope that these additional results and clarifications address the reviewer’s concerns about the evaluation protocol, the comparison with joint activity-affinity models, and the robustness of the proposed approach.
Part 1
We thank the reviewer for their constructive feedback and for recognising the advantages of jointly predicting activity and potency. We have expanded the evaluation to address the concerns about the selection of active pairs, added the Nesso-1 comparison, and clarified the sensitivity of our approach to sampling hyperparameters. We address each concern below. The changes in the revised pdf are highlighted in blue.
W1 & Q2: EC50 evaluation, false negatives, and recall–RMSLE trade-off
We agree with the reviewer that evaluating RMSLE on the intersection of correctly predicted active pairs reduces the evaluation set and may select easier interactions. However, this evaluation is intentional: it isolates EC50 estimation accuracy from activity classification by comparing all models on the same successfully identified active pairs. False negatives are not ignored, but accounted for separately through MCC and recall.
We would also like to emphasise that we consider the absence of a reported EC50 estimate for pairs classified as inactive by ASMI-DR to be a desirable property of our formulation rather than its limitation. Experimentally, EC50 is only defined when a measurable response is observed and a dose-response curve can be fitted. Although regression baselines produce a numerical EC50 for every pair, this does not necessarily imply that their estimates for pairs classified as inactive are informative.
In fact, for the regression baselines, the activity decision is based directly on the estimated EC50, with predictions above 10−1M classified as inactive. Since all experimentally measured EC50 values in our test sets are at most 10−2M, every false-negative prediction necessarily has an EC50 error exceeding one log unit by construction. Evaluating these predictions together with correctly identified active pairs would therefore conflate activity-classification failures with the accuracy of potency estimates.
To investigate whether these numerical predictions nevertheless retain meaningful potency information, we additionally evaluated the regression baselines separately on their true positives (TP) and false negatives (FN):
Table 1: EC50 estimation on experimentally active pairs in the i.i.d. setting, separated by each model’s activity decision. Counts are averaged over five splits. ASMI-DR does not report potency estimates for predicted inactive pairs.
While the regression baselines achieve Spearman correlations of 0.34–0.43 on correctly identified active pairs, these correlations decrease to 0.01–0.16 on false negatives, accompanied by RMSLE values of 3.52–4.04. The same pattern is observed across all four OOD scenarios, where Spearman correlations on false negatives range from −0.12 to 0.16. These results suggest that although the regression baselines produce numerical estimates for false-negative pairs, these predictions provide little useful information about their relative potency.
To further address the reviewer’s concern about the reduced evaluation set, we have substantially extended Figure 4 to evaluate the recall-RMSLE trade-off across varying activity thresholds for all models, rather than only ASMI-DR. We vary the decision threshold of ASMI-DR over τ∈[0.02,0.8] and the regression baselines over T∈[10−3,10−0.5]M. Each model is evaluated on its own true-positive predictions at each threshold.
The revised Figure 4 shows that ASMI-DR achieves lower EC50 error than the regression baselines across their shared recall ranges in all five evaluation scenarios. In contrast to regression models, whose RMSLE generally increases substantially as recall increases, ASMI-DR maintains relatively stable potency estimation accuracy. Thus, the advantage of ASMI-DR is not simply a consequence of evaluating fewer, easier active pairs.
We thank the reviewer for pointing out the ambiguity in the original Figure 4, which helped us to greatly enhance the clarity for the reader. The uncertainty bands now represent one standard deviation of RMSLE across the five data splits, evaluated independently at each threshold. Changing the threshold changes the selected true-positive pairs and consequently the recall shown on the horizontal axis. The revised caption also explicitly defines Nactive as the mean number of experimentally active test pairs per split.