- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Given low data regime for biological assay prediction problem, meta-learning could be beneficial by training models using data from multiple assays even though only limited labeled data are available for each assay. The authors argue that the assay heterogeneity could have a negative impact for the meta-learning of bioactivity prediction. They define assay similarity based on target-and description-based relatedness and demonstrate the blindly adding irrelevant assay could have a negative impact to the prediction performance. They formulate a model architecture called MetaHeta that integrates the features embeddings from the target assay and auxiliary tasks of the most relevant assays with attention mechanism. They benchmark on simulated and real-world data for and demonstrate promising results. A retrospective Bayesian optimization experiment also suggests the model can improve compound prioritization.
Critical strengths
- The problem formulation is reasonable and practical to existing challenges in bioactivity prediction. Leveraging information from other assays can potentially improve bioactivity prediction in low-data regime, but not all assays are equally relevant, therefore explicitly modeling for assay heterogeneity should be helpful.
- The experimental evaluation is comprehensive including (1) a controlled synthetic kinase experiment to investigate the effect of increasing task heterogeneity, and (2) evaluation on real-world FS-Mol assays for few-shot bioactivity prediction. The retrospective Bayesian optimization for compound prioritization strengthens the claim as an orthogonal validation.
Critical weaknesses
- There is limited biological validation of the assay relatedness. It’s critical because the assay relatedness is used for auxiliary task suggestion which is central for this paper. The relatedness based on target identity is intuitive, but the relatedness based on text-based assay description requires further investigation. Similarity derived based on the PubMedBERT embedding can be a surrogate and a good starting point for the biological relevance, while it could be affected by how the assays are documented. More analysis would be helpful based on target identify, assay types, experimental conditions, etc.
Questions for authors
- How much of the performance gain comes from including the related auxiliary assays, and how much from the specific MetaHeta architecture?
- To what extent, the degradation in performance comes from increasing more auxiliary data that dilutes the target-specific data rather than the assay heterogeneity? Would it still hurt the performance to include 20 auxiliary tasks even they are all biologically relevant to the assay of interest?
5 comments
Response to Reviewer 4we1 (2/2)
Q1: On the contributions of auxiliary assays and MetaHeta architecture
How much of the performance gain comes from including the related auxiliary assays, and how much from the specific MetaHeta architecture?
In Section 5.4 (Figures 5 and 7), we vary the number of auxiliary assays for both MetaHeta and ICICL-TNP. For both methods, adding a single auxiliary assay substantially improves and reduces NLL relative to using none. Adding more assays yields little further improvement in , while NLL continues to decrease.
The gains from MetaHeta's architecture can be observed in Table 2, by comparing performance of our method directly to that of ICICL-TNP. Given exactly the same auxiliary data, MetaHeta achieves 1.20–1.87 times higher than ICICL-TNP across the four FS-Mol context sizes. Both methods can be viewed as architectural modifications to vanilla TNP that allow them to incorporate auxiliary assays while preserving assay identity. In our FS-Mol experiments, MetaHeta adds only 1,536 parameters to vanilla TNP, all in the input embedder for the one-hot task indicators, while ICICL-TNP adds about 90,000. Preserving assay identity is critical to avoid conflating measurements from auxiliary assays with those from the target assay, as illustrated by the poor performance of RF- in Table 2.
Q2: On whether auxiliary tasks could dilute the target-specific data
To what extent, the degradation in performance comes from increasing more auxiliary data that dilutes the target-specific data rather than the assay heterogeneity?
As discussed in our response to Q1, the ablation in Section 5.4 shows that including auxiliary assays improves predictive performance relative to using none, with no systematic degradation as the number of auxiliary assays increases to five.
Would it still hurt the performance to include 20 auxiliary tasks even they are all biologically relevant to the assay of interest?
We investigate this in the newly added Appendix I (please see the revised PDF). We retrain MetaHeta on the core EPHA meta-dataset , raising the maximum number of auxiliary assays from 5 to 20. We then evaluate the model’s performance as we vary the number of auxiliary datasets included in . As shown in Figure 8, performance generally improves as more auxiliary assays are added, with diminishing gains and no systematic degradation up to 20 assays.
Thank you again for your thoughtful feedback. We hope our responses address your questions, and we are happy to clarify any remaining points. If we have resolved your concerns, we would be grateful if you would consider raising your score.
Thanks for providing the additional experiment that demonstrates the effectiveness on assay retrieval based on their similarity. This proves my assessment that it is easiest to identify relevant assays if there are target involved in the description. The retrieval performance on assays without specific target involvement can be modest. This is understandable, and I encourage the authors to include this in the revised manuscript.
Response to Reviewer 4we1 (1/2)
Thank you for taking the time to review our manuscript. We respond in two parts, first examining the biological information captured by description-based assay relatedness, and then discussing how auxiliary assays and the MetaHeta architecture contribute to predictive performance.
W1: On the biological validation of description-based assay relatedness
Thank you for highlighting this important point, which deepens our understanding of the biological features captured by description similarity. In the newly added Appendix H (please refer to the revised PDF), we analyse what biological and experimental information is captured by the similarity between PubMedBERT embeddings of assay descriptions. Specifically, we use ChEMBL metadata to compare how often FS-Mol tasks and their retrieved auxiliary assays share the same biological target, assay type, assay format, and other experimental characteristics. We compare the agreement rates across all retrieved pairs with those obtained by randomly assigning auxiliary assays to FS-Mol assays 1,000 times. In the table below, we report enrichment as the ratio of the retrieved match rate to the median match rate across the 1,000 random assignments. For clarity, here we omit the 5th and 95th percentiles, which are reported in Table 6.
Overall, assays retrieved by description similarity were about 173 times as likely to share a biological target as randomly paired assays (32.83% vs. 0.19%). Assay type and BioAssay Ontology (BAO) format, which classifies assays by their experimental system (e.g., cell-based or biochemical), were also frequent matches. Both showed more modest enrichment because they also matched often among random pairs. Tissue showed 65-fold enrichment, although tissue matches were rare even among retrieved pairs (1.29% vs. 0.02%). Finally, retrieved pairs were about 21 times as likely to share a ChEMBL source document (28.05% vs. 1.36%), which may reflect shared protocols, documentation conventions, or how assays were grouped for deposition.