- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper introduces KA-MIL, a multiple instance learning framework for patient-level classification using single-cell transcriptomics. It uses Gene Ontology text embeddings to guide attention, with regularisation to preserve semantic relationships and additional learnable queries to capture other relevant information. Built on scMILD, the method is evaluated on four clinical cohorts and shows improvements over several MIL baselines. Ablation studies examine the main components, while an analysis of the COVID-19 cohort provides some insight into the learned attention patterns.
Critical strengths
- Effective integration of biological knowledge into MIL.
The main strength of the method is its use of biological knowledge to guide attention without requiring cell-level annotations. The geometry regularisation helps preserve semantic relationships while allowing the queries to adapt to the task. The additional learnable queries are also useful, as predefined biological concepts may not capture all relevant information. Overall, this is an interesting extension of conventional attention-based MIL.
- Encouraging experimental results.
KA-MIL shows promising results, particularly on COVID-19 and nasal swabs, where it clearly outperforms the matched scMILD baseline. The ablation studies also support the contribution of the main components, including geometry regularisation and residual queries. The additional experiments with reduced training data and a more restrictive vocabulary are useful for assessing the robustness of the approach.
Critical weaknesses
- Dependence on transductive preprocessing.
My main concern is that the GO vocabulary is constructed using data from the entire cohort, including test patients. When restricted to training data, performance drops noticeably, sometimes below the baseline. Even this experiment retains cohort-wide preprocessing. A fully inductive evaluation is needed to establish how well the method generalises to new patients.
- Limited validation of biological interpretability.
The attention patterns are interesting, but they do not necessarily show that the learned queries capture the intended biological pathways. The similar performance of random and GO anchors on COVID also raises questions about the contribution of biological knowledge. Independent validation, such as pathway enrichment in highly attended cells, would strengthen the interpretability claims.
Questions for authors
Could the authors evaluate KA-MIL using a fully inductive pipeline, with all preprocessing and vocabulary construction based only on training data? This is particularly important given the performance drops on COVID and NS.
Could the authors validate whether the learned queries actually capture their intended biological processes, for example through gene enrichment analysis of highly attended cells?
Why does KAA provide less consistent improvements with frozen foundation-model features? Does the method depend on jointly training the encoder?
We thank the reviewer for these questions. The additional experiments are complete. We will update the PDF shortly and post a consolidated comment identifying the relevant content.
Q1: Fully inductive evaluation.
A1: We evaluated a fully inductive pipeline on COVID, NS, UC, and Lupus10, using the same splits as the original experiments. HVGs and GO vocabularies are selected using training patients only, and the AE is retrained using training cells only. The training-derived gene set and fitted encoder are then applied to validation and test cells.
All three models were retrained under this pipeline, retaining the original validation-based checkpoint/threshold selection rule. Each dataset row reports test AUC / macro-F1 averaged over its five splits. The final row is the equal-weight mean of these four setting-level results.
For the two cohorts highlighted by the reviewer:
The mean across four settings is highest for KA-MIL on both test AUC (0.9080) and macro-F1 (0.8396). We will include validation results and hyperparameter configurations in the revised PDF.
Q2: Do queries capture their intended biological processes?
A2: Using the original COVID checkpoints, we contrasted pseudobulk RNA expression between the top and bottom 20% of attention-ranked cells within each patient and cell type. Genes were then ranked by a paired expression statistic and evaluated via GSEA-style running-sum enrichment, using patient-level sign-flip permutations and FDR correction.
In a joint analysis of 90 held-out patients, 13/50 queries showed positive enrichment (FDR < 0.05), including inflammatory response (NES = 1.643, q = 0.0234). Averaged across patients, its attention also showed stronger within-cell-type correlation with its RNA gene-set score than the mean correlation of other queries’ attention with that same score (FDR q = 0.0010). For the 14 hospitalized patients in original Fig. 2 (fold 4), humoral/myeloid queries showed positive enrichment (NES = 1.532/1.501; both case-study FDR q = 0.0028).
To address the random-anchor concern, we used three vocabulary-selection seeds to generate three sets of 50 real, low-cohort-relevance GO biological-process terms, approximately matched on ontology depth, gene-set size, detected-gene count and coverage, and description length. We retained the original cohort-wide protocol, training settings, and frozen BioLORD encoder. COVID test macro-F1 fell from 0.7075 to 0.6067 (three-vocabulary mean), a 10.07-percentage-point advantage supporting the value of cohort-relevant descriptions. Per-vocabulary results and matching details will be included in the revised appendix.
Q3: Does KAA depend on jointly training the encoder?
A3: KAA shows potential on fixed representations, while the full framework is designed to further realize this potential through joint adaptation.
On identical frozen scGPT and State Embedding (SE) features, with validation-based configuration selection, KAA achieves equal-weight mean test AUC / macro-F1 of 0.8607 / 0.7750 across eight settings (four datasets × two embeddings), compared with 0.8434 / 0.7529 for full-covariance MixMIL and 0.8572 / 0.7765 for attention MIL. KAA improves over MixMIL on average and remains competitive with attention MIL.
Mechanistically, the architecture supports task adaptation at two levels:
This synergy is directly corroborated by our local-AE freezing ablation (Tab. 3), where freezing the encoder after reconstruction pretraining reduces macro-F1 across all four cohorts. Together, these results confirm KAA's effectiveness on fixed representations and the decisive benefit of joint adaptation. We will clarify these comparisons in the revised manuscript.