- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
BASILISC estimates uncertainty in cross-modal single-cell matching using Bayesian siamese networks. It evaluates both same-cell-type prediction and exact cell matching, comparing variational inference, Monte Carlo dropout, ensembles, and deterministic controls. The results show benefits for detecting unseen cell types and some cross-type matching errors, alongside important limitations.
Critical strengths
Uncertainty in cell correspondence is a meaningful practical problem, beyond simply improving matching accuracy.
The experimental design includes deterministic controls, multiple uncertainty methods, held-out cell types and families, and replication across assay types.
The paper clearly reports negative results, including reduced matching accuracy with variational inference and limited novelty detection at the cell level.
Critical weaknesses
The practical benefit of abstention is unclear. Keeping the most certain half removes only 32.8% of cross-type errors. As written, this implies that the retained matches have a higher cross-type error rate than the original set, despite outperforming the deterministic control.
The added value of Bayesian inference is modest. At the cell-type level, error detection essentially ties the deterministic margin. At the cell level, the overall AUROC improvement is only 0.013, and the best matching accuracy comes from the deterministic ensemble.
Questions for authors
Can you report cross-type error rates among retained matches across coverage levels, including comparison with random rejection?
Can a simple combination of deterministic confidence and embedding-based novelty scores match the proposed uncertainty methods?
Were held-out cells excluded from Liam pretraining, and were all thresholds and hyperparameters selected without test labels?
Does uncertainty identify meaningful within-type mismatches, and is the proposed association with transitional states supported by independent biological evidence?
2 comments
Dear Reviewer X3NT,
As the discussion period comes to its end, we would like to briefly draw your attention to our recent interaction with Reviewer RmTW.
Several of the key technical questions you raised overlapped with the concerns raised by Reviewer RmTW, specifically regarding the cross-type error rates during filtering and whether held-out cells were excluded from the Liam pre-training. In our rebuttals, we provided the requested controls, baselines, and clarifications to address these shared points. Reviewer RmTW has since reviewed these additions and noted that the new controls and reported outcomes successfully answered those technical questions.
We have integrated all of these analyses into the revised manuscript and supplement. We hope our responses and the updated results can address your concerns as well.
Response to Reviewer X3NT
We are grateful for the careful review and the valuable feedback. We attempted to address each analysis raised in the review and answer your points in order; the full analyses are in the revised Supplement.
W1 / Q1. We agree with your point on filtering at the cell level. In the cell-level model, filtering on any posterior uncertainty score does not lower the cross-type error share below random at any coverage; the deterministic margin is slightly below it, and filtering on the matcher's own similarity of the assigned pair does better than random on lowering cross-type errors. The reason is what we have already reported in our manuscript: cross-type errors are the confident assignments. We discarded the claim about filtering at the cell level in the manuscript (abstract, contributions, Section 5.2, Discussion; Supplement E.1, Table S12, Figure S11) and reported our findings as a diagnostic of which errors the posterior uncertainty separates from correct matches (Table 1).
At the cell-type level, abstention removes most of the errors at 80% coverage, compared with a fifth for random rejection (Section 5.1, number corrected), and query-level label transfer keeps its error below random from 20% coverage up (below).
W2. We agree that the gain from MFVI is modest, and the revised text states this where it applies. However, where the gain is positive, it has practical use. At the cell-type level, a single sampled score can flag both the model's errors and the cells of a type that was not seen during the matcher's training, whereas no single deterministic score does both as often (Section 5.1). In query-level label transfer, the uncertainty score is higher on the cells whose label is incorrect (Supplement E.4). The manuscript also reports where MC dropout and the deep ensemble perform better than MFVI. For the matching task itself, our ensemble is in the range of CLUE and MatchCLOT, which we trained with their own code and evaluated on the same test set (Supplement E.9). In small-data settings, when we trained cell-level models on a training set that is one tenth of the size of the original one, BASILISC largely retains its accuracy while MatchCLOT's accuracy drops sharply (Supplement E.10). In summary, BASILISC performs comparably to other state-of-the-art methods on the matching task, while outperforming them in low-data regimes in this application, and additionally provides an uncertainty score on the cross-modal correspondence, which other methods do not. This work also presents a comprehensive evaluation protocol that shows where the uncertainty score is useful and where it is not.
Q2. We evaluated the fusion score you suggested: a rank average of the deterministic margin and the embedding-based cosine novelty score, once with equal weights and once with the weight chosen leave-one-type-out. The fusion score detects both the model's errors and the unseen types in as many runs as the mean-field posterior, but in fewer than MC dropout or the deep ensemble. Therefore, we have updated the text to say that this fusion of the two deterministic scores reaches MFVI, but not the other two approximations (Section 5.1; Supplement E.3, Table S14).
Q3. We agree and have clarified the scope: "unseen" refers to the matcher's supervision and not Liam's. Liam was trained on all cell types but in an unsupervised manner, therefore the embeddings are label-free (Supplement B.1). We ran the controls you suggested. (a) Inductive: on a PCA/LSI backbone fit without the held-out type, the gain of psd over the margin stays positive in three of four types but shrinks; (b) on Liam itself refit without the type, the posterior still improves on the margin for two of the four types, but not for the other two, while the same refit on every cell reproduces the released backbone, so we state the result as transductive (Supplement E.4, Tables S16–S17). (c) Query level without label screening: filtering on psd removes about twice the random share of held-out cells at 80% coverage, more than the margin, and keeps the transfer error below random from 20% coverage up, but only when several cells of the novel type are in the query set (Supplement E.4, Table S18). In addition, the Sinkhorn temperatures are now chosen on the validation, and Table 1 does not change (Supplement E.5, Table S20).
Q4. We separated the two types of error as they matter for different use cases of a cross-modal matcher. For label transfer and any analysis at the cell-type level, a same-type partner carries the correct label. For matching at the level of cell state, every error counts, and there the posterior's improvement over the deterministic control is below our predefined threshold. The statement on cells in transition is only kept as a hypothesis in the Discussion (Supplement E.7, Table S22).