- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CellMaster, a collaborative LLM-based agent for cell-type annotation in single-cell RNA-seq analysis. The framework uses three specialized agents for hypothesis generation, marker selection, and expression-based annotation, iteratively refining cluster labels through marker memory, confidence pinning, and abstention. It also supports selective escalation to external annotation tools and human experts. Experiments on nine public datasets, with deeper comparisons on PBMC3k, Liver, and Retina, show improvements over several existing methods. Additional ablation studies and human-in-the-loop experiments examine the contributions of individual components and collaborative annotation.
Critical strengths
The integration of iterative annotation, marker memory, confidence pinning, and expression-based evidence inspection provides a practical framework for single-cell annotation. Allowing uncertain clusters to remain unresolved and supporting targeted sub-clustering or expert intervention are useful features for analyzing ambiguous cell populations. The collaborative interface also makes intermediate annotation decisions and their supporting evidence accessible to users.
Critical weaknesses
-Compared to existing large language model (LLM)-based annotation agents, this approach offers limited innovation. Although CellMaster integrates several practical components, its core architecture is primarily a combination of established techniques—including multi-agent task decomposition, iterative hypothesis refinement, marker-based expression analysis, memory mechanisms, and human-in-the-loop interaction. Other relevant studies, such as CellTypeAgent, scPilot, and MAT-Cell, have also explored agent-based reasoning or multi-agent collaboration methods for single-cell annotation.
-
The nine-dataset benchmark in Table 1 reports CellMaster+AF, which uses CellTypist to annotate unresolved clusters, rather than the reference-free CellMaster-Core. Core is evaluated separately on only three datasets in Table 2, where its advantage over contemporary agents is not consistent; for example, scPilot achieves a CL score of 0.675 on Retina compared with 0.667 for Core. Without Core results on the remaining six datasets, it is difficult to assess whether the proposed reference-free agent consistently generalizes across tissues and how much of the broader benchmark performance depends on external annotation assistance.
-
Section 3.3 acknowledges that the LLM-generated confidence scores are not calibrated probabilities, while the experiments mainly report overall CL scores rather than the accuracy of retained, abstained, or escalated predictions. The specialist-routing experiment in Appendix B (Table 8) is limited to a single 28-cluster Liver dataset, and the human-in-the-loop case studies are largely qualitative. It therefore remains unclear whether the proposed confidence-based decisions reliably identify clusters that should be retained, reconsidered, or referred to specialists.
-Evaluation schemes may fail to fully capture the quality of biological annotations and their generalizability. A primary evaluation approach is based on consistency with the "Cell Ontology": a perfect match is assigned 1.0 point, a direct parent-child relationship match receives 0.5 points, and all other predictions are awarded 0 points. While this metric offers determinism and interpretability, it may not adequately reflect the biological relationships between cell states, nor does it effectively account for variations in the granularity of annotations across the hierarchy.
Questions for authors
-
Table 1 reports CellMaster+AF results across nine datasets, whereas CellMaster-Core is evaluated on only three datasets in Table 2. Could the authors provide Core results without AutoFill for the remaining six datasets to clarify the generalization performance of the reference-free framework?
-
Section 3.3 uses LLM-generated confidence scores for label stabilization and abstention. Could the authors report the annotation accuracy of pinned versus unresolved clusters across iterations, or provide a coverage–accuracy analysis to assess whether the confidence-based decisions reliably identify incorrect or uncertain annotations?
-
Appendix B (Table 8) evaluates specialist routing on only the Liver dataset. How does the selective-escalation strategy perform on the other benchmark datasets? Could the authors compare it with directly applying specialist annotators or AutoFill to clarify whether confidence-based routing provides additional benefits beyond simply incorporating external predictions? -As shown in Supplementary Table 13, the confidence intervals for some of the performance gains—relative to comparable agent-based systems—include zero. How do the authors interpret these results? Could further evaluation be conducted to determine whether these performance improvements are statistically robust?
-How sensitive is CellMaster to the initial resolution parameter used in Leiden clustering? In particular, how does the system perform when rare cell populations are merged into larger clusters? Could the authors provide a more systematic robustness assessment regarding variations in clustering granularity and preprocessing workflows?
CellMaster General Response (Part 1 of 2)
Dear Reviewers,
Thank you for your careful reading and constructive feedback. The core vision of CellMaster is to build scientific AI agents able to test their own hypotheses against data, revise their judgments, and ask for help under uncertainty.
We study these choices in single-cell annotation. Two cell types may share a marker; the agent can check more evidence, revise its label, or ask a scientist for help. We appreciate the reviewers’ recognition of the approach and its practical value.
Corecan leave a cluster unlabeled, investigate it further, or seek expert input.marker memoryand agent roles. Iteration studies motivate a stopping rule.Corescored aboveBiomnion all three datasets in the submitted detailed comparison.Coreand assisted results separately.Major problems we addressed through the rebuttal
Our individual replies address all 14 questions. We consolidate the reviewers’ overlapping concerns into four questions below. Part 2 covers the additional experiments and clarifications.
The reviewers also asked for more direct evidence about
Coreand its annotation decisions.Coreuses expression data and pretrained knowledge without external references at inference. Cell Ontology (CL) scoring gives 1 for an exact match, 0.5 for a direct parent or child, and 0 for a wrong label or an abstention. The main questions and rebuttal responses follow.Coreachieve across all nine datasets withoutAutoFill?Coreresults on the six remaining datasets and separate accounting of CellTypist AutoFill (AF).Corenow covers all nine datasets with five runs each. It reached 0.569 meanCLat 91.1% mean coverage, weighting datasets equally.AFfilled 59 abstentions across 615 cluster–run decisions, raising meanCLto 0.602 at full coverage. Of its additions, 14 were exact, 12 partial, and 33 wrong.expression feedbackimprove annotation?PBMC3k,Liver, andRetina, meanCLwas 0.639 forCoreand 0.568 for a single agent with the same evidence, three rounds, and budget limits. Removing iterativeexpression feedbackreduced the score to 0.532. Removingmarker memoryproduced the largest drop, to 0.483. These scores excludeAF.pinned labels, kept fixed across rounds.Liverlabels retained per run, exact-match error was 24.3% forCore’s ranking and 31.4% for confidence alone. The rankings tied at 9 of 19 onRetina. At round three, 20/54 pinnedLiverlabels and 3/38 pinnedRetinalabels were non-exact across five runs, including partial matches.direct fallbackandconfidence-only referral.Direct fallbackimproved 22 and worsened none.Confidence-only referralimproved 46 and worsened 8, with higher pooled exact accuracy (66.6% vs. 62.8% for CellMaster). With the same acceptance rule, specialist queries on 112 of 290 cluster–run decisions yielded the same labels as queries on all 290.