acceptodds
Under review as a conference paper at ICLR 2027

CellMaster: Collaborative Cell Type Annotation in Single-Cell Analysis

Abstract

Cell-type annotation remains difficult for rare, transitional, and context-dependent populations because fixed references have incomplete tissue and state coverage. We present CellMaster, an expression-grounded LLM agent that iteratively generates hypotheses, proposes marker panels, inspects cluster-level expression and detection patterns, and assigns labels with auditable rationales. Given a preprocessed scRNA-seq object and short study context, CellMaster-Core annotates without querying an external annotated atlas, fixed marker database, or specialist tool at inference time. Rather than forcing a label, it preserves unresolved clusters for another iteration, targeted sub-clustering, or selective escalation to a specialist or human expert. We evaluate this design with a nine-dataset breadth benchmark and a deeper comparison on three biologically distinct datasets against contemporary agents, zero-shot scGPT, and reference-based specialists. Core outperforms Biomni on each depth dataset, with up to a 17.0% relative increase in Cell Ontology score, while optional CellTypist closure provides smaller additional gains. Under fixed closure, removing marker memory causes the largest degradation among the tested components. The collaborative interface exposes the same expression evidence used by the agent and records interventions, making uncertainty and escalation auditable rather than hiding them behind a single final label.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Human-in-the-loop learning, LLM agents, Single-cell RNA sequencing, Cell-type annotation, Expert–AI collaboration, Interpretable biological reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kXauOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CellMaster, a collaborative LLM-based agent for cell-type annotation in single-cell RNA-seq analysis. The framework uses three specialized agents for hypothesis generation, marker selection, and expression-based annotation, iteratively refining cluster labels through marker memory, confidence pinning, and abstention. It also supports selective escalation to external annotation tools and human experts. Experiments on nine public datasets, with deeper comparisons on PBMC3k, Liver, and Retina, show improvements over several existing methods. Additional ablation studies and human-in-the-loop experiments examine the contributions of individual components and collaborative annotation.

Critical strengths

The integration of iterative annotation, marker memory, confidence pinning, and expression-based evidence inspection provides a practical framework for single-cell annotation. Allowing uncertain clusters to remain unresolved and supporting targeted sub-clustering or expert intervention are useful features for analyzing ambiguous cell populations. The collaborative interface also makes intermediate annotation decisions and their supporting evidence accessible to users.

Critical weaknesses

-Compared to existing large language model (LLM)-based annotation agents, this approach offers limited innovation. Although CellMaster integrates several practical components, its core architecture is primarily a combination of established techniques—including multi-agent task decomposition, iterative hypothesis refinement, marker-based expression analysis, memory mechanisms, and human-in-the-loop interaction. Other relevant studies, such as CellTypeAgent, scPilot, and MAT-Cell, have also explored agent-based reasoning or multi-agent collaboration methods for single-cell annotation.

  • The nine-dataset benchmark in Table 1 reports CellMaster+AF, which uses CellTypist to annotate unresolved clusters, rather than the reference-free CellMaster-Core. Core is evaluated separately on only three datasets in Table 2, where its advantage over contemporary agents is not consistent; for example, scPilot achieves a CL score of 0.675 on Retina compared with 0.667 for Core. Without Core results on the remaining six datasets, it is difficult to assess whether the proposed reference-free agent consistently generalizes across tissues and how much of the broader benchmark performance depends on external annotation assistance.

  • Section 3.3 acknowledges that the LLM-generated confidence scores are not calibrated probabilities, while the experiments mainly report overall CL scores rather than the accuracy of retained, abstained, or escalated predictions. The specialist-routing experiment in Appendix B (Table 8) is limited to a single 28-cluster Liver dataset, and the human-in-the-loop case studies are largely qualitative. It therefore remains unclear whether the proposed confidence-based decisions reliably identify clusters that should be retained, reconsidered, or referred to specialists.

-Evaluation schemes may fail to fully capture the quality of biological annotations and their generalizability. A primary evaluation approach is based on consistency with the "Cell Ontology": a perfect match is assigned 1.0 point, a direct parent-child relationship match receives 0.5 points, and all other predictions are awarded 0 points. While this metric offers determinism and interpretability, it may not adequately reflect the biological relationships between cell states, nor does it effectively account for variations in the granularity of annotations across the hierarchy.

Questions for authors
  • Table 1 reports CellMaster+AF results across nine datasets, whereas CellMaster-Core is evaluated on only three datasets in Table 2. Could the authors provide Core results without AutoFill for the remaining six datasets to clarify the generalization performance of the reference-free framework?

  • Section 3.3 uses LLM-generated confidence scores for label stabilization and abstention. Could the authors report the annotation accuracy of pinned versus unresolved clusters across iterations, or provide a coverage–accuracy analysis to assess whether the confidence-based decisions reliably identify incorrect or uncertain annotations?

  • Appendix B (Table 8) evaluates specialist routing on only the Liver dataset. How does the selective-escalation strategy perform on the other benchmark datasets? Could the authors compare it with directly applying specialist annotators or AutoFill to clarify whether confidence-based routing provides additional benefits beyond simply incorporating external predictions? -As shown in Supplementary Table 13, the confidence intervals for some of the performance gains—relative to comparable agent-based systems—include zero. How do the authors interpret these results? Could further evaluation be conducted to determine whether these performance improvements are statistically robust?

-How sensitive is CellMaster to the initial resolution parameter used in Leiden clustering? In particular, how does the system perform when rare cell populations are merged into larger clusters? Could the authors provide a more systematic robustness assessment regarding variations in clustering granularity and preprocessing workflows?

Reviewer_VFhjOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents CellMaster, an iterative LLM-agent framework for cluster-level cell-type annotation of single-cell RNA-seq data. In each iteration, the agent generates hypotheses, proposes marker genes, and assigns labels based on cluster-level expression and detection fractions. Marker memory, confidence-based pinning, and abstention carry state from one iteration to the next, and clusters that remain uncertain can be routed to specialist tools or human experts. The authors separate the reference-free Core setting from CellTypist AutoFill, specialist-assisted annotation, and human-in-the-loop refinement. The evaluation includes a nine-dataset breadth benchmark, a three-dataset comparison with recent methods, component ablations, and collaborative case studies. The paper targets a relevant question: besides producing a plausible label, an annotator should be able to judge whether the available evidence is strong enough to keep it.

Critical strengths
  1. Besides predicting labels, the framework decides whether to keep a label, gather more evidence, or leave a cluster unresolved. This matters in practice, since annotations that look reasonable often still need validation.

  2. The workflow links hypothesis generation, marker selection, expression inspection, and label revision, and allows a human to step in along the way. The paper also reports the automatic, reference-assisted, and human-assisted settings separately, which makes the results easier to interpret and compare.

  3. The ablations give some support for the design. In Table 3, removing Marker Memory lowers the CL score by 0.142, 0.157, and 0.161 on PBMC3k, Liver, and Retina, respectively, and removing Multi-agent Decomposition or Confidence Pinning also lowers the scores on all three datasets. These results suggest that carrying information across iterations helps.

Critical weaknesses
  1. Uncertainty management is not evaluated directly. The confidence values are generated by the model and are not calibrated, and pinned labels cannot be revised later. Because the CL metric gives zero to both incorrect labels and abstentions, a higher score does not show that the system makes more reliable keep-or-abstain decisions. The paper should report error rates at matched coverage and the number of labels that were pinned incorrectly.
  2. The contribution of expression-grounded reasoning is unclear. All ablation variants share the dot-plot inspection step, and the paper does not compare against a simpler agent given the same evidence and inference budget. All variants also use AutoFill, so some of the differences between them may simply reflect how many clusters end up being labeled by the fallback.
  3. The gains over existing baselines are small and inconsistent. In Table 2, CellMaster-Core exceeds scPilot by only 0.062 on PBMC3k and 0.024 on Liver, and is slightly below it on Retina (0.667 vs. 0.675). With AutoFill the gains are still limited, and CellMaster+AF scores lower than PopV on PBMC3k (0.725 vs. 0.813). Given how complex the multi-agent pipeline is, the paper needs stronger statistical evidence and an analysis of computational cost to justify its practical value.
Questions for authors
  1. The main concern is that the uncertainty-management mechanism is not evaluated directly. The experiments mostly report overall CL scores and annotation coverage, and do not show whether confidence-based pinning, abstention, and selective escalation make the annotations more reliable. Since these decisions shape later iterations and determine when external tools or experts are called, how do they compare with simpler alternatives, such as abstaining below a confidence threshold or sending uncertain clusters directly to a specialist? Risk–coverage curves and an error analysis stratified by confidence would also help here.
  2. Are the backbone model, expression evidence, number of iterations, and inference budget matched between the single-agent variant in Table 3 ("Without multi-agent decomposition") and the full framework? A controlled single-agent comparison, together with an ablation that removes expression feedback, would clarify whether the gains come from the architecture itself or from the extra evidence and computation. Could the authors also report the Table 3 results before AutoFill, including Core coverage and the number of clusters that later receive fallback labels?
  3. In Section 3.3, why is a small number of differing genes between two clusters used as the criterion for further sub-clustering? Two clusters being similar does not imply that either of them is heterogeneous. Could the authors test the refinement step on clusters with known mixed composition, for example controlled mixtures of labeled cell types, and check whether the original populations are recovered? Does the procedure also split homogeneous clusters unnecessarily?
  4. The CL score gives 1.0 for exact matches, 0.5 for direct parent–child matches, and 0 otherwise, so a biologically reasonable label at a different level of granularity can be penalized. How sensitive are the method rankings to this scoring scheme? Could the authors also report exact-match accuracy and the fractions of exact, partial, incorrect, and abstained predictions?
  5. The biological reliability of the generated rationales is not evaluated directly. The experiments mainly score the final labels, yet a plausible explanation does not guarantee that the chosen markers actually support the assigned cell type. How often do the rationales contain unsupported biological claims, and how specific are the proposed markers? An independent expert assessment, especially for ambiguous or rare populations, would be valuable here.
  6. Table 1 reports only CellMaster+AF, and Core is evaluated on its own only for PBMC3k, Liver, and Retina (Table 2). Since AutoFill relies on an external annotator, could the authors report Core-only results, with coverage and abstention rates, on all nine datasets?
Reviewer_X6KqOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CellMaster, an iterative, expression-grounded LLM agent for scRNA-seq cell-type annotation. It proposes marker panels, inspects cluster-level expression, maintains marker memory, and can abstain, subcluster, or escalate unresolved clusters. The authors distinguish reference-free Core from modes using CellTypist AutoFill (AF), specialist tools, or human input. Evaluation spans nine datasets, but Core-versus-agent comparisons with repeated runs focus on PBMC3k, Liver, and Retina. Core Cell Ontology scores on those datasets are 0.708, 0.536, and 0.667; optional AF raises them to 0.725, 0.550, and 0.705. The paper also reports component ablations, a post-cutoff Unseen Liver evaluation, and a small two-expert study.

Critical strengths
  1. Unlike a one-shot annotator that must assign a label, CellMaster can retain unresolved clusters, propose further marker checks or subclustering, and escalate to specialists or humans. This is useful for rare or ambiguous populations where forcing a confident label could be misleading.

  2. There is clear separation of operating modes.Reporting reference-free Core separately from CellTypist AutoFill, specialist tools, and human-in-the-loop results helps readers distinguish what the agent itself achieves from improvements contributed by external resources.

  3. The component studies show that marker memory, multi-agent decomposition, and confidence pinning affect performance; the marker-memory ablation has the largest degradation under fixed AutoFill. This gives some empirical support for the workflow's design rather than relying only on an overall score.

  4. Evidence traceability and iteration controls are useful. The interface exposes expression evidence and records interventions, making annotations easier to inspect. The reported decline after about four iterations also motivates the default iteration limit and highlights why explicit stopping rules matter.

Critical weaknesses
  1. The breadth benchmark does not validate Core across nine datasets. CellMaster+AutoFill is reported on all nine datasets, but reference-free Core results are available for only three. The breadth results therefore do not establish that the central iterative agent generalizes across the full benchmark.

  2. Many baselines are point estimates while CellMaster+AF is reported over five runs, and the paper includes zero-shot scGPT without a directly comparable fine-tuned setting.

  3. The paper describes manual adjudication of unresolved labels and records biologically plausible disagreements in Appendix J, but does not clearly report the number of cases reviewed, who adjudicated them, and also the criteria used. This limits the transparency and independent verifiability of the supplementary assessment.

  4. The two-expert study lacks repeated annotations by the same expert, so reliability and time saved are unknown. Performance also declines after about four iterations, while the post-cutoff Unseen Liver result does not eliminate memorization concerns.

Questions for authors
  1. Can you report reference-free Core results on the remaining six breadth-benchmark datasets, or explain why AF is required there? Please distinguish Core performance from the contribution of CellTypist AutoFill.

  2. Can you provide repeated-run results for key baselines and clarify the comparison with zero-shot scGPT? If fine-tuned scGPT is not evaluated, please discuss how that limits the baseline claim.

  3. In Appendix J, it mentions biologically plausible ontology disagreements. How many cases were reviewed, who adjudicated them, and what procedure was used? Please summarize these details and disputed-case results in the main text.

  4. For the HITL study, can you report annotation time and blinded agreement, and clarify the two experts' roles and independence? If unavailable, please narrow claims about collaboration efficiency and reliability.

  5. Given that performance peaks at iterations 3–4 and declines at iteration 5, how sensitive are results to the iteration limit and initial Leiden partition/resolution? How robust is the Unseen Liver result to the stated residual memorization concern?

Comments on the paper

2 comments
AuthorsOpenReview ↗

CellMaster General Response (Part 1 of 2)

Dear Reviewers,

Thank you for your careful reading and constructive feedback. The core vision of CellMaster is to build scientific AI agents able to test their own hypotheses against data, revise their judgments, and ask for help under uncertainty.

We study these choices in single-cell annotation. Two cell types may share a marker; the agent can check more evidence, revise its label, or ask a scientist for help. We appreciate the reviewers’ recognition of the approach and its practical value.

Recognized strengthIts role in CellMasterReviewers’ comments
Reasoning grounded in expression dataThe agent checks candidate labels against measured expression.“The workflow links hypothesis generation, marker selection, expression inspection, and label revision” (VFhj)
Practical support for ambiguous populationsCore can leave a cluster unlabeled, investigate it further, or seek expert input.“keep a label, gather more evidence, or leave a cluster unresolved” (VFhj). “targeted sub-clustering or expert intervention are useful features for analyzing ambiguous cell populations” (kXau)
Empirical support for the designAblations test marker memory and agent roles. Iteration studies motivate a stopping rule.“The ablations give some support for the design.” (VFhj). “Evidence traceability and iteration controls are useful.” (X6Kq)
Gains over several baselinesCore scored above Biomni on all three datasets in the submitted detailed comparison.“show improvements over several existing methods” (kXau)
Clear attribution of external contributionsWe report Core and assisted results separately.“There is clear separation of operating modes.” (X6Kq)
Traceable annotation decisionsScientists can revisit the markers, label revisions, and interventions.“The collaborative interface also makes intermediate annotation decisions and their supporting evidence accessible to users.” (kXau)

Major problems we addressed through the rebuttal

Our individual replies address all 14 questions. We consolidate the reviewers’ overlapping concerns into four questions below. Part 2 covers the additional experiments and clarifications.

The reviewers also asked for more direct evidence about Core and its annotation decisions. Core uses expression data and pretrained knowledge without external references at inference. Cell Ontology (CL) scoring gives 1 for an exact match, 0.5 for a direct parent or child, and 0 for a wrong label or an abstention. The main questions and rebuttal responses follow.

Point addressedReviewer feedbackOur response and evidenceDetailed replies
What does Core achieve across all nine datasets without AutoFill?X6Kq, VFhj, and kXau requested Core results on the six remaining datasets and separate accounting of CellTypist AutoFill (AF).New experiments. Core now covers all nine datasets with five runs each. It reached 0.569 mean CL at 91.1% mean coverage, weighting datasets equally. AF filled 59 abstentions across 615 cluster–run decisions, raising mean CL to 0.602 at full coverage. Of its additions, 14 were exact, 12 partial, and 33 wrong.X6Kq W1/Q1, VFhj Q6, kXau W1/Q1
Do multiple agents and expression feedback improve annotation?VFhj requested a single agent with the same evidence and budget, plus a test without iterative feedback.New controlled experiments. On PBMC3k, Liver, and Retina, mean CL was 0.639 for Core and 0.568 for a single agent with the same evidence, three rounds, and budget limits. Removing iterative expression feedback reduced the score to 0.532. Removing marker memory produced the largest drop, to 0.483. These scores exclude AF.VFhj W2/Q2
Are the agent’s retained labels reliable?VFhj and kXau asked about selection at matched coverage and errors in pinned labels, kept fixed across rounds.New error analysis. With 14 of 28 Liver labels retained per run, exact-match error was 24.3% for Core’s ranking and 31.4% for confidence alone. The rankings tied at 9 of 19 on Retina. At round three, 20/54 pinned Liver labels and 3/38 pinned Retina labels were non-exact across five runs, including partial matches.VFhj W1/Q1, kXau W2/Q2
Does selective referral add value over simpler specialist policies?kXau and VFhj requested comparisons with direct fallback and confidence-only referral.New routing analysis. Across five human datasets, CellMaster’s rule increased ontology credit on 31 decisions and lowered it on none. Direct fallback improved 22 and worsened none. Confidence-only referral improved 46 and worsened 8, with higher pooled exact accuracy (66.6% vs. 62.8% for CellMaster). With the same acceptance rule, specialist queries on 112 of 290 cluster–run decisions yielded the same labels as queries on all 290.kXau W2/Q3, VFhj W1/Q1
AuthorsOpenReview ↗

CellMaster General Response (Part 2 of 2)

Additional experiments and clarifications

Beyond the four main questions in Part 1, the reviewers offered more specific suggestions about the evaluation. We appreciate these suggestions. We followed up with expert assessments, mixture experiments, and checks on scoring, baselines, and cost. The table connects each comment to our response and the detailed results.

Point addressedReviewer feedbackOur response and evidenceDetailed replies
Evidence for markers and written explanationsVFhj asked about marker specificity and unsupported claims.New expert assessment. Of 185 markers, 120 met both detection and specificity thresholds. Two experts independently rated 370 claims with available text. They judged 13.8% and 12.7% unsupported, respectively. Their ratings agreed on 78.1% of claims.VFhj Q5
Partial credit and disputed cell identitiesVFhj asked about scoring sensitivity. X6Kq requested the cases, assessors, and procedure for disputed labels.New scoring and case analyses. Core + AF and Core alone ranked first and second among seven conditions on the three-dataset mean at all four tested partial-credit weights. Two experts reviewed all 18 non-exact Core labels from run 1, with label sources masked and expression evidence provided. They favored Core in 3 cases and the reference in 7. Both labels were plausible in 7, and 1 remained unresolved.VFhj Q4, X6Kq W3/Q3
Agreement between expert annotatorsX6Kq asked about expert independence, repeatability, and time saved.New agreement analysis. Two experts started with the same AF-assisted labels. In one pass each, they agreed on 53 of 55 final labels. Shared starting labels can correlate their judgments. The study did not measure within-expert repeatability or time savings.X6Kq W4/Q4
Recovery of mixed populations and unnecessary splitsVFhj asked for recovery tests on known mixtures and checks for splits of pure populations.New controlled experiments. On Liver at resolution 0.5, recovery reached an adjusted Rand index of 0.381 with a pure cluster of one constituent type present, versus 0.045 without this peer. Selective refinement split 50/360 pure cluster–run cases, versus 113/360 under refinement of every population. Retina showed the same pattern.VFhj Q3
Number of rounds and initial clustering resolutionX6Kq asked about the stopping rule and starting partition.New experiments. Liver and Retina peaked at round four and declined at round five with unchanged coverage. The three-round default was close to both peaks. Coarser partitions scored higher, with fewer clusters as evaluation units. We report this as partition sensitivity.X6Kq W4/Q5(a)
Dependence on study and tissue informationX6Kq raised the possibility of memorization on the unseen Liver dataset.New context-removal experiment. On a separate 11-cluster human liver dataset, Core scored 0.609 with full context, 0.591 without study identity, and 0.545 without study or tissue information. These results quantify dependence on those cues.X6Kq W4/Q5(b)
Baseline differences and cost per runX6Kq requested repeated baselines. VFhj and kXau questioned the comparison with scPilot. VFhj also asked about specialists and cost.New experiments and cost analysis. Across three datasets with five runs each, Core averaged 0.639 CL versus 0.611 for scPilot. The 95% bootstrap interval for their difference included zero (−0.095 to 0.144). Intervals favored Core over GPTCelltype and zero-shot scGPT. On PBMC3k, PopV scored 0.813 versus Core’s 0.713. Mean Core annotation time ranged from 2.76 to 4.95 minutes per run (USD 0.062 to 0.136 at GPT-4o rates).X6Kq W2/Q2, VFhj W3, kXau W1/Q1

Key takeaways from the rebuttal

  • Core’s performance is measured independently across all nine datasets. The expanded evaluation reports Core’s score and coverage, plus the quality of each AF addition.
  • Repeated expression checks and marker memory improved annotation. Removing either component reduced Core’s score on all three comparison datasets. Removing memory produced the largest loss.
  • Referral involves a tradeoff between gains and errors. At the same query count, confidence-only referral yielded more exact labels overall. CellMaster’s rule made fewer changes, and all 31 increased ontology credit.

Together, these results bring us back to the motivation for CellMaster. A scientific agent needs to choose its next step under incomplete evidence. It can inspect the data again, revise an answer, or ask a scientist for help. CellMaster provides a working framework and measurable outcomes for studying these choices. We believe this question is increasingly important for scientific AI and would welcome the opportunity to bring CellMaster to ICLR.

We would be glad to answer any further questions.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.