- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HiLL builds a hierarchy of cell type annotations by recursively clustering cells and asking an LLM whether candidate splits are biologically distinguishable. Comparative marker evidence guides annotation, while semantic regularization links cell and text representations. Different branches can stop at different depths. The experiments cover human pancreas, human brain, and mouse brain datasets, with component ablations and Cell Ontology-based evaluation.
Critical strengths
This method allows different branches to stop splitting at different levels, which matches the varying granularity required in cell annotation. The newly added multi-resolution comparison reuses the same marker-gene extraction pipeline and provides further evidence that the iterative refinement idea is effective. The paper also reports API usage, runtime, and examples of successful and unsuccessful splits, making the overall pipeline clearer.
Critical weaknesses
Although the multi-resolution comparison addresses the attribution issue, the two experiments use different amounts of LLM resources. On the mouse brain dataset, HiLL uses approximately 1.97 times as many calls and 1.92 times as many tokens as the comparison version. It is therefore difficult to determine whether the performance improvement comes from recursive refinement itself or simply from using more LLM computation. This marker-gene selection scheme is more likely to choose genes that distinguish broad cell populations. Its ability to distinguish closely related cell subtypes remains unclear. In addition, the cell ontology curation was performed by only one person, without independent cross-checking or inter-annotator agreement data.
Questions for authors
Given the mismatch in LLM calls and token usage, how much of the performance improvement can be attributed to recursive refinement? Please provide a matched-budget comparison or discuss this limitation explicitly.
How might the single-curator ontology curation affect the reported metrics? If feasible, please conduct an independent review of a small subset of samples.
2 comments
Response to Split-Validation Examples, Sibling-Marker Ranking, Ontology Curation, and Independent-Review Limitations
W1.4 (Role of split validation and representative accepted/rejected examples)
We thank the reviewer for pointing out that the role of split validation needs more concrete examples. We acknowledge that split validation has only a small effect on the final scores, but we have now added representative examples to illustrate its constraining role in accepting or rejecting populations during the recursive process. The revised Appendix A.13 adds an MSN example: MSNs were identified in the first round; in the second round, a split into drd1 and drd2 groups was produced, and because these two markers are mutually exclusive canonical markers, this split was retained; in the third round, D2-MSNs were split into clusters of 1,486 and 18 cells, and because the latter lacked marker support and had an extremely low size ratio, the LLM rejected this split and merged the redundant subpopulations, thereby avoiding over-splitting. Therefore, its role mainly lies in the mechanistic constraint on accepting or rejecting populations during the recursive process, rather than in substantially improving the final scores. Other influencing factors are also discussed in Appendix A.9.
W1.5 (Marker ranking in closely related sibling populations)
We thank the reviewer for reminding us of the potential bias arising from marker ranking in closely related sibling populations. We agree that this concern is reasonable, and we also noticed this potential bias during our study. Although we attempted to take this factor into account, we have not yet found an effective and low-cost implementation, so this version does not explicitly model the marker-difference relationships among sibling populations. We have noted this as a limitation in the revised Discussion section and plan to further explore it in future work (lines 514–517).
W2.1 (Ontology curation workflow, review of ambiguous mappings, and proportion of manual correction)
We thank the reviewer for raising transparency of manual curation. All methods used the same automatic text-processing pipeline. We did not use LLM outputs directly as standard CL terms; instead, candidate CL terms were generated by automatic mapping and then reviewed item by item by a curator. Ambiguous mappings were filtered out as empty labels during automatic mapping and then manually reviewed and corrected by the curator without knowledge of the method source. Taking mouse brain as an example, exact-match rates between raw outputs and CL text were low (GPTCellType 4%, LICT 2%, scPilot 1%, HiLL 1%); the proportions automatically mapped to CL terms were 83%, 81%, 58%, and 47%, respectively; and the proportions still requiring manual correction after automatic mapping were approximately 39%, 34%, 63%, and 75% (denominator includes empty labels). Details are in Appendix lines 894–908.
W2.2 (Independent review, agreement statistics, and limitations)
We thank the reviewer for pointing out the importance of independent review and agreement statistics. We agree that this concern is reasonable. At present, all manual corrections were completed by the same curator, and independent double review has not yet been conducted; therefore, we cannot currently provide inter-rater agreement statistics or an independently verified proportion of ambiguous mappings. This setup helps maintain consistency in processing across methods, but we recognize that it still lacks independent review. We have listed this as a limitation in the revised manuscript and plan to introduce independent review and agreement statistics in subsequent related studies; the relevant content is in Appendix lines 894–908.
(Q1) We thank the reviewer for this comparison request. This comparison is provided in W1.1–W1.2: W1.1 reports the multi-resolution ablation under the same marker-summary construction, and W1.2 reports actual API calls and token usage for budget comparability. Because hierarchy and cluster numbers are data-driven, strictly matching the LLM budget is difficult; on mouse brain, HiLL uses about 1.97× the calls and 1.92× the tokens of multi-resolution, with similar average tokens per call (about 1,326 vs. 1,363). Full results are in Table 2, lines 437–442, and Appendix A.15–A.16.
(Q2) We thank the reviewer for this clarification request and appreciate the concern regarding the transparency of the ontology evaluation. This is addressed in W2.1–W2.2: all methods used the same automatic pipeline; ambiguous mappings were filtered as empty labels and then manually reviewed by a curator blinded to method source. On mouse brain, the proportions still requiring manual correction after automatic mapping were approximately 39%, 34%, 63%, and 75% for the four methods. Full details are in W2.1–W2.2 and Appendix lines 894–908.
Response to the Multi-Resolution Ablation, Actual API/Token Usage, and Runtime
(W1) We thank the reviewer for these thoughtful comments on recursive semantic refinement, its cost, split validation, and marker ranking. We address them in turn below: W1.1 presents the multi-resolution clustering ablation under the same marker summaries; W1.2 discusses API-call and token-usage comparability; W1.3 reports cost and runtime; W1.4 provides representative accepted/rejected split examples; and W1.5 discusses marker ranking among closely related sibling populations as a limitation.
W1.1 (Attribution to recursive semantic refinement itself)
We thank the reviewer for raising this key issue regarding the attribution of recursive semantic refinement. We understand the reviewer’s concern as whether the remaining performance gains indeed arise from recursive semantic refinement itself. To address this, we replaced iterative semantic refinement with multi-resolution clustering in an ablation and compared the two while keeping the construction of marker summaries consistent. The results show that the multi-resolution scheme has slightly lower overall metrics, with a more pronounced gap in macro-F1, indicating that recursive semantic refinement still contributes to fine-grained small categories.The results are as follows:
HP: Human Pancreas; MB: Mouse Brain; Anc: Ancestor; Desc: Descendant.
The multi-resolution scheme is slightly lower than HiLL’s iterative refinement on overall metrics, and the gap is more evident in macro-F1, which better reflects annotation accuracy for small categories. One possible explanation is that the current multi-resolution scheme is less capable than HiLL of perceiving fine-grained clustering across different branches. This ablation has been added to lines 437–442 of the main text and Table 2, with implementation details in Appendix A.15.
W1.2 (API Calls and Token Usage in the Comparison with Multi-Resolution Clustering)
We thank the reviewer for this constructive suggestion regarding a comparison with multi-resolution clustering under the same marker summaries and a matched LLM budget. We agree that such a comparison would be informative. Ideally, it should strictly align the number of calls and token budgets; however, because the hierarchical tree structure and the number of clusters are data-driven, each cluster triggers one LLM call, and the output length varies with the marker set and split decisions, strictly matching budgets is difficult. We therefore report actual API calls and token usage to facilitate the reviewer’s judgment of comparability.. On the mouse brain dataset, the multi-resolution scheme used approximately 976 calls and 1.33 million tokens (593 annotation calls, with input/output of approximately 450k/520k; 383 split-decision calls, with approximately 320k/40k). HiLL used approximately 1,923 calls and 2.55 million tokens (1,327 annotation calls, with approximately 1.00M/1.13M; 596 split-decision calls, with approximately 360k/60k). HiLL’s total calls and token consumption are therefore about 1.97× and 1.92× those of the multi-resolution scheme, while the average token per call is similar between the two (about 1,363 vs. 1,326 tokens/call). The detailed token usage for HiLL is provided in Appendix A.16.
W1.3 (API calls, token usage, and runtime)
We thank the reviewer for reminding us to supplement cost information such as API calls, token usage, and runtime. We have now added cost-related data (Figure 18 and Table 10), including API calls, token usage, and runtime, in the hope that this will help the reviewer judge the actual overhead of the recursive pipeline. Regarding API calls and token usage, although the number of calls varies across the three datasets, the average cost per call is relatively stable (approximately 1,326–1,618 tokens). Annotation dominates token usage, accounting for about 84% of tokens on each dataset, whereas the subcluster stage consumes only about 16%. Token usage also increases with hierarchy depth, with the deepest level accounting for the largest share. Regarding runtime, HiLL averages about 1,045 s (17.4 min) per mouse-brain dataset under a serial configuration; runtime grows with hierarchy depth, and the dominant costs are pairwise DE screening and LLM annotation, both of which are embarrassingly parallel and for which we provide parallel implementations. Full results are reported in Appendix A.16 and A.17, respectively.