- Rating
- 3
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes "LLM-assisted Category Discovery (LCD)", a framework for automatically constructing interpretable category systems from unlabeled texts. % LCD utilizes Large Language Models (LLMs) to iteratively generate, verify, and refine natural-language codebooks (consisting of category names, definitions, rules, and examples) based on strict criteria such as mutual exclusivity, exhaustiveness, and reproducible application. % Through benchmark tests on banking domains and intent understanding, the authors demonstrate that LCD outperforms existing topic modeling and similar methods in classification accuracy and coverage, while significantly reducing the total API cost for large-scale data. % Furthermore, through ablation studies on each component of the proposed method, the effectiveness and reliability of the automatically generated codebooks were verified.
Critical strengths
The method employs a robust mechanism where the LLM itself verifies proposed edits from five operators (SPLIT, MERGE, REVISE, RETIRE, ADD) across all discovery texts. Only edits that pass explicit membership tests are adopted, enabling high-quality, reliable rule formation without human labels.
In benchmark tests, LCD consistently achieves higher classification accuracy across all datasets compared to state-of-the-art baselines in previous studies, such as TnT-LLM, LLooM, and various clustering methods (e.g., UMAP+HDBSCAN).
Critical weaknesses
There are no human-subject experiments evaluating the inspectability or usability of the generated codebooks for human analysts. Furthermore, the evaluation of "reproducible application" is solely measured by the agreement rate between two LLM models, without verification by human annotators.
Regarding the configuration of LCD, the authors note that approximately 25 intermediate configurations were evaluated and tuned on the test split during the development phase. This raises a concern that the results might be overly optimistic and heavily favor the proposed method over the baselines, which were run untuned.
Questions for authors
How sensitive are the results to the specific thresholds used in Equations (5) and (12)-(14) (e.g., the overlap rate of 0.3 and the coverage rate of 0.7) when applied to domains with different granularities or long-form texts? Do you have any insights from sensitivity analyses regarding these hyperparameters?
Given that the test set was referenced approximately 25 times during the development phase, can we expect similar accuracy when applying these hyperparameters unadjusted to a completely unknown external dataset (Out-of-Distribution domains)?
2 comments
Follow-up on threshold sensitivity and transfer
Your question about whether the same thresholds work at different levels of granularity prompted the analysis in Table 43 (p. 74). On Wiki, the changes stayed within ordinary refit variation at the fine-grained level. On Bills and CSL, and at coarser levels, some changes exceeded that range. We have narrowed the claim accordingly. We also agree with your point about human evaluation and now state explicitly that human usability remains untested. Does the analysis in Table 43 address the concern you had about granularity?
Reply to Reviewer 7Nfn: test reuse, thresholds, external data, human evaluation
Thank you. The intent-discovery advantage holds on held-out texts and against a strengthened baseline; on intents, threshold changes stay within refit variation; beyond intents, one F1 gain and one deficit are confirmed; the study with human coders is specified but not yet run. Your Q1 and Q2 prompted the threshold refits and the external evaluation.
W2: test reuse, untuned baselines. Our audit found 41 intermediate configurations and 2,192 offline rule variants scored on the test split, not about 25. No LCD fit reads a label, but these scores informed the design (development-record comment). So we re-tested the margins: the submitted LCD and TnT-LLM fits were applied unchanged to 2,831 additional held-out texts under a plan we recorded beforehand (Table 30; plan: supplement). LCD leads TnT-LLM by +.059/+.293/+.135 ARI and +.091/+.299/+.121 F1 on BANKING77/CLINC150/HWU64. All six planned tests pass (Holm p=.0059; also without 247 BANKING77 texts read earlier), each margin is within .01 of the original, and LCD is ahead of the 13 other comparators scored there (Table 36).
We strengthened the baseline with three further TnT-LLM arms (Table 31), all given LCD's handling criterion: alone, with the true class count, or with LCD's own category count. Given the criterion, TnT-LLM does not improve, and LCD without the criterion keeps all six advantages. Against the arm selected per dataset on development labels, LCD is ahead in all six cells, five with intervals above zero; BANKING77 ARI is inconclusive (+.025 [−.004, +.056]). A limit remains: LCD had more development iterations.
Q1: thresholds. 136 development refits move seven of the twelve thresholds one at a time and eight jointly (coverage .7→.6/.8 and overlap .3→.2/.4 included; Tables 34, 39). No change moves pooled ARI or F1 outside the interval of refitting alone (single changes: ARI −.015 to +.025, against +.012 [−.022, +.048]). With eleven of the twelve switched off, held-out accuracy moves in both directions while codebooks grow by 18–26 categories on average (Table 40): no threshold value is shown to carry the accuracy, and the tests keep codebooks compact.
Other granularities and long texts. On the external datasets (67–125 classes), development sweeps of coverage and overlap stay inside the refit range on Wiki at the fine reference level; on Bills and CSL, and at coarser levels, some changes fall outside it (Table 43; Appendix K.4). Wiki inputs are article openings of up to 4,000 characters; whole long documents are untested.
Q2: external data. We ran LCD on three external datasets, numeric settings unchanged and both methods given the same two domain sentences (ten paired seeds; Tables 32, 41). LCD minus TnT-LLM:
†Adjudicated read fixed in the protocols (descriptive, no test): one choice where LCD gives a text several categories (15.7% of Bills texts). Seed intervals exclude zero on all three datasets (Appendix K.3).
So: not in general. LCD's ARI there is .15–.25; the best label-free ARI is .372, and a classifier trained on discovery labels reaches .258–.495 (Table 37). The large intent margins do not carry over. Told LCD's category count, TnT-LLM moves ahead on Bills under either read and stays ahead on CSL, while Wiki's F1 gain remains (Table 33); an embedding clustering rule, which yields no written rules, scores above both on Bills and Wiki. The advantage is thus largest and most consistent in fine-grained intent discovery.
W1: human evaluation. No study with human coders exists: it needs ethics approval and outside coders, not obtainable in the discussion period. Its protocol is now Appendix L (blinded coders apply entries of both systems; what each outcome will let us say), and we will add the study to the paper. "Interpretable" is now defined operationally (Section 2.1): written rules that a person can read and audit. Measured without people. (i) On the held-out texts one LCD category matches a reference intent at F1≥.8 for 53/77/48% of intents, against 43/42/37% for TnT-LLM and 48/48/36% for TnT-LLM given LCD's criterion and category count (Appendix K.1; no test). (ii) Reproducible application is now measured on all 2,831 held-out texts instead of 128 per benchmark: two LLM readers agree at κ=.848/.926/.855, against .848/.882/.864 for TnT-LLM (Table 38), which is agreement between LLMs, not people. (iii) With names shuffled, 85–88% of single claims follow the written definition, not the misleading name (Table 17).
If these answers resolve your remaining concerns, we would be grateful if you would consider raising your score.