acceptodds
Under review as a conference paper at ICLR 2027

Discovering Interpretable Text Categories with Natural-Language Codebooks

Abstract

Text classification often begins before a useful category inventory exists, and the inventory must also say how later texts are assigned. We formulate interpretable category discovery around three criteria: mutual exclusivity, collective exhaustiveness within a stated scope, and reproducible application from written rules. LLM-assisted Category Discovery (LCD) learns a natural-language codebook from unlabeled texts: each category has a name, a definition, inclusion and exclusion rules, and positive and negative examples. An LLM proposes every split, merge, revision and addition, and each is kept only if it also passes explicit tests on the memberships that an LLM reader verifies on the discovery texts; entries that claim fewer than three texts are retired. The frozen codebook is applied to new texts by a reader that may return zero, one or several categories, so gaps and overlaps remain visible. On BANKING77, CLINC150 and HWU64, whose development-split labels chose LCD's configuration (the label-free comparators ran untuned) and whose test split also scored intermediate configurations, LCD has the highest claim-set ARI and mapped macro-F1 over ten seeds on every dataset against every method we ran: TnT-LLM (ARI +.056, +.293, +.126; macro-F1 +.093, +.306, +.121; Holm-adjusted exact p ≤ .006), its distilled classifiers, every automated LLooM variant, every clustering rule with a data-chosen number of clusters, and four label-informed references. In refits, removing the addition operator lowers pooled ARI by .293 and the split operator by .079, and no removal raises ARI or F1 significantly. Because its reader labels 128 texts per request, labeling costs 11.4–13.7 times less than TnT-LLM's LLM labeler, and total cost is lower at every corpus size on two datasets and from 301 texts on the third. With a weaker reader, LCD stays ahead in macro-F1 on all datasets and in claim-set ARI on two (one with uncovered texts pooled); two-reader agreement is similar for both methods.

Primary area:
interpretability and explainable AI
Keywords:
Text category discovery, natural-language codebooks, interpretable machine learning, large language models, unsupervised text classification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_7NfnOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes "LLM-assisted Category Discovery (LCD)", a framework for automatically constructing interpretable category systems from unlabeled texts. % LCD utilizes Large Language Models (LLMs) to iteratively generate, verify, and refine natural-language codebooks (consisting of category names, definitions, rules, and examples) based on strict criteria such as mutual exclusivity, exhaustiveness, and reproducible application. % Through benchmark tests on banking domains and intent understanding, the authors demonstrate that LCD outperforms existing topic modeling and similar methods in classification accuracy and coverage, while significantly reducing the total API cost for large-scale data. % Furthermore, through ablation studies on each component of the proposed method, the effectiveness and reliability of the automatically generated codebooks were verified.

Critical strengths

The method employs a robust mechanism where the LLM itself verifies proposed edits from five operators (SPLIT, MERGE, REVISE, RETIRE, ADD) across all discovery texts. Only edits that pass explicit membership tests are adopted, enabling high-quality, reliable rule formation without human labels.

In benchmark tests, LCD consistently achieves higher classification accuracy across all datasets compared to state-of-the-art baselines in previous studies, such as TnT-LLM, LLooM, and various clustering methods (e.g., UMAP+HDBSCAN).

Critical weaknesses

There are no human-subject experiments evaluating the inspectability or usability of the generated codebooks for human analysts. Furthermore, the evaluation of "reproducible application" is solely measured by the agreement rate between two LLM models, without verification by human annotators.

Regarding the configuration of LCD, the authors note that approximately 25 intermediate configurations were evaluated and tuned on the test split during the development phase. This raises a concern that the results might be overly optimistic and heavily favor the proposed method over the baselines, which were run untuned.

Questions for authors

How sensitive are the results to the specific thresholds used in Equations (5) and (12)-(14) (e.g., the overlap rate of 0.3 and the coverage rate of 0.7) when applied to domains with different granularities or long-form texts? Do you have any insights from sensitivity analyses regarding these hyperparameters?

Given that the test set was referenced approximately 25 times during the development phase, can we expect similar accuracy when applying these hyperparameters unadjusted to a completely unknown external dataset (Out-of-Distribution domains)?

2 comments
AuthorsOpenReview ↗

Reply to Reviewer 7Nfn: test reuse, thresholds, external data, human evaluation

Thank you. The intent-discovery advantage holds on held-out texts and against a strengthened baseline; on intents, threshold changes stay within refit variation; beyond intents, one F1 gain and one deficit are confirmed; the study with human coders is specified but not yet run. Your Q1 and Q2 prompted the threshold refits and the external evaluation.

W2: test reuse, untuned baselines. Our audit found 41 intermediate configurations and 2,192 offline rule variants scored on the test split, not about 25. No LCD fit reads a label, but these scores informed the design (development-record comment). So we re-tested the margins: the submitted LCD and TnT-LLM fits were applied unchanged to 2,831 additional held-out texts under a plan we recorded beforehand (Table 30; plan: supplement). LCD leads TnT-LLM by +.059/+.293/+.135 ARI and +.091/+.299/+.121 F1 on BANKING77/CLINC150/HWU64. All six planned tests pass (Holm p=.0059; also without 247 BANKING77 texts read earlier), each margin is within .01 of the original, and LCD is ahead of the 13 other comparators scored there (Table 36).

We strengthened the baseline with three further TnT-LLM arms (Table 31), all given LCD's handling criterion: alone, with the true class count, or with LCD's own category count. Given the criterion, TnT-LLM does not improve, and LCD without the criterion keeps all six advantages. Against the arm selected per dataset on development labels, LCD is ahead in all six cells, five with intervals above zero; BANKING77 ARI is inconclusive (+.025 [−.004, +.056]). A limit remains: LCD had more development iterations.

Q1: thresholds. 136 development refits move seven of the twelve thresholds one at a time and eight jointly (coverage .7→.6/.8 and overlap .3→.2/.4 included; Tables 34, 39). No change moves pooled ARI or F1 outside the interval of refitting alone (single changes: ARI −.015 to +.025, against +.012 [−.022, +.048]). With eleven of the twelve switched off, held-out accuracy moves in both directions while codebooks grow by 18–26 categories on average (Table 40): no threshold value is shown to carry the accuracy, and the tests keep codebooks compact.

Other granularities and long texts. On the external datasets (67–125 classes), development sweeps of coverage and overlap stay inside the refit range on Wiki at the fine reference level; on Bills and CSL, and at coarser levels, some changes fall outside it (Table 43; Appendix K.4). Wiki inputs are article openings of up to 4,000 characters; whole long documents are untested.

Q2: external data. We ran LCD on three external datasets, numeric settings unchanged and both methods given the same two domain sentences (ten paired seeds; Tables 32, 41). LCD minus TnT-LLM:

ΔARIΔF1OutcomeAdjudicated ΔARI / ΔF1†
Wiki (article openings)+.031+.070F1 gain confirmed (Holm p=.0078); ARI not+.033 / +.073
Bills (bill summaries)−.003+.039no confirmed difference+.020 / +.066
CSL (Chinese abstracts)−.058−.105TnT-LLM ahead (p=.0039)−.042 / −.081

†Adjudicated read fixed in the protocols (descriptive, no test): one choice where LCD gives a text several categories (15.7% of Bills texts). Seed intervals exclude zero on all three datasets (Appendix K.3).

So: not in general. LCD's ARI there is .15–.25; the best label-free ARI is .372, and a classifier trained on discovery labels reaches .258–.495 (Table 37). The large intent margins do not carry over. Told LCD's category count, TnT-LLM moves ahead on Bills under either read and stays ahead on CSL, while Wiki's F1 gain remains (Table 33); an embedding clustering rule, which yields no written rules, scores above both on Bills and Wiki. The advantage is thus largest and most consistent in fine-grained intent discovery.

W1: human evaluation. No study with human coders exists: it needs ethics approval and outside coders, not obtainable in the discussion period. Its protocol is now Appendix L (blinded coders apply entries of both systems; what each outcome will let us say), and we will add the study to the paper. "Interpretable" is now defined operationally (Section 2.1): written rules that a person can read and audit. Measured without people. (i) On the held-out texts one LCD category matches a reference intent at F1≥.8 for 53/77/48% of intents, against 43/42/37% for TnT-LLM and 48/48/36% for TnT-LLM given LCD's criterion and category count (Appendix K.1; no test). (ii) Reproducible application is now measured on all 2,831 held-out texts instead of 128 per benchmark: two LLM readers agree at κ=.848/.926/.855, against .848/.882/.864 for TnT-LLM (Table 38), which is agreement between LLMs, not people. (iii) With names shuffled, 85–88% of single claims follow the written definition, not the misleading name (Table 17).

If these answers resolve your remaining concerns, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up on threshold sensitivity and transfer

Your question about whether the same thresholds work at different levels of granularity prompted the analysis in Table 43 (p. 74). On Wiki, the changes stayed within ordinary refit variation at the fine-grained level. On Bills and CSL, and at coarser levels, some changes exceeded that range. We have narrowed the claim accordingly. We also agree with your point about human evaluation and now state explicitly that human usability remains untested. Does the analysis in Table 43 address the concern you had about granularity?

Reviewer_9sFKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes LLM-assisted Category Discovery, a framework for discovering interpretable text categories as natural-language codebooks. LCD iteratively applies verified split, merge, revise, retire, and add operations over unlabeled texts, and then uses the frozen codebook to assign categories to new texts while keeping uncovered and overlapping cases visible.

Critical strengths
  1. The formulation of category discovery through natural-language codebooks with explicit inclusion and exclusion rules is clear and interpretable.
  2. The experiments are comprehensive, with strong results against TnT-LLM, LLooM, and clustering baselines, together with cost analysis and ablations.
Critical weaknesses
  1. The test split was repeatedly used to evaluate around 25 intermediate configurations during development. This compromises the independence of the test evaluation and makes the reported gains difficult to interpret as unbiased generalization performance.
  2. The evaluation is limited to three closely related English short-intent benchmarks. It remains unclear how well the method generalizes to broader and more challenging category discovery settings.
  3. The comparison with TnT-LLM is not fully controlled. LCD receives a customer-support handling criterion for SPLIT and MERGE, while TnT-LLM does not, so part of the performance gap may come from this additional prior.
Questions for authors

See the weaknesses above.

2 comments
AuthorsOpenReview ↗

Reply to Reviewer 9sFK: test reuse, the controlled comparison, breadth

Thank you. Each of your three points is now tested directly, with LCD frozen as submitted.

W1. The margins hold on additional held-out texts (Table 30; Appendix K.1). The 30 submitted LCD codebooks and 30 submitted TnT-LLM taxonomies were applied, without refitting, to 2,831 additional texts of the three benchmarks, outside every split of the LCD study. The texts were drawn before any submitted LCD fit existed. From that draw until an evaluation plan was fixed after the reviews, no LLM read them and no score used their labels; they were then scored once. This order of events rests on our records. Method and endpoint were on record with the submission: the 60 fits are those of the code release linked from it, and the six tests are the submitted paper's own (same reader, metrics and test). The plan is in the supplement, verbatim; the read and score records it names will be added to the code release.

LCD − TnT-LLM, ARI / F1Reused test splitHeld-out texts [95% interval over seeds and texts]
BANKING77+.056 / +.093+.059 [.031, .086] / +.091 [.058, .123]
CLINC150+.293 / +.306+.293 [.225, .349] / +.299 [.264, .336]
HWU64+.126 / +.121+.135 [.092, .176] / +.121 [.076, .171]

All six planned tests pass (Holm p=.0059), and every margin is within .01 of the original. LCD is ahead in 59 of the 60 dataset–metric–seed cells and leads all 14 comparators scored on these texts, clustering rules and label-informed references included (Table 36). On the seven seeds fitted after the configuration was chosen, all six gains remain (Holm p=.0469, the smallest value seven seeds allow).

Our audit shows that the reuse was larger than the submitted paper stated: 41 intermediate configurations and 2,192 offline rule variants (parameter settings of decision rules, evaluated on stored reader outputs) were scored on the test split, and these scores informed design choices (development-record comment; Appendix J). No LCD fit reads a label, so reuse could act only through design; the table shows that the margins do not rest on the test texts that development had scored. BANKING77, where the margin is smallest, has two qualifications: 247 of its 741 held-out texts had been read and scored in our earlier studies, before the draw, and without them the gains are +.046 / +.086 (Holm p=.0039). In a later exploratory check, the fits of three seeds, read on 928 new texts, showed no lead there (Appendix K.1). This tests reuse within the three benchmarks; W2 addresses other domains.

W3. The handling criterion (Table 31; Appendix K.2). Six planned controls on the held-out texts (120 new fits) equalize the criterion in both directions, the category count and the reader (four shown; all six in Table 31):

LCD − TnT-LLM, ARI / F1BANKING77CLINC150HWU64
Criterion given to both+.093 / +.132+.419 / +.420+.196 / +.250
Criterion given to neither+.052 / +.087+.302 / +.301+.149 / +.144
Criterion to both, LCD's category count to TnT-LLM+.025 / +.037+.282 / +.236+.108 / +.121
Both read by LCD's reader+.060 / +.091+.273 / +.290+.116 / +.121

Given the criterion, TnT-LLM does not improve, so the criterion closes none of the margin; with it on neither side, LCD keeps all six advantages (Holm p=.0059). LCD is ahead in all 24 cells; one interval includes zero (BANKING77 ARI in the third row: [−.004, +.056]), so we claim no ARI lead there under that control. The number of categories accounts for part of the margins, most on BANKING77 (Appendix K.2).

W2. Breadth: three new domains, numeric settings unchanged (Tables 32, 33, 37; Appendix K.3). We added bill summaries (Bills, 95 classes), Wikipedia article openings (Wiki, 125 classes) and Chinese scientific abstracts (CSL, 67 classes), ten paired seeds each, under protocols that, by our records, predate the downloads (in the supplement). Every dataset that was run is reported. Both methods receive the same two domain sentences (Table 41), so the criterion is matched. LCD minus TnT-LLM:

  • Wiki: F1 +.070, confirmed (Holm p=.0078); ARI +.031, not confirmed (p=.0605).
  • Bills: no confirmed difference (ARI −.003, F1 +.039).
  • CSL: lower on both metrics (−.058 / −.105; p=.0039); the cause is not identified.

With TnT-LLM told LCD's category count, the Wiki F1 gain remains (+.057), Bills turns in TnT-LLM's favor and CSL stays lower; on Bills and Wiki an embedding clustering rule scores above both LLM methods. The paper therefore claims an advantage that is largest and most consistent in fine-grained intent discovery, where it holds on held-out texts and under the controls above, and reports mixed results beyond it (abstract, Sections 5.1 and 7).

Presentation. Sections 2–3 now give every equation in words, an operator table (Table 1) and a worked example.

If these tests resolve W1 and W3, we would be grateful if you would consider raising your score.

AuthorsOpenReview ↗

Follow-up on test-set reuse and the baseline comparison

Thank you again for raising these two concerns. Table 30 (p. 59) reports the results on 2,831 additional texts using the original frozen fits. All six planned comparisons favored LCD, although a separate three-seed check on 928 new BANKING77 texts found no advantage. Table 31 (p. 62) addresses your question about the customer-support criterion: when we removed it from LCD so that neither method received it, LCD still led on both metrics across all three intent datasets. We would appreciate your view on whether either concern remains for the in-domain comparison.

Reviewer_NheEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a new approach, called LCD, for unsupervised text clustering using LLMs to discover interpretable text categories. The idea is to build natural-language codebooks with an LLM trying to split, merge, revise and add codes, whereas LLM reader verifies them.

The paper shows that this LCD method outperforms TnT-LLM and LLooM on 3 datasets (using several evaluation metrics).

Critical strengths

LCD outperforms TnT and LLooM on 3 datasets, across several metrics.

Critical weaknesses

The paper is pretty difficult to read, several equations (e.g. 3, 4, 5, 6) should be explained.

Although the idea is quite intuitive, the implementation is very ad-hoc. There are several rules, coming with hard-coded constants. For instance, in equation 5, first line, where are constants 2, 0.7, 0.3 from. Why take 0.3min(|vk|, |vl|). The experiments use a dev set for configuration selection, but what exactly hyperparameters to tune here? What are their roles?

The paper targets interpretable text categories, but doesn't show how this is achieved. The paper should have an analyse to tell how interpretable the invented text categories are; whether their rules match human intuition and annotations.

Questions for authors
  1. Could you explain the meaning of the equations in the paper?
  2. Could you explain where the constants used are from? How were they chosen?
  3. Could you explain how interpretable the invented text categories are?
4 comments
AuthorsOpenReview ↗

Reply to Reviewer NheE: equations, constants, interpretability

Thank you. Sections 2–3 now give every equation in words, with an operator table (Table 1), a worked example and a quoted entry (Section 2.1). The lead over TnT-LLM that you noted also holds on 2,831 additional held-out texts, in all six tests of our recorded plan (Holm p=.0059; Table 30).

Q1. Equations 3–6 describe one table: a yes or no for every text and every category, filled in by the LLM reader from each category's written rule. No label enters it.

  • Eq. 3 names its parts: the texts a category claims and the texts none claims.
  • Eq. 4 stops a new category from becoming a catch-all (at most 10% of the texts, or as many as the categories it replaces) or from taking more than half of an existing category's texts.
  • Eq. 5 is one counting test per edit (Table 1 gives each in words). A SPLIT is kept if at least two children survive, they keep at least 70% of the parent's texts, and any two share at most 30% of the smaller one.
  • Eq. 6 applies the frozen codebook to a new text: every category whose rule fits is returned (none, one or several; a broad category is replaced by its sub-categories that fit), so gaps and overlaps stay visible.

Example (Section 3.2): an 84-text category is split into children claiming 59, 12 and 8 texts (59, 11 and 7 of them the parent's; one shared). They keep 76 of the parent's 84 (90%, above 70%), and the one shared text is below 30% of 8: accepted.

Q2. The constants. The 2 is structural: a split with one child is a rename. The twelve acceptance thresholds are heuristic defaults, identical on every dataset and seed and not fitted. Each states one of three graded levels or bounds a size (what each encodes, with precedents for the kind of rule, not the values: Appendix G; Table 21):

  • One half, a majority, where a test asks on which side most of a set lies: a child lies mostly inside its parent; an added category is mostly new.
  • .7 and its complement .3, a clear majority, where an edit redraws verified categories: a split keeps at least 70% of the parent's texts, and a text that no child claims is not lost (ADD proposes categories for uncovered texts in the same round). The .3 is taken of the smaller child (hence the min), so at least 70% of each child lies outside the other: a child of 8 wholly inside one of 59 shares 8/8 of the smaller, not 8/59, and fails.
  • .85, nearly all, for MERGE, which asserts that two categories are one: nearly all texts of each side must stay.
  • Size bounds: at least 3 texts and 5% of a parent against fragments; at most 10% of the texts against catch-alls.

Do the values matter? In 136 development refits, twelve single changes to seven of the thresholds (.7 to .6/.8 and .3 to .2/.4 included) and two joint settings move pooled ARI by −.015 to +.025, inside the interval of refitting alone (+.012 [−.022, +.048]; Table 34). The .3 bound rarely binds: 89.4% of the 463 accepted splits have no overlap (Appendix H). With eleven of the twelve switched off, held-out accuracy moves in both directions while every codebook grows, by 18–26 categories on average (Table 40): the thresholds keep codebooks compact.

What labels chose. Before submission one threshold was varied in a fit as a test (ADD's bar of three new texts, against five, on development data). For three thresholds other values were compared offline against held-out labels and none was adopted. We found no record that the SPLIT test's .7 or .3 is among them (Table 21). The development set also chose the cap of six rounds, five first drafts, 128 texts per request and which components run (Table 23). The gates of two post-fit steps came from label-informed studies; removing those steps changes pooled ARI by .001 and .019 (Tables 22, 3; development-record comment).

Q3. Interpretability. Each category is a written rule: name, definition, inclusion and exclusion rules, examples (two in full in Table 28). What can be measured without people:

  • Match with the annotation (Table 27). One LCD category matches a reference intent at F1≥.8 for 58/77/47% of BANKING77/CLINC150/HWU64 intents, against 48/42/36% for TnT-LLM. On the held-out texts: 53/77/48% against 43/42/37%, and against 48/48/36% when TnT-LLM is given LCD's criterion and category count (Appendix K.1; no test).
  • Where name and definition conflict, the definition is followed (Table 17). With names shuffled, 85–88% of single claims stay with the definition and 6–10% follow the moved name. Where they agree, either alone keeps ARI within .013.

Whether the rules match human intuition needs a study with people, hence ethics approval and outside coders, not obtainable in the discussion period. Its protocol is now Appendix L, with what each outcome will let us say; we will add the study to the paper. We claim interpretability only in the operational sense of Section 2.1.

If these answers resolve your three questions, we would be grateful if you would consider raising your score.

Reviewer_NheEOpenReview ↗

response

I would like to thank the authors for the response.

Unfortunately I decided to keep the score as the paper is difficult to read. This problem is also pointed out the the other reviewers.

AuthorsOpenReview ↗

Readability: Eqs. (3)–(6) and the constants are now explained in words

Thank you for your reply. We revised each passage your review named: Eqs. (3)–(6), the constants of Eq. (5) and the choices made on the development set. Sections 2 and 3 were revised throughout, and Section 3 is over 50% longer; what was added is explanation in words, a new table and a worked example. Equation numbers are unchanged; page numbers refer to the revised PDF now on OpenReview.

Right after Eq. (5), on p. 5, the paper now answers your question about the first line of that equation:

"In the first line, k ≠ l are kept children: a split needs 2 of them because one child would only rename p, 70% and 30% are defaults (a clear majority, its complement), and the min measures overlap against the smaller child, so a small child cannot sit inside a large one."

Where else to look:

  • How Eqs. (3)–(6) fit together: one sentence in the opening paragraph of Section 3 (p. 3).
  • Eq. (3): every symbol in words, then a running example from one BANKING77 fit (p. 3).
  • Eq. (4): read as two named guards, a size guard and a take-over guard, with numbers from the example (p. 4).
  • Eq. (5): new Table 1 states each operator's test in words and the failure it blocks; one split is then checked count by count (p. 5).
  • Eq. (6): read in words, with an example (p. 6).
  • Thresholds of Eqs. (4)–(5): three levels (one half; 70%, with 30% its complement; 85%) plus size bounds, explained on p. 4; Appendix G gives the reason for each (pp. 45–46; Table 21, p. 48).
  • Choices made on development data: listed in Section 4 (p. 6): the round cap, ADD's bar of three texts, the number of drafts, the texts per request and which components run; Table 23 gives the role of each (p. 50).

Beyond these passages. The abstract (p. 1) is shorter and defines "interpretable" in plain words, and Section 2.1 (pp. 2–3) quotes the name and rules of one learned category verbatim. Section 3 now says in words how the first draft is chosen, when GROUP reports two entries as one class and how the adjudicated reader picks one claim (pp. 4–6); the formal versions are only in Appendix A. The supplementary material holds the revised manuscript, page for page, with every new or revised paragraph, table and heading in blue.

We believe the paper as a whole now reads more easily, most visibly on pp. 3–6. If any passage is still hard to follow, please name it and we will rewrite it. If the revision resolves your concern, we sincerely hope you will consider raising your score.

AuthorsOpenReview ↗

Thank you for the feedback on readability

Thank you for explaining that readability remains your main concern. If you have time for a second look at just one page, we would suggest p. 5 of the revision, where Table 1 and the worked example explain Eq. (5) step by step. If a particular step there is still hard to follow, we would be grateful to know which one.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

Thank you. LCD is unchanged: all new primary results use the submitted method; original results are kept. New work (Appendix K, with an overview; Tables 30–43): 445 fits, 2,831 additional held-out texts, three external datasets (5,497 test texts) and a log audit (development-record comment). Correction: the submission said about 25 intermediate configurations were scored on the test split; an audit of our score files, stored requests and command logs finds 41 configurations and 2,192 offline rule variants (1,992 of them in the grids of one design study; Table 29).

1. Held-out texts (9sFK W1, 7Nfn W2). The submitted LCD and TnT-LLM fits were applied unchanged to 2,831 held-out texts of the three benchmarks under a plan we recorded beforehand (supplement). LCD is ahead in all six planned tests (Holm p=.0059; margins +.059 to +.299, each within .01 of the original) and leads all 14 comparators scored there (Tables 30, 36).

2. Controlled comparison (9sFK W3, 7Nfn W2). LCD without its handling criterion keeps all six advantages. TnT-LLM given the criterion does not improve; given also LCD's category count, it stays behind in all six cells, one with an interval including zero (BANKING77 ARI, +.025). With one reader for both, LCD leads in all six (Table 31).

3. Acceptance thresholds (NheE Q2, 7Nfn Q1). The twelve are unfitted defaults (three graded levels, size bounds; reasons: Appendix G). In 136 development refits, no threshold change moves pooled ARI or F1 outside the interval of refitting alone (Table 34). With eleven switched off, accuracy moves in both directions and codebooks grow by 18–26 categories on average (Table 40).

4. Scope (9sFK W2, 7Nfn Q2). The advantage is largest and most consistent in fine-grained intent discovery. On three external datasets, numeric settings unchanged, LCD against TnT-LLM has a confirmed F1 gain on Wiki (+.070), no confirmed difference on Bills (TnT-LLM ahead at LCD's category count) and a confirmed deficit on CSL (−.105 F1). We claim no general advantage there: embedding clustering, which yields no written rules, scores above both on Bills and Wiki (Tables 32, 33, 37).

5. Interpretability (NheE Q3, 7Nfn W1). "Interpretable" means written rules that can be read and audited. For 58/77/47% of the reference intents, one LCD category matches at F1≥.8 (TnT-LLM, with fewer categories: 48/42/36%; Table 27). A study with human coders needs ethics approval and outside coders, impossible in the discussion period. Its protocol is Appendix L; we claim nothing about human usability.

6. Presentation (NheE Q1). Sections 2–3: equations in words, operator table (Table 1), worked example, a quoted entry (Section 2.1); equation numbers unchanged. Appendices G–L are new.

AuthorsOpenReview ↗

Development record and held-out re-evaluation

What the submission said. The test split had scored about 25 intermediate configurations; these scores informed design changes, so test results may be optimistic; the final configuration was selected on development data only.

What our audit found. According to our score files and development logs, 41 intermediate configurations and 2,192 offline rule variants (parameter settings of decision rules, evaluated on stored reader outputs) had been scored on the test split. Of the variants, 1,992 belong to one grid study; LCD retains from it the gates for reporting two related categories as one class, and removing that step changes pooled test ARI by .001. Three other held-out samples of these benchmarks were also scored (Table 29; Appendix B.2). So the submitted count was too low, and "development data only" holds for the last selection step but not for every component that step compared. Keyed reading (each verdict returned under its text's identifier) was first chosen on one of these other samples, and the gates of the grouping step came from grids scored on the development split, the test split and another of these samples (Appendix B.2). The appendix sentence that development labels chose the settings shared by all datasets was also wrong for the acceptance thresholds. The twelve are defaults that we did not fit; before submission only one, ADD's bar of three new texts, was varied in a fit as an acceptance test (against five, on development data). Appendix G gives the rationale of each. No LCD fit reads a reference label; the exposure is in design, not in fitting. Section 4 and Appendices B.2 and J give the corrected account. The test split is now called the reused test split.

How we answer it. We re-tested the comparison with TnT-LLM: the submitted fits of both methods were applied, unchanged, to 2,831 additional texts outside every split of the LCD study. These texts had been drawn before any submitted LCD fit existed; they were scored once, under a plan recorded after the reviews and before they were read for this evaluation. LCD is ahead of TnT-LLM in all six planned tests (Holm p=.0059), with margins within .01 of the reused-test margins (Table 30). Every margin is within .008 of the forecast that the plan had fixed, and none of the plan's four flags was raised. On the seven seeds fitted after the configuration was chosen, all six gains remain (Holm p=.0469, the smallest value seven seeds allow).

What can be checked. Method and endpoint were on record with the submission, before these texts were read for this evaluation. The 60 fits are the codebooks and taxonomies of the code release linked from the submission (the six text fields of all 3,713 LCD entries are identical). The six tests use the reader, the metrics, the exact test and the Holm family of the submitted paper. The plan and the protocols of the external datasets are in the supplement, verbatim and with their SHA-256 values, beside the aggregate numbers behind Tables 30–43; the read and score records that the plan names will be added to the code release. What rests on our records is the order of events: that no LLM read these texts and no score used their labels between the draw and the plan, and that they were then scored once. These records are dated by us, not by a third party.

Two qualifications, both on BANKING77 (Appendix K.1). Of its 741 held-out texts, 247 had been read and scored in our earlier studies, before these texts were drawn; without them the gains remain (+.046 ARI, +.086 F1; Holm p=.0039). In a later exploratory check outside the plan, the fits of three seeds were read on 928 newly drawn texts and showed no lead there (ARI .669 against TnT-LLM's .684; F1 .672 against .671). Neither applies to CLINC150 or HWU64, where the held-out gains are +.293 and +.135 ARI.

What remains. LCD was designed on these three benchmarks, so we claim its advantage for intent discovery and report the external datasets separately.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.