acceptodds
Under review as a conference paper at ICLR 2027

Retrieve Label Knowledge, Then Boundary Rules: Label-Centric RAG for Text Classification

Abstract

Flagship large language models (LLMs) offer rich general knowledge and strong reasoning capabilities, but their high inference cost and latency limit their use for large-scale data labeling. We study zero-shot text classification with semantically meaningful labels and task descriptions, without task-specific labeled examples or classifier feedback. We propose SLKR (Semantic Label Knowledge Retrieval), a label-centric retrieval-augmented generation framework that separates offline knowledge construction from lightweight online inference. Offline, a flagship LLM independently generates label definitions, retrieval keywords, and boundary rules for potentially confusable label pairs, forming a reusable label-knowledge store. Because this construction operates over the label space rather than the input corpus, its one-time flagship cost can be amortized across large-scale inference. Online, hybrid semantic and lexical retrieval first identifies the top- candidate labels and then selects up to input-relevant boundary rules connecting candidate-label pairs. A lightweight LLM performs the final classification in a single call using the retrieved definitions and rules. Across three public benchmarks and a real-world multilingual book-classification dataset, SLKR achieves the best results on all four datasets with Qwen3.5-4B and on three of four with Ministral-3-3B, while using compact input contexts and no task-specific labeled examples. Boundary ablations further show that retrieving a small set of input-relevant rules preserves performance close to using all boundaries with substantially less context, supporting efficient reuse of flagship-generated label knowledge for lightweight classification.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, retrieval-augmented generation, zero-shot text classification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_RcczOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SLKR, a framework in which a flagship model generates label definitions, keywords, and class boundary rules offline. Relevant knowledge is then retrieved online to support zero-shot text classification with a smaller model. The paper evaluates the framework on four datasets using two small models and compares its performance and cost against direct flagship-model classification on BANKING77.

Critical strengths
  1. No task-specific labeled data is required, and the generated knowledge is reusable. The separation between offline knowledge construction and online classification is sensible. Table 4 shows improvements over direct zero-shot classification across both model families.
  2. The boundary-retrieval ablation is well designed. Retrieving relevant rules outperforms randomly selecting the same number of rules. It also reduces the average boundary context from approximately 2.51k to 0.35k tokens while maintaining performance close to using all eligible rules on the public benchmarks.
Critical weaknesses
  1. The methodological novelty appears limited. Label-description augmentation, RRF-based hybrid retrieval, and candidate-label selection are established techniques. Boundary rules for confusable labels also have precedents, including Gupta et al. (2026). The main distinction is constructing these rules without task instances or classifier feedback and retrieving them within candidate pairs. The paper should more clearly establish the contribution beyond combining these components.

  2. The baselines do not sufficiently establish competitiveness or isolate the source of improvement. (i) Limited coverage of strong methods. Label-only ZS, 1-shot/label, and Retrieved ICL are reasonable basic baselines, but do not adequately cover representative zero-shot, label-knowledge augmentation, or small–large model collaboration methods. (ii) Unequal access to flagship knowledge. SLKR uses knowledge generated by GPT-5.6 Sol, whereas these baselines rely on the 3B/4B reader and their respective prompts or demonstrations. Table 4 therefore does not isolate gains from flagship knowledge from those of the proposed retrieval framework. A simpler baseline using the same flagship-generated knowledge would help. (iii) The flagship builder’s necessity remains untested. Having Qwen3.5-4B generate the same types of knowledge would reveal whether the stronger builder provides sufficient additional benefit to justify its cost.

  3. Evidence for a competitive cost–performance trade-off is insufficient. (i) The cost comparison is too narrow. Table 6 covers only BANKING77 and two approaches, without placing small-model prompting, SLKR-Desc, and other comparable zero-shot methods under a common cost accounting. (ii) Savings come with a substantial quality loss. Accuracy falls from 86.01% to 77.45%, a drop of 8.56 percentage points. This establishes lower cost, but not a competitive trade-off. Broader comparisons across datasets and workloads are needed.

Questions for authors
  1. Was Gen-Z implemented according to the original method? Does the implementation compute the likelihood of the input text conditioned on each label description, or does it simply add label definitions to the prompt and directly predict the class? Please clarify the scoring procedure and any departures from the original method.
  2. Do SLKR’s advantages hold against representative zero-shot methods? The current comparisons mainly cover basic prompting and retrieval configurations. Could the authors include representative approaches based on label-semantic augmentation, label retrieval, or small–large model collaboration, and clearly specify each method’s use of labeled, unlabeled, and synthetic data?
  3. Is a flagship model necessary for knowledge construction? If Qwen3.5-4B generates the label definitions, keywords, and boundary rules while the subsequent retrieval and classification pipeline remains unchanged, how do accuracy and total cost change? Do the additional gains from the flagship model justify its construction cost?
  4. Does SLKR offer a competitive cost–performance trade-off? On BANKING77, the cost reduction comes with an accuracy drop of 8.56 percentage points. Could the authors compare performance and cost against more comparable zero-shot methods across additional datasets?
3 comments

Methodological novelty. Gupta et al. (2026) appeared on arXiv approximately one month before our ICLR submission; SLKR was developed independently, and we cited this closely related work in our original submission. Unlike reflection-oriented prompting, SLKR explicitly transfers flagship-model knowledge into a reusable label-side knowledge base, enabling candidate-conditioned retrieval and lightweight inference without repeated flagship-model reasoning.

Q1: Clarification of Gen-Z implementation. We thank the reviewer for identifying an inaccurate baseline label in our original submission. The row previously named "Gen-Z (Label + definition)" was in fact a Definition-Augmented Zero-Shot baseline: the reader directly predicts a class from all labels and their GPT-5.6 Sol-generated definitions. It did not implement the conditional-likelihood scoring procedure of Gen-Z. We will correct the name and implementation description in the revised manuscript. We have now implemented two additional conditional-likelihood Gen-Z variants, using the same readers and four datasets. Their results are reported as follows.

ReaderDatasetGen-Z + SLKR single definitionOriginal Gen-Z (multi-description)
Qwen3.5-4BWOS18.58 / 17.7427.28±0.1627.28\pm0.16 / 26.18±0.2126.18\pm0.21
BANKING7750.36 / 47.5263.43±0.3363.43\pm0.33 / 62.85±0.3262.85\pm0.32
CLINC15046.13 / 45.2869.01±0.1869.01\pm0.18 / 67.10±0.2267.10\pm0.22
Book386.15 / 8.7310.61±1.1210.61\pm1.12 / 5.31±1.355.31\pm1.35
Ministral-3-3BWOS27.02 / 25.6643.37±0.2043.37\pm0.20 / 43.55±0.2843.55\pm0.28
BANKING7753.02 / 50.3970.09±0.1970.09\pm0.19 / 69.58±0.2269.58\pm0.22
CLINC15049.89 / 49.0970.83±0.1770.83\pm0.17 / 68.25±0.2068.25\pm0.20
Book3821.79 / 11.9028.10±1.2528.10\pm1.25 / 17.16±1.2717.16\pm1.27

Table caption: New conditional-likelihood Gen-Z experiments (accuracy / macro-F1, %). Multi-description results report mean ±\pm standard deviation over 10 description-subset draws. The single-definition setting uses SLKR's single label description.

Conditional-likelihood scoring. For each class, we use the same GPT-5.6 Sol-generated definition as in the definition-augmented zero-shot baseline and generate 10 additional paraphrases of that fixed definition using GPT-5.6 Sol, following the multi-description construction in Gen-Z. The resulting 11 descriptions are frozen before evaluation, without using test instances or classifier feedback. The single-definition Gen-Z setting uses only the original definition, whereas the multi-description setting uses the original definition and its paraphrases. For description dd and input xx, we score only the input tokens, treating the description as an unscored prefix and using plain text rather than a chat template:

\begin{equation} \ell(x\mid d)=\frac{1}{|x|}\sum_{t=1}^{|x|}\log p_\theta(x_t\mid d,x_{<t}). \end{equation}

For the multi-description adaptation, 10 of the 11 descriptions are sampled per class and their length-normalized conditional likelihoods are aggregated arithmetically:

\begin{equation} S(c,x)=\frac{1}{10}\sum_{j=1}^{10}\exp!\left(\ell(x\mid d_{c,j})\right),\qquad \hat c=\arg\max_{c\in\mathcal C}S(c,x). \end{equation}

We repeat description-subset sampling 10 times and report mean and standard deviation of accuracy and macro-F1. The multi-description setting follows the original Gen-Z conditional-likelihood scoring and description-aggregation procedure, using GPT-5.6 Sol-generated label descriptions for our four evaluation datasets. As shown in above, the multi-description setting improves accuracy over the single-definition setting across all eight reader--dataset combinations.

Q2: Shared knowledge and component-level comparisons.

We clarify that all methods in the original submission that use label definitions employ the same GPT-5.6 Sol-generated definitions; the definition-augmented zero-shot baseline, 1-shot-per-label prompting, Retrieved ICL, SLKR-Desc, and SLKR therefore have access to the same flagship-generated label definitions.

Table: Comparison under shared GPT-5.6 Sol-generated label definitions (accuracy / macro-F1, %). All three methods use the same definition knowledge and reader within each model group.

ReaderMethodWOS-143BANKING77CLINC150Book-38
Qwen3.5-4BDefinition-Augmented ZS57.37 / 57.0672.95 / 71.5885.64 / 84.6661.45 / 60.81
SLKR-Desc (M=0M=0)59.92 / 57.6777.05 / 75.9989.13 / 88.5766.85 / 66.24
SLKR (M=5M=5)60.17 / 58.2077.45 / 76.6389.81 / 89.3370.58 / 70.25
Ministral-3-3BDefinition-Augmented ZS34.49 / 35.5565.91 / 64.7981.36 / 80.3546.37 / 44.87
SLKR-Desc (M=0M=0)52.93 / 51.3167.76 / 66.7084.00 / 83.2253.07 / 49.93
SLKR (M=5M=5)55.83 / 53.7573.44 / 72.8185.69 / 85.0758.66 / 52.29

The above table presents these three methods side by side for both readers. The original row labeled "Gen-Z (Label + definition)" is renamed Definition-Augmented ZS to reflect its direct label-prediction procedure.

Results under matched definition knowledge.

With Qwen3.5-4B, SLKR improves accuracy over Definition-Augmented ZS by 2.80, 4.50, 4.17, and 9.13 percentage points on WOS-143, BANKING77, CLINC150, and Book-38, respectively. With Ministral-3-3B, the corresponding gains are 21.34, 7.53, 4.33, and 12.29 points. The intermediate SLKR-Desc variant, which uses candidate-label retrieval without boundary rules, already improves accuracy over Definition-Augmented ZS on all four datasets for both readers. Adding retrieved boundary rules further improves accuracy over SLKR-Desc on all four datasets for both readers (Qwen3.5-4B: +0.25, +0.40, +0.68, +3.73 points; Ministral-3-3B: +2.90, +5.68, +1.69, +5.59 points). These matched-definition comparisons show that the improvements are not explained solely by access to GPT-5.6 Sol-generated label definitions; candidate-label selection and boundary knowledge contribute additional performance gains.

Q3: Is a flagship builder necessary?

To investigate the dependence on the knowledge builder, we replaced GPT-5.6 Sol with Qwen3.5-4B for offline knowledge construction while keeping the downstream retrieval and classification pipeline unchanged.

Table: Builder ablation using Qwen3.5-4B as reader (accuracy / macro-F1, %).

MethodDatasetGPT-5.6 Sol builderQwen3.5-4B builderGain (GPT − 4B)
SLKR-DescWOS59.92 / 57.6758.72 / 56.62+1.20 / +1.05
BANKING7777.05 / 75.9969.71 / 68.38+7.34 / +7.61
CLINC15089.13 / 88.5782.87 / 81.81+6.26 / +6.76
Book3866.85 / 66.2462.01 / 43.16+4.84 / +23.08
SLKRWOS60.17 / 58.2059.28 / 57.18+0.89 / +1.02
BANKING7777.45 / 76.6372.92 / 72.03+4.53 / +4.60
CLINC15089.81 / 89.3386.16 / 85.21+3.65 / +4.12
Book3870.58 / 70.2566.48 / 44.94+4.10 / +25.31

With the 4B builder, SLKR achieves accuracies of 59.28%, 72.92%, 86.16%, and 66.48% on WOS, BANKING77, CLINC150, and Book38, respectively, corresponding to decreases of 0.89, 4.53, 3.65, and 4.10 percentage points compared with the flagship builder.

Table: Incremental gain from boundary knowledge: SLKR minus SLKR-Desc (accuracy / macro-F1 percentage points).

DatasetGPT-5.6 Sol builderQwen3.5-4B builder
WOS+0.25 / +0.53+0.56 / +0.56
BANKING77+0.40 / +0.64+3.21 / +3.65
CLINC150+0.68 / +0.76+3.29 / +3.40
Book38+3.73 / +4.01+4.47 / +1.78

boundary knowledge remains effective with the smaller builder, improving accuracy over SLKR-Desc by 0.56, 3.21, 3.29, and 4.47 percentage points across the four datasets. Notably, on BANKING77 and CLINC150, the boundary-rule gains increase from 0.40 and 0.68 percentage points with the flagship builder to 3.21 and 3.29 points with the 4B builder, respectively. This indicates that boundary knowledge remains useful even when generated by a smaller builder. These results demonstrate that SLKR is not dependent on a flagship model for knowledge construction. Stronger builders improve overall classification accuracy, while the boundary-knowledge mechanism remains beneficial even when knowledge is generated by a substantially smaller model.

Q4: Cost--accuracy trade-off.

We extend the cost comparison beyond direct flagship classification to eight prompting, retrieval, and conditional-likelihood methods across the two lightweight readers and a GPT-5.6 Sol API-cost reference. Lightweight-reader costs are approximated based on GPU platform pricing, average input token lengths, and model inference throughput. GPT-5.6 Sol costs use published API token rates; the GPT column is a cost projection rather than an additional accuracy evaluation.

Table: Approximate online inference cost per 1,000 classifications (USD).

GPT-5.6 Sol uses standard API rates ($4/1M input tokens; $20/1M output tokens), with 10 output tokens per direct-prediction request. Gen-Z tokens are cumulative across class-description evaluations; GPT-5.6 Sol conditional-likelihood costs are not reported. Offline knowledge construction is excluded.

MethodAvg. input tokensQwen3.5-4BMinistral-3-3BGPT-5.6 Sol
Label-only ZS0.9k0.08250.05163.8000
Definition-Augmented ZS2.7k0.24760.154711.0000
1-shot / label4.7k0.43100.269319.0000
Retrieved ICL1.2k0.11000.06885.0000
SLKR-Desc (M=0M=0)1.0k0.09170.05734.2000
SLKR (M=5M=5)1.4k0.12840.08025.8000
Gen-Z (single definition)15.21k1.39450.8716N/A
Original Gen-Z (multi-description)152.08k13.94498.7155N/A

Cost--performance comparison.

As shown in above, SLKR achieves a favorable accuracy--cost trade-off relative to definition-augmented zero-shot classification. With Qwen3.5-4B, SLKR has an approximate online cost of $0.1284 per 1,000 classifications, compared with $0.2476 for Definition-Augmented ZS, while achieving higher accuracy on all four datasets. With Ministral-3-3B, the corresponding costs are $0.0802 and $0.1547, respectively. SLKR-Desc provides a lower-cost alternative ($0.0917 and $0.0573 for the two readers), whereas full SLKR consistently improves accuracy over SLKR-Desc across the four datasets. The GPT-5.6 Sol column additionally shows the corresponding API-cost projections at published token rates. The conditional-likelihood Gen-Z variants require substantially more cumulative token processing because they score the input against class descriptions, with the multi-description variant aggregating ten descriptions per class. These comparisons extend the evaluation beyond the flagship-direct baseline and characterize the trade-offs among lightweight classification approaches.

Production-scale budget trade-off.

For a production workload of one billion documents, we consider an illustrative online inference budget of $200,000.

Table: Budget-constrained feasibility for classifying one billion documents with Qwen3.5-4B. Costs are approximate online inference expenses in USD; the illustrative budget is $200,000.

MethodCost for 1B documents (USD)Within budget?
Label-only ZS$82,500Yes
Definition-Augmented ZS$247,600No
1-shot / label$431,000No
Retrieved ICL$110,000Yes
SLKR-Desc (M=0M=0)$91,700Yes
SLKR (M=5M=5)$128,400Yes
Gen-Z (single definition)$1,394,500No
Original Gen-Z (multi-description)$13,944,900No

scales the approximate per-1,000-classification costs to this workload. SLKR with Qwen3.5-4B requires approximately 128,400 usd and remains within budget, whereas Definition-Augmented ZS (247,600 usd), 1-shot-per-label prompting (431,000 usd), and the Gen-Z variants exceed the budget. Among the evaluated Qwen3.5-4B methods that can process the full workload within this budget, SLKR achieves the highest accuracy on all four datasets and the original main results). Thus, the relevant trade-off for production-scale classification is accuracy among methods that satisfy the computational budget, rather than accuracy alone. The feasibility comparison concerns online inference costs and excludes offline knowledge construction and infrastructure overhead.

Reviewer_pNYrOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SLKR, a label-centric RAG framework for zero-shot text classification. The main idea is to call a strong LLM once, offline, and let a small LLM do the actual classification. GPT-5.6 Sol only sees the label set and the task description. From these it writes label definitions, retrieval keywords, and boundary rules for label pairs that are easy to confuse. This knowledge base is then frozen and reused for all inputs. At inference time, a hybrid retriever (dense + lexical, fused with RRF) picks the top-K candidate labels. It then picks the top-M boundary rules, but only among rules where both labels are candidates. A lightweight LLM makes the final prediction in a single call. The authors claim three contributions: the label-centric RAG formulation, the candidate-conditioned boundary retrieval, and a cost argument. The cost argument is that the flagship cost depends on the label space (O(N^2) in the worst case), not on the number of inputs. Experiments use WOS-143, BANKING77, CLINC150, and an industrial multilingual book dataset with 38 classes. There are two small readers, Qwen3.5-4B and Ministral-3-3B. The paper also has an ablation on boundary policies and a cost comparison with direct GPT-5.6 Sol classification on BANKING77.

Critical strengths
  1. SLKR builds its knowledge from only the label inventory and task description, so the flagship cost grows with the size of the label space, not with the number of inputs TT. This matters because it targets cold-start labeling, where no labeled data is available and the volume to label is large.
  2. With Qwen3.5-4B, SLKR is best on all four datasets in both accuracy and macro-F1, ahead of the 1-shot and Retrieved ICL baselines that use training labels.
Critical weaknesses
  1. The method is heuristic, and K is a hand-set hyperparameter. Most design choices are fixed by hand, not learned or analyzed. K = 15 is chosen because Recall@15 is above 95%. The same K is used for all datasets. Yet it covers about 39% of the labels in Book-38 and only 10% in CLINC150. M = 5 is set without any justification.
  2. The paper frames SLKR as a solution for large-scale classification many times. But Book-38, the industrial dataset presented as the “large-scale” case, has only 719 books and 179 test items. So, the “large-scale” framing is not backed by the experiments.
  3. The paper does not link to any code and has no reproducibility statement. It does not give the prompts for the three flagship roles or for the reader. Several key settings are missing.
Questions for authors
  1. Since the knowledge base and retriever are frozen, what varies across the three seeds? Did Gen-Z use the same GPT-5.6 Sol definitions as SLKR-Desc, or its own descriptions?
  2. How many boundary rules were generated for each dataset, and what did the build cost on WOS-143 and CLINC150?
3 comments

We thank the reviewer for recognizing the practical relevance of label-only, offline knowledge construction and the consistent improvements of SLKR with Qwen3.5-4B across all four datasets. We address the parameter choices, production-scale evidence, reproducibility, and the two implementation questions below.

W1: Justification of the retrieval budgets KK and MM.

KK and MM control distinct aspects of inference: KK determines candidate-label coverage, while MM limits the number of candidate-conditioned boundary rules supplied to the reader. Our first-stage retrieval already shows that Recall@1515 is 97.18% (BANKING77), 97.47% (CLINC150), 95.90% (WOS-143), and 98.32% (Book-38), compared with Recall@55 of 83.51%, 92.38%, 87.81%, and 81.38%, respectively. Thus, a very small KK imposes a substantial upper bound on end-to-end accuracy. The same absolute KK was used across datasets rather than choosing a different fraction of each label inventory.

To assess candidate-budget sensitivity, we evaluate K∈{5,10,15,20,30}K\in\{5,10,15,20,30\} with M=5M=5 and the same Qwen3.5-4B reader.

Table: Sensitivity to candidate budget KK with M=5M=5 (accuracy / macro-F1, %).

DatasetK=5K=5K=10K=10K=15K=15K=20K=20K=30K=30
BANKING7771.33 / 70.9574.25 / 73.9477.45 / 76.6377.14 / 76.3977.40 / 76.44
CLINC15084.73 / 83.9388.64 / 88.0889.81 / 89.3389.87 / 89.3690.00 / 89.55
WOS-14357.37 / 55.7559.10 / 57.4660.17 / 58.2060.38 / 58.4860.77 / 58.88
Book-3860.89 / 60.6168.72 / 68.3870.58 / 70.2572.63 / 71.8574.30 / 73.20

reports accuracy and macro-F1. Increasing KK from 5 to 15 improves accuracy by 2.80--9.69 percentage points across datasets. Increasing KK further to 30 changes accuracy by −0.05-0.05, +0.19+0.19, +0.60+0.60, and +3.72+3.72 points on BANKING77, CLINC150, WOS-143, and Book-38, respectively; the largest remaining gain is on Book-38. We therefore view K=15K=15 as a fixed candidate-coverage/context-budget trade-off, not a universally optimal choice.

For boundary-rule budget sensitivity, we evaluate M∈{0,1,3,5,10,All}M\in\{0,1,3,5,10,\mathrm{All}\} at K=15K=15 with Qwen3.5-4B.

Table: Boundary-budget sensitivity at K=15K=15 with Qwen3.5-4B (accuracy / macro-F1, %). New M=1,3,10M=1,3,10 results are combined with the original boundary ablation. Boundary tokens denote the average per-input boundary context.

Boundary policyWOS-143BANKING77CLINC150Book-38Boundary tokens
M=0M=0 (no boundary)59.92 / 57.6777.05 / 75.9989.13 / 88.5766.85 / 66.240
M=1M=1 (hybrid)59.68 / 57.7677.11 / 76.3189.49 / 88.9769.83 / 68.460.07k
M=3M=3 (hybrid)59.96 / 58.0977.89 / 77.0589.67 / 89.1569.83 / 69.110.21k
M=5M=5 (hybrid)60.17 / 58.2077.45 / 76.6389.81 / 89.3370.58 / 70.250.35k
M=10M=10 (hybrid)60.62 / 58.6277.79 / 77.0689.71 / 89.2368.72 / 62.890.70k
All boundaries60.36 / 58.3477.47 / 76.8289.95 / 89.4467.04 / 64.712.51k
Random M=5M=558.72 / 56.9276.69 / 75.7889.41 / 88.8866.48 / 62.550.36k

The M=5M=5 hybrid policy improves over M=0M=0 on all four datasets. Compared with M=5M=5, increasing to M=10M=10 improves WOS-143 accuracy by 0.45 points but decreases Book-38 accuracy by 1.86 points. The All-boundaries policy is not consistently better and uses substantially more boundary context (2.51k versus 0.35k tokens per input for hybrid M=5M=5). Thus, M=5M=5 is a fixed, practical retrieval budget rather than a universally optimal value. Random M=5M=5 is included separately to distinguish relevance-based selection from random rule selection.

W2: Billion-scale cost feasibility.

We agree that Book-38 (719 books, including 179 test items) is an industrial quality-evaluation sample, not a billion-document throughput benchmark. To substantiate the relevance of SLKR to large-scale classification, we additionally analyze the projected online inference cost for a workload of one billion documents, under the same Qwen3.5-4B GPU-based pricing and average-token assumptions used in our cost comparison.

Table: Projected online inference cost for classifying one billion documents with Qwen3.5-4B (USD). The budget of $200,000 is illustrative.

MethodCost for 1B documents (USD)Within budget?
Label-only ZS$82,500Yes
Definition-Augmented ZS$247,600No
1-shot / label$431,000No
Retrieved ICL$110,000Yes
SLKR-Desc (M=0M=0)$91,700Yes
SLKR (M=5M=5)$128,400Yes
Gen-Z (single definition)$1,394,500No
Gen-Z (multi-description)$13,944,900No

shows the resulting budget feasibility under an illustrative $200,000 online-inference budget.

Under this projection, SLKR requires approximately $128,400, versus $247,600 for Definition-Augmented ZS. Among the methods meeting the illustrative budget, SLKR achieves the highest accuracy on all four evaluated datasets with Qwen3.5-4B.

Q1: What changes across seeds? Was Gen-Z given the same definitions?

The original three runs use seeds 42, 43, and 44 for boundary-rule randomization, not reader decoding. Reader decoding is configured with temperature=0, and no decoding seed is passed. The full test set and split are fixed, no demonstrations are sampled for SLKR, and the knowledge, code, and tokenizer have matching manifest hashes across runs. In the Random-M=5M=5 policy, the seed changes both the selected rule subset and its ordering; in the all-candidate-boundary policy, it changes rule ordering only. The hybrid-retrieval and no-boundary configurations do not use these seeds. Their run-to-run differences, if any, cannot be attributed to seed-controlled rule randomization: LLM inference can remain nondeterministic even with temperature zero. Thus, the reported variability should not be interpreted as variation over data splits, knowledge construction, or reader decoding seeds. All definition-based comparisons use the same GPT-5.6 Sol-generated label definitions.

We additionally corrected a naming error: the original row called "Gen-Z (Label + definition)" was actually Definition-Augmented Zero-Shot, using direct label prediction rather than Gen-Z conditional-likelihood scoring. We have now implemented conditional-likelihood Gen-Z with (i) the same single definition and (ii) the original definition plus ten GPT-5.6 Sol-generated paraphrases per label. In the latter setting, ten descriptions per label are sampled and scores aggregated, with ten description-subset draws. This sampling procedure is distinct from the three-run protocol for SLKR.

The new Gen-Z results below use the same GPT-5.6 Sol-generated definitions as SLKR. Both variants use conditional-likelihood scoring rather than direct label prediction; the multi-description variant additionally uses ten paraphrases per label and averages over ten description-subset draws.

Table: Conditional-likelihood Gen-Z results (accuracy / macro-F1, %). Multi-description results show mean ±\pm standard deviation over ten description-subset draws.

ReaderDatasetGen-Z (single definition)Gen-Z (multi-description)
Qwen3.5-4BWOS-14318.58 / 17.7427.28 ±\pm 0.16 / 26.18 ±\pm 0.21
BANKING7750.36 / 47.5263.43 ±\pm 0.33 / 62.85 ±\pm 0.32
CLINC15046.13 / 45.2869.01 ±\pm 0.18 / 67.10 ±\pm 0.22
Book-386.15 / 8.7310.61 ±\pm 1.12 / 5.31 ±\pm 1.35
Ministral-3-3BWOS-14327.02 / 25.6643.37 ±\pm 0.20 / 43.55 ±\pm 0.28
BANKING7753.02 / 50.3970.09 ±\pm 0.19 / 69.58 ±\pm 0.22
CLINC15049.89 / 49.0970.83 ±\pm 0.17 / 68.25 ±\pm 0.20
Book-3821.79 / 11.9028.10 ±\pm 1.25 / 17.16 ±\pm 1.27

Q2: How are boundary rules constructed, and what are their counts and costs?

We use a two-stage, label-only LLM procedure. First, for each label, the flagship model ranks the other labels by semantic confusability and returns up to 20 nearest/confusable labels. Second, we traverse proposed label pairs and, before requesting rule generation, check the remaining capacity of both labels. Each label may be associated with at most 20 boundary rules in total. For a pair (a,b)(a,b) with current counts na,nbn_a,n_b, the allowed output budget is

B(a,b)=min⁡{20−na,  20−nb}.B(a,b)=\min\{20-n_a,\;20-n_b\}.

If B(a,b)=0B(a,b)=0, the pair is skipped without an LLM generation call. Otherwise the model generates between min⁡(10,B(a,b))\min(10,B(a,b)) and B(a,b)B(a,b) distinct discriminative rules; thus fewer than ten rules are permitted when the remaining capacity is small. Each generated rule is indexed under both labels and consumes one unit of capacity for each; reversed pairs are not generated again. These constraints apply before generation, rather than trimming over-budget output afterward. Both stages use only the task description and label inventory, not labeled examples or test inputs.

Consequently, with NN labels, the number of unique rules is at most 10N10N when each rule is counted against two label capacities;

We additionally report observed boundary-rule construction statistics and approximate costs for both builders:

DatasetGPT-5.6 Sol RulesTokensCost ($)Qwen3.5-4B RulesTokensCost ($)
BANKING7767843,4870.8768256,5960.0052
CLINC1501,39981,3391.631,35293,4680.0086
WOS v21,376118,8152.381,274103,5240.0095
Book-3828317,7130.3528225,6920.0024
Total3,736261,3545.233,590279,2800.0256

Cost estimation protocol. GPT-5.6 Sol costs use 20 usd per million tokens, treating the reported tokens as output tokens for an approximate upper-rate estimate; the exact bill requires separate input/output counts (input tokens are priced at 4 per million). Qwen3.5-4B costs extrapolate the approximate GPU-based online inference rate of 0.0917 per million input tokens used in our prior comparison.

The 4B builder produces 3,590 rules versus 3,736 for GPT-5.6 Sol (96.1% as many), while consuming 279,280 versus 261,354 reported tokens (6.9% more).

Reviewer_CHNkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SLKR, a label-centric retrieval-augmented framework for zero-shot text classification. Instead of retrieving documents or labeled demonstrations, SLKR constructs a reusable label-side knowledge store offline using a flagship LLM. The store contains semantic definitions, retrieval keywords, and pairwise boundary rules for potentially confusable labels. At inference time, hybrid semantic and lexical retrieval first selects Top-K candidate labels and then retrieves up to Top-M input-relevant boundary rules among those candidates. A lightweight LLM performs the final classification in one call. Experiments on WOS-143, BANKING77, CLINC150, and a 719-instance multilingual industrial book-classification dataset (Book-38), show improvements over several zero-shot and few-shot baselines, while the boundary ablation suggests that selective rule retrieval can reduce context length relative to including all available rules.

Critical strengths
  1. The paper targets a practical and relevant classification setting. Repeated use of a flagship LLM can be expensive for large-scale labeling, particularly when task-specific labeled data are unavailable. SLKR offers a reasonable division of labor: expensive label understanding is performed once for a fixed label inventory, while individual inputs are processed by a lightweight reader.
  2. The label-side knowledge representation is clear and complementary. The three offline outputs serve distinct purposes: definitions make label scope explicit, keywords improve lexical access, and pairwise boundary rules encode distinctions between potentially confusable classes. The online pipeline then retrieves candidate labels before filtering and ranking only the boundaries connecting those candidates. This is a coherent design that avoids placing every label description and every boundary into every prompt.
  3. The evaluation includes several useful baselines and two reader families. The paper evaluates three public datasets and one industrial multilingual book-classification dataset, and compares against label-only prompting, Gen-Z, one-shot-per-label prompting, and retrieved ICL. SLKR obtains the best results on all four datasets with Qwen3.5-4B and on three of four with Ministral-3-3B. The reported Recall@15 is above 95% on all datasets, indicating that the first retrieval stage usually retains the gold label.
  4. The boundary ablation provides evidence for input-conditioned selection. The comparison among no boundaries, all eligible boundaries, random Top-5 rules, and hybrid Top-5 retrieval is useful. The selected rules use about 0.35k boundary tokens compared with 2.51k for all boundaries, while performing competitively on the public benchmarks and substantially better on Book-38. The random-rule control also helps show that simply adding an arbitrary set of rule text is not sufficient.
Critical weaknesses
  1. The methodological novelty is somewhat incremental. The framework combines LLM-generated label descriptions, keyword expansion, pairwise natural-language rules, hybrid dense/lexical retrieval, and a lightweight reader. Each component is intuitive, and the overall pipeline is useful, but the paper does not yet make sufficiently clear which part constitutes the central algorithmic advance beyond organizing known label-description and retrieval techniques into a label-centric pipeline. The distinction from prior boundary-aware and confusion-aware classification methods would benefit from a more direct conceptual and empirical comparison.
  2. The quality of the offline knowledge store is a critical but insufficiently validated assumption. Definitions, keywords, and boundary rules are generated by a single flagship model (GPT-5.6 Sol) and then frozen. Since these artifacts directly guide candidate retrieval and final classification, errors or omissions in the generated knowledge can propagate through the entire pipeline. The paper states that construction uses only the label inventory and task description, but does not provide a systematic human or expert audit of definition correctness, boundary validity, coverage, or consistency. It also remains unclear how sensitive the method is to the choice of knowledge-generation model.
  3. The evidence for large-scale and real-world generalization is limited. The public benchmarks contain 77–150 labels, while Book-38 contains only 38 labels and 719 books, including 179 test instances. This is useful as an industrial case study, but it does not establish performance on genuinely large label inventories. This matters because pairwise boundary construction has a worst-case O(N²) cost, as acknowledged in the paper. The practical scalability of boundary generation, storage, and maintenance for hundreds or thousands of labels is not demonstrated.
  4. The cost comparison exposes a substantial quality–cost trade-off but is too narrow to support a broad deployment claim. The direct GPT-5.6 comparison is reported only on BANKING77. It achieves 85.30 Macro-F1, compared with 76.63 for SLKR with Qwen3.5-4B. SLKR's recurring online cost is much lower, but the amortized figure depends on the assumed workload and the number of instances over which the $3.4 construction cost is distributed. The paper would benefit from a broader quality–cost frontier across datasets and workloads, as well as end-to-end latency and throughput measurements. The current evidence supports lower recurring cost, not equivalent classification quality.
  5. The retrieval stage imposes a hard candidate-set ceiling, while the main results provide limited uncertainty reporting. With K fixed at 15, Recall@15 ranges from 95.90% to 98.32%; consequently, some gold labels are excluded before the reader makes its decision. The paper would be stronger with an oracle or error-decomposition analysis showing how much final error comes from candidate omission versus reader reasoning. In addition, the main classification table reports means over three seeds without corresponding standard deviations or confidence intervals, making small improvements harder to interpret.
Questions for authors
  1. How often are the generated definitions and boundary rules incomplete, incorrect, or inconsistent? Could the authors provide a human/expert audit or an evaluation across multiple flagship generators to establish that SLKR is not overly dependent on GPT-5.6-specific outputs?
  2. Given the worst-case O(N²) pairwise construction cost, how does offline generation scale as the label inventory grows to several hundred or several thousand classes? Could candidate-pair generation reduce this cost without sacrificing boundary coverage?
  3. Could the authors report end-to-end cost and latency across a wider range of input volumes and datasets, including the break-even point at which offline construction becomes worthwhile? How should users choose SLKR over direct flagship inference when the latter provides substantially higher Macro-F1 on BANKING77?
  4. Could the authors provide an oracle-candidate experiment or an error decomposition that separates Top-K retrieval misses from final reader errors? How sensitive are the results to K and M, and do the same settings remain appropriate across all four datasets?
3 comments

We thank the reviewer for recognizing the practical relevance of label-only offline knowledge construction, the complementary roles of definitions, keywords, and boundaries, and the effectiveness of candidate-conditioned boundary retrieval. We provide additional builder, candidate-budget, boundary-budget, construction-cost, and cost--accuracy analyses.

W1: Methodological contribution and relation to prior work.

The central contribution is an explicit mechanism for transferring flagship-model knowledge into a reusable label-side store, followed by candidate-conditioned retrieval of discriminative boundaries and single-call lightweight classification. The contribution is not a new dense retriever or an individual prompting primitive. Unlike reflection-oriented prompting, SLKR materializes reusable definitions, keywords, and pairwise distinctions offline, so flagship reasoning is not repeated for each input. Gupta et al. (2026) appeared on arXiv approximately one month before our ICLR submission; our method was developed independently and that related work was cited in the submission. To isolate the benefit beyond flagship-generated definitions, we compare methods using the same GPT-5.6 Sol definitions and reader.

Table: Matched-definition comparisons (accuracy / macro-F1, %). All methods use the same GPT-5.6 Sol-generated label definitions within each reader group.

ReaderMethodWOS-143BANKING77CLINC150Book-38
Qwen3.5-4BDefinition-Aug. ZS57.37 / 57.0672.95 / 71.5885.64 / 84.6661.45 / 60.81
SLKR-Desc (M=0M=0)59.92 / 57.6777.05 / 75.9989.13 / 88.5766.85 / 66.24
SLKR (M=5M=5)60.17 / 58.2077.45 / 76.6389.81 / 89.3370.58 / 70.25
Ministral-3-3BDefinition-Aug. ZS34.49 / 35.5565.91 / 64.7981.36 / 80.3546.37 / 44.87
SLKR-Desc (M=0M=0)52.93 / 51.3167.76 / 66.7084.00 / 83.2253.07 / 49.93
SLKR (M=5M=5)55.83 / 53.7573.44 / 72.8185.69 / 85.0758.66 / 52.29

Candidate selection (SLKR-Desc) improves accuracy over definition-augmented zero-shot on all four datasets for both readers, and retrieved boundaries add further gains. This provides component-level evidence beyond simply supplying better label definitions.

W2 / Q1: Knowledge-builder dependence and quality.

We replace GPT-5.6 Sol with \qwen{} for offline knowledge construction while keeping the downstream reader and pipeline unchanged . With the smaller builder, SLKR accuracy is 59.28%, 72.92%, 86.16%, and 66.48% on WOS-143, BANKING77, CLINC150, and Book-38. Although the stronger builder improves absolute performance, retrieved boundary rules still improve over SLKR-Desc on all four datasets: +0.56, +3.21, +3.29, and +4.47 accuracy points. On BANKING77 and CLINC150, the incremental boundary gains are larger with the 4B builder (+3.21 and +3.29) than with the flagship builder (+0.40 and +0.68).

To directly examine knowledge quality, we conducted a preliminary paired audit of 120 knowledge units sampled with a fixed random seed across the four datasets. The audit covered label definitions, keyword sets, and groups of boundary rules. As summarized in

Table: Preliminary audit of sampled offline knowledge units. Entries report confirmed errors / audited available units (error rate). Nine missing Qwen units are reported separately and excluded from its content-error denominator.

Knowledge typeGPT-5.6 referenceQwen3.5-4B
Definitions1/20 (5.0%)5/14 (35.7%)
Keyword sets0/20 (0.0%)8/17 (47.1%)
Boundary-rule groups0/20 (0.0%)2/20 (10.0%)
Total available units1/60 (1.7%)15/51 (29.4%)
Missing units09

one error was confirmed among 60 audited GPT-5.6 reference units (1.7%). For Qwen, 15 errors were confirmed among 51 available units (29.4%); another 9 sampled units were missing and are reported separately, not counted as content errors. The largest discrepancies were observed in keyword sets (8/17, 47.1%) and definitions (5/14, 35.7%), while boundary-rule groups had a lower confirmed error rate (2/20, 10.0%).

The audited cases illustrate the semantic nature of these errors. For example, Qwen defined jump_start as starting a car without a conventional key, whereas the corresponding training examples concern jump-starting a car with a dead battery. It defined balance as an outstanding amount rather than the balance of a savings or checking account. It also described verify_top_up as checking top-up progress, whereas the corresponding examples concern verifying a recharge-card PIN. Overall, the builder ablation and audit provide complementary evidence: the former measures downstream classification effects, while the latter directly inspects sampled knowledge artifacts.

W3 / Q2: Pair construction and scaling.

Boundary construction is not exhaustive over all (N2)\binom{N}{2} label pairs. First, for each label, the builder identifies up to 20 potentially confusable labels from the label inventory. Second, proposed pairs are traversed with a capacity constraint: each label can participate in at most 20 stored boundary rules. For a proposed pair (a,b)(a,b) with current rule counts na,nbn_a,n_b, the remaining generation budget is

B(a,b)=min⁡{20−na,20−nb}.B(a,b)=\min\{20-n_a,20-n_b\}.

Pairs with B(a,b)=0B(a,b)=0 are skipped before generation, and reversed pairs are not regenerated. Each unique rule consumes one slot for each endpoint. Therefore, the number of stored unique rules is bounded by 10N10N. This is a storage/rule-count bound; it does not imply that discovering confusable labels from the full inventory has linear time or token complexity.

We acknowledge that SLKR is primarily designed for workloads with a relatively fixed label inventory and a large volume of inputs to classify. In such settings, the number of documents is typically much larger than the number of predefined labels, allowing the one-time label-side knowledge construction cost to be amortized across many inputs. Conversely, SLKR is not intended for extreme regimes in which the label inventory is enormous (e.g., billions of labels) but only a handful of input instances need classification. In that regime, constructing and maintaining the label-side knowledge base and retrieval index could impose disproportionate overhead, making the current approach unsuitable.

**Table: Observed boundary-rule construction statistics. **

DatasetGPT-5.6 Sol RulesTokensCost ($)Qwen3.5-4B RulesTokensCost ($)
BANKING7767843,4870.8768256,5960.0052
CLINC1501,39981,3391.631,35293,4680.0086
WOS1,376118,8152.381,274103,5240.0095
Book-3828317,7130.3528225,6920.0024
Total3,736261,3545.233,590279,2800.0256

reports the observed rule counts and construction-token statistics on our four datasets. We have not measured construction or maintenance at thousand-label or billion-label scale, and do not claim demonstrated scalability to such settings.

W4 / Q3: Cost--quality trade-off and amortization.

We agree that lower cost does not imply equivalent quality to direct flagship classification: on BANKING77, the reported direct GPT-5.6 Sol macro-F1 is 85.30 versus 76.63 for SLKR with \qwen. SLKR targets repeated classification with a fixed label inventory and a constrained online budget. Under the same \qwen{} inference-cost assumptions, SLKR costs approximately 0.1284 usd per 1,000 inputs versus 0.2476 for definition-augmented zero-shot, with higher accuracy on all four datasets.

**Table: Approximate Qwen3.5-4B online inference costs; offline construction and infrastructure overhead excluded. **

MethodUSD / 1,000 inputsUSD / 1B inputs
Label-only ZS0.082582,500
Definition-Augmented ZS0.2476247,600
1-shot / label0.4310431,000
Retrieved ICL0.1100110,000
SLKR-Desc (M=0M=0)0.091791,700
SLKR (M=5M=5)0.1284128,400
Gen-Z (single definition)1.39451,394,500
Original Gen-Z (multi-description)13.944913,944,900

extends the online comparison to other baselines.

W5 / Q4: Retrieval ceiling, K/MK/M sensitivity, and uncertainty.

We agree that gold-label omission in the first stage is irreversible. The original Recall@15 values are 97.18%, 97.47%, 95.90%, and 98.32% for BANKING77, CLINC150, WOS-143, and Book-38. We add a candidate-budget sweep K∈{5,10,15,20,30}K\in\{5,10,15,20,30\} at M=5M=5

Table: Candidate-budget sensitivity at M=5M=5 (accuracy / macro-F1, %), Qwen3.5-4B reader.

DatasetK=5K=5K=10K=10K=15K=15K=20K=20K=30K=30
BANKING7771.33 / 70.9574.25 / 73.9477.45 / 76.6377.14 / 76.3977.40 / 76.44
CLINC15084.73 / 83.9388.64 / 88.0889.81 / 89.3389.87 / 89.3690.00 / 89.55
WOS-14357.37 / 55.7559.10 / 57.4660.17 / 58.2060.38 / 58.4860.77 / 58.88
Book-3860.89 / 60.6168.72 / 68.3870.58 / 70.2572.63 / 71.8574.30 / 73.20

and boundary-budget sweep M∈{0,1,3,5,10,All}M\in\{0,1,3,5,10,\mathrm{All}\} at K=15K=15.

Table: Boundary-budget sensitivity at K=15K=15 (accuracy / macro-F1, %). Boundary tokens are average per-input boundary context.

PolicyWOS-143BANKING77CLINC150Book-38Tokens
M=0M=059.92 / 57.6777.05 / 75.9989.13 / 88.5766.85 / 66.240
M=1M=1 hybrid59.68 / 57.7677.11 / 76.3189.49 / 88.9769.83 / 68.460.07k
M=3M=3 hybrid59.96 / 58.0977.89 / 77.0589.67 / 89.1569.83 / 69.110.21k
M=5M=5 hybrid60.17 / 58.2077.45 / 76.6389.81 / 89.3370.58 / 70.250.35k
M=10M=10 hybrid60.62 / 58.6277.79 / 77.0689.71 / 89.2368.72 / 62.890.70k
All boundaries60.36 / 58.3477.47 / 76.8289.95 / 89.4467.04 / 64.712.51k
Random M=5M=558.72 / 56.9276.69 / 75.7889.41 / 88.8866.48 / 62.550.36k

Increasing KK from 5 to 15 improves accuracy by 2.80--9.69 points; at K=30K=30, Book-38 gains another 3.72 points relative to K=15K=15, whereas changes on the other datasets are smaller. The hybrid M=5M=5 setting improves on M=0M=0 on all datasets; using all boundaries is not consistently better and increases average boundary context from 0.35k to 2.51k tokens. Thus K=15,M=5K=15,M=5 is a shared practical trade-off, not a universally optimal setting.

Error decomposition: candidate retrieval versus reader classification.

To quantify the two sources of error without conflating them, we decompose the final error into (i) retrieval misses, where the gold label is absent from the Top-15 candidates, and (ii) reader errors, where the gold label is retrieved but the reader predicts another label. The conditional reader error rate is computed only over examples whose gold labels are retrieved. The reader-error contribution in the full test set equals this conditional rate multiplied by Recall@15.

Table: Error decomposition at K=15K=15. Reader error (all) is the contribution to the full test set; reader error (conditional) is measured only among examples whose gold labels are retrieved. All values are percentages.

DatasetnnRecall@15Retrieval missReader error (all)Reader error (conditional)Accuracy
BANKING773,08097.182.8219.7320.3077.45
CLINC1504,50097.472.537.667.8689.81
WOS-1439,36795.904.1035.7337.2660.17
Book-3817998.321.6827.7428.2170.58

reports the resulting decomposition. Retrieval misses account for only 1.68%--4.10% of test examples, whereas reader errors among retrieved-gold examples are substantially more frequent (7.86%--37.26%). Thus, under the current K=15K=15 setting, the dominant source of final errors is reader classification rather than candidate omission.

Reviewer_EW69OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SLKR (Semantic Label Knowledge Retrieval), a label-centric RAG framework for zero-shot text classification. The key idea is to leverage a powerful LLM to construct reusable label knowledge offline, including semantic definitions, retrieval keywords, and pairwise boundary rules for potentially confusable labels. During inference, the framework employs two-stage retrieval to identify candidate labels and relevant boundary rules, followed by classification using a lightweight LLM. Experiments on three public benchmarks and one industrial dataset demonstrate improvements in classification performance without task-specific labeled examples, while reducing recurring inference costs through reusable offline knowledge.

Critical strengths
  1. The motivation of the paper is clear and strong enough. It tries to address a practical and importatn challenge that the high inference cost of frontier LLM in large-scale text classification. Separating offline knowledge construction from online classification is a reasonable design.
  2. The authors propose a two-stage retrieval architecture effectively combines semantic and lexical retrieval to reduce the classification context while maintaining high coverage.
  3. The evaluation covers multiple different datasets and multiple lightweight LLMs from multiple model companies. Meanwhile, the ablation studies also shows the effectiveness of different modules within the proposed framework.
Critical weaknesses
  1. The novelty of this paper is smoewhat limited, and it looks like the framework integrates existing techniques, including label-description augmentation, hybrid retrieval, eciprocal rank fusion, and LLM-generated boundary rules. Although the overall architecture is practically useful, the individual components offer limited methodological innovation. The paper would benefit from a clearer demonstration of its unique technical contributions beyond combining established techniques.
  2. While the paper includes boundary-retrieval ablations, it lacks a systematic analysis of the individual contributions of its three core knowledge components: definitions, keywords, and boundary rules. Comparisons involving definitions only, definitions with keywords, and different retrieval configurations would help disentangle their effects. Consequently, it remains unclear how much of the overall improvement can be attributed specifically to the proposed boundary knowledge.
  3. The framework heavily relies on label definitions and boundary rules generated by a powerful LLM, yet the paper provides limited analysis of their correctness, consistency, or potential biases.
Questions for authors
  1. Could you provide a more detailed ablation of label definitions, generated keywords, and boundary rules? It would be helpful to understand which parts contribute most to the final performance.
  2. Since the rules are generated by a flagship LLM without labeled examples, how do you check whether they are correct? Have you observed cases where incorrect or misleading rules actually hurt classification?
  3. For example, could you compare it against an embedding-based classifier using the same generated label descriptions?
2 comments

We thank the reviewer for recognizing the practical motivation of SLKR, the effectiveness of its two-stage retrieval architecture, and the breadth of our evaluation. We appreciate the constructive comments on methodological contribution, the roles of individual knowledge components, and the reliability of LLM-generated knowledge. We address these concerns with additional matched-knowledge comparisons, boundary-retrieval ablations, and a preliminary manual audit of generated knowledge.

W1: Methodological novelty and distinction from existing techniques.

We agree that individual components of SLKR, such as hybrid retrieval, reciprocal rank fusion, and LLM-generated label descriptions, are established techniques. Our contribution is not a new retrieval primitive or an isolated prompting technique. Rather, SLKR provides an explicit knowledge-transfer framework for zero-shot classification: a flagship LLM constructs a reusable label-side knowledge base containing semantic definitions, retrieval keywords, and pairwise discriminative boundaries; at inference time, candidate-conditioned retrieval selects relevant labels and boundary rules for each input, allowing a lightweight reader to classify in a single call without repeatedly invoking the flagship model.

Unlike reflection-oriented prompting, SLKR explicitly materializes flagship-model knowledge into a reusable label-side knowledge base and separates offline knowledge construction from online classification. To demonstrate the contribution beyond simply providing stronger label descriptions, we additionally compare methods using exactly the same generated definitions and downstream reader.

Table: Matched-definition comparison (accuracy / macro-F1, %). All methods within each reader group use the same GPT-5.6 Sol-generated label definitions.

ReaderMethodWOS-143BANKING77CLINC150Book-38
Qwen3.5-4BDefinition-Augmented ZS57.37 / 57.0672.95 / 71.5885.64 / 84.6661.45 / 60.81
SLKR-Desc (M=0M=0)59.92 / 57.6777.05 / 75.9989.13 / 88.5766.85 / 66.24
SLKR (M=5M=5)60.17 / 58.2077.45 / 76.6389.81 / 89.3370.58 / 70.25
Ministral-3-3BDefinition-Augmented ZS34.49 / 35.5565.91 / 64.7981.36 / 80.3546.37 / 44.87
SLKR-Desc (M=0M=0)52.93 / 51.3167.76 / 66.7084.00 / 83.2253.07 / 49.93
SLKR (M=5M=5)55.83 / 53.7573.44 / 72.8185.69 / 85.0758.66 / 52.29

These controlled comparisons show that candidate-label retrieval and boundary knowledge each provide additional gains.

W2/Q1: Contributions of definitions, keywords, and boundary rules.

To disentangle the effects of label knowledge, we compare Definition-Augmented Zero-Shot, \method{SLKR}-Desc (without boundary rules), and full \method{SLKR}. All three methods use the same GPT-5.6 Sol-generated label definitions and the same reader. As shown above, \method{SLKR}-Desc improves accuracy over Definition-Augmented Zero-Shot on all four datasets for both readers. Adding candidate-conditioned boundary retrieval further improves accuracy on all eight reader--dataset combinations. In particular, the boundary component yields additional accuracy gains of 3.73 and 5.59 percentage points on Book-38 with Qwen3.5-4B and Ministral-3-3B, respectively.

We further examine the boundary-rule budget at fixed K=15K=15 using Qwen3.5-4B.

Table: Boundary-retrieval ablation at K=15K=15 with Qwen3.5-4B (accuracy / macro-F1, %). Boundary tokens are average per-input context.

Boundary policyWOS-143BANKING77CLINC150Book-38Boundary tokens
M=0M=0 (no boundary)59.92 / 57.6777.05 / 75.9989.13 / 88.5766.85 / 66.240
M=1M=1 (hybrid)59.68 / 57.7677.11 / 76.3189.49 / 88.9769.83 / 68.460.07k
M=3M=3 (hybrid)59.96 / 58.0977.89 / 77.0589.67 / 89.1569.83 / 69.110.21k
M=5M=5 (hybrid)60.17 / 58.2077.45 / 76.6389.81 / 89.3370.58 / 70.250.35k
M=10M=10 (hybrid)60.62 / 58.6277.79 / 77.0689.71 / 89.2368.72 / 62.890.70k
All boundaries60.36 / 58.3477.47 / 76.8289.95 / 89.4467.04 / 64.712.51k
Random M=5M=558.72 / 56.9276.69 / 75.7889.41 / 88.8866.48 / 62.550.36k

shows that hybrid retrieval with M=5M=5 improves over the no-boundary setting on all four datasets. It also outperforms random selection of five rules on all four datasets, indicating that the gain is not merely due to adding more context, but benefits from relevance-based rule selection. Including all boundary rules is not consistently better and increases the average boundary context from 0.35k to 2.51k tokens per input. Thus, selective, input-relevant boundary retrieval provides a favorable accuracy--context trade-off.

W3/Q2: Reliability of LLM-generated knowledge and potential errors.

We conducted a preliminary manual audit of 120 sampled knowledge units, covering label definitions, keyword sets, and groups of boundary rules.

Table: Preliminary manual audit of sampled offline knowledge units. Entries show confirmed errors / audited available units (error rate).

Knowledge typeGPT-5.6 SolQwen3.5-4B
Definitions1/20 (5.0%)5/14 (35.7%)
Keyword sets0/20 (0.0%)8/17 (47.1%)
Boundary-rule groups0/20 (0.0%)2/20 (10.0%)
Total available units1/60 (1.7%)15/51 (29.4%)
Missing units09

reports the confirmed errors among available audited units. GPT-5.6 Sol had one confirmed error among 60 audited units (1.7%). For Qwen3.5-4B, 15 errors were confirmed among 51 available units (29.4%); nine additional sampled units were missing and are reported separately rather than counted as content errors.

The audit identified semantic mismatches in Qwen-generated knowledge. For example, jump_start was defined as starting a car without a conventional key, whereas the corresponding examples concern jump-starting a car with a dead battery. The definition of balance referred to an outstanding amount rather than the balance of a savings or checking account. The description of verify_top_up referred to checking top-up progress, whereas the corresponding task concerns verifying a recharge-card PIN.

The audit confirms that generated knowledge can contain errors, including errors in boundary-rule groups (2 of 20 audited Qwen groups). However, this preliminary sample is small and should not be interpreted as an estimate of the error rate over the complete knowledge base. Moreover, our current analysis does not isolate the causal effect of each audited erroneous rule on individual classification predictions; we therefore do not claim that every identified error necessarily changes the final prediction. The builder ablation provides complementary downstream evidence: replacing GPT-5.6 Sol with Qwen3.5-4B reduces absolute performance, while boundary knowledge remains beneficial with the smaller builder.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.