- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SLKR, a framework in which a flagship model generates label definitions, keywords, and class boundary rules offline. Relevant knowledge is then retrieved online to support zero-shot text classification with a smaller model. The paper evaluates the framework on four datasets using two small models and compares its performance and cost against direct flagship-model classification on BANKING77.
Critical strengths
- No task-specific labeled data is required, and the generated knowledge is reusable. The separation between offline knowledge construction and online classification is sensible. Table 4 shows improvements over direct zero-shot classification across both model families.
- The boundary-retrieval ablation is well designed. Retrieving relevant rules outperforms randomly selecting the same number of rules. It also reduces the average boundary context from approximately 2.51k to 0.35k tokens while maintaining performance close to using all eligible rules on the public benchmarks.
Critical weaknesses
-
The methodological novelty appears limited. Label-description augmentation, RRF-based hybrid retrieval, and candidate-label selection are established techniques. Boundary rules for confusable labels also have precedents, including Gupta et al. (2026). The main distinction is constructing these rules without task instances or classifier feedback and retrieving them within candidate pairs. The paper should more clearly establish the contribution beyond combining these components.
-
The baselines do not sufficiently establish competitiveness or isolate the source of improvement. (i) Limited coverage of strong methods. Label-only ZS, 1-shot/label, and Retrieved ICL are reasonable basic baselines, but do not adequately cover representative zero-shot, label-knowledge augmentation, or small–large model collaboration methods. (ii) Unequal access to flagship knowledge. SLKR uses knowledge generated by GPT-5.6 Sol, whereas these baselines rely on the 3B/4B reader and their respective prompts or demonstrations. Table 4 therefore does not isolate gains from flagship knowledge from those of the proposed retrieval framework. A simpler baseline using the same flagship-generated knowledge would help. (iii) The flagship builder’s necessity remains untested. Having Qwen3.5-4B generate the same types of knowledge would reveal whether the stronger builder provides sufficient additional benefit to justify its cost.
-
Evidence for a competitive cost–performance trade-off is insufficient. (i) The cost comparison is too narrow. Table 6 covers only BANKING77 and two approaches, without placing small-model prompting, SLKR-Desc, and other comparable zero-shot methods under a common cost accounting. (ii) Savings come with a substantial quality loss. Accuracy falls from 86.01% to 77.45%, a drop of 8.56 percentage points. This establishes lower cost, but not a competitive trade-off. Broader comparisons across datasets and workloads are needed.
Questions for authors
- Was Gen-Z implemented according to the original method? Does the implementation compute the likelihood of the input text conditioned on each label description, or does it simply add label definitions to the prompt and directly predict the class? Please clarify the scoring procedure and any departures from the original method.
- Do SLKR’s advantages hold against representative zero-shot methods? The current comparisons mainly cover basic prompting and retrieval configurations. Could the authors include representative approaches based on label-semantic augmentation, label retrieval, or small–large model collaboration, and clearly specify each method’s use of labeled, unlabeled, and synthetic data?
- Is a flagship model necessary for knowledge construction? If Qwen3.5-4B generates the label definitions, keywords, and boundary rules while the subsequent retrieval and classification pipeline remains unchanged, how do accuracy and total cost change? Do the additional gains from the flagship model justify its construction cost?
- Does SLKR offer a competitive cost–performance trade-off? On BANKING77, the cost reduction comes with an accuracy drop of 8.56 percentage points. Could the authors compare performance and cost against more comparable zero-shot methods across additional datasets?
3 comments
The above table presents these three methods side by side for both readers. The original row labeled "Gen-Z (Label + definition)" is renamed Definition-Augmented ZS to reflect its direct label-prediction procedure.
Results under matched definition knowledge.
With Qwen3.5-4B, SLKR improves accuracy over Definition-Augmented ZS by 2.80, 4.50, 4.17, and 9.13 percentage points on WOS-143, BANKING77, CLINC150, and Book-38, respectively. With Ministral-3-3B, the corresponding gains are 21.34, 7.53, 4.33, and 12.29 points. The intermediate SLKR-Desc variant, which uses candidate-label retrieval without boundary rules, already improves accuracy over Definition-Augmented ZS on all four datasets for both readers. Adding retrieved boundary rules further improves accuracy over SLKR-Desc on all four datasets for both readers (Qwen3.5-4B: +0.25, +0.40, +0.68, +3.73 points; Ministral-3-3B: +2.90, +5.68, +1.69, +5.59 points). These matched-definition comparisons show that the improvements are not explained solely by access to GPT-5.6 Sol-generated label definitions; candidate-label selection and boundary knowledge contribute additional performance gains.
Q3: Is a flagship builder necessary?
To investigate the dependence on the knowledge builder, we replaced GPT-5.6 Sol with Qwen3.5-4B for offline knowledge construction while keeping the downstream retrieval and classification pipeline unchanged.
Table: Builder ablation using Qwen3.5-4B as reader (accuracy / macro-F1, %).
| Method | Dataset | GPT-5.6 Sol builder | Qwen3.5-4B builder | Gain (GPT − 4B) |
|---|---|---|---|---|
| SLKR-Desc | WOS | 59.92 / 57.67 | 58.72 / 56.62 | +1.20 / +1.05 |
| BANKING77 | 77.05 / 75.99 | 69.71 / 68.38 | +7.34 / +7.61 | |
| CLINC150 | 89.13 / 88.57 | 82.87 / 81.81 | +6.26 / +6.76 | |
| Book38 | 66.85 / 66.24 | 62.01 / 43.16 | +4.84 / +23.08 | |
| SLKR | WOS | 60.17 / 58.20 | 59.28 / 57.18 | +0.89 / +1.02 |
| BANKING77 | 77.45 / 76.63 | 72.92 / 72.03 | +4.53 / +4.60 | |
| CLINC150 | 89.81 / 89.33 | 86.16 / 85.21 | +3.65 / +4.12 | |
| Book38 | 70.58 / 70.25 | 66.48 / 44.94 | +4.10 / +25.31 |
With the 4B builder, SLKR achieves accuracies of 59.28%, 72.92%, 86.16%, and 66.48% on WOS, BANKING77, CLINC150, and Book38, respectively, corresponding to decreases of 0.89, 4.53, 3.65, and 4.10 percentage points compared with the flagship builder.
Table: Incremental gain from boundary knowledge: SLKR minus SLKR-Desc (accuracy / macro-F1 percentage points).
| Dataset | GPT-5.6 Sol builder | Qwen3.5-4B builder |
|---|---|---|
| WOS | +0.25 / +0.53 | +0.56 / +0.56 |
| BANKING77 | +0.40 / +0.64 | +3.21 / +3.65 |
| CLINC150 | +0.68 / +0.76 | +3.29 / +3.40 |
| Book38 | +3.73 / +4.01 | +4.47 / +1.78 |
boundary knowledge remains effective with the smaller builder, improving accuracy over SLKR-Desc by 0.56, 3.21, 3.29, and 4.47 percentage points across the four datasets. Notably, on BANKING77 and CLINC150, the boundary-rule gains increase from 0.40 and 0.68 percentage points with the flagship builder to 3.21 and 3.29 points with the 4B builder, respectively. This indicates that boundary knowledge remains useful even when generated by a smaller builder. These results demonstrate that SLKR is not dependent on a flagship model for knowledge construction. Stronger builders improve overall classification accuracy, while the boundary-knowledge mechanism remains beneficial even when knowledge is generated by a substantially smaller model.
Q4: Cost--accuracy trade-off.
We extend the cost comparison beyond direct flagship classification to eight prompting, retrieval, and conditional-likelihood methods across the two lightweight readers and a GPT-5.6 Sol API-cost reference. Lightweight-reader costs are approximated based on GPU platform pricing, average input token lengths, and model inference throughput. GPT-5.6 Sol costs use published API token rates; the GPT column is a cost projection rather than an additional accuracy evaluation.
Table: Approximate online inference cost per 1,000 classifications (USD).
GPT-5.6 Sol uses standard API rates ($4/1M input tokens; $20/1M output tokens), with 10 output tokens per direct-prediction request. Gen-Z tokens are cumulative across class-description evaluations; GPT-5.6 Sol conditional-likelihood costs are not reported. Offline knowledge construction is excluded.
| Method | Avg. input tokens | Qwen3.5-4B | Ministral-3-3B | GPT-5.6 Sol |
|---|---|---|---|---|
| Label-only ZS | 0.9k | 0.0825 | 0.0516 | 3.8000 |
| Definition-Augmented ZS | 2.7k | 0.2476 | 0.1547 | 11.0000 |
| 1-shot / label | 4.7k | 0.4310 | 0.2693 | 19.0000 |
| Retrieved ICL | 1.2k | 0.1100 | 0.0688 | 5.0000 |
| SLKR-Desc () | 1.0k | 0.0917 | 0.0573 | 4.2000 |
| SLKR () | 1.4k | 0.1284 | 0.0802 | 5.8000 |
| Gen-Z (single definition) | 15.21k | 1.3945 | 0.8716 | N/A |
| Original Gen-Z (multi-description) | 152.08k | 13.9449 | 8.7155 | N/A |
Cost--performance comparison.
As shown in above, SLKR achieves a favorable accuracy--cost trade-off relative to definition-augmented zero-shot classification. With Qwen3.5-4B, SLKR has an approximate online cost of $0.1284 per 1,000 classifications, compared with $0.2476 for Definition-Augmented ZS, while achieving higher accuracy on all four datasets. With Ministral-3-3B, the corresponding costs are $0.0802 and $0.1547, respectively. SLKR-Desc provides a lower-cost alternative ($0.0917 and $0.0573 for the two readers), whereas full SLKR consistently improves accuracy over SLKR-Desc across the four datasets. The GPT-5.6 Sol column additionally shows the corresponding API-cost projections at published token rates. The conditional-likelihood Gen-Z variants require substantially more cumulative token processing because they score the input against class descriptions, with the multi-description variant aggregating ten descriptions per class. These comparisons extend the evaluation beyond the flagship-direct baseline and characterize the trade-offs among lightweight classification approaches.
Production-scale budget trade-off.
For a production workload of one billion documents, we consider an illustrative online inference budget of $200,000.
Table: Budget-constrained feasibility for classifying one billion documents with Qwen3.5-4B. Costs are approximate online inference expenses in USD; the illustrative budget is $200,000.
| Method | Cost for 1B documents (USD) | Within budget? |
|---|---|---|
| Label-only ZS | $82,500 | Yes |
| Definition-Augmented ZS | $247,600 | No |
| 1-shot / label | $431,000 | No |
| Retrieved ICL | $110,000 | Yes |
| SLKR-Desc () | $91,700 | Yes |
| SLKR () | $128,400 | Yes |
| Gen-Z (single definition) | $1,394,500 | No |
| Original Gen-Z (multi-description) | $13,944,900 | No |
scales the approximate per-1,000-classification costs to this workload. SLKR with Qwen3.5-4B requires approximately 128,400 usd and remains within budget, whereas Definition-Augmented ZS (247,600 usd), 1-shot-per-label prompting (431,000 usd), and the Gen-Z variants exceed the budget. Among the evaluated Qwen3.5-4B methods that can process the full workload within this budget, SLKR achieves the highest accuracy on all four datasets and the original main results). Thus, the relevant trade-off for production-scale classification is accuracy among methods that satisfy the computational budget, rather than accuracy alone. The feasibility comparison concerns online inference costs and excludes offline knowledge construction and infrastructure overhead.
Methodological novelty. Gupta et al. (2026) appeared on arXiv approximately one month before our ICLR submission; SLKR was developed independently, and we cited this closely related work in our original submission. Unlike reflection-oriented prompting, SLKR explicitly transfers flagship-model knowledge into a reusable label-side knowledge base, enabling candidate-conditioned retrieval and lightweight inference without repeated flagship-model reasoning.
Q1: Clarification of Gen-Z implementation. We thank the reviewer for identifying an inaccurate baseline label in our original submission. The row previously named "Gen-Z (Label + definition)" was in fact a Definition-Augmented Zero-Shot baseline: the reader directly predicts a class from all labels and their GPT-5.6 Sol-generated definitions. It did not implement the conditional-likelihood scoring procedure of Gen-Z. We will correct the name and implementation description in the revised manuscript. We have now implemented two additional conditional-likelihood Gen-Z variants, using the same readers and four datasets. Their results are reported as follows.
Table caption: New conditional-likelihood Gen-Z experiments (accuracy / macro-F1, %). Multi-description results report mean ± standard deviation over 10 description-subset draws. The single-definition setting uses SLKR's single label description.
Conditional-likelihood scoring. For each class, we use the same GPT-5.6 Sol-generated definition as in the definition-augmented zero-shot baseline and generate 10 additional paraphrases of that fixed definition using GPT-5.6 Sol, following the multi-description construction in Gen-Z. The resulting 11 descriptions are frozen before evaluation, without using test instances or classifier feedback. The single-definition Gen-Z setting uses only the original definition, whereas the multi-description setting uses the original definition and its paraphrases. For description d and input x, we score only the input tokens, treating the description as an unscored prefix and using plain text rather than a chat template:
\begin{equation} \ell(x\mid d)=\frac{1}{|x|}\sum_{t=1}^{|x|}\log p_\theta(x_t\mid d,x_{<t}). \end{equation}
For the multi-description adaptation, 10 of the 11 descriptions are sampled per class and their length-normalized conditional likelihoods are aggregated arithmetically:
\begin{equation} S(c,x)=\frac{1}{10}\sum_{j=1}^{10}\exp!\left(\ell(x\mid d_{c,j})\right),\qquad \hat c=\arg\max_{c\in\mathcal C}S(c,x). \end{equation}
We repeat description-subset sampling 10 times and report mean and standard deviation of accuracy and macro-F1. The multi-description setting follows the original Gen-Z conditional-likelihood scoring and description-aggregation procedure, using GPT-5.6 Sol-generated label descriptions for our four evaluation datasets. As shown in above, the multi-description setting improves accuracy over the single-definition setting across all eight reader--dataset combinations.
Q2: Shared knowledge and component-level comparisons.
We clarify that all methods in the original submission that use label definitions employ the same GPT-5.6 Sol-generated definitions; the definition-augmented zero-shot baseline, 1-shot-per-label prompting, Retrieved ICL, SLKR-Desc, and SLKR therefore have access to the same flagship-generated label definitions.
Table: Comparison under shared GPT-5.6 Sol-generated label definitions (accuracy / macro-F1, %). All three methods use the same definition knowledge and reader within each model group.