- Rating
- 1
- Confidence
- 4
- Informativeness
- 1(−1)
- Presentation
- 2
- Soundness
- 1(−1)
Read review
Paper summary
This paper investigates the reliance of large language models on semantic category descriptions in classification tasks. The authors find that as training data and model scale increase, the performance gain from complete category descriptions gradually diminishes, and these gains are primarily concentrated in certain hard-to-distinguish categories. Based on this finding, the authors propose a semantic allocation strategy: first retaining brief semantic descriptions for all categories, then supplementing more detailed descriptions only for the hard-to-distinguish categories, thereby reducing token overhead during inference. In addition, through additional model training, the authors transfer some category semantic information that originally depended on prompts into the model parameters, further lowering inference costs. The authors mainly validate the aforementioned findings and the effectiveness of the method on datasets such as CLINC150, BANKING77, and Wiki80.
Critical strengths
- This paper considers the practical deployment cost of the method.
- The paper is well written and easy to understand.
Critical weaknesses
Weaknesses
-
The abstract is overly lengthy and fails to emphasize the main contributions. Moreover, the central experiments rely heavily on CLINC150, a relatively straightforward intent classification benchmark, which limits the representativeness of the empirical findings.
-
The generalizability of the proposed approach remains questionable. Although the authors provide additional experiments on BANKING77 and Wiki80, the main conclusions are largely drawn from intent classification. It remains unclear whether compact class descriptions can effectively preserve discriminative semantics in more challenging tasks, such as GoEmotions, where fine-grained labels exhibit substantial semantic overlap. Evaluations across more diverse classification tasks and larger label spaces are necessary.
-
The necessity of additional training and the effectiveness of the method on larger LLMs are insufficiently justified. Stronger models with in-context learning may already achieve competitive performance on these relatively simple benchmarks without task-specific fine-tuning. More importantly, the authors' own results show that the benefits of explicit class descriptions diminish as model size increases. Experiments with 8B or larger models, together with strong ICL baselines, are needed to establish whether the proposed approach remains useful at scale.
-
The practical significance and methodological novelty of the proposed approach are not sufficiently demonstrated. The main strategy of preserving short descriptions for all classes while allocating additional details to difficult classes appears relatively intuitive. Although the paper includes token-matched truncation and retrieval baselines, its advantages over stronger long-context methods, adaptive label retrieval, prompt compression, and context distillation remain unclear. A more comprehensive comparison is necessary to clarify what genuinely new insights or practical benefits this work provides.
-
The interpretation of class semantics as residual supervision requires stronger justification. The authors operationalize residual supervision through the performance difference between full descriptions and class names. However, this difference may also reflect prompt formatting, optimization dynamics, or training–inference dependencies rather than unresolved semantic boundaries alone. The observed format dependence further raises questions about whether the proposed measurement accurately captures the claimed phenomenon.
Questions for authors
No
3 comments
Response to Reviewer soAz (Part 2/2: Novelty, Baselines, and Semantic Evidence)
This is Part 2 of our response, addressing W4–W5.
W4. Novelty, stronger baselines, and deployment relevance
The contribution is a measurement-driven framework: quantify remaining semantic demand, identify useful class distinctions, and decide whether semantics belong in inference prompts or model weights.
Unlike compression or retrieval, we ask what information is still worth repeatedly supplying after a model has learned part of the label space. This motivates discriminative coverage before extra depth.
We strengthened the comparisons on CLINC150 (Qwen3-1.7B, five training examples/class):
| Method | Macro-F1 | Tokens |
|---|---|---|
| Names only | 90.52% | 533 |
| Per-input label retrieval | 90.50% | 1,219 |
| Four-word truncation | 90.38% | 1,284 |
| Our compact glosses | 91.42% | 1,256 |
| Our hybrid | 91.61% | 1,826 |
| Full descriptions | 91.59% | 3,162 |
Compact coverage retains 84% of the full-description improvement using only 40% of the catalog tokens, outperforming naive truncation and per-input retrieval at comparable costs. Truncation falling below names-only highlights the importance of preserved distinctions.
We additionally evaluated the official LLMLingua 0.2.2 compressor. In its matched-cost evaluation, it obtains 90.64% F1 at 1,263 tokens, versus 91.47% at 1,256 tokens for our compact glosses. The mean favors ours, but the paired interval crosses zero.
Our adaptation stage also differs from teacher-distribution distillation: it uses gold-label supervision under the intended short deployment prompt. In an equal-training-budget control, continuing optimization under the rich prompt yields only 82.21% F1 when subsequently evaluated with names only, versus 91.22% after names-prompt adaptation. This supports target-prompt alignment over merely adding training steps.
The practical outcome is an accuracy–cost choice: compact prompts preserve much of the full-catalog gain without its recurring length, while adapted names-only deployment improves over direct names-only training by +0.70 F1 points (95% CI [+0.29, +1.10]) using a 533-token catalog.
W5. Does the full-versus-names gap measure residual semantic supervision?
We define residual semantic supervision operationally as the incremental predictive value of class definitions under a specified model, training budget, and inference protocol. Names-only and full-description models are each trained and evaluated with their corresponding catalog. Inference-only catalog removal is measured separately as catalog dependence.
We also added three complementary tests connecting observed gains to semantic content:
(1) Exact-length and structural controls. At 100 training examples/class, neutral replacements matched the full catalog's rendered length for all 4,500 evaluation inputs. Full descriptions still outperform these replacements by +7.49 F1 points (95% CI [+4.44, +10.55]). Restoring repeated class-name structure recovers only 1.22 points of the removal loss. Length and the tested structural factors are insufficient to explain this dependence.
(2) Semantic-content interventions. Ambiguous and conflicting definitions cause targeted per-class F1 declines of 12.3 and 13.7 points, respectively. On Wiki80, rewriting definitions to sharpen class boundaries improves F1 by 2.1 points over the official descriptions (20 examples/class, three seeds). Behavior thus depends on what descriptions say.
(3) Boundary-level evidence. Across 12/12 tested conditions, measured class-boundary repair is positively correlated with class-level description gains (Spearman ρ = 0.33–0.62). This links gains to unresolved class distinctions.
These results support a conditional empirical account, without fully separating semantics from prompt familiarity or optimization. We will remove the overly causal “pure format effect” interpretation of the regression intercept in Section 6.2 and Figure 4.
Importantly, the deployment intervention works independently of that decomposition: removing full definitions from a rich-prompt-trained model lowers F1 from 91.59% to 80.20%; continued training under the names-only deployment prompt restores 91.22%, or approximately 97% of the removal loss.
Conclusion
Our contribution is to connect three questions that conventional compression alone does not answer: when explicit class semantics still help, where residual benefits concentrate, and how to reduce the cost of repeatedly providing them. The new 370-class, 8B, compression, and semantic-intervention experiments strengthen this account. We will clarify its scope in the revision.
Thank you for the response. However, my main concerns remain unresolved: the authors do not evaluate generalization across a sufficiently diverse range of tasks or compare against stronger, more recent baselines. Therefore, I will maintain my score and increase my confidence in this assessment.
Response to Reviewer soAz (Part 1/2: Generalization and Scaling)
We thank the reviewer for recognizing the paper's deployment-cost motivation. We address W1–W3 here and W4–W5 in Part 2. Our contribution is not merely shortening class descriptions: we measure when fixed-label models still benefit from explicit class semantics, identify how that demand varies across classes and supervision regimes, and allocate semantic information between inference prompts and model weights.
W1–W2. Presentation, task diversity, and generalization
The newly completed Icecat-370 study extends our main result beyond intent classification to a larger product taxonomy. We evaluated 370 classes (2.5× CLINC150), Qwen3-1.7B, five training examples per class, and three seeds:
Compact glosses retain 75% of the full-description improvement using only 33% of its prompt tokens. Full descriptions improve over names by +0.66 F1 points (95% CI [+0.28, +1.07]). Crucially, the gain remains highly nonuniform: +2.57 points in the lowest-recall class quartile versus +0.08 in the highest-recall quartile (difference +2.49, 95% CI [+1.24, +3.76]).
Together with BANKING77, Wiki80 (relation classification with Wikidata descriptions), and our Gemma-3 replication, the paper now examines 77–370 labels across intent, relation, and product classification, rather than relying exclusively on CLINC150. These are distinct single-label settings; fine-grained multi-label emotion classification such as GoEmotions remains untested.
We will also replace the lengthy abstract with a concise, single-paragraph summary centered on residual semantic demand, breadth-first semantic budgeting, and deployment-prompt adaptation.
W3. Larger models, in-context learning, and the need for training
The requested 8B and in-context-learning experiments directly support the paper's central prediction: explicit class descriptions become less valuable when the model already receives sufficient task supervision.
We evaluated the unfine-tuned Qwen3-8B under zero-shot prompting and 750 in-context demonstrations:
In the zero-shot regime, full definitions add +2.79 F1 points over names; compact glosses recover 91.1% of that gain at 40% of the catalog tokens. The compact-minus-names improvement is +2.54 points (95% CI [+1.54, +3.59]).
With 750 demonstrations, full definitions offer no reliably detectable additional benefit (full minus names: −0.37 points, 95% CI [−1.04, +0.29]), despite increasing the prompt from 16,563 to 19,192 tokens relative to names-only. This directly illustrates the conditional residual-demand phenomenon: once sufficient class information is supplied by another source, repeatedly adding full definitions has little measured value. We observe the same qualitative pattern in the 1.7B prompt-only comparison, where the full-description advantage under demonstrations is also not reliably detected. This matters scientifically: the value of semantic descriptions must be evaluated relative to existing knowledge and task context, not treated as an intrinsic property of label text.
We do not propose fine-tuning as mandatory. Our framework helps determine when additional descriptions or adaptation are worth their cost—and when neither is needed. In the 8B few-shot setting, names-only prompting uses approximately 16,563 tokens per request, compared with 533 catalog tokens for names-only inference in our trained 1.7B setting. These systems have different accuracy and model sizes, so this is a deployment trade-off rather than a direct performance-equivalent comparison.
For repeated fixed-label decisions, our deployment-prompt adaptation offers another operating point. Rather than relying on descriptions at every request, it explicitly trains the model for the intended shorter prompt: 91.22% Macro-F1 at 533 catalog tokens, outperforming direct names-only training (90.52%) by +0.70 points (95% CI [+0.29, +1.10]). It places useful semantic specification into the model during training instead of repeatedly supplying the full catalog.
In short, the 8B evidence strengthens rather than weakens the paper's contribution: semantic demand is regime-dependent, and an effective deployment policy should pay for explicit class information only when it provides measurable value.