acceptodds
Under review as a conference paper at ICLR 2027

Class Semantics Are Residual Supervision: Breadth First, Depth on Demand

Abstract

Many language-model applications repeatedly make decisions over a fixed output space rather than generate open-ended text, such as intent classification, request routing, and relation classification. These systems often carry an explicit semantic specification of that decision space alongside each input, including class names, definitions, and rules that distinguish neighboring labels. This raises a basic question: how much of that specification must remain explicit once the model has learned part of the decision space? We show that class semantics behave as residual supervision: their value is determined by the boundaries that the model and labeled examples have not yet resolved. As supervision and capability resolve more of these boundaries, the value of explicit semantics decays, and what remains concentrates on the still-unresolved boundaries. On CLINC150, the gain of full descriptions over class names falls from about +2.5 Macro-F1 points with five examples per class to zero with one hundred. Catalog design then becomes a semantic-allocation problem: how much depth each class should carry, and where the remaining depth should be paid for. Per-class benefit rankings change across training seeds, which makes depth-first allocation unreliable at small budgets. This instability motivates a more reliable allocation order in the regimes we study: breadth first, depth on demand. Keep a 2–4 word gloss for every class, then restore full descriptions only at unresolved boundaries. The two-pass verified compact catalog uses 40% of the full-catalog tokens and recovers 84% of its gain; the hybrid uses 58% of the tokens and bounds the loss to the full catalog at 0.4 Macro-F1 points (95% one-sided, ten seeds). Token-matched truncation and retrieval fall back to names-only performance. The remaining burden need not stay in the prompt: continuing training under the deployment prompt moves most of the catalog dependence into the model weights, recovering 97% of the accuracy lost when the catalog is removed, while a tenth of that budget recovers 11%. Catalog dependence also carries a seven-point format component that does not track the description gain. Explicit semantics are a budgeted resource: measure what remains, cover the decision space before adding depth, and move stable semantics into the weights when retraining is available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
class descriptions, semantic priors, supervised classification, large language models, prompt compression, semantic budgeting, intent classification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_soAzOpenReview ↗
Rating
1
Confidence
4
Informativeness
1(−1)
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper investigates the reliance of large language models on semantic category descriptions in classification tasks. The authors find that as training data and model scale increase, the performance gain from complete category descriptions gradually diminishes, and these gains are primarily concentrated in certain hard-to-distinguish categories. Based on this finding, the authors propose a semantic allocation strategy: first retaining brief semantic descriptions for all categories, then supplementing more detailed descriptions only for the hard-to-distinguish categories, thereby reducing token overhead during inference. In addition, through additional model training, the authors transfer some category semantic information that originally depended on prompts into the model parameters, further lowering inference costs. The authors mainly validate the aforementioned findings and the effectiveness of the method on datasets such as CLINC150, BANKING77, and Wiki80.

Critical strengths
  1. This paper considers the practical deployment cost of the method.
  2. The paper is well written and easy to understand.
Critical weaknesses

Weaknesses

  1. The abstract is overly lengthy and fails to emphasize the main contributions. Moreover, the central experiments rely heavily on CLINC150, a relatively straightforward intent classification benchmark, which limits the representativeness of the empirical findings.

  2. The generalizability of the proposed approach remains questionable. Although the authors provide additional experiments on BANKING77 and Wiki80, the main conclusions are largely drawn from intent classification. It remains unclear whether compact class descriptions can effectively preserve discriminative semantics in more challenging tasks, such as GoEmotions, where fine-grained labels exhibit substantial semantic overlap. Evaluations across more diverse classification tasks and larger label spaces are necessary.

  3. The necessity of additional training and the effectiveness of the method on larger LLMs are insufficiently justified. Stronger models with in-context learning may already achieve competitive performance on these relatively simple benchmarks without task-specific fine-tuning. More importantly, the authors' own results show that the benefits of explicit class descriptions diminish as model size increases. Experiments with 8B or larger models, together with strong ICL baselines, are needed to establish whether the proposed approach remains useful at scale.

  4. The practical significance and methodological novelty of the proposed approach are not sufficiently demonstrated. The main strategy of preserving short descriptions for all classes while allocating additional details to difficult classes appears relatively intuitive. Although the paper includes token-matched truncation and retrieval baselines, its advantages over stronger long-context methods, adaptive label retrieval, prompt compression, and context distillation remain unclear. A more comprehensive comparison is necessary to clarify what genuinely new insights or practical benefits this work provides.

  5. The interpretation of class semantics as residual supervision requires stronger justification. The authors operationalize residual supervision through the performance difference between full descriptions and class names. However, this difference may also reflect prompt formatting, optimization dynamics, or training–inference dependencies rather than unresolved semantic boundaries alone. The observed format dependence further raises questions about whether the proposed measurement accurately captures the claimed phenomenon.

Questions for authors

No

3 comments
AuthorsOpenReview ↗

Response to Reviewer soAz (Part 1/2: Generalization and Scaling)

We thank the reviewer for recognizing the paper's deployment-cost motivation. We address W1–W3 here and W4–W5 in Part 2. Our contribution is not merely shortening class descriptions: we measure when fixed-label models still benefit from explicit class semantics, identify how that demand varies across classes and supervision regimes, and allocate semantic information between inference prompts and model weights.

W1–W2. Presentation, task diversity, and generalization

The newly completed Icecat-370 study extends our main result beyond intent classification to a larger product taxonomy. We evaluated 370 classes (2.5× CLINC150), Qwen3-1.7B, five training examples per class, and three seeds:

CatalogMacro-F1Tokens
Names only82.50%1,766
All-class compact glosses83.00%3,464
Hybrid82.97%6,012
Full descriptions83.16%10,377

Compact glosses retain 75% of the full-description improvement using only 33% of its prompt tokens. Full descriptions improve over names by +0.66 F1 points (95% CI [+0.28, +1.07]). Crucially, the gain remains highly nonuniform: +2.57 points in the lowest-recall class quartile versus +0.08 in the highest-recall quartile (difference +2.49, 95% CI [+1.24, +3.76]).

Together with BANKING77, Wiki80 (relation classification with Wikidata descriptions), and our Gemma-3 replication, the paper now examines 77–370 labels across intent, relation, and product classification, rather than relying exclusively on CLINC150. These are distinct single-label settings; fine-grained multi-label emotion classification such as GoEmotions remains untested.

We will also replace the lengthy abstract with a concise, single-paragraph summary centered on residual semantic demand, breadth-first semantic budgeting, and deployment-prompt adaptation.

W3. Larger models, in-context learning, and the need for training

The requested 8B and in-context-learning experiments directly support the paper's central prediction: explicit class descriptions become less valuable when the model already receives sufficient task supervision.

We evaluated the unfine-tuned Qwen3-8B under zero-shot prompting and 750 in-context demonstrations:

RegimeNamesCompactFull
Zero-shot Macro-F180.90%83.45%83.69%
750-shot Macro-F196.05%95.81%95.68%

In the zero-shot regime, full definitions add +2.79 F1 points over names; compact glosses recover 91.1% of that gain at 40% of the catalog tokens. The compact-minus-names improvement is +2.54 points (95% CI [+1.54, +3.59]).

With 750 demonstrations, full definitions offer no reliably detectable additional benefit (full minus names: −0.37 points, 95% CI [−1.04, +0.29]), despite increasing the prompt from 16,563 to 19,192 tokens relative to names-only. This directly illustrates the conditional residual-demand phenomenon: once sufficient class information is supplied by another source, repeatedly adding full definitions has little measured value. We observe the same qualitative pattern in the 1.7B prompt-only comparison, where the full-description advantage under demonstrations is also not reliably detected. This matters scientifically: the value of semantic descriptions must be evaluated relative to existing knowledge and task context, not treated as an intrinsic property of label text.

We do not propose fine-tuning as mandatory. Our framework helps determine when additional descriptions or adaptation are worth their cost—and when neither is needed. In the 8B few-shot setting, names-only prompting uses approximately 16,563 tokens per request, compared with 533 catalog tokens for names-only inference in our trained 1.7B setting. These systems have different accuracy and model sizes, so this is a deployment trade-off rather than a direct performance-equivalent comparison.

For repeated fixed-label decisions, our deployment-prompt adaptation offers another operating point. Rather than relying on descriptions at every request, it explicitly trains the model for the intended shorter prompt: 91.22% Macro-F1 at 533 catalog tokens, outperforming direct names-only training (90.52%) by +0.70 points (95% CI [+0.29, +1.10]). It places useful semantic specification into the model during training instead of repeatedly supplying the full catalog.

In short, the 8B evidence strengthens rather than weakens the paper's contribution: semantic demand is regime-dependent, and an effective deployment policy should pay for explicit class information only when it provides measurable value.

AuthorsOpenReview ↗

Response to Reviewer soAz (Part 2/2: Novelty, Baselines, and Semantic Evidence)

This is Part 2 of our response, addressing W4–W5.

W4. Novelty, stronger baselines, and deployment relevance

The contribution is a measurement-driven framework: quantify remaining semantic demand, identify useful class distinctions, and decide whether semantics belong in inference prompts or model weights.

Unlike compression or retrieval, we ask what information is still worth repeatedly supplying after a model has learned part of the label space. This motivates discriminative coverage before extra depth.

We strengthened the comparisons on CLINC150 (Qwen3-1.7B, five training examples/class):

MethodMacro-F1Tokens
Names only90.52%533
Per-input label retrieval90.50%1,219
Four-word truncation90.38%1,284
Our compact glosses91.42%1,256
Our hybrid91.61%1,826
Full descriptions91.59%3,162

Compact coverage retains 84% of the full-description improvement using only 40% of the catalog tokens, outperforming naive truncation and per-input retrieval at comparable costs. Truncation falling below names-only highlights the importance of preserved distinctions.

We additionally evaluated the official LLMLingua 0.2.2 compressor. In its matched-cost evaluation, it obtains 90.64% F1 at 1,263 tokens, versus 91.47% at 1,256 tokens for our compact glosses. The mean favors ours, but the paired interval crosses zero.

Our adaptation stage also differs from teacher-distribution distillation: it uses gold-label supervision under the intended short deployment prompt. In an equal-training-budget control, continuing optimization under the rich prompt yields only 82.21% F1 when subsequently evaluated with names only, versus 91.22% after names-prompt adaptation. This supports target-prompt alignment over merely adding training steps.

The practical outcome is an accuracy–cost choice: compact prompts preserve much of the full-catalog gain without its recurring length, while adapted names-only deployment improves over direct names-only training by +0.70 F1 points (95% CI [+0.29, +1.10]) using a 533-token catalog.

W5. Does the full-versus-names gap measure residual semantic supervision?

We define residual semantic supervision operationally as the incremental predictive value of class definitions under a specified model, training budget, and inference protocol. Names-only and full-description models are each trained and evaluated with their corresponding catalog. Inference-only catalog removal is measured separately as catalog dependence.

We also added three complementary tests connecting observed gains to semantic content:

(1) Exact-length and structural controls. At 100 training examples/class, neutral replacements matched the full catalog's rendered length for all 4,500 evaluation inputs. Full descriptions still outperform these replacements by +7.49 F1 points (95% CI [+4.44, +10.55]). Restoring repeated class-name structure recovers only 1.22 points of the removal loss. Length and the tested structural factors are insufficient to explain this dependence.

(2) Semantic-content interventions. Ambiguous and conflicting definitions cause targeted per-class F1 declines of 12.3 and 13.7 points, respectively. On Wiki80, rewriting definitions to sharpen class boundaries improves F1 by 2.1 points over the official descriptions (20 examples/class, three seeds). Behavior thus depends on what descriptions say.

(3) Boundary-level evidence. Across 12/12 tested conditions, measured class-boundary repair is positively correlated with class-level description gains (Spearman ρ = 0.33–0.62). This links gains to unresolved class distinctions.

These results support a conditional empirical account, without fully separating semantics from prompt familiarity or optimization. We will remove the overly causal “pure format effect” interpretation of the regression intercept in Section 6.2 and Figure 4.

Importantly, the deployment intervention works independently of that decomposition: removing full definitions from a rich-prompt-trained model lowers F1 from 91.59% to 80.20%; continued training under the names-only deployment prompt restores 91.22%, or approximately 97% of the removal loss.

Conclusion

Our contribution is to connect three questions that conventional compression alone does not answer: when explicit class semantics still help, where residual benefits concentrate, and how to reduce the cost of repeatedly providing them. The new 370-class, 8B, compression, and semantic-intervention experiments strengthen this account. We will clarify its scope in the revision.

Thank you for the response. However, my main concerns remain unresolved: the authors do not evaluate generalization across a sufficiently diverse range of tasks or compare against stronger, more recent baselines. Therefore, I will maintain my score and increase my confidence in this assessment.

Reviewer_9onaOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how much explicit class semantics a language model still needs for classification over a fixed label space after some decision boundaries have already been learned from pretraining and labeled examples. It measures the marginal benefit of full class descriptions relative to class names as data and model capacity increase, studies which classes retain that benefit, and examines the stability of selecting classes for additional semantic detail. These analyses motivate a breadth-first allocation policy that supplies a short gloss to every class before restoring full descriptions to classes with unresolved boundaries. The paper also investigates moving the remaining semantic specification from inference prompts into model weights through a second adaptation stage. On CLINC150 with Qwen3-1.7B and five examples per class, the compact catalog retains 84% of the full-description gain at 40% of its prompt tokens, while the hybrid uses 58% of the tokens and remains close to the full-catalog accuracy. Additional datasets, model families, and controlled comparisons characterize the regimes in which these results hold.

Critical strengths

The paper develops a coherent empirical argument rather than presenting prompt compression as an isolated trick. Sections 4 and 5 connect the reduction of description gains with increasing supervision, the concentration of residual gains at unresolved class boundaries, the instability of class-level gain rankings, and the resulting preference for broad semantic coverage. This distinction between boundary difficulty and the actual benefit of supplying a description is particularly informative for designing fixed-label classification systems.

The empirical comparisons address several plausible alternative explanations. The paper evaluates multiple model scales and data budgets, includes Banking77 and Wiki80, tests token-matched truncation and retrieval alternatives, and reports paired analyses across seeds. In its principal CLINC150 setting, the compact and hybrid catalogs substantially reduce repeated prompt cost while retaining much of the full-description benefit. The second-stage adaptation experiments further distinguish the value of semantics stored in prompts from the value that can be absorbed into model weights.

Critical weaknesses
  • The headline semantic-budget frontier depends materially on how the short glosses are constructed, not simply on their length or coverage. The main compact catalog uses a two-pass generation and verification procedure with a two-to-four-word constraint. Appendix S is a useful robustness check, but it also shows that independently generated catalogs recover only 36% and 48% of the full-description gain, compared with 84% for the verified catalog; a more constrained alternative recovers 70%. These differences are substantial for a paper whose practical recommendation is to distribute inexpensive semantic glosses broadly. The results support the utility of carefully distilled glosses, but do not yet establish that a fixed short-gloss budget is robust to the specific semantic cues retained. The authors should clarify whether the primary contribution is the breadth-first allocation rule, the particular verification recipe, or their combination, and how sensitive the performance–cost frontier is to varying gloss length and content under the same construction protocol.
  • The evidence for the 'depth on demand' targeting rule is less stable than the evidence for breadth-first coverage. Section 4 distinguishes an unresolved boundary from a class whose full description is actually helpful. Nevertheless, the hybrid policy selects the 38 least-resolved classes using recall from the names-only model. In the main catalog this beats a random 38-class allocation by 0.55 Macro-F1 points, but Appendix S reports only a 0.19-point advantage with the independently generated boundary-aware catalog, with a confidence interval that includes zero. The appendix also shows dataset sensitivity in the transfer of that gloss-generation recipe to Banking77. These results do not negate the compact-catalog finding, but they weaken the stronger implication that recall-based targeting reliably identifies where extra semantic depth should be spent. A matched-budget comparison of difficulty, confusion, and directly measured description response within the same hybrid policy would help determine whether the targeting advantage is a general property of unresolved boundaries or an interaction with the chosen gloss catalog.
Questions for authors
  • In light of Appendix S, what properties of the two-pass verified catalog explain its higher captured gain relative to the independent gloss catalogs? Does the breadth-first preference persist when the allowed gloss length is varied while the verification method is held fixed?
  • Can the authors clarify why validation recall is the preferred signal for assigning additional depth, given the paper's distinction between boundary difficulty and description usefulness? How would the hybrid frontier change under an alternative selector evaluated with the same base catalog and token budget?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9ona (Part 1/2: Semantic Coverage and Gloss Quality)

We thank the reviewer for recognizing the empirical motivation and coherent reasoning behind residual semantic supervision and breadth-first allocation. We conducted additional experiments to address both concerns. This first comment focuses on gloss construction and breadth-first coverage; Part 2 addresses depth selection.

Our central contribution is a semantic budgeting framework: measure residual semantic demand, provide discriminative coverage across all classes, and allocate additional depth only when justified. Gloss quality is an important component of making this allocation effective, rather than an interchangeable property of arbitrary short descriptions.

Q1. Does breadth-first coverage depend on the gloss construction recipe and length?

Our new experiments show that discriminative content is essential, while compact all-class coverage remains a competitive low-cost operating point across different construction settings.

1. Controlled gloss-length comparison.

Following the reviewer's suggestion, we constructed exact 2-, 4-, and 8-word catalogs using the same generator (Qwen3-8B), structured generation protocol, and independent verifier (Gemma-4-31B).

We compared all-class glosses against cost-matched selective full descriptions on CLINC150 (Qwen3-1.7B, five training examples per class, three seeds):

LengthAll-class Macro-F1TokensSelective-full Macro-F1
2 words90.80%98790.88%
4 words91.41%1,29590.96%
8 words91.72%1,89691.73%

All-class performance increases with gloss length. At four words, broad coverage achieves a +0.45-point higher mean F1 than selective full descriptions; at two and eight words, performance is similar.

The paired confidence intervals include zero at all three lengths. Accordingly, we claim an effective accuracy–cost trade-off, not universal superiority over selective depth at every budget.

2. Why does the verified catalog perform better?

We compared four independently constructed catalogs:

ConstructionCaptured full-description gain
Independent Gemma generation48%
Independent Qwen generation36%
Boundary-aware generation70%
Two-pass verified catalog84%

Notably, the boundary-aware catalog retains 70% of the full-description gain without the additional model-based verification stage. This supports the usefulness of discriminative content beyond one particular recipe.

At comparable token costs, naive four-word truncation achieves only 90.38% Macro-F1, whereas our compact glosses achieve 91.42%. Thus, what semantic information is preserved matters more than simply shortening the descriptions.

3. Independent human evaluation.

We additionally conducted a blinded human assessment of 240 glosses (60 stratified CLINC150 classes × four frozen catalogs). Two independent annotators evaluated semantic fidelity, scope preservation, and discrimination against lexically neighboring classes, without access to method identities, downstream results, or automated judgments.

ConstructionFidelityBoundaryBoth pass
Gemma88.3%70.0%65.0%
Qwen85.0%63.3%58.3%
Boundary-aware93.3%81.7%78.3%
Two-pass verified95.0%88.3%85.0%

Our verified catalog achieves the highest human-assessed quality across all three measures. Pre-adjudication Cohen's kappa values are 0.72 for fidelity, 0.66 for scope, and 0.65 for boundary preservation.

Importantly, the human quality ranking exactly matches the downstream captured-gain ranking. The primary errors involve semantic overgeneralization, omission of discriminative operations, and reversal of intent.

The human study covers a stratified 60-class subset, and the quality–performance relationship is correlational rather than a causal decomposition of the verification procedure.

Taken together, these experiments clarify our contribution: breadth-first allocation works by distributing concise, discriminative semantics across the decision space. The verified recipe is one effective realization of this principle, while the independent boundary-aware construction demonstrates that the underlying idea is not restricted to a single generation pipeline.

AuthorsOpenReview ↗

Response to Reviewer 9ona (Part 2/2: Depth Selection and Matched-Budget Evaluation)

This is Part 2 of our response, addressing depth selection and matched-budget evaluation.

Q2. Why validation recall, and how does it compare with alternative selectors?

Validation recall is a low-cost, reproducible indicator of unresolved class difficulty. Our new experiments show that broad semantic coverage provides the main accuracy–cost benefit, while selective depth is an optional refinement whose effectiveness depends on the task and budget.

1. Why recall-based selection?

We distinguish boundary difficulty (which classes the model struggles to recognize) from description usefulness (which classes actually benefit from full definitions).

We selected validation recall because it is directly available from names-only predictions, without additional full-description training or per-class intervention estimates. Our earlier analysis also found recall-based rankings more reproducible across seeds than directly measured description gains.

Thus, recall is a practical, inexpensive heuristic—not an assumption that difficulty perfectly predicts semantic benefit.

2. Seven-seed, token-matched comparison.

Following the reviewer's suggestion, we fixed the compact base catalog and compared four selectors under approximately identical prompt-token budgets (four selectors × seven seeds = 28 trained runs):

SelectorMacro-F1TokensExpanded classes
Low recall91.53%1,82638
High confusion91.41%1,825.534–42
Measured description gain91.63%1,825.137–47
Random91.41%1,825.742–49

Measured description gain achieves the highest mean F1, while recall reaches 91.53%, compared with 91.41% for random allocation.

The recall–random difference is +0.117 points (95% CI [−0.403, +0.636]); therefore, we do not claim a statistically established advantage under matched budgets.

The original comparison fixed 38 expanded classes, yielding 91.61% for recall versus 91.06% for random, but consuming 1,826 versus 1,734 tokens. The new comparison instead matches inference token cost, allowing the number of expanded classes to vary with description length.

These are complementary controls: one holds class count fixed; the other evaluates allocation under equal deployment cost.

3. Implications for breadth-first allocation.

The matched-budget experiments reveal an important practical finding: adding more full descriptions does not necessarily improve classification after compact semantic coverage is established.

StrategyMacro-F1Tokens
All-class compact glosses91.42%1,256
Compact + random depth91.41%1,826
Compact + recall depth91.53%1,826
Full descriptions91.59%3,162

Random allocation spends approximately 570 additional tokens and expands 42–49 classes, yet achieves essentially the same mean F1 as compact coverage alone.

Meanwhile, recall-based depth remains close to full-description accuracy at approximately 58% of the token cost. An independently constructed boundary-aware catalog also achieves a competitive hybrid operating point (91.66% Macro-F1 at approximately 1,737 tokens), supporting the applicability of the framework beyond one gloss recipe.

These results clarify the respective contributions:

First, breadth-first coverage establishes the principal accuracy–cost benefit. On CLINC150, compact glosses preserve 84% of the full-description improvement using only 40% of its prompt tokens.

Second, additional depth is a budget-dependent refinement. Recall offers a simple selection signal, but no tested selector has a reliably established universal advantage over alternatives.

We will clarify these distinctions and include the matched-budget comparisons in the revised manuscript.

Conclusion

The new evidence reinforces our central contribution: characterizing nonuniform residual semantic demand and using it to allocate limited semantic information efficiently.

The key principle is to prioritize concise, discriminative coverage across all classes, then introduce additional depth when justified by the expected benefit and deployment cost.

We thank the reviewer for helping us clarify this distinction between the robust value of breadth coverage and the conditional benefit of depth selection.

Thank you for the additional experiments. They helped me understand why preserving the distinctions between similar classes matters when shortening descriptions. I find the evidence for compact descriptions across all classes persuasive, though the benefit of using low recall to decide which descriptions to expand remains less clear.

Given my limited direct experience with this topic, I gave particular weight to Reviewer soAz’s concern about comparisons with existing prompt-compression methods. The added LLMLingua comparison partly addresses this concern. At a matched token budget, the mean favors the proposed approach, but the paired confidence interval includes zero. On balance, I maintain my recommendation of weak acceptance.

Reviewer_qzpvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how much explicit class-level semantic information a model still needs when making decisions over a fixed label space. It argues that class descriptions act as residual supervision: their benefit decreases as labeled data and model capability resolve more class boundaries, and the remaining benefit concentrates on difficult-to-separate classes. The authors propose a breadth-first allocation strategy that gives every class a 2–4 word gloss and restores full descriptions only for classes with unresolved validation boundaries. A second training stage under the deployment prompt further transfers much of the catalog dependence into model weights. The experiments on CLINC150, Banking77, and Wiki80, and evaluates Qwen3 and Gemma-3 models across multiple data and model-capability regimes validate the effectiveness of the approach.

Critical strengths
  1. Rather than treating label descriptions as uniformly useful prompt content, the paper asks how their marginal value changes as a model learns the decision space. The residual-supervision framing gives a useful explanation for why the same catalog can help in low-data or low-capability regimes but become unnecessary in more resolved regimes.

  2. The paper varies both the number of labeled examples and model scale. For example, on CLINC150 with Qwen3-0.6B, the full-description gain over names falls from +2.52 Macro-F1 points at five examples per class to approximately zero at 100 examples. It also shows that the gain is concentrated on less-resolved classes: the least-resolved quartile gains +5.97 points, compared with +0.42 for the most-resolved quartile. The analyses of pairwise confusion and class-level gains make the central claim more informative than a simple aggregate comparison.

  3. The breadth-first strategy is not justified only by intuition. The paper compares compact glosses, selective full descriptions, random allocation, truncation, and per-input retrieval. On CLINC150 at 1.7B, the verified gloss catalog retains 84% of the full-catalog gain at 40% of the tokens, and the hybrid catalog is within 0.4 Macro-F1 points of the full catalog at 58% of its tokens. These controls help separate coverage, semantic content, and targeting.

  4. The second-stage adaptation experiment asks whether the remaining semantic burden must be carried in every prompt. The standard-budget adaptation recovers most of the accuracy lost when the catalog is removed, while the gentle recipe recovers much less. The paper also identifies a format-related component that persists even when the description gain is near zero, which is an interesting observation even though its cause remains unresolved.

Critical weaknesses
  1. The main evidence covers full-parameter supervised fine-tuning with 77–150 labels and 0.6B–4B backbones. The authors explicitly note that larger taxonomies, larger models, and production serving remain unmeasured. The results are persuasive within the tested setting, but it is not yet clear whether the same allocation rule holds for decision spaces with hundreds or thousands of labels, hierarchical taxonomies, or models that are not fine-tuned with the catalog present.

  2. The paper reports that the two-pass AI-verified catalog recovers 84% of the full-catalog gain, whereas independent gloss-generation recipes recover only 36–70%. This suggests that the quality of the generated glosses is a major determinant of the result. The paper would be stronger with a clearer assessment of verification reliability, such as human or expert auditing, inter-verifier agreement, and robustness to different generation and verification models.

  3. The proposed strategy avoids a fragile direct ranking of per-class description benefit, but its depth-on-demand step selects classes using names-only validation recall. The paper acknowledges that this signal is estimated from a finite validation split and that the selected classes need to represent the boundaries encountered at deployment. It would be useful to quantify how the hybrid behaves with smaller validation sets, shifted validation distributions, or domains where a representative development set is unavailable.

  4. The seven-point format component remains unexplained and complicates the interpretation of semantic dependence. Removing the catalog from a rich-trained checkpoint incurs a substantial loss even in regimes where descriptions did not improve matched training accuracy. The authors interpret this as a format effect and show that adaptation removes it, but its mechanism is not established. It is therefore difficult to determine how much of the observed prompt dependence reflects semantic information, prompt-format familiarity, or other distributional effects introduced during training.

Questions for authors
  1. Could the authors provide evidence that the breadth-first allocation rule remains effective when the label inventory is substantially larger than 150 classes, or explain which parts of the method are expected to change in that regime?
  2. Given the reported gap between the two-pass verified catalog and independent gloss-generation recipes, how reliable is the verification stage? Have the authors evaluated gloss correctness with human or domain-expert judgments, or across different generator/verifier models?
  3. How sensitive is the hybrid policy to the size and distribution of the validation set used to identify unresolved classes? Does its advantage persist when the validation set is small or does not reflect the deployment distribution?
  4. Could the authors further isolate the source of the approximately seven-point format component that remains even when the full-description gain is statistically zero? For example, does it arise from prompt structure, exposure during fine-tuning, or another training-distribution effect?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer qzpv Part 1/2

We thank the reviewer for recognizing the importance of residual semantic supervision and semantic budget allocation. We conducted additional experiments addressing all four concerns. We organize our response into two comments: this comment addresses Q1–Q2, and Part 2 addresses Q3–Q4.

Q1. Generalization to larger label spaces

Our new 370-class evaluation supports the effectiveness of compact semantic coverage in larger decision spaces and reproduces the concentration of residual benefits among difficult classes.

We conducted a preregistered Icecat-370 evaluation (370 product categories, Qwen3-1.7B, five training examples per class, three seeds):

CatalogMacro-F1Tokens
Names only82.50%1,766
Compact glosses83.00%3,464
Hybrid82.97%6,012
Full descriptions83.16%10,377

Compact glosses retain 75% of the full-description improvement using only 33% of its prompt tokens. This extends the favorable accuracy–cost trade-off observed on CLINC150 to a substantially larger product classification task.

Residual benefits remain concentrated among difficult classes: full descriptions improve Macro-F1 by +2.57 points in the lowest-recall quartile, compared with only +0.08 points in the highest-recall quartile.

We further evaluated Qwen3-8B without task-specific fine-tuning. In zero-shot classification, compact glosses preserve 91.1% of the full-description gain at approximately 40% of the catalog tokens. With 750 in-context demonstrations, full descriptions no longer provide a reliable additional benefit.

Together, these findings reinforce our central insight: the value of explicit class semantics depends on the supervision already available to the model, while compact all-class coverage remains an effective low-cost configuration. The benefit of selective depth is task-dependent, and thousand-class hierarchical settings remain to be studied.

Q2. Reliability of gloss generation and verification

Our additional human evaluation and independent construction experiments support the importance of preserving semantic fidelity and discriminative class boundaries, rather than merely shortening descriptions.

Independent human assessment. We conducted a blinded assessment of 240 glosses (60 stratified CLINC150 classes × four frozen catalogs). Two independent annotators evaluated semantic fidelity, scope preservation, and boundary discrimination against lexical neighbors. They were blinded to method identities, downstream performance, automated judgments, and each other's ratings.

ConstructionFidelityBoundaryBoth pass
Gemma generation88.3%70.0%65.0%
Qwen generation85.0%63.3%58.3%
Boundary-aware generation93.3%81.7%78.3%
Two-pass verified95.0%88.3%85.0%

Our two-pass catalog achieves the highest human-assessed semantic fidelity, boundary preservation, and joint acceptance rate.

Pre-adjudication inter-annotator agreement is also substantial: Cohen's κ = 0.72 for fidelity, 0.66 for scope, and 0.65 for boundary preservation.

Construction robustness. The four catalogs capture 48%, 36%, 70%, and 84% of the full-description improvement, respectively. Their downstream performance ordering matches the human quality ordering. Moreover, naive four-word truncation achieves only 90.38% Macro-F1, compared with 91.42% for our compact glosses at similar token cost.

Independent automated verification. Qwen3-8B and Gemma-4-31B independently assessed all 600 glosses. Their inter-verifier κ values range from 0.064 to 0.131, substantially below human inter-annotator agreement. This highlights the value of independent human evaluation rather than treating automated verification as semantic ground truth.

The human study covers a stratified 60-class subset, and the observed quality–performance relationship is correlational rather than proof of the second verification pass's causal contribution.

These results strengthen the empirical basis of our method: effective breadth-first allocation relies on concise descriptions that preserve meaningful class distinctions.

AuthorsOpenReview ↗

Author Response to Reviewer qzpv Part 2/2

This is Part 2 of our response to Reviewer qzpv, addressing Q3–Q4.

Q3. Sensitivity to validation-set size and distribution

Our new experiments show that smaller validation subsets change which classes are selected for additional depth, but we do not detect a reliable downstream performance degradation in the tested settings. Importantly, compact all-class coverage does not depend on validation-based class ranking.

We varied the number of validation examples per class used for recall-based selection:

Examples per classSelection Jaccard
20.338
50.501
100.613
201.000 (reference)

We then retrained the hybrid models using five and ten selection examples per class.

Relative to selection using the full validation set, Macro-F1 changes by −0.336 points (95% CI [−2.041, +1.368]) and −0.310 points (95% CI [−1.285, +0.665]), respectively.

Although class selection is sensitive to validation-set size, these changes do not produce a statistically reliable downstream performance loss in our experiments. The confidence intervals remain too wide to establish equivalence.

Crucially, validation-based ranking is required only for the optional depth allocation stage, not for the underlying all-class compact catalog. Thus, when representative validation evidence is limited, the compact catalog remains a practical low-cost configuration without requiring potentially noisy class selection.

These experiments vary the data available for class ranking, while checkpoint selection retains the full validation set. Substantial validation-to-deployment distribution shift remains outside our current evaluation.

Q4. Interpreting the approximately seven-point catalog-removal loss

Our new controls demonstrate that prompt length and superficial structure are insufficient to explain catalog dependence. We therefore refine the original interpretation while preserving the practical contribution of deployment-prompt adaptation.

We conducted additional interventions on CLINC150 with 100 training examples per class, where full descriptions provide negligible improvement under matched training and evaluation conditions.

Repeated class-name structure recovers only 1.22 Macro-F1 points after catalog removal.

More importantly, we constructed neutral replacements whose rendered prompt lengths exactly match the original full catalog across all 4,500 evaluation inputs.

Despite exact length matching, full descriptions outperform neutral replacements by 7.49 Macro-F1 points (95% CI [4.44, 10.55]).

This shows that prompt length and the tested structural factors cannot sufficiently explain the observed dependence. However, semantic content and familiarity with the training prompt remain intertwined.

Accordingly, we will revise Section 6.2 and Figure 4 to describe this phenomenon as catalog-conditioning dependence, rather than attributing the regression intercept to a pure format effect.

Importantly, the practical benefit of our adaptation method remains firmly supported.

On CLINC150 with Qwen3-1.7B, directly removing the full catalog reduces Macro-F1 from 91.59% to 80.20%. Continuing training under the target names-only deployment prompt restores performance to 91.22%, recovering approximately 97% of the catalog-removal loss.

Compared with names-only training from the beginning (90.52%), deployment-prompt adaptation achieves an additional +0.70 points (paired seven-seed 95% CI [+0.29, +1.10]).

Thus, even without fully identifying every mechanism underlying catalog dependence, we establish an effective intervention that substantially reduces repeated semantic input costs while preserving classification quality.

Overall conclusion

The additional experiments reinforce three central contributions:

(1) Residual class semantics are conditional and nonuniform.

(2) Discriminative breadth-first coverage preserves substantial classification utility at lower recurring prompt cost, with additional support from independent human evaluation.

(3) Deployment-prompt adaptation substantially reduces dependence on repeated full descriptions.

We thank the reviewer again for the constructive feedback and will incorporate these findings into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.