acceptodds
Under review as a conference paper at ICLR 2027

Contrastive Label-Embedding Alignment: Adapting Dual Encoders for Text Classification from Label Descriptions

Abstract

Dual-encoder text embedding models provide efficient, reusable representations, but adapting them to a target classification task typically requires labeled documents to fit a classifier head. Alternatives that avoid labeled documents, such as NLI cross-encoders and prompted LLMs, process each document jointly with its candidate labels and therefore cannot cache document representations. We introduce __contrastive label-embedding alignment__ (CLEA), which reaches a point on this supervision-cost / inference-cost frontier that, to our knowledge, no prior method occupies: no labeled documents, no task-specific head, and fully pre-encodable dual-encoder inference. CLEA lightly fine-tunes a shared encoder using only label verbalizers and a small set of natural-language descriptions per label, aligning each verbalizer with its description set through a symmetric multi-positive contrastive objective. Across four benchmarks spanning topic, sentiment, intent, and emotion classification and ten encoders from 22M to 600M parameters, CLEA improves macro-F by __+0.09__ on average over untuned zero-shot embeddings, and on every benchmark the best CLEA-adapted encoder matches or exceeds NLI cross-encoders, a reranker, and prompted LLMs while retaining pre-encodable inference. Five descriptions per label suffice, and LLM-generated descriptions recover most of the gains of hand-written ones. In budget-matched experiments, CLEA outperforms linear probes until several dozen labeled examples per class are available.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
text classification, dual encoders, text embeddings, contrastive learning, dataless classification, label descriptions, low-resource learning, representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rzozOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

CLEA fine-tunes a shared text encoder using label verbalizers and a few descriptions per class, without labeled target-task documents. A bidirectional contrastive loss aligns labels with descriptions; inference uses document-label cosine similarity. Learning rates are selected by embedding uniformity on unlabeled target-domain texts. Across ten encoders and four benchmarks, the paper reports an average absolute macro-F1 gain of about 0.09 over untuned models. The main Banking77 evaluation covers six intents, with a separate experiment on all 77 classes.

Critical strengths
  1. CLEA offers a simple adaptation method without labeled target-task documents or an extra classifier head. Documents and labels remain independently encodable, which is useful for large-scale, repeated classification.
  2. Results across ten encoders suggest that the method generalizes across backbones. It also beats mean-description prototypes using the same descriptions in 18 of 20 combinations, supporting a benefit beyond richer label text alone.
Critical weaknesses

1.The contribution over prior work is not yet clear. Direct comparisons with Gao et al. (EMNLP 2023) and Hu et al. (NLP4ConvAI 2024) are missing. The column loss also shares the positive-sum-inside-log structure discussed in Khosla et al. (2020), Eq. (3), although the formulations differ. Its average gain over rows-only training is only about 0.01 macro-F1, with no uncertainty reported for that difference. 2.The accuracy-cost claim needs more evidence. CLEA receives multiple descriptions per class, while the NLI and LLM baselines use short verbalizers. Appendix O reports uniformity–F1 correlations, but not the gain over fixed learning rates. Reported costs do not clearly account for learning-rate search, and the full 77-class experiment compares only with untuned encoders.

Questions for authors
  1. What does the column loss gain over averaging positive log-probabilities? Please report its improvement over rows-only training and the uncertainty using paired runs.
  2. How do the NLI and LLM results change when these baselines receive the same class descriptions?
  3. Do the training times include learning-rate search? Please provide comparable costs, measured inference latency, and a strong description-based baseline on the full 77-class task.
  4. How does uniformity-based selection compare with a fixed learning rate and reuse of the pretrained model's learning rate? Please match encoders, descriptions, and random seeds, and report F1 differences with uncertainty to isolate the benefit of document-based selection.
5 comments
AuthorsOpenReview ↗

Response to Reviewer rzoz

Thank you for the detailed technical review and the opportunity to clarify our contribution.

We thank the reviewer for highlighting the related work connections:

  • Gao et al. (2023) establish the value of label-description training, but use a prompt-based masked language-model classification framework. CLEA instead adapts pretrained dual encoders, preserving independent, pre-encodable inference. This distinction is central to the supervision–inference trade-off motivating our work. We already cite Gao et al., but will make this distinction more explicit. Their Table 2 reports 77.4% and 79.4% accuracy on AGNews with RoBERTa-base and large, respectively. CLEA achieves 0.79 macro-F1 with 22M-parameter MiniLM and up to 0.85 with Qwen3-Embedding-0.6B. These results are indicative rather than directly comparable.
  • Hu et al. (2024) explore description-augmented embedding similarity without task-specific fine-tuning. Their approach is closely related to our mean-description prototype baseline (Appendix K), against which CLEA improves in 18 of 20 configurations. We will make this connection explicit.
  • Khosla et al. (2020) discuss the positive-sum-inside-log contrastive formulation. We will acknowledge this precedent and clarify that our contribution lies in its use within a complementary rowwise–columnwise objective for label-description alignment, rather than the aggregation principle itself.

Adding the columnwise loss yields a modest but systematic improvement (+0.01 macro-F1 on average), improving or matching rows-only performance in 37 of 40 model–dataset configurations (Appendix G, Table 7). More importantly, it provides two additional properties without sacrificing overall predictive performance:

  • Noise robustness (Appendix H): Under 25% description corruption, macro-F1 is 0.64 (rows-only), 0.71 (columns-only), and 0.67 (symmetric). At 50% corruption, the corresponding scores are 0.59, 0.64, and 0.61. The column term is more robust to moderately noisy descriptions, with the symmetric objective inheriting part of this benefit. Its positive-sum-inside-log formulation places greater gradient weight on strongly aligned, representative descriptions, downweighting vague ones.
  • Description imbalance (Appendix I): Per-label normalization reduces sensitivity to unequal description counts. Adding descriptions to one class largely preserves performance on other classes under the symmetric objective, whereas rows-only training exhibits cross-label interference.

These properties motivate our formulation relative to averaging positive log-probabilities. The rowwise term already emphasizes difficult descriptions, while the averaged-log alternative similarly encourages alignment with individual, potentially noisy positives. Our column term instead emphasizes strongly aligned positives, providing a complementary optimization signal rather than duplicating the rowwise term.

For description-matched baselines, we distinguish two complementary comparisons. Appendix K isolates description information from adaptation: the mean-description prototype baseline uses identical descriptions but classifies through averaged embeddings without fine-tuning. CLEA outperforms it in 18 of 20 configurations. Separately, our NLI, reranker, and LLM comparisons evaluate established approaches to classification without labeled documents under their standard inference setups. We agree that description-augmented variants would provide additional insight and are evaluating them using the same class descriptions. Importantly, these variants substantially increase inference costs for NLI and rerankers, requiring five document–description evaluations per label instead of one, whereas CLEA retains its pre-encodable inference profile. We will quantify this trade-off alongside performance.

Thank you for suggesting a comparison against the oracle-best learning rate, providing a useful measure of selection regret beyond the uniformity–F1 relationships reported in Appendix O. We are comparing uniformity-based selection against a fixed learning rate, the pretrained learning rate, and the oracle-best candidate under matched settings. We will report improvements over simpler alternatives, regret relative to the oracle, and a more precise breakdown of computational costs.

Appendix N evaluates CLEA on all 77 Banking77 classes, demonstrating improvements across four encoders. We agree that a stronger description-based comparison would be useful. Appendix K already implements a mean-description prototype baseline that embeds each description, averages embeddings into class prototypes, and classifies by similarity without fine-tuning. This provides a controlled comparison using identical descriptions, which we will extend to the full 77-class setting.

We will provide quantitative updates as these analyses are completed.

AuthorsOpenReview ↗

Follow-up to Reviewer rzoz (1/3): description-matched baselines

Thank you again for the detailed review. We have completed three of the additional analyses outlined in our initial response and report them in three comments.

1. Description-matched baselines (W2, Q2)

We extended all five baselines to receive the same five descriptions per label used to train CLEA. For NLI models and the reranker, we score each document against every description and average the resulting probabilities per label. For LLMs, each multiple-choice option contains the verbalizer followed by its five descriptions. The verbalizer-only results are from the original Table 5. All values are macro-F1.

MethodAGNewsBanking77EmotionDAIRRottenTom.
DeBERTa NLI, verbalizer0.810.860.440.85
+ descriptions0.110.360.120.75
ModernBERT NLI, verbalizer0.760.830.300.82
+ descriptions0.390.600.280.60
Qwen3-Reranker-0.6B, verbalizer0.790.870.410.78
+ descriptions0.850.810.490.76
Qwen3-4B, verbalizer0.820.910.430.87
+ descriptions0.790.870.430.85
Qwen3-8B, verbalizer0.850.910.400.86
+ descriptions0.800.910.410.76
Qwen3-Embedding-0.6B + CLEA0.850.920.570.88
Best CLEA encoder0.850.960.570.88

The effects of description augmentation differ considerably across architectures.

  • Reranker: Description augmentation substantially improves AGNews (+0.06) and EmotionDAIR (+0.08), but reduces performance on the other two datasets.
  • NLI: Both models deteriorate substantially. One possible explanation is that NLI models expect concise hypotheses expressing individual claims. Longer, multi-clause descriptions may be difficult for a single document to entail in full, reducing the discriminative value of entailment scores.
  • LLMs: Descriptions provide no consistent improvement. Qwen3-8B loses 0.05 on AGNews and 0.10 on RottenTomatoes. Further inspection suggests a shift toward particular labels rather than a simple prompt-length explanation. On RottenTomatoes, correct and incorrect predictions have similar average prompt lengths, but Qwen3-8B predicts "negative" for approximately 75% of reviews despite balanced classes, with approximately 98% of its classification errors consisting of positive reviews classified as negative. Qwen3-4B similarly overpredicts Business on AGNews.

The central conclusion from Table 5 remains supported. Allowing each baseline to use whichever input variant performs better, the strongest alternatives achieve 0.85 / 0.91 / 0.49 / 0.87 across the four datasets, compared with 0.85 / 0.96 / 0.57 / 0.88 for CLEA. A particularly informative within-family comparison is Qwen3-Embedding-0.6B + CLEA versus Qwen3-Reranker-0.6B + descriptions: CLEA matches the reranker on AGNews and exceeds it on the remaining three tasks, while retaining pre-encodable inference.

Why do descriptions benefit CLEA but not necessarily these alternatives? We hypothesize that the distinction lies in how descriptions are used. The baselines consume descriptions directly at inference time, making predictions sensitive to their wording, coverage, and specificity. CLEA instead uses descriptions as training supervision to adapt the shared embedding space through a discriminative contrastive objective, while retaining short verbalizers at inference. This interpretation is consistent with Appendix K: simply averaging description embeddings into prototypes underperforms the untuned encoder in 6 of 20 configurations, whereas CLEA outperforms these prototypes in 18 of 20.

Inference efficiency also remains important. Description-augmented NLI and rerankers require five document–description evaluations per label instead of one (20 rather than 4 for AGNews), while LLMs process longer prompts. CLEA's inference procedure and cost are unchanged by the number of training descriptions.

AuthorsOpenReview ↗

Follow-up to Reviewer rzoz (2/3): learning-rate selection

2. Learning-rate selection (W2, Q4)

We evaluated four LR-selection strategies across all 40 encoder–dataset combinations, using the same nine-candidate learning-rate grid. The training times reported in Appendix C include this learning-rate search, so they cover all nine candidates and not a single run. The table reports mean macro-F1 (%) and regret relative to the empirically best candidate. Brackets denote 95% intervals from 10,000 encoder-block bootstrap resamples, retaining all four datasets within each encoder.

StrategyMean macro-F1 (%)Oracle regret (pp)
Empirical oracle (uses test labels)78.000.00
Uniformity selection (CLEA)77.030.97 [0.62, 1.30]
Published training LR / proxy76.021.98 [1.32, 2.51]
Fixed LR (3e-5)73.514.49 [1.45, 9.96]

Uniformity selection achieves performance within approximately one percentage point of the empirical oracle, while outperforming the published-training-LR/proxy baseline by approximately 1.0 point and the fixed LR by 3.52 points [0.37, 9.19]. The former uses each model's published training LR where available, or the closest documented training recommendation as a proxy when the original LR is undisclosed. The fixed LR (3e-5) is the median of our predefined nine-candidate grid, chosen independently of performance. Its relatively wide interval reflects differences in LR suitability across encoder architectures.

These results strengthen the practical motivation for our uniformity criterion: it provides near-oracle performance without labeled validation data, while the model-specific published LR remains a reasonably effective, lower-cost fallback when an LR search is impractical. We thank the reviewer for suggesting this comparison, which substantially improves the empirical characterization of our selection procedure.

Reviewer_ybbmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CLEA adapts text encoders using label descriptions rather than labeled documents. A contrastive objective aligns descriptions with label representations, preserving efficient independent encoding at inference. Experiments across ten encoders and four datasets show broadly consistent improvements.

Critical strengths

• I appreciate the broad evaluation across ten encoders, including the full 77-class Banking77 experiment.

• The mean-description baseline and loss ablations help establish that training contributes beyond simply providing richer label descriptions.

Critical weaknesses

• The strongest comparisons do not clearly match supervision: CLEA receives curated descriptions, while prompted LLMs receive verbalizers. Description- writing effort is also not directly matched to document-labeling effort, weakening the “budget-matched” claim.

• The novelty framing is too broad. PESCO already classifies through independently encoded document and label representations, although it trains on unlabeled documents. CLEA’s distinction should focus on description-only adaptation.

Questions for authors
  1. Do the advantages remain when competing methods receive the same label descriptions?

  2. How does CLEA compare with PESCO under clearly specified data-access and tuning conditions?

  3. How much performance does uniformity-based selection lose relative to the best tested learning rate, and does it outperform a fixed default?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ybbm

Thank you for the constructive review and for recognizing the breadth of our evaluation and ablations.

When it comes to supervision matching we distinguish two complementary comparisons. First, Appendix K provides a controlled experiment isolating the contribution of description information from that of encoder adaptation: the mean-description prototype baseline uses exactly the same descriptions as CLEA but classifies without fine-tuning. CLEA outperforms it in 18 of 20 configurations, demonstrating gains beyond richer label representations alone. Second, we compare against established alternative paradigms for classification without labeled documents, including NLI cross-encoders, rerankers, and prompted LLMs, under their standard inference setups. We nevertheless agree that description-augmented variants would provide additional insight and are evaluating them using the same descriptions as CLEA. Importantly, this substantially increases inference costs for NLI and rerankers, requiring five document–description evaluations per label instead of one, whereas CLEA retains a single document encoding and pre-encodable label representations. We will quantify this trade-off alongside performance.

PESCO relies on unlabeled target documents and iterative pseudo-labeling/self-training, whereas CLEA adapts a pretrained dual encoder directly from a small, fixed set of label descriptions. CLEA thus avoids the more involved document-based training pipeline, requiring only lightweight fine-tuning on label text. Our contribution is this efficient description-only adaptation procedure, rather than the dual-encoder classification paradigm itself. We will sharpen this positioning in the manuscript. We thank the reviewer for this valuable observation.

Thank you for suggesting a comparison against the best-performing learning rate, which provides a useful measure of selection regret beyond the uniformity–F1 relationships already reported in Appendix O. We are preparing a comparison of uniformity-based selection against a fixed learning rate, the pretrained model's learning rate, and the oracle-best candidate in the search grid, using matched experimental settings. We will report both improvements over simpler alternatives and regret relative to the oracle.

We agree that matching human annotation time is difficult, and will clarify that our existing comparisons match labeled-example counts rather than annotation effort. In practice, acquiring a balanced few-shot training set can require substantially more effort than labeling k×Lk \times L documents, particularly with imbalanced classes. For example, the "surprise" class represents approximately 3.6% of EmotionDAIR, so obtaining 32 examples through random screening would require inspecting around 900 documents on average. Also, performance is sensitive to which examples are selected, as illustrated by the substantial variance of SetFit in Figure 4. Thus, annotation costs depend not only on the number of examples but also on class coverage and example selection. We will make this distinction explicit rather than claim a precise equivalence in human effort.

We will report additional findings as they become available.

AuthorsOpenReview ↗

Follow-up to Reviewer ybbm (1/2): baselines with the same descriptions

Thank you again for the constructive review. We have completed the two additional experiments outlined in our initial response.

1. Description-matched baselines (W1, Q1)

We extended all five baselines to receive the same five descriptions per label used to train CLEA. For NLI models and the reranker, we score each document against every description and average the resulting probabilities per label. For LLMs, each multiple-choice option contains the verbalizer followed by its five descriptions. The verbalizer-only results are from the original Table 5. All values are macro-F1.

MethodAGNewsBanking77EmotionDAIRRottenTom.
DeBERTa NLI, verbalizer0.810.860.440.85
+ descriptions0.110.360.120.75
ModernBERT NLI, verbalizer0.760.830.300.82
+ descriptions0.390.600.280.60
Qwen3-Reranker-0.6B, verbalizer0.790.870.410.78
+ descriptions0.850.810.490.76
Qwen3-4B, verbalizer0.820.910.430.87
+ descriptions0.790.870.430.85
Qwen3-8B, verbalizer0.850.910.400.86
+ descriptions0.800.910.410.76
Qwen3-Embedding-0.6B + CLEA0.850.920.570.88
Best CLEA encoder0.850.960.570.88

The effects of description augmentation differ considerably across architectures.

  • Reranker: Description augmentation substantially improves AGNews (+0.06) and EmotionDAIR (+0.08), but reduces performance on the other two datasets.
  • NLI: Both models deteriorate substantially. One possible explanation is that NLI models expect concise hypotheses expressing individual claims. Longer, multi-clause descriptions may be difficult for a single document to entail in full, reducing the discriminative value of entailment scores.
  • LLMs: Descriptions provide no consistent improvement. Qwen3-8B loses 0.05 on AGNews and 0.10 on RottenTomatoes. Further inspection suggests a shift toward particular labels rather than a simple prompt-length explanation. On RottenTomatoes, correct and incorrect predictions have similar average prompt lengths, but Qwen3-8B predicts "negative" for approximately 75% of reviews despite balanced classes, with approximately 98% of its classification errors consisting of positive reviews classified as negative. Qwen3-4B similarly overpredicts Business on AGNews.

The central conclusion from Table 5 remains supported. Allowing each baseline to use whichever input variant performs better, the strongest alternatives achieve 0.85 / 0.91 / 0.49 / 0.87 across the four datasets, compared with 0.85 / 0.96 / 0.57 / 0.88 for CLEA. A particularly informative within-family comparison is Qwen3-Embedding-0.6B + CLEA versus Qwen3-Reranker-0.6B + descriptions: CLEA matches the reranker on AGNews and exceeds it on the remaining three tasks, while retaining pre-encodable inference.

Why do descriptions benefit CLEA but not necessarily these alternatives? We hypothesize that the distinction lies in how descriptions are used. The baselines consume descriptions directly at inference time, making predictions sensitive to their wording, coverage, and specificity. CLEA instead uses descriptions as training supervision to adapt the shared embedding space through a discriminative contrastive objective, while retaining short verbalizers at inference. This interpretation is consistent with Appendix K: simply averaging description embeddings into prototypes underperforms the untuned encoder in 6 of 20 configurations, whereas CLEA outperforms these prototypes in 18 of 20.

Inference efficiency also remains important. Description-augmented NLI and rerankers require five document–description evaluations per label instead of one (20 rather than 4 for AGNews), while LLMs process longer prompts. CLEA's inference procedure and cost are unchanged by the number of training descriptions.

AuthorsOpenReview ↗

Follow-up to Reviewer ybbm (2/2): learning-rate selection

2. Learning-rate selection (Q3)

We evaluated four LR-selection strategies across all 40 encoder–dataset combinations, using the same nine-candidate learning-rate grid. The table reports mean macro-F1 (%) and regret relative to the empirically best candidate. Brackets denote 95% intervals from 10,000 encoder-block bootstrap resamples, retaining all four datasets within each encoder.

StrategyMean macro-F1 (%)Oracle regret (pp)
Empirical oracle (uses test labels)78.000.00
Uniformity selection (CLEA)77.030.97 [0.62, 1.30]
Published training LR / proxy76.021.98 [1.32, 2.51]
Fixed LR (3e-5)73.514.49 [1.45, 9.96]

Uniformity selection achieves performance within approximately one percentage point of the empirical oracle, while outperforming the published-training-LR/proxy baseline by approximately 1.0 point and the fixed LR by 3.52 points [0.37, 9.19]. The former uses each model's published training LR where available, or the closest documented training recommendation as a proxy when the original LR is undisclosed. The fixed LR (3e-5) is the median of our predefined nine-candidate grid, chosen independently of performance. Its relatively wide interval reflects differences in LR suitability across encoder architectures.

These results strengthen the practical motivation for our uniformity criterion: it provides near-oracle performance without labeled validation data, while the model-specific published LR remains a reasonably effective, lower-cost fallback when an LR search is impractical. We thank the reviewer for suggesting this comparison, which substantially improves the empirical characterization of our selection procedure.

We are incorporating both analyses into the revised manuscript and will upload the updated PDF before the deadline. We would welcome any further feedback on whether these results address your concerns or whether additional clarification would be helpful.

AuthorsOpenReview ↗

Reviewer ybbm: brief check before the discussion closes

As the discussion closes tomorrow, we wanted to check whether the description-matched baselines, the learning-rate comparison, and the narrowed positioning relative to PESCO, now in the revised PDF, address your concerns. If so, we would be grateful if you would consider updating your assessment.

Reviewer_J86QOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors adapt an off-the-shelf text-embedding model into a classifier for a new label set without any labeled documents, while keeping document embeddings cacheable. Its method, CLEA, fine-tunes the encoder only on label text: one short "verbalizer" sentence and five hand-written descriptions per label. A two-way contrastive loss pulls each verbalizer toward its descriptions, and documents are then classified by cosine similarity to the verbalizers.

Critical strengths
  • The paper is easy to read.
  • CLEA provides a cheap operating point that consistently improves on zero-shot use.
  • Extensive ablations let readers check which parts of the method matter, including whether training adds anything beyond the descriptions.
Critical weaknesses
  • The setting is task-specific supervised adaptation, not zero-shot, and it needs one encoder per dataset. If this is the setting considered by this paper, then I would suggest to authors to include comparisons which are trained on the same descriptions as labeled documents, and LLM synthetic-data training

  • The large-label regime that motivates dual encoders is not evaluated, and eXtreme multi-label classification is not discussed. In particular, none of the XMC dataset is considered, and not discussions to the existing XMC methods. Extreme Classification Repository (manikvarma.org/downloads/XC/XMLRepository.html), whose LF-* datasets come with label text.

  • For small label sets, the case against simply using an LLM is not made. With only 2–6 labels, why not prompt a current LLM, or use one to label a few hundred documents? The paper does not show when CLEA is the better choice.

Questions for authors
  • Is the intended deployment one encoder per label set?
  • Is the intended usecase for the scenario with very larger number of labels?
  • For the scenario with small number of labels, what does CLEA compare to directly prompt results using LLM with the label descriptors as candidates?
5 comments
AuthorsOpenReview ↗

Response to Reviewer J86Q

Thank you for the review and for recognizing CLEA's practical value. We would like to clarify its intended setting and address the suggested comparisons.

CLEA performs task-specific adaptation, but requires no labeled target documents. The intended deployment is one adapted encoder per label set, with independently computed document embeddings that can be reused for efficient classification.

CLEA targets efficient, repeated classification, not extreme multi-label classification specifically. Dual encoders are useful beyond large label spaces, as reflected in their broad application across embedding benchmarks such as MTEB (Muennighoff et al., 2023) and MMTEB (Enevoldsen et al., 2025). We additionally evaluate all 77 Banking77 classes in Appendix N, using three descriptions per label. CLEA consistently improves macro-F1 across four encoders: MiniLM (0.59→0.66), EmbeddingGemma (0.52→0.61), BGE-base (0.64→0.67), and E5-large (0.57→0.65). While XMC is outside our experimental scope, we will clarify this distinction and discuss the relevant literature.

We distinguish two complementary evaluations. First, Appendix K isolates the contribution of descriptions from that of adaptation: the mean-description baseline uses identical descriptions but classifies through averaged embeddings without fine-tuning. CLEA outperforms it in 18 of 20 configurations, demonstrating gains beyond richer label representations alone. Second, Appendix F (Table 5) compares CLEA against established approaches to classification without labeled documents, including NLI cross-encoders, a reranker, and prompted LLMs, under their standard inference setups. The best CLEA configuration matches or exceeds the strongest alternative on all four benchmarks while preserving pre-encodable inference. We appreciate the suggestion of a description-trained linear probe as an additional controlled baseline and are investigating it alongside description-augmented LLM, NLI, and reranker comparisons. We are also evaluating the proposed LLM pseudo-labeling pipeline, in which an LLM labels documents used to train a classifier, and hope to be able to report results still during the discussion period.

Why not simply use an LLM? Table 5 already evaluates prompted Qwen3-4B and Qwen3-8B. The best CLEA encoder matches their best performance on AGNews (0.85 vs. 0.85) and exceeds it on Banking77 (0.96 vs. 0.91), EmotionDAIR (0.57 vs. 0.43), and RottenTomatoes (0.88 vs. 0.87), while retaining pre-encodable inference (Reimers & Gurevych, 2019). CLEA is particularly attractive when LLM access is restricted, direct prompting is insufficiently accurate, or repeated classification at scale makes per-document LLM inference costly.

We will share additional results as they become available.

AuthorsOpenReview ↗

Follow-up to Reviewer J86Q (1/2): Description-Trained Linear Probe

Thank you again for the review. We have completed several of the additional comparisons outlined in our initial response and report them in two follow-up comments.

1. Linear probe trained on label descriptions (W1)

Following your suggestion, we trained a linear classifier on the same descriptions used by CLEA, treating them as labeled training examples. We follow the experimental setup of the main paper, using the same pretrained encoders, five descriptions per class, datasets, evaluation splits, and macro-F1 metric. The only difference is the adaptation procedure: the encoder remains frozen, and logistic regression is fitted on the description embeddings. No labeled target documents are used.

The mean-description rows correspond to the prototype baseline from Appendix K, and the CLEA rows to Table 5. All values are macro-F1.

EncoderMethodAGNewsBanking77EmotionDAIRRottenTom.
MiniLM-L6-v2Mean-description0.780.870.380.70
Linear probe0.730.900.390.69
CLEA0.790.900.430.70
e5-base-v2Mean-description0.780.930.400.82
Linear probe0.790.920.400.82
CLEA0.810.960.480.82
bge-base-en-v1.5Mean-description0.810.940.390.82
Linear probe0.810.940.410.82
CLEA0.820.950.470.82
e5-large-v2Mean-description0.690.940.420.84
Linear probe0.720.910.440.84
CLEA0.820.960.530.86
Qwen3-Embedding-0.6BMean-description0.810.950.520.82
Linear probe0.800.950.520.82
CLEA0.850.920.570.88

The linear probe offers no consistent advantage over averaging the descriptions into class prototypes: it performs better in 9 configurations, worse in 9, and ties in 2, with both methods averaging 0.73 macro-F1. CLEA outperforms the probe in 16 of the 20 configurations, ties in 3, and underperforms in only 1 (Qwen3-Embedding on Banking77), achieving an average macro-F1 of 0.77.

These results suggest that, with a frozen encoder, fitting a linear classifier on the descriptions provides little systematic benefit over using them directly as prototypes. In contrast, adapting the encoder through CLEA's contrastive objective yields more consistent improvements, despite using exactly the same supervision.

2. LLM pseudo-labeling (W3)

We have not yet completed the suggested experiment of using an LLM to pseudo-label target documents and subsequently train a classifier. We do not want to promise results we may not be able to deliver before the discussion closes, but will report them if available.

Our existing LLM evaluations in Table 5 provide some relevant context, although they cannot establish the performance of a classifier trained on pseudo-labels. Prompted LLMs perform relatively well on AGNews, Banking77, and RottenTomatoes, but this does not imply that a classifier trained on a few hundred potentially noisy predictions would necessarily match their performance or that of CLEA. The limitations are particularly evident on EmotionDAIR, where Qwen3-4B and Qwen3-8B achieve only 0.43 and 0.40 macro-F1, respectively, compared with 0.57 for CLEA. This suggests substantial pseudo-label noise, making competitive downstream performance far from assured. Whether subsequent classifier training could mitigate these errors remains an empirical question.

More fundamentally, this alternative requires both access to an LLM and an unlabeled target-document corpus, neither of which is needed for CLEA's description-only training. We therefore view pseudo-labeling as a relevant alternative operating under different data-access and computational requirements, rather than a directly equivalent supervision setting. We will clarify this distinction and discuss LLM pseudo-labeling and synthetic-data training in the revised manuscript.

AuthorsOpenReview ↗

Follow-up to Reviewer J86Q (2/2): LLM, NLI and reranker baselines with the same descriptions

3. Baselines given the same descriptions (Q3)

This addresses your question on prompting an LLM with the label descriptions as candidates.

We extended all five baselines to receive the same five descriptions per label used to train CLEA. For NLI models and the reranker, we score each document against every description and average the resulting probabilities per label. For LLMs, each multiple-choice option contains the verbalizer followed by its five descriptions. The verbalizer-only results are from the original Table 5. All values are macro-F1.

MethodAGNewsBanking77EmotionDAIRRottenTom.
DeBERTa NLI, verbalizer0.810.860.440.85
+ descriptions0.110.360.120.75
ModernBERT NLI, verbalizer0.760.830.300.82
+ descriptions0.390.600.280.60
Qwen3-Reranker-0.6B, verbalizer0.790.870.410.78
+ descriptions0.850.810.490.76
Qwen3-4B, verbalizer0.820.910.430.87
+ descriptions0.790.870.430.85
Qwen3-8B, verbalizer0.850.910.400.86
+ descriptions0.800.910.410.76
Qwen3-Embedding-0.6B + CLEA0.850.920.570.88
Best CLEA encoder0.850.960.570.88

The effects of description augmentation differ considerably across architectures.

  • Reranker: Description augmentation substantially improves AGNews (+0.06) and EmotionDAIR (+0.08), but reduces performance on the other two datasets.
  • NLI: Both models deteriorate substantially. One possible explanation is that NLI models expect concise hypotheses expressing individual claims. Longer, multi-clause descriptions may be difficult for a single document to entail in full, reducing the discriminative value of entailment scores.
  • LLMs: Descriptions provide no consistent improvement. Qwen3-8B loses 0.05 on AGNews and 0.10 on RottenTomatoes. Further inspection suggests a shift toward particular labels rather than a simple prompt-length explanation. On RottenTomatoes, correct and incorrect predictions have similar average prompt lengths, but Qwen3-8B predicts "negative" for approximately 75% of reviews despite balanced classes, with approximately 98% of its classification errors consisting of positive reviews classified as negative. Qwen3-4B similarly overpredicts Business on AGNews.

The central conclusion from Table 5 remains supported. Allowing each baseline to use whichever input variant performs better, the strongest alternatives achieve 0.85 / 0.91 / 0.49 / 0.87 across the four datasets, compared with 0.85 / 0.96 / 0.57 / 0.88 for CLEA. A particularly informative within-family comparison is Qwen3-Embedding-0.6B + CLEA versus Qwen3-Reranker-0.6B + descriptions: CLEA matches the reranker on AGNews and exceeds it on the remaining three tasks, while retaining pre-encodable inference.

Why do descriptions benefit CLEA but not necessarily these alternatives? We hypothesize that the distinction lies in how descriptions are used. The baselines consume descriptions directly at inference time, making predictions sensitive to their wording, coverage, and specificity. CLEA instead uses descriptions as training supervision to adapt the shared embedding space through a discriminative contrastive objective, while retaining short verbalizers at inference. This interpretation is consistent with Appendix K: simply averaging description embeddings into prototypes underperforms the untuned encoder in 6 of 20 configurations, whereas CLEA outperforms these prototypes in 18 of 20.

Inference efficiency also remains important. Description-augmented NLI and rerankers require five document–description evaluations per label instead of one (20 rather than 4 for AGNews), while LLMs process longer prompts. CLEA's inference procedure and cost are unchanged by the number of training descriptions.

A correction to our first response. While extending the full 77-class Banking77 evaluation in Appendix N, we identified and corrected an evaluation issue involving documents with missing labels. The corrected macro-F1 values are MiniLM 0.61 to 0.68, EmbeddingGemma 0.54 to 0.63, BGE-base 0.67 to 0.70 and E5-large 0.60 to 0.67, and they supersede the values quoted in our first response. CLEA’s gains over the untuned encoders are essentially unchanged, and the model checkpoints are the same. The full table, including a mean-description baseline, is in our follow-up to Reviewer rzoz (3/3).

We are incorporating these results into the revised manuscript and will upload the updated PDF before the deadline. We would welcome your view on whether the clarified scope and the added baselines address your main concerns, or which of them remain open.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions and additional results

We thank the reviewers for their constructive feedback and the AC for their time and consideration. We have now uploaded the revised manuscript, incorporating the additional analyses and clarifications discussed in our responses. The main findings and conclusions remain unchanged, with the new results providing further support for them.

The main updates, by location:

  • Positioning and scope (Sections 2–3, Limitations). We clarify CLEA's distinction from related approaches, including PESCO, Gao et al. (2023), Hu et al. (2024), and Khosla et al. (2020), and acknowledge extreme multi-label classification as a direction for future work.

  • Additional baselines (Appendices F and P). We include description-matched NLI, reranker, and LLM comparisons, alongside a description-trained linear probe and an LLM pseudo-labeling pipeline. These additional controls reinforce the conclusions of the original evaluation.

  • Learning-rate selection (Appendix O). We quantify selection regret against an empirical oracle and compare uniformity-based selection with fixed and published learning rates, confirming the effectiveness of the proposed criterion.

  • Full Banking77 evaluation (Appendix N). We extend the existing 77-class evaluation with a mean-description prototype baseline using the same descriptions as CLEA, providing a more direct comparison that further supports the benefits of encoder adaptation.

Overall, these additions provide further evidence for the findings established in the original manuscript, while clarifying the scope and practical trade-offs of CLEA.

We appreciate the reviewers' suggestions and welcome any remaining feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.