- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
CLEA fine-tunes a shared text encoder using label verbalizers and a few descriptions per class, without labeled target-task documents. A bidirectional contrastive loss aligns labels with descriptions; inference uses document-label cosine similarity. Learning rates are selected by embedding uniformity on unlabeled target-domain texts. Across ten encoders and four benchmarks, the paper reports an average absolute macro-F1 gain of about 0.09 over untuned models. The main Banking77 evaluation covers six intents, with a separate experiment on all 77 classes.
Critical strengths
- CLEA offers a simple adaptation method without labeled target-task documents or an extra classifier head. Documents and labels remain independently encodable, which is useful for large-scale, repeated classification.
- Results across ten encoders suggest that the method generalizes across backbones. It also beats mean-description prototypes using the same descriptions in 18 of 20 combinations, supporting a benefit beyond richer label text alone.
Critical weaknesses
1.The contribution over prior work is not yet clear. Direct comparisons with Gao et al. (EMNLP 2023) and Hu et al. (NLP4ConvAI 2024) are missing. The column loss also shares the positive-sum-inside-log structure discussed in Khosla et al. (2020), Eq. (3), although the formulations differ. Its average gain over rows-only training is only about 0.01 macro-F1, with no uncertainty reported for that difference. 2.The accuracy-cost claim needs more evidence. CLEA receives multiple descriptions per class, while the NLI and LLM baselines use short verbalizers. Appendix O reports uniformity–F1 correlations, but not the gain over fixed learning rates. Reported costs do not clearly account for learning-rate search, and the full 77-class experiment compares only with untuned encoders.
Questions for authors
- What does the column loss gain over averaging positive log-probabilities? Please report its improvement over rows-only training and the uncertainty using paired runs.
- How do the NLI and LLM results change when these baselines receive the same class descriptions?
- Do the training times include learning-rate search? Please provide comparable costs, measured inference latency, and a strong description-based baseline on the full 77-class task.
- How does uniformity-based selection compare with a fixed learning rate and reuse of the pretrained model's learning rate? Please match encoders, descriptions, and random seeds, and report F1 differences with uncertainty to isolate the benefit of document-based selection.
5 comments
Follow-up to Reviewer rzoz (1/3): description-matched baselines
Thank you again for the detailed review. We have completed three of the additional analyses outlined in our initial response and report them in three comments.
1. Description-matched baselines (W2, Q2)
We extended all five baselines to receive the same five descriptions per label used to train CLEA. For NLI models and the reranker, we score each document against every description and average the resulting probabilities per label. For LLMs, each multiple-choice option contains the verbalizer followed by its five descriptions. The verbalizer-only results are from the original Table 5. All values are macro-F1.
| Method | AGNews | Banking77 | EmotionDAIR | RottenTom. |
|---|---|---|---|---|
| DeBERTa NLI, verbalizer | 0.81 | 0.86 | 0.44 | 0.85 |
| + descriptions | 0.11 | 0.36 | 0.12 | 0.75 |
| ModernBERT NLI, verbalizer | 0.76 | 0.83 | 0.30 | 0.82 |
| + descriptions | 0.39 | 0.60 | 0.28 | 0.60 |
| Qwen3-Reranker-0.6B, verbalizer | 0.79 | 0.87 | 0.41 | 0.78 |
| + descriptions | 0.85 | 0.81 | 0.49 | 0.76 |
| Qwen3-4B, verbalizer | 0.82 | 0.91 | 0.43 | 0.87 |
| + descriptions | 0.79 | 0.87 | 0.43 | 0.85 |
| Qwen3-8B, verbalizer | 0.85 | 0.91 | 0.40 | 0.86 |
| + descriptions | 0.80 | 0.91 | 0.41 | 0.76 |
| Qwen3-Embedding-0.6B + CLEA | 0.85 | 0.92 | 0.57 | 0.88 |
| Best CLEA encoder | 0.85 | 0.96 | 0.57 | 0.88 |
The effects of description augmentation differ considerably across architectures.
- Reranker: Description augmentation substantially improves AGNews (+0.06) and EmotionDAIR (+0.08), but reduces performance on the other two datasets.
- NLI: Both models deteriorate substantially. One possible explanation is that NLI models expect concise hypotheses expressing individual claims. Longer, multi-clause descriptions may be difficult for a single document to entail in full, reducing the discriminative value of entailment scores.
- LLMs: Descriptions provide no consistent improvement. Qwen3-8B loses 0.05 on AGNews and 0.10 on RottenTomatoes. Further inspection suggests a shift toward particular labels rather than a simple prompt-length explanation. On RottenTomatoes, correct and incorrect predictions have similar average prompt lengths, but Qwen3-8B predicts "negative" for approximately 75% of reviews despite balanced classes, with approximately 98% of its classification errors consisting of positive reviews classified as negative. Qwen3-4B similarly overpredicts Business on AGNews.
The central conclusion from Table 5 remains supported. Allowing each baseline to use whichever input variant performs better, the strongest alternatives achieve 0.85 / 0.91 / 0.49 / 0.87 across the four datasets, compared with 0.85 / 0.96 / 0.57 / 0.88 for CLEA. A particularly informative within-family comparison is Qwen3-Embedding-0.6B + CLEA versus Qwen3-Reranker-0.6B + descriptions: CLEA matches the reranker on AGNews and exceeds it on the remaining three tasks, while retaining pre-encodable inference.
Why do descriptions benefit CLEA but not necessarily these alternatives? We hypothesize that the distinction lies in how descriptions are used. The baselines consume descriptions directly at inference time, making predictions sensitive to their wording, coverage, and specificity. CLEA instead uses descriptions as training supervision to adapt the shared embedding space through a discriminative contrastive objective, while retaining short verbalizers at inference. This interpretation is consistent with Appendix K: simply averaging description embeddings into prototypes underperforms the untuned encoder in 6 of 20 configurations, whereas CLEA outperforms these prototypes in 18 of 20.
Inference efficiency also remains important. Description-augmented NLI and rerankers require five document–description evaluations per label instead of one (20 rather than 4 for AGNews), while LLMs process longer prompts. CLEA's inference procedure and cost are unchanged by the number of training descriptions.
Follow-up to Reviewer rzoz (2/3): learning-rate selection
2. Learning-rate selection (W2, Q4)
We evaluated four LR-selection strategies across all 40 encoder–dataset combinations, using the same nine-candidate learning-rate grid. The training times reported in Appendix C include this learning-rate search, so they cover all nine candidates and not a single run. The table reports mean macro-F1 (%) and regret relative to the empirically best candidate. Brackets denote 95% intervals from 10,000 encoder-block bootstrap resamples, retaining all four datasets within each encoder.
| Strategy | Mean macro-F1 (%) | Oracle regret (pp) |
|---|---|---|
| Empirical oracle (uses test labels) | 78.00 | 0.00 |
| Uniformity selection (CLEA) | 77.03 | 0.97 [0.62, 1.30] |
| Published training LR / proxy | 76.02 | 1.98 [1.32, 2.51] |
| Fixed LR (3e-5) | 73.51 | 4.49 [1.45, 9.96] |
Uniformity selection achieves performance within approximately one percentage point of the empirical oracle, while outperforming the published-training-LR/proxy baseline by approximately 1.0 point and the fixed LR by 3.52 points [0.37, 9.19]. The former uses each model's published training LR where available, or the closest documented training recommendation as a proxy when the original LR is undisclosed. The fixed LR (3e-5) is the median of our predefined nine-candidate grid, chosen independently of performance. Its relatively wide interval reflects differences in LR suitability across encoder architectures.
These results strengthen the practical motivation for our uniformity criterion: it provides near-oracle performance without labeled validation data, while the model-specific published LR remains a reasonably effective, lower-cost fallback when an LR search is impractical. We thank the reviewer for suggesting this comparison, which substantially improves the empirical characterization of our selection procedure.
Response to Reviewer rzoz
Thank you for the detailed technical review and the opportunity to clarify our contribution.
We thank the reviewer for highlighting the related work connections:
Adding the columnwise loss yields a modest but systematic improvement (+0.01 macro-F1 on average), improving or matching rows-only performance in 37 of 40 model–dataset configurations (Appendix G, Table 7). More importantly, it provides two additional properties without sacrificing overall predictive performance:
These properties motivate our formulation relative to averaging positive log-probabilities. The rowwise term already emphasizes difficult descriptions, while the averaged-log alternative similarly encourages alignment with individual, potentially noisy positives. Our column term instead emphasizes strongly aligned positives, providing a complementary optimization signal rather than duplicating the rowwise term.
For description-matched baselines, we distinguish two complementary comparisons. Appendix K isolates description information from adaptation: the mean-description prototype baseline uses identical descriptions but classifies through averaged embeddings without fine-tuning. CLEA outperforms it in 18 of 20 configurations. Separately, our NLI, reranker, and LLM comparisons evaluate established approaches to classification without labeled documents under their standard inference setups. We agree that description-augmented variants would provide additional insight and are evaluating them using the same class descriptions. Importantly, these variants substantially increase inference costs for NLI and rerankers, requiring five document–description evaluations per label instead of one, whereas CLEA retains its pre-encodable inference profile. We will quantify this trade-off alongside performance.
Thank you for suggesting a comparison against the oracle-best learning rate, providing a useful measure of selection regret beyond the uniformity–F1 relationships reported in Appendix O. We are comparing uniformity-based selection against a fixed learning rate, the pretrained learning rate, and the oracle-best candidate under matched settings. We will report improvements over simpler alternatives, regret relative to the oracle, and a more precise breakdown of computational costs.
Appendix N evaluates CLEA on all 77 Banking77 classes, demonstrating improvements across four encoders. We agree that a stronger description-based comparison would be useful. Appendix K already implements a mean-description prototype baseline that embeds each description, averages embeddings into class prototypes, and classifies by similarity without fine-tuning. This provides a controlled comparison using identical descriptions, which we will extend to the full 77-class setting.
We will provide quantitative updates as these analyses are completed.