acceptodds
Under review as a conference paper at ICLR 2027

An LLM Hiddens Several Semantic Experts: Probing Task-Specific Attention Values for Text Embedding

Abstract

Different text embedding tasks require different notions of semantic similarity, motivating representations that adapt to the task. In this paper, we investigate whether its attention head modules already contain task-specific semantic signals. Specifically, we introduce a training-free framework that probes and ranks individual modules using augmented text pairs or downstream reference data, then constructs compact task-specific embeddings from their attention values. Two complementary augmentations enable selection using unlabeled data; task instructions can also select and combine dataset-specific experts. Probing is performed offline and routing is decided once per task, while each text requires only one frozen-model forward pass. Across 14 MTEB tasks, eight selected modules yield 1024-dimensional embeddings that exceed the strongest prompt-based ensemble by 4.66 and 5.08 average points on Llama3-8B and Qwen3-8B, respectively. Each text requires one forward pass; the ensemble uses eight. Dynamic routing achieves the highest mean at this dimension on both models. On a trained LLM2Vec embedder, selected modules surpass its native 4096-dimensional representation with either 2048 or 4096 dimensions. These results show that offline task-specific probing can turn existing dense models into flexible, compact embedders.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Text Embedding, Attention Value, Semantics, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_K86eOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a training-free framework that probes and ranks individual attention head modules to construct compact task-specific embeddings from their attention values. Using augmented text pairs or downstream reference data, the framework performs offline probing and task-specific routing while requiring only one frozen-model forward pass per text.

Critical strengths
  1. The paper is clearly written and well organized.
  2. The proposed method is simple and can be applied without additional training.
  3. The paper provides extensive experimental evaluations.
Critical weaknesses
  1. Limited evaluation on dedicated embedding models. The paper focuses on task-specific text embeddings, but most experiments are conducted on general-purpose generative LLMs rather than models specifically trained for embedding tasks. While this is consistent with the training-free setting, it remains unclear how relevant the findings are to existing embedding models.
  2. Marginal improvements on trained embedding models. On LLM2Vec, the improvements over the native embeddings are only 0.15–0.27 points, which are too small to demonstrate a meaningful improvement in embedding quality. The results seem more convincing as evidence of dimensionality reduction than of better task-specific representations.
  3. Questionable necessity of dynamic routing. DR-1 performs worse than static selection, while DR-2 brings only marginal improvements of 0.17–0.45 points. This raises doubts about whether dynamic routing is necessary, especially given the additional complexity of expert selection and combination.
  4. Unclear advantage over instruction-based adaptation. The motivation that different tasks require different embedding representations is already well established. Existing instruction-tuned embedding models can adapt their representations through task instructions. Although the proposed approach avoids additional training, the paper does not clearly establish what head-level selection offers beyond existing instruction-based adaptation.
Questions for authors
  1. Would the proposed method still provide meaningful improvements on modern instruction-tuned embedding models?
  2. Given the marginal gains on LLM2Vec, should the main contribution be viewed as embedding compression rather than improved task-specific representation quality?
  3. What is the actual benefit of dynamic routing over static selection or simple expert averaging? Could the authors isolate the contribution of the routing mechanism itself?
  4. If an instruction-tuned embedding model can already produce different representations by changing task instructions, what additional capability does task-specific head selection provide?
4 comments

Thank you for your valuable feedback. Our responses are as follows:

W1&Q1: Limited evaluation on dedicated embedding models. The paper focuses on task-specific text embeddings, but most experiments are conducted on general-purpose generative LLMs rather than models specifically trained for embedding tasks. While this is consistent with the training-free setting, it remains unclear how relevant the findings are to existing embedding models. Would the proposed method still provide meaningful improvements on modern instruction-tuned embedding models?

This paper primarily focuses on the training-free setting, with an initial exploration of the post-processing setting. We respectfully disagree that evaluation on trained embedding models is required. A substantial body of work studies how to obtain high-quality text embeddings from LLMs without training [1, 2, 3, 4, 5, 6], and our paper follows the same research focus.

W2&Q2: Marginal improvements on trained embedding models. On LLM2Vec, the improvements over the native embeddings are only 0.15–0.27 points, which are too small to demonstrate a meaningful improvement in embedding quality. The results seem more convincing as evidence of dimensionality reduction than of better task-specific representations. Given the marginal gains on LLM2Vec, should the main contribution be viewed as embedding compression rather than improved task-specific representation quality?

LLM2Vec pools the hidden states from its final layer to obtain text embeddings and optimizes these embeddings through contrastive learning during training. The fact that embeddings extracted from attention values can match the performance of its native embeddings is itself an interesting finding. From a practical perspective, we agree that a direct application of our findings is to obtain high-quality, low-dimensional text embeddings without additional training.

W3&Q3: Questionable necessity of dynamic routing. DR-1 performs worse than static selection, while DR-2 brings only marginal improvements of 0.17–0.45 points. This raises doubts about whether dynamic routing is necessary, especially given the additional complexity of expert selection and combination. What is the actual benefit of dynamic routing over static selection or simple expert averaging? Could the authors isolate the contribution of the routing mechanism itself?

Dynamic routing need not outperform static routing, since the latter incorporates human prior knowledge. Our purpose in introducing dynamic routing is not to outperform static routing, but to present an automatic routing approach that adapts when probing or evaluation datasets are added, without requiring people to manually assign the source modules for each target task's embeddings.

W4&Q4: Unclear advantage over instruction-based adaptation. The motivation that different tasks require different embedding representations is already well established. Existing instruction-tuned embedding models can adapt their representations through task instructions. Although the proposed approach avoids additional training, the paper does not clearly establish what head-level selection offers beyond existing instruction-based adaptation. If an instruction-tuned embedding model can already produce different representations by changing task instructions, what additional capability does task-specific head selection provide?

As discussed in the Introduction, existing instruction-following embedders primarily produce different embeddings by prepending an instruction to the input text:

G(t,x)=F(It⨁x)G(t, x) = F(I_t \bigoplus x)

Our approach takes a different route by changing the source modules from which text embeddings are extracted for different tasks, effectively providing a different mapping function for each task:

G(t,x)=Ft(x)G(t, x) = F_t(x)

On the one hand, these two approaches can serve as alternatives. One potential advantage of our approach is that routing can be determined using either a fixed policy (static routing) or similarity between instructions (dynamic routing), without prepending an instruction to each text before encoding. Prepending instructions can introduce substantial additional overhead even when a KV cache is used.

On the other hand, the two approaches can also be combined:

G(t,x)=Ft(It⨁x)G(t, x) = F_t(I_t \bigoplus x)

This is because prepending instructions and selecting task-specific source modules for embeddings are orthogonal choices. LLM2Vec, which we use in our experiments, is one such example: it is trained on the instruction-augmented E5 public dataset and is itself an instruction-following embedder. We use instructions during both probing and embedding extraction for LLM2Vec. As shown in Table 8 of the appendix, downstream performance decreases when instructions are omitted during probing and evaluation.

[1] LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States. ICML 2026

[2] Repetition Improves Language Model Embeddings. ICLR 2025

[3] Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free. ICLR 2025

[4] Meta-Task Prompting Elicits Embeddings from Large Language Models. ACL 2024

[5] Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering. ACL 2025

[6] PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval. EMNLP 2024

Thank you for the response and clarifications. While I appreciate the authors' explanations regarding the scope and motivation of the proposed method, my core concerns remain insufficiently addressed:

  • Limited Evaluation on Trained Embedding Models: The evaluation remains limited to LLM2Vec, without additional experiments on other modern embedding models, such as Qwen3-Embedding or E5-Mistral.
  • Benefits Beyond Instruction-Based Adaptation: Combining head selection with instructions on LLM2Vec yields only marginal improvements over its native embeddings. Moreover, the claim that "Prepending instructions can introduce substantial additional overhead even when a KV cache is used" lacks empirical support. Since task instructions are typically short and shared instruction prefixes can be cached and reused across inputs in causal LLMs, quantitative latency comparisons are needed to substantiate the claim of substantial additional overhead.

Therefore, I keep my original score.

Thank you for your valuable feedback. Our responses are as follows:

W1: Limited Evaluation on Trained Embedding Models: The evaluation remains limited to LLM2Vec, without additional experiments on other modern embedding models, such as Qwen3-Embedding or E5-Mistral.

Thank you for your comment. We initially understood your concern to be primarily about the focus of our paper, so we did not include the results for the additional embedding model in our response to your review. In fact, in addition to LLM2Vec, which is based on Llama3-8B, we have also added experiments on GritLM, which is based on Mistral-7B. The results are as follows:

MethodDim.=256Dim.=512Dim.=1024Dim.=2048
TaskEval (DR-2)66.0467.1567.5468.14
TaskEval66.0666.8067.6067.86
PCA63.9365.7766.3665.93
Gaussian Random Projection64.05 ± 0.2366.03 ± 0.0566.97 ± 0.0167.42 ± 0.10
Whitening64.3164.9363.5561.73
Matryoshka-Adapter64.0565.9466.8867.41

Random projection, whitening, and Matryoshka-Adapter [1] are newly added baselines. The experiments show that Matryoshka-Adapter, which requires additional training and a projection head, performs best among the baselines. TaskEval and TaskEval (DR-2) achieve the best results across all dimensionality-reduction settings on GritLM, consistent with the results on LLM2Vec, demonstrating the competitiveness of our approach.

W2: Benefits Beyond Instruction-Based Adaptation: Combining head selection with instructions on LLM2Vec yields only marginal improvements over its native embeddings. Moreover, the claim that "Prepending instructions can introduce substantial additional overhead even when a KV cache is used" lacks empirical support. Since task instructions are typically short and shared instruction prefixes can be cached and reused across inputs in causal LLMs, quantitative latency comparisons are needed to substantiate the claim of substantial additional overhead.

Thank you for your comment. We do not claim in the paper that current embedders should discard instruction prefixes during training and instead route different tasks to different attention head modules. Our experience with LLM2Vec and GritLM suggests that the two approaches are complementary. We therefore do not consider a comparison with instruction-only methods central to this paper, and we do not believe that its absence should be regarded as a major weakness.

Due to time constraints, we may be unable to provide a quantitative latency comparison. We first provide a theoretical analysis here: concatenating an instruction of length MM with an input text of length NN results in an online inference time complexity of O((M+N)2)O((M+N)^2). With a KV cache, this is reduced to O(NM+N2)O(NM+N^2). Our routing method encodes the instruction and the input text separately. Since instruction encoding and routing can both be performed in advance, the online inference time complexity is O(N2)O(N^2). Moreover, in practice, attention head modules in the last few layers are typically not selected. Text embeddings can therefore be obtained without running the input text through the final layer, which can further reduce encoding time.

[1] Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions. EMNLP 2024

Reviewer_ErXMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds on recent work investigating how pretrained LLMs can be repurposed as text embedding models without additional training. In particular, it extends the value aggregation approach of Zhang et al. (2026), which constructs embeddings from attention-value representations selected at the Transformer layer level.

The authors propose a more fine-grained approach that ranks and selects individual attention head modules across layers. They introduce two training-free probing strategies based on unlabeled text augmentations (CROP/ICT), alongside supervised module selection using downstream reference datasets. They further explore static and dynamic routing mechanisms to construct task-specific embeddings from the selected modules.

Experiments across 14 MTEB tasks and two LLM backbones report improvements over reported training-free embedding baselines while producing compact representations. The authors also apply their approach to a pretrained LLM2Vec embedding model, demonstrating that selected attention-value modules can preserve competitive embedding performance at reduced dimensionality, outperforming PCA-based dimensionality reduction.

Critical strengths
  1. The paper presents an interesting extension of existing attention-value embedding methods by selecting individual attention modules rather than entire Transformer layers. Particularly promising is the use of unlabeled text augmentations to identify useful modules without additional model training or labeled downstream data. The resulting embeddings substantially outperform the layer-wise VA baseline while retaining a single forward pass and a compact representation.

  2. The authors evaluate their approach across multiple task categories, two LLM families, and different model sizes. They also investigate the effects of probing budgets, scoring objectives, embedding dimensionality, and aggregation strategies. The extension to a pretrained LLM2Vec model provides additional evidence that selected attention-value representations can preserve embedding quality at substantially reduced dimensionality.

Critical weaknesses
  1. Insufficient benchmarking and limited understanding of the source of improvements. The paper builds directly on Zhang et al. (2026), but compares primarily against their basic layer-wise Value Aggregation approach, omitting the stronger AlignedWVA method proposed in the same work. This is particularly relevant given that AlignedWVA already achieves average scores of approximately 56.8 on Qwen3-8B, compared with 57.2 for ICT/CROP, 57.6 for static TaskEval, and 57.8 for DR-2 in the present paper. Although these cross-paper results are not directly comparable, they suggest that the improvements over the strongest existing approach may be considerably smaller than the reported gains over VA (LMP). A direct comparison under matched evaluation conditions is therefore important for assessing the incremental contribution. Moreover, the improvements over layer-wise aggregation are not fully disentangled from the change in aggregation strategy (concatenation versus averaging). A comparison with similarly selected intermediate hidden-state representations would help establish whether the gains are specific to attention values. Less severe, but the dimensionality-reduction experiments are limited to a single pretrained embedding model and primarily compare against PCA. It would be useful to generalize the results.

  2. Limited evidence for generalization and task-specific semantic expertise. The supervised module-selection approaches rely on a relatively small collection of reference datasets, some of which overlap with the final evaluation benchmarks at the dataset level, although different splits are used. This raises concerns about generalization to genuinely unseen tasks and sensitivity to the choice of reference datasets. The dynamic-routing methods additionally rely on instruction similarity, without evaluating robustness to instruction reformulations or alternative reference datasets. Their improvements over static TaskEval are generally small, while the proposed rank-wise aggregation of potentially unaligned module representations lacks a clear justification. More fundamentally, the paper's central narrative is that individual attention modules act as task-specific semantic experts, yet the empirical analysis does not convincingly establish this interpretation. Showing that certain modules perform better on particular downstream tasks does not necessarily imply that they encode distinct or specialized semantic information. The observed improvements could instead arise from selecting useful projections of intermediate hidden representations, potentially containing substantial redundancy. Without further analysis of representation geometry, redundancy, or functional specialization, the semantic-expert interpretation remains largely unsubstantiated.

In my view, the most compelling finding is that a relatively simple module-selection procedure based on unlabeled text can produce competitive embeddings, approaching the performance of methods that rely on labeled downstream reference datasets. This is an interesting and practically relevant result. By contrast, the more elaborate dynamic-routing approaches provide relatively limited additional gains, while the central claim that the selected modules constitute distinct, task-specific semantic experts is not convincingly established by the experiments. I therefore see some mismatch between the paper's strongest empirical contribution and its broader narrative. A more focused presentation emphasizing training-free module selection, together with deeper analysis of why it works and how well it generalizes, could make the paper's contribution clearer and more convincing.

Questions for authors
  1. Could the authors clarify why AlignedWVA from Zhang et al. (2026) is not included among the baselines? A comparison under matched evaluation conditions would help establish the incremental contribution of the proposed approach. Similarly, how would the results change if the module-selection procedure were applied to intermediate hidden-state representations?

  2. To what extent do the supervised results depend on the particular reference datasets used for module selection? Could the authors provide a leave-one-dataset-out or cross-dataset evaluation in which the target dataset is entirely excluded from the selection procedure?

  3. How robust are DR-1 and DR-2 to semantically equivalent reformulations of task instructions and changes in the reference-dataset pool? Additionally, what motivates averaging nonshared module representations by rank, rather than combining module scores and selecting the highest-ranked modules?

4 comments

Thank you for your valuable feedback. Our responses are as follows:

W1(a)&Q1(a). Insufficient benchmarking and limited understanding of the source of improvements. The paper builds directly on Zhang et al. (2026), but compares primarily against their basic layer-wise Value Aggregation approach, omitting the stronger AlignedWVA method proposed in the same work. This is particularly relevant given that AlignedWVA already achieves average scores of approximately 56.8 on Qwen3-8B, compared with 57.2 for ICT/CROP, 57.6 for static TaskEval, and 57.8 for DR-2 in the present paper. Although these cross-paper results are not directly comparable, they suggest that the improvements over the strongest existing approach may be considerably smaller than the reported gains over VA (LMP). A direct comparison under matched evaluation conditions is therefore important for assessing the incremental contribution. Could the authors clarify why AlignedWVA from Zhang et al. (2026) is not included among the baselines?

Thank you for this valuable suggestion. We agree that AlignedWVA should at least be discussed and compared in the paper. Beyond performance, our approach offers two advantages over AlignedWVA: (1) Broader applicability. AlignedWVA requires an additional EOL prompt. Although this is usable in training-free settings, it is incompatible with existing instruction-following embedders. As shown in the main text and appendix, our approach can be applied to fine-tuned instruction-following embedders such as LLM2Vec and GritLM. (2) Flexible dimensionality. AlignedWVA has the same fixed dimensionality as the hidden states, namely 4096. Our embeddings are constructed by concatenating the outputs of selected attention head modules, each contributing only 128 dimensions, allowing flexible control over the embedding dimension. The main-table results use eight concatenated modules, yielding 1024 dimensions, substantially fewer than AlignedWVA's 4096.

To address your concern, we developed a variant that incorporates our task-specific approach into FutureEOL. The results are shown below:

MethodModelDim.Avg. Performance
AlignedWVA (FutureEOL)Qwen3-8B409656.78
AlignedWVA (TaskEval + FutureEOL) (Ours)Qwen3-8B409658.31
AlignedWVA (ICT/CROP + FutureEOL) (Ours)Qwen3-8B409658.18
TaskEval (Ours)Qwen3-8B102457.58
ICT/CROP (Ours)Qwen3-8B102457.15

These results show that task-specific selection can further improve AlignedWVA, surpassing both AlignedWVA (FutureEOL) and our reported TaskEval results. We will add a section presenting this comparison to the experimental evaluation. Together with the discussion above, these results demonstrate the competitiveness of our approach, which performs well across a broader range of models and at lower dimensionality.

W1(b)&Q1(b). Moreover, the improvements over layer-wise aggregation are not fully disentangled from the change in aggregation strategy (concatenation versus averaging). A comparison with similarly selected intermediate hidden-state representations would help establish whether the gains are specific to attention values. A comparison under matched evaluation conditions would help establish the incremental contribution of the proposed approach. Similarly, how would the results change if the module-selection procedure were applied to intermediate hidden-state representations?

We would like to clarify two points: (1) module-level attention-value selection and concatenation need to be used together. In LLMs with 7–8B parameters, each attention head module produces only 128 dimensions, so we concatenate the outputs of multiple modules to increase the embedding's representational capacity. (2) The experiments in [1] have already shown that intermediate-layer attention values outperform intermediate-layer hidden states as general-purpose embeddings. We therefore directly use intermediate-layer attention values, namely VA (LMP), as the baseline for comparison.

To address your concern, we also applied task-specific selection to intermediate hidden-state layers. The results are as follows:

MethodDim.Llama3-8BQwen3-8B
HS (Full)409634.0530.78
HS (Half)409636.3532.57
HS (TaskEval, Top-1 layer)409638.8634.68
HS (TaskEval, Top-3 layers)409639.7234.85
HS (TaskEval, Top-5 layers)409638.9135.47
VA (Full)102447.9149.23
VA (Half)102448.6749.37
TaskEval102457.7557.58

These results show that attention values provide a strong starting point and benefit more from task-specific modeling.

W1(c). Less severe, but the dimensionality-reduction experiments are limited to a single pretrained embedding model and primarily compare against PCA. It would be useful to generalize the results.

For a fair comparison, the baselines must use only the probing datasets while supporting dimensionality reduction. We therefore chose the widely used PCA as a baseline. To address your concern, we extended the experiments to GritLM and added comparisons with random projection, whitening, and Matryoshka-Adapter [1]. We ran random projection three times and report the mean and standard deviation.

The results for LLM2Vec-Llama3-8B are as follows:

MethodDim.=256Dim.=512Dim.=1024Dim.=2048
TaskEval (DR-2)65.2466.4766.9967.49
TaskEval63.9765.8866.6267.56
PCA63.0064.8965.6265.41
Gaussian Random Projection63.74 ± 0.1565.50 ± 0.1566.27 ± 0.1166.77 ± 0.08
Whitening62.5862.7561.5959.92
Matryoshka-Adapter64.5665.7166.6067.05

The results for GritLM (based on Mistral-7B) are as follows:

MethodDim.=256Dim.=512Dim.=1024Dim.=2048
TaskEval (DR-2)66.0467.1567.5468.14
TaskEval66.0666.8067.6067.86
PCA63.9365.7766.3665.93
Gaussian Random Projection64.05 ± 0.2366.03 ± 0.0566.97 ± 0.0167.42 ± 0.10
Whitening64.3164.9363.5561.73
Matryoshka-Adapter64.0565.9466.8867.41

The experiments show the following performance ranking among the three training-free baselines in most settings: Random Projection > PCA > Whitening. Matryoshka-Adapter, which requires additional training and a projection head, achieves the best performance among these baselines. TaskEval and TaskEval (DR-2) achieve the best results across all dimensionality-reduction settings on GritLM and LLM2Vec, demonstrating the competitiveness of our approach.

W2(a) & Q2. Limited evidence for generalization and task-specific semantic expertise. The supervised module-selection approaches rely on a relatively small collection of reference datasets, some of which overlap with the final evaluation benchmarks at the dataset level, although different splits are used. This raises concerns about generalization to genuinely unseen tasks and sensitivity to the choice of reference datasets. To what extent do the supervised results depend on the particular reference datasets used for module selection? Could the authors provide a leave-one-dataset-out or cross-dataset evaluation in which the target dataset is entirely excluded from the selection procedure?

We would like to clarify that our probing configuration follows [2], which we retain for a fair comparison. Our proposed CROP and ICT methods rely only on augmented data from public corpora and do not require any downstream-task datasets. They nevertheless outperform VA (LMP), demonstrating the effectiveness of our approach.

To address your concern, we added a probing configuration using ten tasks that are entirely disjoint from the evaluation datasets and report the results on the 14 MTEB subsets. Specifically, we use BBC News and CLINC150 for clustering, AskUbuntuDupQuestions and WikiQA for reranking, BIOSSES and STS22 for STS, MTOPIntentClassification and TweetEval emotion for classification, and CQADupstack Webmasters and FiQA2018 for retrieval. The results on the 14 MTEB task subsets are shown below:

MethodRequires downstream-task data?Dim.Llama3-8BQwen3-8B
VA (LMP) [2] (Original probing configuration)Yes102451.4252.06
TaskEval (Original probing configuration)Yes102457.75 (+6.33)57.58 (+5.52)
VA (LMP) [2] (New probing configuration)Yes102449.2651.86
TaskEval (New probing configuration)Yes102455.07 (+5.81)55.44 (+3.58)
ICT/CROPNo102456.5557.15

The results show that probing with entirely disjoint OOD data reduces performance, but still substantially outperforms VA (LMP), while falling below augmentation-based ICT/CROP. This suggests that when IID task datasets are unavailable, probing with augmented data and InfoNCE generally performs better than probing directly with OOD task datasets.

W2(b) The dynamic-routing methods additionally rely on instruction similarity, without evaluating robustness to instruction reformulations or alternative reference datasets. Their improvements over static TaskEval are generally small, while the proposed rank-wise aggregation of potentially unaligned module representations lacks a clear justification.

Our purpose in introducing dynamic routing is not to outperform static routing, but to present an alternative that adapts automatically when probing or evaluation datasets are added, without requiring people to manually assign the source modules for each target task's embeddings.

W2(c) More fundamentally, the paper's central narrative is that individual attention modules act as task-specific semantic experts, yet the empirical analysis does not convincingly establish this interpretation. Showing that certain modules perform better on particular downstream tasks does not necessarily imply that they encode distinct or specialized semantic information. The observed improvements could instead arise from selecting useful projections of intermediate hidden representations, potentially containing substantial redundancy. Without further analysis of representation geometry, redundancy, or functional specialization, the semantic-expert interpretation remains largely unsubstantiated.

High-quality embeddings are generally understood to compress the semantics of input texts, but semantics itself is difficult to measure concretely. In fact, the term "semantic experts" appears only in the title, primarily to engage readers and convey the connection between our approach and the mixture-of-experts idea. In response to your concern, we will consider discontinuing the use of the potentially ambiguous term "semantic experts."

Q3: How robust are DR-1 and DR-2 to semantically equivalent reformulations of task instructions and changes in the reference-dataset pool? Additionally, what motivates averaging nonshared module representations by rank, rather than combining module scores and selecting the highest-ranked modules?

In the supervised setting, the instructions used for LLM2Vec's TaskEval (DR-1) and TaskEval (DR-2) are officially recommended for that model and are consistent with its training data. For the training-free experiments, we currently use the instructions provided by E5-Mistral. To address your concern, we also evaluated the instructions from Instructor [3]. Below, we illustrate the difference between the two instructions for SciFact and report the corresponding experimental results.

Given a scientific claim, retrieve documents that support or refute the claim  #E5-Mistral Version
Represent a Scientific query for retrieving a supporting passage  # Instructor Version
MethodDim.Llama3-8BQwen3-8B
TaskEval (DR-1) (Original, E5-Mistral Version)102457.4857.23
TaskEval (DR-2) (Original, E5-Mistral Version)102458.5358.01
TaskEval (DR-1) (New, Instructor Version)102455.61 (−1.88)57.00 (-0.23)
TaskEval (DR-2) (New, Instructor Version)102458.29 (−0.24)58.68 (+0.67)

These results show that dynamic routing is generally robust to the choice of instructions. On Qwen3-8B, TaskEval (DR-2) achieves a new state-of-the-art result with the Instructor instructions.

[1] Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions. EMNLP 2024

[2] LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States. ICML 2026

[3] One Embedder, Any Task: Instruction-Finetuned Text Embeddings. ACL 2023 Findings.

Reviewer_ErXMOpenReview ↗

Rebuttal acknowledgement by reviewer ErXM

Thank you for the detailed rebuttal and the clarifying experiments, the effort is well appreciated.

Overall, the additional experiments reinforce my view that the strongest contribution is the training-free selection of useful attention-value modules to produce competitive/compact text embeddings. I find the results with unlabeled CROP/ICT interesting, more so given that they seem to outperform supervised TaskEval when using disjoint reference datasets. I would say this is a valuable contribution in its own right.

My remaining concern is mostly about how the paper positions this contribution. The main narrative places quite a bit of emphasis on task-specific semantic experts and dynamic routing, but the experiments provide stronger evidence for the usefulness of module selection itself. The additional benefits of dynamic routing are relatively small, and I am not entirely convinced that the selected modules necessarily constitute distinct semantic experts. I appreciate the authors' willingness to reconsider this terminology, but the framing goes beyond the title and is also reflected within the manuscript.

I think the paper would be stronger overall if it focused more on training-free module selection and compact embeddings, with dynamic routing presented as a possible extension. The semantic-expert interpretation would benefit from further analysis or better positioning of the work throughout the manuscript. This would also better align the paper with what the experiments establish.

That said, I appreciate the work put into the rebuttal. I will keep my score unchanged for now, but the additional evidence has positively influenced my assessment and will be valuable during the upcoming AC discussion. Thanks again!

Reviewer_m22zOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a training-free text embedding framework. It utilizes unsupervised data augmentation or downstream data to probe the Attn.V modules. It routes and concatenates the top-k modules to generate compact representations. Its effectiveness is validated across 14 MTEB tasks, multilingual retrieval, and model dimensionality reduction.

Critical strengths
  • Refines V-pooling from the full-layer level down to the GQA module level. It can be used as flexible dimensionality control.
  • Compatible with both training-free extraction and post-hoc dimensionality reduction for fine-tuned models.
Critical weaknesses
  • Feels like an ad-hoc engineering assembly; the weighted combination of heterogeneous modules based on their rankings lacks a geometric foundation.
  • Evaluated on only 14 MTEB subsets, with the probing data and test data originating from the same sources.
  • Limited Impact: Underperforms compared to lightweight fine-tuned models. In real-world industrial scenarios, even when an 8B-parameter LLM is used for a single forward pass to extract a 1024-dimensional vector, its inference latency and memory consumption are significantly higher than those of specialized, fine-tuned lightweight embedding models (such as BGE, GTE, or Qwen3-Embedding-0.6B). Furthermore, since its zero-shot (training-free) performance still lags substantially behind that of fine-tuned models, its practical utility for deployment remains limited.
  • Some technical details are vague and code not open-sourced.
Questions for authors
  1. What is the geometric rationale behind the weighted aggregation of modules from different layers and how is the normalization performed?
  2. How does TaskEval perform after excluding data that originates from the same source as the test set?
  3. How are the mini-batch setting and the M=256M=256 configuration?
  4. Will the code be made available?
2 comments

Thank you for your valuable feedback. Our responses are as follows:

W1&Q1 Feels like an ad-hoc engineering assembly; the weighted combination of heterogeneous modules based on their rankings lacks a geometric foundation. What is the geometric rationale behind the weighted aggregation of modules from different layers and how is the normalization performed?

We would like to clarify that features extracted from different modules are concatenated rather than fused. Features from the same module therefore occupy the same dimensions in the final embedding and are used to compute similarity. Accordingly, this operation does not seem to require a geometric foundation. The motivation is simply that different attention head modules encode different semantic features, which we use jointly when computing similarity.

W2&Q2: Evaluated on only 14 MTEB subsets, with the probing data and test data originating from the same sources. How does TaskEval perform after excluding data that originates from the same source as the test set?

We would like to clarify that our probing configuration follows [1], which we retain for a fair comparison. Our proposed CROP and ICT methods rely only on augmented data from public corpora and do not require any downstream-task datasets. They nevertheless outperform VA (LMP), demonstrating the effectiveness of our approach.

To address your concern, we added a probing configuration using ten tasks that are entirely disjoint from the evaluation datasets and report the results on the 14 MTEB subsets. Specifically, we use BBC News and CLINC150 for clustering, AskUbuntuDupQuestions and WikiQA for reranking, BIOSSES and STS22 for STS, MTOPIntentClassification and TweetEval emotion for classification, and CQADupstack Webmasters and FiQA2018 for retrieval. The results on the 14 MTEB task subsets are shown below:

MethodRequires downstream-task data?Dim.Llama3-8BQwen3-8B
VA (LMP) [1] (Original probing configuration)Yes102451.4252.06
TaskEval (Original probing configuration)Yes102457.75 (+6.33)57.58 (+5.52)
VA (LMP) [1] (New probing configuration)Yes102449.2651.86
TaskEval (New probing configuration)Yes102455.07 (+5.81)55.44 (+3.58)
ICT/CROPNo102456.5557.15

The results show that probing with entirely disjoint out-of-distribution (OOD) data reduces performance, but still substantially outperforms VA (LMP), while falling below augmentation-based ICT/CROP. This suggests that when IID task datasets are unavailable, probing with augmented data and InfoNCE generally performs better than probing directly with OOD task datasets. We have incorporated this conclusion into the main text. Computational constraints prevent us from reporting results on the full MTEB benchmark, but we believe that the substantial advantages of our method will extend to other MTEB tasks.

W3 Limited Impact: Underperforms compared to lightweight fine-tuned models. In real-world industrial scenarios, even when an 8B-parameter LLM is used for a single forward pass to extract a 1024-dimensional vector, its inference latency and memory consumption are significantly higher than those of specialized, fine-tuned lightweight embedding models (such as BGE, GTE, or Qwen3-Embedding-0.6B). Furthermore, since its zero-shot (training-free) performance still lags substantially behind that of fine-tuned models, its practical utility for deployment remains limited.

We agree that current training-free methods do not outperform embedders fine-tuned on large-scale datasets, including BERT-sized text encoders. However, obtaining high-quality text embeddings from LLMs or MLLMs without training has long been a research goal in the community [1, 2, 3, 4]. This line of research is motivated by three considerations: (1) training-free methods can produce high-quality text embeddings while preserving an LLM's text-generation capabilities, potentially allowing a single model to support complex workflows such as RAG; (2) these studies help us understand whether LLMs already understand and internally encode the semantics of input texts before being used for embedding tasks; and (3) techniques found useful in training-free settings may further improve trained embedding models.

Our approach primarily applies the mean/last pooling strategies commonly used for hidden-state text embeddings to attention values and introduces task-specific module-selection methods. In principle, these techniques are orthogonal to whether the model is trained and can also benefit trained embedding models. However, this paper already presents several variants and analyzes their robustness and relationships, leaving insufficient space to investigate fine-tuning settings. Instead, our experiments on fine-tuned models, namely LLM2Vec-Llama3-Sup and the newly added GritLM, demonstrate that our approach is effective beyond the training-free setting.

W4&Q4: Some technical details are vague and code not open-sourced. Will the code be made available?

We will release our code upon acceptance. We have also reported the actual attention head modules selected in Table 2 of the appendix, making the results straightforward to reproduce.

Q3: How are the mini-batch setting and the M=256M=256 configuration?

We understand your question as asking whether M=256M=256 is the mini-batch size. To clarify, MM is the number of candidates per query used to compute the contrastive loss: one positive and M−1M-1 negatives. During encoding, GPU memory constraints may prevent us from using a mini-batch size of 256. In such cases, we cache the outputs of multiple encoding mini-batches and compute the InfoNCE loss over the accumulated outputs.

[1] LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States. ICML 2026

[2] Repetition Improves Language Model Embeddings. ICLR 2025

[3] Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free. ICLR 2025

[4] Meta-Task Prompting Elicits Embeddings from Large Language Models. ACL 2024

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.