- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces a training-free framework that probes and ranks individual attention head modules to construct compact task-specific embeddings from their attention values. Using augmented text pairs or downstream reference data, the framework performs offline probing and task-specific routing while requiring only one frozen-model forward pass per text.
Critical strengths
- The paper is clearly written and well organized.
- The proposed method is simple and can be applied without additional training.
- The paper provides extensive experimental evaluations.
Critical weaknesses
- Limited evaluation on dedicated embedding models. The paper focuses on task-specific text embeddings, but most experiments are conducted on general-purpose generative LLMs rather than models specifically trained for embedding tasks. While this is consistent with the training-free setting, it remains unclear how relevant the findings are to existing embedding models.
- Marginal improvements on trained embedding models. On LLM2Vec, the improvements over the native embeddings are only 0.15–0.27 points, which are too small to demonstrate a meaningful improvement in embedding quality. The results seem more convincing as evidence of dimensionality reduction than of better task-specific representations.
- Questionable necessity of dynamic routing. DR-1 performs worse than static selection, while DR-2 brings only marginal improvements of 0.17–0.45 points. This raises doubts about whether dynamic routing is necessary, especially given the additional complexity of expert selection and combination.
- Unclear advantage over instruction-based adaptation. The motivation that different tasks require different embedding representations is already well established. Existing instruction-tuned embedding models can adapt their representations through task instructions. Although the proposed approach avoids additional training, the paper does not clearly establish what head-level selection offers beyond existing instruction-based adaptation.
Questions for authors
- Would the proposed method still provide meaningful improvements on modern instruction-tuned embedding models?
- Given the marginal gains on LLM2Vec, should the main contribution be viewed as embedding compression rather than improved task-specific representation quality?
- What is the actual benefit of dynamic routing over static selection or simple expert averaging? Could the authors isolate the contribution of the routing mechanism itself?
- If an instruction-tuned embedding model can already produce different representations by changing task instructions, what additional capability does task-specific head selection provide?
4 comments
W4&Q4: Unclear advantage over instruction-based adaptation. The motivation that different tasks require different embedding representations is already well established. Existing instruction-tuned embedding models can adapt their representations through task instructions. Although the proposed approach avoids additional training, the paper does not clearly establish what head-level selection offers beyond existing instruction-based adaptation. If an instruction-tuned embedding model can already produce different representations by changing task instructions, what additional capability does task-specific head selection provide?
As discussed in the Introduction, existing instruction-following embedders primarily produce different embeddings by prepending an instruction to the input text:
Our approach takes a different route by changing the source modules from which text embeddings are extracted for different tasks, effectively providing a different mapping function for each task:
On the one hand, these two approaches can serve as alternatives. One potential advantage of our approach is that routing can be determined using either a fixed policy (static routing) or similarity between instructions (dynamic routing), without prepending an instruction to each text before encoding. Prepending instructions can introduce substantial additional overhead even when a KV cache is used.
On the other hand, the two approaches can also be combined:
This is because prepending instructions and selecting task-specific source modules for embeddings are orthogonal choices. LLM2Vec, which we use in our experiments, is one such example: it is trained on the instruction-augmented E5 public dataset and is itself an instruction-following embedder. We use instructions during both probing and embedding extraction for LLM2Vec. As shown in Table 8 of the appendix, downstream performance decreases when instructions are omitted during probing and evaluation.
[1] LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States. ICML 2026
[2] Repetition Improves Language Model Embeddings. ICLR 2025
[3] Your Mixture-of-Experts LLM Is Secretly an Embedding Model for Free. ICLR 2025
[4] Meta-Task Prompting Elicits Embeddings from Large Language Models. ACL 2024
[5] Contrastive Prompting Enhances Sentence Embeddings in LLMs through Inference-Time Steering. ACL 2025
[6] PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval. EMNLP 2024
Thank you for the response and clarifications. While I appreciate the authors' explanations regarding the scope and motivation of the proposed method, my core concerns remain insufficiently addressed:
- Limited Evaluation on Trained Embedding Models: The evaluation remains limited to LLM2Vec, without additional experiments on other modern embedding models, such as Qwen3-Embedding or E5-Mistral.
- Benefits Beyond Instruction-Based Adaptation: Combining head selection with instructions on LLM2Vec yields only marginal improvements over its native embeddings. Moreover, the claim that "Prepending instructions can introduce substantial additional overhead even when a KV cache is used" lacks empirical support. Since task instructions are typically short and shared instruction prefixes can be cached and reused across inputs in causal LLMs, quantitative latency comparisons are needed to substantiate the claim of substantial additional overhead.
Therefore, I keep my original score.
Thank you for your valuable feedback. Our responses are as follows:
W1: Limited Evaluation on Trained Embedding Models: The evaluation remains limited to LLM2Vec, without additional experiments on other modern embedding models, such as Qwen3-Embedding or E5-Mistral.
Thank you for your comment. We initially understood your concern to be primarily about the focus of our paper, so we did not include the results for the additional embedding model in our response to your review. In fact, in addition to LLM2Vec, which is based on Llama3-8B, we have also added experiments on GritLM, which is based on Mistral-7B. The results are as follows:
| Method | Dim.=256 | Dim.=512 | Dim.=1024 | Dim.=2048 |
|---|---|---|---|---|
| TaskEval (DR-2) | 66.04 | 67.15 | 67.54 | 68.14 |
| TaskEval | 66.06 | 66.80 | 67.60 | 67.86 |
| PCA | 63.93 | 65.77 | 66.36 | 65.93 |
| Gaussian Random Projection | 64.05 ± 0.23 | 66.03 ± 0.05 | 66.97 ± 0.01 | 67.42 ± 0.10 |
| Whitening | 64.31 | 64.93 | 63.55 | 61.73 |
| Matryoshka-Adapter | 64.05 | 65.94 | 66.88 | 67.41 |
Random projection, whitening, and Matryoshka-Adapter [1] are newly added baselines. The experiments show that Matryoshka-Adapter, which requires additional training and a projection head, performs best among the baselines. TaskEval and TaskEval (DR-2) achieve the best results across all dimensionality-reduction settings on GritLM, consistent with the results on LLM2Vec, demonstrating the competitiveness of our approach.
W2: Benefits Beyond Instruction-Based Adaptation: Combining head selection with instructions on LLM2Vec yields only marginal improvements over its native embeddings. Moreover, the claim that "Prepending instructions can introduce substantial additional overhead even when a KV cache is used" lacks empirical support. Since task instructions are typically short and shared instruction prefixes can be cached and reused across inputs in causal LLMs, quantitative latency comparisons are needed to substantiate the claim of substantial additional overhead.
Thank you for your comment. We do not claim in the paper that current embedders should discard instruction prefixes during training and instead route different tasks to different attention head modules. Our experience with LLM2Vec and GritLM suggests that the two approaches are complementary. We therefore do not consider a comparison with instruction-only methods central to this paper, and we do not believe that its absence should be regarded as a major weakness.
Due to time constraints, we may be unable to provide a quantitative latency comparison. We first provide a theoretical analysis here: concatenating an instruction of length with an input text of length results in an online inference time complexity of . With a KV cache, this is reduced to . Our routing method encodes the instruction and the input text separately. Since instruction encoding and routing can both be performed in advance, the online inference time complexity is . Moreover, in practice, attention head modules in the last few layers are typically not selected. Text embeddings can therefore be obtained without running the input text through the final layer, which can further reduce encoding time.
[1] Matryoshka-Adaptor: Unsupervised and Supervised Tuning for Smaller Embedding Dimensions. EMNLP 2024
Thank you for your valuable feedback. Our responses are as follows:
This paper primarily focuses on the training-free setting, with an initial exploration of the post-processing setting. We respectfully disagree that evaluation on trained embedding models is required. A substantial body of work studies how to obtain high-quality text embeddings from LLMs without training [1, 2, 3, 4, 5, 6], and our paper follows the same research focus.
LLM2Vec pools the hidden states from its final layer to obtain text embeddings and optimizes these embeddings through contrastive learning during training. The fact that embeddings extracted from attention values can match the performance of its native embeddings is itself an interesting finding. From a practical perspective, we agree that a direct application of our findings is to obtain high-quality, low-dimensional text embeddings without additional training.
Dynamic routing need not outperform static routing, since the latter incorporates human prior knowledge. Our purpose in introducing dynamic routing is not to outperform static routing, but to present an automatic routing approach that adapts when probing or evaluation datasets are added, without requiring people to manually assign the source modules for each target task's embeddings.