- Rating
- 2
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
They introduce Attention Relay, a hybrid method for creating instruction-aware embeddings. LLMs are able to weight the attention of a text given a question but do not give a representation for similarity, while embedders capture the text meaning but cannot weight it for the question. Therefore, they use the weights of the LLMs (given the text + the question) to weight the tokens in the embedding, instead of the classical average pooling over all the tokens. The authors evaluate this method across 6 LLMs from 3 model families (up to 8B parameters) and 10 embedding models. Tested on 6 datasets, the results demonstrate that Attention Relay successfully instills instruction awareness into non-instruction-tuned embedders.
Critical strengths
- The core idea of projecting an LLM's instruction-aware attention onto a separate, specialised embedder's representation space is an interesting training-free intervention
- The authors rigorously test diverse combinations of embedding models and LLM families, proving the method is highly generalisable and works across almost all tested architectures
Critical weaknesses
- Missing Baselines and Scalability Comparisons:
- The only instruction-aware embedder that has been used is Qwen3-Embedding-0.6B, which outperforms its document-mode counterpart on all datasets but InstructSTSB. It would be interesting to compare the results also against other instruction-aware embedding models with more parameters, because the Attention Relay method uses the attention weights of LLMs up to 8B parameters.
- In the introduction and Section 2.2, the authors justify not using LLM hidden states directly by claiming that training-free LLM readout methods (such as PromptEOL, Echo embeddings, and Value Aggregation), “remain well behind trained embedders” (line 045). However, they do not report these methods' performance on the same benchmarks as baselines. Attention Relay runs a forward pass on the LLM using a PromptEOL-style prompt and evaluating the LLM's hidden state at that exact answer position is a natural baseline.
- Text embeddings are designed to be fast, lightweight, and scalable for large retrieval tasks. Requiring an LLM forward pass to extract attention weights introduces massive computational and latency overhead. While the authors briefly quantify the latency in Appendix C.4 they don’t discuss the practical implications of this trade-off in the main text.
Questions for authors
Clarifications:
- Could you provide a direct empirical comparison against LLM-based extraction methods (such as PromptEOL) in your main results to substantiate the claim made in the introduction that they remain "well behind"?
- In Appendix C.4, Table 11 shows that utilising Qwen3-1.7B for Attention Relay increases the total inference time for bge-large-en-v1.5 from 0.70 ms to 5.56 ms per text. Given that this is roughly an 8x latency penalty, why is this massive throughput trade-off not discussed in the main text, especially considering embeddings are typically deployed for high-volume retrieval?
- Why does Qwen3-Embedding-0.6B in native instruction mode still outperform the document-mode + relay variant on most datasets?
5 comments
"…introduces massive computational and latency overhead…" and "…why is this massive throughput trade-off not discussed in the main text…"
Thank you for pressing us on cost! We measured it again, and found the cost lower than the submission's Table 11 suggests. Its 4.58 ms is a full pass through all 28 blocks of Qwen3-1.7B. We read block 24 and never generate a token, so the pass can stop there; and because the text precedes the question, we compute its cache once and reuse it for every question asked of it. Timed this way on the same H100, in ms per text and question, with the total over the embedder alone in parentheses:
| Embedder | Alone | + LLM = total, 1 question per text | + LLM = total, 10 questions per text |
|---|---|---|---|
| bge-large-en-v1.5 | 0.68 | 4.29 (6.3×) | 1.95 (2.9×) |
| Qwen3-Embedding-0.6B | 2.27 | 5.93 (2.6×) | 3.59 (1.6×) |
| Qwen3-Embedding-8B | 11.36 | 15.30 (1.3×) | 12.96 (1.1×) |
The overhead is largest for a small embedder asked one question per text and small for large embedders, and a small LLM suffices: Qwen3-4B or 8B does not necessarily relay better than Qwen3-1.7B (Figure 5). We also see a clear path to removing the LLM from inference entirely: a light module on the frozen embedder, trained to predict the LLM's weights. We have added a discussion on this overhead to Section 6, and the full timing to Appendix C.5.
"Why does Qwen3-Embedding-0.6B in native instruction mode still outperform the document-mode + relay variant on most datasets?"
We would like to clarify that our goal is not to compete with fully developed SOTA embedding systems such as Qwen3-Embedding, which are trained on large amounts of instruction-paired data. Our goal is narrower: to bring instruction following to any embedder without such training, and the two combine. We therefore applied the relay on top of instruction mode (now a row of Table 1), and interestingly, the relay helps most where instruction mode is weak:
| Qwen3-Embedding-0.6B | Instruction mode | + Attention Relay |
|---|---|---|
| InstructSTSB (Spearman) | −1.7 | 16.4 |
| InstructSTSB, same pair under two questions | 50.1% (chance) | 66.2% |
| FewNerd | 58.1 | 62.1 |
| FewEvent | 67.3 | 69.9 |
| NYT / IE / FewRel | 68.4 / 190.0 / 63.4 | 66.2 / 190.2 / 63.1 |
Thank you again for your thoughtful feedback. We hope our clarification and the new results can address your concerns, and we would be grateful if you would take them into account in your final evaluation. Please let us know if we can clarify anything further.
References
- BehnamGhader et al., LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders, COLM 2024.
- Forooghi, Sadeghi, and Lu, Whitening Not Recommended for Classification Tasks in LLMs, RepL4NLP 2024.
- Jiang et al., Scaling Sentence Embeddings with Large Language Models, Findings of EMNLP 2024.
- Zhang et al., LLM-Based Embeddings: Attention Values Encode Sentence Semantics Better than Hidden States, ICML 2026.
Less than two days left in discussion period
Dear Reviewer EAdy,
Thank you again for your review. As the discussion period closes on Nov. 18 (AoE), we wanted to make sure our rebuttal reached you, including the new experiments in response to your questions. If any questions remain, we are happy to clarify before the discussion closes. Thank you!
Thank you for your thoughtful review! We are encouraged that you found our approach "an interesting training-free intervention" and noted that we "rigorously test diverse combinations of embedding models and LLM families, proving the method is highly generalisable." We address the remaining questions and concerns below.
We want to clarify one point first, which we should have made clearer in the paper. Relaying from a large LLM does not make the embedder larger: the LLM never produces the embedding, it only tells the embedder which tokens to weigh more. So the scaling behavior on LLMs doesn't apply here. That is why a 1.7B and an 8B LLM give about the same result below, and why Table 1 compares one 0.6B embedder with itself. The larger trained embedders do better still, as expected from their instruction-paired contrastive training. We evaluated Qwen3-Embedding-8B, E5-Mistral-7B-instruct and GritLM-7B, each with its documented instruction format and the better of the raw question and the task description:
Thank you for asking for this comparison; it turned out to be one of the most informative we ran. We ran the PromptEOL-style baseline you describe (Jiang et al., 2024), taking the LLM's state at the answer position from the same forward pass as the relay. We also ran Value Aggregation (Zhang et al., 2026), the LLM's hidden states pooled by the relay's weights, and the LLM's one-word answer embedded, all with Qwen3-1.7B on all six sets (embedder rows: mean of the five embedders of Table 1):
Trained embedders usually lead, as they do on task-agnostic similarity (Forooghi et al., 2024; BehnamGhader et al., 2024), but we find that which signal wins depends on what the question asks for:
We have revised line 045 to say that trained embedders usually, not always, outperform these readouts.