acceptodds
Under review as a conference paper at ICLR 2027

Lend Me Your Eyes: Instruction-Aware Text Embeddings via Attention Relay

Abstract

Text embedding models trained with contrastive learning learn to follow task instructions from instruction-paired data, while instruction-tuned LLMs already know how to follow them. We show that this instruction-following ability can carry over from an LLM to a Transformer-based embedder without any training. We propose Attention Relay, which passes the attention weights an LLM produces to the embedder's own attention. Across six instruction-tuned LLMs from the Qwen3, Llama 3.1 and OLMo 3 families and ten widely used embedding models that differ in tokenizer, size and pooling type, Attention Relay makes nearly every combination instruction-aware. Experiments that break the method down into its parts show that the LLM's attention weights track the instruction in its later layers and come largely from instruction tuning. They also show that relaying these weights selects which content in the text matters: it makes the aspect of the text that the instruction asks about dominant in the embedding, or restores that aspect where averaging had diluted it.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, large language models, embedding, interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EAdyOpenReview ↗
Rating
2
Confidence
3
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

They introduce Attention Relay, a hybrid method for creating instruction-aware embeddings. LLMs are able to weight the attention of a text given a question but do not give a representation for similarity, while embedders capture the text meaning but cannot weight it for the question. Therefore, they use the weights of the LLMs (given the text + the question) to weight the tokens in the embedding, instead of the classical average pooling over all the tokens. The authors evaluate this method across 6 LLMs from 3 model families (up to 8B parameters) and 10 embedding models. Tested on 6 datasets, the results demonstrate that Attention Relay successfully instills instruction awareness into non-instruction-tuned embedders.

Critical strengths
  1. The core idea of projecting an LLM's instruction-aware attention onto a separate, specialised embedder's representation space is an interesting training-free intervention
  2. The authors rigorously test diverse combinations of embedding models and LLM families, proving the method is highly generalisable and works across almost all tested architectures
Critical weaknesses
  1. Missing Baselines and Scalability Comparisons:
    • The only instruction-aware embedder that has been used is Qwen3-Embedding-0.6B, which outperforms its document-mode counterpart on all datasets but InstructSTSB. It would be interesting to compare the results also against other instruction-aware embedding models with more parameters, because the Attention Relay method uses the attention weights of LLMs up to 8B parameters.
    • In the introduction and Section 2.2, the authors justify not using LLM hidden states directly by claiming that training-free LLM readout methods (such as PromptEOL, Echo embeddings, and Value Aggregation), “remain well behind trained embedders” (line 045). However, they do not report these methods' performance on the same benchmarks as baselines. Attention Relay runs a forward pass on the LLM using a PromptEOL-style prompt and evaluating the LLM's hidden state at that exact answer position is a natural baseline.
  2. Text embeddings are designed to be fast, lightweight, and scalable for large retrieval tasks. Requiring an LLM forward pass to extract attention weights introduces massive computational and latency overhead. While the authors briefly quantify the latency in Appendix C.4 they don’t discuss the practical implications of this trade-off in the main text.
Questions for authors

Clarifications:

  1. Could you provide a direct empirical comparison against LLM-based extraction methods (such as PromptEOL) in your main results to substantiate the claim made in the introduction that they remain "well behind"?
  2. In Appendix C.4, Table 11 shows that utilising Qwen3-1.7B for Attention Relay increases the total inference time for bge-large-en-v1.5 from 0.70 ms to 5.56 ms per text. Given that this is roughly an 8x latency penalty, why is this massive throughput trade-off not discussed in the main text, especially considering embeddings are typically deployed for high-volume retrieval?
  3. Why does Qwen3-Embedding-0.6B in native instruction mode still outperform the document-mode + relay variant on most datasets?
5 comments

Thank you for your thoughtful review! We are encouraged that you found our approach "an interesting training-free intervention" and noted that we "rigorously test diverse combinations of embedding models and LLM families, proving the method is highly generalisable." We address the remaining questions and concerns below.

"…compare the results also against other instruction-aware embedding models with more parameters…"

We want to clarify one point first, which we should have made clearer in the paper. Relaying from a large LLM does not make the embedder larger: the LLM never produces the embedding, it only tells the embedder which tokens to weigh more. So the scaling behavior on LLMs doesn't apply here. That is why a 1.7B and an 8B LLM give about the same result below, and why Table 1 compares one 0.6B embedder with itself. The larger trained embedders do better still, as expected from their instruction-paired contrastive training. We evaluated Qwen3-Embedding-8B, E5-Mistral-7B-instruct and GritLM-7B, each with its documented instruction format and the better of the raw question and the task description:

Model / settingNYTIE
Attention Relay into Qwen3-Embedding-0.6B, no training
from Qwen3-1.7B64.6142.6
from Qwen3-8B63.8148.6
No instruction50–59100.0
Trained with instructions
Qwen3-Embedding-0.6B68.4190.0
Qwen3-Embedding-8B77.6183.2
E5-Mistral-7B-instruct69.5183.5
GritLM-7B73.2172.4

"…claiming that training-free LLM readout methods… 'remain well behind trained embedders' (line 045). However, they do not report these methods' performance…" and "Could you provide a direct empirical comparison against LLM-based extraction methods (such as PromptEOL) in your main results…"

Thank you for asking for this comparison; it turned out to be one of the most informative we ran. We ran the PromptEOL-style baseline you describe (Jiang et al., 2024), taking the LLM's state at the answer position from the same forward pass as the relay. We also ran Value Aggregation (Zhang et al., 2026), the LLM's hidden states pooled by the relay's weights, and the LLM's one-word answer embedded, all with Qwen3-1.7B on all six sets (embedder rows: mean of the five embedders of Table 1):

MethodNYTFewRelFewNerdFewEventIEInstructSTSB
Attention Relay (LLM's reading → embedder)66.857.646.562.9140.715.1
LLM's one-word answer, embedded34.133.839.946.9138.225.5
LLM's hidden states pooled by the relay's weights48.357.651.460.4159.630.2
LLM's state at the answer position (PromptEOL-style)46.656.958.559.6155.628.8
Value Aggregation (AlignedWVA, Zhang et al., 2026; question first)51.970.867.572.4155.221.5
Trained reference: Qwen3-Embedding-0.6B (instruction mode, Table 1)68.463.458.167.3190.0−1.7

Trained embedders usually lead, as they do on task-agnostic similarity (Forooghi et al., 2024; BehnamGhader et al., 2024), but we find that which signal wins depends on what the question asks for:

  • Where the LLM's answer is not the label, the embedder's space wins. Asked where an article about Italy's budget happened, the LLM answers "New York", where the minister gave an interview, while the label is Italy. On NYT the relay comes close to the trained embedder, and every readout of the LLM's own states trails far behind.
  • Where the answer is the label, the LLM's own space tends to win. On entity and relation types (FewRel, FewNerd, FewEvent), Value Aggregation with the question first even leads the trained embedder.
  • A short answer drops the evidence. Passing the LLM's reading beats passing its one-word answer on all four clustering sets.
  • Instruction training is not always enough. On InstructSTSB the trained embedder fails, and every training-free method does better.

We have revised line 045 to say that trained embedders usually, not always, outperform these readouts.

"…introduces massive computational and latency overhead…" and "…why is this massive throughput trade-off not discussed in the main text…"

Thank you for pressing us on cost! We measured it again, and found the cost lower than the submission's Table 11 suggests. Its 4.58 ms is a full pass through all 28 blocks of Qwen3-1.7B. We read block 24 and never generate a token, so the pass can stop there; and because the text precedes the question, we compute its cache once and reuse it for every question asked of it. Timed this way on the same H100, in ms per text and question, with the total over the embedder alone in parentheses:

EmbedderAlone+ LLM = total, 1 question per text+ LLM = total, 10 questions per text
bge-large-en-v1.50.684.29 (6.3×)1.95 (2.9×)
Qwen3-Embedding-0.6B2.275.93 (2.6×)3.59 (1.6×)
Qwen3-Embedding-8B11.3615.30 (1.3×)12.96 (1.1×)

The overhead is largest for a small embedder asked one question per text and small for large embedders, and a small LLM suffices: Qwen3-4B or 8B does not necessarily relay better than Qwen3-1.7B (Figure 5). We also see a clear path to removing the LLM from inference entirely: a light module on the frozen embedder, trained to predict the LLM's weights. We have added a discussion on this overhead to Section 6, and the full timing to Appendix C.5.

"Why does Qwen3-Embedding-0.6B in native instruction mode still outperform the document-mode + relay variant on most datasets?"

We would like to clarify that our goal is not to compete with fully developed SOTA embedding systems such as Qwen3-Embedding, which are trained on large amounts of instruction-paired data. Our goal is narrower: to bring instruction following to any embedder without such training, and the two combine. We therefore applied the relay on top of instruction mode (now a row of Table 1), and interestingly, the relay helps most where instruction mode is weak:

Qwen3-Embedding-0.6BInstruction mode+ Attention Relay
InstructSTSB (Spearman)−1.716.4
InstructSTSB, same pair under two questions50.1% (chance)66.2%
FewNerd58.162.1
FewEvent67.369.9
NYT / IE / FewRel68.4 / 190.0 / 63.466.2 / 190.2 / 63.1

Thank you again for your thoughtful feedback. We hope our clarification and the new results can address your concerns, and we would be grateful if you would take them into account in your final evaluation. Please let us know if we can clarify anything further.

References

  • BehnamGhader et al., LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders, COLM 2024.
  • Forooghi, Sadeghi, and Lu, Whitening Not Recommended for Classification Tasks in LLMs, RepL4NLP 2024.
  • Jiang et al., Scaling Sentence Embeddings with Large Language Models, Findings of EMNLP 2024.
  • Zhang et al., LLM-Based Embeddings: Attention Values Encode Sentence Semantics Better than Hidden States, ICML 2026.
AuthorsOpenReview ↗

Less than two days left in discussion period

Dear Reviewer EAdy,

Thank you again for your review. As the discussion period closes on Nov. 18 (AoE), we wanted to make sure our rebuttal reached you, including the new experiments in response to your questions. If any questions remain, we are happy to clarify before the discussion closes. Thank you!

Reviewer_TH7VOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Attention Relay, which means using an instruction-tuned LLM’s attention over a text to guide token weighting in a separate embedding model, without additional training. Character alignment allows this transfer across tokenizers. Experiments across six LLMs, ten embedders, and six datasets show improvements over instruction-free inputs, with controls suggesting that the gains depend on instruction-sensitive attention to relevant words.

Critical strengths
  • The method is simple and the experimental controls are useful. The proposed method adds task conditioning without retraining the embedder or learning a mapping between representation spaces and the paper shows this method across different tokenizers and pooling methods.
Critical weaknesses
  1. The practical advantage remains unclear. In Table 1, directly using Qwen3-Embedding in instruction mode beats relaying into its document mode on five of six datasets. Relay also requires an additional LLM pass. Without comparisons against conditional LLM embeddings or extracting relevant content before embedding, I am unsure when this approach is preferable.
  2. All experiments extract attention under the same “Answer in one word” prompt. Since this format may favor selecting words useful for predicting a short label, testing prompt sensitivity would help establish whether the method captures instruction relevance more generally.
Questions for authors
  • Why are base-checkpoint controls included for only four of the six LLMs? In particular, why were Qwen3-0.6B and Qwen3-4B omitted, and could this affect the conclusion that instruction tuning largely accounts for the observed attention sensitivity and relay gains?
3 comments

Thank you for your thoughtful review! We appreciate that you recognize the simplicity of the method and the value of our controls. We address the remaining questions and concerns below.

"The practical advantage remains unclear… I am unsure when this approach is preferable."

We ran new experiments for each of the three parts of this question.

Instruction mode against the relay. Contrastive instruction training wins on single benchmarks because it changes how every token is represented, while the relay only reweights which tokens count. We therefore tested whether the two combine, by relaying on top of Qwen3-Embedding-0.6B's instruction mode (now a row of Table 1). They do: the relay helps where instruction mode is weak and changes little where it is strong:

Qwen3-Embedding-0.6BInstruction mode+ Attention Relay
InstructSTSB (Spearman)−1.716.4
InstructSTSB, same pair under two questions50.1% (chance)66.2%
FewNerd58.162.1
FewEvent67.369.9
NYT / IE / FewRel68.4 / 190.0 / 63.466.2 / 190.2 / 63.1

Cost of the extra LLM pass. We read block 24 of Qwen3-1.7B and never generate a token, so the pass can stop there, and we share the text's cache across every question asked of it. On one H100, Qwen3-1.7B then costs 3.33 ms per text and question with one question per text and 0.99 ms with ten, against 2.27 ms for Qwen3-Embedding-0.6B alone. We also see a clear path to removing the pass altogether: a light module trained to predict the LLM's weights.

Conditional LLM embeddings and extract-then-embed. We ran both, with Qwen3-1.7B on all six sets. An LLM can pass on its reading (where it looks) or its conclusion (what it answers), into the embedder's space or into its own, so we compared all four. Embedder rows are the mean of the five embedders of Table 1:

MethodNYTFewRelFewNerdFewEventIEInstructSTSB
Reading → embedder (Attention Relay)66.857.646.562.9140.715.1
Conclusion → embedder (the LLM's answer, embedded)34.133.839.946.9138.225.5
Reading, in the LLM's space (its states pooled by the relay's weights)48.357.651.460.4159.630.2
Conclusion, in the LLM's space (answer-position state, PromptEOL-style)46.656.958.559.6155.628.8
Value Aggregation (AlignedWVA, Zhang et al., 2026; question first)51.970.867.572.4155.221.5
Trained reference: Qwen3-Embedding-0.6B (instruction mode, Table 1)68.463.458.167.3190.0−1.7

Passing the reading beats extract-then-embed on all four clustering sets: a short answer drops the evidence and keeps only the LLM's interpretation of it. Which space wins depends on the label. The embedder's space wins where the label is not what the LLM would answer: asked where an article about Italy's budget happened, the LLM answers "New York", where the minister gave an interview, while the label is Italy. The LLM's own space wins where the label is what it would answer, as with entity types.

We therefore see three cases where the relay is preferable: (1) for embedders trained without instructions; (2) on top of an instruction-trained embedder where its instruction following is weak, as on InstructSTSB; (3) when the instruction should select evidence in the text rather than replace the text with the LLM's answer.

"All experiments extract attention under the same 'Answer in one word' prompt… testing prompt sensitivity would help establish whether the method captures instruction relevance more generally."

Thank you for this suggestion! We read Qwen3-1.7B under four prompts. The relay does not depend on the wording we tested: paraphrased questions and task descriptions keep the whole gain. It does depend on the answer being a single word. Gains are averaged over the five embedders of Table 1:

Prompt used for the readingNYT gainIE gainTexts whose reading stays closer to the same question
"Answer in one word." (paper)+10.7+40.7–
Paraphrased questions+11.2+35.899.5–99.7%
Task descriptions instead of questions+11.0+42.598.4–99.9%
"Answer in one sentence."−1.9+18.32–43%

The last column counts, over the four NYT and IE questions, how often a text's reading under the new prompt stays closer (in total variation) to the paper prompt's reading under the same question than under the other one. When we ask for a sentence instead, the LLM reads a text much more alike under the two questions (cosine 0.70–0.77, against 0.38–0.49 for one word). We think the reason is that to commit to one word, the LLM must settle at the answer position on the aspect asked about, while a sentence lets it defer. PromptEOL (Jiang et al., 2024) used the same one-word constraint on the output side, for the LLM's own state; the relay reads the input side of the same compression.

We also checked that the format does not restrict the relay to simple label words: on IE intent, the LLM's one-word answers pass only 26% of triplets, yet the relay passes 81% of the triplets they fail.

"Why are base-checkpoint controls included for only four of the six LLMs? In particular, why were Qwen3-0.6B and Qwen3-4B omitted…"

Thank you for pointing this out! We have now run both, each read in its instruction-tuned sibling's chat template and at the sibling's block, so instruction tuning is the only difference:

LLMQuestion sensitivity (NYT / IE)Relay gain on IE
Qwen3-0.6B0.41 / 0.26+16.4
Qwen3-0.6B-Base0.06 / 0.05+3.6
Qwen3-4B0.59 / 0.57+46.8
Qwen3-4B-Base0.07 / 0.06+5.3

(Relay gains are the mean of the five embedders of Table 1.) Both new pairs agree with the four in the submission: without instruction tuning, the LLM's attention barely depends on the question, which leaves little for the relay to carry. We have added both pairs to Figures 3 and 6 and to Appendix C.2.

Thank you again for the thoughtful review. We hope the new comparisons, prompt variants and base checkpoints make the practical case clear, and we would be grateful if you would take them into account in your final evaluation. Please let us know if we can clarify anything else.

References

  • Jiang et al., Scaling Sentence Embeddings with Large Language Models, Findings of EMNLP 2024.
  • Zhang et al., LLM-Based Embeddings: Attention Values Encode Sentence Semantics Better than Hidden States, ICML 2026.
AuthorsOpenReview ↗

Less than two days left in discussion period

Dear Reviewer TH7V,

Thank you again for your review. As the discussion period closes on Nov. 18 (AoE), we wanted to make sure our rebuttal reached you, including the new experiments in response to your questions. If any questions remain, we are happy to clarify before the discussion closes. Thank you!

Reviewer_BXpoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a method to enable ordinary text embedders to acquire instruction-following capabilities by obtaining the attention distributions for the same text under the same instructions from an LLM fine-tuned on those instructions, and then transferring those attention distributions to the target text embedder. The method was specifically adapted for different pooling methods. In experiments involving cross-combinations of 6 large models and 10 embedders, the method improved the embeddings’ ability to follow instructions in the vast majority of combinations.

Critical strengths
  1. The Attention Relay approach proposed in the paper is interesting and requires no training.
  2. The experimental design is very thorough, covering various large models, text embedders, and pooling methods.
  3. The paper’s overall logic is coherent and easy to understand.
Critical weaknesses
  1. The use of additional instruction-tuned LLMs results in excessively high forward computation costs, making the practicality of this method and its deployment in large-scale retrieval scenarios unclear.
  2. The performance varies significantly across different combinations of instruction-tuned LLMs and embedders. Furthermore, a more powerful LLM does not necessarily result in better transfer performance, and there is no evidence that LLMs and embedders from the same family will perform better. Therefore, in practical scenarios, it may not be possible to select a suitable LLM for a given embedder, further limiting the method’s practicality.
  3. While the main text of the paper makes the method appear simple, it actually requires selecting the attention layers within the instruction-tuned LLM and determining the weight β\beta. The authors currently use empirical formulas and fixed values to simplify the process, but detailed discussion of these aspects is lacking.
  4. There is a lack of end-to-end analysis of successful and failed samples to demonstrate that the instruction-following capability of the embedders obtained through the proposed method is interpretable.
Questions for authors
  1. Could the authors further clarify which parts of their method are existing and which are novel? I feel that most of the techniques described could be applied in more general scenarios.
  2. For LLM-based embedders that do not use instruction-following data for training—such as SGPT [1], RepLLaMA [2], and LLM2Vec-Unsup [3]—which approach is more effective: directly concatenating the instruction to the input or using an attention relay?
  3. Is it possible for instruction-tuned LLMs to further enhance the performance of instruction-following embedders?

[1] SGPT: GPT Sentence Embeddings for Semantic Search. arXiv:2202.08904

[2] Fine-Tuning LLaMA for Multi-Stage Text Retrieval. arXiv:2310.08319

[3] LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders. COLM 2024

4 comments

Thank you for your thoughtful review! We appreciate your positive assessment of the paper's clarity and your observations that our experimental design is "very thorough, covering various large models, text embedders, and pooling methods" and that the “overall logic is coherent and easy to understand.” We address the remaining questions and concerns below.

"…results in excessively high forward computation costs…"

Thank you for pressing us on cost; it led us to a better measurement. Table 11 of the submission timed a full pass through all 28 blocks of Qwen3-1.7B, and we found two ways to lower it. We read block 24 and never generate a token, so the pass can stop there. And because the text precedes the question (Section 3.1), we compute the text's cache once and reuse it for every question asked of it. Timed this way on the same H100, in ms per text and question, with the total over the embedder alone in parentheses (Qwen3-Embedding in document mode):

EmbedderAloneRelayed+ LLM = total, 1 question per text+ LLM = total, 10 questions per text
bge-large-en-v1.50.680.96+ 3.33 = 4.29 (6.3×)+ 0.99 = 1.95 (2.9×)
Qwen3-Embedding-0.6B2.272.60+ 3.33 = 5.93 (2.6×)+ 0.99 = 3.59 (1.6×)
Qwen3-Embedding-4B7.488.09+ 3.33 = 11.42 (1.5×)+ 0.99 = 9.08 (1.2×)
Qwen3-Embedding-8B11.3611.97+ 3.33 = 15.30 (1.3×)+ 0.99 = 12.96 (1.1×)

We also find that a small LLM suffices: the LLM only selects tokens (Section 6), and Qwen3-4B or 8B does not necessarily relay better than Qwen3-1.7B (Figure 5). Instruction-trained embedders pay at training time and the relay pays at inference time, but the idea also has a trained form, which we are excited about: a light module on the frozen embedder, trained to predict the LLM's weights with the LLM as its only teacher, would remove the LLM from inference altogether. We have added a short paragraph on this overhead at the start of the Discussion, and the full timing to Appendix C.5.

"The performance varies significantly across different combinations… it may not be possible to select a suitable LLM…"

Thank you for pointing this out! We split the variance of the gain (relayed minus alone) over the 6 instruction-tuned LLMs × 10 embedders, and found that it comes almost entirely from the LLM and the embedder separately, not from how they are paired:

Share of variance in gainNYTIE
LLM33%57%
Embedder52%38%
LLM × embedder (interaction)14%5%

Because the effects are nearly additive, the LLMs rank the same on every embedder (Spearman against the mean of the other nine embedders: 1.00 on IE, 0.81 on NYT). So a practitioner can choose one LLM once: using one LLM for all ten embedders, rather than the best one for each, loses nothing on IE and 0.5 V-measure points on average on NYT (at most 2.2). Same-family pairs do no better, so no family matching is needed either. We agree that parameter count alone does not govern transfer. What matters is whether the LLM's attention follows the question: base checkpoints of the same size move their attention a fifth to a tenth as much with the question, and gain little.

"…requires selecting the attention layers within the instruction-tuned LLM and determining the weight β\beta… detailed discussion of these aspects is lacking."

We agree these choices deserve more discussion. Neither is tuned on labels: we choose the block with one label-free rule per LLM, and we fix β\beta = 1 for every LLM, embedder and dataset.

Source block. Our rule picks the upper-half block where the LLM's attention over a text moves most between two questions. We apply it once per LLM, and it transfers across datasets: a block chosen from NYT alone or IE alone keeps on average 94% of the other dataset's gain, with a positive gain in all 24 cases. On the four sets we never used to choose it (FewRel, FewNerd, FewEvent, InstructSTSB), upper-half blocks give a gain in 306 of 344 cases, against 136 of 368 in the lower half. The rule only needs to find the instruction-sensitive region, and that region is broad.

β\beta. β\beta sharpens (β\beta > 1) or flattens (β\beta < 1) the LLM's weights before they are relayed. We swept it with Qwen3-1.7B (mean of the five embedders of Table 1):

β\beta0.51.01.52.03.0
NYT gain+9.1+10.7+7.5+4.1−0.5
NYT: drop on the aspect not asked about6.211.916.119.823.2
IE gain+20.7+40.7+51.6+56.7+61.7

β\beta trades following the question against keeping the rest of the text. NYT scores both aspects and peaks at β\beta = 1, the LLM's weights as they are. IE keeps rising because it rewards following the question and cannot register what is lost, so a sharper β\beta raises IE while embedding the text worse. We therefore keep β\beta = 1.

"There is a lack of end-to-end analysis of successful and failed samples…"

Thank you for this suggestion! We cross-tabulated the relay's successes and failures on IE against the LLM's own one-word answers. The LLM passes a triplet when it gives the anchor and the positive the same answer and the negative a different one; the relay passes when the anchor is closer to the positive than to the negative (mean of the five embedders of Table 1):

AspectLLM's answers passRelay passes where the answers passRelay passes where the answers fail
Emotion66%65%41%
Intent26%90%81%

On emotion, the relay fails mostly where the LLM itself misreads the text, so we can trace its errors back to the LLM's reading. On intent, one word cannot name an open-ended intent (530 distinct answers across 5,241 messages), yet the relay still passes most of the triplets the answer fails: the LLM attends to the right evidence even when one token cannot state it. Figure 9 of the paper adds a complementary breakdown on NYT location: bge-large-en-v1.5 gains 27.2 V-measure points on the 1,749 articles that name the country and 4.8 on the 395 that do not.

"Could the authors further clarify which parts of their method are existing and which are novel?"

We agree that the components are standard: attention weights, alignment through character offsets, weighted pooling. Others already read a model's question-conditioned attention, to rank documents within the same model (ICR; Chen et al., 2025) or to drop tokens before another model generates (Liu et al., 2025; Upasani et al., 2026). To the best of our knowledge, what is new is the whole as a solution for instruction-following embeddings: an instruction-tuned LLM's attention makes an off-the-shelf embedder follow instructions, across tokenizers, with no training and without altering the embedder's representations. We believe that the mechanism can reach beyond text, for example by relaying a vision-language model's attention over image patches into a frozen image encoder, as we discussed in Section 6.

"For LLM-based embedders that do not use instruction-following data for training… which approach is more effective…"

Thank you for this question! We ran all three embedders you mention: LLM2Vec-Unsup (Llama-3.1-8B, MNTP and unsupervised SimCSE, mean pooling), SGPT-5.8B (GPT-J, NLI, position-weighted mean pooling) and RepLLaMA (Llama-2-7B, MS MARCO, last-token pooling). We first checked that our implementation reproduces each released model's embeddings (cosine ≥ 0.999 on 32 texts), then relayed Qwen3-1.7B's weights with the same rules as in Table 1:

EmbedderConditionNYTIE
LLM2Vec-UnsupNo instruction54.9100.0
Instruction in input56.2106.5
Attention Relay59.4157.8
SGPT-5.8BNo instruction39.0100.0
Instruction in input40.3103.3
Attention Relay54.1149.9
RepLLaMANo instruction57.1100.0
Instruction in input57.8103.9
Attention Relay62.2168.8

As one would expect from embedders not trained with instructions, the instruction in the input moves them only slightly. With the relay, all three follow the question much more closely. (SGPT reads at most 75 tokens, as released, hence its lower NYT.)

"Is it possible for instruction-tuned LLMs to further enhance the performance of instruction-following embedders?"

Yes, where the embedder's own instruction following is weak. We relayed Qwen3-1.7B's weights into Qwen3-Embedding-0.6B while it reads the instruction in its instruction mode (now a row of Table 1):

Qwen3-Embedding-0.6BInstruction mode+ Attention Relay
InstructSTSB (Spearman)−1.716.4
InstructSTSB, same pair under two questions50.1% (chance)66.2%
FewNerd58.162.1
FewEvent67.369.9
FewRel63.463.1
NYT68.466.2
IE190.0190.2

The same-pair row takes each InstructSTSB pair under its two questions and counts how often the pair is more similar under the question on which its two sentences agree. Instruction mode is at chance there, and the relay is well above it. Where instruction mode is already strong (NYT, IE, FewRel), the relay changes little. We see the two as complementary: training shapes the embedding space, and the relay selects, text by text, which tokens count.

Thank you again for the thoughtful review. We hope these new analyses resolve your concerns, and we would be grateful if you would take them into account in your final evaluation. Please let us know if we can clarify anything else.

References

  • Chen, Jiménez Gutiérrez, and Su, Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers, ICLR 2025.
  • Liu, Chen, and Zhang, Speculative Prefill: Turbocharging TTFT with Lightweight and Training-Free Token Importance Estimation, ICML 2025.
  • Upasani et al., Cross-Family Speculative Prefill: Training-Free Long-Context Compression with Small Draft Models, ICLR 2026 Workshop on AI with Recursive Self-Improvement.
  • Wang et al., MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers, NeurIPS 2020.
AuthorsOpenReview ↗

Less than two days left in discussion period

Dear Reviewer BXpo,

Thank you again for your review. As the discussion period closes on Nov. 18 (AoE), we wanted to make sure our rebuttal reached you, including the new experiments in response to your questions. If any questions remain, we are happy to clarify before the discussion closes. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.