acceptodds
Under review as a conference paper at ICLR 2027

ASEMbed: Training-Free Embedding Extraction for Order-Robust Retrieval via Internal LLM Signals

Abstract

Training-free embedding methods reuse frozen decoder-only language models for retrieval without parameter updates. However, query content reordering can alter retrieval rankings and degrade retrieval performance. We observe that local attention concentrations remain visible under reordering, while their strength varies across layers. Based on these observations and prior findings linking attention-sink formation to representational compression, we propose ASEmbed, a training-free framework that uses Attention Sink and Projected Value signals to mitigate retrieval degradation under query content reordering. The framework comprises Boundary and Layer. Guided by these signals, Boundary seeks to uncover intrinsic interaction patterns that persist under reordering and uses them to partition each input into contiguous blocks. AS-Attention enables bidirectional attention across these blocks while retaining direct causal attention within each block. Layer traces the evolution of the same signals across layers to determine when to enable bidirectional cross-block interaction, preserving contextual information in earlier layers. It then selects readout layers from the representations computed with AS-Attention. Extensive experiments on three retrieval benchmarks and two reordering benchmarks demonstrate that ASEmbed achieves average nDCG@10 improvements of 1.96 and 4.28 points over the best-performing baseline on retrieval and reordering retrieval tasks, respectively.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Training-free Embedding, Query Content Reordering, Attention Sink, Retrieval Embedding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_do7tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the robustness of ordering in training-free LLM embeddings. It reveals an important gap that reordering content inside a query will shift retrieval rankings badly. The proposed method, ASEmbed, extracts attention-sink and projected-value signals from first pass, using them to cut input into blocks, and then reruns the model with bidirectional attention across blocks. On opensource retrieval tasks and internal reordering tests, it beats training-free baselines.

Critical strengths
  1. The robustness of ordering is a real but mostly ignored problem, and the controlled reordering test are solid.
  2. The interpretability findings and the following two-pass pipelines are clear stories and well motivated.
Critical weaknesses
  1. No ablations for the core claim that ASPV signals find good block boundaries. The paper doesn't tests random blocks of the same size, which is a cheap but strong ablation for claiming the core design.
  2. The paper doesn't say where the calibration set comes from.
  3. Do corpus documents also need two passes?
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer do7t(Part 1 / 2)

We thank you for your careful and constructive review and for recognizing our work. To address your questions, we have added partitioning comparison experiments and further clarified the experimental settings. Below, we provide detailed responses to all the comments:


W1. No ablations for the core claim that ASPV signals find good block boundaries?

A1: Thank you for your suggestion. We have added two types of Boundary Construction controls to the ablation study. First, we combine window sizes of 64, 128, and 256 tokens with overlap ratios of 25%, 50%, and 75%, evaluating nine sliding-window configurations. Second, we randomly shuffle the order of the ASPV-derived blocks three times to evaluate retrieval performance under random order perturbations.

Table R3. Retrieval performance of different boundary construction strategies.

Partitioning strategyWindow / stride (tokens)Overlap rationDCG@10 (%)MRR (%)
Ours——46.44540.202
Ours shuffling (new)——44.99238.412
AST Function——43.76038.271
PINE——46.15239.959
Sliding Window64 / 640%44.49138.826
Sliding Window64 / 4825%43.55938.031
Sliding Window64 / 3250%42.85237.421
Sliding Window64 / 1675%40.28135.137
Sliding Window128 / 1280%43.95738.434
Sliding Window128 / 9625%43.69338.144
Sliding Window128 / 6450%43.17837.736
Sliding Window128 / 3275%42.00236.588
Sliding Window256 / 2560%43.76838.255
Sliding Window256 / 19225%43.62838.034
Sliding Window256 / 12850%43.38537.778
Sliding Window256 / 6475%42.94937.384

The results show that ASPV achieves the highest nDCG@10 and MRR among all partitioning controls. Compared with the control that randomly shuffles the order of the ASPV-derived blocks, ASPV achieves improvements of 1.453 and 1.790 percentage points, respectively, indicating that random perturbations to block order reduce retrieval performance under the current experimental settings.

For the window-based methods, both metrics decrease as the overlap ratio increases for all three window sizes; increasing window overlap provides no performance gains. These results support the retrieval effectiveness of ASPV-guided input-adaptive partitioning on this task and its superiority over random boundary selection.

AuthorsOpenReview ↗

**Response to Reviewer do7t(Part 2 / 2)**

W2. Where does the calibration set come from, and how many samples are used in the main experiments?

A2: Thank you for pointing out that the calibration setup was not sufficiently described. In the main experiments, we sample 24 unlabeled inputs from the test split of the corresponding task for calibration. Throughout the process, model parameters remain frozen, and the attention switching layer and readout layers remain unchanged.

Figure 6(c) compares calibration sizes of 8, 12, 24, 48, and 96. Both retrieval metrics improve as the number of calibration samples increases, with diminishing gains.


W3. Do corpus documents also need two passes?

A3: Yes. Both queries and corpus documents require two forward passes. The first pass extracts the input's ASPV signals and determines its partition. The second pass applies AS-Attention and extracts representations from the selected readout layers.

When the corpus and configuration are fixed, document embeddings can be computed and cached during offline indexing. Online retrieval requires only two forward passes for each new query, after which the cached document embeddings can be searched without re-encoding the corpus.


We are happy to see from your comments that you have an overall positive view of our work, and we respectfully ask if you'd be comfortable to adjust your ratings, based on all the new experiments and justification to your questions. Thank you once again for your time and effort. We look forward to incorporating these insightful discussions into the final camera-ready version.

Dear Reviewer,

I hope this message finds you well. As the discussion period nears its end, I would like to check whether our responses have satisfactorily addressed your concerns. Please let me know if there are any additional points or feedback you would like us to consider. Your insights are invaluable, and I am eager to address any remaining issues to improve our work. I hope that the additional evidence and clarifications help clarify our contributions and support your overall assessment.

Thank you for your time and effort in reviewing our paper!

Reviewer_ban2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a new attention-based method for producing embeddings from decoder-only LLMs without any training. In particular, the paper uses both attention sinks (tokens that receive disproportionate attention) and corresponding token value projections (in the context of attention). To further improve robustness against context reordering, the paper also proposes a boundary method, using attention sinks to demarcate the boundaries, and a two pass layer method to identify embedding-friendly layers in the model, which are then used to actually produce the embeddings. Evaluation shows that the proposed approach beats out existing work.

Critical strengths
  1. Novel approach with some neat insights into automating the process of using attention sink tokens and in finding good layers to produce embeddings
  2. The claims are validated with a wide range of experiments, checking both standard retrieval quality and robustness to content reordering.
Critical weaknesses

I'm not really a huge fan of this problem. On the surface, it seems that it would be a nice way to take advantage of well-trained decoder models for producing embeddings. However, at this scale it shouldn't be too inexpensive to train a proper encoder or to even finetune one out of a decoder model. How do standard embedding models fare against these approaches?

In addition, because the model is decoder-based, order-based problems are magnified, which isn't as much of an issue for standard embedding models (which are normally based on bidirectional attention)

Questions for authors
  1. How were baseline hyperparameters tuned? e.g. For VA the models reported in that paper were different?
  2. Isn't the first "all receivers" score (in equation 2) for the boundary dependent on position still? Is there any bias as a result of that?
  3. Across how many queries have you done the analysis for in figure 1c?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ban2(Part 1 / 2)

We sincerely thank you for your insightful and thoughtful review. We appreciate your recognition of ASEmbed’s core idea of using internal signals to automate partitioning and layer selection for improved embeddings. Your recognition of these contributions means a great deal to us and encourages us to continue refining this research direction. We have carefully considered your constructive feedback and provide our responses below.

W. How do standard embedding models fare against these approaches?

A1: Thank you for suggesting a direct comparison with specialized embedding models. We agree that off-the-shelf specialized embedding models are an important option in practical applications. ASEmbed investigates how to reuse existing frozen decoder-only LLMs to obtain more effective and robust retrieval representations without updating model parameters or performing embedding-specific training.

Representative embedding models, including SFR-Embedding-Code and Qwen3-Embedding, also use decoder-only LLMs as their backbones and improve retrieval capabilities through specialized training. To provide a clearer comparison, we include two models that use bidirectional attention: CodeSage-Large-v2 (CodeSage), which has an encoder architecture, and SFR-Embedding-Code-2B (SFR), which is obtained through specialized training of a decoder backbone. We report their retrieval performance on the main datasets and further evaluate ASEmbed with a frozen Qwen3-Coder-30B backbone. The following table reports MRR / nDCG@10 (%).

Table R1. Retrieval performance of ASEmbed and specialized embedding models. Each cell reports MRR / nDCG@10 (%).

MethodCoREB Code-to-TextCoREB Text-to-CodeCoREB Code-to-CodeArguAnaNFCorpusSciFact
ASEmbed (original setting)63.60 / 45.4448.82 / 39.4449.65 / 57.0818.57 / 26.5736.67 / 20.9556.10 / 60.01
ASEmbed (30B)73.52 / 53.0451.10 / 44.5953.07 / 63.5628.22 / 34.4339.79 / 21.4758.74 / 62.37
SFR82.08 / 59.4353.39 / 52.2055.07 / 66.8253.75 / 62.6260.12 / 39.4375.82 / 79.92
CodeSage80.88 / 58.2453.10 / 50.3756.73 / 66.4026.72 / 34.5741.49 / 22.6746.01 / 49.66

Results analysis.

(1) There remains a performance gap between ASEmbed and specialized embedding models. The advantage of specialized models may partly stem from their targeted training on relevant domain data and retrieval objectives. For example, SFR is initialized from Gemma-2-2B-IT and undergoes multi-stage training on text and code. Its paper reports 3.36 million training pairs covering 12 programming languages. CodeSage-v2 builds on pretrained weights and performs contrastive learning using text–code pairs filtered from The Stack V2. These procedures involve additional data preparation, filtering, and parameter optimization.

(2) ASEmbed can be transferred to a new frozen backbone without additional embedding-specific training and achieve further improvements. After switching to Qwen3-Coder-30B, both MRR and nDCG@10 improve on all six tasks in Table R1. Averaged equally across these six tasks, the two metrics increase by 5.17 and 5.00 percentage points, respectively. In particular, on SciFact, the 30B configuration achieves an MRR / nDCG@10 of 58.74 / 62.37, exceeding CodeSage’s 46.01 / 49.66.

These results show that ASEmbed can benefit from backbone updates to further improve retrieval performance without repeating embedding-specific training for the new backbone, approaching or surpassing some specialized models on certain tasks. This demonstrates the transferability and potential for further improvement of training-free embedding extraction.

AuthorsOpenReview ↗

Response to Reviewer ban2(Part 2 / 2)

Q1. How were baseline hyperparameters determined, particularly VA’s aggregation layers across different backbones?

A2: Thank you for reminding us to provide further reproducibility details. We configure the hyperparameters according to the settings and search rules in the original baseline papers, and use the same frozen backbone within each task (Section 4.1). The discrepancy between our reported results and those in the VA paper arises from the different backbones: the VA paper evaluates natural-language retrieval using Qwen3-8B, whereas our study uses Qwen3-4B. Therefore, these results do not constitute a direct comparison under the same backbone. In the revised Appendix C.2, we will provide the default parameters, search ranges, and final configurations for each method and backbone.


Q2. Does the cumulative score in Eq. (2) depend on position? How does this affect the results?

A3: Thank you for pointing this out. First, we clarify that we adopt the computation used in prior attention-sink studies, which aggregates the attention received by each source token [1, 2]. Second, we acknowledge that, in the cumulative formulation used in our paper, earlier positions have more terms in the sum and therefore retain positional dependence. To examine this factor, we follow the normalization by the number of visible receiving positions in [1] and divide the cumulative score by n−k+1n-k+1 in the ablation study, giving s~ℓ,kall=sℓ,kall/(n−k+1)\tilde{s}^{\mathrm{all}}_{\ell,k}=s^{\mathrm{all}}_{\ell,k}/(n-k+1) . The results are reported in Table R2.

Table R2. Effect of normalizing the cumulative score by the number of visible receiving positions.

SettingnDCG@10 (%)MRR (%)
Ours46.49640.340
Normalized Version46.18340.064

This supplementary experiment confirms that controlling for the number of visible receiving positions leads to only small changes in both metrics, indicating that ASEmbed is insensitive to positional dependence.

Q3. How many queries are included in Figure 1(c)?

A4: One query in two content orders. Figure 1(c) uses Qwen3-Embedding-8B to compare the attention received by each token before and after reordering the same query. Figure 1(b) summarizes ranking changes across queries, while Table 2 reports retrieval results under controlled reordering.


Thank you again for your time and suggestions. We will incorporate the supplementary experiments and implementation details discussed above into the revised manuscript. If these clarifications address your main concerns, we respectfully ask you to reconsider your assessment in light of the additional results. We would also be happy to provide further clarification on any remaining questions.

References

  1. Stephen Zhang, Mustafa Khan, and Vardan Papyan. 2026. Attention sinks: a 'catch, tag, release' mechanism for embeddings. In Proceedings of the 39th International Conference on Neural Information Processing Systems (NIPS '25), Vol. 38. Curran Associates Inc., Red Hook, NY, USA, Article 2513, 83140–83181. https://doi.org/10.52202/085713-2786
  2. Enrique Queipo-de-Llano, Álvaro Arroyo, Federico Barbero, Xiaowen Dong, Michael Bronstein, Yann LeCun, and Ravid Shwartz-Ziv. 2026. Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin. In Proceedings of the Fourteenth International Conference on Learning Representations (ICLR ’26). https://openreview.net/forum?id=c5TFhCJ6fs

Dear Reviewer,

I hope this message finds you well. As the discussion period nears its end, I would like to check whether our responses have satisfactorily addressed your concerns. Please let me know if there are any additional points or feedback you would like us to consider. Your insights are invaluable, and I am eager to address any remaining issues to improve our work. I hope that the additional evidence and clarifications help clarify our contributions and support your overall assessment.

Thank you for your time and effort in reviewing our paper!

Reviewer_ERbxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ASEmbed uses attention-sink and projected-value signals from a frozen decoder-only LLM to partition input into blocks, enable bidirectional attention across blocks, and select readout layers. The authors report retrieval and query-reordering improvements on code and natural-language benchmarks.

Critical strengths

1, The method targets a practical limitation of frozen decoder embeddings. It seeks to reduce sensitivity to content order without fine-tuning model parameters. 2, The paper evaluates several components separately. Boundary construction, attention switching, and readout-layer selection receive targeted comparisons.

Critical weaknesses

1, ASEmbed’s boundary and layer-selection rules rely on attention-sink and projected-value patterns whose relevance may vary across models and input types. The paper does not establish when these signals yield meaningful partitions or reliable intervention points. 2, The evaluation does not clearly separate robustness to surface-level reordering from robustness to changes in meaning or structure. Since the method enables bidirectional attention across detected blocks, its behavior may depend on whether those boundaries preserve the input’s semantics.

Questions for authors

1, How sensitive are the selected boundaries and intervention layers to the backbone and input type, and when do the internal signals fail to identify useful partitions? 2, Does ASEmbed remain robust when reordering changes meaning or disrupts dependencies across the detected blocks?

4 comments
AuthorsOpenReview ↗

Response to Reviewer ERbx (Part 1 / 3)

We thank the reviewer for recognizing the practical value of extracting retrieval representations from frozen decoder-only LLMs. We believe that the additional experiments and clarifications below address most of your concerns regarding the reliability of the internal signals and the semantics of reordering. We respectfully ask whether you would consider reassessing your rating in light of these results. Below, we provide detailed responses to all the comments:

Q1: How sensitive are the selected boundaries and switching layers to the backbone and input type, and when do the internal signals fail to identify useful partitions?

A1: Thank you for this question. Our additional experiments show stable switching-layer selection within each backbone and broadly consistent major boundaries across backbones for the four input examples. Internal Attention Sinks are harder to detect in very short, unstructured queries, yet ASEmbed retains its overall retrieval advantage on NFCorpus, where short queries predominate.

(1)Backbone sensitivity. The Table 3 reports retrieval results for several dense code backbones. We additionally analyze Granite 3.1 2B and two mixture-of-experts (MoE) models: Granite 3.1 MoE and Qwen3-Coder 30B-A3B. For each of the eight backbones in Table R1, we perform 10 calibration runs, randomly sampling 24 unlabeled examples per run. All 10 runs select the same switching layer for each backbone, supporting the calibration rule’s stability across the evaluated dense and MoE architectures.

(2) Input types and boundary locations. Table R1 presents four examples: a code file, function code, English text, and Chinese text. For the code file, all eight backbones detect cutpoints near 12%, 29%, 57%, and 85%; Mistral and Granite MoE additionally detect a cutpoint near 64%. Across the four examples, ASEmbed identifies broadly consistent major boundaries for the same input, alongside model-specific differences in additional cutpoints. We interpret the shared boundaries as reflecting a common understanding of the input query across models, while the differing cutpoints may arise from differences in their capabilities to interpret the query and identify its internal structure.

Table R1: Attention switching layers and boundary locations across backbones and four input examples. Boundary locations are expressed as percentages of the corresponding input length.

BackboneSwitching layerCode file(%)Function code(%)English text(%)Chinese text(%)
Llama 3.1 8B212.21, 29.34, 57.39, 85.0124.38, 43.11, 50.18, 80.57, 95.0520.22, 26.97, 50.19, 75.28, 89.1421.73, 62.97, 81.37
Mistral 7B212.24, 29.79, 56.68, 64.41, 85.6723.73, 42.94, 48.59, 54.52, 82.2019.05, 26.53, 48.64, 74.83, 93.8822.87, 42.03, 48.79, 60.87, 81.48, 95.97
Qwen2.5-Coder 7B511.73, 29.00, 57.36, 84.8622.70, 42.91, 49.65, 80.50, 95.3919.92, 26.69, 49.25, 75.56, 89.4720.34, 42.09, 64.12, 82.49
Qwen3 4B811.94, 29.00, 57.14, 84.8623.76, 42.55, 49.29, 80.50, 95.0425.94, 50.00, 75.56, 89.1021.19, 41.81, 63.84, 82.49, 96.05
DeepSeek-Coder 7B311.74, 28.86, 56.04, 85.7422.61, 42.90, 54.49, 82.32, 95.6519.34, 26.64, 50.00, 74.4520.97, 41.67, 62.63, 81.72
Granite 3.1 2B411.56, 28.99, 56.88, 85.509.68, 24.19, 42.90, 48.71, 55.16, 81.6120.00, 26.45, 48.71, 74.19, 94.1923.97, 43.36, 53.59, 62.96, 81.70
Granite 3.1 MoE511.56, 28.99, 56.88, 64.04, 85.3224.52, 42.90, 49.35, 55.16, 81.6119.68, 26.45, 48.71, 74.1923.75, 43.36, 63.18, 81.70
Qwen3-Coder 30B-A3B311.73, 29.00, 57.36, 84.8624.11, 42.91, 49.65, 80.50, 95.3919.92, 25.94, 49.62, 75.5663.84, 82.77
AuthorsOpenReview ↗

Response to Reviewer ERbx (Part 2 / 3)

(3)Failure-case analysis. Attention Sink is a widely observed phenomenon in pretrained decoder-only models and a core signal used by ASEmbed to determine partitions and the attention switching layer[1,2]. Yu et al. [3] further report that internal Attention Sinks often occur at separators such as periods, newlines, and colons, indicating that their distribution is related to input structure. Motivated by this observation, we additionally examine input conditions under which internal Attention Sinks are difficult to detect. Our query-length analysis finds that unstructured queries with fewer than 10 tokens often lack clearly detectable internal Attention Sinks and therefore provide weak signals for partitioning. To characterize the prevalence of these short inputs in our evaluation, we examined all experimental datasets. Table R2 lists only datasets/tasks containing queries with fewer than 10 tokens; the remaining datasets contain none.

Table R2: Queries with fewer than 10 tokens in the experimental datasets/tasks.

Dataset/taskTotal queriesQueries with fewer than 10 tokensPercentage
CodeSearchNet-CCR-Go182,7354790.2621%
CodeSearchNet-CCR-Java181,0612880.1591%
CodeSearchNet-CCR-JavaScript65,201650.0997%
CodeSearchNet-CCR-PHP268,237170.0063%
CodeSearchNet-CCR-Python280,65210.0004%
CodeSearchNet-CCR-Ruby27,588350.1269%
NFCorpus3,2372,82887.3648%
SciFact1,109413.6970%

These short queries account for less than 0.3% in each of the six CSN-CCR languages, and for 87.36% and 3.70% in NFCorpus and SciFact, respectively. On NFCorpus, where short queries predominate, ASEmbed achieves 36.67 MRR and 20.95 nDCG@10, exceeding the strongest baseline, Value Aggregation (VA), by 2.32 and 1.38 percentage points, respectively (Table 1). When no internal boundary is detected, the input forms a single block and Eq. (3) retains native causal attention. These results show that ASEmbed retains its retrieval advantage even in the challenging setting of a high proportion of short queries, supporting its overall effectiveness and robustness to variation in input length.

AuthorsOpenReview ↗

Response to Reviewer ERbx (Part 3 / 3)

Q2: Does ASEmbed remain robust when reordering changes meaning or disrupts dependencies across the detected blocks?

A2: Thank you for highlighting this distinction. ASEmbed aims to mitigate order bias when task-relevant semantics are preserved. When reordering changes semantics and candidate relevance, the retrieval ranking should change accordingly; our current evaluation does not allow us to reliably quantify performance in this setting.

Appendix C.1 describes the construction of the current reordering evaluation, in which all reordered samples pass AST parsing and compilation checks. In response to your question, we will carefully review and clarify the scope of our experimental conclusions and investigate embedding extraction under meaning-changing reordering in future work.


We hope that these responses address your concerns. We will incorporate the additional experiments, cross-backbone and short-query analyses, and clarifications regarding the scope of the reordering evaluation into the camera-ready version.

References

  1. Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is Not Only a Weight: Analyzing Transformers with Vector Norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7057–7075. https://aclanthology.org/2020.emnlp-main.574/
  2. Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. 2024. Efficient Streaming Language Models with Attention Sinks. In The Twelfth International Conference on Learning Representations. https://arxiv.org/abs/2309.17453
  3. Zhongzhi Yu, Zheng Wang, Yonggan Fu, Huihong Shi, Khalid Shaikh, and Yingyan Celine Lin. 2024. Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration. In Proceedings of the 41st International Conference on Machine Learning, PMLR 235, pp. 57659–57677. https://proceedings.mlr.press/v235/yu24l.html

Dear Reviewer,

I hope this message finds you well. As the discussion period nears its end, I would like to check whether our responses have satisfactorily addressed your concerns. Please let me know if there are any additional points or feedback you would like us to consider. Your insights are invaluable, and I am eager to address any remaining issues to improve our work. I hope that the additional evidence and clarifications help clarify our contributions and support your overall assessment.

Thank you for your time and effort in reviewing our paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.