acceptodds
Under review as a conference paper at ICLR 2027

TRIM: Training-Free Retrieval from Intermediate Model States

Abstract

Intermediate layers of LLMs often perform surprisingly poorly in dense retrieval when read out as a single vector, raising a fundamental question: do these layers lack retrieval-relevant information, or is the information present but not yet accessible to the native representation? We introduce training-free retrieval from intermediate model states (TRIM), a framework combining theoretical analysis and retrieval interventions to investigate this question without training or modifying the underlying model. First, we bypass single-vector pooling and perform token-level matching on shallow states. Surprisingly, even very shallow layers recover substantial retrieval performance despite near-zero performance with the native readout, revealing that retrieval-relevant evidence is already present but remains distributed across tokens. We then ask whether this evidence can be recovered into a single vector. Re-aggregating token states using the model's own attention substantially improves retrieval at middle depths, showing that insufficient aggregation is a major bottleneck. Finally, we find that the remaining gap can be largely explained by the geometry of the resulting embedding space: a training-free geometric calibration increasingly recovers retrieval quality at deeper layers. Together, these results reveal a depth-dependent transition in intermediate representations, i.e., from distributed retrieval evidence, to incomplete aggregation, to geometric mismatch. Building on this observation, we introduce a fully training-free early-exit retrieval pipeline that combines attention-based readout with geometric calibration. Without updating the backbone or using relevance labels, it retains 91.6% of full-depth nDCG\@10 across seven datasets while reducing query encoding time by 26.8–30.5% in our timed setting. Our findings suggest that the poor retrieval performance of intermediate LLM states is not simply due to missing retrieval information, but to how that information is progressively aggregated and geometrically organized across depth.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dense Retrieval, Text Embeddings, Intermediate-Layer Representations, Efficient Inference, Attention-Based Pooling, Geometric Calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hE8jOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates if intermediate hidden states in retrieval models can be used directly to perform retrieval without additional training. Using intermediate hidden states enables more efficient retrieval by using early exiting to skip computation when it isn’t needed. The authors first identify that using the standard last-token or CLS pooling on intermediate states results in poor performance. They then show that using no pooling with MaxSim similarity works well for certain layers while attention-based aggregation is more potent at deeper levels. They also introduce a training-free method to calibrate the embedding space of intermediate representations. They find that applying there methods can recover a large fraction of the final hidden state quality while reducing compute by skipping later layers.

Critical strengths

The question of how well intermediate states can function as retrieval representations is pretty interesting and something that hasn't been studied very much (to my knowledge). Studying novel questions is useful as it provides new insights to the community and helps to build new methods for the future. The experiments to answer the question provide the community some new insights about retrieval models and how information is learned and composed to form retrieval representations.

Critical weaknesses

The writing and narrative of the paper is confused and hard to follow which makes reading it an unpleasant experience. The writing has several AI writing tropes which dilute useful information and add to the unpleasantness. It doesn't feel as if the authors have invested time to polish the writing to make there ideas easy to understand and their contribution clear.

Although novel, the core question the paper is answers is unlikely to be directly impactful to retrieval applications. If someone wanted to use intermediate representations for retrieval they could pretty easily train a model to do this so the need for a training free method is unclear. Additionally, in general early exiting is not widely used in either Information Retrieval or NLP, as far as I'm aware, which suggests the savings are not generally worth the additional architectural and serving complexity. The paper also doesn't address a necessary component of achieving the best performance with early exiting which is an early exiting criteria which says when the model should exit and skip the remaining layers. Without this the main benefit is to just stop at a certain layer to reduce encoding time but it risks making some queries perform worse if they needed additional layers for optimal performance.

Questions for authors
  • What do you see as the main contribution of this work? Is it more an analysis or a method?
  • What kind of results do you see on other dataset such as TREC DL 19 and 20 where BM25 is not a strong baseline like it is for BEIR? For datasets where term matching is sufficient additional depth may not be needed but for other more "semantic" datasets higher levels may be lacking many crucial pieces of information.
  • Did you try mean pooling as an alternative to attention-based pooling? That might produce better results in early layers before the model has been able to aggregate information into the CLS or last token.
  • Without a method to determine that the model should exit at a certain layer how do you see this method being used to improve search latency in real-world uses?
  • How would your method compare with training a model to produce meaningful intermediate retrieval representations?
  • If you use models that use mean pooling instead of an aggregation token do you see the same results?

Below are some more specific comments and suggestions:

  • Writing doesn’t seem that polished “intermediate layers can retrieve extremely poorly even when the same model performs strongly at its final layer”
  • In the introduction the authors mention the “native pooled representation” but it is unclear what the native pooling representation is. Also, I think this phrasing is confusing as natively the model does not use intermediate representations as all so perhaps native pooling technique is more accurate.
  • Writing is kind of hard to follow in some places for example: “TRIM combines a depth-dependent account of retrieval, three training-free probes and ranking analysis. With the encoder frozen (Figure 1), MaxSim tests token evidence, Attention2 Readout tests re-aggregation, and calibration tests geometric recovery.”
    • At this point TRIM hasn’t been introduced except in the title and several of these terms are not obvious, for example: “depth-dependent account”. Also, “MaxSim tests token evidence” is not very clear.
  • Section 3 seems to be trying to balance between being an analysis of models and a new useful method. I think the better framing is this is a study to understand what is going on in retrieval models intermediate state for the helpful task of being able to early exit. The latency experiments help to confirm the usefulness of the approach, but I think the analysis is the main contribution.
  • In Section 3.2 it is unclear what pooling position means in “be the attention from the pooling position p to position j“. This is not a standard term and should be defined clearly.
  • Some AI slop kinds of language: “Its benefit is measured, not assumed”
  • The section transitions in Section 3 don’t flow very clearly and feel abrupt or unconnected. There needs to be clearer connecting text between these.
  • In Section 3 notation is used a lot without a clear text description which makes it harder to follow than it should be. The notation could also be made clearer by making it more obvious what each symbol represents for example $v_l^r$ why does $r$ need and if $z_L$ is the same model but at the final layer why not use $v_L$? Also, $v$ is not an obvious character to represent a dense retrieval model. Similar issues exist throughout Section 3.
3 comments

Thank you for taking the time to respond. The additional experiments help to ease some of my concerns, but some still remain.

Direct Training The direct training comparison is useful, but comparing MaxSim versus standard pooling seems odd as MaxSim is naturally more expressive and has larger representations than pooled representations. There are also not many details about how the trained model is trained and it seems plausible that there are additional confounding factors. For example, only training on MSMARCO potentially degrades generalization on the BEIR tasks. This seems to be supported by your own results: "Training also lowered the full-depth Qwen 4B SciFact score in all three runs". Evaluating on TREC DL or MSMARCO Dev would have been a clearer test. Alternatively, you could train a model in the same way without intermediate representation training (i.e. standard retrieval training) and then compare your methods on that model.

It would also be interesting to see if training complements TRIM.

Overall, I don't know how reliable the trained models are given they perform pretty poorly even at the final layer and aren't trained on relevant data for the tasks they are evaluated on. In general, training should do better than untrained methods with enough data, so to not see that result raises some questions.

TREC Experiments Thank you for adding the TREC experiments. However, I don't know if they help me to understand the main question I was asking, which was more about the performance of earlier layers. Layer 24 seems already to be pretty deep in the model. It would have been nice to get some different levels in these experiments.

General Motivation As an analysis paper, I think this paper makes sense, but it still feels like it is being presented as a practical method. I don't see much practical use for the approach except maybe to understand how easy it would be to train intermediate representations for retrieval (although this isn't established by the experiments). From a practical stand point, this method essentially turns a larger embedding model into a smaller one without additional training (although it does require some effort to do different pooling or calibration), but it isn't clear why this would be beneficial over just using a smaller retrieval model. In your comparison with Qwen 0.6 B, you use 24 layers of Qwen 4B, which uses ~4x the parameters that Qwen 0.6B does, and has only marginal quality improvements. The one big advantage early existing would have over just using a smaller model is flexibility based on the query difficulty, but the paper doesn't investigate this detail.

I would suggest for future versions the authors clearly justify their method compared with smaller embedding models or investigate how to have dynamic early exiting. The other option is frame it as a analysis without a direct practical use or perhaps some use for understanding how well intermediate layers could be used for retrieval if they were trained. If it is focused more as an analysis, I think that would make sense, but it would have to compete with other papers that have investigated intermediate retrieval model representations in various ways.

Conclusion I appreciate the effort made by the authors, but I feel that this paper is not ready for publication. The motivation and scope is limited and there is still some doubt about how generalizable the findings are. For that reason, I will be keeping my rating unchanged.

AuthorsOpenReview ↗

Response 1 — What the remaining objections explain

Dear Reviewer hE8j,

We would like to say that we took your original suggestion seriously and revised Section 3 around the diagnostic question you identified. We were therefore frustrated to read that the analysis “makes sense” while the absence of a dynamic exit policy still weighs against the paper.

We think three distinctions matter for the final assessment.

  1. MaxSim only applies to four of twelve comparisons.

MaxSim is naturally more expressive and has larger representations than pooled representations.

Fact. Only four shallow-exit comparisons use MaxSim. In the other middle- and deep-exit comparisons on SciFact and NFCorpus, both Attention² and calibration produce single vectors of the same dimension. The best evaluated frozen TRIM configuration scores higher in all eight. Appendix F.1, Table 41

Our assessment. For the diagnostic question, however, MaxSim’s expressiveness is precisely why we use it: shallow representations performed very poorly, but they still contain retrieval information that only MaxSim can extract. MaxSim is not a choice, but so far the only training-free approach we can find.

A rejection decision based on extending the capacity objection to the other eight results would be a factual mistake. None uses MaxSim.

  1. The earlier TREC exits already answer your original question directly.

"Layer 24 seems already to be pretty deep in the model. It would have been nice to get some different levels in these experiments."

Fact. On the 8.84M-passage index, our L18 (half depth), L20, and L24 runs yield (nDCG@10):

Qwen 4B exitDL19 originalDL19 TRIMDL20 originalDL20 TRIM
L180.03780.51180.01800.3601
L200.02430.59120.01260.4276
L240.56080.67390.44080.6424

Calibrated last-token pooling stays below 0.04 at L18 and L20, and TRIM also outperforms original pooling by a wide margin across all other earlier exits tested (L12 to L23). TRIM already exceeds BM25 on DL19 (0.4818) at L18 and L20, and on both benchmarks at L24 (DL20 BM25: 0.5146).

Our assessment. The results regarding much earlier exits are available. TRIM still outperforms BM25 on TREC, confirming our contribution, and should be able to address your concern.

  1. A dynamic exit policy would answer a separate research question.

As an analysis paper, I think this paper makes sense, [...] but it isn’t clear why this would be beneficial over just using a smaller retrieval model.

Fact. We have made that comparison on the same Qwen 4B index. Calibrated Qwen 0.6B leads at L22, while the 4B exit leads at L24 and L25. Separately, with both queries and documents exiting at Qwen L22, changing the readout raises retention from 2.2% to 80.3% across three datasets while the encoder stays frozen. Small-model comparison, Table 14 · L22 diagnostic result

Our assessment. In your original review, you wrote that “the analysis is the main contribution.” We revised Section 3 accordingly. Your follow-up agrees that “as an analysis paper, I think this paper makes sense,” yet still weighs the absence of query-dependent exiting against the paper. This is the inconsistency we find difficult to reconcile: you ask us to frame the work as a diagnostic study, then judge its significance by whether we have built the adaptive system that the diagnosis could inform. A stopping rule decides which queries exit at L22; we show why frozen L22 works at all (2.2% to 80.3% retention). We understand the value of a dynamic policy. What we find discouraging is that its absence appears to count more heavily toward clear rejection than the diagnostic finding you encouraged us to foreground. We respectfully ask you to reconsider that weighting.

Best regards,

The authors

AuthorsOpenReview ↗

Response 2 — Why we disagree with the judgment about the trained baselines

Dear Reviewer hE8j,

You originally asked how TRIM would compare with training intermediate representations. We trained six exit-specific models to answer that question. We welcome scrutiny of those experiments. What troubled us in the follow-up was the suggestion that the results themselves were doubtful because training was expected to win.

1. The training and evaluation protocol

Reviewer’s comment. “I don't know how reliable the trained models are given they perform pretty poorly even at the final layer and aren't trained on relevant data for the tasks they are evaluated on.”

Fact. We followed the standard protocol of training on MS MARCO and testing on BEIR [1]. Wang et al. [2] used this protocol to evaluate intermediate retrieval representations, including models trained separately at target exits. We trained six exit-specific models. After re-encoding the documents, Qwen retained 94.1–98.5% of its original full-depth SciFact score across the three Qwen runs. Our Appendix F.1 · Wang et al., SIGIR 2025

Our assessment. The objection that the models “aren't trained on relevant data” misunderstands the evaluation protocol [1]. Testing on BEIR after MS MARCO training is an established practice in precisely this line of research [1, 2]. The observed decline is measurable, but neither its magnitude nor the choice of training data establishes that the baselines are unreliable. We respectfully consider that this judgment lacks an empirical basis.

2. The expectation that training should win

Reviewer’s comment. “In general, training should do better than untrained methods with enough data, so to not see that result raises some questions.”

Fact. Both approaches start from a retrieval-trained encoder. TRIM freezes its parameters and changes how its intermediate states are read. Across both models and all three exit depths on SciFact and NFCorpus, frozen TRIM outperforms direct training in all 12 comparisons; at the Qwen L22 exit, for example, frozen TRIM beats direct training on both SciFact (0.6569 vs. 0.6025) and NFCorpus (0.3597 vs. 0.3408). Gao et al. [3] likewise found that HyDE with frozen Contriever outperformed MS MARCO–fine-tuned Contriever on SciFact (0.691 vs. 0.677). Our Table 41 · Gao et al., ACL 2023, Table 2

Our assessment. We find it frustrating that you asked us to test whether direct training would outperform frozen recovery, yet cite the expectation that training should win as a reason to doubt the outcome. That reasoning prejudges the very question you asked us to test. We respectfully request that either outcome receive the same evidentiary weight.

Best regards,

The authors

References

[1] Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, I. (2021). BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks.

[2] Wang, S., Zhuang, S., Koopman, B., and Zuccon, G. (2025). 2D Matryoshka Training for Information Retrieval. SIGIR.

[3] Gao, L., Ma, X., Lin, J., and Callan, J. (2023). Precise Zero-Shot Dense Retrieval without Relevance Labels. ACL.

Reviewer_G6vGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why intermediate LLM layers perform poorly in dense retrieval and proposes three training-free probes: token-level MaxSim, attention-based re-pooling, and geometric calibration. Combining re-pooling with calibration yields 91.6% of full-depth nDCG@10 across seven datasets while reducing query encoding time in the reported setting.

Critical strengths

1, The paper separates several plausible causes of poor intermediate retrieval. Its probes provide evidence that retrieval information can be present in token states even when native pooling fails to expose it.

2, The analysis connects to a practical use case. Attention-based re-pooling and geometric calibration improve retrieval without changing the backbone weights.

Critical weaknesses

The attention-based readout uses the model’s own attention at its designated pooling position, but the paper does not explain why this attention row should be a reliable proxy for retrieval-relevant aggregation across architectures. Its effectiveness may depend on the backbone’s pooling design rather than reflect a general property of intermediate states.

Questions for authors

How sensitive is Attention² to the choice of pooling position and backbone, and what supports using that attention row as a retrieval-relevant aggregation rule?

Reviewer_Zd66OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks why intermediate layers of LLM embedding models retrieve so poorly, and whether the problem is missing information or the way that information is read out. The authors introduce TRIM, three training-free probes on a frozen model: token-level matching (MaxSim) to check whether retrieval evidence exists, an attention-based pooling method (Attention^2) to test better aggregation, and a linear calibration fitted on unlabeled text to test whether the embedding space is simply misaligned. Experiments on four LLM embedders across BEIR datasets show that useful evidence exists even in shallow layers, aggregation is the main bottleneck at middle depths, and calibration alone recovers most quality near the end, with the combined method retaining 91.6% of full-depth nDCG@10 on Qwen3-Embedding-4B while cutting query encoding time by about 27–30%.

Critical strengths

The writing is good and the presentation is clear.

Critical weaknesses
  1. The motivation for the proposed method is weak and under-developed. It is unclear why enabling early-exit for embedding models is useful or desirable. The authors' main argument for early-exit is time savings for encoding. However, for a large pool of vectors, the search time may significantly exceed the the encoding time. Table 4 shows "Encoding time saved" but no end-to-end retrieval time saved.
  2. The overhead of MaxSim seems significant. The ColBERT-style all-to-all token interaction scoring introduce high overhead compared to single-token scoring. It is unclear whether the efficiency gains from early exit can offset this additional overhead of MaxSim.
  3. Comparisons against obvious alternatives are missing; for example, using a smaller but deeper model (as opposed to early exit). Full-depth Qwen3-Embedding-0.6B might achieve higher quality than 4B with early exit. In addition, trained baselines and distilled baselines are cited but not compared.
Questions for authors
  1. What are end-to-end latency and batched throughput savings once ANN search is included?
  2. What is the overhead of MaxSim compared to single-token retrieval?

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We thank the reviewers for their careful reading and for their constructive feedback. We are encouraged that the reviewers found our diagnostic question "pretty interesting and something that hasn't been studied very much" (hE8j), with insights that help "build new methods for the future" (hE8j); that they noted our diagnosis "separates several plausible causes of poor intermediate retrieval", showing "retrieval information can be present in token states even when native pooling fails to expose it" (G6vG) and "aggregation is the main bottleneck at middle depths" (Zd66); and that in practice they found the analysis "connects to a practical use case" to "improve retrieval without changing the backbone weights" (G6vG) and "recover a large fraction of the final hidden state quality while reducing compute" (hE8j), "retaining 91.6% of full-depth nDCG@10 on Qwen3-Embedding-4B while cutting query encoding time by about 27–30%" (Zd66).

The reviews share one question: whether poor intermediate retrieval comes from missing information or from the original pooling, and whether our explanation brings real quality and speed gains across models and tasks without retraining. Following your comments, we clarified in Section 3 how TRIM combines theoretical insight with practical value (explaining why frozen layers fail rather than aiming to beat full-depth upper bounds or train a dynamic router, and lifting Qwen 4B L22 retention from 2.2% to 80.3% without training), and added the experiments summarized below. Unless stated otherwise, they use the protocol of Section 4 (frozen encoders, unlabeled calibration, zero-shot nDCG@10 on the unchanged index), and each item states where the setup differs. The main experiments cover BEIR and TREC DL with frozen Qwen 4B, Qwen 0.6B, Mistral 7B, and BGE models, addressing the questions raised by all three reviewers.

  1. Comparison with trained early exits (hE8j, Zd66). Against six exit-specific models trained on MS MARCO, frozen TRIM wins all 12 comparisons on SciFact and NFCorpus. Only four shallow cases use MaxSim; in all eight middle- and deep-exit cases, TRIM uses a single vector of the same size as original pooling and wins all eight, including Qwen 4B L22 on SciFact (0.6569 vs. 0.6025) and NFCorpus (0.3597 vs. 0.3408). After document re-encoding, trained Qwen 4B retains 94.1% to 98.5% of full-depth SciFact quality, while TRIM keeps the model and index unchanged (Appendix F.1, Table 41).

  2. Pooling design (G6vG, hE8j). On the same Qwen 4B L22 tokens, Attention² beats uniform averaging and shuffled weights on SciFact (0.6858 vs. 0.6585 and 0.5730; Table 42), and tops calibrated mean pooling across seven datasets at L22, L24, and L25 (L25: 0.5047 vs. 0.4761; Tables 8, 13). Over L20 to L25, Qwen 4B's last-token attention beats a random earlier token (3-seed mean) on SciFact (0.6749 vs. 0.1698) and NFCorpus (0.3424 vs. 0.0461; Section 3.2, Table 43). On CLS-pooled BGE, CLS attention beats calibrated CLS pooling by 0.091 to 0.202 points at L15 to L17 (Table 34).

  3. Earlier semantic layers (hE8j). Searching 8.84M passages on TREC DL19 and DL20, single-vector TRIM beats Qwen 4B's calibrated original pooling at every exit from L12 to L23 (24/24 comparisons). Original pooling stays below 0.04 at L18 and L20, whereas at the 36-layer midpoint L18, TRIM already far exceeds original pooling (0.5118 / 0.3601) and surpasses BM25 (0.5118 vs. 0.4818), advancing steadily at L20 (0.5912 / 0.4276), L22 (0.6245 / 0.5220; BM25: 0.4818 / 0.5146), and L24 (0.6739 / 0.6424; Table 44, reply to hE8j). At one-third depth L12, MaxSim also beats BM25 across three BEIR sets (0.422 vs. 0.406; Table 39).

  4. End-to-end search time (Zd66, hE8j). Including query encoding and unchanged index search (8-query batches), fixed Qwen 4B L25 cuts total time by 24.1% on SciFact (41.87 to 31.79 ms, +31.7% throughput, 97.8% quality retained) and 26.3% on NFCorpus (30.05 to 22.15 ms, +35.7% throughput, 95.5% retained), averaging 91.6% of full-depth quality across seven datasets without per-query routing (Section 4.4, Table 4, Appendix F.4; on search-dominated NQ, L32 saves 1.7%). MaxSim only probes shallow tokens: on SciFact, Qwen 4B L8 MaxSim rises from 0.0430 (L8 mean) to 0.6587, matching full-depth mean pooling (0.6590) at +5.6 ms/query (Table 46).

  5. Smaller models (Zd66, hE8j). On the same Qwen 4B index across seven datasets, Qwen 4B at L24 (0.4946) and L25 (0.5047) beats calibrated full-depth Qwen 0.6B (0.4812) without a second model or index rebuild (Tables 8, 14). On Qwen 0.6B itself across three datasets, TRIM lifts L8 from 0.005 to 0.382 and L22 from 0.311 to 0.445, keeping 88.5% of full-depth quality while skipping six layers (Table 27). This confirms TRIM's general applicability, directly improving small models to run faster with less compute.

The complete tables and the list of revisions appear in the revised paper and in our replies to each reviewer.

Dear reviewers,

As rebuttal deadline is approaching, we sincerely appreciate your continued engagement and feedbacks.

Wish you all the best.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.