- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates if intermediate hidden states in retrieval models can be used directly to perform retrieval without additional training. Using intermediate hidden states enables more efficient retrieval by using early exiting to skip computation when it isn’t needed. The authors first identify that using the standard last-token or CLS pooling on intermediate states results in poor performance. They then show that using no pooling with MaxSim similarity works well for certain layers while attention-based aggregation is more potent at deeper levels. They also introduce a training-free method to calibrate the embedding space of intermediate representations. They find that applying there methods can recover a large fraction of the final hidden state quality while reducing compute by skipping later layers.
Critical strengths
The question of how well intermediate states can function as retrieval representations is pretty interesting and something that hasn't been studied very much (to my knowledge). Studying novel questions is useful as it provides new insights to the community and helps to build new methods for the future. The experiments to answer the question provide the community some new insights about retrieval models and how information is learned and composed to form retrieval representations.
Critical weaknesses
The writing and narrative of the paper is confused and hard to follow which makes reading it an unpleasant experience. The writing has several AI writing tropes which dilute useful information and add to the unpleasantness. It doesn't feel as if the authors have invested time to polish the writing to make there ideas easy to understand and their contribution clear.
Although novel, the core question the paper is answers is unlikely to be directly impactful to retrieval applications. If someone wanted to use intermediate representations for retrieval they could pretty easily train a model to do this so the need for a training free method is unclear. Additionally, in general early exiting is not widely used in either Information Retrieval or NLP, as far as I'm aware, which suggests the savings are not generally worth the additional architectural and serving complexity. The paper also doesn't address a necessary component of achieving the best performance with early exiting which is an early exiting criteria which says when the model should exit and skip the remaining layers. Without this the main benefit is to just stop at a certain layer to reduce encoding time but it risks making some queries perform worse if they needed additional layers for optimal performance.
Questions for authors
- What do you see as the main contribution of this work? Is it more an analysis or a method?
- What kind of results do you see on other dataset such as TREC DL 19 and 20 where BM25 is not a strong baseline like it is for BEIR? For datasets where term matching is sufficient additional depth may not be needed but for other more "semantic" datasets higher levels may be lacking many crucial pieces of information.
- Did you try mean pooling as an alternative to attention-based pooling? That might produce better results in early layers before the model has been able to aggregate information into the CLS or last token.
- Without a method to determine that the model should exit at a certain layer how do you see this method being used to improve search latency in real-world uses?
- How would your method compare with training a model to produce meaningful intermediate retrieval representations?
- If you use models that use mean pooling instead of an aggregation token do you see the same results?
Below are some more specific comments and suggestions:
- Writing doesn’t seem that polished “intermediate layers can retrieve extremely poorly even when the same model performs strongly at its final layer”
- In the introduction the authors mention the “native pooled representation” but it is unclear what the native pooling representation is. Also, I think this phrasing is confusing as natively the model does not use intermediate representations as all so perhaps native pooling technique is more accurate.
- Writing is kind of hard to follow in some places for example: “TRIM combines a depth-dependent account of retrieval, three training-free probes and ranking analysis. With the encoder frozen (Figure 1), MaxSim tests token evidence, Attention2 Readout tests re-aggregation, and calibration tests geometric recovery.”
- At this point TRIM hasn’t been introduced except in the title and several of these terms are not obvious, for example: “depth-dependent account”. Also, “MaxSim tests token evidence” is not very clear.
- Section 3 seems to be trying to balance between being an analysis of models and a new useful method. I think the better framing is this is a study to understand what is going on in retrieval models intermediate state for the helpful task of being able to early exit. The latency experiments help to confirm the usefulness of the approach, but I think the analysis is the main contribution.
- In Section 3.2 it is unclear what pooling position means in “be the attention from the pooling position p to position j“. This is not a standard term and should be defined clearly.
- Some AI slop kinds of language: “Its benefit is measured, not assumed”
- The section transitions in Section 3 don’t flow very clearly and feel abrupt or unconnected. There needs to be clearer connecting text between these.
- In Section 3 notation is used a lot without a clear text description which makes it harder to follow than it should be. The notation could also be made clearer by making it more obvious what each symbol represents for example $v_l^r$ why does $r$ need and if $z_L$ is the same model but at the final layer why not use $v_L$? Also, $v$ is not an obvious character to represent a dense retrieval model. Similar issues exist throughout Section 3.
3 comments
Response 1 — What the remaining objections explain
Dear Reviewer hE8j,
We would like to say that we took your original suggestion seriously and revised Section 3 around the diagnostic question you identified. We were therefore frustrated to read that the analysis “makes sense” while the absence of a dynamic exit policy still weighs against the paper.
We think three distinctions matter for the final assessment.
- MaxSim only applies to four of twelve comparisons.
MaxSim is naturally more expressive and has larger representations than pooled representations.
Fact. Only four shallow-exit comparisons use MaxSim. In the other middle- and deep-exit comparisons on SciFact and NFCorpus, both Attention² and calibration produce single vectors of the same dimension. The best evaluated frozen TRIM configuration scores higher in all eight. Appendix F.1, Table 41
Our assessment. For the diagnostic question, however, MaxSim’s expressiveness is precisely why we use it: shallow representations performed very poorly, but they still contain retrieval information that only MaxSim can extract. MaxSim is not a choice, but so far the only training-free approach we can find.
A rejection decision based on extending the capacity objection to the other eight results would be a factual mistake. None uses MaxSim.
- The earlier TREC exits already answer your original question directly.
"Layer 24 seems already to be pretty deep in the model. It would have been nice to get some different levels in these experiments."
Fact. On the 8.84M-passage index, our L18 (half depth), L20, and L24 runs yield (nDCG@10):
| Qwen 4B exit | DL19 original | DL19 TRIM | DL20 original | DL20 TRIM |
|---|---|---|---|---|
| L18 | 0.0378 | 0.5118 | 0.0180 | 0.3601 |
| L20 | 0.0243 | 0.5912 | 0.0126 | 0.4276 |
| L24 | 0.5608 | 0.6739 | 0.4408 | 0.6424 |
Calibrated last-token pooling stays below 0.04 at L18 and L20, and TRIM also outperforms original pooling by a wide margin across all other earlier exits tested (L12 to L23). TRIM already exceeds BM25 on DL19 (0.4818) at L18 and L20, and on both benchmarks at L24 (DL20 BM25: 0.5146).
Our assessment. The results regarding much earlier exits are available. TRIM still outperforms BM25 on TREC, confirming our contribution, and should be able to address your concern.
- A dynamic exit policy would answer a separate research question.
As an analysis paper, I think this paper makes sense, [...] but it isn’t clear why this would be beneficial over just using a smaller retrieval model.
Fact. We have made that comparison on the same Qwen 4B index. Calibrated Qwen 0.6B leads at L22, while the 4B exit leads at L24 and L25. Separately, with both queries and documents exiting at Qwen L22, changing the readout raises retention from 2.2% to 80.3% across three datasets while the encoder stays frozen. Small-model comparison, Table 14 · L22 diagnostic result
Our assessment. In your original review, you wrote that “the analysis is the main contribution.” We revised Section 3 accordingly. Your follow-up agrees that “as an analysis paper, I think this paper makes sense,” yet still weighs the absence of query-dependent exiting against the paper. This is the inconsistency we find difficult to reconcile: you ask us to frame the work as a diagnostic study, then judge its significance by whether we have built the adaptive system that the diagnosis could inform. A stopping rule decides which queries exit at L22; we show why frozen L22 works at all (2.2% to 80.3% retention). We understand the value of a dynamic policy. What we find discouraging is that its absence appears to count more heavily toward clear rejection than the diagnostic finding you encouraged us to foreground. We respectfully ask you to reconsider that weighting.
Best regards,
The authors
Response 2 — Why we disagree with the judgment about the trained baselines
Dear Reviewer hE8j,
You originally asked how TRIM would compare with training intermediate representations. We trained six exit-specific models to answer that question. We welcome scrutiny of those experiments. What troubled us in the follow-up was the suggestion that the results themselves were doubtful because training was expected to win.
1. The training and evaluation protocol
Reviewer’s comment. “I don't know how reliable the trained models are given they perform pretty poorly even at the final layer and aren't trained on relevant data for the tasks they are evaluated on.”
Fact. We followed the standard protocol of training on MS MARCO and testing on BEIR [1]. Wang et al. [2] used this protocol to evaluate intermediate retrieval representations, including models trained separately at target exits. We trained six exit-specific models. After re-encoding the documents, Qwen retained 94.1–98.5% of its original full-depth SciFact score across the three Qwen runs. Our Appendix F.1 · Wang et al., SIGIR 2025
Our assessment. The objection that the models “aren't trained on relevant data” misunderstands the evaluation protocol [1]. Testing on BEIR after MS MARCO training is an established practice in precisely this line of research [1, 2]. The observed decline is measurable, but neither its magnitude nor the choice of training data establishes that the baselines are unreliable. We respectfully consider that this judgment lacks an empirical basis.
2. The expectation that training should win
Reviewer’s comment. “In general, training should do better than untrained methods with enough data, so to not see that result raises some questions.”
Fact. Both approaches start from a retrieval-trained encoder. TRIM freezes its parameters and changes how its intermediate states are read. Across both models and all three exit depths on SciFact and NFCorpus, frozen TRIM outperforms direct training in all 12 comparisons; at the Qwen L22 exit, for example, frozen TRIM beats direct training on both SciFact (0.6569 vs. 0.6025) and NFCorpus (0.3597 vs. 0.3408). Gao et al. [3] likewise found that HyDE with frozen Contriever outperformed MS MARCO–fine-tuned Contriever on SciFact (0.691 vs. 0.677). Our Table 41 · Gao et al., ACL 2023, Table 2
Our assessment. We find it frustrating that you asked us to test whether direct training would outperform frozen recovery, yet cite the expectation that training should win as a reason to doubt the outcome. That reasoning prejudges the very question you asked us to test. We respectfully request that either outcome receive the same evidentiary weight.
Best regards,
The authors
References
[1] Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, I. (2021). BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks.
[2] Wang, S., Zhuang, S., Koopman, B., and Zuccon, G. (2025). 2D Matryoshka Training for Information Retrieval. SIGIR.
[3] Gao, L., Ma, X., Lin, J., and Callan, J. (2023). Precise Zero-Shot Dense Retrieval without Relevance Labels. ACL.
Thank you for taking the time to respond. The additional experiments help to ease some of my concerns, but some still remain.
Direct Training The direct training comparison is useful, but comparing MaxSim versus standard pooling seems odd as MaxSim is naturally more expressive and has larger representations than pooled representations. There are also not many details about how the trained model is trained and it seems plausible that there are additional confounding factors. For example, only training on MSMARCO potentially degrades generalization on the BEIR tasks. This seems to be supported by your own results: "Training also lowered the full-depth Qwen 4B SciFact score in all three runs". Evaluating on TREC DL or MSMARCO Dev would have been a clearer test. Alternatively, you could train a model in the same way without intermediate representation training (i.e. standard retrieval training) and then compare your methods on that model.
It would also be interesting to see if training complements TRIM.
Overall, I don't know how reliable the trained models are given they perform pretty poorly even at the final layer and aren't trained on relevant data for the tasks they are evaluated on. In general, training should do better than untrained methods with enough data, so to not see that result raises some questions.
TREC Experiments Thank you for adding the TREC experiments. However, I don't know if they help me to understand the main question I was asking, which was more about the performance of earlier layers. Layer 24 seems already to be pretty deep in the model. It would have been nice to get some different levels in these experiments.
General Motivation As an analysis paper, I think this paper makes sense, but it still feels like it is being presented as a practical method. I don't see much practical use for the approach except maybe to understand how easy it would be to train intermediate representations for retrieval (although this isn't established by the experiments). From a practical stand point, this method essentially turns a larger embedding model into a smaller one without additional training (although it does require some effort to do different pooling or calibration), but it isn't clear why this would be beneficial over just using a smaller retrieval model. In your comparison with Qwen 0.6 B, you use 24 layers of Qwen 4B, which uses ~4x the parameters that Qwen 0.6B does, and has only marginal quality improvements. The one big advantage early existing would have over just using a smaller model is flexibility based on the query difficulty, but the paper doesn't investigate this detail.
I would suggest for future versions the authors clearly justify their method compared with smaller embedding models or investigate how to have dynamic early exiting. The other option is frame it as a analysis without a direct practical use or perhaps some use for understanding how well intermediate layers could be used for retrieval if they were trained. If it is focused more as an analysis, I think that would make sense, but it would have to compete with other papers that have investigated intermediate retrieval model representations in various ways.
Conclusion I appreciate the effort made by the authors, but I feel that this paper is not ready for publication. The motivation and scope is limited and there is still some doubt about how generalizable the findings are. For that reason, I will be keeping my rating unchanged.