TRIM: Training-Free Retrieval from Intermediate Model States

Intermediate layers of LLMs often perform surprisingly poorly in dense retrieval when read out as a single vector, raising a fundamental question: do these layers lack retrieval-relevant information, or is the information present but not yet accessible to the native representation? We introduce training-free retrieval from intermediate model states (TRIM), a framework combining theoretical analysis and retrieval interventions to investigate this question without training or modifying the underlying model. First, we bypass single-vector pooling and perform token-level matching on shallow states. Surprisingly, even very shallow layers recover substantial retrieval performance despite near-zero performance with the native readout, revealing that retrieval-relevant evidence is already present but remains distributed across tokens. We then ask whether this evidence can be recovered into a single vector. Re-aggregating token states using the model's own attention substantially improves retrieval at middle depths, showing that insufficient aggregation is a major bottleneck. Finally, we find that the remaining gap can be largely explained by the geometry of the resulting embedding space: a training-free geometric calibration increasingly recovers retrieval quality at deeper layers. Together, these results reveal a depth-dependent transition in intermediate representations, i.e., from distributed retrieval evidence, to incomplete aggregation, to geometric mismatch. Building on this observation, we introduce a fully training-free early-exit retrieval pipeline that combines attention-based readout with geometric calibration. Without updating the backbone or using relevance labels, it retains 91.6% of full-depth nDCG\@10 across seven datasets while reducing query encoding time by 26.8–30.5% in our timed setting. Our findings suggest that the poor retrieval performance of intermediate LLM states is not simply due to missing retrieval information, but to how that information is progressively aggregated and geometrically organized across depth.