Monosemantic Retrieval

A retrieval system fundamentally pairs a representation method with a similarity metric that compares the representations to retrieve relevant documents. Modern retrieval is increasingly converging to using a dense vector from Large Language Model embeddings, with cosine similarity as the metric. However, polysemanticity in LLMs is a representational constraint for such systems, leaving the property that makes a document the right answer superpositioned with other features that may not be relevant to the query. At the same time, cosine similarity in high dimensions of the embeddings dilutes the weight of components that decide nuanced relevance during retrieval. Interpretability research in LLMs has emphasized methods which recover monosemantic features from polysemantic representations. We hypothesize that such methods can solve the foundational issues in using LLM based embeddings for retreival. We introduce Monosemantic Retrieval (MR), where we learn per-feature retrieval weights over over monosemantic representations from Sparse Autoencoders and use them to index documents for retrieval. We score retrieval over the few sparse features a query and document share in place of cosine similarity, thus overcoming the “curse of dimensionality”. Across 66 benchmarks spanning eleven domains, we show that this sits on the quality-versus-compute Pareto frontier of retrieval, matching the performance of state-of-the-art trained dense retrieval, with 28× fewer query-time operations at the median corpus, a margin that grows with collection size, and 43% of its index bytes, while improving on existing lexical and learned-sparse retrieval methods. Leveraging the interpretable capabilities of these representations, we show that MR unlocks inference time steering capabilities that dense embedders lack. We simulate a real-world deployment over a mixed multi-domain corpus and show that MR unlocks inference-time steering in retrieval systems, which improves performance over state-of-the-art dense retrievers with a relative 20.9% improvement, while retaining the frontier on both index size and inference compute.