acceptodds
Under review as a conference paper at ICLR 2027

Monosemantic Retrieval

33 views

Abstract

A retrieval system fundamentally pairs a representation method with a similarity metric that compares the representations to retrieve relevant documents. Modern retrieval is increasingly converging to using a dense vector from Large Language Model embeddings, with cosine similarity as the metric. However, polysemanticity in LLMs is a representational constraint for such systems, leaving the property that makes a document the right answer superpositioned with other features that may not be relevant to the query. At the same time, cosine similarity in high dimensions of the embeddings dilutes the weight of components that decide nuanced relevance during retrieval. Interpretability research in LLMs has emphasized methods which recover monosemantic features from polysemantic representations. We hypothesize that such methods can solve the foundational issues in using LLM based embeddings for retreival. We introduce Monosemantic Retrieval (MR), where we learn per-feature retrieval weights over over monosemantic representations from Sparse Autoencoders and use them to index documents for retrieval. We score retrieval over the few sparse features a query and document share in place of cosine similarity, thus overcoming the “curse of dimensionality”. Across 66 benchmarks spanning eleven domains, we show that this sits on the quality-versus-compute Pareto frontier of retrieval, matching the performance of state-of-the-art trained dense retrieval, with 28× fewer query-time operations at the median corpus, a margin that grows with collection size, and 43% of its index bytes, while improving on existing lexical and learned-sparse retrieval methods. Leveraging the interpretable capabilities of these representations, we show that MR unlocks inference time steering capabilities that dense embedders lack. We simulate a real-world deployment over a mixed multi-domain corpus and show that MR unlocks inference-time steering in retrieval systems, which improves performance over state-of-the-art dense retrievers with a relative 20.9% improvement, while retaining the frontier on both index size and inference compute.

Primary area:
interpretability and explainable AI
Keywords:
Information Retrieval, Interpretability

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9a6SOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Monosemantic Retrieval (MR), which learns per-feature retrieval weights over monosemantic representations from Sparse Autoencoders and scores retrieval over shared sparse features instead of cosine similarity. Across 66 benchmarks, MR matches state-of-the-art dense retrieval with 28× fewer query-time operations and 43% of its index bytes, while outperforming existing lexical and learned-sparse methods. MR also enables inference-time steering, achieving a 20.9% relative improvement over state-of-the-art dense retrievers in a mixed multi-domain corpus.

Critical strengths
  1. The paper proposes a meaningful research direction by using SAE feature coordinates as retrieval units and further leveraging interpretable features for dynamic control at inference time.
  2. The paper provides extensive experimental evaluations across 66 benchmarks, covering retrieval tasks in a wide range of domains.
  3. The paper conducts comprehensive ablation studies to validate the effectiveness of its individual components.
Critical weaknesses
  1. The claimed 28× reduction in query-time arithmetic may not fully reflect the practical computational efficiency of MR. The cost analysis mainly accounts for the sparse postings scan and the dense scoring of the top-200 candidates, without reporting the complete end-to-end retrieval latency, including the cost of computing dense embeddings on the fly.
  2. The number of queries in the mixed-index steering simulation appears inconsistent. Section 6.1 and Appendix E state that the index includes nine collections with 100 queries per collection, implying 900 queries in total. However, the inference-time steering experiment uses 978 queries, and Table 10 reports more than 100 queries for several collections.
  3. The normalization procedure for facet projections is described inconsistently. Section 6.1 states that each document's projection is standardized over the index, whereas Appendix G states that the median and spread are computed from the same positive and negative examples used for facet construction. These two normalization strategies may produce different steering effects.
Questions for authors
  1. Could the authors provide end-to-end latency and computational cost measurements, including the dense embedding computation required for reranking? Does the reported 28× reduction in per-query arithmetic translate into comparable practical speedups?
  2. Could the authors clarify how the 978-query stream was constructed and explain why some collections contribute more than 100 queries despite the stated sampling protocol?
  3. Could the authors clarify whether the facet projections are normalized using statistics from the entire index or from the mined positive and negative examples? Which normalization procedure was used in the reported steering experiments?
4 comments

Thank you again for your review. We wanted to follow up on our responses. We clarified that the key results use stored dense vectors for the rerank, and explained the cost calculations. We also explained how the 978-query stream was built, and we confirmed that facet normalization uses statistics from the full index. We will update the paper and appendices to match these descriptions.

We would be happy to discuss anything further. If these responses address your concerns, we would appreciate it if you would consider updating your score.

Thank you for your rebuttal.

However, your responses confirm that all three issues I identified were actual errors and self-contradictions in the original manuscript (especially conflating the 43% and 68% storage settings in the headline claims, while still not providing the complete end-to-end retrieval latency). Essentially, you are patching these flaws during the rebuttal phase rather than resolving misunderstandings.

So I will maintain my score.

Thank you for engaging in the discussion.

We would like to clarify again that we did report both of these numbers in the original submitted manuscript. The settings which give the 43% and 68% storage gains are presented in the Section 5 of the main paper in lines 279-282: "against the trained retriever on 53 shared benchmarks, the sparse index alone reaches 0.445 against 0.473 from 43% of the index bytes, and a dense rerank of its top 200 candidates closes the gap to within 0.005 at 28×to 55× less query-time arithmetic, with the index at 68% of the dense bytes when a 4-bit copy is stored for the rerank." as well as in Appendix L. Our proposal was to state this more explicitly in the paper. Specifically, we will update the claim in the abstract so that the main storage gains also come from the stored-copy setting. The same applies to the other two points in the discussion. In each case, the reported numbers and the implementations are unchanged. The edits we proposed restate what the paper already reports more explicitly, so we respectfully see them as clarifications and not patchwork or actual errors.

End-to-end latency. We reported operation counts because wall-clock latency depends heavily on the hardware, the search kernels and the index library, which are not the focus of this work. Operation counts give a more reliable unit that is comparable across systems.

We report the end-to-end latency including the query encoding, in milliseconds per query:

CorpusDocsCPU: MR-GCPU: DenseGPU: MR-GGPU: Dense
SciFact5k232623.623.0
FollowIR News2131k323832.331.6
FiQA58k294023.923.1
Quora (hard negatives)177k276726.125.0
Quora523k4017229.427.3

On the CPU, MR-G runs on Seismic (a sparse engine), while the dense model runs on FAISS. MR-G is faster at every index size. The latency gap grows with collection size, as we mentioned earlier. On the GPU, the dense search runs on cuBLAS, which is heavily optimized for dense matrix products. Our sparse search there has no comparable kernel, yet it stays within 1 to 3 ms end to end. We will add these measurements to Appendix L with the caveats we noted.

We hope this can address your concerns.

Thank you for the clarification, I have raised my score.

Reviewer_t1LgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper performs retrieval over sparse autoencoder features extracted from dense embeddings. It learns per-feature gains and biases, uses weighted Jaccard similarity for sparse retrieval, and optionally reranks 200 candidates with dense embeddings. It also mines reusable document facets from simulated feedback and routes later queries to these facets to steer retrieval.

Critical strengths
  • Feedback-driven facets offer a concrete way to distinguish documents that share a topic but serve different purposes. This is a useful direction for making retrieval more controllable.

  • The modular design and ablations separate feature adaptation, sparse candidate generation, and reranking. The evaluation across several embedding models also helps reveal the method’s dependence on the underlying representation.

Critical weaknesses
  • The main quality–cost comparison is not consistently accounted for. Table 5 counts sparse retrieval and 200 dense dot products, but Appendix L states that candidate embeddings are computed on the fly. The cost of encoding those documents is omitted. Storing 4-bit embeddings instead raises storage from the claimed 43% to 68% of the dense index. Table 5 also averages the reranked results over 36 benchmarks and the dense results over 40, so the matching averages do not establish matching quality.

  • The novelty discussion contains substantive errors. Section 2 describes SPLARE as training the SAE dictionary and lacking multidomain evaluation, although SPLARE freezes the SAE while adapting the backbone and reports multidomain retrieval results. The claim that SAEs have not previously been used for indexing, querying, and retrieval also conflicts with CL-SR in Decoding Dense Embeddings. These are close enough to require accurate discussion and meaningful comparisons.

Questions for authors
  • What are the full latency and storage costs when SAE projection and any candidate-document encoding are included?

  • Do the quality–cost conclusions hold when all systems are evaluated on the same benchmark intersection?

  • After correcting the descriptions of SPLARE and CL-SR, which contributions remain distinct, and what comparisons support them?

2 comments

We thank the reviewer again for their review. We wanted to follow up on our responses. We clarified the cost calculations and will report encoding and projection costs separately. We also showed that the conclusions are retained on the same benchmark intersections. In addition, we will correct our descriptions of SPLARE and CL-SR, and as we explained, we will clarify which contributions remain distinct alongside the ablations that support them.

We would be glad to clarify anything further. If these responses address your concerns, we would appreciate it if you would consider updating your score.

The storage and matched-benchmark clarifications address much of my original concern. Please confirm that the reported two-stage quality uses the same stored 4-bit vectors included in the 68% storage figure. My remaining reservation is the interpretation of the ablations: MR-F and the untrained Jaccard index do not directly reproduce SPLARE and CL-SR, respectively. They support the proposed design, but not the stronger claim that effective steering requires monosemanticity or is unique to MR. I would like to remain scoring 2, but with a more positive assessment than in my original review.

Reviewer_jx3ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper (Monosemantic Retrieval) indexes documents by the features a frozen sparse autoencoder (SAE) extracts from an LLM embedding, instead of comparing dense vectors. It trains only a per-feature gain and bias, and optionally reranks the top 200 with the dense model. It claims dense-level quality at 43% of the index bytes and 28–55× fewer query-time operations across 66 benchmarks. It also claims a "steering" loop (an LLM learns document properties from user feedback and reweights results) that beats a strong dense retriever by 20.9%.

Critical strengths
  1. The framing is new. Keeping the SAE dictionary frozen means each feature keeps its meaning and can be used at query time. A matched control (same backbone and training data, full dictionary retrained) isolates this choice.
  2. The evaluation is broad and transparent: 66 benchmarks, 9 SAE backbones, a full range of compressed and approximate dense indexes, bootstrap confidence intervals, measured operation counts, the LLM prompts given verbatim, and code included.
  3. The ablations are useful. A random dictionary roughly halves quality. As a candidate generator, the sparse stage recovers 98.4% of the dense top-10 versus 57.3% for BM25 at the same depth.
  4. The steering analysis is careful. It separates facet quality from intent detection, states when steering helps, and reports its losses.
Critical weaknesses
  1. The bytes and compute claims come from two different setups. The 28–55× figure assumes stored dense vectors for the rerank, which makes the index 68% of the dense size, not 43%. Appendix L instead says rerank embeddings are "computed on the fly." That means running 200 documents through an 8B model on every query, and that cost isn't counted anywhere.

  2. The query encoding is left out of the operation counts, and no latency is reported. The 8B forward pass per query is far larger than the scoring operations being compared. That matters most against BM25 and SPLADE, which use much smaller encoders.

  3. The "margin grows with collection size" only holds against exhaustive dense search. The paper's own Table 5 shows a standard graph index (HNSW) using about 5× fewer operations at the largest corpus, at the same quality. That HNSW was also stored at fp32.

  4. The held-out test is contaminated. All design choices were tuned on the 66 main benchmarks. The SAE dictionary was trained on documents from Quora, Touche, ArguAna and SciFact. Subsamples of those same corpora make up four of the ten "held-out" tasks. Those tasks are also tiny (2–6k documents), and 8 of 10 differences round to zero.

  5. Some baselines look broken. E5-Mistral scores far below its published numbers (SciFact 0.358, CoSQA 0.016), yet the paper claims a "51 of 51" win over it. The same Qwen3-Embedding-8B model gives two different exact-search results (text2sql 0.601 vs 0.849). Two R2MED benchmarks appear twice and are both counted in the 66. The steering benchmark was built for the dense baseline to fail.

  6. Collections were admitted only if the dense retriever confuses them. Steering settings were tuned on those same collections. Feedback comes from clean gold labels. The paper's own label-noise result shows a loss bigger than the entire gain. Facets learned from earlier queries' gold documents may simply be steering later queries to those same documents.

  7. The natural dense-side steering baselines are missing. These include putting the facet into Qwen3-Embedding's instruction prompt, a linear probe, Rocchio relevance feedback, and an LLM reranker. The one dense baseline tested actually beats MR on 5 of 9 collections, and each loop was run only once.

  8. The claim that steering keeps the compute advantage ignores the LLM. A frontier LLM is called on every query and after every round.

  9. The mechanism story isn't tested. Sparse inner product works about as well as weighted Jaccard, so set overlap isn't what "overcomes the curse of dimensionality." The trained bias raises the median active features from 128 to 578, so it's unclear the deployed features keep their single meanings, and this is never checked.

Questions for authors

Address all the weakness

2 comments

We thank the reviewer again for their thorough review. We wanted to follow up on our responses. In them, we report results with the overlapping benchmarks removed and find that the results stay the same. We also added the instruction-prompt dense steering baseline as suggested, where MR still comes out ahead. We separated steering gains on queries whose gold documents were never seen. Finally, we verified that the trained features keep their SAE directions.

We would be happy to discuss any remaining concerns. If these responses address them, we would appreciate it if you would consider updating your score.

Reviewer_jx3ZOpenReview ↗

Response to the authors

I appreciate the efforts put by the authors and after reading the responses, i have raised my score.

Comments on the paper

1 comment

Dear Reviewers,

Thank you again for the time and care you have put into reviewing our paper. We have posted detailed responses to each review. As the discussion period comes to a close, we wanted to gently check whether these responses address your concerns.

In brief, we clarified the storage and compute calculations. The main paper will now report 68% of the dense index bytes, and it will show search costs separately from the per-query encoding cost. We also ran several new analyses. These include results with the benchmarks that overlap with the dictionary’s training corpora removed, comparisons on matched benchmark sets, an instruction-prompt dense steering baseline, and a steering breakdown on queries whose gold documents were not previously revealed. We also added a check confirming that trained features keep their monosemantic directions.

If anything is still unclear, we would be glad to discuss it further. If our responses resolve your concerns, we would be grateful if you would consider revisiting your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.