- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SimSparse, a graph-based approximate retrieval index that operates directly on learned latent sparse embeddings. When constructing the graph, for each document, a few strong, rare features (considering activation and IDF) are selected. SimSparse then proposes candidate neighbors from three sources: 1) local windows around the document's position in activation-ordered posting lists; 2) per-feature "landmarks", i.e., the most strongly activated documents; 3) signature buckets of documents sharing the same top-L feature identifiers. Candidates are rescored with an IDF-weighted sparse inner product and pruned to M=16 out-edges. At query time, the query's active features retrieve short per-feature router lists. The best E entry points are selected by sparse score and used to start a standard best-first traversal. Experiments on six BEIR datasets with two backbones report an nDCG@10 at 30-40x lower storage than HNSW/DiskANN. On a 60M-passage RPJ-Wiki corpus, the authors report about 3.5-4 GiB of storage and about 9 ms/query at a matched 90% Recall@3 against a hybrid dense+sparse reference.
Critical strengths
- The motivation is clear and relevant.
- Experiments are performed on a mid-sized collection of 60M documents.
Critical weaknesses
-
The paper does not actually consider an entire line of literature developed in the Information Retrieval community, which is the one researching for efficient retrieval over learned sparse representations (LSR) [some examples: 1,2,3,4,5,6,7,8]. Only a few papers in this line are cited. Nothing is used for comparison in the reported experiments. In this line, the authors will find inverted index-based solutions for LSR [1,2,3,5,6,7], along with fast HNSW-based adaptations for LSR (starting from BigANN challenge 2023 [4] up to the last months [8]). I invite the authors to reference the papers and, most importantly, to align and borrow from this line, the evaluation methodology which is built around three main directions: 1) small datasets, MSMARCOv1 (8M docs); 2) out-of-domain evaluation (BEIR), 3) very important, indeed: large datasets (MSMARCOv2, 145M docs). From the referenced papers, the contributions in this line are very strong competitors to SimSparse (reported latencies in the microsecond range for MSMARCOv1 and below 4ms for MSMARCOv2, which is far larger than the 60M used by the authors here). I am recommending testing these solutions in the same experimental evaluation to provide a more comprehensive analysis of the proposed solution.
-
Some choices in the evaluation methodology are questionable: What is the rationale behind using hybrid reference top-3 neighbors as ground truth? Given k, why not report recall@k over the pre-computed list of k closest documents?
-
Important experiments are missing: no index build time or peak construction memory are reported for SimSparse and competitors.
-
Several experimental details are missing:
- What is the hardware used?
- What is the HNSW used? Did you compare with SOTA implementations, e.g., kANNolo [9], or other less competitive libraries?
- How do you tune the hyperparameters for competitors?
- No parameters reported for the solutions or the quantizations used in the main experiments.
-
Code is not accessible at the time of the submission, although the authors promised to release it after.
-
In terms of presentation, the paper is very dense and sometimes difficult to understand.
Overall, although the idea is interesting, the paper is not mature enough to be accepted at ICLR 2027.
[1] Bruch et al. Pairing Clustered Inverted Indexes with κ-NN Graphs for Fast Approximate Retrieval over Learned Sparse Representations. SIGIR 2024. [2] Bruch et al. Investigating the Scalability of Approximate Sparse Retrieval Algorithms to Massive Datasets. ECIR 2025. [3] Bruch et al. Pairing Clustered Inverted Indexes with κ-NN Graphs for Fast Approximate Retrieval over Learned Sparse Representations. CIKM 2024. [4] https://big-ann-benchmarks.com/neurips23.html [5] Mallia et al. Faster Learned Sparse Retrieval with Block-Max Pruning. SIGIR 2024. [6] Carlson et a. Efficient Sparse Retrieval with Lightweight Superblock Pruning. SIGIR 2026. [7] Li et al. SINDI: An Efficient Index for Sparse Vector Approximate Maximum Inner Product Search. ICDE 2026. [8] Martinico et al. Multivector Reranking in the Era of Strong First-Stage Retrievers. ECIR 2026. [9] Martinico et al. Two Indexes, One Foundation: kANNolo and Seismic for Efficient Learned Sparse Retrieval at SISAP 2026.
Questions for authors
- How does SimSparse behave against efficient graph-based and inverted-index-based solutions for LSR?
- What is the build time or peak construction memory for SimSparse and competitors?
- What are the performance of SimSparse over MSMARCOv2?
6 comments
W2: Why use hybrid top-3 neighbors as the retrieval ground truth?
Some choices in the evaluation methodology are questionable: What is the rationale behind using hybrid reference top-3 neighbors as ground truth? Given k, why not report recall@k over the pre-computed list of k closest documents?
A2: We thank the reviewer for this question. RPJ-60M lacks human relevance judgments, so we use pre-computed exact top-3 neighbors as a proxy ground truth. However, using dense or sparse top-3 alone would inherently favor methods based on the corresponding representation. We therefore construct the reference using an equal-weight combination of normalized dense and sparse scores. As shown in Table II, dense and sparse top-3 overlap by only 10.58% on average, while the hybrid reference captures information from both representations.
Table II: Top-3 overlap among exact dense, sparse, and hybrid rankings on RPJ-60M.
| Query set | Dense/Sparse | Dense/Hybrid | Sparse/Hybrid |
|---|---|---|---|
| NQ | 13.17% | 48.67% | 32.17% |
| GPQA | 8.17% | 45.50% | 28.83% |
| TriviaQA | 10.33% | 45.00% | 31.83% |
| HotpotQA | 10.67% | 44.33% | 34.50% |
| Avg. | 10.58% | 45.88% | 31.83% |
We have clarified the representation-relative matched-performance criterion in Appendix C.2 and included the dense/sparse/hybrid ranking-overlap analysis in Appendix D.5, with the main findings summarized in Section 5.4 of the revised paper.
W3 & Q2: What are the index construction time and peak memory costs of SimSPARSE and its competitors?
W3: Important experiments are missing: no index build time or peak construction memory are reported for SimSPARSE and competitors.
Q2: What is the build time or peak construction memory for SimSPARSE and competitors?
A3: Thank you for the suggestion. We have conducted additional experiments measuring index construction time and peak memory usage across eight corpora and seven indexing methods. HNSW and DiskANN build their indexes using dense Contriever embeddings, while the remaining methods use sparse Contriever-CSR representations. The results are reported in Table III and Table IV.
Table III: Index construction time (s) across eight corpora for different indexing methods.
| Corpus | SimSPARSE | HNSW | DiskANN | Seismic | SeismicWave | kANNolo | SINDI |
|---|---|---|---|---|---|---|---|
| HotpotQA | 401.10 | 1743.32 | 2214.74 | 625.69 | 406.61 | 418.45 | 305.62 |
| NQ | 209.24 | 936.75 | 1288.19 | 371.58 | 356.75 | 214.07 | 187.66 |
| ClimateFever | 451.73 | 1895.41 | 2572.75 | 587.29 | 671.27 | 590.75 | 404.62 |
| Fever | 478.30 | 2032.79 | 3126.83 | 612.35 | 492.11 | 490.42 | 512.89 |
| DBpedia | 362.68 | 1672.18 | 1947.06 | 423.43 | 395.86 | 417.66 | 385.13 |
| MSMARCO | 716.59 | 3945.87 | 4434.53 | 693.19 | 669.97 | 721.94 | 872.18 |
| RPJ-60M | 2043.18 | 17589.50 | 22923.38 | 2130.44 | 1812.05 | 2639.42 | 2796.29 |
| MSMARCOv2 | 3837.76 | 32654.26 | 37820.69 | 5892.92 | 3668.72 | 5234.73 | 4783.97 |
Table IV: Peak index construction memory (GiB) across eight corpora for different indexing methods.
| Corpus | SimSPARSE | hnsw | diskANN | Seismic | SeismicWave | kANNolo | SINDI |
|---|---|---|---|---|---|---|---|
| HotpotQA | 2.05 | 32.53 | 85.77 | 2.02 | 5.37 | 1.93 | 1.74 |
| NQ | 1.28 | 16.84 | 48.46 | 1.21 | 2.92 | 1.25 | 0.91 |
| ClimateFever | 2.13 | 33.65 | 85.39 | 2.25 | 5.61 | 2.12 | 1.79 |
| Fever | 2.13 | 33.70 | 85.43 | 2.25 | 5.62 | 2.12 | 1.83 |
| DBpedia | 1.89 | 28.87 | 80.80 | 1.91 | 4.81 | 1.96 | 1.59 |
| MSMARCO | 3.18 | 54.63 | 81.54 | 3.79 | 9.11 | 2.82 | 3.12 |
| RPJ-60M | 25.94 | 402.95 | 584.18 | 23.48 | 57.69 | 24.30 | 55.70 |
| MSMARCOv2 | 53.99 | 848.03 | 977.34 | 52.86 | 141.50 | 52.15 | 59.15 |
As shown above, SimSPARSE remains competitive with sparse indexing methods. Its peak construction memory is also dramatically lower than that of dense graph indexes. These results demonstrate that SimSPARSE's low deployment storage and query latency do not come at the expense of prohibitive index construction overhead. We have incorporated the complete index construction time and peak memory comparisons into Appendix D.8 (Tables 18 and 19), with the key results summarized in Section 5.4 of the revised paper.
(continue below)
W4: What hardware, baseline implementations, tuning settings, and quantization configurations were used?
Several experimental details are missing: What is the hardware used? What is the HNSW used? Did you compare with SOTA implementations, e.g., kANNolo [9], or other less competitive libraries? How do you tune the hyperparameters for competitors? No parameters reported for the solutions or the quantizations used in the main experiments.
A4: Thank you for pointing out the missing experimental details. We clarify the hardware, baseline implementations, hyperparameter tuning, and numerical precision below.
Hardware. All experiments were conducted on a server with dual Intel Xeon Platinum 8275CL CPUs (48 physical cores, 96 threads) and 1.10 TiB RAM, running Ubuntu 22.04.5.
Baselines and implementations. Our original HNSW baseline uses FAISS IndexHNSWFlat. To strengthen the comparison with state-of-the-art implementations, we additionally evaluated kANNolo, with results reported in A1. We also use the official DiskANN implementation (diskannpy, v0.7.0), Seismic (pyseismic-lsr, v0.5.1), and SINDI/VSAG (v0.18.5) for the corresponding baselines.
Hyperparameter tuning. We follow the standard tuning strategy for each method, varying the main search-time parameters to evaluate the quality–latency trade-off: efSearch for HNSW and kANNolo, nprobe for IVF, search-list size L_search for DiskANN, and candidate/pruning budgets for Seismic and SeismicWave. For SeismicWave, we additionally account for its KNN graph expansion. Construction parameters are fixed as summarized in Table V.
Table V: Baseline implementations and index-construction parameters.
| Method | Implementation | Main construction parameters |
|---|---|---|
| HNSW | FAISS IndexHNSWFlat | M=16, efConstruction=200 |
| IVF | FAISS IndexIVFFlat | nlist=4096 |
| DiskANN | diskannpy 0.7.0 | R=16, L=200 |
| Seismic | SeismicIndexRaw 0.5.1 | n_postings=3500, max_fraction=1.5, centroid_fraction=0.1, min_cluster_size=2, summary_energy=0.4, doc_cut=16 |
| SeismicWave | Seismic native binding | Same Seismic configuration, plus build_knn(32) |
| kANNolo | SparsePlainHNSW 0.15.1 | m=16, ef_construction=200, graph_type="standard" |
Quantization. The main experiments use FP8 quantization for CSR. The alternative precision settings in Section 5.3 are evaluated separately to analyze the storage–accuracy trade-off.
The newly added comparisons with specialized sparse retrieval implementations, including kANNolo, are reported in Section 5.1 of the revised paper. The hardware environment, implementation versions, tuning procedures, and quantization settings are specified above to clarify the experimental setup.
W5: The implementation code was not publicly accessible at submission time.
code is not accessible at the time of the submission, although the authors promised to release it after.
A5: Thank you for highlighting the importance of code availability and reproducibility. We have made the core implementation of SimSPARSE available in an anonymous repository: https://anonymous.4open.science/r/SimSPARSE/.
The repository includes sparse feature-based graph construction, query-conditioned entry-point routing, index building, and top-k graph retrieval. It supports both standard CSR sparse representations and compressed RPJ-style indexes, with CPU and optional CUDA-accelerated construction. The README provides installation instructions, command-line examples, and a Python API for building indexes, loading them, and executing retrieval queries.
We hope this release facilitates inspection of the core indexing and search mechanisms of SimSPARSE.
(continue below)
We sincerely thank reviewer
kxTnfor the constructive feedback and for recognizing the importance of large-scale retrieval evaluation. We appreciate the reviewer’s guidance on efficient learned sparse retrieval methods and acknowledge the need for stronger comparisons. In response, we add specialized inverted-index and graph-based baselines, extend evaluation to the 145M-document MSMARCOv2 corpus, report index construction costs, and clarify implementation details.W1 & Q1 & Q3: How does SimSPARSE compare with established learned sparse retrieval methods and scale to MSMARCOv2?
A1: We thank the reviewer for highlighting this important line of work. We agree that comparisons with specialized learned sparse representation (LSR) retrieval systems are essential. To address this concern, we have discussed these works in related work and conducted additional experiments against four representative methods: Seismic, SeismicWave, SINDI, and kANNolo, covering both inverted-index-based and graph-based approaches. For a fair comparison, all methods use exactly the same Contriever-based CSR representations, isolating the effectiveness of their indexing and search strategies. We evaluate on six BEIR benchmarks, including MSMARCOv1, and further extend the evaluation to the 145M-document MSMARCOv2 dataset.
Table I: Comparison of learned sparse retrieval methods on six BEIR benchmarks and MSMARCOv2. Each cell reports nDCG@10 (%) / average query latency (ms).
Shown in Table I, SimSPARSE achieves the highest nDCG@10 on 6 of 7 datasets (28.25% on average) and the lowest latency on 3 of 7 datasets. On the 145M-document MSMARCOv2, it achieves the highest nDCG@10 (3.30%) with 13.085 ms latency, demonstrating competitive retrieval quality and efficiency at scale. We have expanded the related work in Section 2 to better cover specialized learned sparse retrieval methods and added these comparisons, including the 145M-document MSMARCOv2 evaluation, to Section 5.1 (Table 2) of the revised paper.
(continue below)