- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Binary Semantic Indexing (BSI), a framework that aims to establish a unified binary semantic space supporting both direct and generative retrieval. Specifically, the authors employ retrieval supervision and Binary Spherical Quantization (BSQ) to compress documents into binary codes, which are subsequently partitioned into local token pairs to construct an inverted index. Based on this representation, the authors develop two generative query interfaces: a Global Generator that predicts complete document codes and a Local Generator that directly predicts local index addresses. The Local Generator further incorporates index-access and candidate-retention objectives to better align generation with the actual retrieval process. Experiments are conducted on NQ, MS MARCO, BEIR, and MAIR to evaluate retrieval effectiveness, quantization quality, and the tradeoffs associated with index storage and access budgets.
Critical strengths
The research problem is meaningful, and the overall framework is reasonably designed. The paper attempts to connect direct retrieval, inverted indexing, and generative retrieval through a unified binary semantic representation, allowing different query mechanisms to access the same document representations. Separating document representations from query mechanisms is an interesting direction and potentially enables flexible adaptation of query models without reconstructing the underlying document index.
The technical framework is relatively comprehensive and supported by extensive experiments. Beyond binary representation learning and local index construction, the authors introduce optimization objectives that explicitly consider index access and candidate retention, adding technical depth to the proposed approach. The experiments cover multiple retrieval benchmarks and demonstrate the feasibility of the framework on million-scale document collections. The analyses of code length, access budgets, and storage costs are also informative.
Critical weaknesses
-
The paper introduces a relatively complex pipeline involving binary quantization, local-address indexing, multiple generative query interfaces, and index-aware optimization. However, it remains unclear what substantial benefits this complexity provides over conventional dense retrieval. For example, Table 9 shows that BSQ achieves substantially lower retrieval effectiveness than the original dense representations, while Table 14 shows nearly identical performance between BSI and Dense retrieval under shared scoring. More importantly, on MS MARCO, the simpler Direct BSQ interface outperforms both generative interfaces. These results raise a fundamental question: why should we adopt this elaborate framework instead of simply using a conventional dense retriever? Although the authors emphasize representation compactness and the flexibility of a shared semantic space, they do not provide convincing evidence of corresponding improvements in retrieval effectiveness, end-to-end efficiency, or practical utility. Without demonstrating such advantages, the motivation for the proposed framework remains insufficiently justified.
-
Since one of the central motivations of BSI is to construct compact binary representations that support efficient retrieval, a natural question is how it compares with existing binary dense retrieval and learning-to-hash approaches. Such methods can also compress document embeddings into binary codes and perform efficient retrieval using Hamming distance or specialized indexing structures. Although the paper compares BSQ with PQ, OPQ, and RQ in terms of representation quality, these comparisons do not adequately establish the advantages of BSI over existing binary retrieval systems. In particular, the paper lacks controlled comparisons with representative binary dense retrieval baselines under matched encoder capacity, code length, and retrieval budgets. Comparisons of retrieval effectiveness, index memory, and end-to-end latency would be essential to determine whether BSI provides meaningful improvements beyond existing binary retrieval approaches.
Questions for authors
-
What concrete advantages does the proposed framework offer over conventional dense retrieval, particularly considering its additional complexity and the competitive performance of the simpler Direct BSQ interface?
-
How does BSI compare with existing binary dense retrieval and learning-to-hash methods under matched settings?
-
I also have some concerns about the fairness of the experimental comparisons. Specifically, the main NQ results are evaluated on 15,324 calibration queries, whereas the reported baselines use the standard 7,830-query test set. On BEIR, the authors adopt target-aware per-dataset configuration selection using relevance judgments, which differs from standard zero-shot evaluation. Furthermore, the final results involve additional dense/text reranking components, making it difficult to isolate the contribution of the proposed method. Could the authors provide more rigorous comparisons under consistent evaluation protocols?
4 comments
Official Comment by Authors
W2. Matched binary retrieval and learning-to-hash baselines
The added learning/deployment comparison starts all binary systems from the same dense-only parent, with a common backbone, 256-bit payload, TRAIN/CAL data, encoder/query search and Hamming-access/asymmetric-ranking procedure. Each selected encoder has its corpus freshly encoded (Table 5 (p. 9)):
| Binary system | R10 | MRR10 | State (MiB) | p50 (ms) |
|---|---|---|---|---|
| BPR | 88.13 | 73.13 | 11.19 | 34.93 |
| BPR+EMA | 88.20 | 73.57 | 11.19 | 35.60 |
| BSI-Direct | 88.93 | 74.27 | 11.19 | 35.37 |
BSI-Direct gains +0.72 R10 points [0.44, 1.01] and +0.70 MRR10 [0.38, 1.03] over BPR+EMA, from unrounded seed-averaged query outcomes. This improves quality at the same binary-state size and similar latency. BPR adapts its published objective to the common Qwen backbone. Q2 describes the complementary fixed-feature hashing/compression controls.
Q1. Why support generative access alongside BSI-Direct?
The shared representation supports compact retrieval through BSI-Direct and learned query access through the generative interfaces. BSI-Local provides the fixed-index improvements demonstrated in W1. With identical adjacent-pair postings and 4,096 addresses, it gives 90.20 R10 versus BSI-Global's 88.66 (Table 14 (p. 29)). BSI-Global supplies complete-code predictions whose local components also benefit from the index: resolving the same completed beams through pair postings improves R10 from 88.40 to 89.49 over complete-SID lookup. These experiments show how the common code vocabulary supports both efficient address prediction and sequence-based access.
In the complete canonical NQ native + common public 4B text-scoring pipeline, BSI-Local/BSI-Global generation costs 0.13/2.25 seconds and full p50 is 11.39/15.88 seconds; Dense/BSI-Direct full p50 is 12.26/12.17 seconds (Table 28 (p. 39)). Serial QPS is 0.087/0.063 for BSI-Local/BSI-Global and 0.082/0.083 for Dense/BSI-Direct. Timings include encoding through CPU-ID return, exclude loading and use mean duration for QPS. The lower generation overhead favors BSI-Local when efficient query adaptation is the priority.
The compact BSI-Direct deployment stores binary document codes and ranks them with continuous queries, with process GPU peaks of 1.850/1.873 GiB for BSI-Direct/Dense. For the MS hybrid pipeline, the complete retrieval-state ledger is 1.984 GiB access + 134.982 GiB vector cache = 136.966 GiB, with model weights and runtime buffers additional (Table 27 (p. 39)). Reporting these components alongside the quality and timing results specifies the resources used by each deployment.
Official Comment by Authors
Q2. What do the hashing and compression controls show?
The fixed-encoder study holds continuous features constant and compares how alternative methods compress and search them. Asymmetric BSI-Direct achieves 87.14/69.80 R10/MRR10, compared with BPR+EMA's 84.55/65.80 and OPQ ADC's 86.48/68.54 (Table 17 (p. 30)). The table also includes RP, ITQ and PQ. Supervised binary heads share TRAIN/CAL data; the other compressors fit unlabeled corpus vectors, and Hamming/asymmetric scoring is identified for each result. Together with W2's encoder-adapted comparison, this provides evidence for the learned binary construction at both the representation-learning and fixed-feature levels.
Q3. Consistent protocols and component attribution
NQ. Table 1 (p. 7) compares BSI with 25 external methods. BSI evaluates all 7,830 official query IDs/texts/qrels on the 109,739 released documents, achieving native BSI-Local/BSI-Global R10 of 89.87/87.85 without text scoring or Dense insertion. BSI-Global resolves frozen canonical-trie proposals through newly built released postings. Appendix E.2 (from p. 26) specifies the released-corpus evaluation, the 109,650-document canonical control and the protocols of the published baselines.
MS MARCO. Table 2 (p. 7) includes TSGen, LTRGR, DGR, RIPOR, PAG, DOGR and PAG reproduction. Document R10 averages the fraction of relevant passages in Top10; MRR10 averages the first relevant reciprocal rank within ten. All 6,980 queries enter both denominators, including misses. Reconstructing the complete BSI-Local system gives 68.82/41.98; BSI-Global reports 71.67/41.23. With the same text-only scorer applied to each method's candidate pool, Dense/BSI-Direct/BSI-Local gives 69.61/40.21, 69.16/40.05 and 66.26/38.84 (Table 7 (p. 26)).
BEIR now uses one complete thirteen-task policy per row. Holding the encoder and candidate pools fixed, equal native/text weighting gives 63.91 macro nDCG@10, compared with 62.97 at 1:3 using the same terminal 8B scorer. The source-selected 8B scorer reaches 65.02 under the common 1:3 rule (Table 24 (p. 35)). A prespecified public4B/context2048 text-only run gives 62.76 on the same native100 pools, with all four public scoring rules included in that table.
Representation transfer. Native 8B BSI-Direct retains 97.4% of matched Dense with native cosine ordering and the vector cache: macro nDCG@10 is 62.23 versus 63.86; adding the fixed 8B text scorer gives 62.97 (Table 3 (p. 8)). The 0.6B native configuration gives 41.37, including 21.14 on Touché. Completing Touché with the same MS-selected 8B text model used on the other tasks gives 33.00 and a full BEIR-13 mean of 56.86, under fixed 1:3 fusion. Our runs use original qrels and linear gain; SSR and other published baselines retain their source conditions.
The expanded MAIR study provides a matched Dense comparison across six tasks. Using the same public 4B model, context 2048 and Top200 text-only scoring, BSI-Direct achieves 45.18 versus Dense's 44.27 macro nDCG@10; the original shared-scoring configuration gives 38.92/38.91 (Table 4 (p. 9)). Table 25 (p. 37) and Table 26 (p. 38) report both gain definitions and all 36 paired configurations across the tested model, context and candidate-depth settings.
Controlled comparisons and selection. The canonical common-scorer comparison (Table 8 (p. 27)), text/insertion interventions (Table 12 (p. 28)), and released TSGen/SEAL comparison (Table 15 (p. 30)) connect document access to retrieval quality under specified scoring conditions. Current NQ selection uses TRAIN/CAL, and the reported comparisons fix their execution policies. Earlier development included evaluation on the NQ holdout and BEIR targets; original MS settings used development judgments.
Thank you again for the thoughtful review. The revised comparisons make BSI's practical contribution concrete: retrieval-aware binary quality at compact state, indexed access through a common code vocabulary, and measurable query improvements while document codes and postings remain fixed.
Official Comment by Authors
Dear Reviewer vAVh,
Thank you again for your time and thoughtful feedback. We have provided point-by-point responses to your concerns and updated the manuscript with corresponding clarifications and additional experimental evidence.
As the discussion period approaches its close, we would be grateful to hear whether these responses address your concerns, and whether any questions remain. We would be happy to provide further clarification during the remaining discussion period.
Thank you for considering our responses and revisions.
Best regards,
The Authors
Official Comment by Authors
Thank you for recognizing the potential of adapting query models over a fixed document index. We address your concerns about the practical value beyond Dense retrieval and binary hashing with matched quality, storage and latency measurements, together with controlled query updates. The revised paper explains how one learned binary representation supports compact retrieval and trainable access to the same document index (Section 4 and Appendices E–H). Below we connect each architectural choice to its measured benefit. Scores are percentages; R10 and MRR10 denote Recall@10 and MRR@10.
W1. Framework benefits and the added value of query adaptation
BSI organizes the framework around a shared document representation. Retrieval supervision learns binary semantic coordinates, and lossless packing exposes those coordinates as index keys and prediction targets. BSI-Direct retrieves through query encoding; BSI-Global predicts complete codes; BSI-Local predicts addresses. This organization provides two practical benefits: compact document storage and query learning that reuses an existing index. The comparisons below evaluate each benefit with its corresponding resource and quality measurements.
The first study compares BSI-Direct with Dense using a matched representation-learning and deployment setup (Table 5 (p. 9)):
BSI-Direct reduces persistent retrieval state by 19.8-fold, with similar measured query latency. The table jointly reports retrieval quality, all persistent retrieval tensors and latency from a raw query to 2,000 document IDs. This quantifies the compact operating point available through the framework's direct interface.
The generative interface adds the ability to learn improved access over that fixed document organization. Refining the BSI-Local head while holding document codes, postings, 4,096 addresses, 2,000 candidates and the additional query scorer fixed gives (Table 6 (p. 10)):
Head refinement improves R10 by +0.294 points, with a 95% paired-query CI of [0.145, 0.456], and MRR10 by +0.154 points [0.035, 0.281], after averaging seeds within each query. The update reuses the document codes and postings, with native scoring and no text scoring or Dense insertion in either configuration.
A complementary experiment freezes the existing query adapter at CAL-selected β=1 and freshly retrieves the released corpus with the refined heads (Table 22 (p. 33)):
On identical pools, the query scorer adds +0.40 R10 points [0.17, 0.63] and +1.46 MRR10 points [1.18, 1.76]. The seven-strength selection uses all 15,324 CAL queries; document codes/postings and candidate caps stay fixed. These controls demonstrate two routes to improved retrieval: learning address access with the scorer fixed, and adapting query evidence on fixed pools. Both improvements reuse document organization, establishing the practical value of a trainable query interface to the semantic index.