acceptodds
Under review as a conference paper at ICLR 2027

Binary Semantic Indexing for Generative Retrieval

Abstract

Retrieval systems must organize large collections of knowledge while allowing queries to reach relevant information efficiently. As retrieval increasingly relies on learned representations and generative models, the representation itself becomes central to both semantic modeling and access. This raises a broader question: can different retrieval mechanisms share a common semantic foundation? We introduce **Binary Semantic Indexing (BSI)**, a framework that connects sparse indexing and generative retrieval through a **shared binary semantic space**. A jointly trained semantic encoder and binary spherical quantizer compress each document into a compact code that preserves retrieval information. The code's local components form addressable units in an inverted index. Complete-code and local-address generators provide complementary query interfaces, while direct query encoding accesses the same representation. By **separating document organization from query modeling**, BSI allows these mechanisms to operate over a common document space. Experiments show effective retrieval across query interfaces and domains, while controlled analyses characterize the tradeoffs between code capacity, index access, and retrieval quality.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Generative retrieval, Semantic representations, Binary spherical quantization, Sparse retrieval, Semantic indexing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vAVhOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Binary Semantic Indexing (BSI), a framework that aims to establish a unified binary semantic space supporting both direct and generative retrieval. Specifically, the authors employ retrieval supervision and Binary Spherical Quantization (BSQ) to compress documents into binary codes, which are subsequently partitioned into local token pairs to construct an inverted index. Based on this representation, the authors develop two generative query interfaces: a Global Generator that predicts complete document codes and a Local Generator that directly predicts local index addresses. The Local Generator further incorporates index-access and candidate-retention objectives to better align generation with the actual retrieval process. Experiments are conducted on NQ, MS MARCO, BEIR, and MAIR to evaluate retrieval effectiveness, quantization quality, and the tradeoffs associated with index storage and access budgets.

Critical strengths

The research problem is meaningful, and the overall framework is reasonably designed. The paper attempts to connect direct retrieval, inverted indexing, and generative retrieval through a unified binary semantic representation, allowing different query mechanisms to access the same document representations. Separating document representations from query mechanisms is an interesting direction and potentially enables flexible adaptation of query models without reconstructing the underlying document index.

The technical framework is relatively comprehensive and supported by extensive experiments. Beyond binary representation learning and local index construction, the authors introduce optimization objectives that explicitly consider index access and candidate retention, adding technical depth to the proposed approach. The experiments cover multiple retrieval benchmarks and demonstrate the feasibility of the framework on million-scale document collections. The analyses of code length, access budgets, and storage costs are also informative.

Critical weaknesses
  1. The paper introduces a relatively complex pipeline involving binary quantization, local-address indexing, multiple generative query interfaces, and index-aware optimization. However, it remains unclear what substantial benefits this complexity provides over conventional dense retrieval. For example, Table 9 shows that BSQ achieves substantially lower retrieval effectiveness than the original dense representations, while Table 14 shows nearly identical performance between BSI and Dense retrieval under shared scoring. More importantly, on MS MARCO, the simpler Direct BSQ interface outperforms both generative interfaces. These results raise a fundamental question: why should we adopt this elaborate framework instead of simply using a conventional dense retriever? Although the authors emphasize representation compactness and the flexibility of a shared semantic space, they do not provide convincing evidence of corresponding improvements in retrieval effectiveness, end-to-end efficiency, or practical utility. Without demonstrating such advantages, the motivation for the proposed framework remains insufficiently justified.

  2. Since one of the central motivations of BSI is to construct compact binary representations that support efficient retrieval, a natural question is how it compares with existing binary dense retrieval and learning-to-hash approaches. Such methods can also compress document embeddings into binary codes and perform efficient retrieval using Hamming distance or specialized indexing structures. Although the paper compares BSQ with PQ, OPQ, and RQ in terms of representation quality, these comparisons do not adequately establish the advantages of BSI over existing binary retrieval systems. In particular, the paper lacks controlled comparisons with representative binary dense retrieval baselines under matched encoder capacity, code length, and retrieval budgets. Comparisons of retrieval effectiveness, index memory, and end-to-end latency would be essential to determine whether BSI provides meaningful improvements beyond existing binary retrieval approaches.

Questions for authors
  1. What concrete advantages does the proposed framework offer over conventional dense retrieval, particularly considering its additional complexity and the competitive performance of the simpler Direct BSQ interface?

  2. How does BSI compare with existing binary dense retrieval and learning-to-hash methods under matched settings?

  3. I also have some concerns about the fairness of the experimental comparisons. Specifically, the main NQ results are evaluated on 15,324 calibration queries, whereas the reported baselines use the standard 7,830-query test set. On BEIR, the authors adopt target-aware per-dataset configuration selection using relevance judgments, which differs from standard zero-shot evaluation. Furthermore, the final results involve additional dense/text reranking components, making it difficult to isolate the contribution of the proposed method. Could the authors provide more rigorous comparisons under consistent evaluation protocols?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for recognizing the potential of adapting query models over a fixed document index. We address your concerns about the practical value beyond Dense retrieval and binary hashing with matched quality, storage and latency measurements, together with controlled query updates. The revised paper explains how one learned binary representation supports compact retrieval and trainable access to the same document index (Section 4 and Appendices E–H). Below we connect each architectural choice to its measured benefit. Scores are percentages; R10 and MRR10 denote Recall@10 and MRR@10.


W1. Framework benefits and the added value of query adaptation

BSI organizes the framework around a shared document representation. Retrieval supervision learns binary semantic coordinates, and lossless packing exposes those coordinates as index keys and prediction targets. BSI-Direct retrieves through query encoding; BSI-Global predicts complete codes; BSI-Local predicts addresses. This organization provides two practical benefits: compact document storage and query learning that reuses an existing index. The comparisons below evaluate each benefit with its corresponding resource and quality measurements.

The first study compares BSI-Direct with Dense using a matched representation-learning and deployment setup (Table 5 (p. 9)):

SystemR10MRR10State (MiB)p50 (ms)
Dense FP1691.3678.85221.0034.71
BSI-Direct88.9374.2711.1935.37

BSI-Direct reduces persistent retrieval state by 19.8-fold, with similar measured query latency. The table jointly reports retrieval quality, all persistent retrieval tensors and latency from a raw query to 2,000 document IDs. This quantifies the compact operating point available through the framework's direct interface.

The generative interface adds the ability to learn improved access over that fixed document organization. Refining the BSI-Local head while holding document codes, postings, 4,096 addresses, 2,000 candidates and the additional query scorer fixed gives (Table 6 (p. 10)):

Query interfaceR10MRR10
BSI-Local (Original)90.2877.50
BSI-Local (Refined)90.5777.65

Head refinement improves R10 by +0.294 points, with a 95% paired-query CI of [0.145, 0.456], and MRR10 by +0.154 points [0.035, 0.281], after averaging seeds within each query. The update reuses the document codes and postings, with native scoring and no text scoring or Dense insertion in either configuration.

A complementary experiment freezes the existing query adapter at CAL-selected β=1\beta=1 and freshly retrieves the released corpus with the refined heads (Table 22 (p. 33)):

Query configurationR10MRR10
BSI-Local (Refined)90.1075.54
BSI-Local (Refined + query scorer)90.4977.00

On identical pools, the query scorer adds +0.40 R10 points [0.17, 0.63] and +1.46 MRR10 points [1.18, 1.76]. The seven-strength selection uses all 15,324 CAL queries; document codes/postings and candidate caps stay fixed. These controls demonstrate two routes to improved retrieval: learning address access with the scorer fixed, and adapting query evidence on fixed pools. Both improvements reuse document organization, establishing the practical value of a trainable query interface to the semantic index.

AuthorsOpenReview ↗

Official Comment by Authors

W2. Matched binary retrieval and learning-to-hash baselines

The added learning/deployment comparison starts all binary systems from the same dense-only parent, with a common backbone, 256-bit payload, TRAIN/CAL data, encoder/query search and Hamming-access/asymmetric-ranking procedure. Each selected encoder has its corpus freshly encoded (Table 5 (p. 9)):

Binary systemR10MRR10State (MiB)p50 (ms)
BPR88.1373.1311.1934.93
BPR+EMA88.2073.5711.1935.60
BSI-Direct88.9374.2711.1935.37

BSI-Direct gains +0.72 R10 points [0.44, 1.01] and +0.70 MRR10 [0.38, 1.03] over BPR+EMA, from unrounded seed-averaged query outcomes. This improves quality at the same binary-state size and similar latency. BPR adapts its published objective to the common Qwen backbone. Q2 describes the complementary fixed-feature hashing/compression controls.


Q1. Why support generative access alongside BSI-Direct?

The shared representation supports compact retrieval through BSI-Direct and learned query access through the generative interfaces. BSI-Local provides the fixed-index improvements demonstrated in W1. With identical adjacent-pair postings and 4,096 addresses, it gives 90.20 R10 versus BSI-Global's 88.66 (Table 14 (p. 29)). BSI-Global supplies complete-code predictions whose local components also benefit from the index: resolving the same completed beams through pair postings improves R10 from 88.40 to 89.49 over complete-SID lookup. These experiments show how the common code vocabulary supports both efficient address prediction and sequence-based access.

In the complete canonical NQ native + common public 4B text-scoring pipeline, BSI-Local/BSI-Global generation costs 0.13/2.25 seconds and full p50 is 11.39/15.88 seconds; Dense/BSI-Direct full p50 is 12.26/12.17 seconds (Table 28 (p. 39)). Serial QPS is 0.087/0.063 for BSI-Local/BSI-Global and 0.082/0.083 for Dense/BSI-Direct. Timings include encoding through CPU-ID return, exclude loading and use mean duration for QPS. The lower generation overhead favors BSI-Local when efficient query adaptation is the priority.

The compact BSI-Direct deployment stores binary document codes and ranks them with continuous queries, with process GPU peaks of 1.850/1.873 GiB for BSI-Direct/Dense. For the MS hybrid pipeline, the complete retrieval-state ledger is 1.984 GiB access + 134.982 GiB vector cache = 136.966 GiB, with model weights and runtime buffers additional (Table 27 (p. 39)). Reporting these components alongside the quality and timing results specifies the resources used by each deployment.

AuthorsOpenReview ↗

Official Comment by Authors

Q2. What do the hashing and compression controls show?

The fixed-encoder study holds continuous features constant and compares how alternative methods compress and search them. Asymmetric BSI-Direct achieves 87.14/69.80 R10/MRR10, compared with BPR+EMA's 84.55/65.80 and OPQ ADC's 86.48/68.54 (Table 17 (p. 30)). The table also includes RP, ITQ and PQ. Supervised binary heads share TRAIN/CAL data; the other compressors fit unlabeled corpus vectors, and Hamming/asymmetric scoring is identified for each result. Together with W2's encoder-adapted comparison, this provides evidence for the learned binary construction at both the representation-learning and fixed-feature levels.


Q3. Consistent protocols and component attribution

NQ. Table 1 (p. 7) compares BSI with 25 external methods. BSI evaluates all 7,830 official query IDs/texts/qrels on the 109,739 released documents, achieving native BSI-Local/BSI-Global R10 of 89.87/87.85 without text scoring or Dense insertion. BSI-Global resolves frozen canonical-trie proposals through newly built released postings. Appendix E.2 (from p. 26) specifies the released-corpus evaluation, the 109,650-document canonical control and the protocols of the published baselines.

MS MARCO. Table 2 (p. 7) includes TSGen, LTRGR, DGR, RIPOR, PAG, DOGR and PAG reproduction. Document R10 averages the fraction of relevant passages in Top10; MRR10 averages the first relevant reciprocal rank within ten. All 6,980 queries enter both denominators, including misses. Reconstructing the complete BSI-Local system gives 68.82/41.98; BSI-Global reports 71.67/41.23. With the same text-only scorer applied to each method's candidate pool, Dense/BSI-Direct/BSI-Local gives 69.61/40.21, 69.16/40.05 and 66.26/38.84 (Table 7 (p. 26)).

BEIR now uses one complete thirteen-task policy per row. Holding the encoder and candidate pools fixed, equal native/text weighting gives 63.91 macro nDCG@10, compared with 62.97 at 1:3 using the same terminal 8B scorer. The source-selected 8B scorer reaches 65.02 under the common 1:3 rule (Table 24 (p. 35)). A prespecified public4B/context2048 text-only run gives 62.76 on the same native100 pools, with all four public scoring rules included in that table.

Representation transfer. Native 8B BSI-Direct retains 97.4% of matched Dense with native cosine ordering and the vector cache: macro nDCG@10 is 62.23 versus 63.86; adding the fixed 8B text scorer gives 62.97 (Table 3 (p. 8)). The 0.6B native configuration gives 41.37, including 21.14 on Touché. Completing Touché with the same MS-selected 8B text model used on the other tasks gives 33.00 and a full BEIR-13 mean of 56.86, under fixed 1:3 fusion. Our runs use original qrels and linear gain; SSR and other published baselines retain their source conditions.

The expanded MAIR study provides a matched Dense comparison across six tasks. Using the same public 4B model, context 2048 and Top200 text-only scoring, BSI-Direct achieves 45.18 versus Dense's 44.27 macro nDCG@10; the original shared-scoring configuration gives 38.92/38.91 (Table 4 (p. 9)). Table 25 (p. 37) and Table 26 (p. 38) report both gain definitions and all 36 paired configurations across the tested model, context and candidate-depth settings.

Controlled comparisons and selection. The canonical common-scorer comparison (Table 8 (p. 27)), text/insertion interventions (Table 12 (p. 28)), and released TSGen/SEAL comparison (Table 15 (p. 30)) connect document access to retrieval quality under specified scoring conditions. Current NQ selection uses TRAIN/CAL, and the reported comparisons fix their execution policies. Earlier development included evaluation on the NQ holdout and BEIR targets; original MS settings used development judgments.


Thank you again for the thoughtful review. The revised comparisons make BSI's practical contribution concrete: retrieval-aware binary quality at compact state, indexed access through a common code vocabulary, and measurable query improvements while document codes and postings remain fixed.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer vAVh,

Thank you again for your time and thoughtful feedback. We have provided point-by-point responses to your concerns and updated the manuscript with corresponding clarifications and additional experimental evidence.

As the discussion period approaches its close, we would be grateful to hear whether these responses address your concerns, and whether any questions remain. We would be happy to provide further clarification during the remaining discussion period.

Thank you for considering our responses and revisions.

Best regards,

The Authors

Reviewer_dA3WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript proposes Binary Semantic Indexing (BSI) - for learning shared binary document representations that support sparse indexing, direct similarity based retrieval and generative retrieval. The authors train a encoder jointly with a binary spherical quantization (BSQ) module to produce compact document codes which are divided into tokens and token pair addresses to construct an inverted index. They present two generative retrieval interfaces - Local and Global - with Local directly predicting index addresses and global autoregressively predicting complete document codes. The authors evaluate BSI on various datasets and compare the retrieval performance against existing retrieval methods. They also investigate representation quality under quantization, code length, index access budgets and retrieval quality.

Critical strengths

Interesting Problem formulation - The authors lay out an interesting problem of formulating a representation learning system that caters to all three retrieval mechanisms - sparse, dense and generational retrieval. This is a worthwhile direction as it could reduce the need for maintaining separate representations for different retrieval mechanisms

Unified representations - The manuscript shows that a single compact binary document representation can support direct similarity search and two different generative retrieval interfaces through a shared index. This provides a prof that such a unification is possible and feasible.

Critical weaknesses

Insufficient Justification, missing baselines - The authors study an interesting problem of unification of representations across sparse and generative retrieval, but they fail to establish why existing representations cannot be adapted to provide this functionality. There is som much literature on all three aspects - dense retrieval, sparse retrieval and generative retrieval - it would hav been very interesting to see how can one adapt representations from one/all of these retreival systems to serve as valid representations in the other retrieval systems, and use such adaption as baselines to motivate the need for learning a combined representation system across the two retrieval modes. L140-L142 mention that SEAL and TSGen does achieve some level of unification across index-based retrieval and generative retrieval, and hence they should serve as baselines under matched circumstances (see next weakness) to be compared against this system or have a meaningful discussion on why such an adaptions would be insufficient. In the absence of matched comparisons or adapted baselines, it's hard to assess BSI in a rich field of retrieval systems.

Unmatched comparisons - the experiments on BSI do not sufficiently establish the practical advantages as : 1. Tables 1, 2 cannot establish reliable relative performance as they are evaluated on either different splits or their definition of the Recall metric is different. 2. Evaluation of retrieval systems is done on various axes like latency, storage space, quality, etc. The authors explore some of them in Section 4.4 and 4.5 - but the manuscript lacks a systematic comparison of latency-storage-quality between different baselines and BSI on the different query mechanisms they study. 3 The main related-work section does not include a dedicated discussion of dense retrieval. While Table 3 includes dense retrieval baselines, it compares complete retrieval systems and not isolating the quality of the underlying dense and binary representations directly. BSI further uses target aware per dataset BSI selection, making the comparison further muddy. The direct dense-versus-binary comparison is limited to six MAIR tasks, making it unclear how well the learned representations generalize.

Presentation - 1. The authors suggest that document organisation can be decoupled from choice of query mechanism, making it feel like the method would work for any query mechanism, while the evaluations are centered only around a limited set of direct binary and generative query interfaces, this scope should be explicitly mentioned in the manuscript. 2. Recent works on hierarchical and alternative dense representation approaches could be cited and possibly compared against, as they explore representation structures and efficiency trade-offs relevant to BSI. 3 Section 3.1 introduce entropy and decorrelation losses, would be beneficial to have an analysis on how these losses aid in learning better representations by doing ablations with and without these, to establish if they are indeed needed for the method to work. 4 Motivation for having both local and global learned retrieval mechanisms is not very clear, and the cost-latency-performance trade-offs between these versions, which use different underlying hyperparameters, are not systematically explored through controlled comparisons, making it unclear when to use which system.

Questions for authors
  1. Motivation for the proposed representation: What limitations prevent existing dense or generative retrieval representations from being adapted to support the same query interfaces as BSI? Have the authors explored such adaptations as baselines?

  2. Matched retrieval comparisons: Can the authors provide comparisons against representative existing generative retrieval methods using the same query splits and metric definitions and comment if this would affect the conclusions drawn from Tables 1 and 2?

  3. Efficiency comparisons: How does BSI compare against existing sparse, dense, and generative retrieval systems in terms of retrieval quality, end-to-end storage, and query latency under comparable resource budgets?

  4. Local versus Global generation: What are the practical benefits of supporting both generative interfaces? Under comparable inference budgets, when does one interface outperform the other?

5 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for recognizing the importance of a shared retrieval representation. Your concerns call for adapted baselines, consistent comparisons and a clearer account of the query interfaces. We have added matched binary-learning and fixed-feature comparisons, released-generator controls, and joint quality–storage–latency measurements. The revised manuscript uses these studies to explain how retrieval-trained coordinates become both index addresses and learnable query targets, and how query access improves over fixed document organization (Sections 2–4 and Appendices A and E–H). Scores are percentages; R10 and MRR10 denote Recall@10 and MRR@10.


W1. The shared representation and comparisons with adapted alternatives

BSI connects representation learning to the operations used to retrieve documents. Continuous and binary relevance supervision shape a common set of semantic coordinates. Lossless packing exposes those coordinates as tokens: token pairs become posting keys, complete sequences become BSI-Global targets, and local addresses become BSI-Local targets. The BSQ inner-product/Hamming relation links code comparison to this binary geometry. The same learned coordinates therefore support similarity, indexed lookup and query prediction, while fixed codes and postings allow subsequent query adaptation. This motivates comparisons at the representation, compression and access levels.

We evaluate representation learning, fixed-feature compression and generated access in complementary comparisons. First, Table 5 (p. 9) starts Dense, BPR, BPR+EMA and BSI-Direct from the same dense-only parent. Binary arms share the backbone, 256-bit payload, TRAIN/CAL data, encoder/query search budgets and retrieval/scoring procedure; each selected encoder gets a freshly encoded corpus:

SystemR10MRR10State (MiB)p50 (ms)
Dense FP1691.3678.85221.0034.71
BPR88.1373.1311.1934.93
BPR+EMA88.2073.5711.1935.60
BSI-Direct88.9374.2711.1935.37

At the same binary-state size, BSI-Direct improves over BPR+EMA by +0.72 R10 points, with a 95% paired-query CI of [0.44, 1.01], and +0.70 MRR10 points [0.38, 1.03]. Three seeds are averaged within each query before resampling. BPR uses its published objective adapted to the common Qwen backbone. Reporting quality, state and latency for these same systems shows the benefit of retrieval-aware binary learning at matched storage and execution conditions.

The fixed-feature study then compares RP, ITQ, PQ, OPQ and supervised binary heads using the same continuous features, with their Hamming or asymmetric scoring rules identified (Table 17 (p. 30)). Released TSGen/SEAL and BM25/SPLADE comparisons examine alternative document-access mechanisms with common relevance scoring (Table 15 (p. 30); Table 16 (p. 30)). These controls connect the learned code's quality to the different ways a query can access documents.

AuthorsOpenReview ↗

Official Comment by Authors

W2. Fair comparisons and representation transfer

The BEIR comparison now applies one complete policy per row across all thirteen tasks. Keeping the encoder and candidate pools fixed, equal native/text weighting gives 63.91 macro nDCG@10, compared with 62.97 at 1:3 using the same terminal 8B scorer. The source-selected 8B scorer gives 65.02 under the common 1:3 rule (Table 24 (p. 35)). A prespecified public4B/context2048 text-only run gives 62.76 on the same native100 pools; the table includes all four public scoring rules.

The expanded MAIR study compares Dense and BSI-Direct with the same text models, context lengths and candidate depths across all six tasks. With the public 4B model, context 2048 and Top200 text-only scoring, BSI-Direct reaches 45.18 versus Dense's 44.27 macro nDCG@10; the original shared-scoring comparison gives 38.92/38.91 (Table 4 (p. 9)). Both gain definitions and all 36 paired configurations are reported in Table 25 (p. 37) and Table 26 (p. 38).

The NQ main comparison evaluates all 7,830 official queries on the released 109,739-document corpus, with native BSI-Local/BSI-Global R10 of 89.87/87.85 (Table 1 (p. 7)). It aligns query IDs, raw texts and qrels to the release; BSI-Global resolves frozen canonical-trie proposals through rebuilt released postings (Appendix E.2 (from p. 26)). A control on the canonical 109,650-document corpus holds document identity and scoring fixed across the BSI interfaces. Q2 gives the matched generative comparisons and MS metrics; Q3 reports resource use.

The native 8B comparison measures the relevance quality retained through binary access: BSI-Direct gives 62.23 macro nDCG@10 versus Dense's 63.86, retaining 97.4% with native cosine ordering and the vector cache; adding the fixed 8B text scorer gives 62.97 (Table 3 (p. 8)). The 0.6B encoder gives 41.37 natively, including 21.14 on Touché. Using the same MS-selected 8B text model and fixed 1:3 fusion across tasks gives 33.00 on Touché and 56.86 over the complete collection. Each mean equally weights all thirteen tasks using original qrels and linear gain. ColBERTv2, SPLADE-v2/v3, SSR-tok/CLS and large-model baselines retain their published evaluation conditions.


W3. Presentation, related work, objectives and interface choice

Section 3 (p. 4) defines BSI-Direct, BSI-Local and BSI-Global over the shared document space. Section 2 and Appendix A (from p. 17) discuss dense/compressed retrieval, including DPR, ANCE, ColBERTv2, PLAID and BPR, and hierarchical alternatives such as MEVI and GDR.

For the requested entropy/decorrelation ablations, we use a common parent, matched training and freshly encoded corpora (Table 18 (p. 30)). With both regularizers, Hamming R10/MRR10 is 83.21/64.46; removing both gives 83.48/64.34. Asymmetric R10 is 87.74/87.73, respectively. These results characterize sensitivity to regularization within retrieval-supervised code learning. The matched binary study in W1 evaluates the resulting representation against BPR-based alternatives, while the fixed-index experiment evaluates query-head refinement with the scorer held constant (Table 6 (p. 10)). Q4 compares the quality and generation cost of the two generative interfaces.

AuthorsOpenReview ↗

Official Comment by Authors

Q1. Why learn this representation, and have adaptations been tested?

Yes. The matched BPR-based experiments adapt an established binary-retrieval objective to the common backbone, and the fixed-encoder study compares hashing and quantization on shared continuous features. Under asymmetric scoring, BSI-Direct gives 87.14/69.80 R10/MRR10, compared with BPR+EMA's 84.55/65.80 and OPQ ADC's 86.48/68.54 (Table 17 (p. 30)). Supervised binary heads share TRAIN/CAL data; RP/ITQ/PQ/OPQ fit unlabeled vectors. These results motivate the learned binary construction, whose coordinates also provide the index addresses and prediction targets used by BSI's query interfaces.

The shared-space design also permits query updates while keeping document codes, postings and budgets fixed. With the same additional query scorer, BSI-Local refinement improves 90.28/77.50 to 90.57/77.65 (Table 6 (p. 10)). Paired-query gains are +0.294 R10 [0.145, 0.456] and +0.154 MRR10 [0.035, 0.281].

The released-corpus experiment selects β=1\beta=1 on all 15,324 CAL queries and then fixes it for evaluation. On identical refined-head pools, native versus adapted queries give 90.10/75.54 versus 90.49/77.00 (Table 22 (p. 33)), gains of +0.40 R10 [0.17, 0.63] and +1.46 MRR10 [1.18, 1.76]. The first control learns address access with a fixed scorer; the second learns query evidence on fixed candidate pools. Together they show how the shared representation supports useful query updates while retaining document codes and postings.


Q2. Consistent generative comparisons and main-table conclusions

We have added a released-model comparison alongside the NQ main table's 25 external methods. TSGen, SEAL and the direct retrievers use the same official queries, released documents and qrels. Each access mechanism produces a candidate pool, which is then evaluated with its native scorer, common cosine scoring, and cosine plus text scoring (Table 15 (p. 30)):

Access methodNative R10Common cosine R10Cosine + text R10
TSGen87.9991.3591.80
SEAL15.4990.8093.09
BSI-Direct90.8390.8393.73
Dense90.8290.8293.97

The pools contain 18–311 candidates for TSGen, 1,500 for SEAL and 2,000 for BSI-Direct/Dense. Common cosine uses the encoder and vector cache; the final column additionally uses the text model. This comparison shows how each access pool supports a common relevance scorer. Table 8 (p. 27) provides the corresponding fixed-corpus, common-scorer comparison across all BSI interfaces.

For MS, let Q\mathcal Q be all 6,980 queries, P(q)P(q) their relevant passages, R10(q)R_{10}(q) the first ten unique retrieved IDs, and rqr_q the first relevant rank (rq=∞r_q=\infty for a miss). We use:

= \frac{1}{\lvert\mathcal Q\rvert} \sum_{q\in\mathcal Q} \frac{\lvert R_{10}(q)\cap P(q)\rvert}{\lvert P(q)\rvert}.$$ $$\mathrm{MRR@10} = \frac{1}{\lvert\mathcal Q\rvert} \sum_{q\in\mathcal Q:\,r_q\leq 10} \frac{1}{r_q}.$$ Misses contribute zero to both metrics. Reconstructing the complete BSI-Local system gives **68.82/41.98**; BSI-Global reports **71.67/41.23** (Table 2 (p. 7)). Both improve on source-reported PAG **67.00/38.50** and PAG reproduction **67.10/38.60** in the full-system comparison, which also includes TSGen, LTRGR, DGR, RIPOR and DOGR. Applying the same text-only scorer to each method's candidate pool gives Dense/BSI-Direct/BSI-Local **69.61/40.21, 69.16/40.05 and 66.26/38.84** (Table 7 (p. 26)), identifying candidate-access performance under shared scoring. Current NQ selection uses TRAIN/CAL, and the revised evaluations use fixed execution policies. Earlier development included evaluation on the NQ holdout and BEIR targets; original MS settings used development judgments.
Reviewer_weYeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Binary Semantic Indexing (BSI), which lets sparse retrieval and generative retrieval share a single binary semantic space. A query encoder and a Binary Spherical Quantizer (BSQ) are trained jointly to turn each document into a compact 256-bit binary code; the code is packed into 32 tokens, and an inverted index is built with token pairs as keys. With the document codes and index fixed, a query accesses the index in one of three ways—direct encoding (Direct), complete-code generation (Global), or local-address generation (Local)—to find candidate documents, which are then ranked by a common scoring rule and a reranker. The paper reports competitive performance on NQ, MS MARCO, and BEIR together with a small index (1.98 GiB on MS MARCO), and further analyzes the trade-off between code length and address budget.

Critical strengths
  1. Novel indexing unit and decoupling of document organization from query modeling.
  • Using token pairs of a binary code, learned jointly with the query encoder and BSQ, as inverted-index keys is novel. In addition, because the document codes and index are fixed first and only the query interface is varied, different query mechanisms (direct encoding, complete-code generation, local-address generation) can be compared over the same document representation. This perspective is the paper's main conceptual contribution.
  1. A single representation serving multiple roles.
  • The same binary code provides a similarity space (Hamming distance), index keys usable as addresses (token pairs), and prediction targets (tokens), tying a sparse index and a generative model to one representation.
Critical weaknesses
  1. Limitations of the evaluation make it hard to verify the method's effectiveness.
  • A. NQ (Table 1): BSI's numbers (Local 94.28, Global 94.15) are measured on the calibration queries (15,324) that were also used for model selection, while the baselines use the 7,830-query test set (the caption notes the comparison is not matched). BSI's numbers also include an 8B reranker and dense-retrieval mixing (Appendix E.3) that the baselines lack. On the holdout, Local reaches 90.22 (embedding ordering only), just 0.5 points above NOVO (89.70), so it is hard to tell whether the 4+ point gap comes from the method itself. Scoring alone shifts Recall@10 by about 10 points (Tables 7 and 8).

  • B. BEIR (Table 3): Configurations were selected per dataset using target judgments. The uniform configuration scores 67.43, 2.3 points below the selected one (69.72) (Table 13); re-computed over the same 12 datasets, SSR-CLS is effectively on par with the uniform configuration. Moreover, the “0.6B” BSI can also use external text models of up to 8B, so the comparison does not isolate encoder capacity.

  1. The claimed benefits (usefulness of the generative interface, efficiency) are insufficiently supported.
  • A. In Table 7 (MS MARCO), Direct (Recall@10 72.95, MRR@10 43.36) outperforms Local (69.86, 41.98) and Global (71.67, 41.23), so the practical benefit of the stated contribution that “a generative model can serve as a query interface to a semantic index” is not clearly demonstrated. The paper shows that the generative interface is feasible, but not yet that it is useful; the “complementary” claim in the abstract is likewise made without per-query analysis.

  • B. The “32 bytes per document, 64× smaller” claim in Section 4.4 refers to the code payload only. According to Table 15, the access structure takes 240.95 B per document, and the FP32 embedding cache used for final ranking (134.98 GiB) is needed in addition. The comparison with SSR's 18.5 GB would be fairer if this were taken into account.

Questions for authors
  1. On the NQ holdout (7,830 queries) not used for model selection, what is the performance when the reranker and the dense mixing are each removed (ablation), with no further tuning? Is this holdout identical to the test set used by prior methods? If the dense mixing is kept, please specify the dense retrieval setup (encoder, retrieval target), the criterion for selecting the added documents, and their insertion rank. Please also state which queries were used for model selection on MS MARCO and BEIR.

  2. Can the results on all 13 BEIR datasets be reported under a single uniform configuration? Why was Touché excluded from Table 13?

  3. What is the latency (online query latency and throughput)? What is the cost of Global/Local generation relative to Direct (beam 2,048; 0.66–1.29M posting reads per query), and what is the total system memory including the embedding cache?

6 comments
AuthorsOpenReview ↗

Official Comment by Authors

Thank you for recognizing the value of a shared document representation. We address your concerns about evaluation fairness, generative access and full system costs with official-query comparisons, controlled query updates and measured resource use. The revised manuscript connects these results to BSI's central contribution: learning one binary document space that supports compact retrieval, indexed generation and query adaptation over a fixed index. Section 4 and Appendices E–H present the evidence discussed below. Scores are percentages; R10 and MRR10 denote Recall@10 and MRR@10.


W1. Consistent benchmark comparisons and component attribution

NQ and MS MARCO. We have updated the NQ main comparison to the official test protocol. Table 1 (p. 7) evaluates all 7,830 official queries against the 109,739 released documents, with query texts, document IDs and positive qrels aligned to that release. BSI-Local and BSI-Global achieve 89.87/87.85 R10 using native ranking, without text scoring or Dense insertion. A matched control on the canonical 109,650-document corpus compares the three BSI interfaces with the same corpus and scoring conditions (Table 8 (p. 27)).

For MS MARCO, both metrics average over all 6,980 development queries: document R10 measures the fraction of relevant passages retrieved, and MRR10 uses the first relevant rank, with misses contributing zero. Reconstructing the complete BSI-Local ranking gives 68.82/41.98; BSI-Global reports 71.67/41.23 (Table 2 (p. 7)). Table 7 (p. 26) adds a common text-only comparison on each method's candidate pool. The main tables retain the broad external-baseline comparison, with source variants and evaluation settings documented in the appendix.

BEIR. Table 3 (p. 8) reports a single policy per BSI row over all thirteen tasks. Encoder scale and additional text-scoring models are identified separately; Q2 gives the complete results.


W2. What the framework provides and what generative access adds

BSI makes the learned document code useful at three connected levels. Retrieval supervision gives the binary coordinates semantic structure; lossless packing turns those coordinates into tokens and posting keys; query models learn to predict codes or addresses in that same vocabulary. The practical benefit is that query access can improve while document codes and postings are reused. BSI-Direct provides compact retrieval from this representation, while BSI-Global and BSI-Local supply learned query interfaces. The following controls measure the gains from adapting that query interface.

First, we refine the BSI-Local head while holding the document codes, postings and additional query scorer fixed. Both configurations use 4,096 addresses and 2,000 candidates (Table 6 (p. 10)):

Query interfaceR10MRR10
BSI-Local (Original)90.2877.50
BSI-Local (Refined)90.5777.65

Head refinement improves R10 by +0.294 points, with a 95% paired-query CI of [0.145, 0.456], and MRR10 by +0.154 points [0.035, 0.281]. Three seeds are averaged within each query before resampling. These gains come from updating query access while reusing the document representation and postings; both configurations use native scoring without text scoring or Dense insertion.

A complementary released-corpus experiment fixes refined-head candidate pools and compares native versus adapted queries (Table 22 (p. 33)). A seven-strength check on all 15,324 CAL queries retained β=1\beta=1 before evaluation:

Query configurationR10MRR10
BSI-Local (Refined)90.1075.54
BSI-Local (Refined + query scorer)90.4977.00

The scorer adds +0.40 R10 points [0.17, 0.63] and +1.46 MRR10 points [1.18, 1.76], using the native vector cache and fixed document codes/postings. The two controls address complementary parts of query modeling: the first learns which addresses to access with the scorer fixed; the second adapts query evidence on identical candidate pools. Together, they demonstrate effective query learning over unchanged document organization. The BEIR and MAIR experiments extend the evaluation to frozen configurations across collections.

The matched binary deployment also gives BSI-Direct 11.19 MiB persistent state versus Dense FP16's 221.00 MiB, a 19.8-fold reduction, with R10 88.93/91.36 (Table 5 (p. 9)). At the same binary-state size, BSI-Direct exceeds BPR+EMA by +0.72 R10 [0.44, 1.01] and +0.70 MRR10 [0.38, 1.03]. Q3 gives process-level and hybrid-pipeline costs.

AuthorsOpenReview ↗

Official Comment by Authors

Q1. Holdout ablations, corpus identity, insertion and model selection

Table 8 (p. 27) fixes the canonical corpus, 7,830 holdout queries/qrels and scoring conditions, with Dense insertion disabled:

Scoring conditionDenseBSI-DirectBSI-LocalBSI-Global
Native cosine R1091.0991.0390.2089.49
Common text fusion R1093.9893.5592.1291.21

The original-source ablation independently removes final text fusion and bounded insertion (Table 12 (p. 28)):

InterfaceNativeTextNative + gateText + gate
BSI-Local90.2092.5892.0194.39
BSI-Global89.6291.9292.4994.09

Each row keeps its source pipeline's scoring depth, including the BSI-Global native result of 89.62. The insertion gate preserves ranks 1–8 and inserts at 9/10. Exact FP32 Dense retrieval uses the joint encoder; BSI-Global draws from Dense Top20. BSI-Local excludes native Top100, text-scores the next 100 with the trained 8B model, selects 20 using cosine:text z-scores 1:3, and applies public-text margins log⁡3\log 3 and zero. Here the gate uses text scores for candidate admission, while the Text column denotes final ranking fusion. Appendix F (from p. 28) additionally reports the fully text-free Dense2 control.

The main released-corpus evaluation uses the same raw query texts as the canonical holdout, with documents, IDs and qrels aligned to the released namespace. BSI-Global resolves 2,048 frozen canonical-trie proposals through newly built released postings (Appendix E.2 (from p. 26)). Published baselines are grouped by their documented corpus and evaluation protocol.

Current NQ selection uses TRAIN/CAL, and the final query strength and each BEIR policy are fixed for evaluation. Earlier development included evaluation on the NQ holdout and BEIR targets; the original MS settings were selected using development judgments.


Q2. Complete BEIR-13 and Touché under one policy

All BEIR rows now cover the complete thirteen-task collection under one policy per row. With the encoder and candidate pools fixed, equal native/text weighting gives 63.91 macro nDCG@10, compared with 62.97 at 1:3 using the same terminal 8B scorer. The source-selected 8B scorer gives 65.02 with the common 1:3 rule (Table 24 (p. 35)). A prespecified public4B/context2048 text-only run gives 62.76 on the same native100 pools; all four public scoring rules are reported in that table.

The matched MAIR study provides additional cross-collection evidence. With the same public 4B model, context 2048 and Top200 text-only scoring, BSI-Direct reaches 45.18 versus Dense's 44.27 macro nDCG@10 (Table 4 (p. 9)). The original shared-scoring comparison gives 38.92/38.91. Table 25 (p. 37) and Table 26 (p. 38) report both gain definitions and all 36 paired configurations across the tested model, context and candidate-depth settings.

Table 3 (p. 8) and Table 24 (p. 35) report all thirteen datasets and their equal-task means:

Evaluation conditionMacro nDCG@10Touché nDCG@10
0.6B BSI-Direct, native41.3721.14
0.6B BSI-Direct, source-selected 8B text model56.8633.00
8B Dense, native63.8632.45
8B BSI-Direct, native62.2332.51
8B BSI-Direct, additional fixed 8B text scorer62.9727.84
8B BSI-Direct, equal native/text fusion63.9130.89
8B BSI-Direct, source-selected text model65.0236.22
8B BSI-Direct, public4B text-only62.7634.49

Touché is now included in both 0.6B conditions. Native retrieval encodes all 382,545 documents and evaluates all 49 judged queries. Applying the same MS-selected 8B text model used for the other twelve tasks gives 33.00 on Touché and a complete BEIR-13 mean of 56.86, with fixed native:text weights 1:3. Our runs use the original BEIR qrels and linear gain; Table 23 (p. 34) provides the task-level breakdown.

The main table presents ColBERTv2, SPLADE-v2/v3, SSR-tok/CLS and BSI in a continuous method-by-dataset comparison. With native cosine ordering and the vector cache, 8B BSI-Direct retains 97.4% of the same encoder's Dense macro score. This cross-collection result complements the fixed-index query-learning gains in W2, connecting the shared representation's transfer quality to the adaptability of its query interface.

AuthorsOpenReview ↗

Official Comment by Authors

Q3. Latency, throughput and full costs

BSI-Local's direct address prediction gives 0.13 seconds of generation time, compared with 2.25 seconds for BSI-Global. The complete pipeline measurements below include their shared downstream work, using canonical NQ and the same public 4B text scorer on an isolated A100 (Table 28 (p. 39)):

InterfaceEnd-to-end p50 (s)Serial QPSGeneration (s)
Dense12.260.082—
BSI-Direct12.170.083—
BSI-Local11.390.0870.13
BSI-Global15.880.0632.25

Timings cover encoding, generation/access, ordering, text scoring and CPU-ID return, after model loading. QPS uses mean request duration. BSI-Local has the lowest end-to-end p50 in this comparison, pairing efficient address prediction with the fixed-index adaptation demonstrated above. Table 21 (p. 32) reports posting reads and candidate coverage as access budgets vary.

The compact deployment ranks continuous queries against stored binary documents and requires no full-precision document-vector cache. The binary-only BSI-Direct/Dense study separately measures 35.37/34.71 ms p50 and 1.850/1.873 GiB process GPU peaks. The MS hybrid ledger includes 1.984 GiB access + 134.982 GiB vector cache = 136.966 GiB, with model weights and runtime buffers additional (Table 27 (p. 39)). The matched Dense study quantifies retrieval-state compression; the resource appendix also lists SSR's source-reported 18.5 GB persistent index.


Thank you again for the constructive questions. The revisions connect the shared-space design to three measured benefits: compact binary retrieval, effective generated access and query improvements that reuse an existing document index.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Author Statement: Binary Semantic Indexing and Its Contributions – Submission 5697

Dear AC, SAC, PCs, and Reviewers,

We sincerely thank all reviewers for their thoughtful feedback and the AC, SAC, and PCs for their time and effort throughout the review process. In response, we have substantially strengthened our experiments, added comprehensive controlled comparisons, and further clarified the motivation and contributions of our work. We would like to highlight the central idea of BSI and the key evidence from our revisions.

1. Motivation and Method.

Our work addresses a fundamental question in information retrieval: Can a unified semantic representation support multiple retrieval mechanisms while enabling flexible query adaptation over a shared document index?

We introduce Binary Semantic Indexing (BSI), a framework that connects semantic representation learning, sparse indexing, and generative retrieval through a shared binary semantic space. By combining retrieval-aware learning with Binary Spherical Quantization (BSQ) and lossless token packing, BSI transforms binary document representations into both searchable index addresses and generative prediction targets. It supports three complementary query interfaces: BSI-Direct, BSI-Global, and BSI-Local, all operating over the same underlying document representation.

2. Addressing the Reviewers' Main Concerns.

We conducted additional experiments to strengthen the empirical evidence for representation quality, generative access, and system efficiency.

  • Matched binary baselines. Under matched training and retrieval settings, BSI-Direct achieves 88.93 Recall@10, outperforming BPR+EMA (88.20) at the same 11.19 MiB retrieval-state size. Additional comparisons against RP, ITQ, PQ, OPQ, and BPR further demonstrate the effectiveness of retrieval-aware binary learning.
  • Fair evaluation and broader benchmarks. We added official NQ evaluation on all 7,830 queries, controlled comparisons under identical corpus and scoring conditions, and expanded BEIR evaluation to all 13 datasets, including Touché. We also added matched comparisons with released generative retrieval models, including TSGen and SEAL.
  • Generative access and query adaptation. With document codes and postings fixed, BSI-Local head refinement improves Recall@10 from 90.28 to 90.57. Query-scorer adaptation further improves Recall@10 from 90.10 to 90.49 and MRR@10 from 75.54 to 77.00. These results demonstrate effective query adaptation without reindexing.
  • Retrieval efficiency. BSI-Direct reduces persistent retrieval state from 221.00 MiB to 11.19 MiB, achieving a 19.8× reduction compared with Dense FP16, while maintaining comparable query latency. We further provide comprehensive measurements of generation cost, end-to-end latency, throughput, and system memory.

3. Key Findings and Contributions.

Our experiments establish three key findings. First, retrieval-aware binary learning produces high-quality semantic representations with substantially reduced storage requirements. Second, a shared binary semantic space naturally connects similarity-based retrieval, inverted-index lookup, and generative prediction. Third, query interfaces can be effectively adapted to improve retrieval while keeping the underlying document representation and index unchanged.

Together, these findings establish a unified retrieval framework in which one learned binary semantic space connects how documents are represented, indexed, and accessed.

4. Closing Remarks.

We are grateful for the reviewers' constructive feedback, which has helped us further strengthen our empirical evidence and clarify the significance of our contributions. We sincerely hope the final assessment will consider the motivation, unified representation and indexing framework, extensive controlled experiments, and demonstrated practical benefits of BSI as a whole.

Thank you again for your time and consideration.

Best regards,

The Authors of Submission 5697

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.