Binary Semantic Indexing for Generative Retrieval

Retrieval systems must organize large collections of knowledge while allowing queries to reach relevant information efficiently. As retrieval increasingly relies on learned representations and generative models, the representation itself becomes central to both semantic modeling and access. This raises a broader question: can different retrieval mechanisms share a common semantic foundation? We introduce **Binary Semantic Indexing (BSI)**, a framework that connects sparse indexing and generative retrieval through a **shared binary semantic space**. A jointly trained semantic encoder and binary spherical quantizer compress each document into a compact code that preserves retrieval information. The code's local components form addressable units in an inverted index. Complete-code and local-address generators provide complementary query interfaces, while direct query encoding accesses the same representation. By **separating document organization from query modeling**, BSI allows these mechanisms to operate over a common document space. Experiments show effective retrieval across query interfaces and domains, while controlled analyses characterize the tradeoffs between code capacity, index access, and retrieval quality.