Decoupled Multi-Codebook Quantization

Multi-codebook quantization commonly ties code selection to codebook geometry, although individually nearest codewords need not form the best joint representation. We propose Decoupled Multi-Codebook Quantization (DMQ), which separates what codewords represent from how they are selected. A learned router coordinates assignments without input–codeword distance matching, unifying parallel and residual quantization. Differentiable training jointly optimizes assignments and codebooks while accommodating downstream objectives. We establish the potential suboptimality of independent Euclidean assignment, prove strictly greater assignment expressivity under suitable capacity assumptions at unchanged code budgets, and derive a sufficient condition for initialization-independent global convergence of Iterated Conditional Modes. Experiments on four million-scale benchmarks demonstrate improvements over representative baselines in reconstruction and retrieval quality, while maintaining efficient inference. These results establish decoupled assignment as an expressive and effective design principle for learning compact discrete representations. Our implementation is anonymously provided at https://anonymous.4open.science/r/DMQ.