- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
DMQ separates multi-codebook representations from the rule used to select their codewords. A learned assignment network coordinates choices across codebooks, with parallel and residual variants trained using soft assignments and reconstruction objectives. Discrete inference uses direct assignments with ICM refinement or beam search. The theoretical results discuss limitations of independent nearest-codeword assignment, the expressivity of learned assignment functions, and a sufficient condition for initialization-independent ICM convergence. Experiments evaluate reconstruction and retrieval on four vector benchmarks at two code budgets, with component and inference-strategy ablations.
Critical strengths
Coordinating assignments across codebooks is a meaningful problem, and the parallel/residual formulation offers a useful way to study it. Table 1 contains substantial improvements in some settings: on BigANN1M at 16 bytes, Parallel DMQ reduces MSE from 0.98 for LSQ to 0.75 while improving retrieval. This is a positive result worth understanding.
The component and inference ablations are helpful. In particular, Table 3 exposes the contribution of post-processing rather than hiding it inside the final system, and Table 2 shows that initialization, attention, and soft assignments affect performance.
Critical weaknesses
There is an unresolved inconsistency between the stated refinement and Table 3. Equation (23) minimizes reconstruction error over one codebook while fixing the others. With fixed codebooks and the same initialization, each exact update must satisfy . Yet Residual DMQ changes from MSE 1.96 with Argmax to 2.02 with Argmax + ICM. Equation (23) is introduced for the parallel variant, so the residual experiment may use a different procedure, but that difference is not specified. This needs an explicit explanation before the refinement results can be interpreted.
The benefit attributable to learned assignment is not sufficiently separated from codebook learning and subsequent search. For Parallel DMQ in Table 3, Random + ICM achieves MSE 1.78 in 17.0 seconds, essentially matching Argmax + ICM at 1.77 in 18.4 seconds. This does not negate the quality of the learned codebooks, but it weakens the case for the learned router at inference. The theoretical comparisons do not settle this issue: independent nearest-codeword assignment is a narrower comparator than joint search, and the margin condition in Theorem 4.3 is not shown to hold for the fitted models. The novelty discussion should also account for Shu and Nakayama's "Compressing Word Embeddings via Deep Compositional Code Learning" [1], which already learns neural multi-codebook assignments jointly with compositional representations.
[1] Shu, R., & Nakayama, H. (2017). Compressing word embeddings via deep compositional code learning. arXiv preprint arXiv:1711.01068.
Questions for authors
-
What exactly is the residual Argmax + ICM procedure in Table 3? Does it optimize the same fixed-codebook objective as Equation (23), and how can its reconstruction error increase? Please clarify whether this is a reporting issue or a difference in the implemented algorithm.
-
Can you separate the router's contribution during codebook training from its contribution during inference? In particular, how should the near equality of Random + ICM and Argmax + ICM affect the interpretation of the main results, and are the baseline search budgets matched?
-
What is the precise methodological advance over earlier neural compositional code learning? Is the principal contribution cross-codebook coordination, the parallel/residual architecture, or a demonstrated quality–cost advantage? A focused comparison would help establish this.
2 comments
Thank you for your response, which partially answered my questions. I maintain 2.
Rebuttal
We thank the reviewer for the constructive and valuable feedback, which helps us further improve our work.
R1. Correction of Residual DMQ Results
We apologize for a transcription error in Table 3, where the Residual DMQ Argmax and Argmax + ICM results were swapped. The corrected results are:
ICM performs deterministic coordinate-wise minimization over fixed codebooks, without random perturbations. Therefore, it cannot increase reconstruction error. We will correct the table accordingly.
R2. Router Contributions During Training and Inference
The router serves two distinct roles: during training, it jointly learns assignments with the codebooks; during inference, it directly predicts code indices without iterative search.
We provide the following comparison on BigANN1M (8 bytes):
Random + ICM bypasses the router entirely at inference. Under the same initialization and refinement strategy, DMQ outperforms LSQ (1.78 vs. 2.39), indicating that the advantage is not solely due to inference-time routing but is also associated with the learned codebooks. The near equality of DMQ's Random + ICM and Argmax + ICM (1.78 vs. 1.77) further shows that the refined result is largely insensitive to router initialization.
Meanwhile, direct Argmax achieves an MSE of 1.83 without geometric search, outperforming LSQ with full Faiss search (1.91). Thus, the router's inference-time value lies in producing high-quality assignments without iterative refinement. We acknowledge that baseline search budgets are not strictly matched and will clarify this distinction. Isolating the router's training-time contribution would require a controlled training ablation.
R3. Novelty Relative to Deep Compositional Code Learning
We thank the reviewer for pointing out Shu and Nakayama (2018). We acknowledge that neural code assignment, Gumbel-Softmax, and additive multi-codebook reconstruction were previously explored.
Our distinction lies in explicitly modeling cross-codebook dependencies through Transformer attention over codebook positions, rather than predicting multiple indices from a shared hidden representation. Residual DMQ further introduces autoregressive assignments conditioned on preceding reconstruction decisions.
The attention ablation supports this distinction: removing attention increases MSE from 8.77 to 9.15 for Parallel DMQ and from 8.88 to 9.21 for Residual DMQ on FB-ssnpp1M (8 bytes). We will clarify the relationship to prior work and refine our novelty claims.
We appreciate these helpful suggestions and will address the identified issues in the next revision.