acceptodds
Under review as a conference paper at ICLR 2027

Decoupled Multi-Codebook Quantization

Abstract

Multi-codebook quantization commonly ties code selection to codebook geometry, although individually nearest codewords need not form the best joint representation. We propose Decoupled Multi-Codebook Quantization (DMQ), which separates what codewords represent from how they are selected. A learned router coordinates assignments without input–codeword distance matching, unifying parallel and residual quantization. Differentiable training jointly optimizes assignments and codebooks while accommodating downstream objectives. We establish the potential suboptimality of independent Euclidean assignment, prove strictly greater assignment expressivity under suitable capacity assumptions at unchanged code budgets, and derive a sufficient condition for initialization-independent global convergence of Iterated Conditional Modes. Experiments on four million-scale benchmarks demonstrate improvements over representative baselines in reconstruction and retrieval quality, while maintaining efficient inference. These results establish decoupled assignment as an expressive and effective design principle for learning compact discrete representations. Our implementation is anonymously provided at https://anonymous.4open.science/r/DMQ.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vector quantization, multi-codebook quantization, discrete representation learning, codebook learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qkpMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DMQ separates multi-codebook representations from the rule used to select their codewords. A learned assignment network coordinates choices across codebooks, with parallel and residual variants trained using soft assignments and reconstruction objectives. Discrete inference uses direct assignments with ICM refinement or beam search. The theoretical results discuss limitations of independent nearest-codeword assignment, the expressivity of learned assignment functions, and a sufficient condition for initialization-independent ICM convergence. Experiments evaluate reconstruction and retrieval on four vector benchmarks at two code budgets, with component and inference-strategy ablations.

Critical strengths

Coordinating assignments across codebooks is a meaningful problem, and the parallel/residual formulation offers a useful way to study it. Table 1 contains substantial improvements in some settings: on BigANN1M at 16 bytes, Parallel DMQ reduces MSE from 0.98 for LSQ to 0.75 while improving retrieval. This is a positive result worth understanding.

The component and inference ablations are helpful. In particular, Table 3 exposes the contribution of post-processing rather than hiding it inside the final system, and Table 2 shows that initialization, attention, and soft assignments affect performance.

Critical weaknesses

There is an unresolved inconsistency between the stated refinement and Table 3. Equation (23) minimizes reconstruction error over one codebook while fixing the others. With fixed codebooks and the same initialization, each exact update must satisfy Enew≤EoldE_{\mathrm{new}} \leq E_{\mathrm{old}}. Yet Residual DMQ changes from MSE 1.96 with Argmax to 2.02 with Argmax + ICM. Equation (23) is introduced for the parallel variant, so the residual experiment may use a different procedure, but that difference is not specified. This needs an explicit explanation before the refinement results can be interpreted.

The benefit attributable to learned assignment is not sufficiently separated from codebook learning and subsequent search. For Parallel DMQ in Table 3, Random + ICM achieves MSE 1.78 in 17.0 seconds, essentially matching Argmax + ICM at 1.77 in 18.4 seconds. This does not negate the quality of the learned codebooks, but it weakens the case for the learned router at inference. The theoretical comparisons do not settle this issue: independent nearest-codeword assignment is a narrower comparator than joint search, and the margin condition in Theorem 4.3 is not shown to hold for the fitted models. The novelty discussion should also account for Shu and Nakayama's "Compressing Word Embeddings via Deep Compositional Code Learning" [1], which already learns neural multi-codebook assignments jointly with compositional representations.

[1] Shu, R., & Nakayama, H. (2017). Compressing word embeddings via deep compositional code learning. arXiv preprint arXiv:1711.01068.

Questions for authors
  1. What exactly is the residual Argmax + ICM procedure in Table 3? Does it optimize the same fixed-codebook objective as Equation (23), and how can its reconstruction error increase? Please clarify whether this is a reporting issue or a difference in the implemented algorithm.

  2. Can you separate the router's contribution during codebook training from its contribution during inference? In particular, how should the near equality of Random + ICM and Argmax + ICM affect the interpretation of the main results, and are the baseline search budgets matched?

  3. What is the precise methodological advance over earlier neural compositional code learning? Is the principal contribution cross-codebook coordination, the parallel/residual architecture, or a demonstrated quality–cost advantage? A focused comparison would help establish this.

2 comments

We thank the reviewer for the constructive and valuable feedback, which helps us further improve our work.

R1. Correction of Residual DMQ Results

We apologize for a transcription error in Table 3, where the Residual DMQ Argmax and Argmax + ICM results were swapped. The corrected results are:

MethodMSE ↓R@1 ↑R@10 ↑
Residual DMQ (Argmax)2.0229.977.0
Residual DMQ (Argmax + ICM)1.9630.978.6

ICM performs deterministic coordinate-wise minimization over fixed codebooks, without random perturbations. Therefore, it cannot increase reconstruction error. We will correct the table accordingly.

R2. Router Contributions During Training and Inference

The router serves two distinct roles: during training, it jointly learns assignments with the codebooks; during inference, it directly predicts code indices without iterative search.

We provide the following comparison on BigANN1M (8 bytes):

MethodMSE ↓R@1 ↑R@10 ↑
Parallel DMQ (Argmax)1.8332.079.6
Parallel DMQ (Argmax + ICM)1.7732.380.0
Parallel DMQ (Random + ICM)1.7832.280.1
LSQ (Random + ICM)2.3924.1869.60
LSQ (Full Faiss Search)1.9131.779.4
RQ (Greedy)2.4927.575.4

Random + ICM bypasses the router entirely at inference. Under the same initialization and refinement strategy, DMQ outperforms LSQ (1.78 vs. 2.39), indicating that the advantage is not solely due to inference-time routing but is also associated with the learned codebooks. The near equality of DMQ's Random + ICM and Argmax + ICM (1.78 vs. 1.77) further shows that the refined result is largely insensitive to router initialization.

Meanwhile, direct Argmax achieves an MSE of 1.83 without geometric search, outperforming LSQ with full Faiss search (1.91). Thus, the router's inference-time value lies in producing high-quality assignments without iterative refinement. We acknowledge that baseline search budgets are not strictly matched and will clarify this distinction. Isolating the router's training-time contribution would require a controlled training ablation.

R3. Novelty Relative to Deep Compositional Code Learning

We thank the reviewer for pointing out Shu and Nakayama (2018). We acknowledge that neural code assignment, Gumbel-Softmax, and additive multi-codebook reconstruction were previously explored.

Our distinction lies in explicitly modeling cross-codebook dependencies through Transformer attention over codebook positions, rather than predicting multiple indices from a shared hidden representation. Residual DMQ further introduces autoregressive assignments conditioned on preceding reconstruction decisions.

The attention ablation supports this distinction: removing attention increases MSE from 8.77 to 9.15 for Parallel DMQ and from 8.88 to 9.21 for Residual DMQ on FB-ssnpp1M (8 bytes). We will clarify the relationship to prior work and refine our novelty claims.

We appreciate these helpful suggestions and will address the identified issues in the next revision.

Thank you for your response, which partially answered my questions. I maintain 2.

Reviewer_rQ6HOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

In standard vector quantization, a single shared feature matrix is used for both computing the similarity/distance scores for code assignment and representing the quantized features. This study challenges it by experimenting separate feature matrices for the two. It also proposed a Transformer-based quantization module while typical implementations simply quantize raw or non-parametrically transformed features. The proposed method was evaluated on vector retrieval tasks.

Critical strengths

The proposed method outperformed baselines at least on the vector retrieval benchmark.

Critical weaknesses

Limited Novelty

The core argument in this paper is that the feature matrix for computing code-assignment vectors (i.e., similarity/distance metric) can be (or should be) distinguished from the one for representing quantized features (codebook). It is true that many of well-adopted methods use a single matrix for these two purposes, but their decoupling has already been explored and adopted in previous studies. A representative example is Wav2Vec 2.0 (Baevski et al., 2020, NeurIPS); their official code computed the assignment logits by nn.Linear (named self.weight_proj) while the codebook vectors are represented by a different nn.Parameter (named self.vars). https://github.com/facebookresearch/fairseq/blob/main/fairseq/modules/gumbel_vector_quantizer.py#L11 Note that Wav2Vec 2.0 also utilizes product quantization.

Also, the proposed method heavily relies on the pretraining (Section 5.3), and their main pretraining method utilizes "conventional quantizers" (which I assumed to be a matrix-shared model). This indicates that the standard matrix-shared models are more stable.

Unclarity of the Implementation Details

Immediately above the title of Section 4.4.1, the authors mention "Gumbel noise", which is explained nowhere else. Similarly, the ablation study (Section 5.3) also mention STE, but the use of hard quantization (plus soft gradient computation) is never reported elsewhere.

Moreover, the authors state "Hard codeword selection is applied only at inference time" in Section 4.2. This sounds contradicting with the previous points.

The unclarity of the exact implementation is critical especially for understanding the residual quantization. Equation 14 uses the soft quantization, and I wondered if the disparity between the soft vs. hard quantization magnifies for the downstream portion of the residuals. If hard quantization (plus soft gradient computation) was used instead, this concern no longer matters.

Research Scope and Contribution are Overclaimed

This paper focuses on a specific application of vector quantization to the vector retrieval task, while it is also actively studied for building generative models (e.g., Esser, 2021, CVPR). In this context, I think the current presentation of the achievements is overhyped; statements like:

We evaluate DMQ on million-scale benchmarks and demonstrate improvements over representative baselines in both reconstruction and retrieval quality.

sounds as if the proposed model has proved effective for generative tasks as well (because generative models are also typically benchmarked by reconstruction quality of discrete autoencoding). I would suggest that the authors clarify that their research scope is limited to the vector retrieval task in the abstract and introduction.

Transformer-Based Quantization Looks Expensive

The proposed method utilizes Transformer for quantization itself, which looks expensive and seems to limit its empirical applicability to generative tasks where deep encoder/decoder or large batch size is required. The authors should report the parameter size and VRAM consumptions of their proposed method in comparison with the baseline models.

Suboptimality of Independent Assignment is Misleading

The authors argue that independent code assignment is suboptimal compared to the proposed method. I think this is a bit misleading. In a setting where vector quantization is sandwiched between an encoder and decoder of raw data (e.g., image, audio), these backbone modules may arrange the latent pre-quantized features so that independent code assignment works effectively. The claimed suboptimality appears only valid when the pre-quantized features are given by some pretrained encoder and the error is evaluated based on the quantization itself rather than the overall reconstruction of raw data.

Questions for authors
  • What is the exact implementation of the code-assignment vectors for training?
  • I don't understand how Equation 17 prevents codebook collapse. Specifically, this loss appears to rather encourage collapse when zero hard frequencies are encountered.
  • Can you prove that independent code assignment is suboptimal even when the representation of the pre-quantized vectors and decoding of the quantized vectors are jointly optimized by an encoder and decoder?
2 comments

We thank the reviewer for the constructive and valuable feedback, which helps us further improve our work.

R1. Novelty and Differences from Wav2Vec 2.0

We acknowledge that Wav2Vec 2.0 separates assignment projections from codebook parameters. However, DMQ differs in two key aspects. First, DMQ focuses on additive multi-codebook quantization, where multiple codewords jointly contribute to the reconstructed representation, rather than group-wise product quantization. Second, DMQ explicitly introduces cross-codebook interactions through a shared Transformer, enabling coordinated code selection instead of independently predicting assignments for different codebook groups. This supports both parallel joint assignment and autoregressive residual assignment.

Regarding initialization, Table 2 shows that load-balancing pretraining achieves nearly identical performance to PQ/RQ-supervised initialization (MSE 8.79 vs. 8.77 for Parallel DMQ). Thus, conventional quantizer supervision facilitates initialization but is not essential to final performance. We will clarify these distinctions and acknowledge the relevant prior work.

R2. Clarification of Training and Inference

During training, DMQ employs Gumbel-Softmax with temperature annealing to obtain differentiable soft assignments:

pm=softmax((Zm+gm)/τ),p_m=\mathrm{softmax}\left((Z_m+g_m)/\tau\right),

where gmg_m denotes Gumbel noise. Quantized representations are constructed through probability-weighted codewords, and residual DMQ uses these soft assignments to update residual states.

At inference, Gumbel noise is disabled, and code indices are directly selected using the highest router logits. STE is used only in the "w/o Soft Assignment" ablation, not in the main training procedure. We will clarify these implementation details in the revision.

R3. Soft–Hard Discrepancy in Residual DMQ

We agree that soft–hard discrepancy may accumulate across residual stages. DMQ mitigates this through temperature annealing, which progressively sharpens assignment distributions. During supervised initialization, subsequent residuals are generated from the model's own predictions rather than ground-truth codes, exposing the router to its autoregressive dynamics.

At inference, hard codewords update residuals, while beam search explores multiple assignment sequences to improve reconstruction quality. We will further clarify this training–inference distinction.

R4. Research Scope and Computational Efficiency

Our experiments focus on vector compression and retrieval rather than generative modeling. We will explicitly clarify this scope in the abstract and introduction.

DMQ employs only a single Transformer block. Under our experimental setting, LSQ requires approximately 3.3× the total inference time of Parallel DMQ with ICM refinement (61.3 s vs. 18.4 s). We will additionally report model parameters, VRAM consumption, and computational costs to better characterize efficiency.

R5. Scope of the Independent Assignment Theorem

Theorem 4.1 demonstrates that independent Euclidean assignment can be suboptimal for additive reconstruction because it ignores cross-codebook interactions. While jointly optimized encoders and decoders may alleviate this limitation by adapting the latent representation, they do not inherently account for such interactions in the assignment rule.

More importantly, DMQ provides a more flexible assignment mechanism. As established in Theorem 4.2, decoupled assignment can represent Euclidean nearest-code rules under the stated capacity assumptions, while also supporting more expressive coordinated assignments. When cross-codebook coordination is unnecessary, DMQ can recover independent assignment behavior; otherwise, it can exploit interactions to improve code selection.

Thus, our results establish a representational advantage rather than guaranteed superiority under arbitrary end-to-end optimization. We will clarify this distinction in the revision.

R6. Interpretation of the Load-Balancing Loss

Eq. (17) combines detached hard assignment frequencies with differentiable soft routing probabilities. Although unused codes have zero hard frequency, softmax normalization couples their probabilities with those of frequently selected codes. Minimizing the loss therefore discourages concentrated routing: complete collapse yields a loss approaching KK, whereas balanced routing yields 11.

Furthermore, we observe that codebook utilization progressively increases during training, providing empirical evidence that the overall framework promotes effective codebook usage rather than reinforcing collapse. While this does not constitute a strict non-collapse guarantee, it supports the practical effectiveness of the approach. We will include utilization statistics in the revision.

We appreciate these helpful suggestions and will address the identified issues in the next revision.

Reviewer_rQ6HOpenReview ↗

Comment by rQ6H

I appreciate the authors' comments. I expected a revised manuscript to be uploaded at some point, but it has not become available yet. I cannot check whether R2-4 are sufficiently addressed without it.

And I'm still unconvinced about R1 and R5. Regarding R1, the following statement seems to presuppose that the decoder (and encoder) is not jointly trained with the codebooks:

multiple codewords jointly contribute to the reconstructed representation, rather than group-wise product quantization

If the decoder can learn how to use the multiple codebooks, the advantage of the proposed method appears to diminish. I think the authors should clarify how much model complexity is allowed to be in this game. Without it, improvements by introducing additional layers look trivial.

Similarly, I don't understand why jointly optimized encoder/decoder remain suboptimal (R5). Detailed proof in a revised manuscript might have helped but again, this is unavailable.

For these reasons, I will retain my original score.

Reviewer_UBxAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets code assignment in multi-codebook quantization for compressed-domain nearest-neighbor search. Instead of choosing codes by distance to codewords, DMQ uses a small Transformer router that outputs a distribution over each codebook's 256 entries. The parallel variant uses full attention over tokens projected from the input. The residual variant uses causal attention over the running residual. Router and codebooks are trained jointly through an annealed softmax mixture of codewords, after a pretraining stage. At inference, the main results add ICM sweeps for the parallel variant and beam search for the residual one. Three theorems motivate the design. Experiments cover Deep1M, BigANN1M, FB-ssnpp1M and Contriever1M at 8 and 16 bytes.

Critical strengths

S1. A new way to assign codes, aimed at a real limitation. Tying code selection to codebook geometry is a real limitation: individually nearest codewords need not form the best joint representation (Thm 4.1). DMQ is the first multi-codebook method I know of whose assignment network never computes a distance to codewords, which separates it from AQ, LSQ and the softmax-over-distance encoders such as DPQ. One router covers both additive and residual quantization by changing only the attention mask and the input stream, and the softmax mixture makes joint training straightforward.

S2. Released code and a protocol that matches prior work. An anonymous repository provides the parallel and residual implementations and the data preparation. Table 1 uses the four standard datasets at two budgets under the 500K-training protocol, and its OPQ/RQ/LSQ rows match published figures, so the results are directly comparable with the literature.

Critical weaknesses

W1. The best-performance claim does not hold against the field: the comparison stops at 2023 and omits the learned quantizers that define the state of the art on this exact suite. Table 1 includes no method newer than DeepQ (2023). QINCo, QINCo2, UNQ and RQ-MoE are cited but not compared, although the paper adopts QINCo's protocol and these methods have public code. Published QINCo results under the same protocol beat DMQ on BigANN1M and Deep1M at both budgets, for example BigANN1M at 16 bytes (MSE 0.47 vs 0.75, R@1 65.5 vs 55.9). The claim of best overall performance (l.419-420) is therefore not supported; it holds only against an older baseline set.

W2. The benefit of decoupling is not shown: the gains over LSQ are small, neither variant is consistently better, and there is no unified method that keeps the strengths of both. Against LSQ, R@1 improves by at most 0.6 points on Deep1M, BigANN1M and FB-ssnpp1M at 8 bytes; larger gains appear only in a few cells, such as BigANN1M at 16 bytes for the parallel variant (55.9 vs 51.1). At 16 bytes each variant falls behind LSQ where the other leads: parallel trails on Deep1M R@1 (41.9 vs 42.3) where residual leads (45.0), and residual trails on BigANN1M MSE (1.08 vs 0.98), FB-ssnpp1M R@1 (5.7 vs 6.2) and Contriever1M R@1 (25.3 vs 25.6) where parallel leads or ties. The flexibility thus costs accuracy in some settings, the paper gives no rule for choosing a variant, and a principled formulation that unifies the two is missing. In addition, the main results add codeword-based refinement at inference (App. B, l.745-747), and in the one cell where this is analyzed (Table 3), ICM from random codes matches ICM from the router's codes (1.78 vs 1.77), while residual router-only decoding is worse than LSQ (1.96 vs 1.91). The benefit of the decoupled assignment itself is therefore not demonstrated.

W3. The experiments are too limited to support the claims. The ablation covers only FB-ssnpp1M at 8 bytes, where all recalls are about 3%, with a single seed. The inference analysis (Table 3) covers a single cell, and no result in the paper reports seeds or standard deviations. All experiments use 1M databases with 500K training vectors, with no 10M-training or billion-scale setting. The claimed support for downstream objectives (l.015) is never tested, codebook utilization is not reported despite the load-balancing loss, and the training cost of the multi-stage pipeline is not given.

W4. The theory is correct but does not reach the reported gains. Thm 4.1 restates the known motivation for joint encoding in AQ and LSQ. Thm 4.2 is a capacity-inclusion result and does not show that better assignments are learned. Thm 4.3 relies on a margin condition (Eq. 47) that is never measured, and the residual rows of Table 3 (2.44 from random codes vs 2.02) indicate it does not hold there.

Minor points, listed separately from the weaknesses above and not weighed in the assessment:

  • In Table 3, ICM raises residual MSE from 1.96 to 2.02, which an exact coordinate minimization (Eq. 23) started from the argmax codes should not do; please explain.
  • Table 3 lists beam search for parallel DMQ, but Sec. 4.5 defines it only for the residual variant.
  • Residual beam search with B=10 beats the default B=5 (1.87 vs 1.92) at nearly the same time.
  • The caption of Table 3 should state the dataset and budget.
  • Typos: "implemenation" (l.499) and "Contrieval1M" (l.399).

Further requests, not decisive for the assessment:

  • Please state whether the LSQ timing in Sec. 5.4 used the GPU or the CPU, and split inference time into encoding and search.
  • Please report training cost and router size.
Questions for authors

Q1 (W1). Please report QINCo and QINCo2, ideally also UNQ, on all four datasets at 8 and 16 bytes under your protocol, with encoding time for each method. Competitive accuracy, or a measured accuracy-versus-encoding-cost advantage, is needed to support the best-performance claim; the published gaps on BigANN1M and Deep1M holding with no cost advantage would confirm W1.

Q2 (W2). Please report router-only results and ICM from random codes for every Table 1 cell, and explain how a user should choose between the parallel and residual variants, or whether one formulation can combine them. Router-only results that stay ahead of LSQ across settings, together with a single variant or rule that does not fall behind LSQ, are needed to show a benefit from decoupling; otherwise the gains belong to the learned codebooks plus refinement, and the contribution must be reframed that way.

3 comments
AuthorsOpenReview ↗

Rebuttal 1

We thank the reviewer for the constructive and valuable feedback, which helps us further improve our work.

R1. Comparison with Recent Learned Quantizers

We would like to clarify that DMQ focuses on a complementary direction to QINCo, QINCo2, and RQ-MoE. These methods primarily enhance codebook expressiveness through input-dependent codebook construction or modulation, while retaining geometry-based codeword selection. In contrast, DMQ focuses on how codewords are selected rather than how they are constructed, replacing geometric matching with a learned router.

Our comparisons therefore focus on representative methods with conventional codebook representations to isolate the contribution of decoupled assignment. Nevertheless, we acknowledge the importance of broader comparisons. Since these two directions are complementary, we plan to investigate replacing the geometry-based assignment mechanisms in QINCo and RQ-MoE with DMQ's learned routing mechanism, while preserving their codebook modeling components. This would further evaluate the applicability of decoupled assignment across different quantization architectures. We will also clarify the scope of our performance claims.

R2. Effectiveness of Decoupled Assignment

We appreciate the reviewer's concern regarding whether the improvements originate from learned assignment or inference-time refinement. To clarify this distinction, we provide the following comparison on BigANN1M with an 8-byte code budget.

MethodEncoding StrategyMSE ↓R@1 ↑R@10 ↑
Parallel DMQArgmax1.8332.079.6
Parallel DMQArgmax + ICM1.7732.380.0
LSQRandom + ICM2.3924.1869.60
LSQFull Faiss local search1.9131.779.4
RQGreedy2.4927.575.4

Importantly, Argmax in DMQ directly selects the code index with the highest router logit, without geometric matching or additional search. DMQ's direct predictions already outperform the conventional baselines, achieving an MSE of 1.83 versus 1.91 for LSQ with full Faiss local search. ICM further reduces the MSE to 1.77 through deterministic coordinate-wise refinement without random perturbations.

We further analyze cross-codebook interactions using the margin-dominance formulation in Theorem 4.3. Parallel DMQ exhibits substantially smaller average worst-case perturbations (0.7982) than LSQ (2.1969) and RQ (1.8812). This advantage persists under random assignments (0.7630 versus 2.1634 and 1.9117, respectively), suggesting that the reduced perturbation is associated with the learned codebook geometry rather than solely with router-predicted assignments.

Together with the competitive Argmax-only performance, these results highlight the effectiveness of jointly learned assignments and codebooks, while ICM serves as an optional refinement mechanism.

R3. Complementary Roles of Parallel and Residual DMQ

DMQ unifies parallel and residual quantization through the same decoupled assignment principle, rather than requiring a single structure to dominate across all settings.

Parallel DMQ uses full attention to jointly predict assignments without predefined codebook ordering, making it particularly suitable for representations without explicit hierarchical structure. Residual DMQ progressively refines representations through sequential residual updates, naturally supporting coarse-to-fine or hierarchical discrete representations, which can be beneficial for downstream applications such as recommendation systems.

The two variants therefore exhibit different inductive biases and may perform differently depending on the data distribution and task requirements. Their complementary behavior reflects the flexibility of the decoupled assignment framework.

We appreciate these helpful suggestions and will address the identified issues in the next revision.

AuthorsOpenReview ↗

Rebuttal 2

R4. Experimental Scope and Computational Efficiency

We emphasize that DMQ employs only a single Transformer block, introducing limited architectural complexity. Moreover, LSQ requires approximately 3.3× the total inference time of Parallel DMQ with ICM refinement (61.3 s vs. 18.4 s), demonstrating the efficiency of our approach.

We acknowledge that additional multi-seed experiments, broader ablations, codebook utilization statistics, and detailed training and inference costs would strengthen the evaluation. We also clarify that while DMQ supports differentiable downstream objectives, our current experiments focus on reconstruction-based training.

R5. Scope of Theoretical Contributions

Theorem 4.1 formalizes the suboptimality of independent Euclidean assignment for additive reconstruction, motivating coordinated assignment rather than claiming novelty over existing joint encoding methods.

Theorem 4.2 establishes that decoupled assignment strictly expands the representable assignment function class under the stated assumptions. We agree that greater expressivity does not guarantee improved optimization; the theorem characterizes representational flexibility, while the experiments investigate its practical effectiveness.

Theorem 4.3 provides a sufficient condition for initialization-independent global convergence of exact ICM over fixed codebooks, rather than an unconditional convergence guarantee. We will clarify these distinctions to avoid overinterpreting the theoretical results.

R6. Corrections and Implementation Clarifications

We thank the reviewer for identifying the inconsistency in Table 3. The residual DMQ results for Argmax and Argmax + ICM were inadvertently swapped due to a transcription error. The correct values are:

  • Argmax: MSE = 2.02, R@1 = 29.9, R@10 = 77.0.
  • Argmax + ICM: MSE = 1.96, R@1 = 30.9, R@10 = 78.6.

We appreciate these helpful suggestions and will address the identified issues in the next revision.

I acknowledge the authors' efforts in the rebuttal. However, the rebuttal does not change my two main concerns. By the authors’ own evidence, random and router assignments give the same result after refinement (1.78 versus 1.77), so the gains come from the learned codebooks, not the decoupled assignment. Moreover, the authors withdraw the best-performance claim rather than defend it, since they still have not run QINCo, QINCo2, and UNQ, and they concede limited theory and experiments. Therefore, I maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.