- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EmbedCodec, a retrieval-aware codec for compressing frozen multimodal embeddings. It introduces a shared query–candidate adapter trained through the hard product-quantization assignments used at deployment, aiming to preserve retrieval rankings after quantization. To further reduce storage, it models quantized indices with a task-conditioned Gaussian mixture model and applies rANS entropy coding. The authors evaluate the method on five retrieval tasks with two frozen embedding encoders, measuring NDCG@10 and serialized storage. The results indicate substantial storage reduction over FP32 embeddings with only a small drop in five-task average NDCG@10. Additional experiments examine adapter placement, retrieval scoring functions, entropy modeling, and cross-domain performance.
Critical strengths
-
The paper addresses the substantial storage cost of frozen multimodal embeddings in retrieval systems and evaluates compression using actual serialized bytes rather than relying solely on theoretical code lengths. This makes the problem formulation practically relevant and the reported storage–quality trade-offs more informative for deployment.
-
The evaluation covers five retrieval tasks and two embedding encoders, while the supplementary material examines adapter placement, deployment scoring functions, entropy-table alternatives, and exact decoding. These analyses provide useful evidence that the proposed pipeline is operationally viable and that its components affect the resulting compression–retrieval trade-off.
Critical weaknesses
1.The proposed framework mainly integrates retrieval-aware adaptation, OPQ/PQ quantization, task-conditioned entropy modeling, and rANS coding. The current experiments do not sufficiently isolate the contribution of the rate-aware optimization or establish a clear methodological advance over existing retrieval-oriented compression methods.
2.The training objective is not fully aligned with the deployment ranking function, and comparisons retain different methods' native scoring functions and rate definitions. Consequently, the results do not fully establish the method's advantage under consistent retrieval protocols and comparable storage costs.
Questions for authors
1.What is the key methodological advance of EmbedCodec over existing retrieval-aware quantization and entropy coding methods? Specifically, can the authors isolate the contribution of rate-aware optimization from retrieval-aware adaptation and entropy modeling?
2.How sensitive are the results to the mismatch between the training objective and the deployment ranking function? Would optimizing with the same negative squared Euclidean distance used at inference improve retrieval performance?
3.How would the reported results change under consistent storage accounting and retrieval evaluation protocols across competing methods, including the costs required for practical deployment?
2 comments
Official Response to Reviewer DnTF (Part 2/2)
Q2: Training uses cosine-prefix losses, whereas deployment ranks with negative squared Euclidean distance. Does this mismatch affect retrieval, and what happens when the model is trained with the deployment-aligned L2 objective?
A2: On Seed1.6 M128 FIQA, cosine-prefix training with L2 retrieval gives the highest score in the matched comparison: 0.7161. Keeping L2 retrieval and changing the training objective to L2 gives 0.7024. Keeping cosine-prefix training and changing the retrieval scorer to cosine gives 0.7047.
-
On FIQA, L2 retraining lowers retrieval quality even when evaluation uses L2.
We change only the score in the retrieval loss on quantized candidates. The analysis-transform initialization, fixed OPQ, training batches, optimizer, other losses, and 6,808 updates are unchanged. The L2 loss retains decoded candidate norms, as at inference.
Training score Cosine retrieval NDCG@10 L2 retrieval NDCG@10 Original cosine prefixes 0.7047 0.7161 L2 prefixes 0.6903 0.7024 See Line 971-992 in the paper (Appendix C.2, Tables C.4–C.5, pp. 18–19).
-
The effect is encoder-dependent: Qwen's mean is almost unchanged, whereas all five Seed tasks decrease.
Encoder / M128 Original training, L2 retrieval L2 training, L2 retrieval Change Qwen3-VL 0.7000 0.7001 +0.000110 Seed1.6 0.7817 0.7741 −0.007616 Scores are five-task mean NDCG@10; changes use unrounded values. See Line 971-992 in the paper (Appendix C.2, Tables C.4–C.5, pp. 18–19).
-
With the model fixed, L2 remains the better deployment scorer in most comparisons.
Holding the checkpoint, queries, candidates, and compressed indices fixed, L2 gives higher NDCG@10 in 24 of 30 task–width comparisons and five of six five-task means across both encoders and M128/M256/M512. Only the scorer changes; stored bytes remain identical. See Line 945-969 in the paper (Appendix C.2, Figure C.2, p. 18).
Q3: Under a common retrieval protocol, a common scorer, and complete deployment storage accounting, how do the reported results change relative to competing methods?
A3: With the same retrieval protocol and negative squared L2 scorer, EmbedCodec retains higher retrieval scores than matched JPQ-style. After including all stored model and index objects, its shared five-task package remains about 26× smaller than FP32.
-
Common scoring preserves the retrieval advantage over matched JPQ-style, with a clear storage trade-off.
At M128, both methods use the same frozen encoder outputs, evaluation queries and full task candidate pools. The matched training uses the same sampled positives and in-batch negatives, batch size 512, and 6,808 updates. Scores are five-task mean NDCG@10; package sizes include all five indexes and one copy of each method's shared model.
Method Scorer Qwen NDCG@10 Seed NDCG@10 Qwen package MB Seed package MB JPQ-style, matched Native inner product 0.5219 0.6154 119.684 119.684 JPQ-style, matched Common negative squared L2 0.4452 0.5721 119.684 119.684 EmbedCodec Negative squared L2 0.7000 0.7817 156.224 156.746 Under common L2, EmbedCodec gains 0.2548 NDCG@10 on Qwen and 0.2096 on Seed, with an additional 36.5–37.1 MB of complete five-task storage. See Line 693-750 in the paper (Appendix B.2, Table B.1, pp. 13–14).
-
Complete package storage remains substantially below FP32 and 16-bit storage. Across the five tasks, it is about 26× smaller than FP32 and 13.2× smaller than 16-bit storage.
Totals count compressed indices, the analysis transform, OPQ state, probability tables, candidate identifiers, offsets, and framing. The five-task deployment contains 501,964 candidate entries and shares one transform and OPQ state; MB denotes bytes.
Deployment FP32 package MB 16-bit package MB EmbedCodec Qwen MB EmbedCodec Seed MB WebQA, 377,482 candidates 3,097.568 1,551.401 129.303 129.646 Five tasks, shared 4,128.277 2,072.233 156.224 156.746 WebQA alone retains about a 24× storage reduction relative to FP32 after all these costs are included. See Line 1239-1276 in the paper (Appendix D.1, Table D.2, pp. 23–24).
Official Response to Reviewer DnTF (Part 1/2)
We thank the reviewer for the valuable comments and seek to address reviewer's concerns below:
Q1: Beyond combining a retrieval-aware adapter, OPQ/PQ, a task-conditioned entropy model, and rANS, what component is methodologically new, and which ablation isolates the contribution of rate-aware optimization from adaptation and final coding?
A1: The methodological contribution is to train a pre-OPQ analysis transform jointly for post-quantization retrieval and the model code length of the selected indices, while the encoder, OPQ rotation, and codebooks remain fixed. Removing only the rate term increases mean code size for both encoders at all three OPQ widths after matched coder fitting.
This design differs from JPQ, which optimizes its query encoder and PQ centroids for ranking (Zhan et al., 2021, Section 3.1). In EmbedCodec, the forward pass uses hard assignments, and soft-assignment gradients carry both the retrieval loss and the geometric GMM's coding cost back to the analysis transform. See Line 232-284 in the paper (Eqs. 6–9, pp. 5–6).
A matched supervised control on Seed1.6 VDR-en shows better retrieval after the same fixed OPQ.
The ordinary supervised control matches the architecture, initialization, training batches, optimizer, and 6,808 updates, with quantization and the rate term removed during training. Both models then use the same fixed M128 OPQ.
Nearly identical pre-quantization scores become different post-quantization scores at comparable code sizes. See Line 869-897 in the paper (Appendix C.1, Table C.2, p. 17).
Removing only the rate term increases actual code size for both encoders at all three OPQ widths.
This control retains quantization and the matched training conditions, changing the rate weight from 0.001 to 0. After training, both models separately fit the same type of task-conditioned three-component GMM on the same 65,536 training candidates, with 64 fitting updates and identical rANS settings.
Removing the rate term increases the five-task mean code size by 0.106%–0.226% across all six settings. The sum of marginal index entropies also rises by 0.114%–0.249% across the six settings. These entropies are computed directly from index frequencies, independently of the fitted GMM.
Two task examples also improve retrieval with the rate term:
Code sizes count actual serialized main + hyper bytes per candidate. See Line 996-1012 in the paper (Appendix C.3, Table C.6, p. 19); Line 1015-1068 in the paper (Appendix C.3, Table C.7 and Figure C.3, pp. 19–20); Line 1069-1101 in the paper (Appendix C.3, Figure C.4, pp. 20–21).
Task conditioning reduces code size without changing retrieval scores.
Holding the transform, OPQ, and indices fixed, we independently fit shared and task-conditioned three-component GMMs under identical conditions. The shared model requires 1.57%–3.05% more bytes on the six five-task means and increases all 30 task streams; every model recovers the same indices and preserves the same retrieval scores. See Line 1106-1127 in the paper (Appendix C.4, Table C.8, p. 21).