acceptodds
Withdrawn from ICLR 2027

Unified Semantic Product Quantization with Category-Aware Implicit Codebooks for Cross-Modal Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Cross-modal retrieval (CMR) requires compact and discriminative representations for efficient large-scale search. Although product quantization (PQ) is effective for compact representation learning, existing PQ-based CMR methods typically treat codebooks primarily as feature-approximation prototypes, limiting the direct incorporation of semantic supervision into codebook learning. To address this issue, we propose P2PQ, a unified semantic PQ framework with category-aware implicit codebooks for CMR. Instead of independently parameterizing quantization codebooks and semantic predictors, P2PQ shares a single matrix between these two roles, so that the codebook cardinality is induced by the dimensionality of the semantic prediction space rather than specified independently. This dual-role parameterization allows category-level supervision to directly shape the quantization prototypes without imposing label-constrained codeword assignments, thereby intrinsically coupling discrete quantization with semantic discrimination. Furthermore, P2PQ learns a shared latent semantic space for heterogeneous modality alignment and preserves geometric consistency between continuous and quantized representations through an implicit label-guided affinity formulation without explicit graph construction. Extensive experiments on three benchmark datasets demonstrate that P2PQ consistently outperforms representative shallow CMR methods under various code lengths, validating its effectiveness and scalability.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Cross-Modal Retrieval, Product Quantization, Implicit Codebook

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nW6YOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes P2PQ, a supervised shallow product-quantization framework for cross-modal retrieval. Its central design shares each subspace codebook with the corresponding semantic regression weights, making the number of codewords equal to the label dimension. The framework jointly learns modality-specific projections, orthogonal transformations, shared semantic factors, and quantized database representations. A label-guided affinity objective is evaluated implicitly without materializing a dense similarity matrix. Optimization alternates between continuous and discrete updates, and retrieval uses continuous query representations against quantized database codes. Experiments cover three benchmarks, four code lengths, component ablations, parameter sensitivity, training time, and an additional deep-feature setting.

Critical strengths

The shared codebook/regression parameterization provides a concrete mechanism for incorporating semantic supervision into quantization prototypes. The implicit computation of label-guided affinities also avoids storing an explicit N × N similarity matrix, which is practically useful for full-batch supervised retrieval.

The reported retrieval improvements are substantial and consistent. In Table 1, P2PQ outperforms the strongest displayed baseline in all 24 dataset/direction/code-length settings, with improvements ranging from 1.11 to 9.34 percentage points. The additional deep-feature comparison suggests that the framework can also benefit from stronger input representations.

Critical weaknesses

The evidence does not sufficiently isolate the central contribution. The main comparison includes hashing baselines but omits directly related quantization methods. Prior work such as ACQH already combines quantization with label supervision, making the specific benefit of the proposed parameter sharing important to establish. However, P2PQ-2 removes both alignment and regression, while P2PQ-3 changes both the affinity objective and database encoding. Neither isolates sharing the codebook and regression weights. The absence of retrieval-time and complete storage comparisons also limits the claimed accuracy–efficiency advantage.

The stated objective has an unresolved scaling issue. With the other variables fixed, the feasible transformation Z_p → Z_p/a, H → H/a², and S_p → aS_p leaves the regression loss, affinity loss, and query representations unchanged. However, it reduces the alignment loss by a⁻² and the H regularizer by a⁻⁴. Thus, objective reduction can occur without improving the retrieval representation. The provided formulation and algorithm do not specify factor normalization or bounds that prevent this behavior, weakening the interpretation of the optimization and convergence results.

Questions for authors

Could you clarify the distinction from prior supervised quantization approaches such as ACQH, and explain how the existing ablations establish the benefit of sharing W_m? A controlled variant with the same codebook cardinality c, features, affinity objective, and retrieval rule, but separate quantization prototypes and regression weights, would be informative. A comparison in one existing 32-bit setting would provide a focused way to address this concern.

Does the implementation normalize or bound Z_p, H, or S_p during training? Under the constraints stated in Eq. (6), Z_p → Z_p/a, H → H/a², and S_p → aS_p can reduce the objective while preserving the retrieval representations. Please document any additional normalization in the formulation and Algorithm 1, or explain how this scaling behavior is handled and how the convergence curves should be interpreted.

At matched stored code lengths, could you clarify the retrieval trade-off relative to binary hashing, including codebook storage, query projection, lookup-table construction, and database distance computation? Table 3 reports training time only. A concrete cost breakdown for one existing 32-bit configuration would help establish the scope of the efficiency claim and distinguish asymptotic training complexity from practical search efficiency.

Reviewer_6A2VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes P2PQ, a supervised quantization framework for cross-modal retrieval. Each subspace codebook serves both as quantization prototypes and as label-regression weights, with its size determined by the number of categories. The framework combines semantic alignment, label regression, and label-guided relation preservation. Experiments on three datasets show improvements over the evaluated hashing baselines.

Critical strengths
  1. Sharing parameters between quantization prototypes and label-regression weights provides a clear way to apply category supervision directly to codebook learning. The design also allows codeword assignments to remain independent of the sample labels.
  2. P2PQ achieves strong retrieval results across three datasets and multiple code lengths. The additional experiments with deep visual features suggest that the framework can also benefit from stronger input representations.
Critical weaknesses
  1. ACQH[1] already combines quantization, label regression, and pairwise semantic preservation. P2PQ introduces parameter sharing between codebooks and semantic predictors, but the paper does not clearly compare this design with the objective and parameterization of ACQH. The specific contribution beyond existing supervised quantization methods needs clearer explanation. [1] Lu Wang, Masoumeh Zareapoor, Jie Yang, and Zhonglong Zheng. “Asymmetric Correlation Quantization Hashing for Cross-Modal Retrieval.” IEEE Transactions on Multimedia, 24: 3665–3678, 2022.
  2. The codebook size is set to the number of semantic categories. However, the number of categories does not necessarily indicate how many codewords are needed to represent the data. The paper should explain why this choice provides enough capacity and when it offers a useful reduction in codebook size.
  3. The experiments mainly compare P2PQ with hashing methods and omit closely related quantization baselines. The ablations also do not directly compare shared and separate codebook and prediction parameters. Since some variants change several components at once, the results do not isolate the benefit of the central parameter-sharing design.
Questions for authors

Please find the questions or concerns in Critical Weaknesses.

Reviewer_mLTxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a product quantization method for cross-modal retrieval. It addresses the limited use of semantic supervision in codebook learning by sharing codebooks between quantization and semantic prediction. The method also incorporates cross-modal alignment and label-guided preservation of semantic relationships. Experiments on three benchmark datasets report improvements over the compared shallow hashing methods, with ablation studies and additional analyses examining the proposed components.

Critical strengths

This paper couples category discrimination with discrete representation learning via shared parameters. The proposed method employs the same matrix both as quantization prototypes and as semantic prediction weights, allowing semantic supervision to directly guide codebook learning. This gives codewords two well-defined, complementary functions: constructing compact representations of data and discriminating semantic categories.

Critical weaknesses

The joint objective can decrease through parameter rescaling while all three fitting terms remain unchanged, calling into question the intended role of the regularizer. This does not prove that training is unstable, but the paper does not explain how this scaling freedom is controlled or whether parameter norms remain stable. Since this concern impacts the central optimization formulation, the authors should provide clarification and supporting evidence.

Questions for authors
  • Section 2.4 adopts asymmetric distance computation for Euclidean-distance retrieval, while Eq. (5) learns label-based semantic affinities through inner products. Could the authors explain why Euclidean distance is preferred here? A comparison of Euclidean-distance and inner-product retrieval using the same trained representations would help clarify the effect of this choice.

  • How is the scaling freedom in Eq. (6) controlled during training? Are there additional constraints or regularization steps that are not described in the paper? If not, how should the role of (μ∣H∣F2)(\mu|H|_F^2) be interpreted when this penalty can be reduced without changing the fitting terms?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.