acceptodds
Under review as a conference paper at ICLR 2027

Relevance-Resolution Transfer via Scale-Decomposable Fractional Diffusion for Multi-Length Cross-Modal Hash Retrieval

Abstract

Cross-modal hashing enables efficient retrieval by encoding heterogeneous data into compact binary codes. Recent methods increasingly exploit fine-grained relations encoded in multi-label training structure; however, they do not explicitly ensure that these relations are preserved as consistent candidate rankings in finite, multi-length Hamming spaces, revealing a relevance resolution bottleneck (RRB). To address the RRB, we propose MultiBit for transferring relevance resolution from multi-label structure to multi-length Hamming spaces. On the one hand, MultiBit constructs a scale-decomposable fractional relation teacher from dataset-level label co-occurrence and label specificity, modeling dependencies from local to long-range over continuous diffusion scales to produce query-conditioned relevance targets. On the other hand, it maps discretized diffusion scales and their contribution weights to scale-aware bit subblocks, organizes multiple code lengths as nested prefixes, and aligns their Hamming candidate rankings with the teacher relations. Experiments on four benchmarks consistently demonstrate improvements in retrieval accuracy, verifying the relevance-resolution advantages brought by scale-decomposable fractional diffusion in multi-length cross-modal hashing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Deep Hashing, Cross-modal Retrieval, Graph Diffusion, Multi-label Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BhX5OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work examines fine-grained semantic degradation arising when multi-label semantic relations get mapped into finite Hamming spaces. It looks at distance collisions, ranking inversion, and inconsistent ranking behavior across different code lengths. The proposed MultiBit framework introduces a fractional relation teacher built from label co-occurrence and specificity, scale-aware bit organization, plus a multi-length prefix alignment strategy. With nested prefix design, a single model can produce multiple code lengths in one forward pass. The authors test image-to-text and text-to-image retrieval over several datasets and four code lengths. They also run ablations to study relevance resolution, quantization error, and ranking consistency.

Critical strengths
  • The paper studies a meaningful problem in multi-length cross-modal hashing, and the proposed components fit together reasonably well. It investigates preserving graded semantic relations within binary hash codes of variable lengths. FRT, SBO and MPA each tackle different subproblems in this setup. Overall, I find the method sufficiently novel in the context of cross-modal retrieval.
  • Evaluations span multiple datasets, both retrieval directions, and several code-length settings. Ablations and ranking consistency analyses help support the method’s benefits. Released code and supplementary implementation details also aid reproducibility. Overall, the experiments support the main claims, although I still have a few methodological questions.
Critical weaknesses
  • In Section 3.4 and Appendix B, 16-bit, 32-bit and 64-bit prefixes only use the first scale subblock; multi-scale diffusion only appears for the 128-bit code. For shorter hash codes, gains from SBO mostly come from supervision within that first block, not cross-scale fusion. The authors should make this explicit in the main text. Comparisons with alternative bit allocation or scale ordering would help quantify how much performance depends on the current prefix split.
  • The paper frames collisions, inversions and cross-length ranking inconsistency as a unified motivation, which is intuitive. Still, limited discrete distances and ranking shifts are known intrinsic properties of binary hashing. To me, the more interesting part is the connection to graded semantic relations and the attempt to preserve ranking structure across code lengths. I recommend separating these well-known limitations from the new problem formulation in the Introduction or Section 3.1.
Questions for authors
  • Why can mini-batch-based normalization guarantee stable ranking on the full retrieval database? It would be helpful to report sensitivity to batch size or batch composition, or to provide some intuition for why the current approximation works well.
  • How is the discrete sequence extended to a continuous function in the integral formulation(Eq.2)? Please supply a more detailed derivation and proof.
3 comments
AuthorsOpenReview ↗

Response to Reviewer BhX5

Q1. Mini-batch normalization and full-database ranking.

Thank you for your recognition of our work and your constructive suggestions. Your feedback is very encouraging to us. We address your question below.

(1) Scope of the ranking property. Equation (10) states a conditional property for a given candidate set: when the listwise divergence vanishes, the student's continuous Hamming ranking preserves the teacher's strict ordering within that set. The full-database performance of the learned binary codes is evaluated empirically.

(2) Why mini-batch alignment provides a consistent ranking reference. The integrated teacher g∗g^\ast is constructed from the training-set label graph and supplies the same relevance scores across batches. For a query qq and two candidates a,ba,b present in a batch,

P∗(a∣q)P∗(b∣q)=exp⁡(g∗(q,a)−g∗(q,b)Tt).\frac{P^{\ast}(a\mid q)}{P^{\ast}(b\mid q)} =\exp\left(\frac{g^\ast(q,a)-g^\ast(q,b)}{T_t}\right).

The normalization denominator cancels, so adding other candidates leaves this pair's teacher ordering unchanged. For fixed model parameters, the same argument applies to the student's distribution. Random mini-batches therefore expose different candidate comparisons against a shared teacher, and every code length learns from this same reference. Batch composition can still affect the loss, gradients and candidate coverage.

(3) Empirical evaluation. We train with batch size 128 and evaluate mAP@all over the complete retrieval database. The table summarizes the mean Kendall's τb\tau_b over the six unordered pairs of 16/32/64/128-bit codes from Table F-2 (higher is better). Full denotes MultiBit with MPA.

DatasetI2T: w/o MPAI2T: FullT2I: w/o MPAT2I: FullMIRFlickr-25K0.8200.8550.8060.845IAPR TC-120.8010.8170.7900.803MS-COCO0.8330.8390.8220.824\begin{array}{lcccc}\text{Dataset}&\text{I2T: w/o MPA}&\text{I2T: Full}&\text{T2I: w/o MPA}&\text{T2I: Full}\\\\\hline\text{MIRFlickr-25K}&0.820&\mathbf{0.855}&0.806&\mathbf{0.845}\\\\\text{IAPR TC-12}&0.801&\mathbf{0.817}&0.790&\mathbf{0.803}\\\\\text{MS-COCO}&0.833&\mathbf{0.839}&0.822&\mathbf{0.824}\end{array}

MPA improves cross-length ranking agreement in both directions on all three datasets. These results support its effectiveness under the reported training configuration; Table F-2 includes the full comparison.

Q2. Continuous diffusion on the discrete label graph.

Thank you for requesting this clarification. In our retrieval setting, two candidates can share the same number of labels with a query yet differ in relevance because of label specificity and relationships among their labels. FRT uses the training-label graph to construct a graded ranking reference for transferring these distinctions into finite Hamming codes.

(1) In our formulation, tt denotes diffusion time on a graph with finitely many label nodes. We write its symmetric positive-semidefinite Laplacian as L=Udiag⁡(λk)UTL=U\operatorname{diag}(\lambda_k)U^\mathsf T, with λk≥0\lambda_k\ge0. For any real t≥0t\ge0,

H(t)=e−tL=Udiag⁡(e−tλk)UT.H(t)=e^{-tL} =U\operatorname{diag}(e^{-t\lambda_k})U^\mathsf T.

This matrix-valued function is continuous in tt and satisfies H′(t)=−LH(t)H'(t)=-LH(t) and H(0)=IH(0)=I. Applied to a weighted label vector, it describes propagation through label relationships. Small tt emphasizes local neighborhoods, while larger tt incorporates longer propagation paths. Thus, continuous time provides different relational scales on the same discrete graph.

(2) Equation (2) expresses how FRT combines these scales with the weight tα−1e−ηt/Γ(α)t^{\alpha-1}e^{-\eta t}/\Gamma(\alpha). For each eigenvalue, the substitution u=(λk+η)tu=(\lambda_k+\eta)t in the Gamma integral gives

1Γ(α)∫0∞tα−1e−(λk+η)t,dt=(λk+η)−α.\frac{1}{\Gamma(\alpha)} \int_0^\infty t^{\alpha-1}e^{-(\lambda_k+\eta)t}\\,\mathrm dt =(\lambda_k+\eta)^{-\alpha}.

Here α>0\alpha>0 and η>0\eta>0 ensure convergence. Applying this identity to the finite set of eigenvalues yields

1Γ(α)∫0∞tα−1e−ηte−tL,dt=Udiag⁡((λk+η)−α)UT=(L+ηI)−α.\frac{1}{\Gamma(\alpha)} \int_0^\infty t^{\alpha-1}e^{-\eta t}e^{-tL}\\,\mathrm dt =U\operatorname{diag}((\lambda_k+\eta)^{-\alpha})U^\mathsf T =(L+\eta I)^{-\alpha}.

Equation (2) therefore integrates continuous-time diffusion on a discrete graph.

(3) In MultiBit, the integrated kernel defines the query--candidate relevance score g∗g^\ast through the kernel cosine in Eq. (3), combining label specificity with relations across propagation scales. MPA uses this shared teacher to supervise every code-length prefix. To expose a finite scale structure for the available bits, Eq. (4) uses Gauss--Laguerre nodes and weights to guide SBO and the scale-specific objectives. The integrated teacher is computed from (L+ηI)−α(L+\eta I)^{-\alpha}, and each scale teacher from e−τℓLe^{-\tau_\ell L}. This connects multi-scale label relations to the bit organization and ranking objectives used for binary retrieval.

Reviewer_BhX5OpenReview ↗

Authors have addressed most of my concerns.

Thank you for your detailed rebuttal. The authors have answered most of my questions with their clarification. I have decided to maintain this positive review.

AuthorsOpenReview ↗

Response to Reviewer BhX5

Thank you for your positive feedback and support. This is very encouraging to us. Thank you again.

Reviewer_kWxpOpenReview ↗
Rating
3(+1)
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes MultiBit, which aims to transfer fine-grained correlations in multi-label structures to binary hash codes of different lengths, reducing correlation collisions, order reversals, and ranking inconsistencies across code lengths. The method utilizes label co-occurrence and label specificity to construct a fractional-order diffusion teacher, organizes scale-specific bit sub-blocks according to the diffusion scale and its product weight, and jointly supervises each scale sub-block and 16, 32, 64, and 128 bit nested prefixes through list-level distribution matching. The inference phase only requires encoding, prefix truncation, and Hamming distance sorting. The paper validates the proposed design on MIRFlickr, IAPR TC-12, MS-COCO, and NUS-WIDE (in the appendix), comparing it with fixed-length and multi-length hashing methods. The design is evaluated through ablation, quantization analysis, correlation ranking metrics, and cross-code length consistency experiments. The reported results show that it achieves the highest mAP in 16 out of 24 settings in the main experiments, while also demonstrating good multi-code length deployment efficiency and ranking consistency.

Critical strengths

This paper addresses the problem of preserving hierarchical relevance and consistent ranking across different code lengths in compact retrieval. In addition to mAP, the paper evaluates collision, reversal, and cross-code length ranking consistency, and examines whether ranking in continuous representations can be preserved after quantization. This paper achieves state-of-the-art results on most settings of the main experiments, demonstrating the usability of the proposed scheme.

Critical weaknesses
  1. The stated objective differs from the code implementation. The reverse teacher distribution, by directly transposing the normalized probability, typically no longer satisfies query-based normalization. The actual implementation used in the experiment and its impact need to be confirmed.

  2. The candidate pools for the relevance resolution metric are not comparable. The baseline uses the union of the top-100 with MultiBit, while MultiBit itself only uses its own top-100. Therefore, differences in τ_b, collision rate, and inversion rate may be influenced by the composition of the candidate set.

  3. The independent contributions of multi-scale organization have not been adequately isolated. While the paper includes some ablation experiments, the existing evidence is insufficient to clearly distinguish whether the gains come from teacher relevance, prefix training, or scale organization and supervision. In particular, according to the reported bit allocation, bits 16, 32, and 64 are all located within the first scale block. Therefore, further verification of the multi-scale design's gains is needed in a truly cross-scale 128-bit setting, by maintaining consistent teacher, prefix loss, and parameter sizes.

Questions for authors
  1. Inconsistency exists between the paper's method and code implementation: In the multibit/method/losses.py file of the Supplementary Material, the inverse loss uses the transpose of the teacher distribution, which has already been row-normalized. However, the transpose typically no longer satisfies row-normalization. Did the paper's experiments use this implementation? If so, do the main experimental results still hold? If not, please provide the corresponding code.

  2. Are the comparisons of relevance resolution based on the same candidate set? Appendix E.1 states that each baseline uses the union of its own Top-100 and MultiBit's Top-100, while MultiBit only uses its own Top-100 for each row. Was MultiBit re-evaluated on the candidate pool of the union of each baseline? Please provide Kendall's τ_b, collision rate, and inversion rate calculated by both methods on the exact same candidate set for each query in each comparison. Under this unified evaluation, does the advantage reported in the paper still exist?

  3. What evidence can isolate the independent contributions of multi-scale designs, distinguishing the contributions of scale organization and scale-specific supervision from the contributions of overall teacher and prefix alignment? Since the 128-bit prefix is ​​the only evaluated prefix that spans multiple scale blocks, should we consider adding a 128-bit control to maintain consistency in overall teacher, prefix objectives, and model capacity, while removing scale-specific designs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer kWxp (Part 1/2): (Q1, Q2)

Thank you for your careful review and recognition of our work. Your constructive suggestions have helped clarify the implementation, evaluation and scale-specific design.

Q1. Reverse teacher distribution.\textbf{Q1. Reverse teacher distribution.}

Thank you for pointing out this discrepancy. The submitted code used an older branch that transposed normalized probabilities for the reverse teacher. All reported experiments followed the paper: transpose raw scores, then apply row-wise softmax for the reverse queries. For raw scores SS,

PI→T=softmax⁡row(S/Tt),PT→I=softmax⁡row(ST/Tt).\begin{aligned} P^{I\rightarrow T}&=\operatorname{softmax}_{\mathrm{row}}(S/T_t),\\\\ P^{T\rightarrow I}&=\operatorname{softmax}_{\mathrm{row}}(S^\mathsf{T}/T_t). \end{aligned}

Both integrated and scale-specific teachers use this normalization. Corrected code: multibit/method/losses.py (MultiBitObjective.forward; supplement).

Q2. Candidate pools for relevance resolution.\textbf{Q2. Candidate pools for relevance resolution.}

Thank you for raising this question. Your feedback has helped us improve the quality of the paper. In the original evaluation, each baseline was scored on the union of its own and MultiBit’s Top-100 lists, while MultiBit was scored on its own Top-100 list. We agree that this made the candidate pools incomparable, and we have recomputed the results.

(1) Shared candidate pool for Table 3.\textbf{Shared candidate pool for Table 3.} For each query, direction and code length, all five methods are scored on the same deduplicated union of their Top-100 lists, using their own Hamming distances. Kendall’s τ_b, collision and inversion use this shared pool. NDCG@100 uses each method’s full-database ranking. We show the first two datasets for space.

N = NDCG@100; C = collision; I = inversion. Arrows indicate better values.

MIRFlickr-25K\textbf{MIRFlickr-25K}

MethodDECHDGHDGHCMCLVH-ABLMultiBit16 bitsN↑τb↑C↓I↓0.6320.4780.4170.2700.6240.4660.4250.2760.6430.4890.4080.2620.6510.5010.3920.2540.6600.5090.3940.25032 bitsN↑τb↑C↓I↓0.6500.5160.3900.2490.6440.5050.3980.2550.6620.5280.3820.2410.6700.5400.3690.2330.6780.5460.3660.22964 bitsN↑τb↑C↓I↓0.6660.5340.3710.2350.6590.5260.3780.2410.6770.5460.3620.2270.6860.5570.3500.2180.6970.5620.3470.214\begin{array}{c}{}\\\\\hline\begin{array}{l}\text{Method}\\\\\hline\text{DECH}\\\\\text{DGHDGH}\\\\\text{CMCL}\\\\\text{VH-ABL}\\\\\text{MultiBit}\end{array}\end{array}\quad\begin{array}{c}\text{16 bits}\\\\\hline\begin{array}{cccc}N↑&\tau_b↑&C↓&I↓\\\\\hline0.632&0.478&0.417&0.270\\\\0.624&0.466&0.425&0.276\\\\0.643&0.489&0.408&0.262\\\\0.651&0.501&\mathbf{0.392}&0.254\\\\\mathbf{0.660}&\mathbf{0.509}&0.394&\mathbf{0.250}\end{array}\end{array}\quad\begin{array}{c}\text{32 bits}\\\\\hline\begin{array}{cccc}N↑&\tau_b↑&C↓&I↓\\\\\hline0.650&0.516&0.390&0.249\\\\0.644&0.505&0.398&0.255\\\\0.662&0.528&0.382&0.241\\\\0.670&0.540&0.369&0.233\\\\\mathbf{0.678}&\mathbf{0.546}&\mathbf{0.366}&\mathbf{0.229}\end{array}\end{array}\quad\begin{array}{c}\text{64 bits}\\\\\hline\begin{array}{cccc}N↑&\tau_b↑&C↓&I↓\\\\\hline0.666&0.534&0.371&0.235\\\\0.659&0.526&0.378&0.241\\\\0.677&0.546&0.362&0.227\\\\0.686&0.557&0.350&0.218\\\\\mathbf{0.697}&\mathbf{0.562}&\mathbf{0.347}&\mathbf{0.214}\end{array}\end{array}

IAPR TC-12\textbf{IAPR TC-12}

MethodDECHDGHDGHCMCLVH-ABLMultiBit16 bitsN↑τb↑C↓I↓0.6110.4020.4370.3220.6040.3880.4440.3290.6200.4210.4240.3100.6290.4280.4150.3020.6400.4240.4080.29632 bitsN↑τb↑C↓I↓0.6340.4310.4120.3000.6270.4200.4190.3060.6470.4410.4000.2890.6560.4520.3880.2780.6680.4560.3900.27464 bitsN↑τb↑C↓I↓0.6530.4480.3910.2820.6460.4400.3980.2880.6630.4580.3820.2720.6760.4690.3680.2610.6880.4740.3650.257\begin{array}{c}{}\\\\\hline\begin{array}{l}\text{Method}\\\\\hline\text{DECH}\\\\\text{DGHDGH}\\\\\text{CMCL}\\\\\text{VH-ABL}\\\\\text{MultiBit}\end{array}\end{array}\quad\begin{array}{c}\text{16 bits}\\\\\hline\begin{array}{cccc}N↑&\tau_b↑&C↓&I↓\\\\\hline0.611&0.402&0.437&0.322\\\\0.604&0.388&0.444&0.329\\\\0.620&0.421&0.424&0.310\\\\0.629&\mathbf{0.428}&0.415&0.302\\\\\mathbf{0.640}&0.424&\mathbf{0.408}&\mathbf{0.296}\end{array}\end{array}\quad\begin{array}{c}\text{32 bits}\\\\\hline\begin{array}{cccc}N↑&\tau_b↑&C↓&I↓\\\\\hline0.634&0.431&0.412&0.300\\\\0.627&0.420&0.419&0.306\\\\0.647&0.441&0.400&0.289\\\\0.656&0.452&\mathbf{0.388}&0.278\\\\\mathbf{0.668}&\mathbf{0.456}&0.390&\mathbf{0.274}\end{array}\end{array}\quad\begin{array}{c}\text{64 bits}\\\\\hline\begin{array}{cccc}N↑&\tau_b↑&C↓&I↓\\\\\hline0.653&0.448&0.391&0.282\\\\0.646&0.440&0.398&0.288\\\\0.663&0.458&0.382&0.272\\\\0.676&0.469&0.368&0.261\\\\\mathbf{0.688}&\mathbf{0.474}&\mathbf{0.365}&\mathbf{0.257}\end{array}\end{array}

Under this shared-pool evaluation, MultiBit has the lowest inversion rate in all nine dataset–length settings, the highest τ_b in seven, and the lowest collision rate in seven. It also has the highest NDCG@100 in all nine settings. We revised the conclusions and marked VH-ABL’s better results in the main paper’s Table 3.

(2) Pairwise comparisons in Appendix E.\textbf{Pairwise comparisons in Appendix E.} We also re-evaluated MultiBit separately against each baseline on the union of their two Top-100 lists for every query. Table E-3 reports both methods’ scores on the same pool under the IDF-weighted reference. On MIRFlickr-25K, MultiBit has a higher τ_b in 17 of 18 comparisons and lower collision and inversion rates in 16 of 18 each. Table E-4 keeps the same candidate pools and Hamming rankings and switches to the label-name semantic reference. MultiBit performs better in 17 of 18 comparisons for each metric under this reference. The advantage therefore persists in most pairwise comparisons under both relevance references.

Evaluation code: scripts/evaluate_shared_pool.py (supplement).

AuthorsOpenReview ↗

Response to Reviewer kWxp (Part 2/2): (Q3)

Q3. Contribution of the scale-specific design.\textbf{Q3. Contribution of the scale-specific design.}

Thank you for your thoughtful and constructive suggestion. Following your advice, we added matched 128-bit controls on MIRFlickr-25K, keeping the integrated teacher, all prefix objectives, model capacity and common training settings fixed to assess the additional contribution of the scale-specific design. Results are mAP (%).

Config.Original bit allocationBlock teacherPrefix supervision (MPA)I2TT2IMeanC1×—✓g∗85.33480.52882.931C2✓g∗✓g∗86.02780.46383.245C3×gℓ✓g∗85.54480.76883.156C4 (Full)✓gℓ✓g∗85.92381.25783.590\begin{array}{lcccccc}\text{Config.}&\text{Original bit allocation}&\text{Block teacher}&\text{Prefix supervision (MPA)}&\text{I2T}&\text{T2I}&\text{Mean}\\\\\hline\text{C1}&{\times}&\text{—}&{\checkmark}g^{\ast}&85.334&80.528&82.931\\\\\text{C2}&{\checkmark}&g^{\ast}&{\checkmark}g^{\ast}&\mathbf{86.027}&80.463&83.245\\\\\text{C3}&{\times}&g_\ell&{\checkmark}g^{\ast}&85.544&80.768&83.156\\\\\text{C4 (Full)}&{\checkmark}&g_\ell&{\checkmark}g^{\ast}&85.923&\mathbf{81.257}&\mathbf{83.590}\end{array}

Here, g∗g^{\ast} is the integrated teacher and gℓg_\ell is the corresponding scale teacher. C1 uses one 128-bit head without block loss; C2 retains the original bit allocation but supervises every block with the integrated teacher; C3 uses four equal 32-bit blocks with scale-specific supervision; C4 is full MultiBit. All four configurations use the same integrated teacher for their prefixes, and C2–C4 share the same scale-loss weights and coefficient.

(1) C4 improves Mean mAP over C1 by 0.659 pp, showing that the scale-specific design provides additional gains when the integrated teacher, MPA and model capacity are held fixed.

(2) C4 exceeds C2 by 0.345 pp. Both use the same bit allocation, so this comparison supports the additional benefit of supervising each block with its corresponding scale teacher.

(3) C4 exceeds C3 by 0.434 pp. Both retain scale-specific supervision, so this comparison supports the benefit of the original scale-based bit allocation over equal allocation.

Reproduction code: scripts/scale_design_control.py (supplement).

Reviewer_mgzxOpenReview ↗
Rating
3(+1)
Confidence
4(+1)
Informativeness
3(+1)
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes MultiBit, a multi-length cross-modal hashing framework designed to address the bottleneck of relevance resolution within a constrained Hamming space. The MultiBit framework consists of three components: Fractional Relation Teacher (FRT), Scale-aware Bit Organization (SBO), and Multi-length Prefix Alignment (MPA). Specifically, FRT integrates label co-occurrence and label specificity through fractional diffusion and decomposes the result into query-conditioned relevance targets together with the quadrature weight of each diffusion scale. SBO assigns relation scales to dedicated bit subblocks of the maximum-length code. MPA organizes the target code lengths as nested prefixes and constrains their Hamming candidate distributions to follow the same teacher-induced ordering. Experiments on multiple benchmark datasets demonstrate that this framework not only achieves competitive retrieval performance but also enhances ranking consistency across different code lengths and reduces deployment costs.

Critical strengths
  1. This paper identifies the relevance resolution bottleneck and explicitly addresses the need to maintain consistent candidate rankings across different hash code lengths.

  2. Extensive experiments on multiple benchmark datasets demonstrate the competitive retrieval performance of MultiBit, along with improved ranking consistency across different code lengths and reduced deployment costs.

  3. Clear and well-designed figures and tables. The figures and tables are visually appealing and well organized, making the proposed framework and experimental results easy to understand.

Critical weaknesses
  1. According to Appendix Table B-1, the first diffusion-scale sub-block contains 71 or 80 bits, which means that the 16-, 32-, and 64-bit codes are derived entirely from this single sub-block. Only the 128-bit codes directly integrate multiple scale-specific sub-blocks.

  2. Although the proposed method outperforms most baseline models, in certain settings, the improvement in mAP over strong competitors remains quite limited, which raises questions regarding the significance of its performance advantages.

Questions for authors
  1. Given that the 16-, 32-, and 64-bit codes are derived entirely from the first scale-specific sub-block, could the authors provide further evidence that multi-scale information is effectively preserved in these shorter codes?

  2. For the main comparisons and ablation studies, could the authors report results across multiple random training seeds? Furthermore, when performance improvements are below 0.3 percentage points, could the authors clarify whether the relative performance rankings of the compared methods remain stable?

  3. Some ablation variants show only marginal improvements in retrieval performance. Could the authors further clarify the individual contributions of these components to retrieval accuracy?

4 comments
AuthorsOpenReview ↗

Response to Reviewer mgzx (Part 1/2): (Q1, Q2)

Thank you for carefully reviewing our paper and recognizing the value of our work. Your valuable feedback has been helpful in refining our research. We respond to your questions below.

Q1 (W1). Do short prefixes retain relations from the integrated teacher?

(1) Training objective. We agree that the 16-, 32-, and 64-bit prefixes lie entirely in the first scale block, and only the 128-bit code explicitly concatenates multiple scale blocks (Table B-1). Our point is that short prefixes still receive direct supervision from the integrated teacher. Here g1g_1 is the first-scale teacher; g∗g^{\ast} integrates diffusion scales. Eqs. (8)–(9) supervise different bit ranges of the same code. Lpre,B(g)\mathcal L_{\mathrm{pre},B}(g) is the bidirectional ranking loss for the first BB bits, and Lblk,ℓ(g)\mathcal L_{\mathrm{blk},\ell}(g) for complete block ℓ\ell, both aligned with teacher gg:

\sum_{B\in\\{16,32,64,128\\}}\beta_B\\,\mathcal L_{\mathrm{pre},B}(g^{\ast}) +\lambda_s\sum_{\ell=1}^{M}\bar\rho_\ell\\,\mathcal L_{\mathrm{blk},\ell}(g_\ell).$$ At 64 bits, the prefix loss aligns rankings from bits 1–64 with $g^{\ast}$. The first-block loss aligns the full block with $g_1$ (bits 1–71 on MIRFlickr-25K, 1–80 on IAPR TC-12 and MS-COCO). Both losses backpropagate through bits 1–64, giving them direct integrated-teacher supervision. (2) **Controlled ablation.** A and B use $g^{\ast}$ and $g_1$, respectively, for 16/32/64-bit prefixes; bit layout, scale losses and the 128-bit target stay fixed. We use all 5,000 MIRFlickr-25K test queries (fixed seed-42 split). Both methods use the per-query union of their Top-100 lists. We retain only pairs strictly ordered oppositely by $g^{\ast}$ and $g_1$, excluding teacher ties. Binary agreement with $g^{\ast}$ and Hamming ties share all retained pairs as the denominator. We pool counts per direction, calculate percentages and average I2T/T2I equally. $\Delta$ is the A–B agreement gap (pp). Counting code: `scripts/short_prefix_control.py` (supplement). $$\begin{array}{cccccc}\rm Bits&\text{Agree A (\\%)}&\text{Agree B (\\%)}&\Delta\\,\text{(pp)}&\text{Ties A (\\%)}&\text{Ties B (\\%)}\\\\\hline16&43.84&41.67&+2.17&16.41&16.84\\\\32&47.36&44.72&+2.64&12.33&12.05\\\\64&49.09&47.13&+1.96&9.14&8.77\end{array}$$ The integrated target raises agreement with $g^{\ast}$ by 2.17/2.64/1.96 pp at 16/32/64 bits. Thus, short binary prefixes retain some additional integrated-teacher ordering information within the first scale block. **Q2. Results across random seeds.** We show five-seed mAP@all on MIRFlickr-25K (MIR) and MS-COCO (COCO). $$\begin{array}{c}{}\\\\\hline\begin{array}{ll}\rm Task&\rm Method\\\\\hline&\rm DSSH\\\\\rm I2T&\rm DECH\\\\&\rm DPSIH\\\\\hline&\rm CMCL\\\\\text{I2T (multi)}&\text{VH-ABL}\\\\&\rm MultiBit\\\\\hline&\rm DSSH\\\\\rm T2I&\rm DECH\\\\&\rm DPSIH\\\\\hline&\rm CMCL\\\\\text{T2I (multi)}&\text{VH-ABL}\\\\&\rm MultiBit\end{array}\end{array}\quad\begin{array}{c}\rm MIR\\\\\hline\begin{array}{cccc}16&32&64&128\\\\\hline75.24±0.57&75.54±0.12&80.21±0.46&80.92±0.13\\\\81.44±0.36&84.03±0.34&85.10±0.27&85.48±0.23\\\\77.27±0.28&80.60±0.25&81.90±0.26&82.42±0.20\\\\\hline\mathbf{83.33}±0.39&84.22±0.16&84.99±0.32&85.45±0.26\\\\80.45±0.33&81.88±0.25&82.52±0.23&82.77±0.22\\\\82.85±0.50&\mathbf{84.54}±0.38&\mathbf{85.64}±0.26&\mathbf{86.03}±0.11\\\\\hline74.44±0.36&73.34±0.32&78.69±0.21&79.49±0.24\\\\78.03±0.30&79.70±0.33&80.58±0.33&81.12±0.21\\\\73.73±0.30&77.42±0.41&77.97±0.46&78.51±0.50\\\\\hline79.25±0.40&79.97±0.42&80.50±0.39&80.42±0.27\\\\75.05±0.35&75.91±0.19&77.43±0.20&78.73±0.33\\\\\mathbf{79.32}±0.61&\mathbf{80.20}±0.25&\mathbf{80.90}±0.34&\mathbf{81.16}±0.33\end{array}\end{array}\quad\begin{array}{c}\rm COCO\\\\\hline\begin{array}{cccc}16&32&64&128\\\\\hline62.97±0.56&67.54±0.20&73.57±0.41&76.32±0.18\\\\64.22±0.33&68.57±0.06&72.35±0.18&73.85±0.13\\\\63.59±0.25&67.93±0.63&70.86±0.26&72.45±0.32\\\\\hline\mathbf{70.48}±0.45&\mathbf{72.92}±0.36&75.25±0.16&75.89±0.17\\\\63.15±0.39&70.19±0.24&73.48±0.16&75.26±0.14\\\\69.85±0.25&72.74±0.10&\mathbf{75.89}±0.10&\mathbf{77.00}±0.08\\\\\hline63.42±0.52&68.66±0.23&74.88±0.33&77.58±0.33\\\\64.74±0.38&69.76±0.08&72.44±0.48&74.36±0.43\\\\63.37±0.49&67.95±0.26&71.02±0.35&72.72±0.42\\\\\hline\mathbf{69.98}±0.31&\mathbf{73.02}±0.60&75.31±0.29&75.73±0.31\\\\63.70±0.10&70.41±0.20&73.74±0.25&75.34±0.32\\\\68.63±0.19&71.77±0.31&\mathbf{76.27}±0.16&\mathbf{77.95}±0.24\end{array}\end{array}$$ In the new five-seed means, MultiBit's lead over the strongest baseline is below 0.3 pp for MIRFlickr-25K T2I at 16, 32 and 128 bits. These three settings give 15 runs (five seeds each). In five runs, at least one baseline has a slightly higher mAP. Averaged over the five seeds, MultiBit remains ahead of every baseline in all three settings; its relative ranking based on mean performance remains stable.
AuthorsOpenReview ↗

Response to Reviewer mgzx (Part 2/2): (Q3)

Q3. Component contributions to retrieval accuracy.

Thank you for this constructive comment. To further clarify the incremental contributions of the components, we summarize the progressive ablation results in Table 2 by short and long codes below. Each column reports the additional Mean mAP gain over the preceding configuration when FRT, SBO and MPA are added in that order, in pp. The short-code group averages the gains at 16/32 bits, and the long-code group averages those at 64/128 bits.

DatasetCode-length groupAdd FRTThen add SBOThen add MPAMIRFlickr-25KShort: 16/32 bits+0.330+0.244+0.062Long: 64/128 bits+0.805+0.545+0.187IAPR TC-12Short: 16/32 bits+0.105+0.091+0.025Long: 64/128 bits+0.910+0.872+0.243MS-COCOShort: 16/32 bits+1.082+0.601+0.418Long: 64/128 bits+1.842+1.529+0.677\begin{array}{llccc}\text{Dataset}&\text{Code-length group}&\text{Add FRT}&\text{Then add SBO}&\text{Then add MPA}\\\\\hline\text{MIRFlickr-25K}&\text{Short: 16/32 bits}&+0.330&+0.244&+0.062\\\\&\text{Long: 64/128 bits}&+0.805&+0.545&+0.187\\\\\hline\text{IAPR TC-12}&\text{Short: 16/32 bits}&+0.105&+0.091&+0.025\\\\&\text{Long: 64/128 bits}&+0.910&+0.872&+0.243\\\\\hline\text{MS-COCO}&\text{Short: 16/32 bits}&+1.082&+0.601&+0.418\\\\&\text{Long: 64/128 bits}&+1.842&+1.529&+0.677\end{array}

All three components bring additional accuracy gains, but their magnitudes vary with code length and dataset. Some short-code gains on MIRFlickr-25K and IAPR TC-12 are small, whereas COCO already benefits more at short lengths. On all three datasets, each component has a larger average gain in the long-code group than in the short-code group.

We believe this pattern is consistent with the components' joint design around our paper's motivating problem, the "relevance resolution bottleneck". To address this problem, our method uses "continuous-scale integration" to provide finer relational distinctions. Whether these distinctions translate into accuracy gains is also limited by the expressive capacity of Hamming codes. Finer ranking distinctions supplied by FRT may still map to identical Hamming distances in short codes, so additional supervision may not fully translate into accuracy gains. SBO's bit allocation faces the same capacity limit, and both components yield greater gains in the long-code group. MPA further improves accuracy after FRT and SBO have already improved the ranking, with larger incremental gains in the long-code group as well. Enabling short codes to better preserve these fine-grained ranking distinctions remains a direction we hope to improve in future work.

I appreciate the authors' thorough and thoughtful response. All my concerns have been satisfactorily addressed. I will therefore raise my score and recommend acceptance of the paper.

AuthorsOpenReview ↗

Response to Reviewer mgzx

Thank you for recognizing our work. Your support is a great encouragement to us. Thank you again.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.