acceptodds
Under review as a conference paper at ICLR 2027

Reliable Supervision and Semantic Initialization for Noisy Multi-Label Cross-Modal Hashing Retrieval

Abstract

Supervised multi-label cross-modal hashing relies on label information to construct semantic relations across heterogeneous modalities. However, false-positive and missing-positive labels introduce spurious positive relations and missing associations, respectively, directly distorting cross-modal semantic alignment and degrading binary representation learning. Existing robust methods mainly focus on identifying, reweighting, or correcting noisy supervision, but often underexploit the fine-grained multi-label structure after correction. Consequently, corrected supervision is not fully preserved across continuous semantic relation modeling and discrete hash optimization. To address this issue, we propose Reliability-Based Cross-modal Deep Hashing (RBCDH), which unifies supervision reliability modeling with multi-resolution semantic learning. RBCDH estimates instance–class-level label reliability by integrating intra-modal neighborhood label support with cross-modal consistency, and performs controlled bidirectional correction to suppress false positives while recovering missing positives. To preserve fine-grained semantics after correction, we construct continuous Soft-Jaccard relations from corrected label overlaps, providing a reliability-aware initialization before discrete optimization. We further derive binary reliable relations from the same corrected supervision to constrain the final hash representations, enabling reliable semantics to be consistently exploited across continuous and discrete spaces. Experiments on NUS-WIDE TC-21, and MM-IMDb under different noise levels and code lengths show that RBCDH consistently outperforms the baseline methods in overall cross-modal retrieval performance under noisy supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Cross-modal Hashing, Noisy Multi-label Learning, Reliable Supervision Reconstruction, Hash Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_PUY9OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies supervised multi-label cross-modal hashing when the training annotations contain false-positive and missing-positive labels. The authors propose RBCDH, which first estimates label reliability using intra-modal neighborhood support and cross-modal consistency, then uses the corrected labels to guide continuous semantic initialization and subsequent binary hash refinement. The Soft-Jaccard objective is intended to preserve differences in label overlap that would otherwise be lost when the corrected annotations are converted into binary relations. Experiments on NUS-WIDE TC-21 and MM-IMDb show that RBCDH achieves the best Avg mAP in 23 of the 24 experimental settings across the two datasets, three noise levels, and four hash-code lengths. The method is straightforward to follow, and the paper addresses a relevant issue in robust cross-modal retrieval.

Critical strengths

(1) Correcting noisy annotations does not automatically preserve the degree of semantic overlap between instances. RBCDH addresses this issue by using continuous Soft-Jaccard relations during initialization and binary relations for the final retrieval-oriented optimization. This gives the method a clear motivation beyond simply filtering unreliable labels.

(2) PF-BLC produces a fixed corrected label matrix, which is then reused in the two learning stages. The auxiliary neighborhood encoder is not transferred to the hashing network, keeping the stages distinct. This sequential design makes the training procedure relatively easy to understand and implement.

(3) RBCDH performs well on both datasets under varying annotation noise and code lengths. The ablations also offer useful insight into the method: replacing Soft-Jaccard initialization with binary supervision leads to a substantial drop on NUS-WIDE, supporting the motivation for preserving graded semantic relations.

Critical weaknesses

(1) PF-BLC obtains neighborhood evidence from representations learned using the observed labels. If the noise has already influenced these representations, the resulting neighborhoods may sometimes reinforce incorrect label associations. Cross-modal consistency helps reduce this risk, but agreement between two modalities does not necessarily mean that the shared label is correct. The paper would benefit from a clearer discussion of this limitation and of the circumstances in which the two sources of evidence are expected to be reliable.

(2) The label-correction diagnostics show a meaningful precision-recall trade-off. On both datasets, PF-BLC increases relation precision and reduces the false-positive rate, but recall also decreases. On MM-IMDb, the reported element-wise label error changes from 0.04348 to 0.04368. These results do not negate the usefulness of the correction strategy, particularly given its precision-first design, but they suggest that the corrected labels should not be interpreted as uniformly more accurate. It would be useful to explain why this particular trade-off is beneficial for the subsequent hashing objective.

(3) The contribution of final hash refinement is not consistent across the two datasets. On MM-IMDb, full RBCDH improves Avg mAP by 0.508 percentage points over the variant without the final hash refinement stage. On NUS-WIDE TC-21, however, removing refinement yields a higher score, 72.508% versus 71.751%. The authors acknowledge this difference in the manuscript, which is appropriate. A little more discussion of why binary refinement can be helpful on one dataset but detrimental on the other would make the role of this stage clearer.

Questions for authors

Q1. Since the neighborhood encoder is trained on noisy labels, how does PF-BLC avoid reinforcing incorrect associations when noisy labels also influence the learned neighborhood structure? A short discussion of this potential feedback effect would be helpful.

Q2. The reliability diagnostics suggest that PF-BLC is more effective at reducing false-positive relations than at recovering missing-positive relations. Do the authors have an explanation for why this precision-oriented correction remains beneficial for retrieval, despite the associated reduction in recall?

Q3. Given that the final binary refinement does not improve NUS-WIDE TC-21 in the reported ablation, is there a property of the corrected relation distribution or the dataset that might explain the difference between the two benchmarks?

2 comments

Thank you for these insightful comments. Our responses are provided below:

Response to Q1:

The Neighborhood Encoder is used as an auxiliary component to extract sample neighborhood relationships from the image and text modalities. These relationships provide additional evidence for PF-BLC. To reduce the influence of noisy neighborhood information, PF-BLC evaluates neighborhood label support separately in the two modalities, rather than relying on either modality alone. It also adopts a conservative correction strategy by applying class-level and instance-level constraints to label deletion and completion.

The corrected labels are not used to retrain the Neighborhood Encoder or update its learned neighborhood structures. Thus, correction errors are not fed back into the encoder through repeated updates. However, when correlated label noise affects both modalities, they may still provide consistent but incorrect neighborhood evidence. This limitation cannot be completely avoided, particularly under high noise rates.

Response to Q2:

False-positive relations can pull semantically unrelated image-text pairs closer in the learned hash space, increasing the likelihood that irrelevant samples are retrieved as matches. By removing these unreliable positive constraints, PF-BLC aims to improve semantic discrimination and reduce incorrect matches.

We acknowledge that deleting some positive relations may also remove genuine semantic associations, causing related samples to lose their positive constraints and potentially reducing retrieval recall. However, false-positive relations introduce misleading similarity constraints, whereas missing positive relations mainly result in insufficient supervision for some genuinely related pairs. This difference helps explain why reducing false-positive relations can still benefit retrieval despite some loss of recall. The benefit of this precision-oriented correction strategy therefore depends on whether the reduction in incorrect matches outweighs the loss of genuine positive relations.

Response to Q3:

We believe the difference may be related to the different label co-occurrence patterns of the two datasets. NUS-WIDE TC-21 contains more diverse multi-label combinations, leading to varying degrees of label overlap between sample pairs. Soft-Jaccard initialization can preserve these graded relationships, whereas binary refinement compresses different degrees of positive overlap into uniform binary constraints. Such discretization may discard useful fine-grained semantic information and introduce overly rigid constraints, explaining why the final binary refinement does not further improve retrieval performance on NUS-WIDE TC-21.

For MM-IMDb, binary constraints may provide a clearer distinction between relevant and irrelevant pairs, making binary refinement more beneficial for retrieval. These results suggest that the effect of binary refinement depends on the label structure of the dataset and is not necessarily consistent across different datasets.

Thank you for your detailed rebuttal. The authors have answered most of my questions with their clarification. I have decided to maintain this positive review.

Reviewer_mi2BOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Aiming at the false positive and missing positive labels in noisy multi label cross modal hashing, this paper proposes RBCDH: first, use neighborhood support and cross modal consistency to make two-way corrections to labels with budget constraints, then use the continuous Soft Jackard relationship of the modified label to initialize the dual stream hash network, and finally fine tune the binary relationship derived from the same label. The effectiveness of the proposed algorithm is verified by several groups of experiments.

Critical strengths
  1. RBCDH first calculates the continuous Soft Jaccard relationship with the correction tag for semantic initialization, and then derives the fine tuning of the binary relationship with the same correction tag, so that the same supervision can be used consistently in the continuous and discrete spaces. The sequential rather than joint optimization method is simple, avoiding mutual interference between the two types of targets.
  2. PF-BLC estimates reliability at the instance class level in combination with the support of neighborhood labels within the mode and the cross-modal consistency, and simultaneously processes false positive and missing positive noises. The precision first threshold and class level and instance level budget constraints limit the risk of over correction and ensure that every instance will not lose all positive labels.
  3. Compared with recent robust methods under multiple data sets, noise rate and code length settings, RBCDH achieves the best in most settings, and verifies the role of key designs such as continuous semantic initialization through ablation experiments, which effectively supports the core proposition.
Critical weaknesses
  1. PF-BLC can only correct a small part of the noise under a fixed and small budget. The label reliability diagnosis shows that the label error rate has almost not decreased, but the relationship F1 has decreased. Ablation showed that after removing PF-BLC, it was still significantly better than the baseline, and its gain and variance were of the same magnitude, indicating that the lead might mainly come from Jaccard initialization rather than label correction.
  2. The noise model is symmetrical synthetic noise, which is highly corresponding to the budget design of the method, and lacks case related noise or real noise data. The number of baselines is small, and a variety of related robust methods are not included. The assessment also uses only two value related mAP, which does not fully match the motivation of "retaining hierarchical semantics".
Questions for authors
  1. Why are the values of the same configuration in Table 1 and Table 2 different (71.780 vs 71.751; 57.418 vs 57.553)?
  2. Soft Jaccard initialization on NUS-WIDE is 8.487 higher than binary initialization, while MM-IMDb is only 0.653. How to explain this difference?
  3. Is the validation set label also corrected? The Neighborhood Encoder trains and selects models with noise labels, and whether there is cyclic dependence under high noise
2 comments

Thank you for these insightful comments. Our responses are provided below:

Response to Q1:

Both experiments were conducted under the same settings, including a noise rate of ρ=0.5, a hash length of 64 bits, and three fixed random seeds (123, 2024, and 3407). For each dataset, the two experiments also used the same input features, data split, noise generation process, and PF-BLC-corrected labels. The difference arose because the results reported in the two tables were obtained from different model checkpoints, leading to slight variations in the reported values. To address this inconsistency, we reran the relevant ablation experiments using the same checkpoints in the revised version and updated the corresponding results.We have rewritten the ablation section in the article, and some of the smaller numerical differences compared to the previous experiment have been completely modified, located on pages 8-10 of the article. The new ablation table is on page 10 of the article.

Response to Q2:

The difference may be related to the distinct label co-occurrence patterns of the two datasets. NUS-WIDE TC-21 exhibits more diverse multi-label co-occurrence patterns, with varying degrees of semantic overlap between samples. Soft-Jaccard initialization preserves these graded relationships, whereas binary initialization maps positive relationships with different degrees of label overlap to the same binary target, losing some fine-grained semantic information. As a result, Soft-Jaccard initialization provides a greater benefit on NUS-WIDE TC-21. In contrast, the smaller performance gap on MM-IMDb suggests that preserving graded label relationships offers a more limited advantage on this dataset. This difference may also depend on how the initial semantic relationships interact with subsequent hash optimization.

Response to Q3:

We do not modify the labels in the validation set. The Neighborhood Encoder serves as an auxiliary component that extracts sample-level neighborhood relationships from the image and text modalities and provides semantic evidence for the subsequent PF-BLC module. Importantly, the corrected labels produced by PF-BLC are not fed back to retrain the Neighborhood Encoder or update the learned neighborhood structures. Therefore, there is no iterative feedback loop between neighborhood learning and label correction.

The author provided a detailed response to my concerns, and I maintain my original score.

Reviewer_FutSOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates cross-modal hashing under noisy multi-label supervision and proposes a reliability-driven framework, termed RBCDH. The key idea is to improve the quality of supervision before hash learning by jointly exploiting intra-modal neighborhood support and cross-modal consistency for bidirectional label correction at both the instance and class levels. Based on the corrected labels, the authors further construct continuous Soft Jaccard similarities for semantic initialization, and then refine the learned hash representations using the corresponding binary relations. Overall, the proposed framework is reasonably well motivated and the different components are connected in a coherent manner. In particular, the staged use of the corrected multi-label structure provides a natural link between reliable supervision construction, continuous semantic learning, and binary retrieval optimization. The experimental evaluation is also relatively comprehensive, covering two benchmark datasets, three noise levels, and four code lengths. In addition, the ablation studies and reliability analyses provide further evidence for the effectiveness of the proposed design.

Critical strengths

1.One advantage of this paper lies in the fact that it simultaneously considers two closely related issues in noisy multi-label cross-modal hashing: one is the distortion of cross-modal semantic relations caused by label noise, and the other is the loss of fine-grained semantic information when binary supervision is directly adopted. The proposed framework addresses the above issues in a progressive manner by combining instance-level and class-level bidirectional label correction, continuous semantic initialization, and subsequent binary relation refinement. Such a phased design is conceptually reasonable and also makes the overall learning process have better coherence.

2.Another meaningful aspect of this method lies in constructing the Soft Jaccard similarity using the corrected labels. Compared with binary pairwise supervision, this modeling approach can better preserve the varying degrees of semantic overlap among multi-label samples, thereby providing more informative semantic supervision for hash learning. Compared with the initialization method based on binary relations, this approach has achieved stable performance improvements on both datasets, which provides more convincing experimental support for this design choice.

3.Overall, the experimental results are relatively sufficient. Among the 24 evaluation settings composed of two datasets, multiple noise levels and different hash code lengths, the proposed method achieved the optimal average mAP in 23 of them, indicating that it has relatively stable performance under different conditions. The authors further provided component-level ablation experiments, multiple-random-seed experiments, and parameter sensitivity analysis, making the experimental evaluation more convincing. Furthermore, the paper not only reports favorable experimental results but also discusses the contributions and limitations of each component, which is worthy of recognition.

Critical weaknesses
  1. The appendix specifies the fixed FQ-PGNR-C features for NUS-WIDE TC-21 and the Qwen/ColQwen Full-Fusion features for MM-IMDb. Since these representations were prepared before RBCDH training, they constitute an important part of the experimental setup. This paper points out that the comparison methods use the same pre-extracted features and data splits, but readers would benefit from a clearer description of how these features are obtained and how the baseline implementations are adapted to them.
  2. Under the same settings, some values in the main result table and the ablation table are slightly different, and there are also ambiguities in the description of the ablation configurations in the text. It is suggested to explain the source of the numerical differences, unify the descriptions of the experimental configurations, simplify repetitive analysis, and enhance the clarity of the paper's presentation.
  3. Existing work has addressed the issues of neighborhood reliability and continuous semantic relations. This paper needs to more directly illustrate the unique contributions brought by the use of phased supervision. Furthermore, the current noise setting only removes and adds one label for each corrupted sample, and its effectiveness under asymmetric noise and multiple missing labels has not yet been verified.
Questions for authors
  1. Compared with existing methods such as SCBCH, how do the main methodological improvements in this paper differ in terms of label correction, training-stage division, and initialization objectives? Could the authors further explain why sequential training is superior to jointly using continuous and binary relations?
  2. Can the appendix provide more details about the fixed image and text features, including their sources and preprocessing procedures? This would help readers reproduce the experiments and better understand the role of the upstream representations throughout the pipeline.
  3. How many labels are actually removed and added in PF-BLC? What are the accuracies of these two operations, respectively? Do the existing results support the contribution of “restoring missing positive labels,” rather than merely suppressing false-positive relations?
1 comment

Thank you for these insightful comments. Our responses are provided below:

Response to Q1:

We acknowledge that SCBCH has already explored sample reliability and soft semantic relationships based on Jaccard similarity. However, our RBCDH differs in how it handles noisy labels and organizes the subsequent hash learning process. Specifically, PF-BLC explicitly corrects the observed label matrix through constrained label deletion and completion at both the class and instance levels.

RBCDH consists of two successive stages: Reliable Semantic Initialization and Cross-modal Hash Refinement. In the first stage, we use the corrected label matrix to construct continuous Soft-Jaccard relationships. Rather than directly generating binary relationships, this stage preserves different degrees of label overlap between sample pairs to provide fine-grained semantic supervision for hash initialization. In the second stage, we start from the hash representations learned in the first stage and retain the same fixed corrected label matrix. We then replace the continuous Soft-Jaccard objective with binary reliable relations to further refine the hash representations.

We adopt this sequential design because the two objectives serve different purposes. Continuous relationships first guide the hash network to capture fine-grained semantic similarities, while binary relationships are subsequently used to strengthen the distinction between relevant and irrelevant pairs. Jointly optimizing these two objectives may introduce competing constraints, particularly for sample pairs with partial label overlap. Sequential training allows the network to learn graded semantic relationships before applying binary constraints for further refinement.

Response to Q2:

We have added details about the feature sources, preprocessing procedures, and fusion protocols in the revised AppendixA.1- A.2 and Table 3.

For NUS-WIDE TC-21, the image features are the VGG19 fc7 features after ReLU activation, while the text features are obtained from TagAll1k. These text features are distinct from the 21-category labels used for supervision. For MM-IMDb, both image and text features are taken from the pre-extracted feature arrays provided with the GMU dataset. The corresponding dataset documentation describes the visual features as VGG-based representations and the text features as word-embedding-based representations.

Regarding preprocessing, the original feature blocks retain their stored values. For MiniLM- and ColQwen-based features, we follow the normalization and token-compression procedures specified in their respective feature-generation protocols. The feature blocks are then concatenated without applying additional global L2 normalization to the resulting fused vector.

Response to Q3:

To answer this question, we conducted an additional statistical analysis of all label correction operations performed by PF-BLC on the Fit subset under ρ=0.5 and a 64-bit setting, using three fixed random seeds. The results are summarized below.

DatasetDeleted labels (correct deletions)Deletion PrecisionAdded labels (correct additions)Addition Precision
NUS-WIDE TC-212,424 (1,748)0.721181 (49)0.6049
MM-IMDb6,005 (2,957)0.492411 (8)0.7273

Here, deletion precision is the proportion of removed labels that are false positives according to the original clean annotations, while addition precision is the proportion of added labels that are true positives according to the same reference. The clean annotations are used only to evaluate the correction operations, not to guide PF-BLC.

The results show that PF-BLC performs substantially more deletions than additions on both datasets. We think this phenomenon reflects a deletion-dominated correction strategy, in which substantially more labels are removed than added while allowing missing positives to be restored when sufficient evidence is available. Importantly, the addition operations are not merely nominal: 49 and 8 of the added labels are verified as correct on NUS-WIDE TC-21 and MM-IMDb, respectively. Thus, PF-BLC does recover some missing positive labels, although this operation plays a more limited role than false-positive suppression in the current experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.