- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies supervised multi-label cross-modal hashing when the training annotations contain false-positive and missing-positive labels. The authors propose RBCDH, which first estimates label reliability using intra-modal neighborhood support and cross-modal consistency, then uses the corrected labels to guide continuous semantic initialization and subsequent binary hash refinement. The Soft-Jaccard objective is intended to preserve differences in label overlap that would otherwise be lost when the corrected annotations are converted into binary relations. Experiments on NUS-WIDE TC-21 and MM-IMDb show that RBCDH achieves the best Avg mAP in 23 of the 24 experimental settings across the two datasets, three noise levels, and four hash-code lengths. The method is straightforward to follow, and the paper addresses a relevant issue in robust cross-modal retrieval.
Critical strengths
(1) Correcting noisy annotations does not automatically preserve the degree of semantic overlap between instances. RBCDH addresses this issue by using continuous Soft-Jaccard relations during initialization and binary relations for the final retrieval-oriented optimization. This gives the method a clear motivation beyond simply filtering unreliable labels.
(2) PF-BLC produces a fixed corrected label matrix, which is then reused in the two learning stages. The auxiliary neighborhood encoder is not transferred to the hashing network, keeping the stages distinct. This sequential design makes the training procedure relatively easy to understand and implement.
(3) RBCDH performs well on both datasets under varying annotation noise and code lengths. The ablations also offer useful insight into the method: replacing Soft-Jaccard initialization with binary supervision leads to a substantial drop on NUS-WIDE, supporting the motivation for preserving graded semantic relations.
Critical weaknesses
(1) PF-BLC obtains neighborhood evidence from representations learned using the observed labels. If the noise has already influenced these representations, the resulting neighborhoods may sometimes reinforce incorrect label associations. Cross-modal consistency helps reduce this risk, but agreement between two modalities does not necessarily mean that the shared label is correct. The paper would benefit from a clearer discussion of this limitation and of the circumstances in which the two sources of evidence are expected to be reliable.
(2) The label-correction diagnostics show a meaningful precision-recall trade-off. On both datasets, PF-BLC increases relation precision and reduces the false-positive rate, but recall also decreases. On MM-IMDb, the reported element-wise label error changes from 0.04348 to 0.04368. These results do not negate the usefulness of the correction strategy, particularly given its precision-first design, but they suggest that the corrected labels should not be interpreted as uniformly more accurate. It would be useful to explain why this particular trade-off is beneficial for the subsequent hashing objective.
(3) The contribution of final hash refinement is not consistent across the two datasets. On MM-IMDb, full RBCDH improves Avg mAP by 0.508 percentage points over the variant without the final hash refinement stage. On NUS-WIDE TC-21, however, removing refinement yields a higher score, 72.508% versus 71.751%. The authors acknowledge this difference in the manuscript, which is appropriate. A little more discussion of why binary refinement can be helpful on one dataset but detrimental on the other would make the role of this stage clearer.
Questions for authors
Q1. Since the neighborhood encoder is trained on noisy labels, how does PF-BLC avoid reinforcing incorrect associations when noisy labels also influence the learned neighborhood structure? A short discussion of this potential feedback effect would be helpful.
Q2. The reliability diagnostics suggest that PF-BLC is more effective at reducing false-positive relations than at recovering missing-positive relations. Do the authors have an explanation for why this precision-oriented correction remains beneficial for retrieval, despite the associated reduction in recall?
Q3. Given that the final binary refinement does not improve NUS-WIDE TC-21 in the reported ablation, is there a property of the corrected relation distribution or the dataset that might explain the difference between the two benchmarks?
2 comments
Thank you for your detailed rebuttal. The authors have answered most of my questions with their clarification. I have decided to maintain this positive review.
Thank you for these insightful comments. Our responses are provided below:
Response to Q1:
The Neighborhood Encoder is used as an auxiliary component to extract sample neighborhood relationships from the image and text modalities. These relationships provide additional evidence for PF-BLC. To reduce the influence of noisy neighborhood information, PF-BLC evaluates neighborhood label support separately in the two modalities, rather than relying on either modality alone. It also adopts a conservative correction strategy by applying class-level and instance-level constraints to label deletion and completion.
The corrected labels are not used to retrain the Neighborhood Encoder or update its learned neighborhood structures. Thus, correction errors are not fed back into the encoder through repeated updates. However, when correlated label noise affects both modalities, they may still provide consistent but incorrect neighborhood evidence. This limitation cannot be completely avoided, particularly under high noise rates.
Response to Q2:
False-positive relations can pull semantically unrelated image-text pairs closer in the learned hash space, increasing the likelihood that irrelevant samples are retrieved as matches. By removing these unreliable positive constraints, PF-BLC aims to improve semantic discrimination and reduce incorrect matches.
We acknowledge that deleting some positive relations may also remove genuine semantic associations, causing related samples to lose their positive constraints and potentially reducing retrieval recall. However, false-positive relations introduce misleading similarity constraints, whereas missing positive relations mainly result in insufficient supervision for some genuinely related pairs. This difference helps explain why reducing false-positive relations can still benefit retrieval despite some loss of recall. The benefit of this precision-oriented correction strategy therefore depends on whether the reduction in incorrect matches outweighs the loss of genuine positive relations.
Response to Q3:
We believe the difference may be related to the different label co-occurrence patterns of the two datasets. NUS-WIDE TC-21 contains more diverse multi-label combinations, leading to varying degrees of label overlap between sample pairs. Soft-Jaccard initialization can preserve these graded relationships, whereas binary refinement compresses different degrees of positive overlap into uniform binary constraints. Such discretization may discard useful fine-grained semantic information and introduce overly rigid constraints, explaining why the final binary refinement does not further improve retrieval performance on NUS-WIDE TC-21.
For MM-IMDb, binary constraints may provide a clearer distinction between relevant and irrelevant pairs, making binary refinement more beneficial for retrieval. These results suggest that the effect of binary refinement depends on the label structure of the dataset and is not necessarily consistent across different datasets.