- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies robust learning for cross-modal hashing under noisy labels. Existing methods typically rely on signals produced by the current model, such as training loss, prediction confidence, neighborhood relations, or semantic representations, to assess sample reliability. However, these signals are themselves affected by noisy supervision, which can cause reliability-estimation bias to accumulate throughout iterative optimization. To address this issue, the authors propose Robust Memory-guided Hashing (RMH), which leverages cross-modal semantic information accumulated from historically reliable samples to provide more stable references for current reliability estimation and robust hash learning.
Specifically, Memory-guided Reliability Estimation (MRE) constructs modality-specific class centers from historically reliable samples and evaluates the reliability of current samples through bidirectional cross-modal matching. Reliable Cross-batch Learning (RCL) jointly exploits reliable semantic relations from the current batch and historical memory for cross-batch learning, while uncertainty-aware learning reduces the influence of potentially erroneous supervision, thereby balancing robustness and data utilization.
The authors conduct systematic experiments on four cross-modal retrieval datasets under multiple noise patterns and noise rates. The results show that RMH consistently improves retrieval performance and robustness to noisy labels, and can be flexibly integrated into different cross-modal hashing methods.
Critical strengths
The motivation is clear, and the problem formulation and technical contributions are well defined.
The most insightful aspect of this paper is that it revisits a key premise in noise-robust learning, whether the evidence used to judge the reliability of supervision is itself sufficiently reliable. Existing methods assess reliability using signals such as loss, confidence, or semantic representations produced by the current model, yet these signals are themselves affected by noisy supervision. As the model gradually fits noisy labels, bias in reliability estimation can continue to accumulate, noisy samples that are mistakenly retained as reliable then re-enter optimization and further distort the Hamming space, forming a self-reinforcing error loop that aggravates noise propagation and error accumulation.
The key value of RMH lies in breaking this self-dependent mechanism. It uses cross-modal semantic information accumulated from historically reliable samples to provide a more stable reference for current reliability estimation and to further guide robust hash learning. Particularly noteworthy is that memory is not treated merely as a cache for expanding training relations. Instead, it serves two closely connected roles: reliability estimation and robust cross-batch learning. Reliable samples continually maintain the historical semantic reference, the resulting reliable historical information in turn supports subsequent sample selection and representation learning, uncertainty-aware learning controls the influence of potentially erroneous supervision from unreliable samples. This design re-examines noise-robust learning from the perspective that reliability estimation itself can be continuously contaminated by noise, resulting in a technically clear, coherent, and insightful framework.
The empirical validation is comprehensive and forms a rigorous validation loop with the paper's core technical claims.
The paper systematically evaluates the overall retrieval performance of RMH across four datasets, multiple noise patterns, different noise rates, and different hash-code lengths, and further provides a direct analysis of the core reliability-estimation mechanism. The authors use four metrics, for example AUROC, Reliable Precision, False-retention Rate, and Memory Purity to evaluate, from complementary perspectives, MRE's ability to separate clean and noisy samples, maintain high-quality reliable-sample selection, reduce the erroneous retention of noisy samples, and preserve a high-quality historical memory.
These metrics directly correspond to the paper's core motivation of stabilizing reliability estimation and suppressing contamination of historical memory. The paper further evaluates integration with different hashing objectives, generalization across different backbones, efficiency, and statistical significance, providing additional evidence for the method's generality, practical cost, and reliability of the reported gains. Overall, the experiments provide a fairly complete validation of the proposed technical contributions and establish a rigorous logical chain from motivation to method design to empirical evidence.
Critical weaknesses
The problem formulation and technical contributions are clear, and the empirical evaluation is comprehensive. I did not identify any technical flaw that would justify rejection. The following two points are mainly implementation details that I would like the authors to clarify in the rebuttal.
- For uncertainty-aware learning, I would like the authors to clarify how triplets involving unreliable samples are constructed. The paper currently describes them as cross-modal triplets involving unreliable samples, but it would be helpful to specify which roles unreliable samples may take, such as anchor, positive, or negative, and whether the same uncertainty modulation is applied in these different cases. Clarifying this detail would improve reproducibility.
- The paper already provides a fairly complete overall efficiency analysis, but the sources of the additional training-time computation could be described more explicitly. For example, it would be useful to indicate whether the extra cost mainly comes from memory matching, cross-batch triplet construction, or the uncertainty-aware branch, so that readers can better assess the training cost of applying the method to larger-scale datasets.
Questions for authors
Q1. In uncertainty-aware learning, can unreliable samples serve as anchor, positive, and negative samples, and is the same uncertainty modulation applied when they appear in different roles?
Q2. Which component contributes most to the additional training cost of RMH, such as memory matching, cross-batch triplet construction, or the uncertainty-aware branch, and what are their approximate relative costs?
5 comments
Response to Reviewer AwMW [1/2]
Response to W1 & Q1
We thank the reviewer for closely examining triplet construction for unreliable samples and uncertainty-aware learning. Clear sample roles, candidate sets, and similarity modulation rules are essential for understanding and reproducing the method. We therefore clarify these details based on the implementation.
a. Triplet Construction and Active Constraints
MRE partitions each mini-batch into reliable () and unreliable () subsets. Equation (6) constructs in-batch triplets from ; Eq. (7) uses anchors from and positives/negatives from historical Memory. In Eq. (9), anchors, positives, and negatives all come from the current , without reliable samples or historical memory as candidates, and include both cross-modal and intra-modal relations. Positive pairs share at least one observed positive label; negative pairs share none. We enumerate all admissible triplets, average only those with nonzero margin loss, and perform no additional semi-hard negative mining. The loss is zero if no active triplets exist.
b. Uncertainty Modulation and Memory Updates
Equation (8) defines pairwise uncertainty-aware similarity from normalized hash-representation distance and joint uncertainty norm . Equation (9) averages the margin loss over active triplets in the current unreliable subset:
\frac{1}{|\mathcal T_u|} \sum_{(a,p,n)\in\mathcal T_u} \left[\mu+S_u(a,n)-S_u(a,p)\right]_+.$$ Here, $\mathcal T_u$ is the set of triplets with nonzero margin loss in the current unreliable subset; the loss is zero if this set is empty. Positive and negative pairs use the same similarity function, with independent modulation by their respective uncertainties and no additional role-specific weights or gates. Unreliable samples contribute only to current-batch uncertainty-aware learning and do not update historical Memory; historical representations receive no cross-batch gradients. During cold start, the first mini-batch initializes Memory; thereafter, only samples deemed reliable are admitted to limit contamination of historical semantic references. The revised manuscript clarifies these loss and update rules.Response to Reviewer AwMW [2/2]
Response to W2 & Q2
We thank the reviewer for examining RMH's training efficiency and computational overhead. Identifying module costs and their sources is important for evaluating practical efficiency. We therefore supplement the original overall-efficiency analysis with module-level timing experiments.
Table 16 in Appendix E.8 of the original manuscript reported RMH's overall efficiency on MIRFlickr-25K with 32-bit codes under 80% symmetric noise: 11.15 G FLOPs, 151.34 M parameters, an estimated 0.88 h for 100 training epochs, and 1.270 ms to encode an image–text query pair, with average mAP of 78.60%. To identify training overhead sources, we measure MRE, CBL, UA, and other Memory operations separately.
MRE includes historical class-center updates, reliability scoring, and sample selection; CBL covers cross-batch relation construction, loss evaluation, and gradient computation; UA covers uncertainty modulation, loss evaluation, and gradient computation; and other Memory operations include reads and maintenance. Timings use synchronized, separate segments: gradients with respect to intermediate representations are computed for each loss branch and then aggregated, while shared-network forward and backward passes are excluded from module-exclusive time.
Table 1: Per-module timing on MIRFlickr-25K (32 bits, 80% symmetric noise). Share denotes the percentage of measured module time, excluding shared-network computation.
| Module | Forward (ms) | Backward (ms) | Total (ms) | Share (%) |
|---|---|---|---|---|
| MRE | 4.046 | 0.000 | 4.046 | 20.23 |
| CBL | 2.281 | 8.927 | 11.208 | 56.04 |
| UA | 1.685 | 2.887 | 4.572 | 22.86 |
| Other Memory | 0.173 | 0.000 | 0.173 | 0.87 |
| Total | 8.185 | 11.814 | 19.999 | 100.00 |
As Table 1 shows, CBL has the largest module cost, mainly from computing gradients for the cross-batch loss rather than reading or maintaining historical memory. MRE and UA cost less, and other Memory operations are minor. This analysis identifies the relative module costs and their main sources. The reported shares concern only measured module operations, not total training time.
Thanks for your review
Dear Reviewer AwMW,
We sincerely thank you for your thoughtful review and valuable suggestions on triplet construction, uncertainty-aware learning, memory updates, and computational efficiency. Your feedback has helped us clarify key implementation details and strengthen our analysis of computational overhead. We provide detailed responses to your concerns below.
Sincerely,
The Authors