acceptodds
Under review as a conference paper at ICLR 2027

Robust Memory-guided Hashing for Cross-modal Retrieval with Noisy Labels

Abstract

Cross-modal hashing (CMH) provides an efficient solution for large-scale multimodal retrieval, but its performance is vulnerable to noisy labels. Existing noise-robust CMH methods typically estimate sample or label reliability based on signals derived from the current model, such as training losses, prediction confidence, neighborhood relations, or learned semantic representations. However, these signals are themselves affected by noisy supervision, and biases in reliability estimation can be progressively amplified during iterative optimization, leading to noise propagation and error accumulation. To address this issue, we propose Robust Memory-guided Hashing (RMH), which exploits semantic information accumulated from historically reliable samples to provide stable references for current reliability estimation and hash learning. Specifically, Memory-guided Reliability Estimation (MRE) constructs modality-specific historical class centers from historically reliable samples, estimates the reliability of current samples through bidirectional cross-modal matching, and employs a smooth threshold to partition samples into reliable and unreliable subsets while updating the memory only with reliable samples. Furthermore, we develop Reliable Cross-batch Learning (RCL), which jointly exploits in-batch and historical cross-batch semantic relations for learning from reliable samples, while introducing an uncertainty-aware loss to attenuate potentially erroneous supervision associated with unreliable samples, thereby improving data utilization and the robustness of hash representations. Extensive experiments under multiple noise patterns and noise rates demonstrate that RMH achieves consistently competitive retrieval performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Deep Hashing, Cross-modal Retrieval, Noisy-label Learning, Memory-guided Reliability Estimation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AwMWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies robust learning for cross-modal hashing under noisy labels. Existing methods typically rely on signals produced by the current model, such as training loss, prediction confidence, neighborhood relations, or semantic representations, to assess sample reliability. However, these signals are themselves affected by noisy supervision, which can cause reliability-estimation bias to accumulate throughout iterative optimization. To address this issue, the authors propose Robust Memory-guided Hashing (RMH), which leverages cross-modal semantic information accumulated from historically reliable samples to provide more stable references for current reliability estimation and robust hash learning.

Specifically, Memory-guided Reliability Estimation (MRE) constructs modality-specific class centers from historically reliable samples and evaluates the reliability of current samples through bidirectional cross-modal matching. Reliable Cross-batch Learning (RCL) jointly exploits reliable semantic relations from the current batch and historical memory for cross-batch learning, while uncertainty-aware learning reduces the influence of potentially erroneous supervision, thereby balancing robustness and data utilization.

The authors conduct systematic experiments on four cross-modal retrieval datasets under multiple noise patterns and noise rates. The results show that RMH consistently improves retrieval performance and robustness to noisy labels, and can be flexibly integrated into different cross-modal hashing methods.

Critical strengths

The motivation is clear, and the problem formulation and technical contributions are well defined.

The most insightful aspect of this paper is that it revisits a key premise in noise-robust learning, whether the evidence used to judge the reliability of supervision is itself sufficiently reliable. Existing methods assess reliability using signals such as loss, confidence, or semantic representations produced by the current model, yet these signals are themselves affected by noisy supervision. As the model gradually fits noisy labels, bias in reliability estimation can continue to accumulate, noisy samples that are mistakenly retained as reliable then re-enter optimization and further distort the Hamming space, forming a self-reinforcing error loop that aggravates noise propagation and error accumulation.

The key value of RMH lies in breaking this self-dependent mechanism. It uses cross-modal semantic information accumulated from historically reliable samples to provide a more stable reference for current reliability estimation and to further guide robust hash learning. Particularly noteworthy is that memory is not treated merely as a cache for expanding training relations. Instead, it serves two closely connected roles: reliability estimation and robust cross-batch learning. Reliable samples continually maintain the historical semantic reference, the resulting reliable historical information in turn supports subsequent sample selection and representation learning, uncertainty-aware learning controls the influence of potentially erroneous supervision from unreliable samples. This design re-examines noise-robust learning from the perspective that reliability estimation itself can be continuously contaminated by noise, resulting in a technically clear, coherent, and insightful framework.

The empirical validation is comprehensive and forms a rigorous validation loop with the paper's core technical claims.

The paper systematically evaluates the overall retrieval performance of RMH across four datasets, multiple noise patterns, different noise rates, and different hash-code lengths, and further provides a direct analysis of the core reliability-estimation mechanism. The authors use four metrics, for example AUROC, Reliable Precision, False-retention Rate, and Memory Purity to evaluate, from complementary perspectives, MRE's ability to separate clean and noisy samples, maintain high-quality reliable-sample selection, reduce the erroneous retention of noisy samples, and preserve a high-quality historical memory.

These metrics directly correspond to the paper's core motivation of stabilizing reliability estimation and suppressing contamination of historical memory. The paper further evaluates integration with different hashing objectives, generalization across different backbones, efficiency, and statistical significance, providing additional evidence for the method's generality, practical cost, and reliability of the reported gains. Overall, the experiments provide a fairly complete validation of the proposed technical contributions and establish a rigorous logical chain from motivation to method design to empirical evidence.

Critical weaknesses

The problem formulation and technical contributions are clear, and the empirical evaluation is comprehensive. I did not identify any technical flaw that would justify rejection. The following two points are mainly implementation details that I would like the authors to clarify in the rebuttal.

  • For uncertainty-aware learning, I would like the authors to clarify how triplets involving unreliable samples are constructed. The paper currently describes them as cross-modal triplets involving unreliable samples, but it would be helpful to specify which roles unreliable samples may take, such as anchor, positive, or negative, and whether the same uncertainty modulation is applied in these different cases. Clarifying this detail would improve reproducibility.
  • The paper already provides a fairly complete overall efficiency analysis, but the sources of the additional training-time computation could be described more explicitly. For example, it would be useful to indicate whether the extra cost mainly comes from memory matching, cross-batch triplet construction, or the uncertainty-aware branch, so that readers can better assess the training cost of applying the method to larger-scale datasets.
Questions for authors

Q1. In uncertainty-aware learning, can unreliable samples serve as anchor, positive, and negative samples, and is the same uncertainty modulation applied when they appear in different roles?

Q2. Which component contributes most to the additional training cost of RMH, such as memory matching, cross-batch triplet construction, or the uncertainty-aware branch, and what are their approximate relative costs?

5 comments
AuthorsOpenReview ↗

Thanks for your review

Dear Reviewer AwMW,

We sincerely thank you for your thoughtful review and valuable suggestions on triplet construction, uncertainty-aware learning, memory updates, and computational efficiency. Your feedback has helped us clarify key implementation details and strengthen our analysis of computational overhead. We provide detailed responses to your concerns below.

Sincerely,
The Authors

AuthorsOpenReview ↗

Response to Reviewer AwMW [1/2]

Response to W1 & Q1

We thank the reviewer for closely examining triplet construction for unreliable samples and uncertainty-aware learning. Clear sample roles, candidate sets, and similarity modulation rules are essential for understanding and reproducing the method. We therefore clarify these details based on the implementation.

a. Triplet Construction and Active Constraints

MRE partitions each mini-batch into reliable (BrB^r) and unreliable (BuB^u) subsets. Equation (6) constructs in-batch triplets from BrB^r; Eq. (7) uses anchors from BrB^r and positives/negatives from historical Memory. In Eq. (9), anchors, positives, and negatives all come from the current BuB^u, without reliable samples or historical memory as candidates, and include both cross-modal and intra-modal relations. Positive pairs share at least one observed positive label; negative pairs share none. We enumerate all admissible triplets, average only those with nonzero margin loss, and perform no additional semi-hard negative mining. The loss is zero if no active triplets exist.

b. Uncertainty Modulation and Memory Updates

Equation (8) defines pairwise uncertainty-aware similarity Su(i,j)S_u(i,j) from normalized hash-representation distance αij\alpha_{ij} and joint uncertainty norm βij\beta_{ij}. Equation (9) averages the margin loss over active triplets in the current unreliable subset:

\frac{1}{|\mathcal T_u|} \sum_{(a,p,n)\in\mathcal T_u} \left[\mu+S_u(a,n)-S_u(a,p)\right]_+.$$ Here, $\mathcal T_u$ is the set of triplets with nonzero margin loss in the current unreliable subset; the loss is zero if this set is empty. Positive and negative pairs use the same similarity function, with independent modulation by their respective uncertainties and no additional role-specific weights or gates. Unreliable samples contribute only to current-batch uncertainty-aware learning and do not update historical Memory; historical representations receive no cross-batch gradients. During cold start, the first mini-batch initializes Memory; thereafter, only samples deemed reliable are admitted to limit contamination of historical semantic references. The revised manuscript clarifies these loss and update rules.
AuthorsOpenReview ↗

Response to Reviewer AwMW [2/2]

Response to W2 & Q2

We thank the reviewer for examining RMH's training efficiency and computational overhead. Identifying module costs and their sources is important for evaluating practical efficiency. We therefore supplement the original overall-efficiency analysis with module-level timing experiments.

Table 16 in Appendix E.8 of the original manuscript reported RMH's overall efficiency on MIRFlickr-25K with 32-bit codes under 80% symmetric noise: 11.15 G FLOPs, 151.34 M parameters, an estimated 0.88 h for 100 training epochs, and 1.270 ms to encode an image–text query pair, with average mAP of 78.60%. To identify training overhead sources, we measure MRE, CBL, UA, and other Memory operations separately.

MRE includes historical class-center updates, reliability scoring, and sample selection; CBL covers cross-batch relation construction, loss evaluation, and gradient computation; UA covers uncertainty modulation, loss evaluation, and gradient computation; and other Memory operations include reads and maintenance. Timings use synchronized, separate segments: gradients with respect to intermediate representations are computed for each loss branch and then aggregated, while shared-network forward and backward passes are excluded from module-exclusive time.

Table 1: Per-module timing on MIRFlickr-25K (32 bits, 80% symmetric noise). Share denotes the percentage of measured module time, excluding shared-network computation.

ModuleForward (ms)Backward (ms)Total (ms)Share (%)
MRE4.0460.0004.04620.23
CBL2.2818.92711.20856.04
UA1.6852.8874.57222.86
Other Memory0.1730.0000.1730.87
Total8.18511.81419.999100.00

As Table 1 shows, CBL has the largest module cost, mainly from computing gradients for the cross-batch loss rather than reading or maintaining historical memory. MRE and UA cost less, and other Memory operations are minor. This analysis identifies the relative module costs and their main sources. The reported shares concern only measured module operations, not total training time.

Reviewer_FPj3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Robust Memory-guided Hashing (RMH) for image-text retrieval with corrupted multi-label annotations. RMH maintains a FIFO memory of samples deemed reliable, constructs modality-specific historical class centers for bidirectional reliability estimation, learns reliable examples using current- and cross-batch triplets, and applies an uncertainty-aware triplet objective to examples deemed unreliable.

Critical strengths

S1. The use of historically reliable samples to assess current sample reliability is intuitive and well motivated. RMH constructs separate historical class centers for images and texts, then uses bidirectional cross-modal matching to assess whether a sample’s labels are supported.

S2. The experiments cover a broad range of settings and analyze the method's components. The paper reports results on four datasets, multiple hash code lengths, and three types of synthetic label noise. It also includes reliability analyses, ablations, and parameter studies.

Critical weaknesses

W1. The reliability threshold depends on the estimated noise rate, but the paper does not explain how to estimate it when the true noise rate is unknown. In addition, samples enter the memory during warm-up without reliability filtering, and subsequent reliability estimates depend on class centers built from that memory. The current experiments do not show when this feedback process recovers or when it may be biased by early contamination, especially under high noise or different noise types.

W2. The reliability score sums probabilities over positive labels, which may favor samples with more labels. The uncertainty vectors are also neither calibrated nor constrained, so they could act as gates that suppress difficult constraints. In addition, the paper does not report results across multiple random seeds or compare against several closely related robust cross-modal hashing methods, making it difficult to assess the method’s robustness and relative contribution.

Questions for authors

Q1. How is ρ hat obtained in practice when clean labels and the true noise rate are unavailable? Could you report performance when the estimated rate is misspecified, and analyze how warm-up memory contamination affects reliability selection and subsequent memory purity?

Q2. Are samples with more labels more likely to be selected as reliable because the score sums probabilities over positive labels? Could you report selection metrics stratified by label cardinality and compare the current score with normalized or cardinality-adjusted alternatives? Could you also analyze the gradients and norm distributions of the uncertainty vectors to verify that they reflect label noise rather than simply suppressing the loss for difficult samples?

Q3. Could you compare RMH with NrDCMH, DCGMH, MGSH, and historical-reliability-based methods under the same backbone, data splits, and noise settings, and report means and variances across multiple random seeds? Please also specify how cross-batch and unreliable-sample triplets are constructed, the minimum number of samples needed to establish a class center, and whether the CB indicator for “Base + MRE + CB” in Table 4 is incorrect.

6 comments
AuthorsOpenReview ↗

Thanks for your review

Dear Reviewer FPj3,

We sincerely thank you for your careful review and valuable suggestions regarding reliability estimation, memory recovery, uncertainty-aware learning, and experimental validation. Your feedback has helped us clarify key methodological details and strengthen our evaluation of robustness and reproducibility. The corresponding clarifications and additional experimental results have been incorporated into the revised manuscript. We provide detailed, point-by-point responses below.

Sincerely,
The Authors

AuthorsOpenReview ↗

Response to Reviewer FPj3 [1/3]

Response to W1 & Q1

We thank the reviewer for these valuable comments. We agree that threshold selection under unknown noise rates and recovery from initially contaminated memory are central to RMH's practical robustness. We therefore add noise-rate misspecification experiments and high-noise memory comparisons to assess the stability of reliability selection and the effectiveness of selective memory updates, respectively.

a. Threshold Selection and Robustness under Unknown Noise Rates

In Eq. (4), ρ^\hat{\rho} is a preset noise-rate estimate used to set the quantile threshold of reliability scores. In the main experiments with synthetic label noise, we set ρ^=ρ\hat{\rho}=\rho, where ρ\rho is the predefined noise injection rate. Since the true noise rate is unknown in practice, we further vary ρ^\hat{\rho} across different true noise rates to assess how estimation mismatch affects retrieval performance, reliable selection precision, and memory purity.

Table 1: Noise-rate sensitivity on MIRFlickr-25K (64 bits, symmetric noise). Results are in percentages; Reliable Precision and Memory Purity are measured at epoch 100.

ρ/ρ^\rho/\hat{\rho}I2T/T2IAverageReliable PrecisionMemory Purity
0.2/0.284.27/83.0283.6589.5290.94
0.2/0.584.48/83.8684.1784.0391.84
0.2/0.883.63/82.8883.2694.8594.78
0.5/0.282.49/82.0882.2973.8274.37
0.5/0.581.52/81.9581.7476.9777.50
0.5/0.880.97/81.3881.1782.7984.22

As Table 1 shows, at each true noise rate, the maximum difference in average mAP across ρ^\hat{\rho} settings is about 1.1 percentage points. Increasing ρ^\hat{\rho} improves Memory Purity but does not consistently improve retrieval, reflecting a trade-off between reliable selection and retaining useful supervision. Notably, fixing ρ^=0.5\hat{\rho}=0.5 yields retrieval performance comparable to the known-rate setting at both true noise rates. Thus, under the tested mismatches, RMH is relatively insensitive to the noise-rate prior and maintains stable retrieval without knowing the true rate precisely.

b. Recovery from Contaminated Initial Memory

We agree that unfiltered samples during cold start may contaminate historical memory, warranting an examination of whether subsequent selective updates improve memory quality. Under the stated noise assumptions, Eq. (15) in Appendix B.1 gives the conditional expected fraction of incorrect positive-class assignments at initialization as ρ/Lˉ0\rho/\bar L_0, where Lˉ0\bar L_0 is the average label cardinality. Thus, sample-level and class-assignment contamination rates differ. Equation (23) further describes memory contamination updates: ηt+1=ηt+ωt(at−dt),\eta_{t+1}=\eta_t+\omega_t(a_t-d_t), Here, ata_t and dtd_t denote the contamination rates of newly admitted and evicted samples, respectively, and ωt\omega_t is the replacement fraction. Memory contamination decreases when at<dta_t<d_t. This characterizes a condition for memory purification, not an unconditional recovery guarantee. Figure 5 in the original manuscript showed that, on NUS-WIDE under 50% symmetric noise, RMH's Memory Purity rises from a low initial level and remains above unfiltered Historical-all. To test recovery under more severe and varied noise, we add comparisons under 80% symmetric and asymmetric noise. For each setting, we maintain three shadow memories on the same trained model's representation stream: RMH admits reliability-selected samples; Historical-all admits all samples; and Random-admission admits the same number as RMH at random. All use the same capacity, isolating reliability-based admission from mere replacement.

Table 2: Memory recovery on MIRFlickr-25K (64 bits, 80% noise). AUROC/MP (%) are reported for symmetric and asymmetric noise; MP is measured at epoch 100.

MethodSym. AUROC/MPAsym. AUROC/MP
RMH73.97/42.8168.76/41.09
Historical-all69.74/18.3456.43/19.69
Random-admission64.91/20.9456.13/19.84

Table 2 shows that RMH achieves higher AUROC and Memory Purity than both controls under either noise type. The new asymmetric-noise trajectories further show an early increase in sample-level memory purity and consistently higher class-assignment purity; these curves will appear in Figure 6 of the revised manuscript. Thus, under the tested high-noise settings, memory improvement is associated with reliability-guided selection, not merely FIFO replacement or admission count. This agrees with the conditional purification analysis in Appendix B.1 and supports gradual improvement of initially contaminated historical semantic references.

AuthorsOpenReview ↗

Response to Reviewer FPj3 [2/3]

Response to W2 & Q2

We thank the reviewer for raising two important concerns: potential label-cardinality bias in reliability estimation and possible oversuppression of hard triplets by uncertainty-aware learning. These concern selection fairness and the actual effect of uncertainty modulation, respectively. We therefore complement the original ablations with cardinality-stratified analyses and triplet-level gradient diagnostics.

a. Label-Cardinality Bias and Reliability Selection

We agree that Sum Support aggregates the total probability mass of observed positive labels and may favor samples with more positive labels. Table 8 in Appendix E.3 of the original manuscript compared Sum and Mean Support: their average mAP differs by only 0.18 and 0.15 percentage points under 50% and 80% symmetric noise, respectively. Similar overall retrieval performance, however, does not rule out cardinality-dependent selection bias. We therefore train both variants under identical settings and stratify samples by observed positive-label count (k=1k=1, k=2k=2, and k≥3k\geq3), comparing AUROC and Reliable Precision (RP) within each group.

Table 3: Reliability analysis by observed label cardinality on MIRFlickr-25K (128 bits, 50% symmetric noise). C/N denotes clean/noisy sample counts; AUROC and RP are reported in percentages.

kn (C/N)Sum AUROC/RPMean AUROC/RP
11235 (655/580)71.30/78.4368.49/77.79
21616 (809/807)76.41/75.7468.45/73.93
≥37149 (3536/3613)68.32/66.1066.88/61.63

As Table 3 shows, Sum Support achieves higher AUROC and RP in all three cardinality groups, indicating that simple mean normalization does not improve within-group reliability discrimination or selection precision. These results do not, however, exclude selection bias across cardinality groups; we clarify this limitation in Appendix E.3 of the revised manuscript.

b. Uncertainty-aware Learning and Gradient Attenuation

We understand the concern that uncertainty-aware learning may benefit from suppressing gradients of all hard triplets rather than selectively attenuating erroneous supervision. RMH's learned uncertainty representations modulate semantic relations; they are not calibrated predictions of label corruption. Appendix B.2 analyzes local gradient modulation by uncertainty-aware similarity. Under 80% symmetric noise, Table 9 in Appendix E.4 of the original manuscript reports 79.62% mAP for joint uncertainty-aware optimization, versus 77.04% for Fixed Weight, 72.58% for Learned Weight, and 77.19% for Detached. Figure 8 of the original manuscript further contrasts the uncertainty trajectories and gradient behavior of joint optimization and direct weight learning.

To examine whether modulation is associated with corrupted supervision, we compare relation uncertainties and local gradient-norm ratios between Clean-active and Flipped triplets. Specifically, β+=∥ua+up∥2\beta^+=\lVert u_a+u_p\rVert_2 and β−=∥ua+un∥2\beta^-=\lVert u_a+u_n\rVert_2 measure positive and negative relation uncertainties, respectively; r=∥∇hLUA∥2/∥∇hLraw∥2r=\lVert\nabla_h L_{\mathrm{UA}}\rVert_2/\lVert\nabla_h L_{\mathrm{raw}}\rVert_2 measures the relative local gradient magnitude after modulation.

Table 4: Relation uncertainty and local gradient ratios for matched triplets on MIRFlickr-25K (64 bits). C/F denotes Clean-active/Flipped; Pairs is the number of matched triplet pairs.

NoisePairsβ+\beta^+ C/Fβ−\beta^- C/Frr C/F
50%5425.540/6.1783.664/4.6410.540/0.489
80%4205.168/5.7473.467/3.9580.578/0.523

As Table 4 shows, under both noise rates, Flipped triplets have higher positive and negative relation uncertainties but lower local gradient-norm ratios than Clean-active triplets. Within the evaluated triplets, modulation is thus associated with label-reversal errors rather than merely uniform gradient scaling. Together with the original ablations, this supports the attenuation of some erroneous semantic constraints by uncertainty-aware learning.

Reviewer_HQTROpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses noisy-label cross-modal hashing by proposing RMH, a framework that mitigates the bias of model-dependent reliability estimation through historical semantic references. RMH comprises two modules: (1) MRE maintains a reliable-sample FIFO memory bank to establish historical class centers, estimates reliability via bidirectional cross-modal matching, and identifies reliable samples using smoothed quantile thresholds; (2) RCL combines intra- and inter-batch learning for reliable samples while suppressing noisy gradients through uncertainty-aware learning. Extensive experiments demonstrate RMH’s effectiveness and robustness across multiple datasets and noise settings.

Critical strengths
  1. The design is well justified and empirically supported. RMH leverages historical semantics from reliable samples to reduce accumulated estimation bias and memory contamination. Figure 5 compares RMH with variants such as Current-batch and Historical-all using AUROC, Precision, and Memory Purity, demonstrating the effectiveness of historical semantic references and selective memory updates in improving estimation stability and reducing noise accumulation.

  2. Figure 2 clearly illustrates the RMH framework and aligns well with the methodology. Extensive experiments show consistent improvements over recent baselines across datasets and noise settings, particularly under severe noise. Significance tests over 10 independent runs further strengthen the credibility of the results.

Critical weaknesses
  1. RMH relies on an estimated noise rate ρ^\hat{\rho} for reliability selection, but its effectiveness when the noise rate is unknown remains insufficiently validated. The sensitivity analysis in Figure 6 covers only one noise setting, limiting its practical implications.

  2. Appendix E.1 compares multiple historical reference strategies, but does not specify the update interval for periodic recomputation or the momentum coefficient for EMA.

  3. Most experiments use a fixed batch size of 128, although reliability estimation and triplet construction depend on mini-batch composition. A small sensitivity analysis with different batch sizes would help further validate the stability of RMH.

  4. The experiments use pre-generated noisy labels, but it is unclear whether the reported results cover different noise realizations. Additional experiments with different noise seeds would further strengthen the robustness evaluation.

  5. The memory bank is initially populated with unfiltered samples during warm-up, potentially introducing substantial contamination under severe noise. Although Appendix B provides conditional theoretical analysis, empirical evidence on how reliably the memory recovers from heavily corrupted initialization remains limited.

Questions for authors
  1. In Table 6, what are the update interval for periodic recomputation and the momentum coefficient for EMA, respectively, and what is the basis for selecting them?

  2. How does RMH perform under different batch sizes, such as 64 and 256?

  3. Do the reported experiments cover different noise realizations, and can results with additional noise seeds be provided?

  4. How does memory purity evolve from heavily contaminated initialization under severe noise, such as 80% corruption?

7 comments
AuthorsOpenReview ↗

Thanks for your review

Dear Reviewer HQTR,

We sincerely appreciate your thorough review and constructive suggestions concerning noise-rate assumptions, historical memory updates, batch-size sensitivity, and statistical reliability. In response, we have clarified the relevant implementation details and conducted additional experiments to assess RMH's robustness and effectiveness. These clarifications and experimental results have been incorporated into the revised manuscript. We address your comments in detail below.

Sincerely,
The Authors

AuthorsOpenReview ↗

Response to Reviewer HQTR [1/4]

Response to W1

We thank the reviewer for examining reliability selection when the noise rate is unknown. We agree that dependence of threshold selection on the true noise rate is important for RMH's practical applicability. We therefore clarify the parameter setting and add noise-rate misspecification experiments to evaluate robustness without an accurate noise-rate prior.

In Eq. (4), ρ^\hat{\rho} is a preset noise-rate estimate used to set the reliability-score quantile threshold. In the main experiments with synthetic label noise, we use ρ^=ρ\hat{\rho}=\rho, where ρ\rho is the predefined noise injection rate. To address unknown true noise rates, we further assess retrieval performance, reliable selection precision, and memory purity across different ρ\rho and ρ^\hat{\rho} combinations.

Table 1: Noise-rate sensitivity on MIRFlickr-25K (64 bits, symmetric noise). Results are in percentages; Reliable Precision and Memory Purity are measured at epoch 100.

ρ/ρ^\rho/\hat{\rho}I2T/T2IAverageReliable PrecisionMemory Purity
0.2/0.284.27/83.0283.6589.5290.94
0.2/0.584.48/83.8684.1784.0391.84
0.2/0.883.63/82.8883.2694.8594.78
0.5/0.282.49/82.0882.2973.8274.37
0.5/0.581.52/81.9581.7476.9777.50
0.5/0.880.97/81.3881.1782.7984.22

As Table 1 shows, at each true noise rate, average mAP varies by at most about 1.1 percentage points across ρ^\hat{\rho} settings. Increasing ρ^\hat{\rho} generally improves Memory Purity, but not retrieval performance monotonically, reflecting a trade-off between reliable selection and retaining useful supervision. Notably, a fixed ρ^=0.5\hat{\rho}=0.5 achieves retrieval performance comparable to the known-rate setting at both true noise rates. These results show that RMH tolerates noise-rate estimation errors and maintains stable cross-modal retrieval without precise knowledge of the true rate under the tested conditions.

AuthorsOpenReview ↗

Response to Reviewer HQTR [2/4]

Response to W2 & Q1

We thank the reviewer for examining historical semantic-reference updates and their parameter choices. Clear update rules and sensitivity analyses are important for interpreting the ablations and ensuring reproducibility. We therefore clarify implementation details and add parameter-sensitivity experiments to further assess the update strategies.

Periodic Re-computation initializes class centers when historical references first become available, then recomputes them from representations stored in FIFO Memory every Δ=10\Delta=10 mini-batches. Centers remain fixed between updates; classes absent from the current memory are excluded from matching, and training data need not be re-encoded. EMA Center instead updates class centers using currently selected reliable samples: cc,newm=normalize⁡(αcc,oldm+(1−α)bcm),c_{c,\mathrm{new}}^m=\operatorname{normalize}\left(\alpha c_{c,\mathrm{old}}^m+(1-\alpha)b_c^m\right), Here, bcmb_c^m is the normalized class center of current reliable samples, with default momentum α=0.9\alpha=0.9. New classes are initialized directly; classes absent from the current batch retain their centers, and updates occur after reliability assessment. EMA does not use FIFO Memory to construct reliability references, but still uses historical memory for cross-batch learning. By contrast, FIFO Memory updates relevant class centers from current memory contents without an additional update interval or momentum coefficient. Table 6 in Appendix E.1 of the original manuscript showed that FIFO Memory outperforms Periodic Re-computation and EMA Center under the tested settings. We further test different update intervals Δ\Delta and momentum coefficients α\alpha.

Table 2: Sensitivity to reference-update parameters on MIRFlickr-25K (64 bits, 50% symmetric noise). Results are reported as mAP (%).

MethodParameterI2TT2IAverage
PeriodicΔ=1\Delta=181.27481.45481.364
PeriodicΔ=5\Delta=581.55081.36281.456
PeriodicΔ=10\Delta=1081.16081.72081.440
PeriodicΔ=20\Delta=2081.17381.40981.291
EMAα=0.5\alpha=0.581.65881.00881.333
EMAα=0.7\alpha=0.781.15481.47881.316
EMAα=0.9\alpha=0.981.30081.77081.535
EMAα=0.99\alpha=0.9981.41181.49281.452

As Table 2 shows, average mAP ranges from 81.29% to 81.46% for Periodic Re-computation and from 81.32% to 81.54% for EMA Center, both below FIFO Memory's 81.74%. Thus, FIFO Memory's advantage is not solely due to the default parameters of the alternatives, further supporting its historical-reference update strategy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.