acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Noisy-Label Cross-Modal Hashing with Dual-Source Semantic Consensus

Abstract

Most cross-modal hashing (CMH) methods encourage semantic consistency across modalities, implicitly assuming that cross-modal agreement is uniformly beneficial. Under noisy labels, however, different modalities can respond differently to the same supervision, making indiscriminate semantic alignment susceptible to propagating unstable predictions across modalities. This motivates a key question: how can cross-modal semantic interaction be conditionally regulated when modality-specific predictions exhibit heterogeneous reliability? To address this issue, we propose a dual-source semantic consensus framework, termed DUET, which regulates cross-modal semantic interaction from complementary temporal and instantaneous perspectives. Specifically, Temporal Consensus Learning (TCL) aggregates predictions across training epochs to characterize their relative temporal stability, while Modality Consensus Learning (MCL) exploits the concentration of current prediction distributions to adaptively determine each modality's contribution to the semantic consensus and the corresponding consensus constraint. By jointly considering these complementary cues, DUET avoids treating all modality predictions as equally influential and enables conditional semantic interaction during cross-modal learning. Extensive experiments on three benchmark datasets demonstrate that DUET consistently improves retrieval performance across a broad range of noise levels and exhibits effective adaptive behavior under heterogeneous modality-specific responses to noisy supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Cross‑modal Hashing, Noisy Labels, Semantic Consensus

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.25 over 4 reviews, above 96% of the others.
Reviewer_KtsJOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DUET for cross-modal hashing under noisy labels. Temporal Consensus Learning maintains historical predictions and weights modalities according to their temporal stability. Modality Consensus Learning constructs an entropy-weighted consensus and applies stronger consistency constraints to less confident modalities. Both components operate through shared learnable class prototypes in the continuous hash space. The evaluation covers three image–text datasets, multiple noise rates and hash lengths, with additional backbone and three-modal experiments. The reported retrieval performance is strong, although the relationship between the specified training procedure and noisy-label supervision requires clarification.

Critical strengths

The paper addresses a relevant limitation of cross-modal agreement: different modalities can provide evidence of different reliability for the same example. Combining historical stability with current prediction uncertainty offers a useful approach to making consensus adaptive rather than treating all modalities equally.

The experimental coverage is substantial. DUET achieves the highest reported average bidirectional mAP in 46 of the 48 main dataset–noise–hash-length settings. The appendix reports means and standard deviations over three independent runs and includes backbone comparisons and three-modal retrieval results. These results support the potential practical value of the method.

Critical weaknesses

The complete supervision pathway is missing from the specified method. Equation (8) initializes temporal targets from model predictions, Equation (19) contains only TCL and MCL, and Algorithm 1 randomly initializes the hashing networks and prototypes. None of these steps explicitly uses the observed noisy labels. With fixed features, initialization, and sampling, changing only the labels would therefore leave the described optimization unchanged. This makes it difficult to interpret the reported robustness as an effect of learning under corrupted supervision. The objective also permits constant, concentrated predictions with zero MCL and small TCL loss, without useful retrieval semantics. An omitted supervised component could resolve these concerns, but it must be stated explicitly.

Questions for authors

Where exactly do the noisy labels enter training? Please provide the complete implemented objective and any label-dependent initialization, pretraining, or sampling steps omitted from Equation (19) and Algorithm 1. A small check holding features, initialization, and sampling fixed while changing only the labels would help establish this dependence. Please also identify the component that anchors the class prototypes and discourages constant semantic predictions. If labels are intentionally excluded, please clarify the interpretation of the noise-rate experiments.

Are α in Equation (14) and the coefficient (1-\beta) in Equation (18) detached during optimization? Detaching the consensus target alone does not specify the gradient treatment of these coefficients. Please state the implemented policy and, on one existing configuration such as XMedia with 64-bit codes and 0.8 noise, compare it with detached loss weights. This would clarify whether the gains come from adaptive weighting or additionally from gradients through the weighting functions.

4 comments
AuthorsOpenReview ↗

Response to Reviewer KtsJ (1/2)

We appreciate the reviewer for raising these critical questions regarding the supervision pathway, the role of noisy labels, and the gradient treatment of adaptive weighting coefficients. We clarify these aspects and address each concern below.

(1) Explicit supervision pathway and complete training objective

We agree that the supervision pathway should be explicitly stated. In our implementation, the observed noisy labels y~i\tilde{\boldsymbol{y}}_i are directly incorporated into the temporal target update in Eq. (9), which was inadvertently omitted from the original manuscript. The complete update is given by p~im[t]=ηp~im[t−1]+(1−η)(pim[t]+y~i)\tilde{\boldsymbol{p}}_i^{m[t]} = \eta\tilde{\boldsymbol{p}}_i^{m[t-1]}+ (1-\eta) \left(\boldsymbol{p}_i^{m[t]}+\tilde{\boldsymbol{y}}_i\right).

Here, pim[t]\boldsymbol{p}_i^{m[t]} denotes the current prediction of modality mm, while y~i\tilde{\boldsymbol{y}}_i represents the observed noisy label. Consequently, the temporal target is jointly determined by the model prediction and the provided annotation, rather than being generated solely from model predictions.

In addition, our class prototypes are initialized using statistical information extracted from the training noisy labels, rather than being initialized independently of the annotations. This introduces label-dependent semantic information into the prototype initialization and establishes an initial correspondence between the prototypes and the annotated classes.

Importantly, the noisy labels are incorporated through the label-dependent temporal target construction and prototype initialization; there is no additional cross-entropy loss in our implementation. TCL exploits the label-informed temporal targets to stabilize intra-modal learning, while MCL coordinates modality-specific predictions through reliability-weighted cross-modal consensus.

To further verify the dependence of training on observed noisy labels, we conducted an additional label replacement experiment on XMedia using 16-bit hash codes under a label noise rate of 0.4, following the reviewer's suggestion. Specifically, we fixed the input features, network initialization, prototype initialization, mini-batch sampling order, and all other training configurations. We then replaced the original noisy labels with randomly shuffled labels while preserving the overall label distribution. Only the labels used in the temporal target update were changed. This controlled setting isolates the effect of label information in Eq. (9) from that of label-dependent prototype initialization.

Under the original and replaced labels, the temporal targets are respectively updated as $\tilde{\boldsymbol{p}}_i^{m[t]}=\eta\tilde{\boldsymbol{p}}_i^{m[t-1]}+(1-\eta)\left( \boldsymbol{p}_i^{m[t]}+\tilde{\boldsymbol{y}}_i\right)$, and $\tilde{\boldsymbol{p}}_i'^{m[t]}=\eta\tilde{\boldsymbol{p}}_i'^{m[t-1]}+ (1-\eta)\left(\boldsymbol{p}_i'^{m[t]}+\tilde{\boldsymbol{y}}_i'\right)$.

We compared the resulting temporal targets, TCL gradients, and final retrieval performance. The results are summarized below.

SettingTemporal target differenceTCL gradient differencemAP
Original noisy labels––89.6
Replaced noisy labels0.310.2776.2

The differences in temporal targets and TCL gradients directly demonstrate that changing the observed labels changes the optimization signals, even when the features, initialization, and sampling order remain fixed. The retrieval results further illustrate the effect of preserving or disrupting the correspondence between training samples and their observed labels. These results provide direct evidence that DUET is trained under the provided noisy label supervision rather than being label-agnostic.

AuthorsOpenReview ↗

Response to Reviewer KtsJ (2/2)

(2) Class-prototype anchoring and prevention of trivial predictions

We acknowledge that the reviewer raises an important concern about whether consistency-based objectives alone can guarantee meaningful semantic predictions. Indeed, TCL and MCL primarily impose temporal stability and cross-modal consistency, respectively. If considered without the label-dependent components, these objectives alone would not provide sufficient class-specific supervision and could admit uninformative solutions.

In DUET, semantic anchoring comes from two label-dependent operations. First, the class prototypes are initialized using statistical information extracted from the observed noisy labels, providing an initial correspondence between the prototypes and the annotated semantic classes. Second, the noisy labels are explicitly incorporated into the temporal targets used by TCL, allowing annotation-related information to influence the learning process throughout training. Consequently, the temporal learning objective is influenced by class-dependent annotation information, rather than merely encouraging predictions to remain unchanged over time.

MCL further exploits the resulting modality-specific predictions to establish cross-modal semantic consensus. Therefore, these components play distinct but complementary roles: label-dependent prototype initialization establishes the initial class correspondence, label-informed temporal targets provide annotation-related guidance during optimization, and MCL coordinates predictions across modalities.

We emphasize that TCL and MCL alone are not intended to guarantee non-degenerate semantic predictions. Their effectiveness relies on the label-dependent information introduced through prototype initialization and temporal target construction.

(3) Gradient treatment of α\boldsymbol{\alpha} and (1−β)(1-\boldsymbol{\beta})

We thank the reviewer for pointing out that detaching the consensus target alone does not fully specify the gradient treatment of the adaptive weighting coefficients. In our implementation, both α\boldsymbol{\alpha} in Eq. (14) and (1−β)(1-\boldsymbol{\beta}) in Eq. (18) are detached from the computational graph during backpropagation. They are used as adaptive weighting statistics calculated from the forward-pass predictions, rather than as differentiable optimization variables. Therefore, gradients are not propagated through these coefficients, while the corresponding prediction and representation pathways remain differentiable. The consensus target is also detached as described in our original formulation.

To examine the effect of this design choice, we conducted an additional comparison on XMedia using 64-bit hash codes under a label noise rate of 0.8. Specifically, we compared the original implementation, in which both adaptive weighting coefficients are detached, with a variant that allows gradients to propagate through α\boldsymbol{\alpha} and β\boldsymbol{\beta}. All other experimental settings and training procedures were kept identical. The results are reported below.

SettingmAP
Ours (detached α\boldsymbol{\alpha} and 1−β1-\boldsymbol{\beta})72.6
Variant (gradients through α\boldsymbol{\alpha} and β\boldsymbol{\beta})64.1

The results show that allowing gradients to propagate through the adaptive weighting coefficients leads to a substantial performance drop. This suggests that treating these coefficients as differentiable optimization variables can introduce additional gradient pathways through which the model may alter the estimated reliability weights to reduce the objective, rather than improve the underlying semantic predictions. By detaching the coefficients, their role is restricted to adaptively regulating the relative contributions of different samples and modalities, while the network parameters are optimized through the corresponding weighted objectives. This comparison supports our design choice and demonstrates the benefit of using detached adaptive weighting coefficients, rather than allowing gradients to propagate through the weighting functions.

Reviewer_KtsJOpenReview ↗

I have raised my score

my questions solved

AuthorsOpenReview ↗

Thanks to Reviewer KtsJ

We are glad that our revisions have addressed all your questions. Should you have any further concerns, we are happy to provide additional clarifications.

Reviewer_NSRmOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

DUET addresses noisy‑label cross‑modal hashing via dual‑source semantic consensus. TCL builds temporal‑consensus targets from past model outputs and computes temporal‑stability weights. MCL uses prediction entropy to compute modality weights for building consensus distribution, and applies asymmetric KL loss: high‑confidence modalities contribute more to consensus while receiving weaker constraints, low‑confidence modalities get stronger regularization. Training proceeds in two phases: warm‑up uses only TCL; afterwards both TCL and weighted MCL loss are active. Two theorems analyze the bound of temporal aggregation and weighted consensus error. The paper validates contributions via ablation studies, hyper‑parameter sweep, convergence analysis and comparisons with SOTA hashing methods.

Critical strengths

This paper makes solid progress for noisy‑label cross‑modal hashing by proposing the DUET framework. Specifically:

  1. Instead of symmetric cross‑modal alignment loss widely used in CMH, MCL assigns different contribution and constraint strength for each modality dynamically per‑sample, which is a meaningful methodological novelty for noisy multimodal learning.

  2. Even with simplifying assumptions, the theoretical analysis helps understand the behavior of temporal aggregation and weighted consensus.

Critical weaknesses

I have minor and fixable concerns regarding this submission.

  1. The method relies on multiple manually tuned hyper‑parameters (\eta, \lambda, E_{warm}). While appendix provides sensitivity plots, there is no automatic adaptation mechanism. Performance may drop when transferring to unseen datasets without careful grid search.

  2. TCL runs in warm‑up phase and shapes the starting point for MCL. It is hard to tell how much final performance comes purely from intra‑modal temporal smoothing versus the cross‑modal MCL component. More fine‑grained ablations would help.

Questions for authors

Three main questions for the authors:

  1. All reliability weighting and consensus calculations operate on a per‑sample basis without exploiting intra‑batch relationships. Could you discuss whether batch‑level semantic context would further improve the modality reliability estimation? Would there be potential risks when introducing batch‑aware computation?

  2. What would you expect to happen under asymmetric label noise, where image and text modalities suffer different noise corruption rates?

  3. You adopt tanh as the continuous relaxation for binary hash representations. Could you discuss whether different hash relaxation strategies would affect TCL and MCL's reliability weight estimation?

4 comments
AuthorsOpenReview ↗

Response to Reviewer NSRm (1/2)

We thank the reviewer for these constructive comments and valuable questions. We appreciate the recognition that the concerns are minor and addressable. We provide the following point-by-point responses to clarify the design choices, component contributions, and broader applicability of DUET.

(1) Manually tuned hyperparameters and the lack of automatic adaptation

We appreciate this reasonable concern regarding the manually selected hyperparameters η\eta, λ\lambda, and EwarmE_{\mathrm{warm}}. Our appendix provides sensitivity analyses showing that DUET maintains stable performance over a relatively broad range of values rather than relying on a narrowly tuned configuration. Nevertheless, we acknowledge that the current framework does not explicitly adapt these hyperparameters to different datasets. Directly transferring DUET to unseen datasets without appropriate parameter adjustment may therefore lead to suboptimal performance. We acknowledge this limitation in F.10 (LIMITATION STATEMENT). Developing self-adaptive strategies that adjust the learning schedule and loss contributions according to training dynamics is a promising direction for future work.

(2) Disentangling the contributions of intra-modal TCL and cross-modal MCL

Thank you for pointing out the difficulty of separating the contributions of intra-modal temporal smoothing and cross-modal consensus learning. Our existing ablation studies, particularly DUET-1 (TCL-only) and DUET-2 (MCL-only), provide high-level comparisons of the two components.

In DUET, TCL operates throughout training to regularize intra-modal prediction stability. Its warm-up phase suppresses short-term prediction fluctuations and establishes more reliable modality-specific predictions before MCL is activated. After warm-up, TCL remains active to continuously stabilize intra-modal learning. Consequently, MCL operates on TCL-regularized representations rather than directly relying on unstable predictions.

In contrast, MCL introduces explicit cross-modal interaction. It assigns modality-wise weights according to prediction entropy, constructs an aggregated semantic consensus, and applies asymmetric KL-divergence constraints to reconcile inconsistent predictions across modalities. While TCL improves the reliability of individual modality predictions, it cannot explicitly resolve cross-modal semantic discrepancies. MCL addresses this limitation by coordinating information from different modalities according to their estimated reliability.

Therefore, TCL mitigates modality-specific overfitting to noisy labels, whereas MCL improves semantic consistency through reliability-aware cross-modal alignment. Their contributions are complementary rather than independent or simply additive, since MCL partly depends on the quality of the modality-specific predictions established by TCL.

(3) Per-sample reliability estimation versus batch-level semantic context

We agree that the current reliability weighting and consensus computation operate at the per-sample level without explicitly exploiting intra-batch semantic relationships. In principle, incorporating batch-level context could provide additional information for reliability estimation. For example, semantic similarities among samples within a mini-batch could help determine whether neighboring samples provide supporting evidence for a modality-specific prediction.

However, batch-aware computation also involves potential risks. First, reliability estimates may become sensitive to batch composition, causing weights to fluctuate across iterations even when the underlying representations change only slightly. Second, under noisy supervision, incorrect predictions or unreliable neighboring samples may contaminate batch-level semantic relationships and propagate errors across samples. Third, relying explicitly on batch context may introduce a discrepancy between training and inference, particularly when predictions are evaluated independently for individual image-text pairs.

Therefore, although batch-level semantic context may provide complementary information, it also introduces additional dependencies and potential sources of noise. We adopt per-sample reliability estimation to maintain a direct and stable correspondence between each sample's prediction behavior and its reliability weight.

AuthorsOpenReview ↗

Response to Reviewer NSRm (2/2)

(4) Behavior under asymmetric label noise across modalities

We appreciate this important question. Asymmetric label noise, in which image and text modalities are corrupted at different rates, is closely related to the modality-heterogeneous noise scenario motivating DUET. Our framework does not assume identical noise levels across modalities. Specifically, TCL independently tracks temporal prediction stability for each modality, while MCL estimates modality reliability based on the concentration of current predictions. When one modality is more heavily corrupted and produces less stable or less concentrated predictions, its corresponding αim\alpha_i^m and βim\beta_i^m can decrease. This reduces its influence on intra-modal learning and cross-modal consensus, allowing the relatively more reliable modality to provide stronger guidance.

To further examine this behavior, we conducted additional asymmetric-noise experiments on XMedia. We compared a symmetric setting, where both modalities are corrupted at a noise rate of 0.4, with an asymmetric setting, where image and text modalities are corrupted at rates of 0.2 and 0.6, respectively. Both settings have the same average noise rate of 0.4.

SettingImage noiseText noisemAP
Symmetric0.40.489.6
Asymmetric0.20.687.0

The results show that retrieval performance decreases from 89.6 to 87.0 under asymmetric noise, despite the same average noise rate. This suggests that an imbalanced noise distribution introduces additional challenges for cross-modal semantic alignment. Although DUET can adaptively reduce the contribution of a less reliable modality, annotations in the more heavily corrupted modality may still affect cross-modal semantic correspondence. Nevertheless, DUET maintains meaningful retrieval performance under this challenging setting, demonstrating its applicability beyond the symmetric-noise assumption.

(5) Influence of hash relaxation strategies on reliability weight estimation

We thank the reviewer for raising this question. In DUET, tanh⁡\tanh is adopted as the continuous relaxation function for binary hash representations. The semantic predictions used to compute αim\boldsymbol{\alpha}_i^m and βim\boldsymbol{\beta}_i^m are obtained from these relaxed representations. Therefore, different relaxation strategies may affect the distribution and optimization behavior of continuous hash features, further influencing semantic predictions and the resulting reliability estimates.

For example, sigmoid and tanh⁡\tanh may produce different feature distributions and gradient magnitudes, potentially affecting prediction concentration and temporal stability. Straight-through estimators (STEs), on the other hand, use a different gradient approximation strategy and may introduce different optimization behavior. If a relaxation strategy produces poorly calibrated or unstable semantic predictions, the resulting entropy and temporal stability statistics may become less reliable, affecting the weighting mechanisms in TCL and MCL.

Nevertheless, DUET's reliability estimation principles are not specific to tanh⁡\tanh. As long as the adopted representation strategy preserves meaningful semantic discrimination and produces sufficiently stable predictions, TCL and MCL can still estimate modality reliability and construct cross-modal consensus. We adopt tanh⁡\tanh because its bounded output range naturally approximates the binary hash space −1,+1\\{-1,+1\\} while providing a smooth optimization pathway.

Thank you for your detailed response. My concerns have been resolved and I will keep the original rating.

AuthorsOpenReview ↗

Thanks to Reviewer NSRm

We are very happy that our revisions have addressed all your concerns. Thanks for your hard work again.

Reviewer_Lur2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DUET for noisy‑label cross‑modal hashing. Instead of applying identical cross‑modal consistency loss on all modalities, DUET adaptively assigns modality weights from temporal prediction history and current prediction entropy. TCL aggregates historical predictions to measure intra‑modal temporal stability; MCL constructs weighted semantic consensus and imposes asymmetric KL‑driven constraints. The model is evaluated on XMedia, XMediaNet, INRIA‑Websearch under noise ratios from 0.2‑0.8. Extensive ablations verify each component. Results show better mAP compared to recent SOTA noisy‑label cross‑modal hashing methods.

Critical strengths

(1) DUET outperforms many recent SOTA approaches consistently on multiple benchmarks, and hash lengths and noise levels. The statistical repeatability with multiple random seeds further strengthens experimental credibility. (2) The learning pipeline is easy to implement and can be plugged on top of existing cross‑modal hashing backbones, shown by experiments with CLIP‑based features.

Critical weaknesses

(1) The class prototypes are shared across modalities in the current implementation. Although an ablation against modality‑specific prototypes is given in the appendix, there is little discussion on how shared prototypes may implicitly couple image and text representations before MCL is activated. It is hard to isolate whether performance gains stem from explicit MCL loss or this pre‑existing parameter‑level coupling. (2) Limited discussion against non‑hashing cross‑modal retrieval baselines. The paper focuses exclusively on hashing methods. It would be helpful to briefly position DUET against general noisy‑label cross‑modal retrieval literature outside the hashing domain. (3) The method is validated on 16‑128 bit hash lengths, but does not discuss behavior for extremely short hash bits (e.g., 8‑bit) with very limited representation capacity. (4) In MCL, the consensus distribution (q_i[t]) uses stop‑gradient on the aggregated consensus. It is not fully discussed what risks stop‑gradient brings to gradient flow and representation updating.

Questions for authors

Could you briefly discuss how your core idea could be migrated to non‑hashing cross‑modal retrieval setups? Also please clarify the key differences when removing the binary hashing constraint.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Lur2 (1/2)

We thank the reviewer for pointing out these critical weaknesses, which help us clarify key design choices and the broader applicability of the DUET framework. We address each concern point-by-point below.

(1) Shared class prototypes and implicit parameter-level coupling before MCL activation

We agree that this requires clearer elaboration. As reported in our appendix, shared and modality-specific prototypes yield only marginal performance differences at the end of the warm-up stage. During warm-up, the TCL loss optimizes each modality against its own historical predictions and does not explicitly align image-text representations. Thus, explicit cross-modal alignment primarily originates from the subsequent LMCL\mathcal{L}_{MCL}, rather than prototype sharing. Although shared prototypes introduce a certain degree of implicit parameter-level coupling, our existing ablation results suggest that the performance gains are unlikely to be dominated by this effect. Instead, MCL provides explicit prediction-level interaction by aggregating modality-specific predictions and enforcing cross-modal consistency.

(2) Limited positioning against non-hashing noisy-label cross-modal retrieval baselines

Our experiments focus on cross-modal hashing, targeting efficient Hamming-space retrieval for large-scale applications. In contrast, non-hashing methods learn continuous real-valued representations and perform retrieval through similarity computation in the embedding space, serving different application requirements. Importantly, the core idea of DUET is not inherently dependent on binary hashing. It derives reliability signals directly from model predictions through temporal prediction stability and instantaneous prediction concentration, without explicitly estimating label corruption.

Therefore, DUET can be extended to non-hashing cross-modal retrieval by replacing binary hash codes with continuous embeddings while retaining the TCL and MCL objectives. Specifically, temporal stability can still regulate intra-modal learning, and prediction-based reliability weights can still guide cross-modal consensus construction and consistency learning. The main differences lie in removing the sign-based binarization and quantization constraints and replacing Hamming-distance retrieval with continuous similarity matching.

(3) Missing discussion and evaluation for extremely short hash codes such as 8-bit

We appreciate this suggestion. Following the reviewer's comment, we additionally evaluated DUET using 8-bit hash codes. The results are compared with those obtained using longer hash codes, as shown below.

MethodBitNoise-0.2Noise-0.4Noise-0.6Noise-0.8
DUET885.969.770.015.4
DUET1690.289.689.787.9
DUET3291.691.491.390.9
DUET6492.292.092.091.6

The results show that 8-bit codes can still achieve reasonable performance under relatively low noise, but their performance deteriorates substantially under increasing label noise, particularly dropping to 15.4 mAP under 80% noise. In comparison, longer hash codes maintain considerably more stable performance across different noise levels.

This degradation is mainly attributable to the limited representation capacity of ultra-short binary codes. An 8-bit code provides only 282^8 possible binary patterns, making it difficult to preserve diverse semantic distinctions and distinguish samples with ambiguous or corrupted labels. Under severe label noise, the limited code space further amplifies semantic confusion and code collisions. Although TCL and MCL improve learning by exploiting temporal stability and cross-modal prediction consensus, they cannot fully overcome the intrinsic representation bottleneck imposed by an extremely short code length. These additional results provide a clearer understanding of the limitations of DUET under ultra-compact representations.

AuthorsOpenReview ↗

Response to Reviewer Lur2 (2/2)

(4) Potential risks of stop-gradient on the consensus distribution qi[t]q_i[t] in MCL

We agree that the gradient-flow behavior deserves further clarification. In MCL, stop-gradient is applied to the aggregated consensus qi[t]q_i[t] when it serves as the target distribution in the KL-divergence loss. Without this operation, each modality could simultaneously update its own prediction and modify the consensus target that guides cross-modal learning, introducing coupled feedback between the predictions and the consensus. Under noisy supervision, such feedback may cause unreliable predictions to alter the learning target, potentially leading to unstable optimization or consensus drift.

With stop-gradient, qi[t]q_i[t] acts as a fixed target within each gradient update, while the modality-specific predictions still receive gradients from the KL-divergence loss and their corresponding representations continue to be optimized. Importantly, the consensus is recomputed from the current predictions during training; thus, it evolves as the representations improve, rather than remaining permanently fixed. Moreover, βim\beta_i^m is a statistic computed from prediction entropy rather than a learnable parameter. Therefore, blocking gradients through the aggregated consensus does not prevent the network parameters from being optimized through the modality-specific prediction paths.

(5) Extension of DUET to non-hashing cross-modal retrieval

We thank the reviewer for this valuable question. The core idea of DUET is not inherently dependent on binary hashing, as its key components operate on model predictions rather than binary representations. Therefore, DUET can be extended to non-hashing cross-modal retrieval by replacing binary hash codes with continuous-valued embeddings while retaining the TCL and MCL objectives. Specifically, TCL can still exploit temporal prediction stability to regulate intra-modal learning, while MCL can continue to use prediction-concentration-based weights to construct cross-modal consensus and enforce semantic consistency. The main differences after removing the binary hashing constraint lie in the representation and retrieval stages: the sign-based binarization and associated quantization constraints are no longer required, and Hamming-distance-based retrieval is replaced by similarity computation in the continuous embedding space. Although this extension sacrifices the compact storage and efficient Hamming-distance matching offered by hashing, it preserves the central reliability-aware learning principle of DUET.

Thank you for the detailed response and clarifications which address several of my concerns. I will keep my original score unchanged.

AuthorsOpenReview ↗

Thanks to Reviewer Lur2

Thank you for your hard work. If you have any further questions, we would be happy to provide additional clarification.

Reviewer_GykbOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies noisy‑label cross‑modal hashing. The authors observe that different modalities exhibit heterogeneous responses to identical label noise, so blindly enforcing uniform cross‑modal alignment may propagate erroneous predictions across modalities. They propose DUET with two components: Temporal Consensus Learning computes temporal‑stability weights from historical predictions, and Modality Consensus Learning builds weighted semantic consensus based on per‑instance prediction entropy. The two modules work in a two‑stage training pipeline. The paper provides two corresponding theoretical bounds, and conducts extensive experiments across three main benchmarks plus a tri‑modal dataset under multiple noise ratios.

Critical strengths
  1. Insight about modality‑heterogeneous noise susceptibility. Most prior noisy‑label cross‑modal hashing treats label corruption as modality‑agnostic disturbance. This work points out that image and text can fit noisy labels at very different speeds, which is an important real‑world observation and motivates the whole method.
  2. Complete supporting materials. The paper includes formal theorem analysis, rich ablations, parameter sensitivity, convergence curves and t‑SNE visualization, which makes empirical validation thorough.
Critical weaknesses

While the paper has promising results, I also identify two key limitations that should be addressed in revision.

  1. The paper uses two complementary weighting mechanisms αim\alpha_i^m from TCL and βim\beta_i^m from MCL, but it remains unclear how these two weights interact with each other during training. It is not analyzed whether the two weights tend to agree or conflict for individual samples, and how their conflict would influence the final semantic consensus. The manuscript only evaluates the full combined pipeline, lacking sample‑level analysis of weight interaction.
  2. How would performance degrade if warm‑up epochs are set too large or too small under heavy label noise? Could you give empirical guidance for choosing EwarmE_{warm}?
Questions for authors

See Critical Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Gykb

We thank the reviewer for these encouraging comments and constructive suggestions. We are pleased that our observation of modality-heterogeneous noise susceptibility, which motivates the proposed DUET framework, is recognized as a valuable real-world insight. We address the reviewer's concerns point-by-point below.

(1) Interaction between TCL weight αim\boldsymbol{\alpha}_{i}^{m} and MCL weight βim\boldsymbol{\beta}_{i}^{m}

We appreciate this insightful comment regarding the interaction between the TCL-derived temporal stability weights αim\boldsymbol{\alpha}_{i}^{m} and the MCL-derived prediction-concentration weights βim\boldsymbol{\beta}_{i}^{m}. These two weights characterize different aspects of modality reliability: αim\boldsymbol{\alpha}_{i}^{m} captures the long-term temporal stability of predictions, whereas βim\boldsymbol{\beta}_{i}^{m} reflects the concentration of current predictions. Accordingly, they serve distinct purposes in DUET: αim\boldsymbol{\alpha}_{i}^{m} reweights LTCL\mathcal{L}_{TCL} to stabilize intra-modal learning, while βim\boldsymbol{\beta}_{i}^{m} determines each modality's contribution to cross-modal consensus and regulates the corresponding asymmetric KL-divergence constraint in LMCL\mathcal{L}_{MCL}.

Importantly, the two weights do not directly compete through element-wise weighting. Instead, they influence different optimization components and jointly shape the learned representations through the overall objective. Their modality preferences may either agree or differ, since temporal stability and current prediction concentration are not necessarily correlated. For example, a modality may produce temporally stable but relatively diffuse predictions, or concentrated predictions that fluctuate across training epochs. In the former case, temporal stability does not necessarily indicate that the current prediction is sufficiently informative; in the latter, high prediction concentration does not necessarily imply reliable temporal behavior. By exploiting these complementary signals, TCL stabilizes intra-modal learning, while MCL adaptively regulates cross-modal information aggregation and consistency constraints.

Our existing analysis in Fig. 5 further illustrates their distinct behaviors. At the early training stage, the text modality exhibits higher temporal stability and prediction-concentration weights than the image modality, whereas the differences between modalities gradually diminish as training proceeds. Moreover, the ablation studies involving DUET-1, DUET-2, and DUET-4 show that using either weighting mechanism alone does not achieve the performance of the full model, supporting the complementary contributions of TCL and MCL.

(2) Warm-up hyperparameter EwarmE_{\mathrm{warm}} analysis

Thank you for raising this practical question regarding the warm-up hyperparameter EwarmE_{\mathrm{warm}}. We adopt Ewarm=3E_{\mathrm{warm}}=3 in our experiments to allow each modality to establish preliminary prediction stability through LTCL\mathcal{L}_{TCL} before introducing cross-modal consensus learning. The warm-up duration involves a trade-off, particularly under heavy label noise. If EwarmE_{\mathrm{warm}} is too small, MCL may be activated before modality-specific predictions become sufficiently stable, increasing the risk of propagating unreliable predictions through cross-modal consensus. Conversely, an excessively large EwarmE_{\mathrm{warm}} delays cross-modal interaction and limits the opportunity for MCL to exploit complementary semantic information and correct modality-specific prediction biases.

We adopt three warm-up epochs as a practical setting throughout our experiments, under which DUET achieves consistently strong retrieval performance across the evaluated benchmarks and noise levels. We clarify that this value is an effective empirical setting for our experimental configuration, rather than claiming it to be universally optimal.

The authors answered my questions, and I maintain my original acceptance.

AuthorsOpenReview ↗

Thanks to Reviewer Gykb

Thank you very much for maintaining your positive recommendation. We appreciate your careful evaluation of our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.