- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DUET for cross-modal hashing under noisy labels. Temporal Consensus Learning maintains historical predictions and weights modalities according to their temporal stability. Modality Consensus Learning constructs an entropy-weighted consensus and applies stronger consistency constraints to less confident modalities. Both components operate through shared learnable class prototypes in the continuous hash space. The evaluation covers three image–text datasets, multiple noise rates and hash lengths, with additional backbone and three-modal experiments. The reported retrieval performance is strong, although the relationship between the specified training procedure and noisy-label supervision requires clarification.
Critical strengths
The paper addresses a relevant limitation of cross-modal agreement: different modalities can provide evidence of different reliability for the same example. Combining historical stability with current prediction uncertainty offers a useful approach to making consensus adaptive rather than treating all modalities equally.
The experimental coverage is substantial. DUET achieves the highest reported average bidirectional mAP in 46 of the 48 main dataset–noise–hash-length settings. The appendix reports means and standard deviations over three independent runs and includes backbone comparisons and three-modal retrieval results. These results support the potential practical value of the method.
Critical weaknesses
The complete supervision pathway is missing from the specified method. Equation (8) initializes temporal targets from model predictions, Equation (19) contains only TCL and MCL, and Algorithm 1 randomly initializes the hashing networks and prototypes. None of these steps explicitly uses the observed noisy labels. With fixed features, initialization, and sampling, changing only the labels would therefore leave the described optimization unchanged. This makes it difficult to interpret the reported robustness as an effect of learning under corrupted supervision. The objective also permits constant, concentrated predictions with zero MCL and small TCL loss, without useful retrieval semantics. An omitted supervised component could resolve these concerns, but it must be stated explicitly.
Questions for authors
Where exactly do the noisy labels enter training? Please provide the complete implemented objective and any label-dependent initialization, pretraining, or sampling steps omitted from Equation (19) and Algorithm 1. A small check holding features, initialization, and sampling fixed while changing only the labels would help establish this dependence. Please also identify the component that anchors the class prototypes and discourages constant semantic predictions. If labels are intentionally excluded, please clarify the interpretation of the noise-rate experiments.
Are α in Equation (14) and the coefficient (1-\beta) in Equation (18) detached during optimization? Detaching the consensus target alone does not specify the gradient treatment of these coefficients. Please state the implemented policy and, on one existing configuration such as XMedia with 64-bit codes and 0.8 noise, compare it with detached loss weights. This would clarify whether the gains come from adaptive weighting or additionally from gradients through the weighting functions.
4 comments
Response to Reviewer KtsJ (2/2)
(2) Class-prototype anchoring and prevention of trivial predictions
We acknowledge that the reviewer raises an important concern about whether consistency-based objectives alone can guarantee meaningful semantic predictions. Indeed, TCL and MCL primarily impose temporal stability and cross-modal consistency, respectively. If considered without the label-dependent components, these objectives alone would not provide sufficient class-specific supervision and could admit uninformative solutions.
In DUET, semantic anchoring comes from two label-dependent operations. First, the class prototypes are initialized using statistical information extracted from the observed noisy labels, providing an initial correspondence between the prototypes and the annotated semantic classes. Second, the noisy labels are explicitly incorporated into the temporal targets used by TCL, allowing annotation-related information to influence the learning process throughout training. Consequently, the temporal learning objective is influenced by class-dependent annotation information, rather than merely encouraging predictions to remain unchanged over time.
MCL further exploits the resulting modality-specific predictions to establish cross-modal semantic consensus. Therefore, these components play distinct but complementary roles: label-dependent prototype initialization establishes the initial class correspondence, label-informed temporal targets provide annotation-related guidance during optimization, and MCL coordinates predictions across modalities.
We emphasize that TCL and MCL alone are not intended to guarantee non-degenerate semantic predictions. Their effectiveness relies on the label-dependent information introduced through prototype initialization and temporal target construction.
(3) Gradient treatment of and
We thank the reviewer for pointing out that detaching the consensus target alone does not fully specify the gradient treatment of the adaptive weighting coefficients. In our implementation, both in Eq. (14) and in Eq. (18) are detached from the computational graph during backpropagation. They are used as adaptive weighting statistics calculated from the forward-pass predictions, rather than as differentiable optimization variables. Therefore, gradients are not propagated through these coefficients, while the corresponding prediction and representation pathways remain differentiable. The consensus target is also detached as described in our original formulation.
To examine the effect of this design choice, we conducted an additional comparison on XMedia using 64-bit hash codes under a label noise rate of 0.8. Specifically, we compared the original implementation, in which both adaptive weighting coefficients are detached, with a variant that allows gradients to propagate through and . All other experimental settings and training procedures were kept identical. The results are reported below.
| Setting | mAP |
|---|---|
| Ours (detached and ) | 72.6 |
| Variant (gradients through and ) | 64.1 |
The results show that allowing gradients to propagate through the adaptive weighting coefficients leads to a substantial performance drop. This suggests that treating these coefficients as differentiable optimization variables can introduce additional gradient pathways through which the model may alter the estimated reliability weights to reduce the objective, rather than improve the underlying semantic predictions. By detaching the coefficients, their role is restricted to adaptively regulating the relative contributions of different samples and modalities, while the network parameters are optimized through the corresponding weighted objectives. This comparison supports our design choice and demonstrates the benefit of using detached adaptive weighting coefficients, rather than allowing gradients to propagate through the weighting functions.
I have raised my score
my questions solved
Thanks to Reviewer KtsJ
We are glad that our revisions have addressed all your questions. Should you have any further concerns, we are happy to provide additional clarifications.
Response to Reviewer KtsJ (1/2)
We appreciate the reviewer for raising these critical questions regarding the supervision pathway, the role of noisy labels, and the gradient treatment of adaptive weighting coefficients. We clarify these aspects and address each concern below.
(1) Explicit supervision pathway and complete training objective
We agree that the supervision pathway should be explicitly stated. In our implementation, the observed noisy labels y~i are directly incorporated into the temporal target update in Eq. (9), which was inadvertently omitted from the original manuscript. The complete update is given by p~im[t]=ηp~im[t−1]+(1−η)(pim[t]+y~i).
Here, pim[t] denotes the current prediction of modality m, while y~i represents the observed noisy label. Consequently, the temporal target is jointly determined by the model prediction and the provided annotation, rather than being generated solely from model predictions.
In addition, our class prototypes are initialized using statistical information extracted from the training noisy labels, rather than being initialized independently of the annotations. This introduces label-dependent semantic information into the prototype initialization and establishes an initial correspondence between the prototypes and the annotated classes.
Importantly, the noisy labels are incorporated through the label-dependent temporal target construction and prototype initialization; there is no additional cross-entropy loss in our implementation. TCL exploits the label-informed temporal targets to stabilize intra-modal learning, while MCL coordinates modality-specific predictions through reliability-weighted cross-modal consensus.
To further verify the dependence of training on observed noisy labels, we conducted an additional label replacement experiment on XMedia using 16-bit hash codes under a label noise rate of 0.4, following the reviewer's suggestion. Specifically, we fixed the input features, network initialization, prototype initialization, mini-batch sampling order, and all other training configurations. We then replaced the original noisy labels with randomly shuffled labels while preserving the overall label distribution. Only the labels used in the temporal target update were changed. This controlled setting isolates the effect of label information in Eq. (9) from that of label-dependent prototype initialization.
Under the original and replaced labels, the temporal targets are respectively updated as $\tilde{\boldsymbol{p}}_i^{m[t]}=\eta\tilde{\boldsymbol{p}}_i^{m[t-1]}+(1-\eta)\left( \boldsymbol{p}_i^{m[t]}+\tilde{\boldsymbol{y}}_i\right)$, and $\tilde{\boldsymbol{p}}_i'^{m[t]}=\eta\tilde{\boldsymbol{p}}_i'^{m[t-1]}+ (1-\eta)\left(\boldsymbol{p}_i'^{m[t]}+\tilde{\boldsymbol{y}}_i'\right)$.
We compared the resulting temporal targets, TCL gradients, and final retrieval performance. The results are summarized below.
The differences in temporal targets and TCL gradients directly demonstrate that changing the observed labels changes the optimization signals, even when the features, initialization, and sampling order remain fixed. The retrieval results further illustrate the effect of preserving or disrupting the correspondence between training samples and their observed labels. These results provide direct evidence that DUET is trained under the provided noisy label supervision rather than being label-agnostic.