- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies supervised multi-label cross-modal hashing that simultaneously supports pairwise image-text, image-only, and text-only inputs. The authors dynamically adjust the relationship weights based on the historical losses of nine directed modal relationships, and further adjust the sampling probabilities of the three types of input views.
Critical strengths
The paper focuses on the training imbalance problem of different modal relationships, and proposes a solution that links relationship weights with view sampling. The experiments cover three datasets and various hash lengths, and basic component ablation is also provided.
Critical weaknesses
W1. The paper simulates missing modalities by masking complete multimodal data; however, this approach fails to account for real-world scenarios involving incomplete pairings or non-random missingness. It is recommended to include experiments using datasets with varying missingness rates, non-random missing patterns, or naturally occurring incomplete multimodal data.
W2. The claim that "relationships with higher loss should be assigned greater weight and sampling priority" lacks sufficient justification, as high loss may also stem from label noise or inherently unlearnable relationships. It is suggested to compare this approach against alternatives such as "easy-first" sampling, inverse-difficulty weighting, and fixed-curriculum strategies.
W3. Although the paper highlights nine types of relationships, it only evaluates eight protocols; no explanation is provided for the omission of the T2T (text-to-text) protocol.
W4. The paired image-text features are combined using element-wise multiplication; however, CLIP does not guarantee that corresponding dimensions possess independent semantic meanings. It is recommended to conduct ablation studies on alternative fusion methods, such as concatenation followed by projection, gated fusion, or cross-attention.
W5. effectively applies a relation-dependent scaling to the similarity scores, rather than acting as a direct weight for the relation loss; consequently, the behavior of its gradient may shift depending on the sign of the similarity score. It is suggested to compare this approach against standard relation-loss weighting and to analyze the resulting gradient behavior.
W6. There is a lack of genuine cross-modal baselines. The ablation results are unstable; for instance, in the 32-bit setting on the MIRFlickr25K dataset, removing the "view curriculum" component actually leads to an improvement from 84.38 to 84.73.
W7. The paper also omits key information, such as data partitioning, the architecture of "HashCoder", the update frequency of the controller, and the criteria for model selection, which necessitates the inclusion of comprehensive details to ensure reproducibility.
Questions for authors
Please refer to Weakness.
2 comments
Response to W4: We thank the reviewer for this valuable suggestion.
- Feature fusion is not a central focus of this work. We adopt element-wise multiplication as a lightweight, parameter-free fusion strategy. It preserves the original feature dimension, allowing paired and unimodal representations to share the same HashCoder.
- Our framework is robust to different fusion choices. To verify this, we compare element-wise multiplication with two alternative fusion strategies on COCO using 32-bit codes. We do not include cross-attention, as it requires token-level features and additional interaction modules, which are less suited to our lightweight architecture.
| Fusion Strategy | Avg. AsymHamming mAP@5000 |
|---|---|
| Concat + Projection | 90.67 |
| Gated Fusion | 91.41 |
| Element-wise Multiplication (Ours) | 91.26 |
The comparable performance across all three variants demonstrates that our framework is not tied to a particular fusion operator. Element-wise multiplication offers a favorable balance between retrieval performance and architectural simplicity.
Response to W5: We thank the reviewer for this insightful comment.
- Both formulations follow the same adaptive relation-weighting principle. They use the same estimated relation weight to adjust the contribution of each relation, yet apply it at different stages: our method scales similarity scores before the softmax, whereas standard relation-loss weighting scales the resulting loss. Since , it does not directly reverse the gradient direction according to the sign of the similarity score. Instead, it changes the gradient magnitude and the softmax distribution.
- The two formulations achieve comparable performance. We additionally evaluate standard relation-loss weighting while keeping the remaining training settings unchanged. Given the limited rebuttal period, we report this comparison on COCO with 32-bit codes.
| Weighting Strategy | Avg. AsymHamming mAP@5000 |
|---|---|
| Relation-Loss Weighting | 91.59 |
| Similarity Scaling (Ours) | 91.26 |
The results demonstrate that both weighting formulations effectively exploit the adaptive relation weights and achieve comparable retrieval performance.
Response to W6: We thank the reviewer for these valuable comments.
-
We have included additional cross-modal hashing baselines. Following the reviewer's suggestion, we evaluate DSPH, DDBH, and PromptHashing using the same data splits and evaluation protocol as our method. Our method achieves the best performance across all six COCO I2T/T2I settings. The results are reported below: | Method | I2T 16-bit | I2T 32-bit | I2T 64-bit | T2I 16-bit | T2I 32-bit | T2I 64-bit | | ------------------- | ---------: | ---------: | ---------: | ---------: | ---------: | ---------: | | DSPH (TCSVT'24) | 87.31 | 89.74 | 90.81 | 87.90 | 90.29 | 91.06 | | DDBH (TCSVT'25) | 80.32 | 85.84 | 87.85 | 79.55 | 85.08 | 87.06 | | PromptHash(CVPR'25) | 88.50 | 89.73 | 90.65 | 88.13 | 89.55 | 90.45 | | Ours | 88.62 | 90.80 | 91.64 | 88.28 | 90.79 | 91.77 |
-
The ablation results show that View Curriculum is beneficial in most settings. The full model outperforms its counterpart without View Curriculum in 7 of the 9 dataset–bit-length combinations, with consistent gains on COCO and NUS-WIDE. We acknowledge a decrease on MIRFlickr25K at 32 bits (84.38 vs. 84.73), which may be attributable to variations in relation difficulty and view-sampling dynamics across datasets. Overall, View Curriculum provides complementary benefits, although its impact varies across datasets and hash lengths.
Response to W7: We thank the reviewer for pointing out these missing details. We will address them in the revised manuscript to improve the reproducibility of our work.
- We will provide the missing experimental and implementation details. The fixed train/database/query splits are for COCO, for MIRFlickr25K, and for NUS-WIDE. The shared HashCoder consists of a lightweight two-layer MLP with ReLU activations and Batch Normalization. The controller is updated every 12 training iterations. All models are trained for 12 epochs, with the final-epoch checkpoint used for evaluation, rather than selecting checkpoints based on validation or test retrieval performance.
- We will release the code and experimental configurations. The source code, training and evaluation configurations, and dataset split files will be made publicly available to facilitate reproducibility and future research.
Response to Reviewer 9JXi
Response to W1: We thank the reviewer for highlighting the gap between our modality construction and realistic incomplete multimodal data.
Our P/I/T construction is not intended to simulate real-world missingness.
The goal of this work is to uncover cross-modal relationships under different modality-availability conditions and learn a unified retrieval framework. P/I/T therefore serves as modality-conditioned training views that expose the model to diverse modality combinations and encourage shared cross-modal representation learning.
Persistent missingness does not outperform our adaptive P/I/T training.
As an additional robustness check, we assign each training sample once to P, I, or T and keep the assignment fixed throughout training. Due to the limited rebuttal period, we evaluate this setting on COCO with 32-bit codes, as shown below.
The results show that performance degrades as persistent missingness increases, while our adaptive P/I/T training achieves the best retrieval accuracy. This supports the effectiveness of adaptive modality-conditioned training for unified retrieval.
Future work.
We leave realistic missingness for future study and will investigate multi-level cross-view relation modeling under incomplete observations. The model can learn shared and view-specific relationships from available modalities and adapt retrieval to the observed modality pattern, enabling more robust recovery of implicit cross-modal relationships.
Response to W2: We thank the reviewer for this insightful comment.
Response to W3: We thank the reviewer for pointing this out. To provide a complete evaluation of all nine directed retrieval protocols, we have added the T2T results under the same evaluation settings. The results are reported below: