acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Relation Weighting and View Curriculum for Supervised Multi-Label Cross-Modal Hashing

Abstract

Cross-modal hashing maps heterogeneous data into compact binary codes for efficient retrieval. Existing methods mainly assume fixed image/text input configurations, while practical systems may need to jointly handle paired, image-only, and text-only inputs. This unified setting introduces heterogeneous and dynamically changing optimization difficulty across the nine directed relations among these input conditions. To address this issue, we propose a unified supervised multi-label cross-modal hashing framework that couples adaptive directed relation weighting with an adaptive view curriculum. Relation weights are estimated from raw contrastive statistics to calibrate multi-label contrastive learning, while the learned relation states further guide the sampling of paired, image-only, and text-only views for subsequent training. Experiments on COCO, NUS-WIDE, and MIRFlickr25K demonstrate the effectiveness of the proposed framework across multiple hash-code lengths and eight directed retrieval protocols.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Supervised Cross-Modal Hashing, Multi-Label Learning, Adaptive Relation Weighting, Curriculum Learning, Contrastive Learning, Vision-Language Retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9JXiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies supervised multi-label cross-modal hashing that simultaneously supports pairwise image-text, image-only, and text-only inputs. The authors dynamically adjust the relationship weights based on the historical losses of nine directed modal relationships, and further adjust the sampling probabilities of the three types of input views.

Critical strengths

The paper focuses on the training imbalance problem of different modal relationships, and proposes a solution that links relationship weights with view sampling. The experiments cover three datasets and various hash lengths, and basic component ablation is also provided.

Critical weaknesses

W1. The paper simulates missing modalities by masking complete multimodal data; however, this approach fails to account for real-world scenarios involving incomplete pairings or non-random missingness. It is recommended to include experiments using datasets with varying missingness rates, non-random missing patterns, or naturally occurring incomplete multimodal data.

W2. The claim that "relationships with higher loss should be assigned greater weight and sampling priority" lacks sufficient justification, as high loss may also stem from label noise or inherently unlearnable relationships. It is suggested to compare this approach against alternatives such as "easy-first" sampling, inverse-difficulty weighting, and fixed-curriculum strategies.

W3. Although the paper highlights nine types of relationships, it only evaluates eight protocols; no explanation is provided for the omission of the T2T (text-to-text) protocol.

W4. The paired image-text features are combined using element-wise multiplication; however, CLIP does not guarantee that corresponding dimensions possess independent semantic meanings. It is recommended to conduct ablation studies on alternative fusion methods, such as concatenation followed by projection, gated fusion, or cross-attention.

W5. WuvW_{uv} effectively applies a relation-dependent scaling to the similarity scores, rather than acting as a direct weight for the relation loss; consequently, the behavior of its gradient may shift depending on the sign of the similarity score. It is suggested to compare this approach against standard relation-loss weighting and to analyze the resulting gradient behavior.

W6. There is a lack of genuine cross-modal baselines. The ablation results are unstable; for instance, in the 32-bit setting on the MIRFlickr25K dataset, removing the "view curriculum" component actually leads to an improvement from 84.38 to 84.73.

W7. The paper also omits key information, such as data partitioning, the architecture of "HashCoder", the update frequency of the controller, and the criteria for model selection, which necessitates the inclusion of comprehensive details to ensure reproducibility.

Questions for authors

Please refer to Weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 9JXi

Response to W1: We thank the reviewer for highlighting the gap between our modality construction and realistic incomplete multimodal data.

Our P/I/T construction is not intended to simulate real-world missingness.
The goal of this work is to uncover cross-modal relationships under different modality-availability conditions and learn a unified retrieval framework. P/I/T therefore serves as modality-conditioned training views that expose the model to diverse modality combinations and encourage shared cross-modal representation learning.

Persistent missingness does not outperform our adaptive P/I/T training.
As an additional robustness check, we assign each training sample once to P, I, or T and keep the assignment fixed throughout training. Due to the limited rebuttal period, we evaluate this setting on COCO with 32-bit codes, as shown below.

SettingP/I/T ratioAvg. AsymHamming mAP@5000
Persistent missingness (20%)80/10/1091.07
Persistent missingness (40%)60/20/2090.53
Persistent missingness (60%)40/30/3088.92
OursAdaptive P/I/T91.26

The results show that performance degrades as persistent missingness increases, while our adaptive P/I/T training achieves the best retrieval accuracy. This supports the effectiveness of adaptive modality-conditioned training for unified retrieval.

Future work.
We leave realistic missingness for future study and will investigate multi-level cross-view relation modeling under incomplete observations. The model can learn shared and view-specific relationships from available modalities and adapt retrieval to the observed modality pattern, enabling more robust recovery of implicit cross-modal relationships.

Response to W2: We thank the reviewer for this insightful comment.

  • Relation loss serves as a heuristic signal for weight allocation. We use relation-specific losses as indicators of relative learning difficulty, rather than assuming that every high-loss relation is inherently learnable. To avoid overemphasizing transient high-loss observations, which may arise from label noise, we employ EMA smoothing and constrain relation weights within predefined bounds.
  • We compare our approach with alternative weighting and curriculum strategies. Given the limited rebuttal period, we evaluate inverse-difficulty weighting, easy-first sampling, and fixed curriculum on COCO with 32-bit codes. The results are reported below:
StrategyAvg. AsymHamming mAP@5000
Fixed curriculum (w/o)89.57
Inverse-difficulty weighting89.33
Easy-first sampling90.83
Adaptive strategy (Ours)91.26
  • Our adaptive strategy consistently outperforms all three alternatives. These results support the effectiveness of emphasizing relatively difficult relations and dynamically adapting the view sampling strategy over inverse-difficulty weighting and fixed or easy-first curricula.

Response to W3: We thank the reviewer for pointing this out. To provide a complete evaluation of all nine directed retrieval protocols, we have added the T2T results under the same evaluation settings. The results are reported below:

Dataset16-bit32-bit64-bit
COCO88.6790.7191.24
NUS-WIDE76.8578.2677.76
MIRFlickr25K76.1076.7476.45

Response to W4: We thank the reviewer for this valuable suggestion.

  • Feature fusion is not a central focus of this work. We adopt element-wise multiplication as a lightweight, parameter-free fusion strategy. It preserves the original feature dimension, allowing paired and unimodal representations to share the same HashCoder.
  • Our framework is robust to different fusion choices. To verify this, we compare element-wise multiplication with two alternative fusion strategies on COCO using 32-bit codes. We do not include cross-attention, as it requires token-level features and additional interaction modules, which are less suited to our lightweight architecture.
Fusion StrategyAvg. AsymHamming mAP@5000
Concat + Projection90.67
Gated Fusion91.41
Element-wise Multiplication (Ours)91.26

The comparable performance across all three variants demonstrates that our framework is not tied to a particular fusion operator. Element-wise multiplication offers a favorable balance between retrieval performance and architectural simplicity.

Response to W5: We thank the reviewer for this insightful comment.

  • Both formulations follow the same adaptive relation-weighting principle. They use the same estimated relation weight WuvW_{uv} to adjust the contribution of each relation, yet apply it at different stages: our method scales similarity scores before the softmax, whereas standard relation-loss weighting scales the resulting loss. Since Wuv>0W_{uv}>0, it does not directly reverse the gradient direction according to the sign of the similarity score. Instead, it changes the gradient magnitude and the softmax distribution.
  • The two formulations achieve comparable performance. We additionally evaluate standard relation-loss weighting while keeping the remaining training settings unchanged. Given the limited rebuttal period, we report this comparison on COCO with 32-bit codes.
Weighting StrategyAvg. AsymHamming mAP@5000
Relation-Loss Weighting91.59
Similarity Scaling (Ours)91.26

The results demonstrate that both weighting formulations effectively exploit the adaptive relation weights and achieve comparable retrieval performance.

Response to W6: We thank the reviewer for these valuable comments.

  • We have included additional cross-modal hashing baselines. Following the reviewer's suggestion, we evaluate DSPH, DDBH, and PromptHashing using the same data splits and evaluation protocol as our method. Our method achieves the best performance across all six COCO I2T/T2I settings. The results are reported below: | Method | I2T 16-bit | I2T 32-bit | I2T 64-bit | T2I 16-bit | T2I 32-bit | T2I 64-bit | | ------------------- | ---------: | ---------: | ---------: | ---------: | ---------: | ---------: | | DSPH (TCSVT'24) | 87.31 | 89.74 | 90.81 | 87.90 | 90.29 | 91.06 | | DDBH (TCSVT'25) | 80.32 | 85.84 | 87.85 | 79.55 | 85.08 | 87.06 | | PromptHash(CVPR'25) | 88.50 | 89.73 | 90.65 | 88.13 | 89.55 | 90.45 | | Ours | 88.62 | 90.80 | 91.64 | 88.28 | 90.79 | 91.77 |

  • The ablation results show that View Curriculum is beneficial in most settings. The full model outperforms its counterpart without View Curriculum in 7 of the 9 dataset–bit-length combinations, with consistent gains on COCO and NUS-WIDE. We acknowledge a decrease on MIRFlickr25K at 32 bits (84.38 vs. 84.73), which may be attributable to variations in relation difficulty and view-sampling dynamics across datasets. Overall, View Curriculum provides complementary benefits, although its impact varies across datasets and hash lengths.

Response to W7: We thank the reviewer for pointing out these missing details. We will address them in the revised manuscript to improve the reproducibility of our work.

  • We will provide the missing experimental and implementation details. The fixed train/database/query splits are 10,000/117,218/5,00010{,}000/117{,}218/5{,}000 for COCO, 4,000/20,000/1,0004{,}000/20{,}000/1{,}000 for MIRFlickr25K, and 10,500/193,734/2,10010{,}500/193{,}734/2{,}100 for NUS-WIDE. The shared HashCoder consists of a lightweight two-layer MLP with ReLU activations and Batch Normalization. The controller is updated every 12 training iterations. All models are trained for 12 epochs, with the final-epoch checkpoint used for evaluation, rather than selecting checkpoints based on validation or test retrieval performance.
  • We will release the code and experimental configurations. The source code, training and evaluation configurations, and dataset split files will be made publicly available to facilitate reproducibility and future research.
Reviewer_YHwZOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies supervised multi-label cross-modal hashing with paired, image-only, and text-only inputs in a unified retrieval setting. Joint learning of these inputs gives rise to nine directed relations. Their learning difficulties can differ and change during training. To address these challenges, the paper introduces adaptive directed relation weighting and an adaptive view curriculum. Experiments on COCO, NUS-WIDE, and MIRFlickr25K cover multiple hash-code lengths and eight directed retrieval protocols. Ablation and sensitivity analyses further validate the proposed adaptive mechanisms.

Critical strengths

1.The paper studies a practical setting that goes beyond the usual assumption of fixed modality configurations. 2.The proposed weighting and curriculum mechanisms fit the stated motivation well and form a coherent adaptive training strategy. 3.The experimental evaluation is broad, covering multiple datasets, code lengths, retrieval directions, and targeted ablations.

Critical weaknesses

1.Since the method is motivated by changing relation difficulty during training, it would be useful to show how the learned relation weights and P/I/T sampling probabilities evolve over time. 2.The P/I/T states are constructed from complete image-text records. The paper could briefly clarify how this construction relates to the practical cases described in the introduction, where both modalities, only an image, or only text may be available.

Questions for authors

1.What motivates the initial sampling probabilities (πP,πI,πT)=(0.8,0.1,0.1) and the constraint πP ∈ [0.4,0.8]? 2.Why are raw similarities used to estimate relation difficulty, while weighted similarities are used for network optimization?

1 comment
AuthorsOpenReview ↗

Response to Reviewer YHwZ

Response to W1:

  • The learned relation weights and P/I/T sampling probabilities evolve adaptively during training.
    To make the training dynamics explicit, we summarize representative controller states from the COCO 32-bit run:
Controller stateπP\pi_PπI\pi_IπT\pi_TRange of WuvW_{uv}
Initialization0.8000.1000.1001.000--1.000
Early stage0.8000.1000.1000.848--1.135
Intermediate stage0.6140.1930.1930.987--1.008
Final stage0.5160.2430.2410.994--1.012

The sampling distribution gradually shifts from the initially pair-dominated state toward increased image-only and text-only sampling. Meanwhile, the directed relation weights show clear differentiation in the early stage and gradually stabilize as training proceeds. These dynamics demonstrate that both relation weighting and view sampling are adaptively adjusted rather than remaining fixed. We will include the full training trajectories in the revised manuscript.

Response to W2:

  • Our P/I/T construction is not intended to simulate real-world missing modalities.
    Our training data consist of complete image--text records. P/I/T are modality-conditioned training views introduced to jointly learn a shared hash space and improve retrieval under different modality-availability conditions. Therefore, our goal is not to model naturally incomplete or non-randomly missing multimodal data.
  • Persistent missingness does not provide an advantage over our adaptive P/I/T training.
    As an additional robustness check, we assign each training sample once to P, I, or T and keep this assignment fixed throughout training. Given the limited rebuttal period, we evaluate persistent missingness on COCO with 32-bit codes, as shown below.
SettingP/I/T ratioAvg. AsymHamming mAP@5000
Persistent missingness (20%)80/10/1091.07
Persistent missingness (40%)60/20/2090.53
Persistent missingness (60%)40/30/3088.92
OursAdaptive P/I/T91.26

These results show that persistent missingness does not outperform our adaptive P/I/T training, while performance decreases as the missing rate increases. This supports our use of P/I/T as adaptive training views rather than as a simulation of a real-world missing-data process. Naturally incomplete and non-random missingness are beyond the scope of this work.

Response to Q1:

  • The initialization and constraint provide a stable starting point while preserving adaptation.
    The initialization (πP,πI,πT)=(0.8,0.1,0.1)(\pi_P,\pi_I,\pi_T)=(0.8,0.1,0.1) gives higher priority to the more informative paired view at the beginning of training, while non-zero unimodal probabilities ensure that I/T relations are observed from the start. The constraint πP∈[0.4,0.8]\pi_P\in[0.4,0.8] prevents the curriculum from collapsing to either an almost fully paired or excessively unimodal regime, while still allowing substantial adaptation during training.

Response to Q2:

  • Raw similarities are used for relation-state estimation to avoid self-referential feedback.
    Raw similarities provide a relation-difficulty signal before the current weighting intervention, whereas weighted similarities are used for network optimization. If weighted similarities were also used for difficulty estimation, the current WuvW_{uv} would directly affect the statistics used to determine the next WuvW_{uv}, creating a self-referential feedback loop. Separating raw statistics for estimation from weighted similarities for optimization therefore provides a cleaner and more stable controller signal.
Reviewer_WVMnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified supervised multi-label cross-modal hashing framework that couples adaptive directed relation weighting with an adaptive view curriculum. The framework uses a single hashing model for paired image-text, image-only, and text-only inputs. It adjusts relation weights based on contrastive losses and dynamically changes the proportions of different views sampled during training. The authors report results for eight directed retrieval protocols on COCO, NUS-WIDE, and MIRFlickr25K, along with ablation experiments.

Critical strengths
  1. A single model handles paired image-text, image-only, and text-only inputs, supporting several retrieval settings.

  2. In Table 1, the proposed method achieves the highest reported I2I scores across the three datasets and three code lengths.

  3. The paper is clearly motivated by the need to handle heterogeneous input conditions and dynamically changing learning difficulties in cross-modal hashing.

Critical weaknesses
  1. The experimental comparisons are limited to image-to-image (I2I) retrieval. Although the paper reports results for eight directed retrieval protocols, comparisons with existing methods on image-to-text (I2T) and text-to-image (T2I) retrieval are missing, making it difficult to assess the proposed method's competitiveness on conventional cross-modal retrieval tasks.

  2. The relation-weight estimator used for NUS-WIDE is undefined. Lines 369–371 on page 7 state that this dataset uses a “progress-aware variant,” but the manuscript provides no formula or update procedure for it.

  3. The ablation studies do not isolate the contribution of directional relation weighting. Although the method assigns separate weights to opposite directions, no comparison with a symmetric weighting strategy is provided, leaving the benefit of explicitly modeling directionality insufficiently validated.

  4. The presentation of the paper could be further improved. The authors should refine the writing, improve the visual quality and readability of the figures and tables, particularly Figure 1, and ensure consistent formatting of the references.

Questions for authors
  1. Could the authors provide additional I2T and T2I comparisons with representative cross-modal hashing methods under consistent experimental settings?

  2. Please provide the formula, update procedure, and parameters of the “progress-aware variant” used for NUS-WIDE, and explain how it differs from Equations (10)–(12). What results do you obtain on NUS-WIDE with the standard estimator defined in the paper?

  3. Could the authors compare the proposed directed relation weighting with a symmetric variant that assigns the same weight to opposite directions, while keeping the view sampling schedule unchanged?

1 comment
AuthorsOpenReview ↗

Response to Reviewer WVMn

Response to Q1: We thank the reviewer for this valuable suggestion. We agree that comparisons with representative cross-modal hashing methods under I2T and T2I protocols provide a valuable complement to the evaluation of our unified P/I/T framework. Accordingly, we have included three representative methods, DSPH (TCSVT'24), DDBH (TCSVT'25) and PromptHashing (CVPR'2025), using their official implementations. Due to the limited rebuttal time, we conduct this comparison on COCO only. All methods are evaluated on COCO under the same data splits, query/database partitions, hash-code lengths, and evaluation metric (Asymmetric Hamming mAP@5000). As shown in Table 1, our method achieves superior performance over three baselines across I2T and T2I retrieval, further demonstrating its effectiveness in cross-modal hash learning.

MethodI2T 16-bitI2T 32-bitI2T 64-bitT2I 16-bitT2I 32-bitT2I 64-bit
DSPH (TCSVT'24)87.3189.7490.8187.9090.2991.06
DDBH (TCSVT'25)80.3285.8487.8579.5585.0887.06
PromptHash(CVPR'2025)88.5089.7390.6588.1389.5590.45
Ours88.6290.8091.6488.2890.7991.77

Response to Q2: We thank the reviewer for raising this point. The progress-aware variant is a dataset-specific stabilization of relation-weight estimation. Compared with COCO and MIRFlickr25K, NUS-WIDE exhibits less stable relation statistics. Therefore, we introduce three modifications to make relation-weight updates more robust, while keeping the overall framework and training objective unchanged.

Specifically, the update procedure consists of three steps:

  1. Reliability-aware state update. We estimate relation reliability from the number of valid positive terms: ruv(t)=nuv(t)nuv(t)+κ.r_{uv}^{(t)}=\frac{n_{uv}^{(t)}}{n_{uv}^{(t)}+\kappa}. The relation state is updated as
=E_{uv}^{(t)}+(1-\mu_W)r_{uv}^{(t)} (c_{uv}^{(t)}-E_{uv}^{(t)}).$$ 2. **Progress-aware gating.** We maintain fast and slow EMAs of relation losses with decay factors $\mu_F$ and $\mu_L$. Their relative gap defines $$p_{uv}^{(t+1)}=\max\left( \frac{L_{uv}^{(t+1)}-F_{uv}^{(t+1)}} {|L_{uv}^{(t+1)}|+\epsilon},0 \right).$$ The corresponding gate is $$g_{uv}^{(t+1)} =g_{\min}+(1-g_{\min}) \frac{p_{uv}^{(t+1)}}{p_{uv}^{(t+1)}+\tau_p}.$$ 3. **Robust difficulty estimation.** Let $M_E$ denote the median of $E_{uv}$ and $R_E$ its median-based robust scale. We compute $$d_{uv}^{(t+1)} =\tanh\left( \frac{E_{uv}^{(t+1)}-M_E}{T_dR_E} \right).$$ The final relation weight is $$W_{uv}^{(t+1)} =1+\beta r_{uv}^{(t)} g_{uv}^{(t+1)}d_{uv}^{(t+1)}.$$ **The corresponding parameters are set to** $\kappa=[\,256]$, $\mu_W=[\,0.8]$, $\mu_F=[\,0.5]$, $\mu_L=[\,0.9]$, $g_{\min}=[\,0.15]$, $\tau_p=[\,0.02]$, $\alpha=[\,0.01]$, $T_d=[\,1.0]$, and $\beta=[\,1.0]$. **We also evaluated the standard estimator on NUS-WIDE.** As shown in Table 2, under the 32-bit setting, the standard estimator achieves comparable performance, with only a marginal decrease relative to the progress-aware variant. This suggests that the variant provides additional stabilization for NUS-WIDE, while the standard estimator remains effective. | Estimator | Avg. AsymHamming mAP@5000 | |---|---:| | Standard estimator | 83.85 | | Progress-aware estimator | 84.30 | **Response to Q3:** We thank the reviewer for this insightful suggestion. - **The directed estimator preserves flexibility across opposite directions.** Each $W_{uv}$ is independently estimated from the relation-specific mean positive contrastive loss $\ell_{uv}$, computed using raw similarities before relation weighting. Thus, opposite directions can receive similar weights when their learning difficulties are comparable, while retaining the flexibility to capture asymmetric learning dynamics. - **Explicit symmetry provides no consistent performance gain.** We construct the symmetric variant by averaging the weights of opposite directions: $$W_{uv}^{\mathrm{sym}} = W_{vu}^{\mathrm{sym}} = \frac{W_{uv}+W_{vu}}{2}.$$ To isolate the effect of directed relation weighting, we keep the view sampling schedule and all other training settings unchanged. The results under 32-bit codes (average AsymHamming mAP@5000 across eight retrieval protocols) are as follows: | Weighting Scheme | COCO | NUS-WIDE | MIRFlickr25K | |---|---:|---:|---:| | Symmetric | **91.43** | **84.33** | 84.03 | | Directed (Ours) | 91.26 | 84.30 | **84.38** | These results show that **symmetric weighting offers no consistent advantage over directed weighting**. Our directed formulation retains the flexibility to model potentially asymmetric relation difficulties without imposing an explicit symmetry constraint.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.