acceptodds
Under review as a conference paper at ICLR 2027

Cross-Domain Cross-Modal Hashing via Semantic Relation Transfer and Inverse-Free Broad Learning

Abstract

Cross-domain cross-modal hashing aims to leverage semantically rich knowledge from an auxiliary domain to improve cross-modal retrieval in an unlabeled or weakly labeled target domain. Existing methods face two major challenges: the lack of reliable semantic supervision in the target domain, and the increased complexity of cross-domain transfer caused by modality discrepancies and disjoint label spaces, which further impedes the exploitation of semantic information. To address these challenges, we propose a hashing framework based on semantic relation transfer and QR-based incremental learning. For cross-domain semantic transfer, we learn transferable cross-modal representations from the semantically rich auxiliary domain and bridge the auxiliary and target domains through parameter transfer and cross-domain semantically aligned samples. Meanwhile, we construct an auxiliary-domain semantic relation graph and a weak semantic relation graph for the target domain using CLIP-generated pseudo-labels. By jointly modeling semantic relations across the two domains, our method provides effective semantic guidance for the supervision-deficient target domain, thereby facilitating knowledge transfer and target-domain adaptation. For representation learning, we introduce an incrementally extensible broad learning-based hashing layer that dynamically expands the representation space by appending feature nodes and enhancement nodes. We further develop an incremental QR decomposition-based update of the R-factor and employ triangular solves to analytically compute the expanded-layer weights, eliminating the need for the explicit pseudoinverse computation required by conventional broad learning methods. By integrating auxiliary-to-target knowledge transfer with incremental target-domain learning in a lightweight hashing framework, together with dual semantic relation graph modeling, the proposed method simultaneously enables cross-domain knowledge transfer and target-domain semantic enhancement. Extensive experiments on two benchmark datasets demonstrate significant improvements over existing methods, validating the effectiveness and superiority of the proposed framework.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Cross-modal Hashing, Cross-domain transfer, Broad Learning System, Weakly-supervised

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pxnPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SRIF-BLS for cross-domain cross-modal hashing, using a labeled auxiliary domain to support retrieval in an unlabeled target domain. The framework combines transferred representations and semantic relations with CLIP-based weak semantic guidance. An expandable broad learning hashing layer is adapted through QR-based ridge refitting followed by gradient-based semantic optimization. The appendix establishes that the expansion update recovers the exact ridge solution for a fixed feature-and-target snapshot. Experiments evaluate both transfer directions between MIRFlickr-25K and NUS-WIDE, three hash lengths, and several component ablations.

Critical strengths

The framework provides a concrete way to combine auxiliary-domain knowledge with target-domain semantic guidance without requiring directly matched class indices. The reported ablations suggest that relation transfer and transferred initialization contribute beyond the target semantic graph, making the integration potentially useful for supervision-scarce retrieval.

The QR update is specified with an appropriately limited theoretical guarantee. The paper explicitly refreshes the factor state when encoder features change and distinguishes exact ridge refitting from optimization of the semantic objective. Experiments also cover both transfer directions, with the full method achieving the highest reported mAP in 10 of the 12 dataset/direction/hash-length settings.

Critical weaknesses

The model-selection protocol raises a substantive evaluation concern. Section 4.1 states that target-domain retrieval mAP is evaluated every epoch and used for model selection, but no independent validation split is described. The released implementation evaluates target_test_dl using ground-truth query and retrieval labels and saves the checkpoint with the highest resulting score. Unless the reported experiments used an additional, undocumented validation split, this introduces test-set feedback into checkpoint selection. It therefore weakens the evidence for generalization under the stated unlabeled-target setting.

The empirical evidence does not sufficiently isolate or quantify the claimed advantages. The “w/o IF-BLS” variant is described only as “conventional optimization,” leaving unclear whether width expansion, initialization, and training budgets are matched. No runtime, peak-memory, or numerical-accuracy comparisons substantiate the efficiency claims, including the cost of rebuilding the old QR state. Baseline feature and supervision comparability is also insufficiently documented. These omissions matter because several gains are small: on MIRFlickr, the improvements over the strongest listed baseline are only 0.10–0.28 percentage points in four settings, while the method underperforms that baseline at 64-bit text-to-image retrieval. Only one random seed is reported.

Questions for authors

Could you provide a compact table identifying each method’s image/text features, pretrained backbone, access to auxiliary and target labels, and use of CLIP-derived semantic information? Please also specify how the candidate vocabulary is constructed and which split supplies its text. If the strongest baselines use different features or semantic resources, a matched comparison with CDTH or VTM-UCH at one representative hash length would help establish whether the improvement comes from the proposed framework.

What exactly changes in “w/o IF-BLS”? Are node widths, expansion schedules, binary targets, ridge coefficients, and subsequent optimization matched? On one existing expansion snapshot, could you compare the proposed update with a direct batch ridge solve, reporting solution agreement or residuals, runtime, and peak memory? Please include snapshot collection and old-factor reconstruction in the timing. This would distinguish the benefit of analytical refitting and added capacity from the benefit of the incremental solver itself.

1 comment
AuthorsOpenReview ↗

Evaluation independence and controlled BLS comparison

We thank the reviewer for examining evaluation independence, semantic resources, and the traditional BLS comparison. The revised manuscript reports a fixed-schedule five-seed evaluation and matched expansion-snapshot solver measurements.

Q1. Model selection and seed stability. Tables 1–2 report the final epoch-40 checkpoint. Intermediate checkpoints were saved on the fixed schedule; the highest mAP displayed in earlier logs monitored retrieval accuracy and did not determine the reported checkpoint. We additionally ran a pre-specified 40-epoch target schedule with expansions at epochs 10 and 20; the epoch-40 checkpoint was fixed before evaluation loaded query and retrieval labels. Across five target-adaptation seeds, two-direction mean mAP is 75.29 ± 0.34% on MIRFlickr-25K and 68.95 ± 0.48% on NUS-WIDE (mean ± sample standard deviation, with a fixed source checkpoint in each transfer direction). On NUS-WIDE, the I→T and T→I means exceed the strongest corresponding baselines by 1.84 and 1.03 percentage points. Appendix A.7 reports these results separately from the single-run main tables.

Q2. Feature and supervision resources. SRIF-BLS uses L2-normalized 512-dimensional CLIP ViT-B/32 image and text embeddings. Source-domain labels supervise auxiliary training; target-domain hash learning uses paired examples without target training labels. For the target weak graph, we directly adopt the fixed candidate-term embedding files flickr_tags_tr.mat and nus_tags_tr.mat from the CDTH authors' public code and take the first TopK candidates. Our target protocol uses 2,000 query pairs and 5,000 training pairs sampled from the retrieval set. This shared candidate resource makes CDTH the closest comparison for source-to-target semantic transfer. VTM-UCH supplies a complementary CLIP-based unsupervised hashing comparison.

MethodInput and semantic resourcesAuxiliary labelsTarget training labels
SRIF-BLS512D CLIP ViT-B/32 inputs; CDTH candidate embeddingsUsedUnused
CDTHPublished CLIP-split inputs; released candidate embeddingsUsedUnused
VTM-UCHCLIP-guided vision–text miningUnusedUnused

Q3. Traditional BLS and controlled solver comparison. We renamed the earlier w/o IF-BLS ablation row Traditional BLS to identify the comparison directly: it uses pseudoinverse-based incremental updating. It retains the broad hashing layer and the scheduled widths 912 → 1176 → 1440, with expansions at epochs 10 and 20. SRIF-BLS instead rebuilds the current old-width R factor, updates it for the appended nodes, and solves triangular systems. The ablation compares their retrieval outcomes; Appendix A.8 directly compares the update algorithms on recorded expansion snapshots.

Across four real 32-bit events, SRIF-BLS and full QR use the same expanded design matrix, binary targets, ridge coefficient ρ=10−4\rho=10^{-4}, and float64 arithmetic. SRIF-BLS includes old-factor reconstruction in its measurement. Its maximum relative output-weight difference from the full QR ridge solution is 1.83 × 10⁻¹⁰; its maximum absolute relative objective difference is 3.99 × 10⁻¹⁶. Cumulative solver times are 981.9 ms for SRIF-BLS, 1187.2 ms for full QR, and 8066.3 ms for traditional pseudoinverse updating. Thus SRIF-BLS is 8.21× faster than the traditional update at the solver stage, with 7.87–9.09% lower peak allocated GPU memory.

Reviewer_QL5TOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses unsupervised cross-domain cross-modal hashing with disjoint label spaces and proposes SRIF-BLS, which transfers auxiliary-domain mappings to initialize the target network, constructs a dual semantic supervision graph from a CLIP-based weak-semantic graph and a consistency-filtered auxiliary-domain relation-transfer graph for unlabeled target data, and employs QR-based R-factor updates with triangular solving to replace pseudo-inverse computation during scheduled width expansion of a tanh-binarized hashing network. The method is evaluated on MIRFlickr and NUS-WIDE using mAP and related retrieval metrics against prior cross-modal hashing baselines, with ablations on the transfer module, weak-semantic graph, and inverse-free block update; however, the claimed efficiency and lightweight advantages are not directly supported by training/inference time, FLOPs, parameter count, or memory measurements, and several gains are marginal without multi-seed variance or significance tests.

Critical strengths

The paper’s cross-domain semantic relation transfer mechanism is a well-motivated and coherent way to provide supervision for an unlabeled target domain. By constructing a dual semantic graph — a CLIP-based weak-semantic graph capturing the target domain’s intrinsic structure and a consistency-filtered relation-transfer graph injecting auxiliary-domain semantic priors — the method avoids the strict requirement of aligned category spaces, which is a common bottleneck in cross-domain cross-modal hashing.

Critical weaknesses
  1. The abstract claims “significant improvements,” yet the empirical comparison does not consistently outperform all competing methods; in some settings the gains are marginal or even reversed. Moreover, the paper highlights “QR-based R-factor updates and triangular solving” as a key contribution and describes them as efficient and lightweight, but there is no direct experimental evidence to substantiate this claim. Please provide concrete efficiency measurements, such as training/inference time, FLOPs, parameter count, and memory consumption, and preferably a direct comparison with the standard pseudo-inverse-based BLS variant.
  2. The image and text features fed into the SRIF-BLS network are only described as “precomputed,” without specifying the feature extractor (backbone architecture, extracted layer, or whether BoW/tag representations are used) or the original feature dimensionality. Please clarify the exact feature extraction pipeline and report the input feature dimensions for both modalities.
Questions for authors
  1. The abstract claims “significant improvements,” yet the empirical comparison does not consistently outperform all competing methods, and in some settings the gains are marginal or even reversed. Given that “QR-based R-factor updates and triangular solving” are presented as a key contribution and described as efficient and lightweight, what concrete efficiency measurements (training/inference time, FLOPs, parameter count, memory consumption) can be provided, and how do they compare with a standard pseudo-inverse-based BLS variant?
  2. The image and text features fed into the SRIF-BLS network are only described as “precomputed.” What is the exact feature extraction pipeline (backbone architecture, extracted layer, or whether BoW/tag representations are used), and what are the original feature dimensionalities for both modalities?
  3. In Figure 2, do the “Compatibility QR-BLS Hash Layer” in the “Auxiliary Transfer Network” and the “Compatibility QR-BLS Hash Layer” in the “Target Cross-modal Hash Learning Network” refer to the same module or to two separate instances, and why do the arrows connecting these components appear misaligned?
1 comment
AuthorsOpenReview ↗

Retrieval performance, measured solver cost, and CLIP feature inputs

We thank the reviewer for requesting quantitative evidence and an exact account of the multimodal inputs. The revised appendix now provides both.

Q1. Retrieval performance and computational evidence. The abstract now describes retrieval improvements in several settings, reflecting the direction-specific results. In the additional five-seed 32-bit study, SRIF-BLS reaches 68.54 ± 0.38% image-to-text and 69.36 ± 0.57% text-to-image mAP on NUS-WIDE. These exceed the strongest corresponding values in Table 1 by 1.84 and 1.03 percentage points, respectively. On MIRFlickr-25K, the corresponding results are 75.41 ± 0.37% and 75.17 ± 0.31%. Appendix A.7 reports these target-adaptation means and sample standard deviations; the main tables retain the single-run results.

Appendix A.8 measures four real expansion events at 32 bits, with 5,000 training pairs per event. SRIF-BLS takes 981.9 ms cumulatively, including old-R reconstruction, compared with 1187.2 ms for direct full QR and 8066.3 ms for traditional BLS pseudoinverse updating. This is an 8.21× faster analytical expansion than the traditional update and a 17.3% reduction against full QR. Peak allocated GPU memory during solving falls by 7.87–9.09% relative to the traditional update. On the same expanded ridge problem in float64, the largest relative output-weight difference from full QR is 1.83 × 10⁻¹⁰.

The final two-branch SRIF-BLS network contains 24,589,760 parameters and uses approximately 49,097,984 dense-matrix FLOPs per paired precomputed-feature input (Appendix A.6). Together, these measurements quantify the model and the analytical update at their stated units.

Q2. Feature extraction. Both inputs are precomputed, L2-normalized 512-dimensional CLIP ViT-B/32 embeddings. Images are processed by the CLIP image encoder; dataset tag text is tokenized and processed by the CLIP text encoder. SRIF-BLS therefore learns hash functions over CLIP embeddings. The weak semantic graph uses fixed candidate-term embeddings from the CDTH public release. Appendix A.6 records the backbone, modalities, dimensions, and normalization.

Q3. Framework structure. Figure 2 depicts two QR-BLS hash-layer instances with distinct roles. The auxiliary network learns source-domain parameters and relations; its parameters initialize the target network. The target hash layer subsequently expands and refits its output projection on the current target-training representation. The connection between the two instances represents this parameter-transfer step.

Reviewer_2SF8OpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SRIF-BLS, a method designed to transfer knowledge from a labeled auxiliary domain to an unlabeled target domain, with the aim of improving cross-domain, cross-modal hash retrieval. The approach integrates parameter transfer, a transfer relation graph constructed from semantically consistent samples, and a target-domain weak semantic graph derived from CLIP-generated pseudo-labels. For representation learning, the model expands the width-learning layer by incorporating additional feature nodes and enhancing existing ones. It then re-estimates the output weights using QR decomposition, block-wise updates, and triangular system solving, followed by gradient-based semantic optimization. The authors evaluate the method on two transfer directions between the MIRFlickr-25K and NUS-WIDE datasets using 16, 32, and 64 bit hash codes. The results demonstrate that the proposed method outperforms the listed baselines in terms of mean Average Precision (mAP) across most settings, and ablation studies confirm the contribution of each individual component. Furthermore, the appendix provides a proof demonstrating that, for a fixed snapshot, the block-wise update approach is equivalent to the optimal solution for an extended ridge regression problem.

Critical strengths
  1. This paper investigates a practically significant problem: improving cross-modal retrieval of unlabeled target domains using supervised information from labeled auxiliary domains. The overall design is coherent, combining transfer relations, weak semantic information of the target domain, and a scalable hash layer. Experimental results in two transfer directions demonstrate the potential for practical applications.

  2. Ridge regression updates under fixed feature snapshots have a clear mathematical definition. The derivation proves that the expanded solution is equivalent to the optimal solution of the corresponding ridge regression problem, while clearly distinguishing this guarantee from improvements in nonlinear semantic objectives or retrieval accuracy. This provides a reliable mathematical foundation for the proposed refitting step.

Critical weaknesses
  1. The independence between model selection and final evaluation has not been confirmed. Section 4.1 states that the retrieval mAP is evaluated in each round of target domain training, and the model is selected using the average mAP of the two retrieval directions, but it does not describe an independent validation set. In the code repository, _evaluate_target_map() uses target_test_dl for evaluation, and the training loop saves the best model based on these scores. Since Tables 1–2 report the performance of the model selected through this process on the same batch of queries, the results may suffer from optimistic bias due to model selection. This is especially true since the paper only reports results from a single random seed, and some performance improvements are minimal.

  2. The new contributions relative to existing methods, particularly the claimed efficiency and numerical stability, have not been fully validated. CDTH already incorporates major parameter transfer and bi-semantic graph mechanisms, and InvF-BLS has proposed methods based on R-factor and triangulation. Therefore, this paper needs stronger evidence to demonstrate the added value brought by combining these two approaches. The paper does not report comparisons of runtime, memory usage, or numerical stability. Since the proposed method reconstructs the matrix factors of the old width before each expansion and continues gradient training after expansion, it cannot be inferred that the whole method has higher computational efficiency based solely on the block update steps.

Questions for authors
  1. Model Selection and Evaluation: Were the models in Tables 1–2 selected using the same query set and relevance labels as the final evaluation models? Section 4.1 does not describe the independent validation set, while _evaluate_target_map() uses target_test_dl. Please clarify the actual model selection process. If the same batch of queries was indeed reused, can you provide results obtained through the independent validation set or a pre-fixed training plan to determine if the performance improvement is always valid?

  2. Contributions Compared to Existing Work: Compared to CDTH's parameter transfer and dual semantic graph mechanism, and InvF-BLS's lack of an explicit inverse solver, what specific methodological contributions does this paper make?

  3. Efficiency and Numerical Stability: What evidence supports the improved efficiency and numerical stability of the full adaptation process? Runtime and memory comparisons should take into account the reconstruction of the old width matrix factor and subsequent gradient training. Under the same extended design matrix, regularization coefficients, and numerical accuracy, how does the proposed block update perform numerically compared to the full scheme?

1 comment
AuthorsOpenReview ↗

Evaluation protocol, methodological contribution, and solver evidence

We appreciate the reviewer's scrutiny of evaluation, methodological contribution, and computational cost. We have added a fixed-schedule five-seed evaluation and measured the analytical update on real target-training snapshots.

Q1. Model selection and evaluation. Tables 1–2 report the final epoch-40 checkpoint. Intermediate checkpoints were saved on the fixed training schedule, and the highest mAP shown in earlier logs served to monitor retrieval accuracy; it did not determine the reported checkpoint. We further verified the result with supplementary 32-bit experiments using a pre-specified 40-epoch schedule and expansions at epochs 10 and 20. For these experiments, the epoch-40 checkpoint was fixed before target query and retrieval labels were read for evaluation. Appendix A.7 reports five target-adaptation seeds: 75.29 ± 0.34% two-direction mean mAP on MIRFlickr-25K and 68.95 ± 0.48% on NUS-WIDE (mean ± sample standard deviation, with a fixed source checkpoint in each transfer direction). On NUS-WIDE, the I→T and T→I means exceed the strongest corresponding baselines by 1.84 and 1.03 percentage points. Thus the original logging convention did not determine the reported checkpoint, and the independent supplementary evaluation confirms the retrieval results without test-mAP-based model selection.

Q2. Methodological contribution. CDTH establishes source-to-target semantic transfer and a CLIP-guided weak target graph. InvF-BLS establishes R-factor updating and triangular solving for broad learning. SRIF-BLS extends these foundations to cross-modal target adaptation in which the encoder changes between expansion stages. At each stage, it appends one 64-node feature window, 75 enhancement nodes linked to that window, and 125 enhancement nodes linked to all available windows. Existing blocks retain their original input mappings, so their design columns are preserved on the current encoder snapshot. SRIF-BLS rebuilds the old-width R factor from that snapshot, updates the factor for the appended columns, and analytically refits the output weights. The expanded hashing network then continues semantic training. This synchronized sequence connects relation-guided target learning with width expansion on an evolving representation.

Q3. Efficiency and numerical accuracy. Appendix A.8 evaluates four real 32-bit expansion events: epochs 10 and 20 in both transfer directions, with 5,000 training pairs per event. Each event combines the image and text solver times, measured three times per branch. Cumulative time is 981.9 ms for SRIF-BLS, 1187.2 ms for direct full QR, and 8066.3 ms for traditional BLS pseudoinverse updating. The SRIF-BLS measurement includes reconstruction of the old R factor. It achieves an 8.21× speedup over the traditional update and a 17.3% time reduction against full QR for the analytical expansion step. Peak allocated GPU memory is 7.87–9.09% lower than for the traditional update.

On identical expanded designs and binary targets, with ridge coefficient ρ=10−4\rho=10^{-4} and float64 arithmetic, the largest relative output-weight difference from full QR is 1.83 × 10⁻¹⁰ and the largest absolute relative ridge-objective difference is 3.99 × 10⁻¹⁶. The block update thus attains the full QR ridge solution to high numerical accuracy at lower measured solver cost.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.