acceptodds
Withdrawn from ICLR 2027

Robust Unsupervised Cross-Modal Hashing via Causal Subject-Context Intervention

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In unsupervised cross-modal hashing, similarity relations are used to guide hash learning and thus directly shape the learned Hamming space. Although existing methods seek to obtain more reliable similarity relations through various estimation and refinement strategies, they generally derive such relations from representations where retrieval-relevant subject semantics may still be entangled with contextual factors, such as shared backgrounds, textual styles, or incidental co-occurrences. Consequently, context-induced associations may be encoded as semantic similarities and propagated through iterative similarity updates, thereby misleading subsequent hash learning. To address this issue, we propose Causal Subject-Context Intervention Hashing (CSCIH). Specifically, Causal Subject-Context Intervention (CSCI) learns subject- and context-oriented representations and performs an approximate differentiable intervention by restricting residual refinement to context-oriented dimensions while preserving subject-oriented ones. It further evaluates whether subject information remains consistent across alternative contextual states and summarizes this evidence as a subject consistency score. Moreover, CSCIH derives complementary similarity relations from subject-oriented representations and continuous hash representations. Health-Aware Structure Refresh (HASR) adaptively balances these relations according to subject consistency and training progress, enabling more reliable similarity updates throughout training. Finally, Geometry-Aware Joint Hash Learning (GJHL) captures modality-local and cross-modal relational structures in both Euclidean and hyperbolic spaces and integrates them into a continuous teacher representation to guide modality-specific hash learning. Experiments on three benchmark datasets demonstrate that CSCIH consistently outperforms representative unsupervised cross-modal hashing methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Unsupervised Cross-Modal Hashing, Robust Cross-Modal Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tPzQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CSCIH for unsupervised cross-modal hashing, motivated by the propagation of context-induced errors through iterative similarity updates. CSCI learns subject- and context-oriented representations and applies an approximate residual refinement. HASR combines subject-derived and hash-derived similarity relations using a consistency score and a fixed initial anchor. GJHL integrates modality-local and cross-modal structures through Euclidean and hyperbolic branches to construct a teacher representation. Modality-specific hashing heads are retained for inference. Evaluation on three benchmarks reports improvements across retrieval directions and hash lengths, supplemented by ablations, sensitivity analyses, and efficiency measurements.

Critical strengths

The paper addresses a relevant issue in unsupervised hashing: errors in estimated similarities can influence representation learning and propagate through subsequent updates. Combining complementary similarity sources with an initial anchor provides a useful approach to this feedback process.

Table 1 reports consistently favorable performance across three datasets, two retrieval directions, and three hash lengths. The additional component ablations provide useful evidence for the complete framework. Keeping the additional modules confined to training also preserves straightforward modality-specific inference.

Critical weaknesses

The residual refinement uses the sample’s own learned context representation, while alternative contexts are used only for consistency assessment. Moreover, the consistency score combines alternative availability and assignment confidence. These constructions support a useful design, but the connection to semantic stability under contextual intervention remains insufficiently explained. Explicit assumptions and limitations would strengthen the conceptual contribution.

Some experimental reporting needs reconciliation. For example, the full model’s NUS-WIDE results differ between the main comparison and the ablation tables. The baseline list in Section 5.1 also differs from the methods displayed in Table 1. These inconsistencies make it harder to establish the exact comparison settings and interpret component contributions.

Questions for authors

Could you clarify the assumptions under which the residual correction in Eq. (2) can be interpreted as a contextual intervention, given that it uses the sample’s own context representation and alternative samples are used only for assessment? A brief discussion of the intended causal interpretation and its limitations would help calibrate the main claim.

In Eq. (4), the score combines contextual-alternative availability with prototype-assignment confidence. How should a relatively high score be interpreted when assignments are confident but few eligible alternatives are available? Please explain the rationale for balancing these terms through λ_h and how this situation should influence HASR.

Could you clarify whether the main comparisons and ablations use the same full-model configuration and hash length? On NUS-WIDE, Table 1 reports 32-bit results of 65.9/66.7, whereas Tables 2 and 3 report 65.3/66.2. A short explanation of any configuration differences, or correction of the reported values, would help reconcile the performance and ablation conclusions.

Reviewer_7tXxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CSCIH, an unsupervised cross-modal hashing framework intended to reduce misleading contextual associations. It learns subject- and context-oriented representations, applies context-guided feature refinement, and adaptively updates similarity relations. A joint learning module combines Euclidean and hyperbolic representations to guide hash learning. Experiments on three datasets show improved retrieval performance.

Critical strengths
  1. The paper addresses a relevant problem: contextual associations can distort estimated similarities and affect later training through repeated similarity updates.
  2. Extensive experiments support the usefulness of the proposed components for cross-modal retrieval.
Critical weaknesses
  1. The proposed CSCI module uses a learned gate to separate subject and context features. It is unclear how this differs fundamentally from conventional feature disentanglement and perturbation methods. More importantly, the paper does not directly verify whether the learned representations truly separate subject information from contextual factors.
  2. The paper argues that context-induced spurious similarities can accumulate during training. There is no direct analysis of how similarity errors change during training or whether the method is more robust when the background or context changes.
  3. Although the paper compares CSCIH with several representative methods, some recent approaches, such as ENHash[1], BRAINHash[2], UWMCH[3], and HINT[4], are not included. These methods address related problems in semantic relation learning and cross-modal hashing. In addition, the author mentioned in Lines 346–348, such as UCCH、CDTH、VTMUCH、EGICH, are not included in the comparisons presented in Table 1. [1] Hao Fu, Zebing Yao, Chuangchuang Tan, and Guanghua Gu. Enhash: Error notebook-guided finegrained learning for unsupervised cross-modal hashing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 21136–21144, 2026. [2] Hao Fu, Guanghua Gu, YunchaoWei, and Yao Zhao. Brainhash: Brain-inspired region-aligned interaction network for unsupervised cross-modal hashing. IEEE Transactions on Image Processing, 2026. [3] Fan Yang, Yuanzhi Zhao, Haimei Zhao, Yudong Zhao, and Haikun Xu. Mask to align, weight to disambiguate: Reliable unsupervised cross-modal hashing with masked-weight contrast. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 30151–30161, 2026. [4] Zhiping Xiao, Junyu Luo, Hang Zhou, Yusheng Zhao, Xiao Luo, PengyunWang, Wei Ju, Siyu Heng, and Ming Zhang. Hierarchical encoding tree with modality mixup for cross-modal hashing. In International Conference on Learning Representations, pp. 126617–126641, 2026.
Questions for authors

Please find the questions or concerns in Critical Weaknesses.

Reviewer_RA6GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CSCIH, an unsupervised cross-modal hashing framework for image-text retrieval. The method consists of three main components. First, CSCI decomposes image and text features into subject-oriented and context-oriented representations using sample-dependent gates, then applies a context-modulated residual refinement and estimates a subject-consistency score across inferred contextual states. Second, HASR constructs similarity relations from continuous hash representations and subject-oriented representations, and adaptively combines them according to the estimated subject consistency and training progress while anchoring the updates to an initial similarity matrix. Third, GJHL models modality-local and cross-modal structures using Euclidean and hyperbolic graph propagation, and uses the resulting fused continuous representation as a teacher for modality-specific hash functions. The method is evaluated on MIRFlickr-25K, NUS-WIDE, and MS-COCO using image-to-text and text-to-image retrieval with 16-, 32-, and 64-bit codes. The paper reports improvements over several unsupervised cross-modal hashing baselines, together with additive and fine-grained ablations, parameter sensitivity experiments, efficiency measurements, and a qualitative retrieval example.

Critical strengths

Please identify one or two critical strengths of the paper that support acceptance. For each, explain briefly The paper addresses a meaningful problem in unsupervised cross-modal hashing: similarity matrices are repeatedly estimated from evolving representations, and context-induced errors can propagate across updates. The idea of anchoring similarity updates to an initial matrix and regulating them with a consistency signal is technically reasonable and relevant to this failure mode.

Critical weaknesses
  1. The causal subject-context intervention claim is not established. Equation (2) does not replace the context with an alternative state or construct a counterfactual representation; it adds a residual generated from the same sample’s learned context representation. In addition, the learned gate and the subject/context encoders are not identifiable. Without context annotations, controlled context changes, or a structural causal assumption, the paper cannot establish that the learned subject representation is causal or context-invariant.
  2. The performance gains cannot be reliably attributed to the proposed causal or geometric components. GJHL is a high-capacity teacher architecture with many trainable transformations and fusion mechanisms. The reported ablations do not include parameter-matched Euclidean-only teachers, controlled capacity baselines, or direct context-shift experiments. The paper also lacks multiple random seeds, variance estimates, and statistical significance tests. Therefore, the reported improvements may reflect model capacity, graph refinement, or hyperparameter choices rather than the claimed causal intervention.
Questions for authors
  1. What exactly are the contextual prototypes pkp_k? Are they learnable parameters, batch-level cluster centers, or externally computed prototypes? How are they initialized and updated across epochs, and how is prototype collapse prevented?
  2. Please clarify the causal meaning of Eq. (2). Since the residual is generated from the current sample’s own context representation cic_i, what is the formal intervention variable, and where is the alternative contextual state introduced? Can the authors provide a context-swap or counterfactual experiment showing that subject representations remain stable when context changes?
  3. How are the gate values gimg_i^m distributed during training? Please report their mean, variance, and per-dimension statistics, and compare against fixed, random, and uniform gates to rule out all-subject or all-context degeneration.
  4. The experimental setup lists one group of baselines, whereas Table 1 reports another group. Which baselines were actually implemented, and were all methods trained with identical CLIP features, training subsets, optimization budgets, and hyperparameter tuning procedures?
  5. Can the authors provide results over at least three to five random seeds with standard deviations or confidence intervals? Several reported improvements are relatively small and may not be statistically robust.
  6. Can the authors provide parameter-matched and capacity-matched comparisons between the full GJHL teacher, a Euclidean-only teacher, a hyperbolic-only teacher, and a simpler MLP teacher?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.