acceptodds
Under review as a conference paper at ICLR 2027

High-Order Interaction and Modality-Specific Distillation for Unsupervised Cross-Modal Hashing

Abstract

Unsupervised cross-modal hashing aims to encode heterogeneous data into a shared compact Hamming space without relying on semantic annotations. A key design challenge is to exploit interactions between paired image and text features during training while transferring the resulting multimodal knowledge to independently deployable unimodal hash encoders. To address this limitation, we propose a High-Order Cross-Modal Hashing framework, termed HOCH, which combines progressive cross-modal interaction with relational knowledge distillation. Its cascaded multi-head interaction blocks form products of normalized low-rank projections and use these interaction features to update the image and text representations through separate gated residual updates. Each subsequent block computes interactions from the updated representations, without explicitly constructing full outer products. An adaptive modality fusion module then combines the enhanced representations using sample-dependent weights to construct a multimodal teacher. Modality-specific relational distillation transfers both cross-modal and intra-modal similarity structures from the teacher to image and text student hash branches, enabling each branch to generate hash codes independently at inference. Experiments on MIRFlickr, NUS-WIDE, and MSCOCO with hash-code lengths from 16 to 128 bits show that HOCH achieves the best results in most evaluated settings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Cross-modal hashing, cross-modal retrieval, high-order interaction, relational knowledge distillation, adaptive modality fusion

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_XY4FOpenReview ↗
Rating
3
Confidence
1(−2)
Informativeness
2
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper proposes HOCH for unsupervised cross-modal hashing. Cascaded interaction blocks combine normalized low-rank multiplicative features with gated residual updates to progressively refine paired image and text representations. Adaptive modality fusion constructs a multimodal teacher, whose cross-modal and intra-modal similarity structures are distilled into independently deployable unimodal students. Teacher and student branches share modality-specific encoders but use separate hash heads. Experiments cover three datasets, two retrieval directions, and four hash lengths using fixed CLIP features, supplemented by component ablations, parameter sensitivity analyses, and efficiency measurements.

Critical strengths

The framework addresses a practical training–deployment distinction: paired modalities can provide richer supervision during training, while each deployed student requires only its own modality. The architecture makes this distinction explicit, and the reported inference paths exclude the interaction and fusion modules. This gives the approach a useful deployment rationale beyond improving a jointly evaluated multimodal representation.

The empirical evaluation is reasonably broad. The paper states that all methods use identical partitions, fixed CLIP features, hash lengths, and evaluation protocols. HOCH achieves the highest reported mAP@50 in 20 of 24 settings, with particularly substantial improvements on MSCOCO at short code lengths. The mAP@K curves provide additional evidence beyond one retrieval depth. Table 3 also reports measured training costs relative to FirstOrder, supporting the practical value of the proposed compact interaction design in that setting.

Critical weaknesses

The paper does not specify the number of independent training runs or report variability for retrieval accuracy; the latency standard deviations do not address this uncertainty. Several important settings are also missing, including the pseudo-similarity quantile, hash sharpness, ranking margins and temperatures, regularization weights, and teacher/student weighting schedules. Encoder and aggregation architectures are not fully specified, and checkpoint-selection and tuning procedures remain unclear. These omissions matter when interpreting small component-level gains and reproducing the method without the planned code release.

Questions for authors

Could you provide the exact FirstOrder formulation and explain the parameter-count difference in Table 3? On one representative dataset at 64 bits, could you compare HOCH with an additive variant retaining the same low-rank projections, normalization, gates, and output dimensions? A single-block multiplicative variant would also help determine whether recursive refinement provides an additional benefit beyond multiplicative interaction itself.

Could you report mean and standard deviation over at least three training seeds for HOCH, FirstOrder, and MeanFusion on one representative 64-bit setting, keeping the data split fixed? Reporting the paired differences between variants would be particularly informative. For example, on NUS-WIDE, the reported gains over FirstOrder and MeanFusion are 0.0037 and 0.0007, respectively; their consistency across runs could materially affect the assessment of the individual modules.

Could you provide a compact configuration table specifying the pseudo-similarity quantile, hash sharpness, DropPath probabilities, ranking margins and temperatures, loss coefficients, and teacher/student weighting schedules, together with the encoder and aggregation architectures? Please also explain how checkpoints and hyperparameters were selected, including the dataset-specific feature-preservation weights, and whether any labeled validation data were used. This would clarify the reproducibility and supervision requirements of the reported results.

5 comments
AuthorsOpenReview ↗

Response to Reviewer XY4F [1/5]

Dear Reviewer XY4F,

We sincerely thank you for your careful review and constructive questions about the FirstOrder comparison, the reliability of the ablation results, and the reproducibility of our experimental protocol. In response, we clarified the FirstOrder formulation and parameter counts, conducted controlled interaction comparisons and three-seed ablations, and brought the training settings and model-selection procedure together in a concise account.

We hope these additions make the evidence behind our conclusions easier to assess, and we would be grateful if you would reconsider our work.

Sincerely, Co-authors

AuthorsOpenReview ↗

Response to Reviewer XY4F [2/5]

Response to Weakness 1 & Question 1: FirstOrder formulation, parameter-count differences, and controlled interaction analysis.

Thank you for asking us to separate the effects of the interaction operator and recursive refinement, and to make the parameter comparison reproducible. We audited the models module by module. At 64 bits, HOCH contains 12,549,890 parameters: 4,726,784 in the two modality encoders, 3,150,336 in the shared aggregation network, 4,540,416 in the four interaction blocks, 131,328 in the four teacher/student hash heads, and 1,026 in adaptive fusion.

In the originally submitted FirstOrder variant, the entire multiplicative interaction branch in each block is replaced by a full-width additive projection:

ul=GELU(LN(Waddl(zil−1+ztl−1)))u^l = GELU(LN(W_{add}^l(z_i^{l-1}+z_t^{l-1})))

where WaddlW_{add}^l is a learnable 512-by-512 projection matrix. This additive projection and its LayerNorm contain 263,680 parameters per block, compared with 83,456 in HOCH's multiplicative interaction branch. The difference is 180,224 per block and 720,896 across four blocks, giving FirstOrder 13,270,786 parameters (13.27M). The gates and residual updates are identical in these two variants; the parameter difference is confined to the interaction branch.

We also report the A4* additive control from our controlled experiments. A4* retains HOCH's four-block recursive architecture, normalized low-rank projections, gates, output dimensions, aggregation, fusion, and distillation, replacing only element-wise multiplication with addition. Both variants have 12.55M active parameters.

Table R1(a). Parameter-matched interaction comparisons at 64 bits (three-seed mean ± sample SD of bidirectional average mAP@50).

MethodOperatorInteraction inputsMIRFlickrNUS-WIDEMSCOCO
HOCHMultiplicationUpdated0.9181 ± 0.00100.8260 ± 0.00190.9411 ± 0.0026
A4*AdditionUpdated0.9174 ± 0.00050.8263 ± 0.00190.9414 ± 0.0016

We additionally evaluated M1, a single-block multiplicative variant that retains the first interaction block's gate and residual update and omits the subsequent three blocks. At 64 bits, the three-seed means ± sample standard deviations of bidirectional average mAP@50 are:

Table R1(b). Single-block versus four-block multiplicative interaction at 64 bits. Values are the mean ± sample standard deviation of bidirectional average mAP@50 over three training seeds. The last column reports the difference between the HOCH and M1 means.

DatasetM1 (one block)HOCH (four blocks)HOCH − M1
MIRFlickr0.9176 ± 0.00120.9181 ± 0.0010+0.0005
NUS-WIDE0.8250 ± 0.00210.8260 ± 0.0019+0.0010

At 64 bits, four-block HOCH improves mean bidirectional mAP@50 over the single-block M1 variant on MIRFlickr and NUS-WIDE, showing that additional interaction stages contribute beyond a single multiplicative block in these settings. In the revised manuscript, we will clarify the FirstOrder formulation and parameter-count difference in Appendix C.4, and briefly summarize the parameter-matched A4* and single-block M1 comparisons there.

AuthorsOpenReview ↗

Response to Reviewer XY4F [3/5]

Response to Weakness 2 & Question 2: multi-seed ablation results and statistical significance analysis.

Thank you for pointing out that the small single-run ablation gains need a variability estimate. We repeated HOCH, FirstOrder, and MeanFusion on NUS-WIDE at 64 bits with the same data split and three training seeds (2024, 2025, and 2026).

Table R2(a). Three-seed NUS-WIDE ablation at 64 bits.

MethodI2TT2IAverageHOCH − variant
HOCH0.8385 ± 0.00150.8135 ± 0.00250.8260 ± 0.0019—
FirstOrder0.8321 ± 0.00290.8111 ± 0.00260.8216 ± 0.0009+0.0044 ± 0.0012
MeanFusion0.8399 ± 0.00150.8128 ± 0.00420.8263 ± 0.0024−0.0003 ± 0.0012

Table R2(b). Teacher-relation Precision@50 with original and shuffled AMF weight assignments on NUS-WIDE at 64 bits.

seedOriginal P@50Shuffled P@50Paired difference
20240.8262100.824166+0.002044
20250.8392070.829679+0.009528
20260.8240070.824009−0.000002

HOCH exceeds FirstOrder in all three training seeds; its mean paired gain is +0.0044 ± 0.0012. HOCH and MeanFusion are approximately tied in bidirectional retrieval performance. However, the sample-specific fusion weights still help the teacher model preserve semantic neighbors. To test whether the learned sample-to-weight assignment matters to the multimodal teacher, we held the teacher checkpoint and input features fixed and shuffled the learned image-text fusion weights across samples. The original assignment gave higher teacher-relation Precision@50 in two seeds and a negligible difference in the third; its mean paired advantage was +0.003857 ± 0.005017.

In the revised manuscript, we will add the three-seed results and paired differences to the ablation appendix and update the discussion accordingly.

Reviewer_a4dROpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how to effectively leverage feature interactions between images and text during the training phase of unsupervised cross-modal hashing, and how to transfer multimodal knowledge to independently deployable unimodal hashing encoders. To this end, the authors propose the HOCH framework, which enhances image-text representations through multi-layer low-rank multiplicative interactions and constructs a multimodal teacher model using adaptive fusion. The model employs relational distillation to transfer the teacher's cross-modal and intra-modal relationships to unimodal student models capable of independent inference. The authors conducted retrieval experiments across three datasets and various hash code lengths, providing analyses on component ablation, parameter sensitivity, visualization, and efficiency. The results demonstrate that HOCH achieves superior retrieval performance in most evaluation settings.

Critical strengths

1.The multi-layer low-rank multiplicative interaction design stands out as a distinctive feature of the proposed method, by recomputing interactions from layer-wise updated vision-language representations, it offers a concrete approach to modeling cross-modal features. Furthermore, by incorporating relational distillation, the method leverages multimodal information during training while preserving the capability for independent unimodal inference, demonstrating a well-integrated design across its various modules.

2.The empirical evaluation covers solid and diverse experimental settings across three mainstream datasets, bidirectional retrieval tasks, and multiple hash code lengths. Extensive ablation studies, efficiency analyses, parameter sensitivity tests, and qualitative visualizations further validate the method’s effectiveness. The proposed HOCH achieves particularly strong performance on MSCOCO. Crucially, the distilled student branches incur no additional inference overhead from intermediate interaction or fusion modules.

Critical weaknesses

1.The current experiments compare HOCH mainly with a FirstOrder variant, but provide little evidence on how the progressive interaction behaves as the number of HOI blocks increases. Since the full model uses four cascaded interaction blocks, it would be useful to show whether each additional block consistently improves retrieval performance, or whether the gains saturate after only a few stages. Such an analysis would also make the role of the layer-by-layer representation updates easier to understand.

2.After adaptive fusion, HOCH still produces teacher outputs through two modality-specific hash heads, each combining the enhanced unimodal representation with the fused representation. The paper explains how these two branches are constructed, but not why this design is preferable to generating a single teacher output directly from the fused feature. A comparison with a single fused teacher, or further discussion of what the two teacher heads preserve for relational distillation, would make this design choice more convincing.

3.The pseudo-similarity supervision is constructed from mini-batch-specific quantile thresholds computed separately for the image and text similarities. This makes the positive-pair assignment potentially dependent on batch composition and batch size, yet the paper does not examine how stable these pseudo relations are under different sampling conditions. Since the ranking objectives of both teacher and student branches ultimately rely on this supervision, additional sensitivity analysis on the quantile threshold or batch configuration would help assess the robustness of the training signal.

Questions for authors

1.The full model uses four cascaded HOI blocks. How was this depth determined? It would be helpful to report results with different numbers of interaction blocks and, if possible, the performance after each stage to show whether progressive interaction continues to add useful information or quickly saturates.

2.Why are two modality-specific teacher hash heads retained after adaptive fusion? Since the fused representation already combines image and text information, I wonder whether a single hash head operating directly on the fused representation would provide a comparable distillation target. A comparison between these two designs would help justify the current teacher architecture.

3.The teacher and student share a modality encoder, while MSRD applies a stop-gradient to the teacher's similarity matrix. Does the student loss still indirectly affect the teacher's representations via the shared encoder? Please explain the gradient propagation path under this setup.

7 comments
AuthorsOpenReview ↗

Response to Reviewer a4dR [1/5]

Dear Reviewer a4dR,

We sincerely appreciate the reviewer careful review and constructive questions about the choice of HOI depth, the rationale for two teacher hash heads, and the gradient flow between the teacher and student. In response, we have completed a three-seed depth comparison and explained how the two teacher heads provide distinct relational targets and how student gradients influence the teacher through shared encoders.

We hope these results and explanations address your concerns, and we would be grateful if you would reconsider your assessment of our work.

Sincerely, Co-authors

AuthorsOpenReview ↗

Response to Reviewer a4dR [2/5]

Response to Weakness 1 & Question 1: HOI block depth selection and progressive interaction analysis.

Thank you for asking how the four-block depth was chosen and whether progressive interaction continues to help. Four blocks were fixed as the default architecture during method design. This design combines the initial representation and four successive updates in the shared aggregation network. The four blocks use DropPath probabilities of 0, 0.1, 0.3, and 0.5, respectively.

To assess the effect of depth, we trained separate models with one, two, three, and four multiplicative interaction blocks on MIRFlickr at 64 bits using seeds 2024, 2025, and 2026. All entries are bidirectional average mAP@50. The last column gives the seed-paired difference between four-block HOCH and each shallower variant.

Table R4. Interaction-depth comparison on MIRFlickr at 64 bits (mean ± sample standard deviation over three seeds).

Interaction blocksBidirectional average mAP@50HOCH (4 blocks) − variant
10.917592 ± 0.001191+0.000464 ± 0.000195
20.917146 ± 0.001094+0.000910 ± 0.000097
30.917531 ± 0.001060+0.000525 ± 0.000279
4 (HOCH)0.918056 ± 0.000998—

On MIRFlickr, four-block HOCH achieves the highest mean score and exceeds each shallower model at every paired seed. The benefit is not monotonic at intermediate depths: the two-block model scores below the one-block model, while the four-block model performs best. Thus, the results support the four-block default on this representative 64-bit setting without establishing it as a universal optimum. We will add this depth comparison to the revised appendix.

AuthorsOpenReview ↗

Response to Reviewer a4dR [3/5]

Response to Weakness 2 & Question 2: modality-specific teacher hash heads and single-head fusion comparison.

Thank you for asking whether one hash head on the fused representation can replace the two modality-specific teacher heads. AMF produces a shared fused representation, but the two teacher heads do not receive that representation alone. For an image–text pair, their outputs are

hiT=tanh⁡(β[(zi∗+zf)WiT+ciT])\mathbf{h}_{i}^{T}=\tanh(\beta[(\mathbf{z}_{i}^{*}+\mathbf{z}_{f})\mathbf{W}_{i}^{T}+\mathbf{c}_{i}^{T}])

htT=tanh⁡(β[(zt∗+zf)WtT+ctT])\mathbf{h}_{t}^{T}=\tanh(\beta[(\mathbf{z}_{t}^{*}+\mathbf{z}_{f})\mathbf{W}_{t}^{T}+\mathbf{c}_{t}^{T}])

Thus, each teacher code incorporates the same fused information while retaining a path through its own enhanced modality representation. This distinction is used directly by MSRD, whose teacher targets are

SitT=Cos⁡(HiT,HtT)\mathbf{S}_{it}^{T}=\operatorname{Cos}(\mathbf{H}_{i}^{T},\mathbf{H}_{t}^{T})

SiiT=Cos⁡(HiT,HiT)\mathbf{S}_{ii}^{T}=\operatorname{Cos}(\mathbf{H}_{i}^{T},\mathbf{H}_{i}^{T})

SttT=Cos⁡(HtT,HtT)\mathbf{S}_{tt}^{T}=\operatorname{Cos}(\mathbf{H}_{t}^{T},\mathbf{H}_{t}^{T})

The cross-modal target relates the two teacher code spaces, while the intra-modal targets retain the neighborhood structures within each modality. These distinct targets supervise the corresponding image and text student similarity matrices.

If one head instead hashes only zf\mathbf{z}_{f} and the resulting code HfT\mathbf{H}_{f}^{T} is reused as both teacher outputs, all three teacher targets become Cos⁡(HfT,HfT)\operatorname{Cos}(\mathbf{H}_{f}^{T},\mathbf{H}_{f}^{T}). The student losses would still be three separate terms, but they would be matched to the same teacher Gram matrix. This changes the relational supervision supplied to the students as well as the number of teacher heads. We retain two heads to preserve modality-specific teacher relations after fusion. We will clarify this design rationale in Sections 3.3 and 3.4 of the revised manuscript.

Reviewer_i8YROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose HOCH which combines high-order cross-modal interaction, adaptive modality fusion, and modality-specific relational distillation. A multimodal teacher transfers similarity structures to independent image and text hashing branches. Experiments cover three datasets and hash-code lengths from 16 to 128 bits, using fixed image and text features.

Critical strengths
  • The teacher uses paired information during training while the student branches remain independently deployable. Theis provides a practical way to transfer multimodal information without adding the interaction modules at inference.

  • The comparisons use matched features and data partitions, cover several code lengths, and examine retrieval beyond the first 50 results. HOCH performs particularly well on MSCOCO.

Critical weaknesses
  • The related work mentions UDCH, ENHash, CIRH, CMGCH, and HuggingHash as recent strong methods, but none appears in Table 1.

  • The contribution of the interaction and fusion choices is not established convincingly. Table 2 reports average gains of 0.00288 over FirstOrder and 0.00182 over MeanFusion, without variation across training runs. Given the established factorized-interaction and distillation methods discussed in the paper, stronger isolation of the recursive interaction would help establish the contribution.

  • The FirstOrder replacement needs a more precise description. Table 3 reports 13.27 million active parameters for FirstOrder and 12.55 million for HOCH, while Appendix C.4 describes an additive replacement with the remaining structures unchanged. This difference does not itself establish an unfair comparison, but its source matters when interpreting both the accuracy and efficiency claims.

Questions for authors

Please address the points raised in the critical weaknesses section.

5 comments
AuthorsOpenReview ↗

Response to Reviewer i8YR

Dear Reviewer i8YR,

We sincerely thank the reviewer for the careful assessment and constructive suggestions. In response, we expanded the comparison to include five recent methods under a unified evaluation protocol, added three-seed ablations and controlled experiments to clarify the effects of recursive interaction and adaptive fusion, and provided a module-level parameter audit with the exact FirstOrder formulation.

We would be profoundly grateful if you would consider re-evaluating our work.

Sincerely, Co-authors

AuthorsOpenReview ↗

Response to Reviewer i8YR [2/5]

Response to Weakness 1 & Question 1: comparison with recent methods under a unified evaluation protocol.

We thank the reviewer for checking which methods cited in the related work actually enter the quantitative comparison. We agree that a table omitting recently published competitors weakens the empirical claim. We have therefore given all five named methods an explicit disposition, separated re-runs from quoted numbers, and made the provenance of every entry explicit.

We audited the official releases through the paper pages, GitHub, Gitee and the authors' publication pages. As of this response, only CIRH has a public implementation (XizeWu/CIRH). Its pipeline hard-codes the original handcrafted feature dimensions ori_featI and ori_featT in .mat files and builds the first encoder as Linear(ori_feat, 512), so the input dimension is fixed by the shipped data. We reproduced UDCH, ENHash, CMGCH, and HuggingHash based on the architectures, loss functions, and training configurations described in their respective papers. All methods were evaluated under the same experimental protocol, with their inputs adapted to 512-dimensional CLIP features. The reproduced results are reported as obtained, while the results originally reported in the corresponding papers are presented separately in Panel B.

Since the performance of hashing methods is constrained by the information contained in their input features, for the sake of fair comparison, Table 1 includes only methods evaluated using the same 512-dimensional CLIP features, identical dataset splits, and the same evaluation script.

Table R1: The five methods named by the reviewer, at 64 bits. Panel A entries (†) are re-run by us under the unified protocol.

Panel A — unified protocol (mAP@50)

MethodVenueBackboneMIRFlickr I2T / T2INUS-WIDE I2T / T2IMSCOCO I2T / T2I
HOCHours512-d CLIP0.933 / 0.9010.837 / 0.8110.936/0.949
CIRHTKDE'23512-d CLIP0.917 / 0.9000.845 / 0.8240.930 / 0.945
HuggingHash†IJCV’24512-d CLIP0.731/0.7270.604/0.5870.934/0.940
CMGCH†AAAI’24512-d CLIP0.847/0.8590.755/0.7660.757/0.766
ENHash†AAAI’26512-d CLIP0.847/0.8650.775/0.7230.801/0.839
UDCH†AAAI’26512-d CLIP0.909/0.8900.798/0.7750.790/0.802

Panel B — quoted from original paper

MethodVenueBackbone in the original paperReported metric and value
UDCHAAAI'26Image and text features as inputsMIR 0.9054/0.9025、NUS 0.8838/0.8551、COCO 0.9011/0.9005(MAP@50)
ENHashAAAI'26Image and text features with a Transformer encoder and MLPMIR 0.756/0.747、NUS 0.734/0.755、COCO 0.669/0.667(MAP@ALL)
CMGCHAAAI'24Faster R-CNN/BUTD image-region features and BERT word embeddingsCOCO 0.776/0.763(MAP@50)
HuggingHashIJCV'24ViT-Base/16 image encoder and BERT-Base text encoderFlickr25K 0.764/0.766; NUS 0.662/0.663; COCO 0.526/0.675(MAP@ALL)
CIRHTKDE'23AlexNet image features and BoW text featuresMIR 0.929/0.900; NUS 0.854/0.810; COCO 0.830/0.872(MAP@50)

As Table R1 shows, under the unified protocol HOCH is higher than CIRH in both MIRFlickr and MSCOCO, where as CIRH is higher on NUS-WIDE by 0.008 in I2T and 0.013 in T2I. The Panel B methods are competitive on their own backbones. The slight performance gap on NUS-WIDE stems from differences in how the two methods preserve dataset-specific similarity structures. CIRH explicitly optimizes correlation and identity reconstruction, whereas HOCH trains independently deployable unimodal student networks through a multimodal teacher and relational distillation. These methods impose different constraints on the learned hash codes, leading to different advantages across datasets and retrieval directions.

In the revised paper, Section 4.1 states which methods are excluded and why, and the related-work wording no longer implies a comparison the table does not contain.

AuthorsOpenReview ↗

Response to Reviewer i8YR [3/5]

Response to Weakness 2 & Question 2: run-to-run variability and controlled evaluation of recursive interaction and adaptive fusion.

Thank you for pointing out that the original ablation table did not report run-to-run variability and that the FirstOrder comparison did not isolate recursive interaction. We address these concerns with two sets of controlled comparisons. All results below use 64-bit codes and the same data split across the three training seeds. For each run, we select the checkpoint with the highest bidirectional average mAP@50 and report both retrieval directions from that checkpoint.

Table R2(a). Three-seed ablation results on NUS-WIDE (mean ± sample standard deviation). The last column reports paired differences in bidirectional average mAP@50, computed as HOCH minus the variant at the same seed.

MethodI2TT2IBidirectional averagePaired HOCH − variant
HOCH0.8385 ± 0.00150.8135 ± 0.00250.8260 ± 0.0019—
FirstOrder0.8321 ± 0.00290.8111 ± 0.00260.8216 ± 0.0009+0.0044 ± 0.0012
MeanFusion0.8399 ± 0.00150.8128 ± 0.00420.8263 ± 0.0024−0.0003 ± 0.0012

Table R2(b). Teacher-relation Precision@50 with original and shuffled AMF weight assignments on NUS-WIDE at 64 bits.

seedOriginal P@50Shuffled P@50Paired difference
20240.8262100.824166+0.002044
20250.8392070.829679+0.009528
20260.8240070.824009−0.000002

On NUS-WIDE, HOCH outperforms the FirstOrder variant from the original submission across all three training seeds. Although the results of the MeanFusion variant are roughly on par with those of HOCH, the sample-specific fusion weights still help the teacher model preserve semantic neighbors. To investigate whether the correspondence between fusion weights and individual samples is meaningful, we kept the teacher model’s checkpoint and input features fixed, while randomly shuffling the learned image-text fusion weights across different samples. At 64 bits, the original assignment yielded higher teacher-relation P@50 than the shuffled assignment in two of the three seeds, with a negligible difference in the third. The mean paired difference was +0.0039 ± 0.0050 (mean ± sample standard deviation).

Table R2(c). Parameter-matched interaction comparisons at 64 bits. Entries are the three-seed mean ± sample standard deviation of bidirectional average mAP@50.

MethodInputs to the interaction operatorMIRFlickrNUS-WIDEMSCOCO
HOCHUpdated representations0.9181 ± 0.00100.8260 ± 0.00190.9411 ± 0.0026
A4*Updated representations0.9174 ± 0.00050.8263 ± 0.00190.9414 ± 0.0016
MFH-styleFixed initial representations0.9171 ± 0.00060.8273 ± 0.00320.9409 ± 0.0020

To isolate recursive interaction, we compare HOCH with MFH-style. Both variants retain four multiplicative interaction blocks, the same gates, residual updates, and aggregation. Only the inputs to the interaction operator change: HOCH uses the representations updated by the preceding block, whereas MFH-style uses the initial representations in every block. On MIRFlickr, updating the interaction inputs improves mAP@50 in all three seeds, with a paired gain of +0.0010 ± 0.0005. We will revise the manuscript to state the narrower conclusion supported by these controls: recursive interaction gives a small, directionally consistent gain on MIRFlickr at 64 bits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.