acceptodds
Under review as a conference paper at ICLR 2027

Reliability-Disentangled Distillation Hashing: Selective Bit and Semantic Purification for Multimodal Retrieval

Abstract

Unsupervised multimodal hashing is critical for retrieval efficiency, yet suffers from excessive parameters and redundant code bits. Standard distillation methods treat the teacher model as a perfect supervisor, forcing the student model to mimic representations that inherently contain stochastic noise and errors. To break this, we propose a Reliability-Disentangled Distillation Hashing (RDDH) framework for multimodal retrieval, which energizes the student to selectively inherit reliable knowledge via a dual-granularity purification mechanism. At the micro level (bit-wise), we present an information-theoretic reliability module based on secondary clustering that explicitly disentangles discriminative bits from redundant noise via entropy and polarity analysis, focusing the student on semantically stable directions. At the macro level (semantic-wise), we rectify the geometry of teacher through a filtered bidirectional contrastive objective that acts as a gatekeeper, masking the false-positive alignment and false-negative repulsion to preserve the intrinsic topological structure. Specifically, to balance the supervision sparsity from strict filtering, an intra-student contrastive task is integrated to ensure global data utilization. Extensive experiments on three benchmark datasets demonstrate that RDDH effectively purifies the distilled knowledge, realizing a lightweight student to significantly surpass the performance of teacher.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal retrieval, Hash learning, Cross-modal hashing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CB94OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript proposes a novel distillation framework for cross-modal hashing that decouples reliable knowledge from inherent teacher noise. Recognizing that pre-trained teacher models often produce redundant and unstable hash codes, especially when dealing with complex, multi-label semantic relations, the manuscript designs a dual-granularity purification system. At the micro-level, the method employs an elbow-based and label-cardinality-guided clustering approach to compute the entropy and polarity of each bit, which subsequently informs a continuous reliability mask. At the macro-level, the framework adjusts the contrastive learning objectives by filtering out false-positive supervision and shielding latent semantic neighbors from incorrect false-negative repulsion. Comprehensive evaluations illustrate its capability to yield efficient students that consistently surpass their teacher counterparts across multiple code lengths.

Critical strengths

(1) The core mechanism of decoupling reliability from noisy supervision offers an insightful perspective in the domain of multimodal hashing. Rather than treating the output of teacher as an absolute standard, the framework utilizes information-theoretic measures to assess the trustworthiness of each individual hash dimension. This design prevents the student from inheriting the representational biases of teacher and memorizing random noise, which bridges the gap between information theory and discrete optimization in cross-modal retrieval. (2) The manuscript provides thorough ablation studies that isolate and validate the individual contributions of the reliability masks, semantic filtering mechanisms, and internal contrastive objectives. The logical flow from identifying the problem of teacher noise to addressing it via bit-wise and semantic-wise purification is coherent. The consistent improvements, especially the performance leaps observed when distilling from weaker, noisier teachers, corroborate the theoretical claims of the paper.

Critical weaknesses

The problem formulation and technical contributions are well articulated, but there are a few areas regarding algorithmic robustness, parameter sensitivity, and computational transparency that require clarification: (1) The framework heavily depends on an offline dual-clustering step, particularly relying on the elbow method to determine the coarse partitions and pseudo-labels. Heuristic clustering can be quite fragile under extreme noise distributions, the potential failure modes of this initialization phase are underexplored. An incorrect clustering granularity could systematically propagate erroneous ones. (2) Furthermore, although the paper mentions the offline nature of the preprocessing steps, it fails to explicitly quantify the exact computational time cost it adds to the overall pipeline. While bit-wise calculations on centroids might be efficient, performing K-means clustering over the entire dataset could become a bottleneck. (3) The framework introduces several heuristic thresholds for bit masking (e.g., vminv_{min}, vmaxv_{max}, and λ\lambda parameters). Although setting defaults promotes cross-dataset generalization, the lack of deep discussion on how these parameters affect the accuracy.

Questions for authors

Q1. How robust is the elbow-based clustering approach when applied to varying noise levels? Are there scenarios where incorrect clustering granularity severely degrades the downstream semantic filtering? Q2. Explicitly quantify the additional time cost introduced by the offline secondary clustering and bit-wise entropy/polarity calculations. How does this preprocessing overhead scale computationally as the size of the dataset increases? Q3. The overall method relies on several threshold values. Is the framework highly sensitive to small perturbations in these heuristic thresholds? Besides, the current methodology focuses on image-text pairs. If this framework were to be extended to more than two modalities (e.g., integrating video, audio, and text) simultaneously, would the dual-granularity purification still hold?

7 comments
AuthorsOpenReview ↗

Response to Reviewer CB94 [1/5]

Dear Reviewer CB94,

We appreciate your thorough review and constructive suggestions concerning the noise robustness of elbow-based clustering, offline preprocessing overhead, threshold sensitivity, and extension to more than two modalities. In response, we have quantified the exact computational complexity and runtime of the offline clustering and reliability calculations, and provided detailed analyses on parameter sensitivity and multi-modal generalizability. We have addressed your comments in five sections. We will release the corresponding code and experimental logs to support reproducibility.

We would be profoundly grateful if you would consider re-evaluating our work.

Sincerely,

Co-authors

AuthorsOpenReview ↗

Response to Reviewer CB94 [2/5]

Response to Weakness 1 & Question 1: robustness of elbow-based clustering and safeguards against noise.

1. Low-dimensional discrete space avoids the disadvantage of dimensionality.

Different from traditional clustering applied to high-dimensional continuous visual or text features, where heuristic elbow criteria often become unstable. The proposed dual-clustering operates strictly on compact, low-dimensional binary hash codes (k∈{16,32,64,128}k \in \{16, 32, 64, 128\} bits). Specifically, the elbow-based clustering employs standard kk-means with kk-means++ initialization, searching within a bounded range [1,40][1, 40] guided by dataset label cardinality to establish a stable coarse baseline (MaskE\text{MaskE}).

2. Multi-layered error mitigation and graceful degradation.

Even if the initial elbow clustering produces imperfect boundaries under noisy teacher outputs, three built-in safeguards prevent error propagation:

  1. Complementary second granularity and reliability filtering: any boundary ambiguity in the coarse partition is actively rectified by the secondary label-cardinality clustering (MaskC\text{MaskC}) and dual-criterion bit filtering (polarization and entropy).
  2. In macro-level semantic filtering, cluster assignments are used to check cross-modal consensus (fpi=I(yIt(i)=yTt(i))fp_i = \mathbb{I}(y_I^t(i) = y_T^t(i))). An imperfect partition simply causes uncertain sample pairs to fail the consensus check and be excluded from distillation candidate sets.
  3. Compensation via intra-student contrastive learning: any supervision sparsity caused by strict filtering is compensated by our holistic intra-student contrastive objective (Lcontra\mathcal{L}_{contra}), ensuring continuous global manifold utilization.

3. Empirical verification across varying noise levels.

In our setting, noise directly originates from the inherent imperfections of different teacher models. Across all seven teachers in Table 1, RDDH achieves consistent gains, notably delivers the largest improvements on weaker, noisier teachers, e.g., +13.4% average I2T mAP gain on DJSRH vs. +2.8% on DHRL.

AuthorsOpenReview ↗

Response to Reviewer CB94 [3/5]

Response to Weakness 2 & Question 2: quantitative preprocessing overhead and dataset scaling analysis.

1. Strict offline execution with zero inference overhead.

All dual-clustering, entropy, polarization, and mask construction operations are executed strictly once during an offline initialization phase prior to iterative network training. Consequently, they add 0 ms of computational overhead during query retrieval, where the lightweight student simply performs a single forward pass.

2. Theoretical complexity and empirical runtime measurements.

  1. kk-means clustering on compact binary hash codes has a time complexity of O(I⋅n⋅c⋅k)\mathcal{O}(I \cdot n \cdot c \cdot k), where II is the number of iterations, cc is the cluster count, and kk is the code length. Because it operates on low-dimensional binary vectors, it scales linearly with the number of training samples nn.
  2. On an NVIDIA RTX 4070 GPU, the complete preprocessing pipeline takes only about 4 seconds on MIRFlickr (n=5,000n = 5,000, 128 bits) and about 2 minutes on the large-scale NUS-WIDE dataset (n≈180,000n \approx 180,000, 128 bits).
  3. Crucially, bit-wise polarization and entropy are computed exclusively on the cc cluster centroids. This yields a constant complexity of O(c⋅k)\mathcal{O}(c \cdot k), which is completely independent of dataset size nn.
  4. Even on large datasets, total offline preprocessing accounts for less than 1.5% of the overall training time, confirming that it does not become a computational bottleneck.
Reviewer_WAQvOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the negative impact of noisy supervision in unsupervised multimodal hashing when transferring knowledge from a teacher to a student model. The authors observe that existing methods suffer from severe semantic degradation, because they blindly inherit both the valid correlations and the inherent redundancy produced by teacher networks. To mitigate the accumulation of incorrect semantic signals, the authors introduce a novel reliability-aware distillation framework named RDDH. This approach features a meticulously designed two-stage dual-granularity mechanism: a micro-level bit purification process leveraging secondary clustering alongside entropy and polarization metrics to mask unreliable hash dimensions, and a macro-level semantic filtration that suppresses false-positive and false-negative alignments during bidirectional contrastive distillation. This ensures that the student selectively assimilates only the most informative and stable semantic structures. Extensive experiments across various benchmark datasets (MIRFlickr, NUS-WIDE, IAPR-TC12) validate the method’s potential as a highly adaptable plug-and-play module that consistently enhances multiple existing teacher architectures.

Critical strengths

1.Moving away from the conventional paradigm where a student blindly mimics a teacher network, the authors introduce a proactive diagnostic mechanism to filter out low-quality supervision before it can corrupt the student’s learning process. The hierarchical design that addresses both individual bit stability and broader relational semantics is logically sound and theoretically appealing. 2.By showing consistent performance gains across a wide array of diverse teacher models, the authors provide strong evidence for the framework's versatility, scalability, and broad applicability as a universal enhancement module. The ablation studies meticulously verify the contributions of reliability masking, bidirectional distillation, and intra-student contrastive learning, forming a rigorous experimental closed loop.

Critical weaknesses

1.While the proposed distillation mechanism is structurally sound, the empirical setup feels somewhat disconnected from contemporary representation learning paradigms. The feature extraction relies on traditional backbones (VGG-19 for images and Bag-of-Words for text) rather than modern foundation models (CLIP or ViT). Given that modern architectures like CLIP already provide highly robust, well-aligned, and less noisy multi-modal embeddings out-of-the-box, it remains uncertain if the proposed bit-purification is still effective when the semantic ceiling of the teacher is significantly higher. 2.A central premise of the manuscript is the lightweight nature of the student model, claiming it achieves a superior Pareto frontier between real-time efficiency and retrieval accuracy. However, the current draft lacks a rigorous, quantitative efficiency analysis in the main text. Without direct comparisons of parameter counts, computational overhead, and inference latency between the complex teacher networks and the distilled student, the core motivation of deploying this method in resource-constrained environments remains unsubstantiated.

Questions for authors
  1. Provide empirical evidence demonstrating the framework’s efficacy when integrated with modern, foundation-model-based feature extractors (e.g., CLIP ViT-B/32). Does the performance gap between teacher and student remain significant in this scenario, or does the stronger backbone diminish the necessity of reliability purification?
  2. To fully substantiate the lightweight claim, present a comprehensive quantitative comparison detailing the parameter sizes, FLOPs, and end-to-end inference latencies for both the teacher models and your proposed student model. How do these metrics scale with increasing hash code lengths?
  3. The approach relies heavily on hyperparameter settings for controlling the distillation thresholds. Elaborate on the rationale behind fixing certain scaling weights (like the lambda parameters) to a default value of 1 across all datasets, rather than utilizing the seemingly optimal values (e.g., near 100) shown in your sensitivity analysis curves.
  4. The t-SNE visualizations provided in the supplementary materials do not seem to exhibit a dramatic improvement in cluster separability between the baseline and your proposed method. Provide a more detailed discussion that better illustrates the local semantic structural improvements brought by RDDH.
5 comments
AuthorsOpenReview ↗

Response to Reviewer WAQv [1/5]

Dear Reviewer WAQv,

We appreciate your thorough review and constructive suggestions concerning modern foundation backbones, quantitative efficiency analysis, hyperparameter scaling defaults, and semantic structural visualizations. In response, we have conducted additional experiments using the pretrained CLIP (ViT-B/32) backbone, provided a comprehensive comparison of parameter counts, FLOPs, and end-to-end inference latency, and clarified the hyperparameter choices and multi-label manifold properties. We addressed your comments in five sections. We will release the corresponding code and experimental logs to support reproducibility.

We would be profoundly grateful if you would consider re-evaluating our work.

Sincerely, Co-authors

AuthorsOpenReview ↗

Response to Reviewer WAQv [2/5]

Response to Weakness 1 & Questions 1: efficacy and necessity of bit-purification with modern foundation models (CLIP ViT-B/32).

1. Rationale for standard benchmarks and model-agnostic design.

In the original submission, we adhered to canonical benchmark settings in unsupervised cross-modal hashing (VGG-19 for images and BaW for text) to ensure strict, fair, and reproducible comparisons with baselines. Crucially, RDDH is entirely model-agnostic: our bit-level reliability purification and semantic-filtered distillation operate directly on latent hash representations and discrete binary code spaces. This makes RDDH inherently compatible with modern foundation backbones.

2. Empirical validation on pretrained CLIP (ViT-B/32).

We have conducted controlled experiments integrating the pretrained CLIP (ViT-B/32) backbone (following standard CLIP-based hashing protocols) on MIRFlickr and NUS-WIDE at 128 bits:

BackboneMethod / ModelI2T (MIRFlickr)T2I (MIRFlickr)I2T (NUS-WIDE)T2I (NUS-WIDE)
CLIP (ViT-B/32)DJSRH (Teacher)0.8240.8060.7220.711
DJSRH-RDDH (Ours)0.930 (+10.6%)0.880 (+7.4%)0.831 (+10.9%)0.790 (+7.9%)
CIRH (Teacher)0.9030.8840.8310.815
CIRH-RDDH (Ours)0.938 (+3.5%)0.911 (+2.7%)0.863 (+3.2%)0.831 (+1.6%)

3. Stronger backbones do not diminish the necessity of purification.

As demonstrated above, the performance gap between the distilled student and its CLIP-based teacher remains highly significant, e.g., +10.9% I2T mAP on NUS-WIDE for DJSRH, and +3.5% / +3.2% over the strong CIRH teacher. Even when foundation models provide well-aligned continuous embeddings out-of-the-box, the subsequent severe dimensionality compression and continuous-to-discrete quantization (sgn\text{sgn} binarization) inevitably introduce bit-level stochastic noise, boundary ambiguity, and cross-modal semantic instability. Stronger backbones elevate the semantic ceiling, while RDDH provides essential reliability purification to unlock the retrieval potential.

AuthorsOpenReview ↗

Response to Reviewer WAQv [3/5]

Response to Weakness 2 & Question 2: comprehensive quantitative efficiency analysis and code-length scaling.

1. Student network architecture.

Our student model comprises two symmetric, lightweight subnetworks (img_net and txt_net). Each branch consists of two linear hidden layers interleaved with LayerNorm, ReLU, Dropout (0.1), and a final Tanh activation, to generate continuous representations prior to binary quantization.

2. Quantitative complexity and inference latency comparison.

Using an NVIDIA RTX 4070 GPU, we profiled the exact parameter counts, computational overhead (FLOPs), and end-to-end query inference latency of the RDDH student against all seven teacher architectures on NUS-WIDE (128-bit):

ModelRoleParams (M)FLOPs (G)Latency (ms)SpeedUp (vs. RDDH)
DJSRHTeacher7.150.00740.691.17$\times$
DGCPNTeacher21.930.02190.791.34$\times$
PT-FUCHTeacher110.980.11102.023.42$\times$
CIRHTeacher18.430.01840.741.25$\times$
DHRLTeacher15.260.033613.9023.56$\times$
SMSHTeacher43.870.04381.552.63$\times$
USFTHTeacher11.590.01815.729.69$\times$
RDDH (Ours)Student9.780.00980.591.00$\times$ (Baseline)
  1. Our student requires only 9.78M parameters and 0.0098 GFLOPs, representing over a 91% parameter reduction compared to PT-FUCH (110.98M), and 15.6% fewer parameters with 45.8% fewer GFLOPs compared to the strong recent teacher USFTH (11.59M, 0.0181 GFLOPs).
  2. The student achieves an end-to-end latency of 0.59 ms, yielding a 23.56$\times$ speedup over DHRL (13.90 ms) and a 9.69$\times$ speedup over USFTH (5.72 ms).
  3. Dual-granularity clustering, entropy/polarization calculation, and mask construction are strictly one-off offline initialization processes executed before training, they introduce 0 ms of overhead at retrieval time.

3. Scaling across hash code lengths (16 to 128 bits).

In the dual-stream MLP student, the hash code length k∈{16,32,64,128}k \in \{16, 32, 64, 128\} only determines the output dimension of the final linear projection layer, whereas the dominant parameters and matrix multiplications reside in the hidden layers. Consequently, scaling kk from 16 to 128 bits adds negligible computational cost (less than 0.05M parameters and < 0.0005 GFLOPs), leaving end-to-end inference latency virtually invariant (fluctuating by less than 0.1 ms). This confirms a superior Pareto frontier between real-time deployment efficiency and retrieval precision across all bit budgets.

Reviewer_eQH2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Reliability-Disentangled Distillation Hashing (RDDH) for unsupervised image–text retrieval. The method aims to transfer reliable knowledge from a teacher hashing model to a lightweight student. It combines bit weighting derived from two clustering granularities, bidirectional contrastive distillation that filters potentially incorrect semantic relations, and an intra-student contrastive objective that compensates for reduced supervision. Additional losses align continuous and binary teacher–student representations and the student’s cross-modal probability distributions.

Experiments cover three datasets, seven teacher models, four code lengths, and both retrieval directions. The reported results consistently improve over the corresponding teachers. The main proposed contribution is selective knowledge transfer at both the bit and semantic-relation levels.

Critical strengths

Broad evaluation across teacher models supports the framework’s practical applicability. Table 1 evaluates seven teachers on three datasets at four code lengths, rather than demonstrating improvement for only one teacher. Some gains are substantial: with DHRL on IAPR-TC12 at 128 bits, image-to-text mAP increases from 0.701 to 0.766, and text-to-image mAP increases from 0.726 to 0.795. This breadth is meaningful evidence that the training framework can improve the evaluated teacher–student configurations.

The method addresses a plausible tension between supervision quality and coverage. Filtering unreliable teacher signals can reduce useful training information, so supplementing filtered distillation with an intra-student objective is well motivated. Tables 2 and 3 provide supporting ablations. For example, removing intra-student contrastive learning reduces DHRL-based image-to-text mAP on IAPR-TC12 at 128 bits from 0.766 to 0.680. This supports the importance of maintaining a complementary learning signal, although it does not independently establish the novelty of the proposed filtering.

Critical weaknesses

The incremental contribution over closely related distillation methods is insufficiently established. BRCD already combines contrastive hashing distillation, clustering-based handling of unreliable positive and false-negative relations, and bit masks derived from cluster-wise bit statistics. Its original setting is image hashing, so RDDH’s cross-modal extension and two-granularity weighting are potentially meaningful additions. However, their specific benefit needs to be isolated. Table 1 compares RDDH students with their teachers, rather than comparing alternative distillation methods using the same teacher and student. Tables 2 and 3 remove relatively large components, but do not isolate the second clustering granularity, soft weighting, or the two relation filters. Consequently, the results establish improvement over the selected teachers but leave the advance over existing robust distillation unclear. A controlled cross-modal adaptation of BRCD would be more informative than directly comparing numbers from its original task.

The claimed connection between the reliability criterion and semantic information is not justified. Appendix A, Proposition A.3, claims that selecting highly polarized, low-entropy bits maximizes a mutual-information lower bound. However, a bit that is constantly +1 across all samples has maximal polarization and zero within-cluster entropy, while its marginal entropy and mutual information with cluster labels are both zero. Thus, the criterion does not ensure the marginal-entropy property asserted in Step 1 of the proof.

Moreover, under Eqs. (7)–(9), the binary entropy is a deterministic, monotonically decreasing function of the absolute cluster mean. Entropy and polarization therefore do not supply independent evidence of reliability in this formulation. This does not invalidate the reported empirical gains, but it undermines the theoretical justification and leaves unresolved whether the mask selects discriminative bits or merely stable ones.

Questions for authors

What evidence isolates the proposed contribution over prior robust distillation? Could the authors provide a comparison with a clearly specified cross-modal adaptation of BRCD, holding the teacher, student, features, code length, and training budget fixed? Targeted ablations of the second clustering granularity and soft weighting would help identify the actual methodological gain.

5 comments
AuthorsOpenReview ↗

Response to Reviewer eQH2 [1/5]

Dear Reviewer eQH2,

We appreciate your thorough review and constructive suggestions concerning comparisons with cross-modal adaptations of BRCD, fine-grained component ablations, and information-theoretic justification of bit reliability in Appendix A. In response, we have conducted controlled benchmark comparisons against BRCD alongside targeted module ablations, and revised Proposition A.3 to incorporate global bit balance and clarify the coupled roles of entropy and polarization. We have addressed your comments in five sections. We will release the corresponding code and experimental logs to support reproducibility.

We would be profoundly grateful if you would consider re-evaluating our work.

Sincerely,

Co-authors

AuthorsOpenReview ↗

Response to Reviewer eQH2 [2/5]

Response to Weakness 1 (Part A) & Questions 1-2: controlled cross-modal comparison with BRCD.

Benchmarking against a controlled cross-modal adaptation of BRCD under identical settings is essential to substantiate our methodological advances.

1. Formulating the controlled cross-modal baseline.

BRCD was proposed strictly for unimodal (image-to-image) hashing, utilizing a single coarse clustering partition to derive binary hard masks and applying single-level relation filtering. To provide a rigorous and informative cross-modal comparison, we implemented two controlled adaptations of BRCD:

  1. Naive BRCD-CM (direct unimodal porting): Strictly mirroring BRCD’s native logic without any student-side multi-modal interaction, i.e., single-granularity hard mask, no FP/FN filtering, and w/o student contrastive learning.
  2. BRCD-CM (+ intra-student contrastive): Enhancing BRCD-CM with our student-side cross-modal contrastive learning objective (Lcontra\mathcal{L}_{contra}) to bridge the multimodal representation gap.

Across all baselines, we strictly maintain:

  • The same teacher models: identical pre-trained DHRL and DJSRH.
  • The same student architectures: identical dual-stream MLP hash heads for image and text.
  • The same features and training budget: standard VGG-19/BoW features under identical training budgets and optimization schedules.

2. The benchmark results on IAPR-TC12 and MIRFlickr are presented (including our targeted ablations under DHRL on IAPR-TC12):

TeacherDatasetCode lengthMetricTeacher baselineNaive BRCD-CM (w/o contra)BRCD-CM (+contra)RDDH (ours)Gain over naive BRCD-CMGain over BRCD-CM (+contra)
DHRLIAPR-TC1264 bitsI2T / T2I0.687 / 0.6970.707 / 0.7290.729 / 0.7470.745 / 0.755+3.8% / +2.6%+1.6% / +0.8%
DHRLIAPR-TC12128 bitsI2T / T2I0.701 / 0.7260.723 / 0.7570.744 / 0.7780.766 / 0.795+4.3% / +3.8%+2.2% / +1.7%
DHRLMIRFlickr128 bitsI2T / T2I0.918 / 0.865—0.933 / 0.8870.948 / 0.904—+1.5% / +1.7%
DJSRHIAPR-TC12128 bitsI2T / T2I0.591 / 0.603—0.657 / 0.6630.666 / 0.678—+0.9% / +1.5%
  1. Directly migrating BRCD without cross-modal student contrast leads to significant performance degradation (dropping by -2.2% / -1.8% on 64-bit and -2.1% / -2.1% on 128-bit compared to BRCD-CM with contrast). This demonstrates that unimodal distillation alone cannot sufficiently bridge the vision-language gap.
  2. Even when BRCD-CM is fortified with our student contrastive objective, RDDH still outperforms it by +2.2% / +1.7% on 128-bit IAPR-TC12. This verifies that our dual-granularity soft weighting and bidirectional symmetric filtering provide substantial methodological advances beyond BRCD’s mask design.
AuthorsOpenReview ↗

Response to Reviewer eQH2 [3/5]

Response to Weakness 1 (Part B) & Question 3: targeted fine-grained component ablations.

To independently substantiate the role of each individual module, we conducted a fine-grained ablation study under DHRL at 128 bits on IAPR-TC12 (full RDDH: I2T = 0.766, T2I = 0.795):

Ablation variantIsolated design / ModificationI2TT2IPerformance drop (Δ\Delta)
Full RDDH (Ours)Complete dual-granularity soft weighting + bidirectional FP/FN filtering + student contrast
0.7660.795—
w/o 2nd granularity (MaskC)Uses only coarse Elbow clustering (MaskE), removing label-cardinality clustering
0.7410.779-2.5% / -1.6%
Hard binary mask (BRCD-style)Replaces continuous soft weights in MaskC with binary {0,1}\{0, 1\} hard masks
0.7520.784-1.4% / -1.1%
RDDH-R (w/o any bit mask)Removes bit-level reliability weighting completely
0.7300.789-3.6% / -0.6%
RDDH-B (w/o bidirectional refinement)Removes bidirectional cross-modal FP/FN relation filtering (Eq. 18)
0.7520.784-1.4% / -1.1%
RDDH-C (w/o intra-student contrast)Removes student-side cross-modal contrastive learning (Eq. 3)
0.6800.720-8.6% / -7.5%
  1. Second clustering granularity (MaskC): disabling label-guided fine-grained clustering drops I2T mAP by -2.5%, demonstrating that coarse elbow clustering alone merges overlapping multi-label sub-categories, whereas the second granularity recovers subtle semantic distinctions.
  2. Continuous soft weighting vs. hard masking: binarizing continuous weights into {0,1}\{0, 1\} masks decreases I2T/T2I by -1.4% / -1.1%. Continuous soft weighting retains valuable gradient supervision from moderately reliable bits.
  3. Bidirectional FP/FN relation filtering: removing bidirectional refinement drops I2T/T2I by -1.4% / -1.1%, confirming that asymmetric false positives and false negatives between vision and text must be filtered in both directions.
  4. Intra-student contrastive objective: removing student-side contrast causes the largest drop (-8.6% / -7.5%), confirming that when unreliable teacher signals are masked out, internal cross-modal contrast acts as the essential anchor preventing semantic collapse.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.