acceptodds
Under review as a conference paper at ICLR 2027

RICH: Preserving Intra-Class Semantic Order in Binary Codes Learned from Coarse Labels

Abstract

Deep hashing compresses images into short binary codes for large-scale retrieval. Its dominant paradigm, hash-center methods, trains every sample of a class toward one binary center, so the code encodes the class label but not the ordering among same-class items: kayaks and sailboats both labeled watercraft collapse onto the same center. That ordering decides what a user sees, since with thousands of same-class items the top-100 of a correct-class query is fixed by it, and class-level mean average precision (mAP) does not measure it. We study this ordering in the coarse-label setting, where training uses only superclass labels and subclass labels are withheld for evaluation, formalize it as a retrieval-quality dimension of its own, and propose two metrics for it: Intra-class Spearman and Semantic Recall@. Keeping the ordering in a binary code is hard because the two objectives compete for a fixed bit budget and encoding both in one bit space causes interference that more bits do not remove. Our proposed Residual-Informed Center Hashing (RICH) resolves the conflict by splitting the bits between a center head trained on the superclass label and a residual head trained on sub-clusters found by -means in the pretrained embedding, with the number of sub-clusters set automatically per dataset from the per-class sample count and the codebook capacity. In two-stage retrieval, the center bits rank by superclass and the residual bits refine the order within it. On CIFAR-100, ImageNet-S20, and ImageNet-S50, RICH outperforms the second-best baseline on all intra-class metrics at every bit width, reaching its subclass mAP@100, scored against held-out subclass labels, and its intra-class , scored against the pretrained partition. It also leads on all nine inter-class settings when comparing its center bits with the baselines' full codes at the same width, because the sub-cluster loss sharpens the shared backbone that the center head reads. Code is available at https://anonymous.4open.science/r/RICH-286C.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
deep hashing, image retrieval, binary code learning, coarse-to-fine representation learning, within-class structure preservation, coarse-label supervision, hierarchical retrieval

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_MJaaOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

RICH addresses the limitation of image hashing with coarse labels. It splits a fixed bit budget between center bits learned from coarse labels and residual bits learned from clusters of frozen features. A graphic Hamming ranking gives coarse-class similarity priority over residual similarity. The experiments examine both coarse retrieval and within-class ordering across datasets and bit budgets, including relevance defined by held-out subclass labels.

Critical strengths
  1. The distinction between class retrieval and within-class ordering are useful. A method can achieve good class-level mAP while returning a poorly ordered list within each class. Evaluating this failure with withheld subclass labels gives the paper a contribution beyond its proposed hashing loss.

  2. RICH keeps inference in a compact binary representation. The comparisons at equal total bit budgets, pretrained-feature and ITQ alternatives, subclass P@100, and tests on additional datasets and backbones support the usefulness of preserving pretrained structure. The size of the headline gain still depends on the evaluation issues below.

Critical weaknesses
  1. The evaluation protocol makes the headline improvements difficult to interpret. Appendix H selects checkpoints on the final query set, uses an additional within-class criterion for RICH, and describes the results as best-epoch estimates. An independent validation set is needed to establish held-out performance. Subclass AP@100 raises a separate concern. Its denominator is the number of relevant items actually retrieved in the top 100, and queries with no hits are excluded. This can favor lists with few relevant hits and give different methods different evaluated query populations. P@100 provides useful additional evidence, but does not settle the comparability of the AP results.

  2. It remains unclear how much the code split and ranking rule add beyond pseudo-fine supervision. For the stated linear heads operating on the same features, stacking their weights reproduces their outputs before binarization. This observation concerns output parameterization; it does not make the losses, codebooks, optimization or ranking rules equivalent. The role of separating supervision and bit allocation therefore needs a matched empirical control. Appendix N replaces coarse labels with pseudo-fine labels; it does not test a shared code trained with both objectives. Reranking exactly 100 candidates also leaves open the standard alternative of retrieving a larger set before reranking.

Questions for authors

Can you report subclass AP@100 over all queries, giving zero to queries with no hits and fixing the relevance denominator independently of the returned list, for example min(100, the number of relevant database items)? Please also report query coverage and confirm which queries enter P@100.

2 comments
AuthorsOpenReview ↗

Response to Reviewer MJaa (Part 1/2)

W1 - Held-Out Checkpoint Selection

We agree that held-out selection is preferable. It leaves the conclusions unchanged. Originally, every method, baselines included, selected checkpoints on the query set, as is common in public hashing code (e.g., the official CSQ code). We retrained RICH, saving checkpoints every 10 epochs, and used the same rule on a query-set-sized held-out database split, evaluating on the original queries; baselines keep query-set selection, the less favorable condition for RICH. Appendix H now states that Table 1 reports best-epoch, not held-out, results.

Sub mAP@100RICH, held-out selection (16 / 32 / 64 bits)Best baseline, Table 1 (16 / 32 / 64 bits)
CIFAR-10044.6 / 49.2 / 44.819.7 / 21.8 / 29.6
ImageNet-S2057.8 / 63.1 / 66.013.9 / 14.8 / 18.5
ImageNet-S5048.5 / 53.0 / 56.413.1 / 13.7 / 18.5

RICH still leads every setting by a wide margin. Both rules chose the same checkpoint in six of nine settings (Table 9 in Appendix H), and removing held-out items shifts subclass mAP@100 by at most 0.70%p and its all-query variant by at most 0.25%p (Appendix H). Under query-set selection, retrained runs differ from Table 1 by −4.7 to +1.7%p in subclass mAP@100, beyond the 1.2–3.0%p two-run spread at the same K only for CIFAR-100 at 64 bits and ImageNet-S20 at 32 bits (Appendix H).

W1 - Tie-Breaker

We agree that the tie-breaker is an extra, RICH-only criterion that can favor its intra-class metrics; the implementation details now say so. The tie-breaker (intra-class ρ) uses no subclass labels, so selection never sees the labels behind our main metrics; it acts only on checkpoints within ±1 percentage point of the best mAP@all, including in the held-out experiment. Selecting on mAP@all alone changes the checkpoint only in the three CIFAR-100 runs: the tie-breaker raises subclass mAP@100 at 16 and 32 bits (44.6 vs. 42.4, 49.2 vs. 47.4) and lowers it at 64 bits (44.8 vs. 50.6). Either way, RICH stays far above the best baseline.

W1, Q - Subclass AP@100 over All Queries

Over all queries, RICH ranks first in all nine settings (zero for queries without hits; denominator min(100, number of relevant items)). Its ratio to the best baseline grows from 1.7–4.5× to 3.7–29.3×; the absolute gap narrows everywhere except CIFAR-100 at 64 bits, mostly because the all-query variant divides each AP by 100 rather than by the number of relevant items retrieved.

32 bitsRICH (paper / all queries)Best baseline (paper / all queries)Coverage (RICH / baselines)
CIFAR-10046.5 / 28.021.8 / 4.2 (SHC)93.9% / 90.8–91.9%
ImageNet-S2066.1 / 48.814.8 / 1.7 (SHC)93.2% / 91.2–92.9%
ImageNet-S5052.3 / 31.613.7 / 1.2 (SHC)87.4% / 83.0–84.8%

Coverage (queries with at least one same-subclass item in the top 100) is similar across methods (81–94% overall), so the original definition excluded a similar share of each method's queries. Subclass P@100 (Appendix C) was already computed over all queries; recomputation gives identical values in all 54 settings. The Table 4 caption now says so, Table 10 in Appendix H gives full all-query results, and the abstract states RICH leads all nine settings under this definition.

AuthorsOpenReview ↗

Response to Reviewer MJaa (Part 2/2)

W2 - Stacked Heads

We agree that the two heads form one linear map of the backbone feature with partitioned outputs. RICH gives each partition its own loss and codebook (now stated in Section 3.3) and its own ranking role (Section 3.7).

W2 - Shared Code Trained with Both Objectives

Partitioning the bits lets both objectives be met. The single-head control (Appendix T) trains one Linear(512→N) head with both losses on the same N bits, matching the stacked heads' output parameters except the partition.

32 bitsmAP@all (single-head / RICH)Sub mAP@100 (single-head / RICH)
CIFAR-10055.2 / 61.843.3 / 46.5
ImageNet-S2065.3 / 84.658.7 / 66.1
ImageNet-S5055.7 / 59.741.0 / 52.3

At matched total length, the control uses all N bits for inter-class retrieval (RICH only its N/2 center bits), yet its mAP@all drops by 2.3–24.6%p in all nine settings. At 64 bits it matches or exceeds RICH on several intra-class metrics, but only by losing the most inter-class mAP (62.5 vs. 87.0 on ImageNet-S20). Sweeping its sub-cluster loss weight α ∈ {0.5, 0.1, 0.01} (27 runs) raises mAP@all, but never to RICH's level at matched inter-class bit budget, and drops subclass mAP@100 to 13.0–22.5% for α ≤ 0.1 in eight of nine settings. No α matches RICH on both, so loss weighting is not the cause (Tables 22 and 23 in Appendix T).

W2 - Ranking Rule

The ranking rule adds no intra-class gain of its own. Single-pass ranking of RICH's full code matches or exceeds two-stage subclass mAP@100 in eight of nine settings; dropping the residual bits lowers it to 12–22% (Table 2). The two-stage rule protects inter-class ranking: single-pass with the full code loses 5.7–31.6%p of mAP@all.

W2 - Re-Ranking a Larger Candidate Set

RICH ranks first in all nine settings at both depths. We re-ranked every method's Hamming top-500 and top-1000 with the same pretrained features, so any difference reflects each code's candidates.

32 bits, Sub mAP@100, allTop-500 (RICH / best baseline)Top-1000 (RICH / best baseline)
CIFAR-10035.9 / 24.2 (SHC)35.8 / 31.2 (SHC)
ImageNet-S2059.2 / 29.8 (SHC)59.3 / 46.2 (SHC)
ImageNet-S5046.6 / 23.1 (SHC)47.6 / 37.7 (SHC)

Baselines improve from 500 to 1000 candidates; RICH saturates at 500, so its code places relevant items near the top. The gap narrows with more candidates and closes when the whole predicted class is re-ranked, where over all queries RICH leads in two of nine settings and stays within 2.8%p of the best baseline (Appendix C). Any re-ranking, however, stores a continuous feature per database item (2 KB for a 512-d float embedding vs. 8 B for a 64-bit code); the introduction now limits its claim to a fixed top-100 and states this cost (Table 5 in Appendix C).

Reviewer_uTKdOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses intra‑class semantic ordering degradation for deep supervised hashing trained only on coarse super‑class labels, where existing hash‑center baselines collapse all samples within one super‑class toward a single binary center and ignore fine‑grained subclass structure. The authors define two new evaluation metrics for intra‑class retrieval quality and propose RICH, a dual‑head hashing architecture that splits total bits into independent center head (for inter‑superclass discrimination) and residual head (for intra‑class fine‑grained structure, supervised by K‑means pseudo sub‑clusters from pretrained embedding without fine‑grained annotations). A two‑stage lexicographic retrieval is proposed to avoid bit‑space interference between two objectives. Comprehensive experiments on CIFAR‑100, ImageNet‑S20/S50 and additional benchmarks show large improvements on intra‑class metrics while keeping competitive inter‑class mAP when superclasses are visually separable. The method relies heavily on the quality of pretrained embedding and requires hyper‑parameter tuning (α) for datasets with visually similar superclasses. Extensive ablations, theoretical propositions and qualitative analysis are provided in appendices.

Critical strengths
  1. A relevant retrieval problem with a clear evaluation gap. The paper highlights that conventional class-level mAP does not capture semantic ordering within a superclass. Its intra-class metrics and subclass-level retrieval evaluation provide a useful perspective for assessing binary retrieval beyond coarse-label relevance.

  2. Extensive empirical evaluation. The paper includes multiple datasets, bit widths, backbone variants, and ablations, including comparisons against unsupervised ITQ residual codes. The consistent gains in subclass retrieval provide evidence that the proposed learnable residual head is effective relative to the selected baselines.

Critical weaknesses
  1. Overly broad claims about existing hashing paradigms. The paper primarily motivates its contribution through the limitations of hash-center methods, which encourage samples within the same class to converge toward a shared binary center. However, deep hashing encompasses other learning paradigms, including pairwise similarity learning, triplet-based ranking, and listwise ranking objectives (e.g., HashNet and DSRPH), which can preserve relative similarities among samples rather than collapsing them to a single class center. These approaches may already support fine-grained intra-class ordering when pretrained features contain such information. The paper should therefore narrow its claims about the limitations of existing hashing methods and compare RICH against representative ranking-based and similarity-preserving hashing baselines under the same coarse-label supervision and bit budget.

  2. Limited methodological novelty beyond coarse-to-fine reranking. The substantial gains in intra-class retrieval appear to stem primarily from the two-stage retrieval strategy, which first ranks samples using center bits and then refines the ordering using residual bits. This is essentially a coarse-to-fine reranking scheme, a well-established strategy in information retrieval, rather than clear evidence of substantially improved binary representations. Table 3 does not adequately address this concern: ITQ is an unsupervised quantization method, not a competitive fine-grained ranking-based hashing baseline. Showing that a learned residual head outperforms ITQ only establishes the benefit of supervised residual encoding over unsupervised binarization; it does not demonstrate an advantage over existing pairwise-, triplet-, or ranking-based hashing methods. Since reranking can substantially improve retrieval quality, the reported gains may primarily reflect the retrieval protocol rather than a sufficiently novel representation-learning contribution.

Questions for authors
  1. How does RICH compare with ranking-based hashing methods? The current evaluation primarily focuses on hash-center baselines, while pairwise-, triplet-, and ranking-based hashing methods can preserve relative similarities between samples without explicitly collapsing each superclass to a single center. Can the authors clarify why these methods are unsuitable for the coarse-label setting and provide comparisons under the same supervision and bit budget?

  2. How much of the improvement comes from the learned representation versus the two-stage retrieval strategy? Since RICH ranks samples using center bits first and residual bits second, the gains may be substantially influenced by this coarse-to-fine ranking procedure. Can the authors quantify the contribution of the learned residual codes independently of the retrieval strategy, ideally by applying equivalent two-stage or reranking protocols to competitive baselines?

4 comments
AuthorsOpenReview ↗

Response to Reviewer uTKd (Part 1/2)

W1 - Scope of Claims

Scope of claims. We agree and narrowed them. The introduction now refers only to hash-center codes, and Related Work states that pairwise and ranking-based hashing take relevance from supervision, so with one coarse label per image and no captions all same-class pairs are equally relevant (Sections 1 and 2).

W1, Q1 - Ranking-Based Hashing under Coarse Labels

Under one coarse label, these methods have no signal that orders items within a class. Pairwise and triplet losses treat all same-class pairs as equally similar; DSRH grades relevance by shared-label count, which reduces to two levels. DSRPH adds caption cosine similarity to label overlap; without captions, it too reduces to two levels. We found no public implementation of DSRPH and did not include it.

HashNet. To test whether pretrained features supply within-class order, we trained HashNet with the same supervision, backbone, bit budget, and checkpoint rule as the Table 1 baselines and, for consistency, their common training loop rather than its original optimizer, with a loss matching the official implementation. With two levels, DSRH's weighted triplet loss sees the same binary similarity, so HashNet stands for this family under our supervision.

32 bitsmAP@all (HashNet / RICH)Sub mAP@100 (HashNet / RICH)
CIFAR-10063.5 / 61.821.1 / 46.5
ImageNet-S2084.1 / 84.614.0 / 66.1
ImageNet-S5048.2 / 59.712.5 / 52.3

HashNet's subclass mAP@100 (10.4–21.1) lies within or below that of the Table 1 hash-center baselines (12.8–29.6), and its ρ (0.06–0.35) overlaps theirs (0.04–0.32). RICH's center bits lead HashNet in inter-class mAP@all in all nine settings at matched inter-class bit budget, and in all but CIFAR-100 at 32 bits (61.8 vs. 63.5) at matched total length (Table 22 in Appendix T).

DSRH-sub. We also trained DSRH with RICH's sub-cluster labels as a second label, giving three relevance levels (2: same sub-cluster, 1: same superclass, 0: otherwise). Its loss matches the official implementation; batches of 8 superclasses × 4 sub-clusters × 4 images guarantee same-sub-cluster pairs per batch, a choice that favors DSRH-sub.

32 bitsmAP@all (DSRH-sub / RICH)Sub mAP@100 (DSRH-sub / RICH)
CIFAR-10063.3 / 61.837.3 / 46.5
ImageNet-S2081.7 / 84.624.5 / 66.1
ImageNet-S5045.4 / 59.721.6 / 52.3

RICH leads DSRH-sub on subclass mAP@100 in all nine settings and on inter-class mAP@all in eight of nine at matched total length (except CIFAR-100 at 32 bits, 63.3 vs. 61.8). DSRH-sub trails HashNet on inter-class mAP@all in every setting (by 0.2–7.5%p): adding the sub-cluster signal to the same bits costs inter-class retrieval. It comes closest to RICH at 64 bits on CIFAR-100, where its subclass mAP@100 approaches RICH's (48.8 vs. 49.5) and its SR@100 is higher (52.51 vs. 49.65). At 16 bits its subclass mAP@100 is 10.7–17.1% versus 43.6–54.7% for RICH (Table 22). We did not run a listwise method: listwise objectives rank graded relevance, and sub-cluster pseudo-labels give only the three levels DSRH-sub already uses.

AuthorsOpenReview ↗

Response to Reviewer uTKd (Part 2/2)

W2 - Residual Code versus Retrieval Strategy

The gain comes from the learned residual code, not the retrieval rule (Table 2). Removing the residual bits (center-only) drops subclass mAP@100 to 12–22%, while single-pass ranking with the full code matches or exceeds two-stage retrieval on this metric in eight of nine settings.

32 bits, Sub mAP@100Center-onlyTwo-stage (ours)Full code
CIFAR-10019.546.549.0
ImageNet-S2013.966.167.4
ImageNet-S5013.152.354.5

Two-stage retrieval protects inter-class ranking: single-pass full-code retrieval loses 5.7–31.6%p of mAP@all. We agree that coarse-to-fine search is well established; Contribution 3 now says so and claims only its use after code splitting. RICH also leads the pairwise and ranking-based HashNet and DSRH-sub on subclass mAP@100 in every setting (Part 1/2, Table 22).

A stronger control than ITQ. We agree that ITQ is a weak control. Under two-stage retrieval, we paired each baseline's superclass code (N/2 bits) with the same baseline trained on our sub-cluster labels (N/2 bits), on CIFAR-100 at 32 and 64 total bits, where Appendix N has sub-cluster models.

CIFAR-100, 64 bitsmAP@allSub mAP@100SR@100Backbones
CSQ + CSQ_sub64.046.741.02
CRH + CRH_sub60.849.346.72
RICH67.149.549.71

At both widths, RICH beats every combination on mAP@all, subclass mAP@100 (both definitions), and SR@100 with one backbone and one forward pass instead of two; CRH + CRH_sub has the highest intra-class ρ (0.365/0.379 vs. 0.257/0.264 at 32/64 bits). CRH, best intra-class, loses 6.3%p of mAP@all at 64 bits; CSQ, best inter-class, is weaker intra-class. This supports splitting the code and supervising the residual space with sub-clusters, which RICH does in one shared model (Table 16 in Appendix N).

Q2 - Two-Stage Protocol for the Baselines

Table 2 separates the residual code from the retrieval rule, and the CSQ/CRH pairs above run baselines with learned residual codes under the same two-stage protocol. Whether the baselines use two-stage retrieval with ITQ residual bits (Appendix A) or re-rank their top-100 with continuous features (Appendix C), RICH leads on subclass mAP@100 in every configuration. Re-ranking the whole predicted class with continuous features closes the gap: RICH leads in three of nine settings and trails the best baseline by at most 0.8%p under the paper definition, and leads in two and trails by at most 2.8%p over all queries (Appendix C).

Thank you for the detailed rebuttal and additional experiments. My main reservation is that the method largely combines existing techniques, and the gains over baselines with matched pseudo-label supervision and two-stage retrieval are relatively modest. Still, preserving intra-class structure under coarse-label supervision is a useful problem for binary retrieval. The two-stage design is reasonable and avoids the cost of dense feature reranking, while achieving good overall results. The additional experiments have addressed some of my concerns, so I have decided to raise my score from 2 to 3.

Thank you for your time, the constructive discussion, and for raising your score. We will reflect the revisions in the final version.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response

We thank all reviewers for their constructive feedback. Revisions are in blue. Below we summarize the major changes; each response reports the results relevant to that reviewer.

TL;DR

  • HashNet, DSRH-sub, two-stage baseline combinations, held-out checkpoint selection, AP over all queries, and top-500/1000 re-ranking all leave RICH ahead in subclass mAP@100.
  • A single-head control with the same backbone and total bits loses inter-class mAP@all in all nine settings; at 64 bits it matches or exceeds RICH on several intra-class metrics, and no α matches RICH on both.
  • Main cases where RICH does not lead: at matched total code length it trails the best baseline at 16 and 32 bits (by up to 3.3%p) and in all six CIFAR68 and CIFAR-SI settings, so the "not a trade-off" claim is removed; re-ranking the whole predicted class removes RICH's lead in most settings; on ImageNet, ρ against CLIP and NDCG cannot separate order from alignment with the backbone. Each thread reports the other cases.

Summary of changes

New experiments

  • A single-head control that trains both objectives on one shared code with the same backbone and total bits, also with the sub-cluster loss down-weighted (α ∈ {0.5, 0.1, 0.01}, 27 runs); results as in the TL;DR [TQwm/MJaa, App. T]
  • HashNet, a pairwise method, under the baselines' protocol [uTKd/TQwm, App. T]
  • DSRH with sub-cluster pseudo-labels (DSRH-sub), a ranking-based method with a weighted triplet loss, under the same protocol. RICH leads in subclass mAP@100 in all nine settings and in inter-class mAP@all in eight of nine at matched total length; DSRH-sub comes close at 64 bits on CIFAR-100 and is higher there in SR@100 [uTKd/TQwm, App. T, Table 22]
  • Two-stage baselines pairing each baseline's superclass code with a learned sub-cluster code [uTKd, App. N]
  • Intra-class order references independent of the training features: ρ against CLIP and graded NDCG from subclass labels [TQwm, App. U]
  • Subclass AP@100 over all queries, with per-method coverage [MJaa, App. H]
  • Re-ranking of the top-500 and top-1000 Hamming candidates: RICH ranks first in all nine settings at both depths. The gap narrows as the candidate set grows and closes when the whole predicted class is re-ranked, which requires storing continuous features for the whole database [MJaa/uTKd, App. C]
  • Checkpoint selection on a held-out database split, and re-evaluation without the held-out items: subclass mAP@100 changes by at most 0.70%p, the all-query variant by at most 0.25%p [MJaa, App. H]
  • A sweep of K beyond 2C on CIFAR-100 and on ImageNet-S20, the only dataset where the cap binds (K = 60, 80, Appendix G). No gain in subclass mAP@100 is consistent across code lengths, a smaller K scores higher on ImageNet-S20, and K is fixed by a label-free rule, not tuned on these results [TQwm, App. G]
  • Sub-cluster stability across 90 superclasses [TQwm, App. P]

Revisions to the text

  • Claims about existing methods now refer to hash-center methods and state the boundary with multi-center and ranking-based hashing [TQwm/uTKd, §1, §2].
  • Section 4.1 states each metric's role: subclass mAP@100 is the primary evidence, ρ and SR are diagnostics [TQwm, §4.1].
  • The inter-class claim in the abstract, introduction, and conclusion now specifies the bit budget and leads with the result at matched total code length [TQwm, §1, §5].
  • The claim that splitting the code is not a trade-off is removed; CIFAR68 and CIFAR-SI are counterexamples [TQwm, §1, §4.1, §5, App. L].
  • The introduction's re-ranking statement is restricted to a fixed top-100 [MJaa, §1].
  • New Appendices T and U and extended Appendices C, G, H, N, and P report the experiments above [all reviewers, App. T, App. U].

Not run: DSTH itself (multi-label cross-modal; Appendix T tests its single-space design), a listwise method (with sub-cluster labels, the only relevance grades are the three levels that DSRH-sub already uses), DSRPH (needs image captions; no public implementation found), and two-stage baseline combinations at 16 bits (no sub-cluster baseline models at that width).

Shared concerns

  • Can one bit space hold both objectives (TQwm, MJaa)? The single-head control above.
  • Does ranking-based hashing already order items within a class (TQwm, uTKd)? HashNet and DSRH-sub above.
  • Does the gain come from the code or from the retrieval rule (uTKd, MJaa)? Table 2 and the re-ranking experiments above.

Pending results: None

AuthorsOpenReview ↗

Response to Reviewer TQwm (Part 1/3)

W1, Q1 - Novelty

Novelty. We agree the claim holds only for the combination of coarse labels, binary codes, intra-class ordering, and the proposed metrics. The introduction now reads, "To the best of our knowledge, it is the first supervised hashing method that learns an intra-class ordering from superclass labels and sub-clusters of a pretrained embedding, without subclass labels." Each cited work covers part of it but not intra-class ordering: DSTH adds centers per class without an ordering objective (below); SHC spaces its one center per class by inter-class semantic similarity, yet trains every sample toward its class center and stays at 12.9–21.8 subclass mAP@100 (Table 1); DSRPH ranks by label overlap and caption similarity, so without captions its relevance has two levels. We corrected Related Work on SHC: "SHC reflects inter-class semantic similarity in the codebook." Baselines and Related Work now state these boundaries, and Appendix T reports the single-head control, HashNet, and DSRH with sub-cluster labels under the same protocol. RICH leads HashNet and DSRH-sub on subclass mAP@100 in every setting.

W1 - DSTH Boundary

The boundary is structural. DSTH, a multi-label cross-modal method, learns several centers per class from class labels in one bit space, so the same bits must separate classes and encode within-class variation, with no intra-class ordering objective. RICH assigns the two roles to disjoint bits. Its cross-modal multi-label setting does not transfer to ours without redesign, so we did not reimplement it and tested its single-space design in two cases.

Without sub-cluster supervision, the objective has no intra-class ordering signal; methods trained only for class discrimination (hash-center baselines, HashNet) stay at 10–30 subclass mAP@100 (Table 1, Table 22).

Q1 - Single-Head Control

With sub-cluster supervision, the single-head control requested in Q1 trains one N-bit head with our center and sub-cluster losses and RICH's backbone and total bits, so it places several centers per class in one bit space, a single-space analogue of DSTH's design.

32 bitsmAP@all (single-head / RICH)Sub mAP@100 (single-head / RICH)
CIFAR-10055.2 / 61.843.3 / 46.5
ImageNet-S2065.3 / 84.658.7 / 66.1
ImageNet-S5055.7 / 59.741.0 / 52.3

At matched total code length, the control uses all N bits for inter-class retrieval and RICH only its N/2 center bits, yet its inter-class mAP@all drops in all nine settings, by 2.3–24.6%p, growing with code length on ImageNet (13.4, 19.3, and 24.6%p on ImageNet-S20). Since the control keeps the superclass loss, this drop, given sub-cluster supervision, comes from sharing one bit space; the larger collapse in Appendix N also reflects removed superclass labels, which those baselines replace with sub-cluster labels. At 64 bits it matches or exceeds RICH on several intra-class metrics only by losing the most inter-class mAP (62.5 vs. 87.0 on ImageNet-S20). Sweeping its sub-cluster loss weight α ∈ {0.5, 0.1, 0.01} (27 runs) raises its inter-class mAP@all, but never to RICH's level at matched inter-class bit budget, and drops its subclass mAP@100 to 13.0–22.5% for α ≤ 0.1 in eight of the nine settings. No α matches RICH on both (Tables 22 and 23 in Appendix T). In one bit space, intra-class order thus costs inter-class retrieval.

AuthorsOpenReview ↗

Response to Reviewer TQwm (Part 2/3)

W2, Q2 - Circularity of ρ and the Role of SR@100

We agree that ρ against the backbone alone cannot rule out imitation of it. We added both references; among the methods of Table 1, RICH ranks first on each in all nine settings (Table 24). On ImageNet, the pretrained features and their direct binarizations score higher (below).

32 bitsρ against CLIP (RICH / best baseline)Graded NDCG@100 (RICH / best baseline)
CIFAR-1000.56 / 0.31 (SHC)50.1 / 34.3 (SHC)
ImageNet-S200.51 / 0.38 (SHC)68.3 / 35.8 (SHC)
ImageNet-S500.53 / 0.41 (SHC)51.4 / 28.1 (SHC)

(a) ρ against CLIP recomputes the sub-clusters, assignments, and reference distances from CLIP ViT-B/16 features, never used in training. (b) Graded NDCG@100 uses only human subclass labels (relevance 2 for same subclass, 1 for same superclass, 0 otherwise); RICH reaches 45.9–68.3 vs. 27.3–39.5 for the best baseline.

The two ρ variants share one definition but play different roles. ρ against the backbone measures how much of its intra-class structure, the source of our sub-cluster supervision, a code retains, so the backbone's direct binarizations (Sign and ITQ, Table 3) score highest on it by construction while retrieving worse: lower subclass mAP@100 in every setting except ITQ on ImageNet-S20 at 64 bits. ρ against CLIP checks that the order is not backbone-specific. SR@100 compares each model's continuous features with its own codes, not with the backbone. None replaces subclass mAP@100, now the primary evidence in Section 4.1 and the only intra-class metric reported in the abstract.

On ImageNet, the references cannot separate order from alignment with the backbone. The ImageNet-S20 and ImageNet-S50 subclasses are ImageNet classes used to pretrain the backbone, and the two references agree. Sign and ITQ exceed RICH in ρ against CLIP at every code length (0.591–0.664 vs. 0.473–0.558); the continuous features exceed it in NDCG@100 (76.97 vs. at most 68.33 on ImageNet-S20, 66.31 vs. at most 54.83 on ImageNet-S50), as does ITQ at 64 bits on ImageNet-S20 (69.02 vs. 67.98, Table 24).

On CIFAR-100, whose subclass labels are independent of pretraining, RICH exceeds the backbone's direct binarizations. It beats Sign and ITQ in ρ against CLIP at every code length and in subclass mAP@100 (46.5 vs. 32.7 for ITQ at 32 bits), and the continuous features in NDCG@100 at 32 and 64 bits (50.1 and 53.6 vs. 48.2), though not at 16 bits (47.6). RICH's contribution is transferring this ordering into binary codes that also separate classes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.