RICH: Preserving Intra-Class Semantic Order in Binary Codes Learned from Coarse Labels

Deep hashing compresses images into short binary codes for large-scale retrieval. Its dominant paradigm, hash-center methods, trains every sample of a class toward one binary center, so the code encodes the class label but not the ordering among same-class items: kayaks and sailboats both labeled watercraft collapse onto the same center. That ordering decides what a user sees, since with thousands of same-class items the top-100 of a correct-class query is fixed by it, and class-level mean average precision (mAP) does not measure it. We study this ordering in the coarse-label setting, where training uses only superclass labels and subclass labels are withheld for evaluation, formalize it as a retrieval-quality dimension of its own, and propose two metrics for it: Intra-class Spearman $\rho$ and Semantic Recall@$k$. Keeping the ordering in a binary code is hard because the two objectives compete for a fixed bit budget and encoding both in one bit space causes interference that more bits do not remove. Our proposed Residual-Informed Center Hashing (RICH) resolves the conflict by splitting the $N$ bits between a center head trained on the superclass label and a residual head trained on sub-clusters found by $K$-means in the pretrained embedding, with the number of sub-clusters set automatically per dataset from the per-class sample count and the codebook capacity. In two-stage retrieval, the center bits rank by superclass and the residual bits refine the order within it. On CIFAR-100, ImageNet-S20, and ImageNet-S50, RICH outperforms the second-best baseline on all intra-class metrics at every bit width, reaching $1.7\sim 4.5\times$ its subclass mAP@100, scored against held-out subclass labels, and $1.2\sim 2.2\times$ its intra-class $\rho$, scored against the pretrained partition. It also leads on all nine inter-class settings when comparing its center bits with the baselines' full codes at the same width, because the sub-cluster loss sharpens the shared backbone that the center head reads. Code is available at https://anonymous.4open.science/r/RICH-286C.