- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RICH addresses the limitation of image hashing with coarse labels. It splits a fixed bit budget between center bits learned from coarse labels and residual bits learned from clusters of frozen features. A graphic Hamming ranking gives coarse-class similarity priority over residual similarity. The experiments examine both coarse retrieval and within-class ordering across datasets and bit budgets, including relevance defined by held-out subclass labels.
Critical strengths
-
The distinction between class retrieval and within-class ordering are useful. A method can achieve good class-level mAP while returning a poorly ordered list within each class. Evaluating this failure with withheld subclass labels gives the paper a contribution beyond its proposed hashing loss.
-
RICH keeps inference in a compact binary representation. The comparisons at equal total bit budgets, pretrained-feature and ITQ alternatives, subclass P@100, and tests on additional datasets and backbones support the usefulness of preserving pretrained structure. The size of the headline gain still depends on the evaluation issues below.
Critical weaknesses
-
The evaluation protocol makes the headline improvements difficult to interpret. Appendix H selects checkpoints on the final query set, uses an additional within-class criterion for RICH, and describes the results as best-epoch estimates. An independent validation set is needed to establish held-out performance. Subclass AP@100 raises a separate concern. Its denominator is the number of relevant items actually retrieved in the top 100, and queries with no hits are excluded. This can favor lists with few relevant hits and give different methods different evaluated query populations. P@100 provides useful additional evidence, but does not settle the comparability of the AP results.
-
It remains unclear how much the code split and ranking rule add beyond pseudo-fine supervision. For the stated linear heads operating on the same features, stacking their weights reproduces their outputs before binarization. This observation concerns output parameterization; it does not make the losses, codebooks, optimization or ranking rules equivalent. The role of separating supervision and bit allocation therefore needs a matched empirical control. Appendix N replaces coarse labels with pseudo-fine labels; it does not test a shared code trained with both objectives. Reranking exactly 100 candidates also leaves open the standard alternative of retrieving a larger set before reranking.
Questions for authors
Can you report subclass AP@100 over all queries, giving zero to queries with no hits and fixing the relevance denominator independently of the returned list, for example min(100, the number of relevant database items)? Please also report query coverage and confirm which queries enter P@100.
2 comments
Response to Reviewer MJaa (Part 2/2)
W2 - Stacked Heads
We agree that the two heads form one linear map of the backbone feature with partitioned outputs. RICH gives each partition its own loss and codebook (now stated in Section 3.3) and its own ranking role (Section 3.7).
W2 - Shared Code Trained with Both Objectives
Partitioning the bits lets both objectives be met. The single-head control (Appendix T) trains one Linear(512→N) head with both losses on the same N bits, matching the stacked heads' output parameters except the partition.
| 32 bits | mAP@all (single-head / RICH) | Sub mAP@100 (single-head / RICH) |
|---|---|---|
| CIFAR-100 | 55.2 / 61.8 | 43.3 / 46.5 |
| ImageNet-S20 | 65.3 / 84.6 | 58.7 / 66.1 |
| ImageNet-S50 | 55.7 / 59.7 | 41.0 / 52.3 |
At matched total length, the control uses all N bits for inter-class retrieval (RICH only its N/2 center bits), yet its mAP@all drops by 2.3–24.6%p in all nine settings. At 64 bits it matches or exceeds RICH on several intra-class metrics, but only by losing the most inter-class mAP (62.5 vs. 87.0 on ImageNet-S20). Sweeping its sub-cluster loss weight α ∈ {0.5, 0.1, 0.01} (27 runs) raises mAP@all, but never to RICH's level at matched inter-class bit budget, and drops subclass mAP@100 to 13.0–22.5% for α ≤ 0.1 in eight of nine settings. No α matches RICH on both, so loss weighting is not the cause (Tables 22 and 23 in Appendix T).
W2 - Ranking Rule
The ranking rule adds no intra-class gain of its own. Single-pass ranking of RICH's full code matches or exceeds two-stage subclass mAP@100 in eight of nine settings; dropping the residual bits lowers it to 12–22% (Table 2). The two-stage rule protects inter-class ranking: single-pass with the full code loses 5.7–31.6%p of mAP@all.
W2 - Re-Ranking a Larger Candidate Set
RICH ranks first in all nine settings at both depths. We re-ranked every method's Hamming top-500 and top-1000 with the same pretrained features, so any difference reflects each code's candidates.
| 32 bits, Sub mAP@100, all | Top-500 (RICH / best baseline) | Top-1000 (RICH / best baseline) |
|---|---|---|
| CIFAR-100 | 35.9 / 24.2 (SHC) | 35.8 / 31.2 (SHC) |
| ImageNet-S20 | 59.2 / 29.8 (SHC) | 59.3 / 46.2 (SHC) |
| ImageNet-S50 | 46.6 / 23.1 (SHC) | 47.6 / 37.7 (SHC) |
Baselines improve from 500 to 1000 candidates; RICH saturates at 500, so its code places relevant items near the top. The gap narrows with more candidates and closes when the whole predicted class is re-ranked, where over all queries RICH leads in two of nine settings and stays within 2.8%p of the best baseline (Appendix C). Any re-ranking, however, stores a continuous feature per database item (2 KB for a 512-d float embedding vs. 8 B for a 64-bit code); the introduction now limits its claim to a fixed top-100 and states this cost (Table 5 in Appendix C).
Response to Reviewer MJaa (Part 1/2)
W1 - Held-Out Checkpoint Selection
We agree that held-out selection is preferable. It leaves the conclusions unchanged. Originally, every method, baselines included, selected checkpoints on the query set, as is common in public hashing code (e.g., the official CSQ code). We retrained RICH, saving checkpoints every 10 epochs, and used the same rule on a query-set-sized held-out database split, evaluating on the original queries; baselines keep query-set selection, the less favorable condition for RICH. Appendix H now states that Table 1 reports best-epoch, not held-out, results.
RICH still leads every setting by a wide margin. Both rules chose the same checkpoint in six of nine settings (Table 9 in Appendix H), and removing held-out items shifts subclass mAP@100 by at most 0.70%p and its all-query variant by at most 0.25%p (Appendix H). Under query-set selection, retrained runs differ from Table 1 by −4.7 to +1.7%p in subclass mAP@100, beyond the 1.2–3.0%p two-run spread at the same K only for CIFAR-100 at 64 bits and ImageNet-S20 at 32 bits (Appendix H).
W1 - Tie-Breaker
We agree that the tie-breaker is an extra, RICH-only criterion that can favor its intra-class metrics; the implementation details now say so. The tie-breaker (intra-class ρ) uses no subclass labels, so selection never sees the labels behind our main metrics; it acts only on checkpoints within ±1 percentage point of the best mAP@all, including in the held-out experiment. Selecting on mAP@all alone changes the checkpoint only in the three CIFAR-100 runs: the tie-breaker raises subclass mAP@100 at 16 and 32 bits (44.6 vs. 42.4, 49.2 vs. 47.4) and lowers it at 64 bits (44.8 vs. 50.6). Either way, RICH stays far above the best baseline.
W1, Q - Subclass AP@100 over All Queries
Over all queries, RICH ranks first in all nine settings (zero for queries without hits; denominator min(100, number of relevant items)). Its ratio to the best baseline grows from 1.7–4.5× to 3.7–29.3×; the absolute gap narrows everywhere except CIFAR-100 at 64 bits, mostly because the all-query variant divides each AP by 100 rather than by the number of relevant items retrieved.
Coverage (queries with at least one same-subclass item in the top 100) is similar across methods (81–94% overall), so the original definition excluded a similar share of each method's queries. Subclass P@100 (Appendix C) was already computed over all queries; recomputation gives identical values in all 54 settings. The Table 4 caption now says so, Table 10 in Appendix H gives full all-query results, and the abstract states RICH leads all nine settings under this definition.