- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Structured Semantic Transfer (SST), a zero-shot image hashing framework built upon a PZSH-based generative retrieval pipeline. The main idea is to transfer semantic relationships into both visual representations and learnable hash centers, rather than relying primarily on holistic feature alignment. SST introduces two components. First, patch-level relational distillation employs learnable attribute queries to aggregate patch tokens and aligns pairwise similarities of the resulting embeddings with class-attribute similarities. Second, attribute geometry alignment constrains the cosine similarities between unseen and seen class centers using attribute-presence information. The authors also provide a theoretical bound relating continuous center geometry to binary Hamming similarities under additional assumptions. Experiments on AWA2, CUB, and SUN, across four hash lengths, demonstrate consistent improvements over the authors' PZSH-based implementation. Ablation studies, selected multi-seed experiments, and alternative design controls further support the effectiveness of the proposed objectives.
Critical strengths
- A coherent and practically motivated approach to structured semantic transfer. The two proposed objectives address complementary aspects of zero-shot hashing: transferring class-attribute relationships into patch-derived visual representations and imposing semantic structure on learnable hash centers. Their integration into a shared visual encoder and hash-learning pipeline is conceptually coherent. The distinction between batch-level representation relations and cross-class center geometry provides a reasonable motivation for the overall framework.
- Consistent empirical improvements supported by meaningful ablations. SST improves over the authors' internal baseline across all twelve dataset–code-length configurations. Both transfer objectives contribute positively, and the full model generally performs better than either single-component variant. The additional multi-seed results in Appendix A.10 and the design controls in Appendix A.11 strengthen the empirical evidence. I also appreciate the authors' explicit discussion of evaluation-protocol differences and the limitations of their theoretical analysis.
Critical weaknesses
- The comparison against PZSH is not sufficiently controlled to support the broader improvement claim. The main concern is that the reported PZSH baseline is not a faithful reproduction of the original PZSH optimization objective. The original PZSH method uses center-based cosine classification, quantization, and proxy losses, whereas the current paper adopts learnable continuous centers supervised through MSE and momentum consistency. Consequently, removing the two proposed transfer losses does not recover the original PZSH method. While the ablations convincingly demonstrate improvements over this modified implementation, they do not establish whether the proposed objectives improve the original PZSH framework under matched conditions. Furthermore, the paper reports mAP@5000, while published PZSH results use mAP@all, and some imported comparisons use different class partitions. The authors acknowledge these differences, but they substantially limit conclusions about competitiveness against established methods. This issue is particularly consequential because PZSH is the primary foundation and strongest directly controlled baseline of the paper.
- The methodological novelty and the evidence for the intended semantic mechanisms remain somewhat limited. Previous zero-shot hashing studies, particularly COMAE (ICMR 2025), RAZH (2025), and DA2H (2026), have already investigated attribute-related local representations and visual-semantic alignment. SHC (TOIS 2025) also considers semantically structured hash centers, although in a different setting. The specific relational objectives proposed here differ from these methods, but their broader motivation substantially overlaps with prior work. Moreover, the patch-level objective matches image-level similarity matrices without directly supervising individual attribute coordinates or spatial correspondence. Consequently, it does not establish that the learned queries identify meaningful attribute-specific regions. The authors appropriately acknowledge this limitation, but additional mechanistic evidence would be valuable. Similarly, the geometry objective constrains continuous unseen–seen center similarities, whereas the principal retrieval task depends on ranking unseen images using binary codes. Theorem 3.3 provides a conditional relationship between these spaces, but it does not directly characterize unseen–unseen retrieval or mAP. The empirical improvements are encouraging, yet the precise role of semantic geometry in improving retrieval discrimination could be more convincingly demonstrated.
Questions for authors
- Could the authors evaluate the two proposed transfer objectives on top of a faithful implementation of the original PZSH framework, retaining its original center construction and hash losses? Reporting the original method, the modified MSE-based implementation, and both variants with the proposed components under identical retrieval protocols would clarify whether the improvement comes from structured semantic transfer rather than other changes to the baseline.
- Could the authors further isolate the contribution of the proposed structural relations? In particular, a semantic-target control using shuffled or mismatched class-attribute relations would help determine whether the geometry loss benefits from meaningful semantic structure rather than generic center regularization. Likewise, more direct evidence connecting patch-level aggregation to attribute-specific visual information, beyond pairwise similarity agreement, would help distinguish SST from existing attribute-aware hashing approaches.
- Can the authors provide stronger evidence of competitiveness against recent attribute-aware methods such as COMAE and DA2H under matched dataset splits and evaluation metrics? The current literature scores are useful contextual references, but are not sufficient for a controlled state-of-the-art comparison.
3 comments
(2/3)
Critical Weakness 2 — methodological distinction from prior attribute-aware hashing.
We acknowledge prior attribute-aware modeling in COMAE, RAZH, and DA2H, and semantic centers in SHC. SST combines batchwise attribute-relation matching on patch-derived embeddings with unseen–seen attribute-presence matching on learnable centers in generative zero-shot hashing.
Q2. Could a shuffled or mismatched semantic target isolate the structural relations, and could more direct evidence connect patch aggregation to local visual information?
On CUB (training seed 42), we separately test shuffled geometry targets (class–attribute permutations within seen/unseen groups) and class-only patch targets (same/different class = 1/0), keeping other settings fixed. Localization compares visible-part hits in top-attention versus random 20% regions on unseen images, using query–part mappings fixed from seen validation. Masking applies identical 20% attention or random masks to SST and CLS queries; attention masks come from frozen SST, database/checkpoints/fill remain fixed, and random results average ten mask seeds.
Table metrics: Geometry/Masking = mAP@5000; Patch = mAP@5000 / @all; Localization = hit rate (%). Retrieval values use the 0–1 scale.
| Test | Variant | 24 bits | 48 bits | 64 bits | 128 bits |
|---|---|---|---|---|---|
| Geometry | No geometry | 0.1571 | 0.2312 | 0.2351 | 0.3035 |
| Geometry | Shuffled | 0.1441 | 0.2200 | 0.2316 | 0.2756 |
| Geometry | SST | 0.1581 | 0.2362 | 0.2547 | 0.3060 |
| Patch | Class-only | 0.1532 / 0.1527 | 0.2272 / 0.2269 | 0.2477 / 0.2481 | 0.2407 / 0.2405 |
| Patch | Attribute (SST) | 0.1581 / 0.1575 | 0.2362 / 0.2357 | 0.2547 / 0.2541 | 0.3060 / 0.3059 |
| Localization | Attention | 39.96% | 35.88% | 45.17% | 49.92% |
| Localization | Random | 20.13% | 20.10% | 20.03% | 20.13% |
| Masking | SST, unmasked | 0.1581 | 0.2362 | 0.2547 | 0.3060 |
| Masking | SST, random | 0.1433 | 0.2101 | 0.2190 | 0.2725 |
| Masking | SST, attention | 0.0969 | 0.2070 | 0.1087 | 0.2633 |
| Masking | CLS, unmasked | 0.1318 | 0.2089 | 0.2383 | 0.2925 |
| Masking | CLS, random | 0.1180 | 0.1828 | 0.2064 | 0.2542 |
| Masking | CLS, attention | 0.0742 | 0.1615 | 0.1035 | 0.2342 |
Correct geometry targets improve mAP@5000 over shuffled targets by 1.40/1.62/2.31/3.04 percentage points; the same ordering holds at @all. Attribute targets outperform class-only targets at both cutoffs. Attention regions yield higher part hit rates and larger masking-induced retrieval drops than random regions. These results support meaningful semantic correspondence and part-associated, retrieval-relevant information.
The target substitution changes loss scale; localization does not certify each query's named attribute. Masking also affects CLS, so this effect is not exclusive to SST. We will add these controls and clarify the contribution accordingly.
(3/3)
Q3. Can COMAE and DA2H be compared under matched dataset splits and evaluation metrics?
We have added the requested comparison on AWA2, CUB, and SUN, retaining all four code lengths. The reproduced methods share the class split, query/database membership, relevance definition, and AP calculation. AP@K averages precision at relevant ranks among the first K retrieved items; queries with no relevant retrieval contribute zero. For each method, @5000 and @all use the same checkpoint and retrieval lists. The comparison controls data and evaluation; method-specific architectures and pretraining are retained.
Our SUN setting uses 574 seen / 143 unseen classes, 1,430 queries, and 7,170 database images, following the default split in the released COMAE loader. The manuscript's 645/72 split and retrieval counts were description errors and will be corrected; this clarification does not change the reported scores. We will explicitly state the implemented setting instead of transferring the textual setting from another paper.
Each entry below is mAP@5000 / mAP@all, at seed 42. The DA2H entries are our reproduction.
| Dataset | Method | 24 bits | 48 bits | 64 bits | 128 bits |
|---|---|---|---|---|---|
| AWA2 | PZSH-original | 0.5117 / 0.4502 | 0.5846 / 0.5406 | 0.6164 / 0.5633 | 0.6412 / 0.5997 |
| AWA2 | PZSH-MSE | 0.5132 / 0.4596 | 0.5877 / 0.5363 | 0.6149 / 0.5754 | 0.6336 / 0.6104 |
| AWA2 | COMAE | 0.3836 / 0.3337 | 0.5121 / 0.4520 | 0.5086 / 0.4683 | 0.5705 / 0.5327 |
| AWA2 | DA2H (our reproduction) | 0.4897 / 0.4313 | 0.5444 / 0.5032 | 0.5613 / 0.5203 | 0.6046 / 0.5736 |
| AWA2 | SST | 0.5769 / 0.5563 | 0.6192 / 0.5939 | 0.6288 / 0.6053 | 0.6643 / 0.6516 |
| CUB | PZSH-original | 0.1318 / 0.1317 | 0.2164 / 0.2164 | 0.2244 / 0.2244 | 0.2691 / 0.2691 |
| CUB | PZSH-MSE | 0.1397 / 0.1392 | 0.2160 / 0.2153 | 0.2281 / 0.2276 | 0.2704 / 0.2701 |
| CUB | COMAE | 0.1119 / 0.1115 | 0.1831 / 0.1827 | 0.2054 / 0.2050 | 0.2496 / 0.2490 |
| CUB | DA2H (our reproduction) | 0.1246 / 0.1241 | 0.1730 / 0.1726 | 0.2024 / 0.2021 | 0.2415 / 0.2412 |
| CUB | SST | 0.1581 / 0.1575 | 0.2362 / 0.2357 | 0.2547 / 0.2541 | 0.3060 / 0.3059 |
| SUN | PZSH-original | 0.1057 / 0.1051 | 0.1437 / 0.1418 | 0.1729 / 0.1717 | 0.1930 / 0.1906 |
| SUN | PZSH-MSE | 0.1055 / 0.1031 | 0.1458 / 0.1433 | 0.1771 / 0.1742 | 0.1949 / 0.1921 |
| SUN | COMAE | 0.0950 / 0.0942 | 0.1233 / 0.1222 | 0.1482 / 0.1477 | 0.1874 / 0.1869 |
| SUN | DA2H (our reproduction) | 0.0855 / 0.0852 | 0.1345 / 0.1335 | 0.1653 / 0.1643 | 0.1873 / 0.1867 |
| SUN | SST | 0.1273 / 0.1260 | 0.1691 / 0.1667 | 0.2097 / 0.2070 | 0.2169 / 0.2154 |
SST is higher than original PZSH, PZSH-MSE, COMAE, and our DA2H reproduction in all twelve dataset–code-length settings under both cutoffs. These matched comparisons support the empirical advantage without mixing literature protocols or evaluation cutoffs. We will place them separately from contextual literature scores and document the implementations used.
(1/3)
Dear Reviewer 9HNw,
Thank you for your careful review and constructive feedback. We respond to this concern below.
Q1. Evaluate the transfer objectives on original PZSH and distinguish them from the MSE modification under identical retrieval protocols.
We now report the original-center/original-loss reproduction separately from PZSH-MSE. The original loss implementation uses fixed centers with center-classification, quantization, and proxy terms. PZSH implementation
We first add patch supervision without changing those fixed centers (O0 versus O1). We then isolate the effect of making centers learnable (O2), adding patch (O3), and adding geometry (O5). O2/O3/O5 retain the original hash losses and the original center initialization construction. This separate adaptation is necessary because a loss defined only on fixed centers cannot update those centers. The M0/M1/M2/SST rows independently isolate the objectives within the MSE branch. CUB mAP@5000 at seed 42 is:
O1 improves over fixed-center O0 at every length. In the original-loss, learnable-center branch, the increments O3−O2 are 0.196/0.034/0.300/0.106 points, and O5−O3 are 0.334/0.130/0.510/0.064 points, in 24/48/64/128-bit order. Both increments are positive under @all as well. The results provide controlled evidence that the objectives' benefits are not confined to the MSE branch, while identifying the required center adaptation. We will rename the internal baseline, retain the distinction between fixed and learnable centers, and report both evaluation cutoffs.