- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses the limitations of existing approaches in which similar regions are repeatedly encoded and the spatial layout is lost during compression. The authors propose a two-stage training framework featuring a shared sketch and photo encoder. Experimental results validate the effectiveness of the proposed approach and demonstrate state-of-the-art (SOTA) performance on several benchmarks.
Critical strengths
-
Learns cross-modal representations from two complementary perspectives: compressing redundant evidence and inferring missing details.
-
Integrates redundancy-aware balanced semantic compression and masked spatial prediction into the shared sketch and photo encoder.
-
Demonstrates consistent improvements over baseline methods across public benchmarks.
Critical weaknesses
-
Limited novelty: The technical novelty is somewhat modest, as similar latent mask prediction designs have appeared in prior work.
-
Lack of visualization: It would be helpful to include visualisations to showcase that repetitive regions have been effectively down-weighted.
Questions for authors
- Dataset evaluation: It is recommended to validate the design choices on the SketchyCOCO dataset, which targets scene-level rather than object-level retrieval.
6 comments
Thank you for raising the novelty concern — Aggregation with spatial fusion (2/5)
Major Weakness 1 · Limited technical novelty (continued)
Table 8 excerpt · Aggregation with spatial fusion (Appendix A.8, p. 17, lines 904–917, and p. 18, lines 918–922)
Each cell reports R@1 / R@5 / R@10 (%, three-seed mean ± sample SD).
| Head | User-disjoint FS-COCO | SketchyCOCO-SL | ShoeV2 |
|---|---|---|---|
| NetVLAD | 59.40±0.29 / 81.06±0.68 / 87.52±0.31 | 89.37±0.27 / 98.25±0.27 / 99.37±0.55 | 45.05±1.82 / 76.33±0.61 / 84.88±1.28 |
| DOLG-style | 60.54±0.32 / 81.71±0.31 / 88.09±0.25 | 92.22±0.27 / 98.41±0.27 / 99.37±0.73 | 44.94±0.46 / 76.13±2.08 / 83.88±1.47 |
| A2GC | 65.21±0.49 / 83.78±0.39 / 89.22±0.41 | 93.81±2.90 / 99.52±0.00 / 99.84±0.27 | 45.90±2.10 / 76.18±1.71 / 84.88±0.77 |
| Full model (E5) | 65.34±1.08 / 84.62±0.30 / 89.67±0.12 | 92.70±0.27 / 99.52±0.48 / 100.00±0.00 | 48.90±1.65 / 78.03±1.20 / 85.79±0.71 |
With spatial fusion, the full model leads A2GC on seven of nine entries and ties on one. A2GC retains the lead in SL R@1. On FS-COCO, our method trails A2GC at all three recall cutoffs in the 768D comparison and leads at all three after fusion. These results show that pairing the compressor with spatial memory contributes to the full model's advantage.
Thank you for suggesting a visualization — Redundancy weighting and retrieval (3/5)
Major Weakness 2 · Lack of visualization of redundancy down-weighting
We agree that recall alone makes it difficult to tell which regions are affected by redundancy weighting. Your suggestion prompted us to show the heatmaps alongside the paired photo's rank in the full gallery. We prepared three Grad-CAM comparisons in Figure 2C, p. 9, lines 463–471. Grad-CAM is a gradient-based visualization method. We use the gradient of the sketch–photo matching score with respect to the Stage 4 features to weight each channel and highlight regions that support the match. See §4.3, p. 8, lines 429–431 for the discussion.
The only configuration difference between E2 and E3 in the two ShoeV2 examples is the use of redundancy-aware token weighting. For the fringed shoe, the response covers more of the fringe and the paired photo rises from rank 18 to rank 1. For the buckled boot, the strongest response moves from the blank area near the toe to the buckle and the paired photo rises from rank 17 to rank 1. In the third example, on FS-COCO, the model's response is more concentrated on the foreground cow after adding compression, and the paired photo rises from rank 4 to rank 1. The buckled-boot example is especially clear. The previously dominant response in the blank area recedes, the response shifts to the buckle, a detail that helps distinguish the shoe, and the paired photo moves to rank 1. This provides qualitative support for the role of redundancy weighting in reducing the influence of low-information regions and preserving local discriminative evidence.
Thank you for your feedback on novelty — Compressor design and aggregation comparisons (1/5)
Thank you for making time in your busy review schedule to read our paper carefully and for recognizing the complementary roles of compressing redundant evidence and predicting missing information. Your comments helped us explain more concretely the compressor's independent contribution, how retrieval responses change after redundancy weighting, and how each component performs on scene-level retrieval. All page and line numbers below refer to the revised manuscript.
Major Weakness 1 · Limited technical novelty
We agree with your observation that prior work has used latent masked prediction. This helped us clarify prediction's auxiliary role and identify our core methodological contribution, the redundancy-aware balanced semantic compressor. It addresses how to allocate capacity in a compact retrieval descriptor so that similar responses can be consolidated without gaining influence simply through repetition.
We estimate each token's redundancy from its similarity to other tokens in the same image and reduce the total mass of more redundant tokens. Sinkhorn soft assignment constrains both the total contribution of each token across slots and the assignment mass received by each slot. Each slot computes a weighted summary of the features assigned to it. The 32 low-dimensional slot outputs are retained separately and concatenated. See §3.2, p. 5, lines 242–269, and p. 6, lines 270–280 for the key and value construction, redundancy-aware mass, and slot assignment.
We ran the comparisons across three seeds using the same DINOv3 backbone, 768D descriptor width, training splits, and test galleries, with the spatial branch disabled for all methods. NetVLAD and DOLG-style replace the 512D complementary component. A2GC is adapted as a whole aggregation head to preserve its design. See Appendix A.8, p. 17, lines 868–898.
Table 6 excerpt · Aggregation without the spatial branch (§4.5, p. 10, lines 507–527)
Each cell reports R@1 / R@5 / R@10 (%, three-seed mean ± sample SD).
E3 outperforms NetVLAD and DOLG-style at all recall cutoffs on all three datasets. It also outperforms A2GC at all three cutoffs on ShoeV2, with an R@1 gain of 2.25 percentage points. A2GC leads on FS-COCO in the 768D comparison. On SL, E2 with uniform mass achieves the highest R@1 and R@5, indicating that the gains from redundancy weighting depend on the composition of the evidence in the scene. See §4.3, p. 8, lines 424–429.