acceptodds
Withdrawn from ICLR 2027

Compress Redundancy, Predict What Is Missing: Beyond Global Pooling for Scene Sketch Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sketch-to-photo retrieval enables users to query images with free-hand drawings, but its central challenge is matching sparse strokes to visually rich photographs. Existing methods emphasize semantic alignment or local matching, yet pay limited attention to two limitations of single-vector aggregation: similar regions are repeatedly encoded, while spatial layout is lost during compression. We argue that learning a compact representation for discriminative retrieval requires consolidating repeated evidence while preserving the spatial ordering of regions. Accordingly, we encode sketches and photos with a shared pretrained DINOv3 ConvNeXt-B backbone and introduce redundancy-aware balanced semantic compression and masked spatial prediction to learn complementary global, regional, and spatial representations. Under identical protocols on FS-COCO, SketchyCOCO-SL, and QMUL-ShoeV2, our method improves upon the previous best results by 8.6%–32.7% in R@1, 5.9%–10.6% in R@5, and 1.9%–6.3% in R@10 across scene-level and fine-grained object retrieval. This provides a new direction for compact representation learning in sketch-driven cross-modal image retrieval, jointly suppressing redundancy while preserving spatial structure.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
sketch-based image retrieval, cross-modal retrieval, representation learning, feature aggregation, predictive representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_E3jgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the limitations of existing approaches in which similar regions are repeatedly encoded and the spatial layout is lost during compression. The authors propose a two-stage training framework featuring a shared sketch and photo encoder. Experimental results validate the effectiveness of the proposed approach and demonstrate state-of-the-art (SOTA) performance on several benchmarks.

Critical strengths
  • Learns cross-modal representations from two complementary perspectives: compressing redundant evidence and inferring missing details.

  • Integrates redundancy-aware balanced semantic compression and masked spatial prediction into the shared sketch and photo encoder.

  • Demonstrates consistent improvements over baseline methods across public benchmarks.

Critical weaknesses
  • Limited novelty: The technical novelty is somewhat modest, as similar latent mask prediction designs have appeared in prior work.

  • Lack of visualization: It would be helpful to include visualisations to showcase that repetitive regions have been effectively down-weighted.

Questions for authors
  • Dataset evaluation: It is recommended to validate the design choices on the SketchyCOCO dataset, which targets scene-level rather than object-level retrieval.
6 comments
AuthorsOpenReview ↗

Thank you for your feedback on novelty — Compressor design and aggregation comparisons (1/5)

Thank you for making time in your busy review schedule to read our paper carefully and for recognizing the complementary roles of compressing redundant evidence and predicting missing information. Your comments helped us explain more concretely the compressor's independent contribution, how retrieval responses change after redundancy weighting, and how each component performs on scene-level retrieval. All page and line numbers below refer to the revised manuscript.

Major Weakness 1 · Limited technical novelty

We agree with your observation that prior work has used latent masked prediction. This helped us clarify prediction's auxiliary role and identify our core methodological contribution, the redundancy-aware balanced semantic compressor. It addresses how to allocate capacity in a compact retrieval descriptor so that similar responses can be consolidated without gaining influence simply through repetition.

We estimate each token's redundancy from its similarity to other tokens in the same image and reduce the total mass of more redundant tokens. Sinkhorn soft assignment constrains both the total contribution of each token across slots and the assignment mass received by each slot. Each slot computes a weighted summary of the features assigned to it. The 32 low-dimensional slot outputs are retained separately and concatenated. See §3.2, p. 5, lines 242–269, and p. 6, lines 270–280 for the key and value construction, redundancy-aware mass, and slot assignment.

  • NetVLAD softly assigns local features to cluster centers and accumulates residuals relative to each center. Our compressor also considers similarities among local responses in the same image to regulate each token's total contribution across all slots.
  • DOLG-style uses orthogonal fusion to extract local components complementary to the global vector. Our complementary component retains multiple slot outputs separately, with redundancy among local responses and the assignment load of each slot jointly determining how these slots organize the evidence.
  • A2GC (CVPR 2026) uses geometry-constrained asymmetric optimal transport and retains global and local readouts. Our design sets the token marginal directly from semantic similarities among tokens in the same image and balances the mass received by the semantic slots.

We ran the comparisons across three seeds using the same DINOv3 backbone, 768D descriptor width, training splits, and test galleries, with the spatial branch disabled for all methods. NetVLAD and DOLG-style replace the 512D complementary component. A2GC is adapted as a whole aggregation head to preserve its design. See Appendix A.8, p. 17, lines 868–898.

Table 6 excerpt · Aggregation without the spatial branch (§4.5, p. 10, lines 507–527)
Each cell reports R@1 / R@5 / R@10 (%, three-seed mean ± sample SD).

HeadUser-disjoint FS-COCOSketchyCOCO-SLShoeV2
NetVLAD52.34±1.83 / 76.07±0.73 / 83.76±0.7180.63±1.98 / 96.83±0.73 / 98.41±1.1040.29±0.23 / 73.37±1.35 / 83.83±2.20
DOLG-style55.22±0.56 / 77.89±0.80 / 84.97±0.5682.86±1.65 / 97.30±0.99 / 98.57±0.8239.74±0.91 / 73.42±2.03 / 82.38±1.74
A2GC64.33±0.30 / 83.38±0.25 / 88.88±0.3188.89±2.35 / 98.10±0.48 / 99.37±0.2742.14±1.94 / 74.27±1.46 / 83.58±0.17
Balanced slots, uniform mass (E2)62.32±1.21 / 82.39±0.69 / 87.90±0.7389.21±0.73 / 98.73±0.27 / 99.52±0.0043.09±2.25 / 74.22±1.07 / 83.73±0.23
Balanced slots, redundancy-aware mass (E3)62.82±0.94 / 82.63±0.58 / 88.69±0.3186.67±2.08 / 98.10±0.48 / 99.68±0.2744.39±1.89 / 76.68±0.53 / 84.98±1.17

E3 outperforms NetVLAD and DOLG-style at all recall cutoffs on all three datasets. It also outperforms A2GC at all three cutoffs on ShoeV2, with an R@1 gain of 2.25 percentage points. A2GC leads on FS-COCO in the 768D comparison. On SL, E2 with uniform mass achieves the highest R@1 and R@5, indicating that the gains from redundancy weighting depend on the composition of the evidence in the scene. See §4.3, p. 8, lines 424–429.

AuthorsOpenReview ↗

Thank you for raising the novelty concern — Aggregation with spatial fusion (2/5)

Major Weakness 1 · Limited technical novelty (continued)

Table 8 excerpt · Aggregation with spatial fusion (Appendix A.8, p. 17, lines 904–917, and p. 18, lines 918–922)
Each cell reports R@1 / R@5 / R@10 (%, three-seed mean ± sample SD).

HeadUser-disjoint FS-COCOSketchyCOCO-SLShoeV2
NetVLAD59.40±0.29 / 81.06±0.68 / 87.52±0.3189.37±0.27 / 98.25±0.27 / 99.37±0.5545.05±1.82 / 76.33±0.61 / 84.88±1.28
DOLG-style60.54±0.32 / 81.71±0.31 / 88.09±0.2592.22±0.27 / 98.41±0.27 / 99.37±0.7344.94±0.46 / 76.13±2.08 / 83.88±1.47
A2GC65.21±0.49 / 83.78±0.39 / 89.22±0.4193.81±2.90 / 99.52±0.00 / 99.84±0.2745.90±2.10 / 76.18±1.71 / 84.88±0.77
Full model (E5)65.34±1.08 / 84.62±0.30 / 89.67±0.1292.70±0.27 / 99.52±0.48 / 100.00±0.0048.90±1.65 / 78.03±1.20 / 85.79±0.71

With spatial fusion, the full model leads A2GC on seven of nine entries and ties on one. A2GC retains the lead in SL R@1. On FS-COCO, our method trails A2GC at all three recall cutoffs in the 768D comparison and leads at all three after fusion. These results show that pairing the compressor with spatial memory contributes to the full model's advantage.

AuthorsOpenReview ↗

Thank you for suggesting a visualization — Redundancy weighting and retrieval (3/5)

Major Weakness 2 · Lack of visualization of redundancy down-weighting

We agree that recall alone makes it difficult to tell which regions are affected by redundancy weighting. Your suggestion prompted us to show the heatmaps alongside the paired photo's rank in the full gallery. We prepared three Grad-CAM comparisons in Figure 2C, p. 9, lines 463–471. Grad-CAM is a gradient-based visualization method. We use the gradient of the sketch–photo matching score with respect to the Stage 4 features to weight each channel and highlight regions that support the match. See §4.3, p. 8, lines 429–431 for the discussion.

The only configuration difference between E2 and E3 in the two ShoeV2 examples is the use of redundancy-aware token weighting. For the fringed shoe, the response covers more of the fringe and the paired photo rises from rank 18 to rank 1. For the buckled boot, the strongest response moves from the blank area near the toe to the buckle and the paired photo rises from rank 17 to rank 1. In the third example, on FS-COCO, the model's response is more concentrated on the foreground cow after adding compression, and the paired photo rises from rank 4 to rank 1. The buckled-boot example is especially clear. The previously dominant response in the blank area recedes, the response shifts to the buckle, a detail that helps distinguish the shoe, and the paired photo moves to rank 1. This provides qualitative support for the role of redundancy weighting in reducing the influence of low-information regions and preserving local discriminative evidence.

Reviewer_qSEBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a sketch-to-photo retrieval framework motivated by two limitations of conventional global aggregation: repeated visual evidence may dominate the final representation, while spatial information can be weakened during compression. The method uses a shared pretrained DINOv3 ConvNeXt-B encoder and introduces redundancy-aware balanced semantic compression to redistribute token contributions across learnable slots. It also learns a spatial representation from intermediate features and applies masked-region semantic prediction to encourage the model to infer missing visual information from surrounding context. The resulting global, complementary semantic, and spatial representations are normalized and concatenated for cosine-similarity retrieval. Experiments on FS-COCO, SketchyCOCO-SL, and QMUL-ShoeV2 report improvements over the selected prior methods. Ablation experiments further investigate the contributions of predictive alignment, balanced slot assignment, redundancy-aware weighting, and spatial prediction.

Critical strengths

The first strength is the paper's central motivation. Rather than simply adding more local matching operations, the authors ask how a fixed-size representation should allocate capacity to repeated and discriminative visual evidence. This is a useful perspective for sketch-based retrieval, where the query is sparse and the target image contains substantially more visual information. Combining redundancy-aware weighting with balanced semantic slots is a reasonable way to investigate this idea.

A second strength is the breadth of the evaluation. The method is tested on FS-COCO, SketchyCOCO-SL, and QMUL-ShoeV2, covering both scene-level retrieval and fine-grained object matching. The reported results are consistently strong, and the progressive ablation on QMUL-ShoeV2 suggests that both semantic compression and the spatial branch contribute to the final representation. In particular, the reported improvement from 44.14 to 48.80 R@1 after adding the spatial branch and its predictive objective is encouraging.

Finally, the paper makes a useful effort toward reproducibility. It specifies training configurations, optimization settings, dataset protocols, and implementation details, and reports a stricter user-disjoint evaluation for FS-COCO. These details make the experimental claims easier to investigate and reproduce.

Critical weaknesses

My main concern is that the paper's methodological novelty is not yet sufficiently distinguished from the combination of established ideas. Redundancy-aware reweighting, Sinkhorn-based balanced assignment, compact feature aggregation, and masked latent prediction all have substantial precedents in representation learning and image retrieval. The paper's particular combination may be effective, but the current evidence does not fully establish what is fundamentally new about the proposed capacity-allocation mechanism, beyond adapting and combining these ingredients for sketch retrieval. A more direct comparison with established aggregation methods would help clarify this point.

The second concern is experimental comparability. The reported gains are impressive, particularly the R@1 result on SketchyCOCO-SL. However, the paper does not provide equally strong evidence that all headline comparisons use fully matched training and evaluation conditions. The authors describe a controlled retraining of a recent baseline on the additional user-disjoint FS-COCO split and on full SketchyCOCO, which is helpful. Comparable controlled results on the other benchmarks would make the overall claims much more convincing. This is particularly important when the gains are large enough to change the interpretation of the state of the art.

The ablation study is useful but somewhat narrow. Most of the component-level analysis is conducted on QMUL-ShoeV2, so it remains unclear whether the same components explain the improvements on complex scene retrieval. Moreover, the main predictive objective does not improve R@1 in isolation: E0 obtains 38.74 R@1, whereas E1 obtains 38.44. The full model eventually performs substantially better, but this makes it important to disentangle the contribution of the main predictive objective from that of the spatial branch and its region-prediction loss. Experiments on at least one scene-level benchmark would help establish whether the proposed mechanisms behave consistently across task settings.

I am also concerned about the absence of multi-seed statistics. The paper reports a fixed seed for the principal experiments, while some evaluation settings have relatively small test sets. Mean and standard deviation across multiple runs would help establish whether the improvements are robust rather than sensitive to initialization or training variability. This is especially relevant for interpreting the very high SketchyCOCO-SL scores.

Finally, the spatial branch is described as learning spatial relations, but its representation is constructed partly through fixed raster-order flattening, and its prediction target consists of pooled regional features. This may encode useful positional structure, but it does not by itself demonstrate that the model learns meaningful relations between objects or regions. The authors should more carefully distinguish spatial preservation from relational reasoning and provide evidence that the geometry-conditioned prediction objective captures information beyond the fixed spatial ordering.

Questions for authors
  1. Could the authors compare the proposed compressor against established aggregation alternatives, such as generalized max pooling, NetVLAD, DOLG-style aggregation, or a Sinkhorn-based retrieval aggregator, under a matched backbone, descriptor dimensionality, training protocol, and evaluation gallery? This would help isolate the contribution of the proposed redundancy-aware mass allocation.

  2. Could the authors provide controlled, same-protocol retraining results for the strongest relevant baseline on SketchyCOCO-SL and QMUL-ShoeV2, and clarify the precise candidate gallery and split used for each comparison? This would be particularly helpful for interpreting the substantial performance gap on SketchyCOCO-SL.

  3. Could the authors report results across multiple random seeds, ideally with mean and standard deviation, and extend the key ablations to at least one scene-level dataset? In particular, why does the main predictive objective slightly reduce R@1 in isolation, and does its contribution become more consistent when evaluated on scene-level retrieval?

  4. What evidence supports the interpretation that the spatial branch learns inter-region relations rather than primarily preserving coarse positional information? An ablation that removes the geometry input, changes the spatial ordering, or compares against a simpler spatial-feature baseline would help clarify the mechanism.

6 comments
AuthorsOpenReview ↗

Thank you for asking us to clarify the compressor design and compare aggregators (Q1)

Thank you for asking us to distinguish our capacity allocation mechanism from prior methods. We agree that redundancy reweighting, Sinkhorn assignment, compact aggregation, and masked feature prediction have precedents. We clarify how our compressor consolidates similar local responses and regulates their contribution to the final representation.

How does the compressor organize repeated evidence?

Balanced multi-slot aggregation organizes the evidence, and redundancy-aware mass regulates how much each token contributes. Learnable queries assign tokens by content similarity, allowing similar responses to be summarized within a slot. Slot marginals balance the mass across slots. We retain and concatenate the compact slot outputs. E2 consolidates evidence with uniform token mass. With the same slot structure, E3 uses similarity to other local responses to set each token's marginal and regulate its total contribution across slots. Both consolidate repeated evidence and retain complementary information in a fixed-length representation. See §1 (p. 2, lines 63–68) and §3.2 (pp. 5–6, lines 246–280, Eqs. 6–10).

Comparison with other aggregators

We added NetVLAD (cluster residual aggregation), DOLG-style (orthogonal local–global fusion), and A2GC (CVPR 2026). A2GC is a recent visual place recognition aggregator and gives us a direct comparison with geometry-constrained asymmetric optimal transport. We match the DINOv3 ConvNeXt-B backbone, token grid, training splits, epochs, seeds (20260816, 84, and 168), final-checkpoint evaluation, and test galleries. We use 768D descriptors without the spatial branch. We adapt the full A2GC aggregation head (Appendix A.8, p. 17, lines 869–898).

Table 6 (§4.5, p. 10, lines 512–519). R@1/R@5/R@10 (%, three-seed mean ± sample SD). Bold marks the best mean.

HeadUser-disjoint FS-COCOSketchyCOCO-SLQMUL-ShoeV2
NetVLAD52.34±1.83/76.07±0.73/83.76±0.7180.63±1.98/96.83±0.73/98.41±1.1040.29±0.23/73.37±1.35/83.83±2.20
DOLG-style55.22±0.56/77.89±0.80/84.97±0.5682.86±1.65/97.30±0.99/98.57±0.8239.74±0.91/73.42±2.03/82.38±1.74
Ours (E2, uniform mass)62.32±1.21/82.39±0.69/87.90±0.7389.21±0.73/98.73±0.27/99.52±0.0043.09±2.25/74.22±1.07/83.73±0.23
Ours (E3, redundancy-aware mass)62.82±0.94/82.63±0.58/88.69±0.3186.67±2.08/98.10±0.48/99.68±0.2744.39±1.89/76.68±0.53/84.98±1.17
A2GC64.33±0.30/83.38±0.25/88.88±0.3188.89±2.35/98.10±0.48/99.37±0.2742.14±1.94/74.27±1.46/83.58±0.17

Balanced multi-slot aggregation yields more consistent gains, and redundancy weighting can improve them further. E2 improves all recall metrics over E1 on all three datasets, with R@1 gains of 3.85–9.05 points. With the slots fixed, E3 further improves all metrics on FS-COCO and ShoeV2. On SL, similar responses from distinct instances of the same category can help distinguish scene configurations. Default weighting lowers R@1 and R@5, but weaker weighting raises them by 1.74 and 0.16 points over uniform mass. See Table 3 and §4.3 (p. 8, lines 403–429) and Appendix A.11 (pp. 20–21, lines 1065–1086).

E3 outperforms NetVLAD and DOLG-style at every cutoff on all three datasets. It also outperforms A2GC at all three cutoffs on ShoeV2 (+2.25 points at R@1) and at R@10 on SL. A2GC leads at all three cutoffs on user-disjoint FS-COCO (§4.5, p. 10, lines 522–527).

Comparison with spatial memory

We add a spatial branch to each aggregator, yielding 1024D descriptors with a one-third weight for the spatial component. A2GC retains its global-to-local weight ratio in the remaining two thirds (Appendix A.8, pp. 17–18, lines 899–924).

Table 8 (p. 17, lines 904–910). Same metrics and statistics as Table 6.

HeadUser-disjoint FS-COCOSketchyCOCO-SLQMUL-ShoeV2
NetVLAD59.40±0.29/81.06±0.68/87.52±0.3189.37±0.27/98.25±0.27/99.37±0.5545.05±1.82/76.33±0.61/84.88±1.28
DOLG-style60.54±0.32/81.71±0.31/88.09±0.2592.22±0.27/98.41±0.27/99.37±0.7344.94±0.46/76.13±2.08/83.88±1.47
A2GC65.21±0.49/83.78±0.39/89.22±0.4193.81±2.90/99.52±0.00/99.84±0.2745.90±2.10/76.18±1.71/84.88±0.77
Ours (E5)65.34±1.08/84.62±0.30/89.67±0.1292.70±0.27/99.52±0.48/100.00±0.0048.90±1.65/78.03±1.20/85.79±0.71

With spatial fusion, our full model leads in seven of nine comparisons and ties for best in one. It outperforms A2GC at every cutoff on FS-COCO and ShoeV2, with R@1 gains of 0.13 and 3.00 points, respectively. On SL, A2GC leads at R@1, both tie at R@5, and ours leads at R@10.

These results support our compressor design. Balanced multi-slot aggregation consolidates similar local evidence while retaining complementary information across slots. Redundancy-aware mass further regulates the total contribution of repeated responses.

AuthorsOpenReview ↗

Thank you for requesting controlled retraining and clarification of the evaluation splits and galleries (Q2)

Thank you for pointing out that controlled retraining on the same split and gallery is important for interpreting the large performance gap on SketchyCOCO-SL. The original manuscript reported controlled reruns on the additional user-disjoint FS-COCO split and full SketchyCOCO. To address your concern about the other two benchmarks, we added three-seed reruns of SketchScape on SketchyCOCO-SL and of ARNet and MCGI on QMUL-ShoeV2.

Splits, galleries, and retraining protocol

  • SketchyCOCO-SL. We follow the published fixed-subset setting and train on 1,015 sketch–photo pairs. Each of the 210 test sketches queries the same full gallery of 210 photos. SketchScape and our method use the same training split, queries, and gallery.
  • QMUL-ShoeV2. We follow the ARNet directory split (trainA/trainB and testA/testB). The training set contains 5,982 sketches and 1,800 photos. Each of the 666 test sketches queries the 200 unique photos in testB. ARNet, MCGI, and our method use the same training split, queries, and gallery.

See §4.1.1 (p. 7, lines 339–348) and Appendix A.4 (p. 16, lines 813–817) for the splits and evaluation. The controlled reruns use seeds 20260816, 84, and 168 and match the training epoch budget for each dataset. Each baseline retains its own loss, optimizer, and sampling procedure. We use the authors' released implementations of SketchScape and MCGI. For ARNet, we use the authors' full implementation with two corrections to parameter registration and input layout in the multi-scale Transformer. See Appendix A.7 (pp. 16–17, lines 854–865).

For ShoeV2, we added MCGI (Pattern Recognition 2026), a recent fine-grained sketch retrieval method. Its public ShoeV2 configuration uses 100 epochs. To match the training epoch budget, we used the authors' implementation to retrain it for 20 epochs per seed on the split and gallery described above, then evaluated the epoch-20 checkpoint. The EUPS-SBIR authors have yet to release training code, so Table 2 retains its published result. MCGI ShoeV2 configuration · EUPS-SBIR repository

Controlled retraining results

Table 2 (§4.2, p. 7, line 377 and p. 8, lines 379–390). Cells report R@1/R@5/R@10 (%). † marks controlled reruns. The reruns and our method report the three-seed mean ± sample SD. Other external entries are published results. Bold marks the highest value for each metric. A dash indicates an unreported result.

SketchyCOCO-SL methodR@1/R@5/R@10QMUL-ShoeV2 methodR@1/R@5/R@10
SceneDiff34.3/69.1/81.4Part-SBIR39.9/68.2/82.9
Partially Does It34.5/–/89.3ARNet†26.18±0.85/58.21±1.51/71.87±0.97
SceneTrilogy38.2/–/87.6NT-SBIR43.7/74.9/–
SceneSketcher-v268.1/87.6/95.2EUPS-SBIR44.2/70.8/84.7
SketchScape†73.02±0.73/93.81±1.72/97.78±1.20MCGI†33.43±0.35/68.22±0.57/80.33±0.40
Ours92.70±0.27/99.52±0.48/100.00±0.00Ours48.90±1.65/78.03±1.20/85.79±0.71

On SL, after matching the training split, epoch budget, test queries, and gallery, our method still leads the retrained SketchScape by 19.68, 5.71, and 2.22 percentage points at R@1, R@5, and R@10, respectively. The large R@1 gap therefore persists under the matched protocol. To assess sensitivity to the training seed, we computed approximate 95% Student-t confidence intervals across the three seeds. The intervals for R@1/R@5/R@10 are [92.02, 93.38]%, [98.34, 100.00]%, and [100.00, 100.00]%, respectively, suggesting limited variation across seeds on this fixed test set. See §4.2 (p. 8, lines 393–396).

On ShoeV2, our method outperforms the controlled reruns of ARNet and MCGI at all three recall cutoffs. Compared with MCGI, the stronger of the two rerun baselines, our gains at R@1, R@5, and R@10 are 15.47, 9.81, and 5.46 percentage points, respectively.

AuthorsOpenReview ↗

Thank you for requesting multi-seed ablations and clarification of the two prediction objectives (Q3)

Thank you for noting that the original ablations focused on ShoeV2 and that main-branch prediction alone slightly reduced R@1. We extended the full ablation to user-disjoint FS-COCO, SketchyCOCO-SL, and QMUL-ShoeV2 to test whether this also occurs in scene retrieval. We examine main-branch prediction (E0→E1) and spatial region prediction (E4→E5) separately.

Three-seed ablation. Main-stage seeds are 20260816, 84, and 168. Spatial seeds are the main seeds plus four on FS-COCO and the same seeds on SL and ShoeV2. Within each dataset, configurations share the full training split, test queries, and gallery. We evaluate prespecified final checkpoints (Appendix A.5, p. 16, lines 819–826).

Table 3 (§4.3, p. 8, lines 403–418). R@1/R@5/R@10 (%, three-seed mean ± sample SD). Bold marks the best mean.

Config.User-disjoint FS-COCOSketchyCOCO-SLQMUL-ShoeV2
E−1 · Frozen DINOv35.20±0.00/13.47±0.00/18.73±0.007.62±0.00/19.05±0.00/35.24±0.006.46±0.00/21.17±0.00/31.98±0.00
E0 · Paired global-only56.13±0.18/78.30±0.09/85.16±0.4083.02±1.67/96.19±0.48/97.62±0.0041.39±2.36/72.42±0.31/82.08±0.77
E0₁₀₂₄ · 1024D global-only56.20±0.62/78.31±0.84/85.26±0.4883.17±1.67/96.19±0.48/97.62±0.4839.79±1.23/72.22±1.97/82.73±1.65
E1 · Main-branch prediction54.10±0.42/77.04±0.70/84.53±0.5080.16±1.98/95.24±1.72/98.41±0.2739.24±1.02/73.27±1.23/83.23±1.13
E2 · Balanced slots, uniform mass62.32±1.21/82.39±0.69/87.90±0.7389.21±0.73/98.73±0.27/99.52±0.0043.09±2.25/74.22±1.07/83.73±0.23
E3 · Redundancy-aware mass62.82±0.94/82.63±0.58/88.69±0.3186.67±2.08/98.10±0.48/99.68±0.2744.39±1.89/76.68±0.53/84.98±1.17
E4 · Spatial memory65.34±0.68/84.61±0.29/89.76±0.1293.17±0.27/99.68±0.27/100.00±0.0048.05±1.45/77.63±1.59/85.94±0.48
E5 · Spatial region prediction (full)65.34±1.08/84.62±0.30/89.67±0.1292.70±0.27/99.52±0.48/100.00±0.0048.90±1.65/78.03±1.20/85.79±0.71

E0→E1. Why does main-branch prediction lower R@1?

We mask part of the intermediate sketch features and use the remaining context to align with the full photo representation (§3.1, p. 5, lines 227–238, Eqs. 4–5). R@1 falls by 2.03, 2.86, and 2.15 percentage points on FS-COCO, SL, and ShoeV2, respectively. Extending the ablation to scenes therefore shows no R@1 gain. At broader cutoffs, SL gains 0.79 points at R@10, and ShoeV2 gains 0.85 and 1.15 points at R@5 and R@10.

We grouped full-gallery results by dataset, recall cutoff, and recovery or regression. After deduplication, we inspected 173 E0→E1 queries and 118 E4→E5 queries. Each case compares the same sketch, its paired photo, and the actual top-1 result for each configuration. The main seed is 20260816, and the FS-COCO spatial seed is 20260820 (Appendix A.14, p. 22, lines 1176–1185).

Figure 5 (p. 23, lines 1215–1220) shows recoveries involving adult and young zebras and the combination of a boot upper and sole. Regressions include confusion over a clock tower's viewpoint, zebra pose and grouping, and sandal strap and sole configurations. Predicting from visible context can help infer missing components, but several subtly different instances can fit the same high-level semantics. This may explain why the correct photo can move into a broader set of top-ranked results while a semantically similar photo with a different configuration takes first place (Appendix A.14, p. 23, lines 1223–1232).

E4→E5. What does spatial region prediction add?

Ordered spatial memory (E3→E4) improves every recall metric on all three datasets, with R@1 gains of 2.52–6.51 points. E4→E5 adds region prediction to this memory. The predictor estimates masked-region features from visible spatial memory, anchor features, and relative geometry. ShoeV2 gains 0.85 and 0.40 points at R@1 and R@5, with a slight R@10 drop. FS-COCO changes little. SL drops slightly at R@1 and R@5 and stays unchanged at R@10 (§3.3, p. 6, lines 292–311 and §4.3, p. 9, lines 432–442).

Figure 6 (p. 24, lines 1280–1284) shows corrections in plane–sun position, dog–sheep composition, and heel–upper configuration, consistent with the goal of learning content–position associations. Regressions still involve similar poses, overlap, or boot constructions. Visible parts of a single shoe can provide more consistent cues to missing parts, whereas scenes contain more varied object combinations and backgrounds. This may explain the gains at the top ranks on ShoeV2 and the limited average gains on the scene benchmarks (Appendix A.14, p. 23, lines 1225–1232).

In summary, main-branch prediction alone does not improve R@1 on either scene benchmark. Ordered spatial memory yields more consistent gains, and spatial region prediction helps some queries involving object composition or relative position and improves early recall on ShoeV2.

Comments on the paper

9 comments
AuthorsOpenReview ↗

Thank you for your feedback on comparison fairness and backbone effects (Q1)

Q1 · Comparison fairness, backbone effects, and statistical uncertainty

Thank you for taking the time to review our paper carefully.

To address your concern about the lack of 2026 comparisons, we reran MCGI (Pattern Recognition 2026) on ShoeV2 using the authors’ code. On the standard FS-COCO and SketchyCOCO-SL benchmarks, SketchScape (BMVC 2025) is the most recent directly comparable baseline we verified.

In response to your request for multi-seed evaluation on SL, we extended the comparison to FS-COCO, SketchyCOCO-SL, and ShoeV2, using seeds 20260816, 84, and 168. Like SketchScape, we fixed the training budget in advance. Appendix A.5 (p. 16, lines 819–826) specifies the seeds, training budgets, and evaluation rule. For each run, we report the final checkpoint without selecting it based on test performance.

Tables 1–2 excerpts (§4.2, pp. 7–8, lines 363–390). Each cell gives R@1 / R@5 / R@10 (%, mean ± sample SD). † denotes controlled reruns with matched splits, galleries, and epoch budgets, retaining each baseline’s loss, optimizer, and sampling procedure. The rerun protocol is detailed in Appendix A.7 (pp. 16–17, lines 854–865). Other external results are quoted from the published papers.

BenchmarkMethodR@1 / R@5 / R@10
User-disjoint FS-COCOSketchScape†61.11±0.12 / 80.11±0.74 / 86.12±0.80
Ours65.34±1.08 / 84.62±0.30 / 89.67±0.12
SketchyCOCO-SLSketchScape†73.02±0.73 / 93.81±1.72 / 97.78±1.20
Ours92.70±0.27 / 99.52±0.48 / 100.00±0.00
QMUL-ShoeV2ARNet†26.18±0.85 / 58.21±1.51 / 71.87±0.97
MCGI† (2026)33.43±0.35 / 68.22±0.57 / 80.33±0.40
Ours48.90±1.65 / 78.03±1.20 / 85.79±0.71

The available materials did not support matched reruns of the three methods you suggested. SceneSketcher-v2 provides retrieval code and test graphs, but its release lacks the full training-graph pipeline. The SceneTrilogy authors have not released the training code or the exact list of 82,783 photo–text pairs used for captioning pretraining. The public materials for SceneDiff do not include training code.

To investigate whether backbone or protocol differences explain the larger gains on scene retrieval than on ShoeV2, we adapted SketchScape, the strongest public baseline on the two scene benchmarks, to DINOv3 while retaining its projection head, I-Con loss, and Adam optimizer. With DINOv3, its R@1 rises from 61.11 to 62.97 on user-disjoint FS-COCO and from 73.02 to 74.92 on SL.

Table 5 excerpt (§4.4, pp. 9–10, lines 483–505). Adaptation details are in Appendix A.10 (p. 19, lines 1004–1025).

MethodUser-disjoint FS-COCOSLShoeV2
DINOv3 global-only (E0)56.13±0.18 / 78.30±0.09 / 85.16±0.4083.02±1.67 / 96.19±0.48 / 97.62±0.0041.39±2.36 / 72.42±0.31 / 82.08±0.77
SketchScape-DINO62.97±0.39 / 81.33±0.52 / 86.97±0.3274.92±3.70 / 93.17±2.79 / 97.30±1.4542.84±1.69 / 74.77±0.26 / 82.88±0.69
Ours-DINO (full)65.34±1.08 / 84.62±0.30 / 89.67±0.1292.70±0.27 / 99.52±0.48 / 100.00±0.0048.90±1.65 / 78.03±1.20 / 85.79±0.71
CLIP global-only38.00±0.93 / 65.64±0.23 / 75.88±0.4059.68±2.15 / 87.30±0.55 / 95.40±1.4538.34±1.11 / 72.67±0.94 / 83.58±1.00
Ours-CLIP (core)56.72±0.53 / 78.87±0.70 / 85.16±0.5083.65±1.53 / 96.83±0.73 / 99.05±0.4848.00±2.26 / 78.83±0.94 / 87.54±0.54

Our full DINOv3 model outperforms E0 and SketchScape-DINO at every recall cutoff, with R@1 gains over SketchScape-DINO of 2.38 / 17.78 / 6.06 percentage points across the three datasets. Our CLIP core variant also outperforms same-backbone global pooling at every cutoff, with R@1 gains of 18.72 / 23.97 / 9.66 points. The gains persist with matched backbones, splits, and galleries. Their magnitude varies by dataset, but a stronger backbone alone cannot explain the improvement from the combined design.

To address your concern about the 210-query SL test set, we report approximate 95% Student-t confidence intervals across three seeds (df = 2, intersected with [0, 100]%). The intervals are [92.02, 93.38] / [98.34, 100.00] / [100.00, 100.00]% for R@1 / R@5 / R@10, respectively (§4.2, p. 8, lines 393–396). The narrow R@1 interval indicates limited variation across seeds on this fixed test set. All three runs reached the R@10 ceiling.

We also addressed the cross-referencing issues you flagged. The main text now cites Table 2 (§4.2, p. 7, line 377) and refers to the relevant equations by number in §§3.1–3.4.

AuthorsOpenReview ↗

Thank you for asking how positional encoding and region prediction differ (Q2)

Thank you for flagging our unclear wording. Since our predictor uses relative geometry, you ask whether positional encoding plus global pooling suffices and what missing-region prediction adds.

We evaluated E−1 through E5, including the 1024D E0₁₀₂₄ control, on user-disjoint FS-COCO, SketchyCOCO-SL, and QMUL-ShoeV2 with main seeds 20260816, 84, and 168. E4→E5 isolates the effect of adding region prediction to ordered spatial memory. On SL, we also trained two positional-encoding-plus-pooling baselines and compared them with E5. We used full training splits, all test queries, full galleries, prespecified final checkpoints for trained runs, and main seed + 4 for FS-COCO's spatial stage. See Appendix A.5 (p. 16, lines 819–826) and Appendix A.9 (p. 18, lines 929–947) for the protocols.

The revised §1 (p. 2, lines 69–77) clarifies that positional encoding records where features occur, while learning spatial associations also depends on how regions fit together. Since the same entity can elicit different local responses in sketches and photos, we fuse local context and jointly read out regional features in fixed raster order to encode content and layout together. Auxiliary region prediction then uses visible regions and their relative positions to constrain the semantics of a masked region. The ordered readout preserves content–position associations, and region prediction adds semantic supervision for those associations. See §3.3 (p. 6, lines 289–311) for the geometry input.

Explicit positional encoding with pooling. We compared Additive PE+GAP (Vaswani et al., NeurIPS 2017) and CoordConv+GAP (Liu et al., NeurIPS 2018). Both reuse E3's global and compressed components with equal-weight fusion.

Table 9 (p. 18, lines 950–955). SketchyCOCO-SL, three-seed mean ± sample SD (%).

Spatial readoutR@1R@5R@10
Additive PE+GAP86.83±0.9998.73±0.7399.52±0.00
CoordConv+GAP86.51±1.5398.73±0.7399.52±0.00
Full model (E5)92.70±0.2799.52±0.48100.00±0.00

E5 improves R@1 by 5.87 and 6.19 percentage points over the two pooled controls, respectively. We checked the full-gallery ranks for all 210 SL queries and visually examined 19 queries whose ranks changed in the same direction against both controls. E5 recovered 10 top-1 hits, improved the paired photo's rank without reaching first place in 5 cases, and lost 4 top-1 hits. Figure 3 shows 8, 3, and 3 of these cases, respectively (Appendix A.9, p. 18, lines 958–963).

In 7 of the 8 top-1 recoveries shown in Figure 3(a), the controls already retrieved the foreground subject or broad category, and E5 corrected the object combination or configuration. For example, E5 retrieved a dog behind a foreground sheep and a formation of two biplanes. These changes concern which objects accompany the subject and how they are arranged. They fit our aim of using visible context and relative geometry to constrain missing content and encode configurations consistent with that context. See Appendix A.9 (p. 19, lines 979–997) and Figure 3 (p. 20, lines 1059–1061).

Increment from region prediction. E4 already uses ordered spatial memory. E5 adds the region-prediction objective.

Table 3 excerpt (§4.3, p. 8, lines 403–418). R@1 / R@5 / R@10, three-seed mean ± sample SD (%).

DatasetE4: Spatial memoryE5: Spatial memory + region prediction
User-disjoint FS-COCO65.34±0.68/84.61±0.29/89.76±0.1265.34±1.08/84.62±0.30/89.67±0.12
SketchyCOCO-SL93.17±0.27/99.68±0.27/100.00±0.0092.70±0.27/99.52±0.48/100.00±0.00
QMUL-ShoeV248.05±1.45/77.63±1.59/85.94±0.4848.90±1.65/78.03±1.20/85.79±0.71

We visually examined 118 E4→E5 queries across the three datasets, comparing each sketch, paired photo, and actual top-1 results before and after adding prediction (Appendix A.14, p. 22, lines 1176–1185). Figure 6(a–c) shows corrections involving a plane's position relative to the sun, a dog–sheep combination, and a heel–upper configuration. These changes are consistent with prediction reinforcing content–position associations. The targets are high-level features without explicit labels for these configurations. Plausible predictions may still leave similar proportions, poses, viewpoints, or layouts ambiguous (Appendix A.14, p. 23, lines 1225–1232; Figure 6, p. 24, lines 1280–1283).

Visible parts of a single shoe can offer more coherent clues to missing parts than scenes with varied objects and backgrounds. This may explain why prediction improves ShoeV2 R@1 and R@5 by 0.85 and 0.40 points, with little change or slight declines on FS-COCO and SL. §4.3 (p. 9, lines 435–442) summarizes this difference and its possible explanation and points readers to the spatial controls in Appendix A.9 and the prediction analyses in Appendix A.14.

AuthorsOpenReview ↗

Thank you for asking us to clarify the definitions and test our choices (Q3)

Thank you for asking about our definitions, parameter sensitivity, and equal fusion. We clarified the equations and tested these settings and the freezing schedule.

Definitions and cross-references. §3.2 (pp. 5–6, lines 246–280) now defines separate LayerNorm branches for each 256D token (Eq. 6), ki=Norm⁡2(WkLN⁡k(xi))k_i=\operatorname{Norm}_2(W_k\operatorname{LN}_k(x_i)) and vi=LN⁡v(xi)v_i=\operatorname{LN}_v(x_i). Keys are 64D (bias-free projection, L2 normalization) and values are 256D. In Eq. (7), σ(t)=1/(1+e−t)\sigma(t)=1/(1+e^{-t}) is the sigmoid. ρ\rho sets the threshold, β\beta smooths redundancy, and γ\gamma controls mass decay via mi∝di−γm_i\propto d_i^{-\gamma}. Eqs. (8)–(10) use normalized query–key cosine logits ℓsi=Norm⁡2(us)⊤ki/τslot\ell_{si}=\operatorname{Norm}_2(u_s)^\top k_i/\tau_{\mathrm{slot}}, with τslot=0.10\tau_{\mathrm{slot}}=0.10, for Sinkhorn assignment. A shared linear projection maps each weighted summary to 16D csc_s. The 32 slots concatenate to 512D. We now cross-reference numbered equations (§§3.1–3.4) and cite Table 2 (§4.2, p. 7, line 377).

Revised motivation in §1 (p. 1, lines 45–49).

Efficient coding and predictive coding in biological vision inspire our research. These ideas suggest two complementary processes: compressing recurring evidence and inferring poorly observed evidence from context (Attneave, 1954; Barlow, 1961; Rao & Ballard, 1999). We study these directions through local aggregation and auxiliary learning from incomplete observations in compact sketch–photo representations.

Table 10. Compressor sensitivity on SketchyCOCO-SL (p. 20, lines 1072–1079). We vary one parameter at a time from E3 (768D). All tables give three-seed means ± sample SD (%).

SettingR@1R@5R@10
Default (0.70,0.08,1)(0.70,0.08,1)86.67±2.0898.10±0.4899.68±0.27
ρ=0.60\rho=0.6086.51±2.6298.25±1.2099.37±0.27
ρ=0.80\rho=0.8086.51±2.7197.94±0.9999.05±0.95
β=0.04\beta=0.0485.24±1.4397.46±0.9999.21±0.73
β=0.16\beta=0.1688.73±1.2097.46±0.7398.89±0.73
γ=0.5\gamma=0.590.95±2.1898.89±0.2799.37±0.27
γ=2\gamma=282.54±2.2596.67±0.4899.05±0.48

Over the tested range, ρ\rho barely affects R@1, β\beta trades off cutoffs, and γ\gamma affects R@1 most. At γ=0.5\gamma=0.5, R@1 rises 86.67→90.95 and R@5 98.10→98.89, while R@10 dips. At γ=2\gamma=2, all three fall. Strong decay may suppress cues that distinguish same-category instances with similar responses in SL. We retain the defaults (A.11, pp. 20–21, lines 1065–1086).

Table 11. Fusion sensitivity (p. 21, lines 1095–1107). We rescore E5 descriptors using Eq. (13), §3.4 (p. 6, lines 313–323), with checkpoints, queries, and galleries fixed. Weights are global/complementary/spatial. Equal weights reproduce E5.

DatasetWeightsR@1R@5R@10
SL(1/3,1/3,1/3)(1/3,1/3,1/3)92.70±0.2799.52±0.48100.00±0.00
(.50,.25,.25)(.50,.25,.25)92.54±0.7399.05±0.8299.84±0.27
(.25,.50,.25)(.25,.50,.25)92.38±1.2699.21±0.73100.00±0.00
(.25,.25,.50)(.25,.25,.50)92.38±0.4899.68±0.2799.84±0.27
ShoeV2(1/3,1/3,1/3)(1/3,1/3,1/3)48.90±1.6578.03±1.2085.79±0.71
(.50,.25,.25)(.50,.25,.25)47.05±0.4377.18±1.3185.34±1.22
(.25,.50,.25)(.25,.50,.25)48.55±1.7577.98±0.4386.24±0.61
(.25,.25,.50)(.25,.25,.50)47.10±0.8877.23±1.2284.83±0.69
User-disjoint FS-COCO(1/3,1/3,1/3)(1/3,1/3,1/3)65.34±1.0884.62±0.3089.67±0.12
(.50,.25,.25)(.50,.25,.25)63.20±1.1583.03±0.4388.77±0.20
(.25,.50,.25)(.25,.50,.25)66.73±0.8185.32±0.0590.18±0.20
(.25,.25,.50)(.25,.25,.50)62.30±0.8082.44±0.4688.27±0.44

Upweighting the complementary component improves all cutoffs in every FS-COCO seed (mean R@1 65.34→66.73). On ShoeV2 it raises R@10 and lowers R@1 in every seed. SL is least sensitive. Maximal absolute mean R@1 shifts are 0.32/1.85/3.04 points on SL/ShoeV2/FS-COCO. We retain prespecified equal weights (A.12, p. 21, lines 1088–1124).

Table 12. Spatial-stage freezing schedule (p. 22, lines 1155–1162). Schedules share E3 parents, budgets, splits, and final-checkpoint evaluation (1024D). After two frozen epochs, the alternative unfreezes the stem and Stages 1–3. Stage 4 and semantic readouts stay frozen.

DatasetScheduleR@1R@5R@10
User-disjoint FS-COCOFrozen (E5)65.34±1.0884.62±0.3089.67±0.12
Partial unfreeze65.77±0.5284.49±0.1089.28±0.10
SLFrozen (E5)92.70±0.2799.52±0.48100.00±0.00
Partial unfreeze94.92±0.9999.37±0.27100.00±0.00
ShoeV2Frozen (E5)48.90±1.6578.03±1.2085.79±0.71
Partial unfreeze45.80±1.4375.98±0.5484.33±1.36

Partial unfreezing improves SL R@1 in all three seeds (92.70→94.92). FS-COCO gains slightly at R@1 and loses at R@5/10. ShoeV2 drops at all cutoffs, consistent with possible overfitting or feature drift (5,982 sketches, 1,800 distinct photos). Effects vary by dataset and cutoff. (A.13, pp. 21–22, lines 1126–1174).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.