acceptodds
Under review as a conference paper at ICLR 2027

Consensus-Anchored Retrieval with Cross-Modal Alignment and Semantic Calibration for Sketch-Based Person Re-Identification

Abstract

Sketch-based person re-identification is challenged by both the sketch–photo modality gap and strong sketch subjectivity, as sketches of the same identity may exhibit substantial style-dependent variations. Existing methods mainly fuse multiple sketches or suppress style discrepancies, but do not explicitly model the latent identity consensus shared across style-biased observations. We therefore propose a consensus-anchored retrieval framework for robust sketch-based person re-identification. Its core component, Style-Conditioned Prototype-Fusion Consensus (SPFC), jointly learns an identity-consensus prototype and a query-adaptive retrieval descriptor through two complementary style-conditioned estimators. Two auxiliary training objectives further support this representation: Global-to-Local Mutual Alignment (GLMA) uses the global descriptor of each modality to guide local aggregation in the other, while Semantic Affinity-Guided Contrastive Calibration (SACC) uses auxiliary identity-level semantics available only during training to calibrate ambiguous negatives. Inference remains entirely visual. Extensive experiments under single-query, multi-query, and cross-style protocols demonstrate the effectiveness and generalizability of the proposed framework. Upon acceptance, we will publicly release the source code and pretrained checkpoints on GitHub.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sketch-based person re-identification, Style-conditioned learning, Cross-modal retrieval, Representation learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZvbgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper primarily investigates the sketch-to-person re-identification task, focusing on addressing the modality gaps between sketches and real photographs, as well as the style variations that arise when different individuals sketch the same target. To this end, the authors propose a retrieval framework based on identity consensus. The SPFC module employs two independent branches to learn, respectively, the shared features across sketches belonging to the same identity and discriminative features for final retrieval; it further utilizes leave-one-out contrastive learning to enhance feature consistency among sketches of the same individual. The GLMA module leverages global features from one modality to guide the aggregation of local features from another modality, thereby improving the matching performance between sketches and photographs. The SACC module utilizes textual description information from the training phase to assist the model in better distinguishing between pedestrians who share similar appearances but belong to different identities. The authors conducted experiments on the Market-Sketch-1K and PKU-Sketch datasets, achieving promising results across single-query, multi-query, and cross-style evaluation settings. Ablation studies also demonstrated that each module contributed to performance improvements. However, the paper currently lacks more direct experimental evidence to quantify the specific contribution of the identity consensus mechanism, and certain configurations in the cross-style experiments require further elaboration

Critical strengths
  1. The paper addresses style variations in sketch-based person re-identification. The research objective is clear and has practical relevance.

  2. SPFC learns the identity-consensus prototype and retrieval features separately, which helps reduce potential interference from directly fusing different sketches. The ablation results also support the benefits of this design.

  3. GLMA and SACC improve the model through cross-modal local feature alignment and hard negative learning, respectively, without introducing additional cross-modal inference overhead.

Critical weaknesses
  1. When training with only a single sketch style, the SPFC fusion weight is always 1, preventing the weighting branch from being effectively trained. The paper does not clearly explain how the model learns to handle unseen sketch styles under this setting.

  2. SPFC assigns weights mainly based on the features of individual sketches, without explicitly modeling the agreement between different sketches. It remains unclear whether the module actually learns identity consensus.

  3. In the PKU-Sketch dataset, each identity has only one sketch, making leave-one-out multi-sketch consensus learning unavailable. Therefore, the performance improvements on this dataset cannot directly demonstrate the effectiveness of the core consensus mechanism.

  4. Some experiments could be more comprehensive. The mAP improvement under the single-query setting is relatively small, but no multi-run stability analysis is provided. In addition, the cross-style multi-query experiments lack comparisons with stronger methods such as OLTM.

Questions for authors
  1. When the training data consists of only one sketch style, the fusion weight of SPFC remains consistently 1. In such cases, how does the model learn fusion weights for different styles? And how does it handle unseen styles during inference?

  2. SPFC primarily assigns weights based on the intrinsic features of each individual sketch; how does the model determine which features constitute shared identity characteristics across different sketches?

  3. In the PKU-Sketch dataset, each identity has only one sketch, making multi-sketch consensus learning infeasible. Could the authors further clarify which components primarily contribute to the performance improvement observed on this dataset?

  4. The improvement in mAP for the single-query task reported in the paper appears limited; were multiple repeated experiments conducted to verify the robustness of these results?

5 comments
AuthorsOpenReview ↗

Response to Reviewer–Part 1: Unseen-style generalization and identity consensus

Thank you for your careful review and for recognizing the practical relevance of addressing sketch-style variation, the separate prototype and retrieval branches in SPFC, and the complementary roles of GLMA and SACC. In response to your comments, we have clarified the mechanism and scope of consensus learning, added a visualization of cross-style feature consistency and repeated-run statistics, and expanded the cross-style MQ comparisons. We address each concern below.

W1 / Q1. Single-style training and unseen-style generalization.

Thank you for highlighting the need to clarify this setting. We agree that training with only one drawing style provides no observations of differences between styles and therefore cannot directly supervise how different drawing styles should be weighted. Our single-style experiments examine transfer under limited training-style coverage, rather than a dedicated mechanism for learning unseen styles.

In Market-Sketch-1K SQ training, we retain the prototype-based supervision used by the framework. Training preferentially samples same-identity sketches from different styles; when only one original sketch is available, it instead uses independently augmented views of that sketch. Although these views share the same style label, their visual features can differ. Consequently, the prototype weighting branch can still receive training signals, and its normalized weights need not be uniform. The resulting supervision encourages consistency across augmented views, without introducing additional drawing styles.

At inference, an unseen-style sketch is processed by the same visual encoder and retrieval path. Under SQ, its descriptor is directly compared with gallery photo descriptors, so handling the unseen style relies on transfer of the learned visual representation. The encoder is initialized from CLIP and trained with identity and cross-modal supervision; no additional adaptation to the held-out style is performed. Under MQ, the existing fusion branch combines the query features, but its use does not establish that single-style SQ training has learned reliable fusion weights for unseen styles. We have clarified these distinctions in Appendix C.3 and identified reliable aggregation under limited training-style coverage as a direction for future research.

W2 / Q2. How SPFC learns identity consensus.

Thank you for asking us to clarify how consensus is learned. Each sketch is indeed scored from its own visual features and style information, without explicitly computing pairwise agreement. However, the prototype weighting branch is trained through objectives relating multiple same-identity observations. Individual scoring therefore does not imply isolated per-sketch learning.

Specifically, Lintra\mathcal L_{\mathrm{intra}} requires an excluded sketch to identify the prototype constructed from its same-identity peers against other identity prototypes. This jointly trains the visual encoder and prototype scorer to capture evidence that remains predictive of identity across observations. Meanwhile, Lpp\mathcal L_{\mathrm{pp}} aligns the prototype with same-identity photos, and Lrp\mathcal L_{\mathrm{rp}} transfers the prototype constraint to the retrieval descriptor. Thus, shared identity information is learned through cross-sketch consistency and cross-modal identity discrimination, rather than specified by the scoring function alone.

Table 4 provides controlled evidence for this contribution, with warm-up, GLMA, and SACC fixed. With uniform retrieval fusion, Prototype Only improves over Mean Fusion by 2.08 mAP and 2.61 Rank-1 points. Without explicit style conditioning, retaining the prototype branch and its objectives improves over Attention Fusion by 1.58 and 3.17 points, respectively. These comparisons support the benefit of prototype-based supervision beyond learned retrieval weighting or explicit style conditioning.

We have additionally included a visualization and analysis in Appendix D.2. For the illustrated identity, six sketches with visibly different drawing styles exhibit coherent deep representations: all off-diagonal cosine similarities exceed 0.52, with B–C and E–F reaching 0.807 and 0.808. This illustrates cross-style consistency within the same identity in the learned feature space. The figure also shows different prototype and retrieval-fusion weight distributions, complementing the ablations by displaying the two branches’ aggregation behavior.

AuthorsOpenReview ↗

Response to Reviewer–Part 2: PKU-Sketch analysis, stability, and cross-style comparisons

W3 / Q3. How the framework operates on PKU-Sketch.

Thank you for highlighting this distinction. We agree that PKU-Sketch, with one sketch per identity, does not directly evaluate multi-sketch consensus learning. In this setting, the prototype reduces to the individual sketch representation, and the leave-one-out objective is inactive. The remaining framework, however, continues to learn sketch–photo retrieval through complementary supervision.

Specifically, CLIP initialization provides a general representation foundation, while the visual contrastive warm-up establishes an initial sketch–photo alignment. Subsequent identity and cross-modal objectives further refine the retrieval representation. GLMA models local sketch–photo correspondence, while SACC provides training-only semantic supervision to calibrate ambiguous negatives. The remaining prototype-related objectives also retain single-sketch representation consistency and photo alignment. These mechanisms remain applicable without multiple same-identity sketches and jointly support the reported PKU retrieval performance.

PKU-Sketch therefore provides a complementary assessment of the framework’s applicability when multiple sketches are unavailable. We have clarified this experimental scope in Sec. 4.3 and explained the active learning mechanisms in Appendix C.1.

W4 / Q4. Multi-run stability and stronger cross-style comparisons.

Thank you for highlighting the need to verify the modest SQ mAP improvement through repeated experiments. We conducted nine additional independent training runs for each protocol, yielding ten runs per protocol, including the run reported in the main comparison tables. All runs use the same experimental settings as the main paper. We have added the summary statistics and stability analysis in Appendix C.6.

Results are reported as mean ± sample standard deviation (%):

ProtocolmAPRank-1Rank-5Rank-10mINP
SQ41.37 ± 0.1639.85 ± 0.2465.02 ± 0.4875.82 ± 0.3831.22 ± 0.26
MQ69.81 ± 0.3974.88 ± 0.5893.98 ± 0.5797.41 ± 0.2256.64 ± 0.74

The SQ and MQ mAP values in Table 1, 41.34% and 69.84%, are close to the corresponding ten-run means. SQ mAP ranges from 41.19% to 41.69%, with a standard deviation of 0.16 percentage points. Together with the limited variation in the other retrieval metrics, these results support the run-to-run stability of the reported performance.

We also agree that OLTM provides a strong and relevant text-guided cross-modal alignment baseline. Following your suggestion, we have added its published MQ-training/MQ-testing results to Table C.5, alongside BDG, CGC, and our method. Our method leads on four of the six configurations reported by OLTM, while OLTM leads on the other two. We have also expanded the analysis in Appendix C.3 to discuss how training-style coverage, drawing-specific correlations, and the identity cues preserved in unseen queries may contribute to these differences.

We thank you again for your constructive comments, which have helped us make the evaluation more complete and explain the learning mechanism more clearly. The additional analyses and experiments strengthen the evidence for prototype-based retrieval and its run-to-run stability, while clarifying the role of each evaluation setting. We hope these revisions address your concerns and provide a clearer basis for reassessing the contribution of the work.

Your questions about identity-consensus learning and the single-sketch setting have been helpful in refining our explanation and evaluation of the framework.

In the revised manuscript, we connect the ablation evidence with a new feature-consistency visualization, clarify single-style training and the PKU-Sketch setting, and provide repeated-run statistics and cross-style MQ comparisons with OLTM.

Before the discussion closes, we would be grateful if you could take a look at these additions and let us know whether any concerns remain. We are happy to discuss them further and make appropriate revisions. Thank you again for your thoughtful feedback and time.

Reviewer_oqtKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a consensus-anchored retrieval framework for robust sketch-based person re-identification. The authors model sketches of the same identity as style-biased observations of a latent identity. Its core component, Style-Conditioned Prototype-Fusion Consensus (SPFC), jointly learns an identity-consensus prototype and a query-adaptive retrieval descriptor through two complementary style-conditioned estimators. Two auxiliary training objectives further support this representation: Global-to-Local Mutual Alignment (GLMA) uses each modality's global descriptor to guide local aggregation in the other, while Semantic Affinity-Guided Contrastive Calibration (SACC) uses identity-level semantics available only during training to calibrate ambiguous negatives. Extensive experiments under single-query, multi-query, and cross-style protocols demonstrate the framework's effectiveness and generalizability.

Critical strengths

This paper proposes a consensus-anchored retrieval framework that unifies sketches of the same identity across different styles as style-biased observations of a latent identity, which is conceptually more aligned with the core challenge of sketch subjectivity than traditional approaches that simply fuse multiple sketches or suppress style discrepancies. The SPFC jointly learns an identity-consensus prototype and a retrieval descriptor through style-conditioned mechanisms, where prototype construction and retrieval aggregation can use their respective independent learning objectives to adapt to different task requirements, and inference is entirely visual without requiring text or additional modality inputs. Moreover, the two auxiliary objectives GLMA and SACC are each focused on different aspects: GLMA targets fine-grained local cross-modal alignment, while SACC calibrates semantically ambiguous negatives during training, making them complementary without adding inference-time overhead.

Critical weaknesses
  1. SACC uses the CLIP text encoder to encode attribute descriptions to obtain identity-level semantic representations, which means that training requires the participation of the text modality (or attribute annotations). Although inference is entirely visual, the training stage requires corresponding text/attribute annotations, which may not be available in certain practical datasets (e.g., sketch datasets with no attribute annotations). It is recommended that the paper more clearly specify SACC's annotation requirements and their limitations in practical deployment.

  2. SPFC uses a learnable style embedding to conditionally weight sketches of different styles. This means that training requires known style labels. For datasets where styles are unknown or unannotated, the method cannot be applied (or would require additional style annotation cost). The Introduction mentions that "sketches of the same identity may exhibit substantial style-dependent variations," but the paper does not discuss how the method performs when style information is unavailable. It is recommended that the limitations section discuss the impact of missing style labels.

  3. Although Table 1 and Table 2 compare with various methods, the cross-style scenario is only systematically compared with FreeStyle. Additionally, FreeStyle outperforms the proposed method in 7 out of 15 cross-style splits, indicating that the current method still has room for improvement in more robust cross-style generalization. Meanwhile, comparisons with training-related methods (e.g., generation-based methods, attention-fusion-based methods) are not comprehensive — some SOTA methods (such as certain diffusion or GAN-based sketch generation methods) are not included. Moreover, as shown in Table 5 (cross-style SQ comparison), among the 15 valid splits, our method outperforms FreeStyle in mAP on 8 splits, while FreeStyle leads in the other 7 splits (FreeStyle leads in all ABC-trained splits). Cross-style generalization is one of the core challenges in sketch Re-ID (since training and testing style sets differ), and the current cross-style results show a notable gap compared to the SQ/MQ protocol results. It is recommended that the authors discuss the sources of the cross-style performance gap and potential mechanisms for improvement.

  4. Although sketch Re-ID datasets are inherently small in scale, the paper does not discuss the method's scaling behavior on larger sketch datasets, nor does it validate the visual descriptor's generality on larger cross-modal Re-ID datasets (e.g., CUHK-PEDES, IUPDB). It is recommended that the authors at least discuss the method's scalability.

  5. Using multiple sketches may lead to high cost for real-world application. It might be somewhat difficult to get accurate and high quality sketches. It may also cause biased consensus due to the poor quality when some of the sketches shares the similar flaws.

Questions for authors

Please refer to the Weakness.

6 comments
AuthorsOpenReview ↗

Response to Reviewer–Annotation Requirements and Style-Conditioning Ablation(Part 1/3)

Thank you for your careful review and for recognizing the value of the identity-consensus formulation and the complementary roles of SPFC, GLMA, and SACC. In response to your comments, we have added a controlled style-conditioning ablation, broadened the comparisons, and clarified the annotation requirements, cross-style behavior, scalability, and practical scope of the framework.

W1. Existing public annotations support an auxiliary training objective. For Market-Sketch-1K, we verbalize the public 27-dimensional Market-1501 identity attributes with a fixed template. For PKU-Sketch, we sample existing English descriptions from each identity's description pool. No new per-sketch captions are collected, and only training identities contribute semantics. These inputs support SACC during training; retrieval uses no text.

When annotations are unavailable, SACC can be omitted: Table 3 already reports SPFC+GLMA without SACC at 69.19% mAP and 73.69% Rank-1, versus 69.84% and 74.50% for the full model. We have clarified the annotation requirements and processing in Sec. 4.1 and Appendix B.1, and their practical implications in Sec. 5 (Limitations).

W2. Explicit style labels improve SPFC, while both aggregation branches also work without label-conditioned scoring. In response to your concern about missing style information, we conducted an additional controlled ablation and added the results to Table 4. We retain the complete SPFC and its objectives, and set the style-embedding inputs to both independent weighting heads to zero during training and evaluation. Warm-up, GLMA, and SACC remain fixed. The updated Table 4 is reproduced below.

VariantPrototypeUses style embeddingFusion weightsmAPRank-1
Mean Fusion——Uniform66.3569.48
Attention Fusion——Learned67.0569.32
Style Fusion—✓Learned67.7070.88
Prototype Only✓✓Uniform68.4372.09
SPFC w/o Style Cond.✓—Learned68.6372.49
Shared Weights✓✓Shared68.96‾\underline{68.96}73.29‾\underline{73.29}
Full SPFC✓✓Learned69.8474.50

SPFC w/o Style Cond. reaches 68.63% mAP and 72.49% Rank-1. Compared with Attention Fusion, retaining the prototype branch adds 1.58 and 3.17 points. Restoring explicit style conditioning adds another 1.21 and 2.01 points.

Thus, the new ablation isolates the benefit of explicit style labels within the complete SPFC. Visual style variation remains in the images, and the original query grouping and valid-sketch masks are preserved. We have added the result to Table 4, analyzed it in Sec. 4.4, and discussed its scope and the implications of missing style labels in Sec. 5 (Limitations).

AuthorsOpenReview ↗

Response to Reviewer–Expanded Comparisons and Cross-Style Generalization(Part 2/3)

W3. Comparison coverage and cross-style generalization. Thank you for encouraging broader comparisons and a deeper analysis of cross-style generalization.

We have expanded Tables 1–2 by adding MSIF to the PKU-Sketch comparison and the sketch-only results of InteractReID to both datasets, while bringing additional PKU-Sketch baselines from Appendix C.1 into Table 2. We have expanded the cross-style SQ comparison in Table 5 to include CGC and OLTM alongside FreeStyle and our method; Appendix C.2 further includes BDG and AIP.

The comparisons cover the generative and attention-based approaches highlighted in your comment, including CGC and DALNet for generative or auxiliary-modality learning, and MSIF and BDG for attention-based fusion. Together with methods addressing semantic alignment and style generalization, these comparisons assess complementary approaches to the modality gap and sketch subjectivity. Table 4 further examines the contribution of consensus supervision and independent prototype/retrieval weighting through controlled ablations with the same backbone.

We have also deepened the discussion in Sec. 4.4 and Appendix C.2. Consensus learned from a limited set of styles may retain drawing-specific correlations shared by those observations. Unseen strokes, abstraction patterns, or omitted details can then change the visibility and reliability of identity cues, weakening the transfer of consensus supervision. In SQ, this concerns how training-time consensus shapes a descriptor extracted from a single sketch at inference. GLMA and SACC strengthen cross-modal correspondence and identity discrimination, but neither explicitly trains this transfer across held-out styles. Cross-style evaluation also reduces available training observations, so the current results do not isolate this effect from style shift.

To improve transfer, we discuss constructing prototypes from a subset of training styles and supervising queries from excluded styles, using the same style split across identities. Excluding query styles from prototype-consistency targets would extend the current sample-level leave-one-out objective toward style-level generalization. Identity-preserving stroke or abstraction perturbations, combined with descriptor consistency and retrieval supervision, offer a complementary direction for broadening style coverage while retaining discriminative identity details. These are our next research directions for strengthening consensus transfer to unseen styles.

AuthorsOpenReview ↗

Response to Reviewer–Scalability, Practical Scope, and Consensus Limitations (Part 3/3)

W4. Scalability and broader applicability. Thank you for highlighting the need to examine scalability beyond the current sketch benchmarks. We agree that computational scaling and applicability to larger, more diverse datasets deserve explicit discussion.

We have added an analysis of training and retrieval costs. Sec. 4.4 now explains that, at fixed image resolution, query encoding and aggregation costs grow approximately linearly with the number of sketches. Appendix B.5 further explains that gallery descriptors can be precomputed, with feature storage and per-query exact similarity computation scaling as O(Gd)O(Gd) for gallery size GG and descriptor dimension dd. Prototypes and semantic affinities are computed within mini-batches, while larger datasets increase the number of training updates and additional training identities enlarge the classification head.

We have also made broader evaluation and transfer explicit future directions. Our empirical evaluation currently covers Market-Sketch-1K and PKU-Sketch; computational scaling alone does not establish performance on larger datasets. CUHK-PEDES uses natural-language queries, so applying the framework there would require adapting the query modality and associated supervision. The Limitations and Future Work paragraph in Sec. 5 identifies evaluation on larger sketch datasets and extension to other multimodal person-retrieval settings as worthwhile next steps.

W5. Sketch acquisition cost and biased consensus. Thank you for drawing attention to the practical cost of obtaining sketches and the possibility of shared drawing errors. We agree that poor-quality sketches, especially those with similar omissions or incorrect identity details, can bias consensus. Agreement among sketches does not necessarily imply accuracy.

Multiple sketches are optional at inference. The Market-Sketch-1K SQ results in Table 1 and the PKU-Sketch results in Table 2 demonstrate retrieval with one query sketch, while MQ uses additional observations when available. As clarified in Appendix A.3, Market-Sketch-1K SQ training still uses same-identity peer sketches for consensus supervision; PKU-Sketch evaluates the single-sketch formulation. We have made the distinction between SQ deployment and optional MQ aggregation explicit in Sec. 5.

We have explicitly discussed the risk of correlated errors. The weighting heads are trained for retrieval and do not provide calibrated sketch-quality estimates. Leave-one-out supervision prevents direct anchor self-inclusion but does not guarantee correction of errors shared by the remaining sketches. We have therefore expanded Sec. 5 (Limitations and Future Work) to acknowledge that shared errors can bias consensus even when sketches agree, and to identify reliability-aware aggregation for correlated sketch errors as a future research direction.

The additional ablation and expanded comparisons strengthen the evidence for consensus-guided visual retrieval, while the revisions clarify the framework’s annotation requirements and practical scope. We thank you again for your constructive comments, which have helped make the paper more complete and deepen our analysis of the framework’s capabilities and limitations. We hope these additions address your concerns and provide a clearer basis for reassessing the contribution of the work.

Reviewer_E6tVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies sketch-based person re-identification under subjectivity differences among sketches and the sketch-photo modality gap. It introduces a CLIP-ViT-based framework with a five-epoch visual contrastive warm-up. The central Style-Conditioned Prototype-Fusion Consensus (SPFC) module uses separate style-conditioned weighting heads to construct a same-identity sketch prototype and a retrieval-oriented fusion descriptor, supported by prototype consistency and leave-one-out contrastive supervision. Global-to-Local Mutual Alignment (GLMA) guides patch aggregation using global features from the paired modality. Semantic Affinity-Guided Contrastive Calibration (SACC) uses training-identity text attributes to strengthen semantically similar negative pairs. Text features are not used for retrieval, although multi-query fusion relies on provided style labels. Experiments on Market-Sketch-1K and PKU-Sketch report competitive single-query (SQ), multi-query (MQ), and cross-style results, with component ablations and additional analyses.

Critical strengths
  1. The problem is meaningful and the central representation distinction is well motivated. Different sketches of one person can emphasize or distort different visual cues. Separating an identity-consensus prototype from a query-adaptive retrieval descriptor is a reasonable response to this problem. The leave-one-out loss in Eq. (7) also attempts to prevent the anchor sketch from directly reconstructing itself in that particular contrastive objective.
  2. The reported retrieval results are strong. On Market-Sketch-1K MQ, the method reaches 69.84 mAP and 74.50 Rank-1, compared with 62.55 and 69.48 for OLTM in Table 1. The SQ setting achieves 41.34 mAP and 40.21 Rank-1. On PKU-Sketch, the method reports 94.21 mAP and 94.40 Rank-1 (Table 2), which are competitive with the strongest listed methods. These results justify further investigation of the approach.
  3. Several experiments help clarify which design choices matter. Table 3 distinguishes visual warm-up, SPFC, GLMA, and SACC, and Table 4 compares uniform fusion, learned attention, style-conditioned fusion, prototypes, shared weighting, and independent weighting. In particular, independent prototype and fusion weights improve MQ mAP from 68.96 to 69.84 over the shared-weight variant. The supplementary material also gives cross-style results, backbone comparisons, hyperparameter sensitivity, and distance distributions.
Critical weaknesses
  1. Small performance margins and between-method comparability would benefit from uncertainty reporting. The Market-Sketch-1K SQ mAP margin over FreeStyle is just 0.18 percentage points, and the PKU-Sketch Rank-1 margin over CGC is 0.40 points. Section 4.1 states that PKU-Sketch averages ten trials, but no dispersion is provided; the paper also lacks cross-seed variability for Market-Sketch-1K. The method uses CLIP pretraining, a dedicated visual warm-up, and training-only semantic annotations. The internal Table 3 ablations control the backbone, but Tables 1 and 2 do not establish which compared methods use identical auxiliary supervision, backbone, query composition, or optimization budgets. This does not show that the published comparisons are unfair; it limits the strength of claims about the reliability and cause of small gains.
  2. Can the authors provide component/loss-level ablations on Market-Sketch-1K SQ and PKU-Sketch, especially the K = 1 setting? How much of the single-sketch improvement persists when the prototype-related losses are removed with all other paths and training settings fixed? I think it is important to show the proposed method do works well in this condition.
  3. The treatment of unseen styles in the MQ protocol is insufficiently specified. Equation (3) computes prototype and fusion weights from visual descriptors and learned style embeddings, whereas Appendix C.2 evaluates on style sets excluded from training. Under the specified MQ sampling rule, training on style A alone yields one valid sketch per query set. For K = 1, both softmax weights equal one, so the scoring heads and style embeddings receive no learning signal through this weighting operation. Yet Table C.3 reports substantial improvements for A-to-EF and other unseen multi-style settings. The manuscript does not state how embeddings of unseen style categories are initialized, updated, or replaced, or how the untrained weighting heads behave when K increases at test time.
Questions for authors

Please refers to the weaknesses. I think this method is not attractive enough when consider from previous works in this topic. The improvements in both method and experiments may be somewhat limited. So, I tend to reject it based on its current version, but will consider the acceptance again after rebuttal.

6 comments
AuthorsOpenReview ↗

Response to Reviewer – Part 1: Reliability and comparison conditions

Thank you for your detailed and constructive review, and for recognizing the motivation behind our identity-consensus formulation, the distinction between consensus prototypes and retrieval representations, and the competitive retrieval results. Your comments helped us identify where stronger experimental support and clearer protocol descriptions were needed. In response, we have added repeated-run statistics, a summary of comparison conditions, component ablations under SQ and single-sketch settings, and explicit initialization and inference details for cross-style evaluation. These revisions have made the empirical evaluation more complete and the scope of our conclusions clearer. We address your concerns below.

W1. Reliability of gains and comparability of evaluation conditions.

Thank you for highlighting the need for uncertainty reporting and clearer comparison conditions. We have expanded the supplementary material to address both the reliability of the reported results and the evidence supporting the contributions of our design.

We have added multi-run statistics in Appendix C.6 (Table C.8). On Market-Sketch-1K, we conduct ten independent training runs under each of the SQ and MQ protocols, including the runs reported in Table 1, with the same experimental settings as the main paper. On PKU-Sketch, we report dispersion over the ten evaluation trials: each trial uses 150 training identities and 50 testing identities, with no training–testing identity overlap within that trial and different testing identity sets across trials. The main metrics are summarized below as mean ± sample standard deviation (%).

DatasetProtocolmAPRank-1
Market-Sketch-1KSQ41.37 ± 0.1639.85 ± 0.24
Market-Sketch-1KMQ69.81 ± 0.3974.88 ± 0.58
PKU-SketchSQ94.21 ± 2.4794.40 ± 4.40

The SQ and MQ mAP values in Table 1, 41.34% and 69.84%, are close to the corresponding ten-run means. Together with the limited Rank-1 variation, these results support the repeatability of our Market-Sketch-1K performance. On PKU-Sketch, the dispersion reflects different identity splits and their associated training runs. Each trial contains only 50 sketch queries, so one additional correct top-1 match changes Rank-1 by two percentage points. Appendix C.6 also reports Rank-5, Rank-10, and mINP statistics for completeness.

We appreciate the distinction between performance comparisons and controlled evidence for their causes. We have added Table C.2 in Appendix C.1, reproduced below, to summarize the visual backbones, pretraining, auxiliary supervision, and additional training data of the methods in Tables 1 and 2. This makes the comparison conditions more explicit: OLTM and InteractReID use CLIP-ViT-B/16, as does our method, whereas FreeStyle uses CLIP-ViT-B/32 and draws on the same attribute-annotation sources as ours. These methods nevertheless differ in how they construct and use semantic supervision. Our attribute/text information is restricted to training identities, and retrieval requires no text input. SQ and MQ results are reported separately, and the InteractReID results use sketch-only retrieval. We also specify our five-epoch visual warm-up and 50-epoch main training schedule in Appendix B.4; the published baselines retain their method-specific optimization settings.

MethodVisual backbonePretrainingAuxiliary supervisionAdditional data
DCLNetResNet-50ImageNetVisualNR
DSCNetResNet-50ImageNetVisualNR
DEENResNet-50ImageNetVisualNR
BDGResNet-50ImageNet; CLIP textAttributes (Market); PKU: NRNR
SketchTrans+ViT-Base/16ImageNet; pretrained generatorGenerated A-sketchesScribble (generator)
OLTMViT-B/16CLIP; ViLT-VQAVQA-generated attributesNR
CGCViT-BaseNRGenerated proto-style sketchesNR
InteractReIDViT-B/16CLIP; task adaptationMultimodal adaptationTri-PEDES
KTCAAResNet-50ImageNet (code)NRNo Market-1501
FreeStyleViT-B/32CLIPAttribute textNR
CDAFLGoogLeNetImageNet; Re-ID pretrainingVisualMarket-1501
LMDIVGG-16ImageNetVisualNo extra Re-ID pretraining
CDACResNet-50ImageNet-1KVisualNo extra Re-ID pretraining
MSIFResNet-50NRVisualNo extra Re-ID pretraining
DALNetResNet-50ImageNet; U²-NetGenerated auxiliary modalityU²-Net source: NR
OursViT-B/16CLIPAttributes/text; style labelsNo extra image dataset

Additional data are listed separately from generic ImageNet/CLIP initialization. Visual denotes supervision derived from sketch/photo inputs without attribute or text annotations; common identity labels are omitted. NR denotes details not reported in the cited sources.

AuthorsOpenReview ↗

Response to Reviewer – Part 2: Comparison analysis and SQ/PKU ablations

We have also expanded the accompanying analysis to clarify what these comparisons support. For example, our SQ mAP is close to FreeStyle’s (41.34% vs. 41.16%), while our Rank-1 is 5.19 percentage points higher and FreeStyle achieves higher Rank-5. We therefore describe our advantage more specifically as improved top-1 matching, rather than a uniform improvement across ranking metrics. Tables 1 and 2 compare complete retrieval systems under their respective training configurations; the repeated-run statistics assess the stability of our results, while the controlled ablations provide more direct evidence for the contributions of our design.

Specifically, Table 4 holds the backbone, visual warm-up, GLMA, and SACC fixed under the Market-Sketch-1K MQ protocol. Prototype Only improves over Mean Fusion by 2.08 mAP and 2.61 Rank-1 percentage points, while independent prototype and retrieval-fusion weights improve over Shared Weights by 0.88 and 1.21 points, respectively. These controlled results support prototype-based supervision and the separation of the two weighting objectives under MQ. The revised analysis thus relates the system-level comparisons to controlled evidence without attributing the differences between published methods solely to an individual component.

W2. Component contributions under SQ and K=1K=1 settings.

Thank you for suggesting additional ablations under single-query and single-sketch conditions. We have added component ablations on Market-Sketch-1K SQ and PKU-Sketch in Appendix C.2, Table C.3, reproduced below. Within each protocol, the visual backbone, single-sketch retrieval path, and optimization settings are held fixed apart from the indicated warm-up and supervision choices. Disabling SPFC removes its associated supervision while retaining the retrieval path. All PKU-Sketch configurations are evaluated on the same ten identity splits.

SettingWarm-upSPFCGLMASACCSQ mAPSQ Rank-1PKU mAPPKU Rank-1
(a)————35.8533.4688.6988.00
(b)✓———37.2635.1590.8690.20
(c)✓✓——39.0937.5191.9392.00
(d)✓—✓—38.8936.2992.8992.80
(e)✓——✓38.5736.5892.2493.40
(f)✓✓✓—40.6439.4992.9593.20
(g)✓✓—✓41.0340.1792.7593.40
(h)✓—✓✓40.4439.7593.7093.60
(i)✓✓✓✓41.3440.2194.2194.40

SQ denotes Market-Sketch-1K SQ. All results are percentages; PKU-Sketch results are averaged over ten trials.

On Market-Sketch-1K SQ, adding SPFC to the warmed baseline improves mAP and Rank-1 by 1.83 and 2.36 percentage points. Although only one sketch is available at inference, training can exploit same-identity sketches across styles to regularize the individual retrieval representation. With warm-up, GLMA, and SACC retained, the model without SPFC supervision achieves 40.44% mAP and 39.75% Rank-1; enabling SPFC further improves these to 41.34% and 40.21%. These results support the contribution of SPFC’s prototype-related design to single-query retrieval.

PKU-Sketch directly examines the K=1K=1 condition: each identity contributes only one valid sketch, so the prototype reduces to that sketch’s deep feature anchor. Multi-sketch aggregation and leave-one-out supervision are inactive, while feature-consistency and cross-modal alignment objectives remain applicable. With the other components retained, removing SPFC supervision leaves 93.70% mAP and 93.60% Rank-1, compared with 94.21% and 94.40% for the full model. The substantial performance retained without SPFC is supported by identity/triplet training, visual warm-up, GLMA, and SACC. The additional gains from SPFC reflect its remaining single-sketch supervision, rather than multi-sketch consensus aggregation.

The full configuration achieves the highest mAP and Rank-1 under both settings. We have added the corresponding analysis in Appendix C.2 to clarify the component contributions and the role of SPFC under different sketch-availability conditions.

AuthorsOpenReview ↗

Response to Reviewer – Part 3: Unseen-style embeddings and inference

W3. Unseen-style embeddings and aggregation in cross-style evaluation.

Thank you for pointing out the need to specify the handling of unseen-style embeddings and aggregation heads. We have added initialization and inference details in Appendix C.3. Our cross-style evaluation examines transfer to styles excluded from training, with all model parameters fixed during testing.

All style embeddings, including entries for held-out styles, are independently initialized from N(0,0.022)\mathcal N(0,0.02^2) before training. The prototype and fusion scoring heads are independently initialized using their linear layers’ default initialization. Held-out embeddings receive no corresponding style-specific training supervision. At inference, each query retains its original style label and indexes the corresponding entry in the existing embedding table. For example, E/F queries use the E/F entries; these are neither reinitialized, replaced with the A-style embedding, nor adapted using test data. The fixed fusion head scores each query sketch using its visual descriptor and corresponding style embedding, and normalizes these scores across the available sketches according to Eq. (3). Increasing the number of query sketches changes the inputs and normalization set, without updating any model parameters. No uniform-fusion fallback is applied.

We agree that genuine singleton aggregation yields a unit weight and no learning signal through that weighting operation. For the reported A→EF result, however, the checkpoint comes from SQ training, where the prototype aggregates independently augmented views of the same A-style sketch. The prototype branch can therefore learn augmentation consistency, and its weights need not be uniform. This should be distinguished from the independent retrieval-fusion head, which does not receive effective multi-query fusion supervision during SQ training.

With multiple styles under MQ training, both scoring heads receive supervision from observed sketch sets, although excluded-style embeddings still lack corresponding supervision. Thus, the cross-style results report retrieval behavior under the specified fixed inference configuration. In particular, A→EF does not establish that single-style SQ training learns reliable unseen-style fusion. The visual encoder learns representations through identity and cross-modal objectives during training and is applied without parameter updates at test time. Reliable aggregation under limited training-style coverage remains a direction for further research.

We sincerely appreciate the opportunity to strengthen the paper through your feedback, as well as your willingness to reconsider the work in light of the rebuttal. Your comments have helped us better substantiate the component contributions, characterize performance variability, and explain the behavior and boundaries of cross-style evaluation. We hope the additional evidence and clarifications address your concerns and provide a stronger basis for reassessing the contribution of separating identity-consensus learning from retrieval-oriented representation construction. Thank you again for your time and thoughtful consideration.

Comments on the paper

1 comment
Area_Chair_KiznOpenReview ↗

Please review the authors’ responses and provide feedback

Dear Reviewers,

The authors have submitted their responses. Please review them carefully, assess whether they address your concerns, and leave a follow-up comment for the authors.

Thank you for your time and thoughtful feedback.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.