acceptodds
Under review as a conference paper at ICLR 2027

Sketch-Omni: Instance-Level Sketch-Based Visual Prompting for Perception MLLMs

16 views

Abstract

Promptable perception, from GroundingDINO to Rex-Omni, puts one model behind one prompt interface and returns coordinates for any localisation task. Every model in that line was built for photographs; none has been given a sketch. We present Sketch-Omni, to our knowledge the first perception MLLM built to work on sketches and from them. On a sketch, the sketch is the image, and text prompts detect, point, resolve object and spatial referring expressions, and ground affordances in it. From a sketch, the sketch is the prompt, and a drawing of one object localises that object in a photograph, even among several of the same category, a query that neither a category name nor a photo exemplar can pose. Two obstacles stand in the way. A photo-trained model's attention is tuned to colour and texture, which a sketch does not have, and no dataset pairs a drawing with the one instance it depicts among others of its kind. We build the training data instead of collecting it. An edge detector turns every labelled photograph into a sketch-like drawing that keeps the photograph's labels, and two constructions produce sketch-to-photo instance pairs from existing data. On top of this, photo-guided relation alignment passes each photograph and its edge sketch through the same model and trains the sketch pass to reproduce how the photograph pass relates phrases to patches and patches to each other. Not one training label is drawn by hand. Sketch-Omni reaches 91.4 F1 on sketch-prompted localisation against 81.8 for the strongest general-purpose MLLM, with a lead that doubles when the photograph holds several instances of the queried category, and it is best in 39 of 40 evaluation cells across six tasks. Code, model, and a hand-annotated benchmark of 2754 sketch-to-photo links will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sketch, Visual Prompting, Multimodal LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UrYuOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper extends Rex-Omni, a coordinate-token perception MLLM, to sketches in two directions: perceiving on a sketch (detection, pointing, object/spatial referring, affordance) and using a sketch as a prompt to find the one drawn instance in a photo. Stage A converts labelled photos into HED edge maps and trains a LoRA "sketch path" whose attention relations (phrase-to-patch and patch-to-patch) are aligned to those of the frozen "photo path". Stage B builds instance-level pairs from Sketchy photos tiled with same-category distractors and from HED crops of Objects365 objects. The authors also annotate 2754 sketch-to-photo instance links on FS-COCO. Sketch-Omni reaches 91.4 [email protected] on FS-COCO instance VP versus 81.8 for Qwen3-VL-8B, and is best in most cells across six tasks.

Critical strengths
  1. The work is solid and carefully executed. The data constructions are described in detail, train/test overlap with SketchyCOCO is filtered at both category and instance level, and the authors openly discuss the limits of SketchyCOCO and annotate a new FS-COCO benchmark with a clear protocol.

  2. The evaluation goes beyond headline numbers. The Easy/Hard split by number of same-category instances (Table 2, Table 6) and the IoU 0.9 results directly test the claims, and the w/o-align ablation is reported on all tasks.

Critical weaknesses
  1. Evaluation is close to the training distribution. Referring and affordance are evaluated only on HED edge maps, the same kind of input used in training, while baselines have never seen edge maps; HED maps are also not free-hand sketches. On FS-COCO, queries are crops from the scene sketch at an object box, which matches the Objects365 HED-crop construction used in Stage B. This may explain why the lead is 9.6 F1 on FS-COCO but only 4.5 on SketchyCOCO.

  2. The instance-level claim rests on a small subset. 2452 of the 2754 FS-COCO queries (89%) are Easy, where the photo has only one instance of the category and the category alone is enough. Only 302 queries test picking one instance among several.

  3. The gain mostly comes from data construction, not from the proposed alignment. The w/o-align model already beats every baseline in every cell, and alignment adds only 0.9–2.9 F1 on instance VP and 0.4–4.3 on sketch-native tasks, from single runs without variance.

Questions for authors

How does the model perform on referring or affordance with free-hand sketches rather than HED maps?

2 comments

We thank the reviewer for the careful reading and the detailed comments. The questions on free-hand sketches led us to new experiments, which we report below, and we answer the three weaknesses in turn.

W1 & Q1. Free-hand referring and affordance: To answer this question, we build a free-hand referring set. FS-COCO sketches depict COCO photographs, and RefCOCO, RefCOCO+ and RefCOCOg annotate many of these photographs with referring expressions. Our sketch-to-photo links move each expression from its target in the photograph to the sketch object. Keeping only expressions and photographs not used in our training gives 302 expressions for 68 sketch objects in 52 sketches (App. E, Table 9):

Model[email protected][email protected]F1@avg
Rex-Omni61.839.949.9
Qwen3.8-27B68.946.957.3
Sketch-Omni (3B)72.448.359.2

The 3B Sketch-Omni stays first in every metric and outperforms the 27B Qwen3.8. It learns referring from HED sketches only and has seen no free-hand sketch with a referring label, yet it still improves on its base model Rex-Omni by 10.6 F1 at IoU 0.5. For affordance, to our knowledge no free-hand sketch dataset carries affordance labels, so we conduct a qualitative experiment on further free-hand sketches (App. E, Figure 12).

W1. FS-COCO sketch crops: We appreciate this careful analysis. A crop at the box of a sketched object cuts the strokes that cross its border. To test whether Sketch-Omni relies on them, we delete every stroke that the crop cuts and keep all strokes that lie fully inside it. Sketch-Omni changes by at most 0.2 F1 at IoU 0.5:

QueryFS-COCOFS-COCO-Hard
Original crop91.480.3
Cut strokes removed91.280.4

Its accuracy therefore does not come from the crop format. This experiment is now included in App. C (Table 7). Two further results use queries that are not crops of a scene sketch. SketchyCOCO queries are standalone object sketches, and Sketch-Omni is still the best model on SketchyCOCO-Hard (Table 2). In the new study of Sec. 4.3 (Table 3), each query is a free-hand sketch drawn while looking at the target, and Sketch-Omni reaches 54.4 F1 at IoU 0.5 in scenes that hold at least five instances of the target's category. The smaller margin on SketchyCOCO comes instead from its sketches, which were drawn for other photographs and do not strictly match the target in shape (Figure 13).

W2. Size of the instance-level evidence: Beyond the 302 FS-COCO-Hard queries, where Sketch-Omni leads the newly added Claude Sonnet 5.5 by 10.5 F1 at IoU 0.5, SketchyCOCO contributes 651 Hard queries, too. Its sketches match the target less closely, but Sketch-Omni still achieves the best performance there. The new study in Sec. 4.3 adds 160 cluttered scenes, where the target is one of at least five instances of its category. This setting is harder than the Hard subsets, and Sketch-Omni with a free-hand sketch still locates the target at 54.4 F1 at IoU 0.5, 20.1 above a narrative description written and grounded by Qwen3.8-27B. The evidence that Sketch-Omni picks one instance among several therefore covers more than 1,100 queries in total and is not limited to the 302 FS-COCO-Hard queries.

W3. Contribution of the alignment: The constructed supervision indeed brings most of the absolute gain, and it is part of our contribution. The alignment transfers the relations the model forms on a photograph to the corresponding sketch. On FS-COCO, it widens the margin over Claude Sonnet 5.5 from 1.0 to 3.8 F1 at IoU 0.5 (Table 1). It also improves every one of the 14 ablation cells in Tables 1 and 11, and its largest gain of 4.3 F1 falls on object referring, which is consistent with its role of aligning how a phrase selects its region.

On variance, we agree that multiple seeds would be better. Two further seeds of the full and w/o-align models need at least 4,000 GPU hours, which we cannot complete within the rebuttal period. Two facts reduce the concern. Decoding is greedy, so the evaluation itself is deterministic. The gain also has the same sign in all 14 cells, which span all six tasks, both instance VP benchmarks and three IoU thresholds, and it is largest on the task the alignment targets most directly. We will state in the paper that the ablation reports single runs.

We hope these results resolve the three concerns, and we are glad to clarify anything further.

Thank you again for asking how Sketch-Omni behaves on free-hand sketches and whether it relies on the crop format. Table 9 (App. E) reports the free-hand referring results, and Table 7 (App. C) reports the test with cut strokes removed. We would welcome any further thoughts on these experiments.

Reviewer_LvbeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors built Sketch-Omni multimodal large language model using photo-guided relational alignment and supervision from photographs like boxes, phrases, affordance labels, etc. They quantitatively and qualitatively analyzed Sketch-Omni against open-source models on affordance pointing, spatial referring, object referring, detection + pointing, and cross-modal and intra-modal relations. Sketch-Omni performs best across six tasks.

Critical strengths
  • Quantitative results are supported by qualitative experiments.
  • The related work section is explained in detail.
  • Training details with dataset are shared.
Critical weaknesses
  • The models compared with Sketch-Omni are only open-sourced and are previous versions. As the performance evaluation does not involve SOTA models, the comparison does not provide a clear conclusion.
  • The Introduction part requires revision for better writing style.
  • The use of sketches but not photographs is not clearly described in the paper. Therefore, the motivation of the study needs more explanation and clarity.
Questions for authors
  • Some part of the writing is not clear and Introduction need better academic writing.
  • I believe Figure 2 shows a misleading example regarding the photo. As the prompt says, "detect all such objects," the result returns every instance of a human being, but in the instance-level example, the prompt requests the specified object. With these differences making an inference of that photograph provides category-level but sketch returns instance-level is not accurate.
  • The sketch-native tasks represented in Figure 1 can be performed with photographs as well. The authors should explain the study's main idea, why they prefer sketches, and what sketches can provide that photographs cannot.
  • Although the instance-level task can be done using text prompting with a narrative of the target object/scene, why do we need to use sketches? As MLLMs are better at textual performance than visual performance, they can perform better on text-based instance-level tasks rather than sketches. Can you conduct a study comparing photographs vs. text vs. sketches for model performance comparison?
  • The evaluated models on instance-level sketches are all open-sourced and outdated versions. The evaluation of current proprietary models like Claude or GPT can provide a better performance comparison, same as current version of Qwen model (3.8).
  • What are the test dataset the authors used for each task and any of these samples included in training set?
7 comments

We thank the reviewer for the constructive feedback. All the requested comparisons have been run, and we address each concern below.

W1 & Q5. Newer and proprietary models: As suggested, we have added Qwen3.8-27B and Claude Sonnet 5.5 on instance-level sketch-based visual prompting, the main task of the paper, and evaluated them on the Easy and Hard subsets of SketchyCOCO and FS-COCO. We report F1 at IoU 0.5:

ModelSketchyCOCO AllSketchyCOCO-EasySketchyCOCO-HardFS-COCO AllFS-COCO-EasyFS-COCO-Hard
Qwen3-VL-8B66.590.247.881.884.363.1
Qwen3.8-27B69.294.542.784.086.067.7
Claude Sonnet 5.573.195.449.687.689.869.8
Sketch-Omni (3B)71.089.551.491.492.780.3

Sketch-Omni outperforms all the other models on FS-COCO and on the two Hard subsets. On the Hard subsets a model must use the shape of the sketch to tell the target from other objects of the same category, and Sketch-Omni beats Claude Sonnet 5.5 by 10.5 F1 on FS-COCO-Hard. Our training recipe thus lets a 3B model on an older backbone (Qwen2.5-VL) outperform a recent 27B open-source model and a frontier commercial model.

The other models score higher on SketchyCOCO-Easy, and Claude Sonnet 5.5 is also ahead on SketchyCOCO overall. Each SketchyCOCO-Easy photograph holds only one object of the queried category, so recognising the category is enough to find it. SketchyCOCO sketches were also drawn for other photographs and do not strictly match the target in shape, so the shape that Sketch-Omni is trained to follow is not a reliable cue there. This subset is therefore not a strict test of instance-level localisation. Examples are shown in Figure 13 (App. F). FS-COCO sketches are drawn from the target scene instead, and Sketch-Omni outperforms Claude Sonnet 5.5 by 2.9 F1 even on FS-COCO-Easy. Both models are now included in Tables 1 and 2 of the revised paper.

Q2. Figure 2: Thank you for raising this point. The two panels in Figure 2 follow the standard text prompt of each task. Rex-Omni asks the model to "find all similar objects in the image" for photo-based visual prompting, because the photo exemplar serves as a reference for the kind of object. Changing the prompt would not make the photo exemplar instance-level: an exemplar of a different person carries no information about which person is meant. To specify one instance, it would have to be another photo of exactly that object. Such a photo is hard to obtain in practice, and it is not provided in datasets such as Objects365. A sketch is instead drawn for one specific object from a user's mind. Like text, it is a convenient way for the user to provide cues about the object to be located, and its outline and pose single out the target from other objects of the same category. The text prompt for sketch-based visual prompting asks for that specific object accordingly. The prompts differ because the two tasks differ. We have revised the caption of Figure 2 to make this clear.

Q4. Comparison with photographs and narrative texts: That is a good point, and we have run the requested study. Narrative text can specify a single instance. It works well when the target has something easy to name, such as a distinctive colour or a salient position. However, many objects in a cluttered scene may share these attributes, and describing one of them so that it stands apart from the others is then not always intuitive. What still sets the target apart is often its outline, pose or layout, which a hand-drawn sketch can capture with a few strokes.

We sampled 160 cluttered scenes from the Objects365 validation set. In each scene the target is one of at least 5 instances of its category. We query it in three ways:

  • Photo exemplar. A photo of another object of the same category, chosen at random in the scene, because Objects365 provides no second photograph of the target (Q2). This is the category-level reference.
  • Narrative text. Qwen3.8-27B sees the photo with the target outlined, describes the target in at most 40 words, and then locates the target in the original photo with its own description. The description is thus written and grounded by the same recent 27B model.
  • Free-hand sketch. A sketch drawn while looking at the target.

Each kind of query is evaluated with a specialist model for it:

QueryModel[email protected][email protected][email protected]
Photo exemplarRex-Omni14.212.911.7
Narrative textQwen3.8-27B36.134.330.5
Free-hand sketchSketch-Omni56.254.446.2

The photo exemplar also retrieves the other objects of the same category and has the lowest F1, consistent with its category-level role (Q2). Narrative text does better but often locates another object of the same category by mistake. The free-hand sketch achieves the highest F1, 20.1 above narrative text at IoU 0.5, because its outline and pose single out the target directly.

The two are also complementary. The sketch locates 52 of the 105 targets that the text misses, the text locates 20 of the 73 targets that the sketch misses, and together they locate 107 of the 160 targets. Text works well for targets with a distinctive attribute, and a sketch extends localisation to targets that are easier to draw than to describe. The study is in Sec. 4.3 (Table 3) of the revised paper, with examples in Figure 6.

W3 & Q3. Why we study sketches: Sketching is a universal form of communication that people of any age or skill can use with each other [A]. A few strokes can express how a person subjectively sees an object, including fine-grained shape such as outline and pose that is hard to describe in words. A sketch is therefore a sparse and abstract modality of its own that differs fundamentally from a photograph [A]. The sketch-native tasks have photo counterparts, but they are needed when there is no photograph to work on. Design drafts and storyboards exist only as drawings, and a model that understands this modality must detect, point and refer on the sketch itself. A sketch also serves as a prompt that complements narrative text and photo exemplars, as the study above shows (Q2, Q4). Yet MLLMs have explored this modality far less. Sketch-Omni is built to understand sketches both as inputs and as prompts.

Q6. Test data: We confirm that the test data of each task is separated from its training data.

  • Detection and pointing are tested on the FS-COCO test split and HED sketches of COCO val2017.
  • Object and spatial referring are tested on the RefCOCOg val and test splits.
  • Affordance is tested on the AGD20K Seen and Unseen test splits.
  • Instance VP is tested on the SketchyCOCO validation scenes and on the FS-COCO test split.

SketchyCOCO takes its object sketches from Sketchy, which is part of our training data. To avoid any overlap, we remove all SketchyCOCO categories when training on Sketchy, and we also remove any other training sketch that appears as a query in SketchyCOCO. This category- and instance-level filtering is also noted by Reviewer UrYu. We have added these details to Sec. 4.1 of the revised paper.

As a further check, FS-COCO and RefCOCOg both build on COCO photographs. 200 of the 1,923 FS-COCO test photographs used for instance VP are also RefCOCOg's training images, which we use only to train referring on their HED sketches. FS-COCO sketches are never used in training. We re-evaluated instance VP on FS-COCO after excluding these photographs, which leaves 2,422 of the 2,754 sketch-to-photo links. We report F1 at IoU 0.5:

ModelAll linksNon-overlapping links
Qwen3-VL-8B81.883.0
Qwen3.8-27B84.084.8
Claude Sonnet 5.587.688.3
Sketch-Omni (3B)91.492.1

Sketch-Omni reaches 92.1 F1 on the non-overlapping links and still outperforms Claude Sonnet 5.5 by 3.8 F1, the same margin as on all links. The overlap therefore gives Sketch-Omni no advantage.

W2 & Q1. Writing of the Introduction: We appreciate this suggestion. We have revised the wording of the Introduction for a more formal tone (marked in blue).

We hope the new comparisons give the clear conclusion the reviewer asked for, and we are glad to clarify anything further.

[A] Xu et al. Deep Learning for Free-Hand Sketch: A Survey. TPAMI, 2023.

Reviewer_Fy4UOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a new framework called Sketch-Omni. This framework includes tasks such as (1) identifying the corresponding bounding box in a photograph based on a sketch image as the query input, and tasks that use a sketch image as input and a natural language query to perform several tasks including object detection and pointing. They synthesized sketch images corresponding to photographs and trained an MLLM capable of understanding sketch images while considering the alignment loss between the photograph and the sketch image.

Critical strengths
  • The authors comprehensively trained and evaluated multiple tasks using sketch images.
  • As a result of training the model using synthesized data, performance improvements were observed compared to baseline models in both settings where sketch images were used as input and where they were used as queries.
Critical weaknesses
  • While there have been several individual studies on sketch image recognition and treating sketch images as queries as described in the related work section, this work can be viewed as an integration of those multiple tasks. On the other hand, the model, data construction steps, and task setup rely heavily on existing methods, so the novelty is limited.
Questions for authors
  • In the implementation details in Section 4.2, why are Stages A and B trained alternately? Would training them all together not work well?
  • How exactly were the 2,754 manually annotated sketch-to-photo links created?
  • Regarding the part where sketches are automatically synthesized from photos, can any photo be successfully converted into a sketch? Have you conducted any quantitative evaluations to determine the differences between the automatically synthesized sketches and those drawn by humans?
  • By training the model on sketches alongside photo images, are there any secondary effects, such as improved object detection performance on photo datasets?
4 comments

Thank you for the thoughtful review. We first address the comment on novelty and then answer each question in turn.

Novelty. The novelty is a new capability rather than an integration of existing tasks. Promptable perception models accept a category name or a photo exemplar as the prompt, and neither can ask for one particular object among several of its kind. Sketch-Omni lets a sketch be that prompt: a drawing of one object localises that object in a photograph. No prior perception MLLM does this, and no dataset pairs a drawing with the one instance it depicts among others of the same category, so the task, the training and the evaluation all had to be built:

  • Direction. Sketching is a universal form of communication among people and can serve as another language between people and machines [A]. A few strokes can express how a person subjectively sees an object, including fine-grained shape such as outline and pose that is hard to describe in words. Yet MLLMs have explored this modality far less.
  • Unified model. Sketch research has been dominated by task-specific models. Sketch-Omni lets a perception MLLM understand sketches as a modality of its own, reading them as inputs and using them as queries, and integrates the sketch tasks into one model.
  • Training. To address the scarcity of labelled sketch data, we construct the training data from labelled photographs without any new label. We also propose photo-guided relation alignment, a training strategy that transfers the relations the model forms on a photograph to its sketch. The standard components (HED, LoRA, the Rex-Omni base) are used deliberately, to show that labelled photographs alone can supervise sketch understanding.
  • Evaluation. We introduce, to our knowledge, the first benchmark of hand-annotated sketch-to-photo instance links, with 2,754 links.

The result is that a 3B model outperforms the newly added Qwen3.8-27B and the commercial Claude Sonnet 5.5 on FS-COCO, by 10.5 F1 over Claude Sonnet 5.5 when the photograph holds several objects of the queried category (Table 2). As Reviewer UrYu's summary also describes, the paper extends perception MLLMs to sketches in two directions, on a sketch and from a sketch.

Q1. Alternating Stages A and B. The two stages are trained together, in one run with one optimiser and one set of LoRA weights. Only the batches alternate, because the training settings of the two stages differ. A Stage-A step also passes the photograph through the frozen model to compute the alignment loss, while a Stage-B step uses next-token prediction alone. The alternation lets all samples in a batch have the same forward passes and losses. We have added this explanation to App. H.

Q2. The 2,754 links. Two people annotated the FS-COCO test split and a third checked every annotation. Every sketched object was boxed by hand and linked to the photographed instance it depicts. A link is kept only when the sketch crop alone distinguishes that instance from the other objects in the photograph. The full procedure is included in App. C.

Q3. Synthesized versus human sketches. Yes, any photograph can be converted. Our conversion uses HED, which turns any photograph into a sketch that stays pixel-aligned with it. The sketch is not only meant to imitate human drawing but also inherits the original labels. To quantify how close HED sketches are to human drawing, we measured their FID to the free-hand sketches of FS-COCO with Inception and CLIP features, using the photographs as the reference:

FID (Inception)FID (CLIP)
HED sketches126.981.6
Photographs229.4159.1

The HED sketches are much closer to free-hand sketches than the photographs are, at about half the FID under both features. This comparison is now included in App. E (Table 8). An additional experiment in App. E (Table 9) shows that training referring on HED sketches also improves the model's understanding of free-hand sketches, with a gain of 10.6 F1 at IoU 0.5 over its base model Rex-Omni.

Q4. Photo detection. This is an insightful point. Sketches emphasise shape over texture, and StructXLIP [B] shows that such structural cues can enhance vision-language models. Testing whether sketches bring improvement to MLLMs on photographs is therefore a natural future direction. Our current work focuses on sketch understanding, so Sketch-Omni is trained specifically on sketch inputs and sketch queries, not alongside photo tasks. Such a study would require joint training on sketch grounding and photo grounding tasks, which goes beyond the scope of this work.

[A] Xu et al. Deep Learning for Free-Hand Sketch: A Survey. TPAMI, 2023.

[B] Ruan et al. StructXLIP: Enhancing Vision-language Models with Multimodal Structural Cues. CVPR, 2026.

Thank you again for your comments on novelty and your questions on the alternating training, the 2,754 links and the comparison between HED and human sketches. The revised paper explains the alternation in App. H, describes the annotation procedure in App. C, and reports the FID comparison in App. E (Table 8). We would be glad to answer any further questions you may have.

Thank you for the detailed response. It has addressed some of my concerns, but I still have questions about the motivation for the task setting in Instance-Level Sketch-Based Visual Prompting. The task appears to use highly detailed sketches of objects. For example, Figure 4 includes a sketch detailed enough to distinguish among several similar objects. I find it difficult to identify practical situations in which someone would draw such a detailed sketch to locate an object in an image. If a sketch is detailed enough to distinguish among multiple objects without ambiguity, it seems likely that the person has already seen the image, and I am not sure what benefit the sketch query provides in that case. Could you further explain the practical value of using sketches as queries and add this explanation to the paper?

Thank you very much for the follow-up and for pinpointing the remaining question. We agree that the paper should state clearly when a sketch query is the right tool, and we answer in three parts.

1. A sketch is the right query when the target is in mind but not in view. We agree that the user has usually seen the target. What they lack is the image being searched in front of them, and the sketch carries the target as they remember it. Prior work has motivated sketch queries by three such settings:

  • Finding a particular object seen earlier. A shopper who has seen a particular shoe can sketch it more easily than describe it in a long text, which motivated instance-level sketch-based image retrieval [A]. Instance VP adds what retrieval does not give, the location of that one instance among others of its kind in a photograph, such as a crowded shop shelf.
  • One query, many images. To crop the target as an exemplar, the user would first have to find it in one of the images, which is the search itself. A sketch is drawn before the search begins and applies to every frame of a video or every photo of a collection, as in sketch-based video object localisation [B].
  • Instructing a robot. A user who instructs a robot does not see the robot's camera view. RT-Sketch [C] uses sketches to specify the robot's goal, since language can be ambiguous and goal images are over-specified, and finds them more robust when language is ambiguous or distractors are present.

2. In cluttered scenes, a sketch singles out the target better than a narrative text or a photo exemplar. When the user is looking at the image being searched, a click or a box is the simplest prompt, and we do not claim that a sketch should replace it. When the image is not in front of the user, the other options are a narrative text and a photo exemplar. The study added for Reviewer Lvbe (Sec. 4.3, Table 3) compares the three on 160 cluttered scenes in which the target is one of at least five instances of its category. A photo exemplar of another instance reaches 12.9 F1 at IoU 0.5, since it indicates the category. Even the exemplar that looks most like the target, chosen in DINOv2 features with the target known, reaches 29.6 with Qwen3.8-27B (see our follow-up to Reviewer Lvbe). A narrative text written by Qwen3.8-27B reaches 34.3, and the sketch 54.4. The narrative text and the sketch were both made with the target in view, so this comparison measures how well each kind of query conveys the target. FS-COCO covers the memory setting.

3. The sketches need not be detailed. An instance-level query has to show what sets the target apart from other objects of its category, which is mostly its outline and pose. A few strokes are usually enough for this. In Figure 6, the jellyfish is drawn as a wavy line and a loop, and the cabinet as a rectangle with one line and two dots. Sketch-Omni locates both among several objects of their kind. Prior work on fine-grained retrieval likewise shows that a partial sketch is often enough to identify one instance [D]. The FS-COCO queries in Figure 4 are also rough. They are crops of scene sketches drawn by non-experts from memory, after viewing the photograph for 60 seconds and within 3 minutes per scene, a limit set to keep the drawings "recognizable but not overly detailed" [E]. In the fourth row, a crouching figure of a handful of strokes is enough to tell the catcher from the standing players. To test how much of such a sketch is needed, we kept only the first 25%, 50% or 75% of the strokes of each FS-COCO query, in the order they were drawn. We report F1 at IoU 0.5:

Strokes kept25%50%75%100%
FS-COCO72.985.290.591.4
FS-COCO-Hard58.771.480.480.3

Without its last quarter of strokes, a query locates the target nearly as well as the full sketch. With half of its strokes, Sketch-Omni still outperforms Qwen3.8-27B given the full sketch (85.2 against 84.0) and, on FS-COCO-Hard, Claude Sonnet 5.5 (71.4 against 69.8; Tables 1–2).

Changes to the paper. As suggested, we will add this explanation to the Introduction and the stroke study to the appendix.

We hope this clarifies the practical value of sketch queries. If it resolves your remaining concern, we would be grateful if you would consider updating your score.

[A] Yu et al. Sketch Me That Shoe. CVPR, 2016.

[B] Woo et al. Sketch-Based Video Object Localization. WACV, 2024.

[C] Sundaresan et al. RT-Sketch: Goal-Conditioned Imitation Learning from Hand-Drawn Sketches. CoRL, 2024.

[D] Bhunia et al. Sketch Less for More: On-the-Fly Fine-Grained Sketch-Based Image Retrieval. CVPR, 2020.

[E] Chowdhury et al. FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context. ECCV, 2022.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.