- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper extends Rex-Omni, a coordinate-token perception MLLM, to sketches in two directions: perceiving on a sketch (detection, pointing, object/spatial referring, affordance) and using a sketch as a prompt to find the one drawn instance in a photo. Stage A converts labelled photos into HED edge maps and trains a LoRA "sketch path" whose attention relations (phrase-to-patch and patch-to-patch) are aligned to those of the frozen "photo path". Stage B builds instance-level pairs from Sketchy photos tiled with same-category distractors and from HED crops of Objects365 objects. The authors also annotate 2754 sketch-to-photo instance links on FS-COCO. Sketch-Omni reaches 91.4 [email protected] on FS-COCO instance VP versus 81.8 for Qwen3-VL-8B, and is best in most cells across six tasks.
Critical strengths
-
The work is solid and carefully executed. The data constructions are described in detail, train/test overlap with SketchyCOCO is filtered at both category and instance level, and the authors openly discuss the limits of SketchyCOCO and annotate a new FS-COCO benchmark with a clear protocol.
-
The evaluation goes beyond headline numbers. The Easy/Hard split by number of same-category instances (Table 2, Table 6) and the IoU 0.9 results directly test the claims, and the w/o-align ablation is reported on all tasks.
Critical weaknesses
-
Evaluation is close to the training distribution. Referring and affordance are evaluated only on HED edge maps, the same kind of input used in training, while baselines have never seen edge maps; HED maps are also not free-hand sketches. On FS-COCO, queries are crops from the scene sketch at an object box, which matches the Objects365 HED-crop construction used in Stage B. This may explain why the lead is 9.6 F1 on FS-COCO but only 4.5 on SketchyCOCO.
-
The instance-level claim rests on a small subset. 2452 of the 2754 FS-COCO queries (89%) are Easy, where the photo has only one instance of the category and the category alone is enough. Only 302 queries test picking one instance among several.
-
The gain mostly comes from data construction, not from the proposed alignment. The w/o-align model already beats every baseline in every cell, and alignment adds only 0.9–2.9 F1 on instance VP and 0.4–4.3 on sketch-native tasks, from single runs without variance.
Questions for authors
How does the model perform on referring or affordance with free-hand sketches rather than HED maps?
2 comments
Thank you again for asking how Sketch-Omni behaves on free-hand sketches and whether it relies on the crop format. Table 9 (App. E) reports the free-hand referring results, and Table 7 (App. C) reports the test with cut strokes removed. We would welcome any further thoughts on these experiments.
We thank the reviewer for the careful reading and the detailed comments. The questions on free-hand sketches led us to new experiments, which we report below, and we answer the three weaknesses in turn.
W1 & Q1. Free-hand referring and affordance: To answer this question, we build a free-hand referring set. FS-COCO sketches depict COCO photographs, and RefCOCO, RefCOCO+ and RefCOCOg annotate many of these photographs with referring expressions. Our sketch-to-photo links move each expression from its target in the photograph to the sketch object. Keeping only expressions and photographs not used in our training gives 302 expressions for 68 sketch objects in 52 sketches (App. E, Table 9):
The 3B Sketch-Omni stays first in every metric and outperforms the 27B Qwen3.8. It learns referring from HED sketches only and has seen no free-hand sketch with a referring label, yet it still improves on its base model Rex-Omni by 10.6 F1 at IoU 0.5. For affordance, to our knowledge no free-hand sketch dataset carries affordance labels, so we conduct a qualitative experiment on further free-hand sketches (App. E, Figure 12).
W1. FS-COCO sketch crops: We appreciate this careful analysis. A crop at the box of a sketched object cuts the strokes that cross its border. To test whether Sketch-Omni relies on them, we delete every stroke that the crop cuts and keep all strokes that lie fully inside it. Sketch-Omni changes by at most 0.2 F1 at IoU 0.5:
Its accuracy therefore does not come from the crop format. This experiment is now included in App. C (Table 7). Two further results use queries that are not crops of a scene sketch. SketchyCOCO queries are standalone object sketches, and Sketch-Omni is still the best model on SketchyCOCO-Hard (Table 2). In the new study of Sec. 4.3 (Table 3), each query is a free-hand sketch drawn while looking at the target, and Sketch-Omni reaches 54.4 F1 at IoU 0.5 in scenes that hold at least five instances of the target's category. The smaller margin on SketchyCOCO comes instead from its sketches, which were drawn for other photographs and do not strictly match the target in shape (Figure 13).
W2. Size of the instance-level evidence: Beyond the 302 FS-COCO-Hard queries, where Sketch-Omni leads the newly added Claude Sonnet 5.5 by 10.5 F1 at IoU 0.5, SketchyCOCO contributes 651 Hard queries, too. Its sketches match the target less closely, but Sketch-Omni still achieves the best performance there. The new study in Sec. 4.3 adds 160 cluttered scenes, where the target is one of at least five instances of its category. This setting is harder than the Hard subsets, and Sketch-Omni with a free-hand sketch still locates the target at 54.4 F1 at IoU 0.5, 20.1 above a narrative description written and grounded by Qwen3.8-27B. The evidence that Sketch-Omni picks one instance among several therefore covers more than 1,100 queries in total and is not limited to the 302 FS-COCO-Hard queries.
W3. Contribution of the alignment: The constructed supervision indeed brings most of the absolute gain, and it is part of our contribution. The alignment transfers the relations the model forms on a photograph to the corresponding sketch. On FS-COCO, it widens the margin over Claude Sonnet 5.5 from 1.0 to 3.8 F1 at IoU 0.5 (Table 1). It also improves every one of the 14 ablation cells in Tables 1 and 11, and its largest gain of 4.3 F1 falls on object referring, which is consistent with its role of aligning how a phrase selects its region.
On variance, we agree that multiple seeds would be better. Two further seeds of the full and w/o-align models need at least 4,000 GPU hours, which we cannot complete within the rebuttal period. Two facts reduce the concern. Decoding is greedy, so the evaluation itself is deterministic. The gain also has the same sign in all 14 cells, which span all six tasks, both instance VP benchmarks and three IoU thresholds, and it is largest on the task the alignment targets most directly. We will state in the paper that the ablation reports single runs.
We hope these results resolve the three concerns, and we are glad to clarify anything further.