- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces sketch–text guided textured 3D model retrieval. In this task, a freehand sketch constrains an object’s geometry and a text description constrains its appearance, such as material, color, texture, and style. The authors argue that the two modalities complement each other: sketches capture structure but cannot express appearance, while text describes appearance but only loosely constrains geometry.
To support the task, they construct ST-Tex3D, a tri-modal benchmark for chairs and lamps that pairs freehand sketches with textured 3D models and retrieval-oriented descriptions. Each geometry appears with multiple texture variants, so the benchmark tests whether a model can distinguish objects that share a shape but differ in appearance.
The authors also propose a unified retrieval framework. It enriches sparse sketches with distilled semantic knowledge, encodes the sketch–text query as a set of intent tokens, and matches these tokens against both geometric and texture evidence from each 3D candidate, adaptively weighting structure against appearance for each query. On ST-Tex3D, the method outperforms sketch-only, text-only, and simple fusion baselines, and ablations support the contribution of each component.
Critical strengths
A well-motivated new task and a benchmark to support it.
- Sketch-based 3D retrieval has so far defined relevance almost entirely by geometry. Text-based retrieval leaves part layout and proportion under-constrained. Treating sketch and text as complementary constraints (geometry from the sketch, appearance from the text) is a natural and practically relevant extension. The test split is disjoint in both geometry and texture, which is a sensible protocol. ST-Tex3D fills a real gap between sketch–shape datasets (SHREC, Doodle2Object) and text–3D datasets (Text2Shape, Cap3D), so it could serve as a starting point for later work.
A coherent architecture whose components are each ablated.
- Several design choices are well suited to the problem. LSSE addresses sketch sparsity without overwriting contour cues, since it adds semantics through a gated residual. The intent tokens with a per-intent geometry/texture gate (Eq. 8) let the model weight structure and appearance differently for each query, instead of using a fixed fusion. Removing any single component lowers performance on both categories, which suggests none of them is redundant.
Critical weaknesses
The benchmark design may allow shortcuts, which limits what the results show.
-
Each category has only 10 textures, applied to every geometry. After the 7:3 split, the test set contains only 3 textures per category.
-
The captions were generated by GPT-5 from a single rendered view together with the material label.
-
As a result, the appearance part of the task may reduce to recognizing one of 3 texture classes from fairly templated text, not matching open-ended appearance descriptions.
-
All descriptions are machine-generated, and no human-written or linguistically varied queries are evaluated. Only two categories are tested, each in a separate gallery. Appearance evidence comes from just 3 fixed views under fixed lighting.
-
Together, these factors make it unclear whether the gains reflect real geometry–appearance composition or exploitation of the controlled texture set. They also limit any claims about generalization or practical use. The paper acknowledges some of this in Sec. 6, but this is the main threat to the validity of the conclusions.
The comparisons are not strong enough to show the architecture is needed.
-
The two sketch+text baselines only fuse similarity scores. On chairs, the best one (Pivot-SBSR + CLIP-MV, R@1 18.38) is below the text-only 3DAlign-DAER (18.92). This suggests the fusion baselines are weak or under-tuned, not that the problem needs the proposed design.
-
A stronger learned-fusion baseline is missing, for example a joint sketch+text transformer trained with the same losses and matching head.
-
Composed image retrieval methods are neither adapted nor discussed, although they address closely related composition problems (e.g., TIRG, ComposeAE, CLIP4Cir).
-
Some sketch-only baselines are reconstruction methods (DeepSDF, PSGN), and their adaptation to retrieval is not explained in the main text.
-
No variance over seeds is reported. There is no failure-mode or sensitivity analysis, such as noisy sketches, sketch–text conflicts, or the behavior of the learned gate m.
Questions for authors
Questions
-
How many queries are in each test gallery? The reported recalls (e.g., R@1 = 29.12% on chairs) do not seem to correspond to an integer count over 720 chair queries. Are results averaged over several runs? If so, how many, and what is the variance?
-
Do the GPT-5 captions directly expose the texture or material label? What R@1 does a simple baseline achieve that predicts only the texture class from text and combines it with sketch-only geometry retrieval? This would show how much of the gain comes from texture-label recognition.
-
In the “w/o Texture-aware Encoding” ablation, how is appearance represented, and how is Ltex computed without texture tokens?
-
How do the CLIP ViT-B/16 sketch backbone and LSSE interact, both at training and at inference? What prompts and data split were used to distill the GPT-4o teacher, and were any test sketches seen during distillation?
-
The 3D representation depends on the query, so candidate embeddings cannot be precomputed and indexed. How does the reported ~509 ms per query scale with gallery size, and is this practical for larger or mixed-category galleries?
Suggestions
-
An evaluation on a small set of human-written sketch+text queries, or on mixed-category galleries, would greatly strengthen the claims.
-
An analysis of the learned gate m across structure-oriented and appearance-oriented queries would directly support the claimed adaptive behavior.