acceptodds
Withdrawn from ICLR 2027

Shape Meets Surface: Compositional Sketch-Text Retrieval of Textured 3D Objects

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sketches provide an intuitive way to express object geometry, but they are insufficient for specifying textured appearance such as material, color, pattern, and surface style. Text offers complementary appearance descriptions, yet it is often less precise in constraining fine-grained geometry. In this paper, we study a new task: sketch-text guided textured 3D model retrieval, where the goal is to retrieve a textured 3D model that simultaneously satisfies sketch-based geometric intent and text-based appearance intent. To support this task, we construct ST-Tex3D, a tri-modal benchmark containing 1,005 textured chair models and 555 textured lamp models, with aligned SVG sketches, ShapeNetCore meshes, texture variants, multi-view renderings, and retrieval-oriented textual descriptions. We further propose a unified retrieval framework that jointly models sketch structure, textual appearance, 3D geometry, and multi-view texture evidence. Specifically, the framework distills high-level semantics for sketch encoding, generates intent-aware sketch-text query tokens, extracts geometry and texture representations for each candidate 3D model, and performs query-aware evidence aggregation with fine-grained late-interaction matching. Experiments on ST-Tex3D validate the effectiveness of the proposed task formulation, dataset, and retrieval framework for texture- and style-sensitive 3D model retrieval. Our code and models will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sketch-Text Guided 3D Retrieval, Textured 3D Objects, Multimodal Representation Learning, Geometry-Appearance Alignment, Query-Aware Evidence Aggregation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_E5LzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces sketch–text guided textured 3D model retrieval. In this task, a freehand sketch constrains an object’s geometry and a text description constrains its appearance, such as material, color, texture, and style. The authors argue that the two modalities complement each other: sketches capture structure but cannot express appearance, while text describes appearance but only loosely constrains geometry.

To support the task, they construct ST-Tex3D, a tri-modal benchmark for chairs and lamps that pairs freehand sketches with textured 3D models and retrieval-oriented descriptions. Each geometry appears with multiple texture variants, so the benchmark tests whether a model can distinguish objects that share a shape but differ in appearance.

The authors also propose a unified retrieval framework. It enriches sparse sketches with distilled semantic knowledge, encodes the sketch–text query as a set of intent tokens, and matches these tokens against both geometric and texture evidence from each 3D candidate, adaptively weighting structure against appearance for each query. On ST-Tex3D, the method outperforms sketch-only, text-only, and simple fusion baselines, and ablations support the contribution of each component.

Critical strengths

A well-motivated new task and a benchmark to support it.

  • Sketch-based 3D retrieval has so far defined relevance almost entirely by geometry. Text-based retrieval leaves part layout and proportion under-constrained. Treating sketch and text as complementary constraints (geometry from the sketch, appearance from the text) is a natural and practically relevant extension. The test split is disjoint in both geometry and texture, which is a sensible protocol. ST-Tex3D fills a real gap between sketch–shape datasets (SHREC, Doodle2Object) and text–3D datasets (Text2Shape, Cap3D), so it could serve as a starting point for later work.

A coherent architecture whose components are each ablated.

  • Several design choices are well suited to the problem. LSSE addresses sketch sparsity without overwriting contour cues, since it adds semantics through a gated residual. The intent tokens with a per-intent geometry/texture gate (Eq. 8) let the model weight structure and appearance differently for each query, instead of using a fixed fusion. Removing any single component lowers performance on both categories, which suggests none of them is redundant.
Critical weaknesses

The benchmark design may allow shortcuts, which limits what the results show.

  • Each category has only 10 textures, applied to every geometry. After the 7:3 split, the test set contains only 3 textures per category.

  • The captions were generated by GPT-5 from a single rendered view together with the material label.

  • As a result, the appearance part of the task may reduce to recognizing one of 3 texture classes from fairly templated text, not matching open-ended appearance descriptions.

  • All descriptions are machine-generated, and no human-written or linguistically varied queries are evaluated. Only two categories are tested, each in a separate gallery. Appearance evidence comes from just 3 fixed views under fixed lighting.

  • Together, these factors make it unclear whether the gains reflect real geometry–appearance composition or exploitation of the controlled texture set. They also limit any claims about generalization or practical use. The paper acknowledges some of this in Sec. 6, but this is the main threat to the validity of the conclusions.

The comparisons are not strong enough to show the architecture is needed.

  • The two sketch+text baselines only fuse similarity scores. On chairs, the best one (Pivot-SBSR + CLIP-MV, R@1 18.38) is below the text-only 3DAlign-DAER (18.92). This suggests the fusion baselines are weak or under-tuned, not that the problem needs the proposed design.

  • A stronger learned-fusion baseline is missing, for example a joint sketch+text transformer trained with the same losses and matching head.

  • Composed image retrieval methods are neither adapted nor discussed, although they address closely related composition problems (e.g., TIRG, ComposeAE, CLIP4Cir).

  • Some sketch-only baselines are reconstruction methods (DeepSDF, PSGN), and their adaptation to retrieval is not explained in the main text.

  • No variance over seeds is reported. There is no failure-mode or sensitivity analysis, such as noisy sketches, sketch–text conflicts, or the behavior of the learned gate m.

Questions for authors

Questions

  • How many queries are in each test gallery? The reported recalls (e.g., R@1 = 29.12% on chairs) do not seem to correspond to an integer count over 720 chair queries. Are results averaged over several runs? If so, how many, and what is the variance?

  • Do the GPT-5 captions directly expose the texture or material label? What R@1 does a simple baseline achieve that predicts only the texture class from text and combines it with sketch-only geometry retrieval? This would show how much of the gain comes from texture-label recognition.

  • In the “w/o Texture-aware Encoding” ablation, how is appearance represented, and how is Ltex computed without texture tokens?

  • How do the CLIP ViT-B/16 sketch backbone and LSSE interact, both at training and at inference? What prompts and data split were used to distill the GPT-4o teacher, and were any test sketches seen during distillation?

  • The 3D representation z3Dqz_{3D}^q depends on the query, so candidate embeddings cannot be precomputed and indexed. How does the reported ~509 ms per query scale with gallery size, and is this practical for larger or mixed-category galleries?

Suggestions

  • An evaluation on a small set of human-written sketch+text queries, or on mixed-category galleries, would greatly strengthen the claims.

  • An analysis of the learned gate m across structure-oriented and appearance-oriented queries would directly support the claimed adaptive behavior.

Reviewer_Vzf9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies sketch–text-guided retrieval of textured 3D objects and introduces the ST-Tex3D dataset. The proposed framework combines sketch semantic distillation, query-aware geometry–texture aggregation, and fine-grained matching. Experiments on chairs and lamps show improvements over the evaluated unimodal and multimodal fusion baselines, while ablation studies assess the contributions of several components.

Critical strengths
  1. The task is well motivated and addresses the practical need to specify both object geometry and appearance.
  2. The benchmark uses genuine freehand sketches and separately partitions geometries and textures to evaluate generalization.
  3. The reported improvements are consistent, and the ablations support the usefulness of the evaluated components.
  4. The video demo is very clear.
Critical weaknesses
  1. The text descriptions contain substantial geometric information. Without unimodal controls using the same backbone, the complementary roles of sketch and text remain insufficiently demonstrated.
  2. The multimodal baselines mainly use score fusion and do not clearly isolate the benefits of additional large-model supervision. Fine-grained matching also lacks a dedicated ablation.
  3. The gallery composition, metric aggregation, and distillation procedure are insufficiently specified, limiting reproducibility.
  4. The evaluation covers only two categories, chairs and lamps, providing limited evidence of generalization across object categories. Experiments on more diverse categories and mixed-category galleries would strengthen the conclusions.
Questions for authors
  1. Does the method still outperform text-only and simple fusion baselines using the same backbone when the text describes appearance only?
  2. Does distillation use training sketches only? How much improvement remains when baselines share the same semantic encoder and supervision?
  3. Do Figure 3 and Table 1 use the same test gallery? Please specify the number of queries and candidates, and whether results are averaged across runs.
Reviewer_MYNoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces sketch-text guided textured 3D object retrieval, a multimodal retrieval task in which sketches specify geometric structure while textual descriptions provide complementary appearance information, including materials, colors, textures, and styles. To support this task, the authors construct ST-Tex3D, comprising 12,000 textured instances derived from 1,200 sketch–geometry pairs in the chair and lamp categories. They also propose a retrieval framework combining semantic sketch distillation, sketch-text intent generation, geometry and multi-view texture encoding, query-aware evidence aggregation, and fine-grained late-interaction matching. The proposed method achieves substantial improvements over the evaluated sketch-only, text-only, and simple sketch-text fusion baselines. I appreciate the task motivation and the effort to construct a controlled multimodal benchmark. However, the dataset's dependence on a small number of reusable texture templates and automatically generated captions raises concerns about whether the evaluation adequately reflects real-world compositional retrieval.

Critical strengths
  1. Sketches and textual descriptions offer complementary constraints on object geometry and appearance. The proposed task addresses a limitation of traditional sketch-based 3D retrieval and has potential applications in 3D asset search and interactive content creation.
  2. ST-Tex3D combines real sketch–geometry pairs with deliberately varied appearances. Importantly, Section 5.1 specifies disjoint geometry and texture partitions for training and testing, which is stronger than a naive random split of textured instances.
  3. The integration of point-cloud geometry, multi-view appearance features, sketch semantics, and query-dependent evidence provides a reasonable approach to fine-grained cross-modal retrieval.
Critical weaknesses
  1. The dataset is generated using ten category-specific textures applied systematically across the underlying geometries. Although the geometry and texture splits are disjoint, the overall appearance space is constructed from a small collection of manually selected templates. This raises the possibility that the model learns to recognize a restricted set of texture and material patterns rather than generalizable compositional relationships between natural language and textured 3D objects. The distinction is important because the paper's primary contribution includes a new benchmark intended to measure fine-grained compositional retrieval.
  2. Each textured instance receives a retrieval-oriented description generated by GPT-5 from a rendered view and material information. Such captions may contain highly standardized material descriptors, object attributes, or linguistic patterns that correlate strongly with the underlying texture templates. This could make matching easier than when using independent human-authored descriptions. The evaluation currently does not establish robustness to linguistic paraphrases, incomplete appearance descriptions, ambiguous instructions, or adversarial combinations of geometry and texture attributes.
  3. The two sketch-text baselines in Table 1 combine existing sketch-based and text-based methods through score fusion. These comparisons are useful, but they do not fully isolate whether the gains come from the proposed query-aware evidence aggregation or simply from using a substantially more expressive joint retrieval architecture. A parameter- and backbone-matched joint-embedding baseline with learned sketch-text fusion would be a particularly informative control.
  4. The proposed framework includes large pretrained vision-language models, semantic distillation, multiple geometry branches, saliency-based texture tokens, intent queries, gated fusion, and late interaction. While Table 1 removes several major modules, it does not separately examine important architectural choices such as the intent-token count, number of rendered views, distillation teacher, local texture-token selection, or late-interaction score. It is therefore difficult to determine which components are essential and whether a simpler architecture could achieve similar performance.
  5. The manuscript references Appendices A and B for texture construction, generation prompts, rendering procedures, and baseline implementation details. However, these appendices are not present in the provided 12-page PDF. Furthermore, code and models are promised upon acceptance, while the reproducibility of the newly constructed dataset is central to evaluating the work. The missing details make it difficult to verify dataset generation, baseline adaptations, and possible train-test dependencies.
Questions for authors
  1. How would the proposed method perform with independent human-written queries or paraphrased captions rather than GPT-generated retrieval descriptions?
  2. Can the authors evaluate compositional generalization using controlled hard negatives, including candidates with identical geometry but different appearances and candidates with identical appearances but different geometries?
  3. How does the proposed method compare against a strong jointly trained sketch-text-3D retrieval baseline with matched pretrained encoders and comparable model capacity?
  4. Can the authors provide sensitivity experiments for the number of intent tokens, rendered views, and texture templates, together with the computational cost of scoring a gallery as its size increases?
  5. Can the authors provide the referenced appendices and clarify the exact dataset construction, caption-generation prompts, and baseline adaptation procedures?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.