acceptodds
Withdrawn from ICLR 2027

StructAlign: Learning What to Preserve and What to Complement for Sketch-Text 3D Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrieving textured 3D objects from freehand sketches and text requires combining complementary yet unequally reliable cues. Sketches provide strong geometric constraints through contours, proportions, and part relationships, whereas text supplies category and appearance information but may be incomplete, ambiguous, or inconsistent with the sketch. Existing fusion methods often lack explicit protection of sketch structure, allowing textual semantics to distort geometry-sensitive representations. We present SketchAnchor, which treats the sketch representation as a geometric anchor and uses multimodal large language model features to complement missing semantics. It constructs a foreground-aware sketch representation and selectively injects locally relevant semantic features through cross-modal attention and gated residual updates. A foreground Structural KL constraint limits changes in the relative importance of key stroke regions. To align semantic completion with downstream retrieval, we further introduce a shape–texture graded ranking reward and retrieval-drift-calibrated GRPO with completion-specific adaptive clipping. We also construct ST-Tex3D v2, containing 15,600 textured 3D instances with aligned sketches, retrieval-oriented descriptions and texture variants. Experiments demonstrate the effectiveness of explicitly modeling asymmetric modality reliability for fine-grained multimodal retrieval

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sketch-Text Retrieval, Textured 3D Retrieval, Multimodal Retrieval, Structure-Preserving Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_urYxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses sketch-based 3D shape retrieval. It introduces an additional modality, textual description, and makes the retrieval task more complex: the method must retrieve not only the correct shape but also the correct texture. The authors propose the StructAlign framework, which includes a 3D shape encoder, a texture encoder, a sketch encoder, and an MLLM text encoder. The goal is to encourage the feature encoders to focus on the sketch strokes themselves while ignoring the blank background regions. The method appears to use contrastive learning to learn shape and texture representations, and it uses GRPO to train the MLLM, specifically with LoRA on Qwen-8B.

Critical strengths
  • The authors provide a video demo, which is valuable because it allows reviewers to directly observe the effect of the proposed method.
  • The attempt to use contrastive learning to address sketch- and text-based 3D shape and texture retrieval is workable.
  • The authors claim to provide a dataset of around 15,000 sketch-text-textured 3D shapes, although the shape categories are limited to two.
Critical weaknesses

I believe there is a clear rejection signal, and I will explain my assessment as follows.

Writing and presentation

The writing has significant problems. In the abstract and introduction, the paper does not disclose core implementation details, such as the use of Qwen-8B, LoRA, and another key component, the Retrieval KL. If readers expect to understand the core logic from the abstract alone, they will be seriously misled or confused. In multimodal and reinforcement learning papers, the base model and the training method, such as LoRA, are critical information. From the perspective of the idea, the paper appears to apply GRPO to sketch-based 3D shape retrieval, but the difference from existing GRPO applications is not clear, and I do not see significant novelty.

Figure quality

The figures do not meet the standard of a research paper.

Regarding Figure 1, I am not satisfied with its presentation. It is not solid and is prone to ambiguity. The middle part of Figure 1 summarizes existing works on sketch and text semantic encoding and shows failure cases from existing works. I would like to know which specific works belong to the summarized encoding ways. The authors should provide actual results and applicable citations. In addition, for the encoding ways summarized in the figure and the proposed encoding way mentioned in the right subfigure, the paper states that existing encoding methods pay weaker attention to the foreground (sketch) than to the background (blank regions). I am skeptical of this assumption. I cannot infer from the fusion way drawn in Figure 1 that this would necessarily prevent attention to the sketch. Since this cannot be inferred from the method logic, the authors need to provide solid failure cases. Otherwise, readers will view Figure 1 as not solid and will question the entire paper.

Figure 2 is the framework overview of the proposed method and is very important. However, the figure contains highly refined content, and I suspect it was generated by an auto research tool or a similar AI tool. Such high-level conceptual description is characteristic of AI generation, and I am concerned about the authenticity of the paper. I have the following specific concerns about this figure. First, the content of Figure 2 does not correspond to the relevant descriptions in the main text. I suggest adding the mathematical notation mentioned in the main text to Figure 2 to improve readability. Second, in subfigure 1, the output of Qwen is labeled as "K candidate semantic completions." This description is extremely inaccurate. What is being completed? Third, in subfigure 2, the output of Qwen is described as "Qwen hidden features," which is inconsistent with the notation in subfigure 1. This inconsistency is critical. Fourth, for the token operations, they are connected only by simple arrows, and I do not know what the specific operations are. Fifth, in subfigure 3, because this is retrieval, features need to be extracted for every item in the entire dataset and then compared with the query to obtain the top-k candidates. I can understand this operation, but how the query is input is not shown. Overall, the figure only shows inputs and outputs without details, and it is unclear what it is trying to convey.

Methodological issues

There are several methodological flaws. One of the contributions is the Structural KL, but this KL constraint is not described at all. What is KL? I assume it refers to Kullback-Leibler Divergence. The only relevant section is 3.6, "FOREGROUND STRUCTURAL CONSISTENCY LOSS," where Eq. 9, 10, and 11 are meaningless. They simply modify existing general formulas into a form that fits the notation of this paper. The main text explicitly states, "We further introduce two KL constraints," namely Structural KL and Retrieval KL. These are mentioned vaguely in the figure and introduction. The section related to Retrieval KL is 3.7, "RETRIEVAL-DRIFT-CALIBRATED ADAPTIVE GRPO." It is very difficult to locate the Retrieval KL here. After careful reading, I found it in Eq. 14. The readability is too low and does not meet the standard of a research paper. In addition, both KL constraints are applications of existing theory, and I do not see any novelty.

Some details of the method are also puzzling. For example, in eq.7 the term "joint" is mentioned. What does "joint" refer to here? Even considering the context, I cannot understand the meaning of "joint." Does it refer to a vertex in geometry? At least it should be related to shape and texture in the context. In addition, the description of contrastive learning in this paper is disastrous. It does not explain how contrastive learning operates in this context, and I find it difficult to understand the mechanism.

Experimental issues

The experimental dataset has too few categories, only chairs and lamps. This means the proposed method cannot handle other categories, and its generalization ability is weak. The amount of data is also small. I suspect the learned network will overfit. Can such a small amount of data be used to fine-tune an LLM? I am very doubtful. In addition, the implementation description is inaccurate. The paper mentions using PointNet++ as the geometric encoder, but the original text says, "while the 3D branch employs PointNet++." This description is inaccurate. I can only infer that PointNet++ is the geometric encoder of this paper. There is also a texture encoder, but what is it? The paper does not mention it. Furthermore, the paper does not clearly describe why LoRA is used. In summary, this does not meet the standard of a qualified paper.

Questions for authors

Please see in "Critical Weaknesses"

Reviewer_rk3oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies textured 3D instance retrieval from a sketch and a textual description. StructAlign uses foreground-aware sketch tokens as an anchor and injects Qwen3-VL semantic-completion features through local attention and gated residual updates. A foreground-importance KL regularizer and a shape–texture ranking reward support learning, while retrieval-distribution drift controls GRPO clipping for the completion policy. The paper introduces ST-Tex3D v2, containing 15,600 textured instances, and reports substantial retrieval improvements on chairs and lamps. This assessment includes the technical supplement and static inspection of the supplied training code; the reported experiments were not independently reproduced.

Critical strengths
  1. The architecture and reward address meaningful, distinct retrieval errors. Sketch geometry and textual appearance provide complementary information. Foreground selection and gated local semantic injection offer a concrete way to combine them, while same-shape/different-texture and same-texture/different-shape negatives distinguish two important failure modes (Section 3.5, Eqs. 7–8). This makes the proposed supervision well aligned with exact-instance textured 3D retrieval.

  2. The reported gains are substantial and supported by useful controls. Table 1 reports chair R@1 of 43.88% versus 25.31% for UME-R1-to-3D, and lamp R@1 of 48.71% versus 25.02% for PASTA-style Semantic Prior. Table 2 supports the utility of the components: removing Structural KL reduces chair R@1 to 39.21%, and removing retrieval-drift calibration reduces it to 40.68%. The factor-level data split and the detailed baseline adaptations in Appendix B strengthen the evaluation. Appendix C also supplies qualitative examples of improved structural matching. These results support a useful retrieval-method contribution.

Critical weaknesses
  1. The specific geometry-preservation interpretation remains less established than the retrieval improvement. Eqs. 9–11 constrain foreground-token importance distributions, while semantic injection changes the token features themselves. The Structural KL ablation and Appendix C's qualitative examples support the method's usefulness, but they do not fully separate geometry preservation from a more general regularization effect. The evaluation primarily reports exact-instance R@K, without separately quantifying shape and texture errors. Appendix D appropriately acknowledges strong sketch–text conflicts as a limitation. The interpretation should remain scoped to the evidence provided; separating these error types would help explain what the regularizer preserves.

  2. The reported Structural KL objective needs a precise mapping to the supplied implementation. Eq. 11 penalizes only KL above a tolerance, whereas grpo_loss.py::structural_distribution_kl returns the mean raw KL and mg_model.py::_compute_generative_grpo_update scales it by a coefficient, whose launcher default is 0.05. This could correspond to a zero-tolerance special case with an explicit loss weight, but the settings used for the reported full model and ablation should be specified. The code already provides separate retrieval-network and LoRA updates, fixed-rollout reuse, and retrieval-drift-dependent clipping; the remaining concern is documenting the exact configuration that connects these implementations to the paper's equations and results.

Overall, I lean toward weak acceptance. The large retrieval gains, task-specific design, component ablations, and supplementary implementation details provide sufficient value. The remaining concerns mainly affect the precision of the mechanism's interpretation and reproducibility of the exact reported configuration.

Questions for authors
  1. Implementation clarification: What Structural KL tolerance, temperature, and loss coefficient were used for Tables 1–2? Does the supplied implementation correspond to the zero-tolerance case of Eq. 11? Please provide the resolved configuration for the reported runs and identify any additional retrieval-network losses used alongside the policy objective.

  2. Interpretation clarification: Is structure preservation intended specifically to mean preservation of foreground-token importance, or is there quantitative evidence that separates shape correctness from texture correctness? If such results are already available, how do these errors change when Structural KL is removed? A controlled sketch–text conflict analysis would strengthen this interpretation, although Appendix D already makes its current limitation clear.

Reviewer_8oX3OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes StructAlign, a framework for retrieving textured 3D objects from freehand sketches and textual descriptions. It treats sketch features as geometric anchors and selectively incorporates semantic features from Qwen3-VL through foreground-aware encoding, local cross-modal attention, and gated residual updates. A foreground Structural KL loss constrains changes in sketch-region importance, while a shape–texture ranking reward guides semantic completion. The method further introduces retrieval-drift-calibrated GRPO, which adapts policy clipping according to changes in retrieval distributions. The authors construct ST-Tex3D v2, containing 15,600 textured instances, and evaluate exact-instance retrieval on chair and lamp categories using R@1, R@5, and R@10. Experiments report improvements over single-modal, sketch–text fusion, and multimodal language model baselines, with ablations on the chair subset examining the contributions of individual components.

Critical strengths

The method explicitly preserves sketch-derived geometric evidence while selectively incorporating semantic and appearance information through gated local attention and a structural consistency constraint. This addresses a consequential challenge in textured 3D retrieval: matching appearance without losing instance-specific shape cues. StructAlign improves R@1 over the strongest reported baseline by 18.57 percentage points on chairs and 23.69 points on lamps. These results provide encouraging evidence that the complete framework offers practically meaningful benefits within the evaluated setting.

Critical weaknesses

Structural KL constrains token-importance distributions, which does not directly establish preservation of geometric information. Likewise, the reported retrieval accuracy and component-removal ablations do not demonstrate improved training stability from adaptive GRPO. Independent geometry-sensitive evaluation and a clearly specified, well-tuned fixed-clipping comparison are needed to substantiate these core contributions.

The paper does not explicitly define how enhanced sketch tokens form the final query or how post-injection importance scores are computed. It also leaves unclear how retrieval-network optimization interacts with the stated GRPO and structural losses.

Questions for authors

How are enhanced sketch tokens aggregated into q_k, and how are post-injection importance scores computed? Is the background token excluded from Structural KL?

Appendix B indicates that Qwen-DPO uses a text-encoding retrieval pipeline, whereas StructAlign directly retains sketch features. What evidence isolates the benefits of the proposed optimization strategy when the query architecture, backbone, reward, supervision, and training budget are held fixed?

Does retrieval-drift-calibrated clipping outperform a sufficiently tuned fixed-clipping GRPO under otherwise identical conditions?

The benchmark captions explicitly describe both geometry and appearance. Does StructAlign retain its advantage when captions contain only appearance attributes, or when textual geometry conflicts with the sketch? What quantitative evidence establishes that the method preserves sketch-derived geometry and handles unequal modality reliability, beyond the qualitative examples in Appendix C?

This assessment is limited to chairs and lamps. Can this method be extended to objects with more complex geometric structures and component relationships?

Could the authors define and report a base retrieval model? Table 2 removes individual components from the full model, but does not establish the overall improvement over this base model. A progressive ablation would help clarify the cumulative contributions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.