acceptodds
Under review as a conference paper at ICLR 2027

SelectionVLM: Enabling VLMs to Interact Programmatically with CAD

Abstract

Modifying CAD geometry programmatically requires first identifying the entities to manipulate and then passing references to them to the appropriate functions in the CAD system's API. Vision-language models (VLMs), given only a canvas screenshot, can perceive these entities but cannot directly reference them, requiring the correspondence to be inferred from geometric or textual clues. We introduce SelectionVLM, a VLM augmented with a new modality that exposes information from the selection buffer: a per-pixel integer-valued image containing the pick IDs which 3D graphics software uses to allow interactive picking. Unlike RGB-based strategies such as object coloring, 2D bounding boxes, or numbered image markers, the selection buffer provides a direct correspondence between image pixels and CAD entities. We show that this representation enables SelectionVLM to substantially outperform RGB-only approaches on a range of entity identification tasks. Because pick IDs can also index structured per-entity metadata, supplied as part of the prompt, SelectionVLM can directly connect visual entities with their attributes, yielding further improvements on relational and spatial reasoning tasks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Aided Design, CAD, Vision Language Model, selection, entity identification, design, agent, spatial reasoning, mechanical comprehension

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_52DvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SelectionVLM augments a VLM with a pixel-aligned CAD selection buffer, allowing it to associate visible entities with API-addressable pick IDs and structured metadata. Experiments on Fusion 360 Gallery assemblies show improvements over color-keyed, bounding-box, and Set-of-Mark baselines on entity identification and metadata-based reasoning tasks.

Critical strengths

The central idea is practical and well motivated. Using the CAD selection buffer directly connects visual entities with programmatic references and associated metadata, providing a useful interface for future CAD agents.

The method consistently improves performance across the evaluated task families. The assembly-level split and architectural ablations provide encouraging evidence that the approach can generalize to unseen assemblies with a relatively lightweight additional encoder.

Critical weaknesses

The baseline comparisons could be better controlled. SelectionVLM is task-trained, whereas the bounding-box and Set-of-Mark baselines are not. Fine-tuning the color-keyed baseline decreases performance; the authors attribute this to distribution shifts, but the explanation is not experimentally validated. Comparably trained alternatives would help establish how much improvement comes specifically from the selection-buffer modality.

The quantitative evaluation focuses on synthetic questions at the body/occurrence level. Face/edge selection and downstream CAD operation success are not quantitatively evaluated. These acknowledged limitations leave the practical scope of the method less established than its entity-identification capability.

Questions for authors

Can you provide comparably trained bounding-box and Set-of-Mark baselines and further investigate why fine-tuning degrades the color-keyed baseline?

How are ambiguous color references and duplicate color-keyed metadata entries handled when colors are reused above 20 visible entities?

How robust is SelectionVLM to changes in rendering appearance, such as using original CAD materials or a uniform color instead of the distinctive training palette? This would help establish its applicability to typical CAD views.

Reviewer_dzU6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes SelectionVLM, which equip the VLM with a new modality for CAD related understanding task. Specifically, the authors propose to combine both CAD renderings with an entity ID rendering and a trainable look-up table. The baseline VLM is then fine-tuned with the proposed modality to take the entity map into the model understanding. To fine-tune the model, the authors also propose a CAD dataset with several thousand samples. The fine-tuned model show superior performance comparing to other baselines.

Critical strengths

The main strengths of the paper include;

  1. The motivation of the work is clean and practical. Existing CAD renderers already produce a pixel-level correct entity ID map, and exposing it gives the model exact grounding of the information it requires for the task, instead of forcing model to infer it implicitly.
  2. The data contribution is also substantial. There are large number of renders with selection buffers and metadata and generated tasks over Fusion 360 Gallery assemblies, split by assembly. This dataset with all its metadata information could be valuable if released.
Critical weaknesses

The main weakness of the paper is the lack of a specific baseline in my mind, which then leads to the actual contribution of the work. Since entity ID map is rendered and considered as given, for the proposed tasks in the paper, a straightfoward baseline is, instead of using some other modalities, asking the VLM to directly point out on the renderings, and map that to the ID in the entity ID map. This baseline's result itself is then fundamentally proving the usefulness of the proposed leverage of the entity modality. And comparing this training-free baseline with fine-tuned model can further show the performance change introduced by fine-tuning. Right now it's not that clear where the improvement gain comes from, by evaluating only a holistic learnt modality.

Questions for authors
  1. As described in the critical weakness, how does a base VLM do with pointing plus an ID lookup at the predicted pixel?
  2. The results in baseline comparison table is strange. Why does fine-tuning the color-keyed baseline lower its accuracy? Are the fine-tuning of baselines under the same data and budget?
  3. Given recent rapid progress of agents, is there an end-to-end agentic task (e.g., create the joint) where SelectionVLM improves success rate or cost over a tool-based agent? Or it can act as a "better tool"?
  4. Are the descriptions generated by Qwen3-VL-32B biased toward the Qwen-based model? Can a different model generate the evaluation descriptions?
1 comment

Thanks the authors for detailed experiments and replies. I also appreciate the effort on spotting the baseline bugs and revise the results which make the tables much more convincing.

I still think "providing a list of 2d boxes and asking VLM to choose" is fundamentally different from "asking VLM to directly write a point location", in the first option the given list can affect the implicit model prediction, and box is fundamentally different from point. Asking it directly is much closer to the proposed SelectionVLM here in terms of input / output.

Reviewer_SVBDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SelectionVLM encodes the CAD renderer's selection buffer alongside RGB images to link visual entities with CAD IDs and metadata. Evaluation covers entity identification, attribute retrieval, and relational tasks on held-out assemblies.

Critical strengths
  1. The selection buffer provides a direct interface between visual entities, CAD references, and metadata.

  2. Table 1 reports 74.12% description-to-entity accuracy versus 67.41% for the strongest baseline, with additional fine-tuned color comparisons.

Critical weaknesses
  1. Reference-collision handling is unspecified. Section 6 uses color names as metadata keys and colors to identify targets, while Table 2 states that colors are reused beyond 20 visible entities. The handling of duplicate keys and ambiguous targets is not reported.

  2. Experimental settings and spatial results are not reconciled. Figure 4 shows approximately 49% spatial accuracy, whereas Appendix C reports 24.2% and 29.3% for the depth variants. Section 5 reports four epochs and 5,684 steps for over 43 million training tasks without defining the sampling procedure or epoch size.

Questions for authors
  1. How are repeated colors, duplicate metadata keys, and ambiguous targets handled? What fraction of the test set is affected?

  2. How do the training and evaluation settings differ between Figure 4 and Appendix C? How are the 43 million training tasks sampled within the reported 5,684 steps?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.