acceptodds
Under review as a conference paper at ICLR 2027

ViSculpt: Visual-Centric Agentic Geometry Editing

Abstract

3D geometry editing is a critical yet labor-intensive part of the graphics pipeline, requiring artists to translate creative intent into precise operations in complex professional software. Large language models (LLMs) have shown promise for script-based 3D creation, but script generation is less suited to perception-driven editing of arbitrary existing meshes, where execution must remain visually grounded and untouched regions should be preserved. We present a visual-centric, training-free multi-agent system that edits existing 3D meshes directly in Blender by emulating the iterative workflow of human artists. Rather than generating scripts or regenerating geometry, our system operates through the Blender GUI: multimodal LLM agents observe the viewport, reason about the current mesh state, and execute localized edits through simulated user interactions. Experiments on a curated benchmark provide initial evidence that this agentic approach can follow natural language instructions, perform representative localized mesh edits, and preserve the overall identity of the input asset. Our results highlight a complementary regime for language-driven 3D editing: direct in-place modification of existing meshes within the native 3D editing workflow. We view this work as an exploratory step toward visual-centric agentic geometry editing in professional graphics software. We will release our code and benchmark to facilitate future research.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Human Computer Interaction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2rt9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ViSculpt is a training-free multi-agent system for editing existing meshes through the Blender GUI. It combines planning and visual reflection with three mouse-trajectory primitives, Smear, Drag, and Draw, and a coarse-to-fine localization procedure. Evaluation includes 20 editing tasks, a blinded output-rating study with 48 participants, auxiliary VLM assessments, and qualitative comparisons with script-based agents.

Critical strengths

The paper addresses a useful problem: localized editing of existing meshes through natural-language instructions while preserving unrelated geometry. The combination of compact action primitives and deterministic trajectory generation is practical and well motivated.

The qualitative results demonstrate a promising range of deformation, smoothing, and surface-detail operations. The human evaluation provides useful initial evidence that the resulting edits are perceptually plausible.

Critical weaknesses

The evaluation needs stronger support for its comparative conclusions. The main study relies on independent 0–10 ratings, but the small mean difference between system and human results is not accompanied by uncertainty estimates for the difference or an analysis accounting for repeated ratings across participants and tasks. The auxiliary VLM evaluators also use noticeably different scoring scales. A supplementary blinded pairwise comparison between human-created and system outputs, with randomized presentation order, ties, and confidence intervals, would provide more direct evidence of relative quality. This could initially be added to the VLM evaluation without repeating the entire human study. The expertise, number, and time budgets of the human reference creators also need clarification.

The experiments establish output quality more clearly than system usability or the contribution of the agentic design. Participants rate completed outputs rather than use the system, so reduced interaction burden and accessibility to non-experts remain insufficiently evaluated. There is an overclaim about the system usability in Section 4, without any comparative study results.

The MCP comparisons and primitive/QuadLoc ablations are primarily qualitative, while the benefit of reflection is not quantitatively isolated. Furthermore, the MCP comparisons change both the backbone and interaction framework, making attribution difficult. Aggregate task success, retries, failure rates, and component ablations would substantially strengthen the evidence.

Questions for authors

Could you supplement the existing VLM ratings with blinded, order-swapped pairwise comparisons between system and human-created outputs, reporting win/tie/loss rates and uncertainty? How were the human reference creators selected, and what editing time budgets were provided?

Did participants interact with ViSculpt, or only evaluate its outputs? If only the latter, can the usability claims be narrowed, or supported by measurements of user effort, correction burden, and task completion?

Can you report aggregate quantitative results for the MCP comparisons and component ablations, particularly with and without reflection? Please clarify task counts, execution budgets, retry limits, and whether reported runtime and cost include failed attempts.

5 comments

Thank you for recognizing the value of localized mesh editing, the practical motivation for compact primitives, and the initial evidence of perceptual plausibility. We further clarify the existing study design and the scope of our conclusions below.

1. Blinded paired ratings, statistical uncertainty, and VLM scores

Concern: The mean-score difference lacks uncertainty estimates accounting for repeated ratings across participants and tasks. VLMs use different scoring scales. Blinded, order-swapped pairwise comparisons with win/tie/loss rates and confidence intervals would provide stronger evidence.

Response: Our study already used blinded, order-controlled paired ratings. For each of the 20 tasks, participants evaluated both the ViSculpt result and the human-created reference without knowing which method produced either result. The presentation order of the two methods' outputs was also varied. The 48 participants evaluated 40 results produced by the initial ViSculpt configuration and the human reference creators, assigning each result one overall 0-10 score considering instruction adherence, visual aesthetics, and geometric plausibility.

Thus, these observations are paired by both evaluator and task, although the questionnaire records a score for each output rather than separately asking participants to choose A, B, or a tie. We will describe the blinding, order control, and paired design more explicitly. Comparisons derived from paired scores differ from directly elicited preference votes.

The auxiliary VLM evaluation follows the same matched design: each VLM evaluated both methods on the same 20 tasks using the same questionnaire. An evaluator's overall tendency to award higher or lower scores does not by itself invalidate this comparison. An additive scoring offset cancels in the difference between the two methods' scores within that evaluator.

The human participants' mean ratings were 7.53 for ViSculpt and 7.20 for the references, providing evidence of competitive perceptual quality on this task set. The reported standard deviations describe score dispersion, rather than uncertainty in the mean difference.

2. Human reference creators and their editing protocol

Concern: How were the human reference creators selected, what was their expertise, how many were involved, and what time budgets were provided?

Response: The human reference edits were created by two PhD students in computer graphics with experience using Blender. No time limit was imposed on individual tasks. The 48 participants and the expert/non-expert groups in Table 1 describe the rating cohort.

We will add these reference-creation details to the manuscript. The comparison evaluates system outputs against these manually created references, rather than against a representative sample of professional artists under matched time budgets.

3. Natural-language interaction and the scope of usability claims

Concern: Did participants use ViSculpt or only rate its outputs? Claims about reduced interaction burden and accessibility to non-experts are not adequately supported without an interactive usability study.

Response: Participants evaluated completed outputs and did not interact with ViSculpt. The natural-language interface already provides a concrete simplification of interaction: given an input mesh in the configured environment, users describe the desired edit in natural language, and the system automatically plans and executes the Blender operations. Users do not need to manually carry out the corresponding Blender operations, write editing scripts, or draw brush trajectories. The pipeline and the demonstrated editing examples support this capability. We will clarify that "reduced manual interaction" refers to the automation of these low-level Blender operations.

4. Editing quality and efficiency with a lightweight backbone

Concern: The MCP comparisons change both the backbone model and the interaction framework, making it difficult to attribute differences to the agentic design.

Response: Figures 6-7 compare complete configurations and demonstrate a practical advantage in the model configuration needed to obtain competitive edits. In Figure 7, Codex + Blender MCP produces visibly degraded results with GPT-6 Astra at medium reasoning effort, while the max-effort configuration produces a competitive result. ViSculpt achieves visually comparable editing quality using the lightweight Gemini 3.5 Flash Lite backbone.

These observations support our argument that, for the demonstrated editing tasks, the proposed framework can reduce the reasoning burden placed on the underlying model. This is consistent with our design: the agents handle semantic decisions and compact parameters, while geometric procedures construct dense mouse trajectories.

This lightweight ViSculpt configuration also provides a substantial runtime and cost advantage in the reported comparison: approximately 1-2 minutes and USD 0.10 per edit, versus about 10 minutes and USD 4.00 for Codex with GPT-6 Astra at max reasoning effort. As clarified below, these measurements include failed attempts and retries.

Our claim is that the system can achieve competitive editing quality with a lightweight backbone and lower observed execution cost. These comparisons are conducted at the complete-system level; controlled comparisons using the same backbone could further isolate the framework's contribution. We will explicitly scope the conclusions to the tasks and configurations evaluated.

5. Primitive, QuadLoc, and reflection ablations

Concern: The primitive and QuadLoc ablations are qualitative, and the incremental benefit of reflection has not been quantitatively isolated.

Response: Figures 8 and 12 address two execution bottlenecks: selecting the intended mesh region and generating a suitable brush trajectory. Figure 8 compares direct coordinate prediction with QuadLoc's recursive quadrant selection; Figure 12 compares directly generated coordinate sequences with trajectories constructed from the primitives.

In the illustrated cases, off-target strokes produced no visible edit, whereas the full method produced the requested edits. We will include the existing before-and-after editing results alongside the localization/trajectory visualizations and explain these failure mechanisms. These cases support the motivation for the component designs.

The Reflection Agent compares pre- and post-edit observations against the sub-goal and provides feedback for parameter adjustment or another execution attempt (Section 3.2.3). We have not yet quantitatively isolated its incremental benefit, and we will add the corresponding experiments in future work. We emphasize that reflection is not a core contribution or source of novelty of this paper, and our arguments and conclusions do not depend on its ablation results.

6. Task counts, budgets, retries, and runtime/cost accounting

Concern: Please report aggregate success, retries, and failures, and clarify task counts, execution budgets, retry limits, and whether runtime and cost include failed attempts.

Response: The MCP comparisons cover the three tasks shown in the paper: the pumpkin and cat examples against the Claude-based baseline in Figure 6, and the armadillo example against Codex in Figure 7. Each tested configuration was repeated approximately five times.

ViSculpt allowed at most three execution attempts per subtask, including the initial attempt, i.e., up to two retries. The MCP baseline had no explicit retry cap. In our ViSculpt experiments, we generally observed completion on the first attempt, with occasional retries.

The runtime and cost figures cited above are averages over repeated experimental runs and include failed attempts and retries. They describe the practical performance of the tested configurations under their respective retry policies.

We will add these protocol details and clarify other execution-budget settings based on the original records.

Thank you for the detailed clarifications. The explanations of the evaluation protocol, human reference creation, and execution budgets address several of my questions.

However, my main concerns remain only partially resolved. In particular, the evaluation still lacks statistical analysis of paired differences, direct pairwise preferences, an interactive usability study, and quantitative ablations (especially for reflection). The MCP comparisons also remain limited and confounded by different backbones.

I appreciate the authors' willingness to clarify and narrow their claims, but stronger quantitative evidence would be needed to fully address my concerns.

Reviewer_2MYcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ViSculpt, a training-free, multi-agent framework that edits 3D assets directly by using agents to control Blender's GUI. Instead of generating code or refining geometry using training-based diffusion models, the framework can take a user's text prompt and operate through three proposed primitives: Smear, Drag (for local deformation using a recursive QuadLoc strategy), and Draw (for high-frequency details/text). The visualizations show the effectiveness of the proposed method in editing shapes.

Critical strengths

1: The framework can directly operate the user's Blender software and perform operations to edit the existing 3D assets, showing the promising computer-use ability for automated 3D asset generation and editing.

2: The visualizations are beautiful and effectively show the process of editing powered by the proposed method.

3: The authors use a coarse-to-fine quadrant-selection strategy to effectively alleviate the spatial hallucinations that potentially exist in the output of VLM coordinate predictions.

Critical weaknesses

1: The writing is somewhat confused and messy. The authors claim that “ViSculpt with the lightweight Gemini 3.5 Flash Lite backbone produces more faithful local edits than the Claude Sonnet 4.6 baseline and results competitive with Codex using GPT-6 Astra at max reasoning effort”, but I haven't seen any quantitative results in the paper.

2: Why did the authors only evaluate VLMs released in 2025? The results only include models released in 2025. Many more advanced vision-language models and coding agents have been released, even in early 2026. The authors don't analyze them, which weakens the paper's conclusion.

3: "Limited ablation studies". Only one table shows quantitative results, with no further analysis of how each proposed module works. The effectiveness of the proposed operation primitives.

Questions for authors

Please review the weaknesses and ensure you include enough results before submission.

1 comment

Thank you for recognizing the potential of GUI-based editing of existing assets, the quality of the visual examples, and QuadLoc's coarse-to-fine localization strategy. We address the three concerns below.

1. Quantitative support and clarity of the MCP comparisons

Concern: The presentation is confusing, and claims of better editing results than Claude Sonnet 4.6 and comparable results to GPT-6 Astra lack quantitative support.

Response: We agree that the initial evaluation against human-created results (Section 4, Table 1) and the subsequent qualitative MCP comparisons (Section 5, Figures 6–7) should be distinguished more clearly. Figure 6 compares ViSculpt with Claude Desktop + Blender MCP + Claude Sonnet 4.6; Figure 7 compares it with Codex + Blender MCP + GPT-6 Astra at medium and max reasoning effort. These examples support observations about the illustrated edits but do not provide aggregate comparisons of quality or success rates.

GPT-6 Astra at max reasoning effort produces competitive edits, primarily through Blender Python scripts. Our contribution is an orthogonal, Visual-Centric approach using a lightweight backbone and compact action representations. Figure 7 shows the baseline's high reasoning-effort requirements in this case, while ViSculpt with Gemini 3.5 Flash Lite produces comparable edits that satisfy the instructions. Our framework assigns semantic decisions and compact parameters to the agents, while geometric procedures generate dense mouse trajectories.

In the reported comparison setting, ViSculpt takes approximately 1–2 minutes and USD 0.10 per edit, versus about 10 minutes and USD 4.00 for the Codex baseline at max reasoning effort. These existing measurements motivate assessing editing quality alongside execution cost. We will clarify the scope of these observations and distinguish the two evaluation settings more clearly. Systematic quantitative MCP comparisons remain future work.

2. The role of the VLMs listed in Table 1

Concern: The results include only VLMs released in 2025, without analyzing newer VLMs and coding agents.

Response: We would like to clarify that the VLMs listed in Table 1 are auxiliary evaluators. They use the same questionnaire as the human participants to rate edits already produced by ViSculpt and human artists; they do not perform editing tasks in this experiment. As the caption states, “VLM rows identify the evaluators.” The two score columns correspond to Human Reference and Ours, rather than the listed VLMs' own editing performance.

This experiment supplements the human evaluation of the initial ViSculpt configuration across 20 tasks. The 48 human participants gave mean ratings of 7.53 to ViSculpt and 7.20 to the human references; the corresponding auxiliary VLM means were 6.51 and 6.54. These similar aggregate scores provide evidence of competitive perceptual quality on this task set, rather than establishing statistical equivalence or ranking the VLMs' editing capabilities.

As noted above, comparisons between editing agents are presented separately in Figures 6–7. Adding newer evaluator models could help test the robustness of the auxiliary ratings, but this is distinct from benchmarking newer models as editing agents. We will clarify these roles and explicitly state the limitations of the auxiliary evaluation. It supplements, rather than replaces, the human evaluation.

3. Limited ablations and the role of action primitives

Concern: Only one table reports quantitative results, with insufficient analysis of individual modules and the effectiveness of the action primitives.

Response: Table 1 evaluates the complete system without isolating component contributions. Figures 8 and 12 address two specific execution bottlenecks: selecting the intended mesh region and drawing a suitable brush trajectory. Figure 8 compares direct coordinate prediction with QuadLoc's recursive quadrant selection; Figure 12 compares directly generated mouse-coordinate sequences with trajectories constructed from action primitives. Smear represents region coverage, Drag anchor-based deformation, and Draw pattern tracing, reducing the need for the language model to generate dense trajectories directly.

In the illustrated ablation cases, off-target brush strokes produced no visible edit, whereas the full method achieved the requested edits. The current figures show localization or trajectories but omit the resulting meshes. We will add these existing before-and-after results to the revised manuscript and explain how off-target operations lead to editing failures.

We agree that these cases do not quantify average benefits or robustness across tasks. They support the motivation for the component designs; broader quantitative ablations remain an important direction for future work.

Reviewer_1fsBOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors propose a training-free multi-agent system able to perform 3D sculpting of meshes through the Blender UI. Their proposed method operates using three primitives: Smear, Drag, Draw. They also propose a zooming operator to enable VLMs to focus on fine details of the 3D geometry. Their multi-agent system is operating on top of Gemini 3.5 Flash Lite, a closed-weights proprietary model.

Critical strengths

This paper is clearly presented and well written. Some of the qualitative examples are interesting. The intervention of human experts and non-experts in the evaluation process is also valuable and interesting.

Critical weaknesses

It is very hard to be convinced by the evaluation method proposed. Some of the issues include: - All ablations in this paper are qualitative only. Why is there no quantitative comparison of edit quality using the same evaluation procedure as in table 1. with different components and design choices ablated. - There is no comparison to the native ability of proprietary LLMs in MCP mode for this task. I don't believe that Opus 5.5 or GPT-6 Astra are completely unable to perform this task. I also don't understand why no quantitative comparison to these models is offered anywhere in the paper. There are multiple Blender-MCP modules for Opus or GPT that enable UI interactions with Blender, since this paper directly competes with these base abilities a quantitative comparison to these models, with reasonable scale, is needed. Only a few figures feature qualitative comparison to frontier models without the agentic wrapper proposed in this paper, which is not sufficient at all. - The edits proposed are not very visually compelling overall and look poor. e.g. 3D meshes of creatures waving their arms upwards or downwards appear at least ten times in this paper. This is not what 3D sculpting is or amounts to.

(minor comments, but it would be appropriate to change the SIGGRAPH example in Fig.9 to something more appropriate for this venue. same for "GeoPT" which appears to be the previous name of this method.)

Questions for authors

The main concerns to address or as follows:

  • A quantitative ablation of the design choices in this agentic framework is needed.
  • A quantitative comparison of the base abilities of Blender-MCPs (or even general MCPs) with frontier models. bpy.ops.sculpt.brush_stroke exists and is exposed in the bpy API for example. I do expect this framework to outperform MCP or bpy-based methods, but it is nonetheless important to feature this comparison. This paper should show that using this agentic framework improves the base abilities of frontier models. Right now, this paper features no scores, no success rates are reported in this paper to make this comparison.
  • Can you show more ambitious sculpting attempts? Besides smoothing/moving a part downwards or upwards/writing text. For example, surface detailization? Surface style changes? (taking a cube and making it look like an old stone wall, etc.).
2 comments

Thank you for recognizing the paper's clear presentation, interesting qualitative examples, and the value of the human evaluation. Below, we further clarify the existing evidence and the scope of our contribution.

1. Quantitative evaluation of Primitive Abstraction and QuadLoc

Concern: The ablations provide only qualitative results. Please quantitatively compare editing quality using the evaluation protocol in Table 1.

Response: Figures 8 and 12 address two specific execution bottlenecks. Figure 8 compares direct coordinate prediction with QuadLoc's recursive quadrant selection; Figure 12 compares directly generated mouse-coordinate sequences with trajectories constructed from action primitives. This distinction matters because selecting the correct mesh region and drawing a suitable brush trajectory are two distinct requirements for a successful edit.

In the illustrated ablation cases, off-target brush strokes produced no visible edit, whereas the full method achieved the requested edits. The current figures show localization or trajectories but omit the resulting meshes. We will add these existing before-and-after results to the revised manuscript and explain how off-target operations lead to editing failures, making the demonstrated failure mechanisms clearer.

We agree that these cases do not quantify average benefits or robustness across tasks. They support the motivation for our design choices, while broader quantitative ablations remain an important direction for future work.

2. Comparisons with frontier models using Blender MCP

Concern: Please provide quantitative comparisons against frontier models using Blender MCP, accounting for their ability to invoke sculpting operations such as bpy.ops.sculpt.brush_stroke.

Response: The submitted paper includes comparisons with Claude Desktop + Claude Sonnet 4.6 in Figure 6, and with Codex + Blender MCP + GPT-6 Astra at medium and max reasoning effort in Figure 7. We agree that these qualitative examples do not replace aggregate comparisons of quality and success rates. We also acknowledge that MCP/API access can support sculpting operations; our argument does not depend on these operations being unavailable to the baselines.

GPT-6 Astra at max reasoning effort produces competitive edits, primarily by constructing Blender Python scripts to complete the task—a Script-Centric approach. Our contribution is an orthogonal, Visual-Centric approach that performs useful local edits with a lightweight backbone and compact action representations. In the reported comparison setting, ViSculpt with Gemini 3.5 Flash Lite takes approximately 1–2 minutes and USD 0.10 per edit, versus about 10 minutes and USD 4.00 for the Codex baseline at max reasoning effort. These existing measurements motivate assessing editing quality alongside execution cost.

The difference between medium and max in Figure 7 reflects the high demands this Script-Centric approach places on the LLM's reasoning capabilities. ViSculpt, using the lightweight Gemini 3.5 Flash Lite LLM with substantially weaker reasoning capabilities, still produces comparable edits that satisfy the prompt, demonstrating that ViSculpt requires substantially less LLM reasoning capability.

Our design assigns semantic decisions and compact parameters to the agents, while geometric procedures generate dense mouse trajectories. This reduces the low-level trajectory detail that the language model must specify directly and explains the rationale for using a lightweight backbone. Systematic quantitative MCP comparisons and controlled component evaluations remain future work, and we will state these limitations explicitly.

3. Practical relevance of pose editing and diversity of examples

Concern: The editing results are visually unconvincing, and the repeated examples of raising or lowering arms do not sufficiently demonstrate meaningful sculpting capabilities.

Response: Local pose adjustment is a legitimate mesh-editing task. Blender's Pose brush in Sculpt Mode explicitly supports posing unrigged models and adjusting mesh proportions. In practice, Maxon's Fallas sculpture production case study documents a workflow that uses Transpose Master to pose characters before correcting local deformations. These workflows demonstrate the practical relevance of pose adjustment within mesh editing.

We agree that the repeated arm examples do not fully represent the task range. Figure 9 also includes nose reshaping, ear elongation, smoothing, surface roughening, and pattern drawing. We will reduce repetitive pose examples, highlight these existing results, and provide closer views of subtle local changes to communicate the demonstrated range of local operations more clearly.

4. Surface detailing and more challenging sculpting tasks

Concern: Beyond smoothing, moving parts, and writing text, can the system add surface details or change surface styles, for example by turning a cube into an aged stone wall?

Response: Figure 9 already includes surface-detailing operations: adding star patterns to the hat and armadillo surfaces, and roughening the rabbit's torso with the Perlin brush. These tasks are also included in the original user-study questionnaire. We will highlight these existing examples and explain how they are implemented.

These examples use predefined procedural brushes, with the agents spatially locating and executing the requested brush operations. They therefore demonstrate local surface detailing, but do not establish that the system can autonomously create complex styles from unrestricted semantic requests. The proposed transformation from a cube into an aged stone wall remains challenging for ViSculpt. Extending the system to such complex sculpting tasks remains future work.

The contribution we seek to establish is the first multi-agent system for 3D mesh editing through perception and interaction in GUI space: compact primitives connect semantic instructions to executable brush trajectories in GUI space, while visual feedback supports iterative refinement.

5. The SIGGRAPH and GeoPT text in Figure 9

Concern: The SIGGRAPH example is inappropriate for this venue, and GeoPT appears to be a previous name for the method.

Response: Thank you for pointing this out. We will replace these examples in Figure 9 with other existing editing results. These presentation changes will not alter the evaluation materials or rating data from the original user study.

Comments on the paper

1 comment

Dear reviewers,

Thanks for your continued efforts. The authors have provided a rebuttal to the initial reviews. Please review it carefully, along with all other reviews and the corresponding author responses, and consider whether you have any further questions for the authors or whether the rebuttal has addressed your concerns. There are only a few days remaining before the discussion period closes, so please use this time to seek any further clarifications or raise any additional questions about this paper.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.