acceptodds
Under review as a conference paper at ICLR 2027

AssemblyVerse: Towards Multimodal 3D Assembly Reasoning & Planning

Abstract

The task of 3D assembly is not merely predicting a sequence of independent part poses; it is a long-horizon, state-conditioned decision problem that couples reasoning about what to assemble next with planning how to move each part into place. Existing benchmarks address either 6-DoF pose reasoning or trajectory prediction in specialized regimes, leaving open the general process-level learning across domains and initial configurations. Towards this end, we introduce AssemblyVerse, a large-scale, multi-domain dataset for process-level multimodal assembly reasoning and planning of diverse objects and complex assembly configurations. AssemblyVerse contains 10,809 assemblies spanning five domains and 192,412 assembly steps. Each step is aligned with rendered instruction diagrams, text instructions, evolving assembly states, and geometrically verified collision-free trajectories. Building on AssemblyVerse, we present AssemblyMind, a vision-language 3D assembly model that grounds each subsequent assembly step through an auxiliary order-grounding head and autoregressively generates 6-DoF trajectories as reordered discrete pose tokens. Our staging-randomized supervised fine-tuning improves robustness to diverse initial part configurations, while GV-GRPO directly optimizes geometric validity of predicted trajectories without sacrificing final-pose accuracy. Extensive experiments across domains demonstrate that strong performance on AssemblyVerse enables transferability of process-level learning. Compared to the previous state of the art, AssemblyMind substantially improves order grounding, final-pose prediction, and collision-free trajectory generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Assembly, MLLMs, Reasoning & Planning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CfPrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies 3D part assembly guided by multimodal instruction manuals, addressing step-to-part matching, final-pose prediction, and assembly trajectory generation conditioned on a given starting pose. The authors introduce AssemblyVerse, which unifies 3D assemblies from five sources and constructs assembly procedures by reversing geometrically verified disassembly trajectories, accompanied by generated visual and textual instructions.

The proposed AssemblyMind builds on Qwen3-VL-2B with a point-cloud encoder and represents trajectories as discrete pose tokens. It predicts the final pose before intermediate waypoints and uses staging-randomized supervised training to improve final-pose accuracy and trajectory generation across starting configurations. GV-GRPO subsequently optimizes trajectories using a reward that requires both a correct final pose and passage of the geometric validity check. The main contributions are a multi-domain dataset of assembly procedures and a multimodal framework integrating part matching, target-pose prediction, and trajectory generation.

Critical strengths
  1. The task formulation is valuable. Beyond predicting where parts belong, the paper considers how each part moves from a given starting pose to its target placement. Subsequent predictions use the previously predicted assembly state, allowing evaluation to reflect the effects of accumulated errors.
  2. The dataset brings together everyday objects, LEGO sets, CAD assemblies, and industrial assemblies, aligning part geometry with step-wise visual and textual instructions and geometrically verified trajectories. Validating candidate disassembly paths before allowing a VLM to select the next part combines semantic selection with geometric feasibility checks and provides a useful resource for further research.
  3. The method is well motivated, and the ablations in Table 4 support the effectiveness of the overall training approach. Predicting the final pose first gives intermediate trajectory generation an explicit goal, while staging randomization reduces reliance on fixed paths from reference starts. Adding these components substantially improves final-pose accuracy and trajectory validity under free-space starts, although the independent contributions of token reordering and final-pose loss weighting require further clarification.
  4. GV-GRPO rewards final-pose correctness and geometric validity without requiring agreement with a single reference path, which is appropriate for tasks with multiple feasible trajectories. Its benefits also appear in assembly-level metrics: in Table 2, E2E success increases from 14.78% to 17.06% under ground-truth order and free-space starts. The cross-dataset experiments provide additional encouraging evidence of transfer.
Critical weaknesses
  1. The benefits of direct trajectory generation over learned target poses followed by conventional motion planning remain unclear. The experiments show that AssemblyMind outperforms the evaluated learning-based baselines, but do not include a comparison in which the same model predicts the final pose and a planner such as RRT-Connect connects the start to that pose. Holding target-pose prediction quality fixed would allow a more direct assessment of the success-rate and inference-time benefits of learned trajectory generation, helping distinguish improvements in final-pose prediction from those in path generation.
  2. The coverage of starting-pose generalization and reinforcement learning optimization is limited. The free-space evaluation described in the appendix primarily changes part translations along the six coordinate-axis directions, while GV-GRPO sampling holds rotations fixed and explores only translations. The results therefore support adaptation within the specified starting-pose distribution, but provide limited direct evidence for arbitrary initial orientations or more general 6-DoF starting conditions. Testing randomized initial rotations and explaining the decision to hold rotations fixed would help establish the method's scope.
  3. The contributions of language inputs and pose-token reordering need clearer ablations. Difference images and point clouds already provide substantial part-identification and geometric information, but the paper does not evaluate removing or shuffling textual instructions, leaving the additional benefit of language unclear. Furthermore, final-pose tokens receive a fivefold loss weight, and Table 4 does not explicitly state whether this weight is held constant across serialization orders. Clarifying this setting is necessary to determine whether the improvement in final-pose accuracy can be independently attributed to token reordering.
Questions for authors
  1. Could the authors add a baseline that uses AssemblyMind to predict the final pose and RRT-Connect to generate the path, comparing success rates and inference time? This would help assess the necessity and practical benefits of directly generating trajectories.
  2. Could the authors evaluate randomized initial rotations? Why does GV-GRPO sampling hold rotations fixed and explore only translations, and could this restriction affect assembly tasks that require substantial rotational motion?
  3. Could the authors provide ablations that remove or shuffle textual instructions to assess their contribution beyond difference images and point clouds? In the pose-token reordering ablation, is the final-pose loss weight held constant across serialization orders?
Reviewer_xCDHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce AssemblyVerse, a large-scale 3D assembly dataset covering five different domains. Each assembly step includes visual and textual instructions, part geometry, target poses, and geometrically verified assembly trajectories. Building on this dataset, the authors propose AssemblyMind, a multimodal assembly model based on Qwen3-VL-2B. The model takes assembly instructions, part point clouds, the current assembly state, and the initial pose of the target part as input to predict the assembly order and 6-DoF trajectories. The authors further introduce pose-token reordering, staging-randomized SFT, and geometry-verified GRPO to improve pose prediction accuracy and trajectory validity.

Critical strengths
  1. The paper provides rich visualizations. The qualitative results across different assembly domains and the full assembly sequences are clearly presented, with additional demonstration videos provided in the supplementary materials.
  2. AssemblyVerse is a valuable dataset contribution. Compared with previous works that mainly focus on final assembly poses, it provides assembly orders, multimodal instructions, and geometrically verified trajectories. It also covers assembly objects with different levels of complexity.
Critical weaknesses
  1. The paper mainly evaluates the geometric validity of the predicted trajectories, which does not necessarily guarantee their physical executability. For example, some screw assembly steps are simplified as pure translations without considering the rotations required for actual fastening or removal. Although the authors provide physics simulation results on AssemblyBench, the current evaluation does not fully demonstrate that the trajectories in AssemblyVerse can be directly executed in real-world assembly tasks.
  2. Error accumulation remains a significant issue in long assembly sequences. Under the predicted-order and free-space-start setting, the end-to-end success rate is only 14.84%. Performance also drops noticeably when using predicted assembly orders instead of ground-truth orders. Moreover, the current framework does not support error recovery or replanning when earlier predictions are inaccurate, which limits its reliability on complex, long-horizon assembly tasks.

I look forward to the authors' responses.

Questions for authors
  1. Do the authors plan to release the dataset, model weights, and code, including the dataset generation and geometric verification pipelines?
  2. What happens with randomized starting orientations, and how does that affect longer sequences with predicted order?
Reviewer_AyLXOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes AssemblyVerse, a dataset of CAD and LEGO assembly procedures. Each assembly step comes with a line-drawing diagram, a VLM instruction, and pose/3D information. Trajectories come from a disassembly planner that certifies each step, and a VLM picks the final candidate. The paper also proposes AssemblyMind, a Qwen-based model with point-cloud tokens. Given a start pose, it outputs the trajectory as discrete pose tokens, with the final pose placed right after the start. It is trained with randomized start positions, then fine-tuned with GRPO using the final-pose and collision checks as a binary reward (GV-GRPO), and a separate head predicts the part order. Results show large gains over retrained AssemblyDyno and Manual-PA on AssemblyVerse, zero-shot results on AssemblyBench and IKEA-Manual, and ablates reordering, staging randomization and GV-GRPO.

Critical strengths
  1. The dataset is well built and valuable, and the procedural pipeline seems quite general and useful. It is more comprehensive than AssemblyBench (CVPR 2026), and adds LEGO and articulated objects (which is a really important addition for complex assembly). The verification process turns existing part-level 3D corpora into verifiable, step-by-step planning data. I can see it being used well beyond this paper, for example as supervision or as an RL environment with a checkable reward for improving the spatial planning and reasoning of VLMs.

  2. The start-conditioned protocol and its ablations are quite informative.

Critical weaknesses
  1. There is no fair reference point for current frontier models, so I cannot tell how much headroom the benchmark has, or whether the proposed recipe still matters on a stronger base model. The best baseline AssemblyMind is built on Qwen3-VL, released in October 2025. It would be valuable to add newer models (Qwen 3.6, 3.6, 3.8) as well as more "frontier" models like Gemini 3.1 Pro, Fable 5, Sol 5.6; and this is already ignoring the slew of even newer models that are significantly better at this family of tasks (released after paper deadline). I'd expect current frontier models to do well on at least parts of this task under a fair protocol. I understand this is a challenge faced by the broader field today, but it is important to understand what the gap between the frontier and proposed method is. If so, most of the paper's value rests on the dataset, and the paper should test this and discuss how the dataset and the method stay useful as base models improve.

  2. It is not clear how much the method adds beyond plain SFT on this data, and the baseline comparisons are not controlled. Table 4's "Baseline" is plain SFT on the same data and backbone with natural token order (its numbers match Qwen3-VL-2B). As far as I can tell, the start pose also keeps the seated orientation for most steps, which would make the final pose mostly a translation problem (Q3). I think the dataset is a real contribution, and I am willing to raise my score if the rebuttal adds a release commitment with a license, a deduplication check between AssemblyBench and the Fusion 360 training data, natural-order SFT with randomized starts, and at least one frontier-model reference under a fair protocol.

Questions for authors
  1. (Suggestion) Please evaluate current proprietary VLMs and the newest open VLMs under a fair protocol, zero-shot and few-shot. Candidates are Gemini 3.1 Pro, Claude Fable 5, GPT-5.6 Sol and Qwen3.8-27B (this time with the same inputs AssemblyMind gets).

  2. (Suggestion) To separate method from data, please add natural-order SFT with the same randomized starts, a final-pose-only output followed by a straight line from the start, and a simple matcher in place of the order head.

  3. (Clarification) Is the start orientation ever different from the seated orientation, in training or at test time? What fraction of test steps use an RRT-Connect path? Please report Table 2 with random start orientations, and say which start pose AssemblyMind receives on AssemblyBench

  4. (Suggestion) Please retrain AssemblyDyno and Manual-PA to convergence (both original papers train the pose model for 1,000 epochs), state Manual-PA's training budget here, and explain how assemblies with more than 20 parts were handled, since the original Manual-PA filters those out.

  5. (Clarification) How many AssemblyBench test assemblies, or near-duplicates of them, appear in your Fusion 360 split, and what does Table 3 look like after removing them?

  6. Will the data, planner, code and weights be released, and under what license? Where do the inverter assemblies come from, and may they be redistributed? How will the non-commercial terms of Fusion 360 Gallery and PartNet-Mobility (which also inherits the ShapeNet terms) be passed on?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.