- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper segments digital LEGO models into semantic parts. Each brick is split into the smallest units called atoms, and labels are assigned per atom. LEGO-SAM renders the model from multiple views, asks an MLLM to paint 2D segmentation masks, lifts them onto the visible atoms, merges them across views, and uses P3-SAM to label the unseen atoms. Retrieved segmentations with high multi-view agreement are removed from the model, and the remaining model is segmented again. LEGO-SAM-Flash distills this into P3-SAM with adapters and an atom pooling head, trained on over 4000 pseudo-labeled models. A benchmark of 300 hand-annotated models is introduced. On this benchmark, the proposed methods reach a bounding box mIoU of around 67, compared to 36 for the best baseline.
Critical strengths
The argumentation for operating on an atom-level makes sense, and the method of isolating them from a big LEGO model is a decent contribution and a reusable concept that can be used for other segmentation methods
The 300 LEGO model benchmark is valuable if it is released as promised, since it is annotated per atom (by LEGO designers) and could be useful for further work in this domain.
Critical weaknesses
There are concerns regarding the evaluation. The only description of the baseline setup is that they are applied using their recommended best practices, plus a cap on their predicted parts in Appendix C. It is not stated what input they receive or whether they are trained on the LEGO data. This matters because these methods were developed for generic shapes and have never seen atom-level annotations, while LEGO-SAM-Flash is trained on over 4000 in-domain models. Also, the importance of operating at the atom level has been stressed throughout this work, but there are no experiments showing that it really matters.
The method section, especially 4.1, is really hard to follow and still not fully clear after several reads.
Questions for authors
You motivate the work with LEGO models reproducing real-world things. Is any part of the method, for example, the atom head or the disassembly loop, meant to be transferred to other assembly data, and has this been tried?
How are the well-segmented parts removed? From my understanding, only the view consistency decides this. Is the brick connectivity used at all?
The evaluation uses bounding box IoU per part, which does not check whether atoms inside a part are assigned correctly. Can you report an atom-level metric?
What input do the baselines receive, what data are they trained on, and how are their outputs turned into parts?
Is the benchmark contained in the 4000 training models? Flash beats its own teacher by 14 mIoU on figurines, which needs an explanation.
4 comments
Q: Importance of the atom-level operation
A: Atom-level operations are essential for correctly segmenting LEGO bricks that span multiple semantic parts, which cannot be handled by brick-level segmentation. To quantify the prevalence of such cases, we identify bricks whose constituent atoms are assigned different part labels in the human-annotated benchmark. We find that 55% of the indoor scene models contain bricks crossing different semantic parts. Similar cases occur in 92% of car models, 69% of ship models, and 100% of plane models. These results show that bricks spanning multiple semantic parts are common in LEGO assemblies, highlighting the necessity of atom-level segmentation to resolve brick-level ambiguity and accurately segment semantic parts. We now report this analysis information in Appendix B.
Q: LEGO-SAM-Flash
A: The 4,000 training models pseudo-labeled by our LEGO-SAM pipeline are entirely separate from the 300 human-labeled models in LEGO-SAM-Bench. The two sets have no overlap. LEGO-SAM-Flash outperforms LEGO-SAM on figurines, since figurines exhibit relatively small structural variability compared with indoor scenes and other LEGO categories. Specifically, figurine models have a mean part count of 9.8 with a standard deviation of 1.4, compared with 26.0 ± 14.5 for indoor scenes and 28.4 ± 35.0 for other categories. This lower variability facilitates effective learning from in-domain supervision. In contrast, the greater structural diversity and complexity of indoor scenes and other categories favor LEGO-SAM's iterative Chain of Disassembly, which progressively exposes occluded structures and simplifies part relationships. We have now included the dataset statistics in Appendix B and the analysis in Appendix F.
Q: Method section
A: Thank you for pointing this out. To improve the readability, we have now included detailed step-by-step pseudocode in Appendix C. Specifically, our LEGO-aware structural segmentation consists of four key stages: (1) MLLM-guided view selection and 2D segmentation, (2) lifting 2D masks into 3D, (3) cross-view merging of lifted masks into global 3D proposals, and (4) proposal-prompted 3D structural segmentation using P3-SAM. Additionally, we will release our code and trained models for reproducibility.
Q: Well-segmented part removal
A: Yes, only the view consistency determines the removal of well-segmented parts. We observe that multi-view consistency is already effective for progressive disassembly, and we do not explicitly utilize the brick connectivity. The motivation for Chain of Disassembly is that many LEGO assemblies contain occluded parts or entirely invisible atoms that cannot be reliably segmented in a single multi-view processing pass. When a part is consistently segmented across multiple views, we assign a semantic label to it and can remove it from the assembly during subsequent rounds of processing. This new approach helps to expose occluded structures, such that we can progressively segment a given 3D assembly until all the parts are consistently segmented.
Q: Generalization to Other Assemblies
A: We acknowledge this as a very interesting direction useful for future research, and have included further discussion in Section 7 of the updated manuscript. While our current implementation focuses on LEGO models, we believe the underlying ideas can potentially generalize to other 3D assembly paradigms, such as robotic construction kits and mechanical CAD assemblies. Specifically, the atom-level segmentation can be adapted to assemblies with discrete primitives, while the Chain-of-Disassembly framework can potentially benefit assemblies with occlusion and complex part relations.
Reviewer Comment
Thank you for the detailed response. Many of my questions were resolved, in particular through the atom-set metrics, the LEGO-adapted P3-SAM baseline, and the baseline details.
I did not appreciate the way parts of the response were communicated. A capitalized statement of contributions does not replace evidence, and I would have preferred the response without it. Beyond this, I am still not convinced of the broader importance outside of LEGO-specific tasks. While the task and the benchmark are new, I see the technical novelty as limited. Flash adds standard adapters and atom pooling to a frozen P3-SAM. LEGO-SAM is a training-free combination of two large pretrained models, where the disassembly loop is the main new element. This makes a clear description of the method all the more important. I flagged Section 4.1 as not understandable, and it is unchanged in the revised main paper. The new pseudocode helps, but it sits only in the appendix. Since this section describes the core of the method, it needs to be addressed in the main paper itself.
Dear Reviewer,
Thank you for your further response and for confirming that your concerns about atom-set metrics, LEGO-adapted P3-SAM baseline, and baseline details have been addressed.
For better clarity, we have added a more detailed description of the method in the revised version. In Section 4.1, we present the pipeline in four steps in a more organized way, provide more details on mask lifting, cross-view merging, and 3D structural segmentation, and refer to the step-by-step pseudocode in Appendix C.1. In Section 4.2, we clarify that part removal is determined by multi-view consistency and refer to the pseudocode in Appendix C.2. In Section 5, we add the inference details of LEGO-SAM-Flash.
Our primary target is LEGO, which already has an active research community with practical relevance. Part-level understanding can support more applications, including instruction generation, part-level editing, and model retrieval. Beyond LEGO, many everyday objects, such as household appliances and mechanical devices, share the same structure: they are solid assemblies of discrete components whose internal parts become occluded after assembly. Since the Chain-of-Disassembly pipeline is designed to handle exactly this kind of occlusion, we believe that our approach could also benefit part understanding in such settings. We have added this discussion as future work in the revised paper.
Dear Reviewer,
Thank you for recognizing our method as "a decent contribution and a reusable concept" and our benchmark as "valuable". The other reviewers also recognize our task as "having practical relevance", our work as "systematically studying semantic part segmentation for 3D LEGO models", and our Chain-of-Disassembly idea as "well tailored to the discrete and heavily occluded nature of LEGO assemblies."
To start, we would like to emphasize that this work introduces a NEW TASK and a NEW SOLUTION. This is the first research work on semantic segmentation of LEGO models. We formulate the task, which has not been explored by any prior work, and propose the LEGO-SAM framework with the novel Chain-of-Disassembly idea and atomized LEGO segmentation; in addition, we propose to distill the semantic understanding capability of LEGO-SAM into an end-to-end Flash model. Our models achieve leading performance compared to all state-of-the-art generic 3D segmentation models and also a LEGO-adapted P3-SAM model fine-tuned on our LEGO data. We will release data and code.
Q: Baseline setup
A: We acknowledge that the baseline settings were not sufficiently detailed in the original manuscript, as we followed their standard inference pipelines and configurations to ensure a fair comparison on our benchmark. We have now provided further details, clarifying their implementations in Appendix C and summarizing their inputs, training supervision, and outputs in Appendix Table 2.
Specifically, SAMesh and GeoSAM2 take multi-view renderings as input, generate 2D segmentation masks, and lift them into 3D mesh-face part labels. PartField, P3-SAM, and PartSAM operate on point clouds sampled from the input meshes. PartField extracts per-face features and groups them into parts using agglomerative clustering. P3-SAM and PartSAM additionally sample prompt points using farthest point sampling, predict point-level masks from these prompts, and apply non-maximum suppression. The resulting point-level labels are then transferred to mesh faces through majority voting.
Regarding the training data, SAMesh is training-free, while the other baselines leverage pretrained models trained on generic 3D datasets. Some are trained on large-scale datasets, such as approximately 3.7M shapes for P3-SAM and 5M for PartSAM. It is unclear whether these datasets contain LEGO-related shapes, as none of these methods explicitly reports training on LEGO-specific datasets. These additional details have been included in the revised submission.
Q: Atom-level metrics
A: This is a good point. In addition to axis-aligned bounding-box IoU (aabbIoU), we now report atom-set IoU (asIoU) and its associated F1 score, precision, and recall to directly evaluate atom-level segmentation accuracy. The following table summarizes the updated asIoU and asF1 results across the three LEGO category groups in the evaluation. All values are expressed as percentages (%), with the best results highlighted in bold.
In the above table, LEGO-adapted P3-SAM is a newly included baseline by fine-tuning P3-SAM on the same 4,000 LEGO models, which are pseudo-labeled data produced by our LEGO-SAM pipeline. These results show that both LEGO-SAM and LEGO-SAM-Flash consistently outperform all the baselines across the three category groups, in terms of both asIoU and asF1, demonstrating that our models achieve more accurate atom-level part assignments, in addition to bounding-box overlap. We have provided the associated definition and computation of asIoU in Appendix E, with detailed quantitative results, including the F1 scores, precision, and recall, reported in Appendix F.