LEGO-SAM: Segment Any LEGO Model through Chain of Disassembly

This paper introduces LEGO-SAM, the first 3D segmentation framework for partitioning LEGO models into semantic parts. To start, we formulate LEGO segmentation at atom level with our generalized LEGO-Atomizer to bypass ambiguities in brick-level semantics. We then design a training-free Chain-of-Disassembly (CoD) method to segment the atomized LEGO assemblies. Our method jointly leverages visual evidence from a 2D MLLM and structural priors from foundational 3D segmentation models, and takes out well-segmented parts iteratively based on multi-view consistency to reduce occlusion and simplify part relations. Also, we take CoD as a data engine to generate LEGO annotations (1.86M atoms and 67K semantic parts) and take them as supervision to train LEGO-SAM-Flash, a fast native end-to-end LEGO segmentor, with residual adapters and an atom-level segmentation head. Further, we build LEGO-SAM-Bench, the first LEGO segmentation benchmark for quantitative evaluation, providing 347K atoms and 6.4K parts for a wide variety of LEGO models, e.g., indoor rooms, animals, buildings, etc. Experiments show that LEGO-SAM achieves leading performance, outperforming the strongest baseline by 31.18%, and also facilitates various assembly-aware downstream applications such as semantic-aware instruction generation, text-guided grounding, and object/part retrieval. Our code and dataset will be publicly available.