acceptodds
Under review as a conference paper at ICLR 2027

LEGO-SAM: Segment Any LEGO Model through Chain of Disassembly

Abstract

This paper introduces LEGO-SAM, the first 3D segmentation framework for partitioning LEGO models into semantic parts. To start, we formulate LEGO segmentation at atom level with our generalized LEGO-Atomizer to bypass ambiguities in brick-level semantics. We then design a training-free Chain-of-Disassembly (CoD) method to segment the atomized LEGO assemblies. Our method jointly leverages visual evidence from a 2D MLLM and structural priors from foundational 3D segmentation models, and takes out well-segmented parts iteratively based on multi-view consistency to reduce occlusion and simplify part relations. Also, we take CoD as a data engine to generate LEGO annotations (1.86M atoms and 67K semantic parts) and take them as supervision to train LEGO-SAM-Flash, a fast native end-to-end LEGO segmentor, with residual adapters and an atom-level segmentation head. Further, we build LEGO-SAM-Bench, the first LEGO segmentation benchmark for quantitative evaluation, providing 347K atoms and 6.4K parts for a wide variety of LEGO models, e.g., indoor rooms, animals, buildings, etc. Experiments show that LEGO-SAM achieves leading performance, outperforming the strongest baseline by 31.18%, and also facilitates various assembly-aware downstream applications such as semantic-aware instruction generation, text-guided grounding, and object/part retrieval. Our code and dataset will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D part segmentation, LEGO assembly understanding, open-world segmentation, chain of disassembly

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_VTt1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper segments digital LEGO models into semantic parts. Each brick is split into the smallest units called atoms, and labels are assigned per atom. LEGO-SAM renders the model from multiple views, asks an MLLM to paint 2D segmentation masks, lifts them onto the visible atoms, merges them across views, and uses P3-SAM to label the unseen atoms. Retrieved segmentations with high multi-view agreement are removed from the model, and the remaining model is segmented again. LEGO-SAM-Flash distills this into P3-SAM with adapters and an atom pooling head, trained on over 4000 pseudo-labeled models. A benchmark of 300 hand-annotated models is introduced. On this benchmark, the proposed methods reach a bounding box mIoU of around 67, compared to 36 for the best baseline.

Critical strengths

The argumentation for operating on an atom-level makes sense, and the method of isolating them from a big LEGO model is a decent contribution and a reusable concept that can be used for other segmentation methods

The 300 LEGO model benchmark is valuable if it is released as promised, since it is annotated per atom (by LEGO designers) and could be useful for further work in this domain.

Critical weaknesses

There are concerns regarding the evaluation. The only description of the baseline setup is that they are applied using their recommended best practices, plus a cap on their predicted parts in Appendix C. It is not stated what input they receive or whether they are trained on the LEGO data. This matters because these methods were developed for generic shapes and have never seen atom-level annotations, while LEGO-SAM-Flash is trained on over 4000 in-domain models. Also, the importance of operating at the atom level has been stressed throughout this work, but there are no experiments showing that it really matters.

The method section, especially 4.1, is really hard to follow and still not fully clear after several reads.

Questions for authors

You motivate the work with LEGO models reproducing real-world things. Is any part of the method, for example, the atom head or the disassembly loop, meant to be transferred to other assembly data, and has this been tried?

How are the well-segmented parts removed? From my understanding, only the view consistency decides this. Is the brick connectivity used at all?

The evaluation uses bounding box IoU per part, which does not check whether atoms inside a part are assigned correctly. Can you report an atom-level metric?

What input do the baselines receive, what data are they trained on, and how are their outputs turned into parts?

Is the benchmark contained in the 4000 training models? Flash beats its own teacher by 14 mIoU on figurines, which needs an explanation.

4 comments

Dear Reviewer,

Thank you for recognizing our method as "a decent contribution and a reusable concept" and our benchmark as "valuable". The other reviewers also recognize our task as "having practical relevance", our work as "systematically studying semantic part segmentation for 3D LEGO models", and our Chain-of-Disassembly idea as "well tailored to the discrete and heavily occluded nature of LEGO assemblies."

To start, we would like to emphasize that this work introduces a NEW TASK and a NEW SOLUTION. This is the first research work on semantic segmentation of LEGO models. We formulate the task, which has not been explored by any prior work, and propose the LEGO-SAM framework with the novel Chain-of-Disassembly idea and atomized LEGO segmentation; in addition, we propose to distill the semantic understanding capability of LEGO-SAM into an end-to-end Flash model. Our models achieve leading performance compared to all state-of-the-art generic 3D segmentation models and also a LEGO-adapted P3-SAM model fine-tuned on our LEGO data. We will release data and code.

Q: Baseline setup

A: We acknowledge that the baseline settings were not sufficiently detailed in the original manuscript, as we followed their standard inference pipelines and configurations to ensure a fair comparison on our benchmark. We have now provided further details, clarifying their implementations in Appendix C and summarizing their inputs, training supervision, and outputs in Appendix Table 2.

Specifically, SAMesh and GeoSAM2 take multi-view renderings as input, generate 2D segmentation masks, and lift them into 3D mesh-face part labels. PartField, P3-SAM, and PartSAM operate on point clouds sampled from the input meshes. PartField extracts per-face features and groups them into parts using agglomerative clustering. P3-SAM and PartSAM additionally sample prompt points using farthest point sampling, predict point-level masks from these prompts, and apply non-maximum suppression. The resulting point-level labels are then transferred to mesh faces through majority voting.

Regarding the training data, SAMesh is training-free, while the other baselines leverage pretrained models trained on generic 3D datasets. Some are trained on large-scale datasets, such as approximately 3.7M shapes for P3-SAM and 5M for PartSAM. It is unclear whether these datasets contain LEGO-related shapes, as none of these methods explicitly reports training on LEGO-specific datasets. These additional details have been included in the revised submission.

Q: Atom-level metrics

A: This is a good point. In addition to axis-aligned bounding-box IoU (aabbIoU), we now report atom-set IoU (asIoU) and its associated F1 score, precision, and recall to directly evaluate atom-level segmentation accuracy. The following table summarizes the updated asIoU and asF1 results across the three LEGO category groups in the evaluation. All values are expressed as percentages (%), with the best results highlighted in bold.

MethodFigurines asIoU↑Figurines asF1↑Indoor asIoU↑Indoor asF1↑Others asIoU↑Others asF1↑
SAMesh18.6212.6523.5015.2127.4017.08
GeoSAM213.7225.7213.4120.4111.9620.03
PartField36.4832.9421.6328.5725.5628.72
P3-SAM46.3541.7731.6734.3630.6130.96
PartSAM45.9632.6934.0228.3135.3627.81
LEGO-adapted P3-SAM64.0350.2443.6127.4033.5129.19
LEGO-SAM-Flash (ours)84.3168.0056.9138.7347.2639.97
LEGO-SAM (ours)68.6552.9161.5448.9254.0543.21

In the above table, LEGO-adapted P3-SAM is a newly included baseline by fine-tuning P3-SAM on the same 4,000 LEGO models, which are pseudo-labeled data produced by our LEGO-SAM pipeline. These results show that both LEGO-SAM and LEGO-SAM-Flash consistently outperform all the baselines across the three category groups, in terms of both asIoU and asF1, demonstrating that our models achieve more accurate atom-level part assignments, in addition to bounding-box overlap. We have provided the associated definition and computation of asIoU in Appendix E, with detailed quantitative results, including the F1 scores, precision, and recall, reported in Appendix F.

Q: Importance of the atom-level operation

A: Atom-level operations are essential for correctly segmenting LEGO bricks that span multiple semantic parts, which cannot be handled by brick-level segmentation. To quantify the prevalence of such cases, we identify bricks whose constituent atoms are assigned different part labels in the human-annotated benchmark. We find that 55% of the indoor scene models contain bricks crossing different semantic parts. Similar cases occur in 92% of car models, 69% of ship models, and 100% of plane models. These results show that bricks spanning multiple semantic parts are common in LEGO assemblies, highlighting the necessity of atom-level segmentation to resolve brick-level ambiguity and accurately segment semantic parts. We now report this analysis information in Appendix B.

Q: LEGO-SAM-Flash

A: The 4,000 training models pseudo-labeled by our LEGO-SAM pipeline are entirely separate from the 300 human-labeled models in LEGO-SAM-Bench. The two sets have no overlap. LEGO-SAM-Flash outperforms LEGO-SAM on figurines, since figurines exhibit relatively small structural variability compared with indoor scenes and other LEGO categories. Specifically, figurine models have a mean part count of 9.8 with a standard deviation of 1.4, compared with 26.0 ± 14.5 for indoor scenes and 28.4 ± 35.0 for other categories. This lower variability facilitates effective learning from in-domain supervision. In contrast, the greater structural diversity and complexity of indoor scenes and other categories favor LEGO-SAM's iterative Chain of Disassembly, which progressively exposes occluded structures and simplifies part relationships. We have now included the dataset statistics in Appendix B and the analysis in Appendix F.

Q: Method section

A: Thank you for pointing this out. To improve the readability, we have now included detailed step-by-step pseudocode in Appendix C. Specifically, our LEGO-aware structural segmentation consists of four key stages: (1) MLLM-guided view selection and 2D segmentation, (2) lifting 2D masks into 3D, (3) cross-view merging of lifted masks into global 3D proposals, and (4) proposal-prompted 3D structural segmentation using P3-SAM. Additionally, we will release our code and trained models for reproducibility.

Q: Well-segmented part removal

A: Yes, only the view consistency determines the removal of well-segmented parts. We observe that multi-view consistency is already effective for progressive disassembly, and we do not explicitly utilize the brick connectivity. The motivation for Chain of Disassembly is that many LEGO assemblies contain occluded parts or entirely invisible atoms that cannot be reliably segmented in a single multi-view processing pass. When a part is consistently segmented across multiple views, we assign a semantic label to it and can remove it from the assembly during subsequent rounds of processing. This new approach helps to expose occluded structures, such that we can progressively segment a given 3D assembly until all the parts are consistently segmented.

Q: Generalization to Other Assemblies

A: We acknowledge this as a very interesting direction useful for future research, and have included further discussion in Section 7 of the updated manuscript. While our current implementation focuses on LEGO models, we believe the underlying ideas can potentially generalize to other 3D assembly paradigms, such as robotic construction kits and mechanical CAD assemblies. Specifically, the atom-level segmentation can be adapted to assemblies with discrete primitives, while the Chain-of-Disassembly framework can potentially benefit assemblies with occlusion and complex part relations.

Reviewer_VTt1OpenReview ↗

Reviewer Comment

Thank you for the detailed response. Many of my questions were resolved, in particular through the atom-set metrics, the LEGO-adapted P3-SAM baseline, and the baseline details.

I did not appreciate the way parts of the response were communicated. A capitalized statement of contributions does not replace evidence, and I would have preferred the response without it. Beyond this, I am still not convinced of the broader importance outside of LEGO-specific tasks. While the task and the benchmark are new, I see the technical novelty as limited. Flash adds standard adapters and atom pooling to a frozen P3-SAM. LEGO-SAM is a training-free combination of two large pretrained models, where the disassembly loop is the main new element. This makes a clear description of the method all the more important. I flagged Section 4.1 as not understandable, and it is unchanged in the revised main paper. The new pseudocode helps, but it sits only in the appendix. Since this section describes the core of the method, it needs to be addressed in the main paper itself.

Dear Reviewer,

Thank you for your further response and for confirming that your concerns about atom-set metrics, LEGO-adapted P3-SAM baseline, and baseline details have been addressed.

For better clarity, we have added a more detailed description of the method in the revised version. In Section 4.1, we present the pipeline in four steps in a more organized way, provide more details on mask lifting, cross-view merging, and 3D structural segmentation, and refer to the step-by-step pseudocode in Appendix C.1. In Section 4.2, we clarify that part removal is determined by multi-view consistency and refer to the pseudocode in Appendix C.2. In Section 5, we add the inference details of LEGO-SAM-Flash.

Our primary target is LEGO, which already has an active research community with practical relevance. Part-level understanding can support more applications, including instruction generation, part-level editing, and model retrieval. Beyond LEGO, many everyday objects, such as household appliances and mechanical devices, share the same structure: they are solid assemblies of discrete components whose internal parts become occluded after assembly. Since the Chain-of-Disassembly pipeline is designed to handle exactly this kind of occlusion, we believe that our approach could also benefit part understanding in such settings. We have added this discussion as future work in the revised paper.

Reviewer_VH3fOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates semantic component segmentation for complete digital LEGO assembly models. The paper first decompose the original bricks into smaller LEGO atoms using integer linear programming to address the issue of a single brick spanning multiple semantic components. It then lifts and merges the multi-view segmentation maps produced by an image model into 3D part proposals, and uses P3-SAM to extend these proposals to include previously unlabeled atoms. Chain-of-Disassembly (CoD) selects and removes reliable parts based on multi-view consistency, progressively exposing occluded structures. The final segmentation aggregates the parts obtained across iterations.

The authors further use this process to generate pseudo-labels for 4,000 models and trained a fast model LEGO-SAM-Flash, using residual adapters and atom-level feature pooling. On a test set containing 300 manually annotated models, both methods significantly outperforme the compared general segmentation methods on bounding box-based metrics.

Critical strengths

The problem has practical relevance, as semantic part segmentation can support part retrieval, replacement, and assembly instruction generation.

CoD allows subsequent rendering to expose more of the previously occluded structure, making the design motivation reasonable. In Table 2, after removing CoD, the bounding box mIoU decreased from 67.40 to 57.04, providing evidence for the effectiveness of iterative processing in this system.

Under the reported evaluation protocol, both methods substantially outperform the evaluated general-purpose baselines across all three category groups.

Critical weaknesses

The paper is motivated by the need to resolve semantic boundaries at the atom level and correctly assign occluded interior atoms to parts. However, all quantitative segmentation metrics are based on axis-aligned bounding boxes. The mIoU measures bounding box overlap, while precision, recall, and F1 use a bounding box IoU ≥ 0.3 to determine correct matching. Misassigning internal atoms does not necessarily change the bounding box of a component. Therefore, improvements in bounding box metrics cannot directly prove that structural completion and CoD improve the attribution of internal atoms. It cannot fully verify the accuracy of detailed boundaries either.

Flash is a major contribution of this paper, but its core technology is described rather vaguely, which affects reproducibility.

I find the paper do not report the results of adapting P3-SAM using the same batch of pseudo-labeled data, nor do they separately conduct ablation studies the two structural modifications. Therefore, it is currently impossible to determine whether the improvement mainly comes from the added domain supervision or from the proposed model design.

Furthermore, the overall average may mask significant class differences. Compared to LEGO-SAM, Flash improves the mIoU for the character category from 71.45 to 85.92, but decreases from 69.57 to 58.51 for indoor scenes, and from 61.18 to 55.64 for other categories. Thus, similar overall performance does not prove that Flash preserves the teacher's segmentation quality consistently across categories.

Questions for authors

Could you report atom-set IoU, including separate results for atoms visible in at least one initially selected view and atoms invisible in all initially selected views?

How does Flash compare with P3-SAM adapted using the same pseudo-labeled training dataset under comparable training conditions? What are the individual contributions of the residual adapters and atom-level pooling?

Does Flash require sampled point prompts or other queries at inference time, and how many decoder calls are needed per model? How are duplicate masks removed, conflicting atom assignments resolved, and uncovered atoms handled? Please clarify which inference steps are inherited unchanged from P3-SAM and which are modified for Flash.

2 comments

Dear Reviewer,

Thank you for your thoughtful and detailed review, particularly for recognizing the practical relevance of LEGO semantic part segmentation and the effectiveness of our Chain-of-Disassembly approach. Your feedback has helped us strengthen the experimental evaluation and improve the clarity of this work. We address your concerns in detail below.

Q: Atom-level metrics

A: This is a good point. In addition to axis-aligned bounding-box IoU (aabbIoU), we now report atom-set IoU (asIoU) and its associated F1 score, precision, and recall to directly evaluate atom-level segmentation accuracy. The following table summarizes the updated asIoU and asF1 results across the three LEGO category groups in the evaluation. All values are expressed as percentages (%), with the best results highlighted in bold.

MethodFigurines asIoU↑Figurines asF1↑Indoor asIoU↑Indoor asF1↑Others asIoU↑Others asF1↑
SAMesh18.6212.6523.5015.2127.4017.08
GeoSAM213.7225.7213.4120.4111.9620.03
PartField36.4832.9421.6328.5725.5628.72
P3-SAM46.3541.7731.6734.3630.6130.96
PartSAM45.9632.6934.0228.3135.3627.81
LEGO-adapted P3-SAM64.0350.2443.6127.4033.5129.19
LEGO-SAM-Flash (ours)84.3168.0056.9138.7347.2639.97
LEGO-SAM (ours)68.6552.9161.5448.9254.0543.21

In the above table, LEGO-adapted P3-SAM is a newly included baseline by fine-tuning P3-SAM on the same 4,000 LEGO models, which are pseudo-labeled data produced by our LEGO-SAM pipeline. These results show that both LEGO-SAM and LEGO-SAM-Flash consistently outperform all the baselines across the three category groups, in terms of both asIoU and asF1, demonstrating that our models achieve more accurate atom-level part assignments, in addition to bounding-box overlap. We have provided the associated definition and computation of asIoU in Appendix E, with detailed quantitative results, including the F1 scores, precision, and recall, reported in Appendix F.

Q: Ablation study of LEGO-SAM-Flash

A: Following your suggestion, we conducted an ablation study to evaluate the individual contributions of the residual adapters and the atom-level segmentation head in LEGO-SAM-Flash. Specifically, we evaluate four configurations: (1) LEGO-adapted P3-SAM fine-tuned on the same pseudo-labeled dataset, (2) Flash without residual adapters, (3) Flash without the atom-level segmentation head, and (4) the full Flash model. The quantitative results are summarized below.

MethodaabbIoU ↑aabbF1 ↑asIoU ↑asF1 ↑
LEGO-adapted P3-SAM46.1129.8147.0535.61
Flash w/o adapter57.4132.8056.7234.42
Flash w/o atom head52.8332.0353.3634.12
LEGO-SAM-Flash (full)66.6944.4162.8348.90

For a fair comparison, we fine-tune both the LEGO-adapted P3-SAM and Flash without residual adapters using LoRA, with the rank and scaling parameter (alpha) both set to 16. These two configurations each have 352,640 trainable parameters, comparable to the 265,603 trainable parameters of our full Flash model. The results show that removing either the residual adapters or the atom-level segmentation head degrades the performance of Flash, demonstrating the effectiveness of both components. In particular, removing the atom-level segmentation head causes a larger performance drop, highlighting the importance of incorporating atom-level geometric priors into segmentation. We have included these ablation results in the revised main paper Section 6.

Q: Visibility-Aware Evaluation

A: Thank you for this neat suggestion. We additionally conducted a visibility-aware evaluation. Specifically, we partition the atoms into visible and invisible subsets based on their visibility in the 12 initially selected views, and compute the atom-set IoU (asIoU) separately on these two subsets. The results are summarized below.

MethodasIoU for visible atoms ↑asIoU for invisible atoms ↑
SAMesh24.1713.88
GeoSAM225.6813.52
PartField28.8342.33
P3-SAM36.4547.44
PartSAM40.5049.05
LEGO-adapted P3-SAM44.4753.41
LEGO-SAM-Flash (ours)59.8060.58
LEGO-SAM (ours)60.1357.05

The results show that lifting-based methods (SAMesh and GeoSAM2) perform better on visible atoms, reflecting their reliance on the rendered views and the 2D segmentation. In contrast, feedforward 3D segmentation methods generally perform better on invisible atoms, benefiting from their learned 3D structural priors. Both our methods outperform all the baselines on both visible and invisible atoms. In particular, LEGO-SAM mitigates the limitations of view-based segmentation through CoD (Chain of Disassembly), which progressively exposes occluded structures and enables more accurate segmentation of the initially invisible atoms. We have now included these results and the associated analysis in Section 6 of the revised manuscript, with detailed category-wise results in Appendix F.

Q: Performance comparison between LEGO-SAM and LEGO-SAM-Flash

A: Thank you for pointing this out. LEGO-SAM-Flash aims to distill the knowledge of LEGO-SAM into an efficient feedforward model, balancing segmentation quality and inference efficiency. Flash substantially outperforms the original baselines while significantly reducing the inference time compared with LEGO-SAM.

We observe that LEGO-SAM-Flash underperforms LEGO-SAM on structurally complex categories, such as indoor scenes and others. This is because LEGO-SAM benefits from its iterative Chain of Disassembly, which progressively exposes occluded structures and simplifies part relationships, making it particularly effective for these challenging categories. We have added the corresponding analysis in Appendix F. As the future work, we plan to explore advanced distillation techniques, such as confidence-aware distillation and complexity-aware importance sampling, to further improve performance of Flash, as discussed in Section 7.

Q: Inference of LEGO-SAM-Flash

A: This is a good suggestion. Thank you! We additionally included the inference details of LEGO-SAM-Flash in Section 5 and provided step-by-step pseudocode in Appendix C.2. We will also open-source our implementation for reproducibility. In detail, Flash follows the overall inference pipeline of P3-SAM, including surface sampling, prompt sampling, feature extraction, mask decoding, non-maximum suppression (NMS), conflict resolution, and completion of uncovered regions. We adapt key procedures for atom-level segmentation.

Flash automatically selects up to 400 atoms using FPS (or all atoms if the model has fewer than 400 atoms), with one sampled point per selected atom serving as a prompt. Point features are extracted only once, and prompts are processed in batches of at most 32, requiring at most 13 decoder calls per assembly.

During post-processing, duplicate masks are suppressed using sampled-point-weighted atom-set IoU with a threshold of 0.9. Conflicting assignments are resolved by processing masks from largest to smallest, allowing smaller masks to overwrite larger ones. For uncovered atoms, connected components exceeding 0.1% of all sampled points receive new part labels. Labels are then iteratively propagated to the remaining unlabeled atoms by majority voting among their labeled atom neighbors.

Reviewer_CyeGOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LEGO-SAM, a first framework for semantic part segmentation of 3D LEGO models. It introduces atom-level segmentation to address brick-level ambiguity and a training-free Chain-of-Disassembly pipeline that combines MLLM-derived 2D semantics with pretrained 3D structural priors while progressively exposing occluded parts. LEGO-SAM is also used to generate pseudo-labels for training the efficient LEGO-SAM-Flash. The authors further contribute the manually annotated LEGO-SAM-Bench. Both models substantially outperform existing 3D segmentation baselines and support applications such as instruction generation, text-guided grounding, and part retrieval.

Critical strengths
  1. The paper systematically studies semantic part segmentation for 3D LEGO models. Unlike prior work that primarily focuses on geometry and brick connectivity, semantic understanding enables downstream applications such as instruction generation, text-guided grounding, and part retrieval and replacement.
  2. Point-level segmentation may over-fragment LEGO bricks, while brick-level segmentation is ambiguous when one brick spans multiple semantic parts. The proposed LEGO-Atomizer decomposes bricks into minimal non-decomposable atoms, enabling finer semantic labeling while preserving geometry, connectivity, and assemblability.
  3. The paper uses LEGO-SAM to generate pseudo-labels for 4,000 models, covering approximately 1.86M atoms and 67K semantic parts. It also introduces LEGO-SAM-Bench, manually annotated by professional LEGO® designers, containing 300 models, 347K atoms, and 6,4K part annotations across diverse categories.
  4. LEGO-SAM combines 2D semantic evidence from an MLLM with structural priors from a pretrained 3D segmentor. Based on multi-view consistency, it iteratively removes reliably segmented parts, progressively exposing occluded internal structures. This design is well tailored to the discrete and heavily occluded nature of LEGO® assemblies.
Critical weaknesses
  1. The five compared baselines (SAMesh, GeoSAM2, PartField, P3-SAM, PartSAM) are general-purpose 3D segmentors with no LEGO-specific adaptation, maybe need some strong baselines.
  2. Benchmark is annotated by two professional LEGO designers, with an open label set, and no inter-annotator agreement (IAA) reported. Yet what counts as one semantic part is highly subjective (is a window a part of the car body or a separate part?). Without IAA, the reliability of the gold standard is questionable.
  3. LEGO-SAM-Flash suffers from a self-bootstrapping / circular-training risk. LEGO-SAM-Flash is trained on 1.86M atoms / 67K parts auto-generated by CoD across 4,000 models. LEGO-SAM-Flash‘s benchmark score is close to CoD's — but is this generalization, or merely replication of CoD's systematic errors?
Questions for authors
  1. LEGO-SAM-Bench was annotated by two professional LEGO designers with an open label set. What is the inter-annotator agreement on the benchmark?
  2. Could you train LEGO-SAM-Flash using only the 300 human-annotated benchmark models(or a split thereof) and report the resulting performance? If LEGO-SAM-Flash trained on human labels is much worse, you should analysis it.
1 comment

Dear Reviewer,

We appreciate your constructive review and thank you for recognizing that our work "systematically studies semantic part segmentation for 3D LEGO models" and our Chain of Disassembly as "well tailored to the discrete and heavily occluded nature of LEGO assemblies." Following your feedback, we have conducted additional experiments and clarified the benchmark annotation procedure in the revised paper. We address your concerns as follows.

Q: Stronger Baselines

A: Thank you for this suggestion. We additionally include LEGO-specific adaptation to produce a new baseline by fine-tuning P3-SAM (see the details below). Importantly, the semantic part segmentation of LEGO assemblies is a newly introduced task. No existing methods have been designed specifically for this setting. Hence, we fine-tune P3-SAM on the same 4,000 LEGO models. These models are pseudo-labeled using our LEGO-SAM pipeline.

In detail, the resulting LEGO-adapted P3-SAM improves upon the original P3-SAM, increasing axis-aligned bounding-box IoU (aabbIoU) from 32.24 to 46.11. This result demonstrates the effectiveness of our pseudo-labeled dataset, which helps to improve various generic 3D segmentation models to handle LEGO assemblies. Yet, our LEGO-SAM-Flash achieves a higher aabbIoU of 66.69, compared with 46.11 for the LEGO-adapted P3-SAM, demonstrating the strengths of our proposed techniques, including residual adapters and atom-level segmentation head. We have revised Table 1 in the main paper and provided details in Appendix F.

Q: Annotation Consistency in Benchmark

A: Thank you for raising this concern. We agree that consistent definitions of semantic parts are essential for benchmark annotations. We had the following strategies to uphold annotation consistency. First, we predefined a part vocabulary for each LEGO model category for the LEGO designers to follow when they annotated the LEGO models. For example, for indoor scenes, the vocabulary includes windows, doors, walls, furniture, etc., and for cars, the vocabulary includes wheels, lights, fenders, windshields, etc. Each part type is accompanied by a name, a description text, and some example LEGO models for reference. Second, we avoided hierarchical part annotations in the vocabulary for granularity consistency. For instance, since a table is defined as a part of the whole scene (model), its leg is not treated as a separate part. Third, a designer can propose a new part type when encountering one that is not covered by the existing vocabulary. We then add the proposed part type, together with its description and example LEGO models, to the vocabulary, such that the other designer can subsequently follow. Furthermore, after the initial annotation, the designers cross-reviewed each other's results to ensure consistency with the predefined part definitions. In addition, we, the authors, also reviewed all the segmentation results for consistency. We now added these details into the revised paper, Section 6.1.

Q: LEGO-SAM-Flash trained on human-labeled dataset

A: Thank you. Following your suggestion, we compared two models with the same LEGO-SAM-Flash architecture but different training datasets: (1) the original Flash model trained on our 4,000 pseudo-labeled models and (2) a variant trained on 210 human-labeled models (70% of our benchmark, with 70 models from each category group). Both models are evaluated on the remaining 90 benchmark models. The results are summarized below.

Training DataPerfect LabelSizeaabbIoU ↑aabbF1 ↑asIoU ↑asF1 ↑
Human-label training✓21054.0237.5052.7938.14
Pseudo-label training✗4,00064.0941.2260.8645.96

The model trained on the 4,000 pseudo-labeled models generalizes better, highlighting the importance of our pseudo-label curation pipeline when accurate human labels are limited. We now added this comparison and corresponding analysis in Appendix G.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes

Dear Chairs and Reviewers,

We thank the reviewers for their constructive feedback. We have revised the manuscript accordingly, with changes highlighted in blue:

  1. Added atom-level evaluation. Added atom-set IoU and corresponding precision, recall, and F1 scores, together with separate evaluations on initially visible and invisible atoms (Sec. 6.3; Appendix E–F).

  2. Added baseline with LEGO adaptation. Added P3-SAM fine-tuned on the same 4,000 pseudo-labeled models. Clarified baseline inputs, supervision, inference configurations, and output conversion (Sec. 6.3; Appendix F).

  3. Added ablation studies of LEGO-SAM-Flash. Separately evaluated the residual adapters and atom-level segmentation head in LEGO-SAM-Flash (Sec. 6.3, Table 2).

  4. Added training-data analyses. Compared LEGO-SAM-Flash trained on 210 human-labeled models versus 4,000 pseudo-labeled models using the same remaining evaluation set. (Appendix G.3).

  5. Clarified implementation details. Added step-by-step descriptions and pseudocode for structural segmentation and Flash inference (Sec. 5; Appendix C).

  6. Clarified benchmark preparation procedures. Clarified the annotation procedures for ensuring consistent part definitions and granularity, and added dataset statistics (Sec. 6.1; Appendix B).

  7. Clarified baseline implementations. Detailed baseline inputs, supervision, inference configurations, and output conversion (Appendix D).

  8. Discussed future work. Discussed potential transfer to other assembly domains, improved distillation for complex assemblies, and integration with LEGO generation and editing (Sec. 7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.