BrickFlow: Connectivity-Guided Brick Reconstruction

Given a set of LEGO parts and an image of an assembled object, our task is to infer the pose of each part to assemble the object. In contrast with traditional 3D reconstruction, doing so requires satisfying discrete physical constraints, which provide structure but render the problem combinatorial and highly sensitive to small pose errors. To learn a prior over how parts fit together, we train a set-conditioned SE(3) flow model that maps noisy poses to valid assemblies. We then finetune this model to inject image conditioning. However, while the learned flow captures global structure, it does not enforce connectivity. To address this, we introduce an analytic connector-field guidance to pull predictions toward the connection-consistent manifold at test time. Through this, we demonstrate improved validity and reconstruction performance, scaling with model size, pretraining, and conditional guidance. We will make our models and data available for research purposes.