Hybrid Reasoning through Compositional Structural Energy Minimization

Many real-world reasoning problems require solving for high-level discrete structure with continuous values. For example, a robot solving a manipulation problem must determine both a high-level task structure and continuous actions; a model for scene understanding infers both the objects' pairwise relations and their poses from a partial camera view. Existing machine learning approaches face important limitations: end-to-end monolithic models often struggle to solve problems that are harder than those seen during training, while compositional methods typically require the correct problem structure to be specified in advance. We introduce Compositional Structural Energy Minimization (CSEM), a general framework that learns from small training instances and jointly solves for the discrete structure and continuous solutions of harder problems. We evaluate CSEM on a wide range of hybrid reasoning problems, including task-and-motion-planning, 3D scene understanding, and reconstructing physical trajectories and contact modes from sparse observations. We demonstrate that CSEM generalizes without additional training to larger structures and longer horizons, outperforming strong learning-based baselines.