PrimLLM: Learning to Assemble 3D Geometric Primitives from Text via Annotation-Free Reinforcement Learning

Geometric primitive assembly provides a compact and structurally editable 3D representation by composing simple parametric shapes. We study text-conditioned primitive assembly: generating executable primitive programs from text without reference geometry. This task requires jointly reasoning about an object's semantic components and their geometric configurations. Existing approaches often rely on reference geometry or paired supervision over predefined component vocabularies, leaving open-ended text-conditioned assembly underexplored. We propose PrimLLM, a two-stage post-training framework that adapts a 9B open-weight language model to assemble general-purpose primitives with limited annotation. At both training and inference, the model is conditioned on dense captions produced by component planning, specifying component geometry and spatial relations. Structure-Aware Supervised Fine-Tuning (SFT) learns program syntax and mappings from component-level semantics to geometric configurations using dense caption-program pairs. Annotation-Free Reinforcement Learning (RL) then improves assembly quality by combining rule-based rewards for program format validity and physical plausibility with vision-language model feedback on multi-view renderings, without additional program annotations. The generation policy remains text-only, with visual feedback used exclusively to compute RL rewards. In our evaluation, PrimLLM matches or surpasses frontier proprietary models in semantic consistency and structural plausibility, and outperforms all evaluated open-weight baselines, including models with up to two orders of magnitude more parameters. Starting from the SFT model, RL improves assembly quality on both seen and unseen categories using only a small set of text descriptions per category. Code and data will be released.