acceptodds
Under review as a conference paper at ICLR 2027

PrimLLM: Learning to Assemble 3D Geometric Primitives from Text via Annotation-Free Reinforcement Learning

Abstract

Geometric primitive assembly provides a compact and structurally editable 3D representation by composing simple parametric shapes. We study text-conditioned primitive assembly: generating executable primitive programs from text without reference geometry. This task requires jointly reasoning about an object's semantic components and their geometric configurations. Existing approaches often rely on reference geometry or paired supervision over predefined component vocabularies, leaving open-ended text-conditioned assembly underexplored. We propose PrimLLM, a two-stage post-training framework that adapts a 9B open-weight language model to assemble general-purpose primitives with limited annotation. At both training and inference, the model is conditioned on dense captions produced by component planning, specifying component geometry and spatial relations. Structure-Aware Supervised Fine-Tuning (SFT) learns program syntax and mappings from component-level semantics to geometric configurations using dense caption-program pairs. Annotation-Free Reinforcement Learning (RL) then improves assembly quality by combining rule-based rewards for program format validity and physical plausibility with vision-language model feedback on multi-view renderings, without additional program annotations. The generation policy remains text-only, with visual feedback used exclusively to compute RL rewards. In our evaluation, PrimLLM matches or surpasses frontier proprietary models in semantic consistency and structural plausibility, and outperforms all evaluated open-weight baselines, including models with up to two orders of magnitude more parameters. Starting from the SFT model, RL improves assembly quality on both seen and unseen categories using only a small set of text descriptions per category. Code and data will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Primitive Assembly, Large Language Model, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_eoPeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

PrimLLM generates editable assemblies of cubes, cylinders, and spheres as executable HTML/Three.js programs. An external language model expands an object description into a component-level caption. A 9B generator is supervised on existing primitive assemblies and then optimized with GRPO using format, geometric plausibility, and multi-view visual rewards. In experiment, the reported gains over the base model are substantial, particularly in executable-output rate and structure scores. The contribution is promising, but stronger controls are needed to isolate the value of RL from planning, templates, and candidate search, and to establish stable structural improvements over strong baselines.

Critical strengths
  1. The representation supports execution and editing. Generating primitive programs provides a direct route to rendering, parameter modification, and outcome-based feedback. Using SFT to establish valid outputs before RL is well motivated.
  2. The improvement over the base generator is obvious.
Critical weaknesses
  1. The comparison does not fully separate post-training gains from the effects of external planning, program scaffolding, and inference configuration.
  2. The evidence for superiority over the strongest baseline is weaker than the evidence for improvement over the base model.
  3. The supported adaptation claim is narrower than unrestricted annotation-free learning or zero-shot category generalization.
Questions for authors
  1. Can the authors provide a per-baseline protocol covering original versus planned descriptions, fixed prefix, output length, candidate count, reasoning settings, and primitive filtering?
  2. How does RL compare with SFT+planning and reward-based best-of-N under a matched online generation or evaluation budget?
  3. What is the quality–cost trade-off between RL and reward-guided search over the same Stage-1 model? It is preferred to report generation, rendering, and scoring costs separately.
  4. If the intended claim extends beyond label-free category adaptation, can the authors evaluate categories or component compositions absent from both SFT and RL?
1 comment

Thank you for the insightful comments. Below, we respond to the questions and provide more results. All of them will be included in the revised paper.

Q1: Provide a per-baseline protocol.

A1: We provide the testing code for each baseline in the supplementary material. All baselines receive the same original user description as input and use the same fixed program prefix, generating code only after this prefix. Each description yields one candidate. The baselines use either API-based inference or local deployment, with configurations summarized in the table. To ensure a fair comparison, all models are prompted to use only cubes, spheres, and cylinders. Unsupported primitives are removed through a common post-processing step before rendering and evaluation.

ModeModelsMax TokensTemperature
API callsGPT-5.5, Claude Sonnet 5, Gemini 3.8 Flash, GLM-5.2, DeepSeek-V4-Pro16,3840.2
Local deploymentQwen3.5-9B, Qwen3.5-27B32,768N/A (greedy decoding)

Q2:How does RL compare with SFT+planning and reward-based Best-of-N?

A2: To clarify the contributions of RL over SFT+planning(Stage-1) and reward-based Best-of-N(BoN) selection, we evaluate four settings: Stage-1 with a single output(N=1), Stage-1 with BoN(N=8), Stage-1+RL with a single output(N=1), and Stage-1+RL with BoN(N=8). BoN scores candidates using the same reward models and weights as RL and selects the highest-reward candidate. The results are summarized below:

1. Stage-1 vs. Stage-1+RL: Introducing RL improves performance beyond SFT+planning. This indicates that reward-based RL helps the model explore better primitive compositions and spatial configurations, improving structural plausibility and semantic consistency. The ablation results in Table 2 of the main paper also support this benefit.

2. Stage-1+BoN vs. Stage-1+RL: From the same Stage-1 checkpoint, RL outperforms inference-time reward-based selection. This indicates that the performance gains stem not only from reward-based selection but also from incorporating reward feedback into model parameter updates, improving the model's single-sample generation capability.

3. Stage-1+BoN vs. Stage-1+RL+BoN: With the same candidate count and reward-scoring procedure, Stage-1+RL+BoN still performs better. This suggests that RL provides additional benefits under matched online candidate-generation and reward-evaluation budgets.

MethodNHTML Valid(%) ↑VQA Score ↑Structure Score ↑
Stage-1197.000.74683.790
+ BoN898.000.78433.877
+ RL1100.000.79514.109
+ RL + BoN8100.000.81954.187

Q3: What is the quality–cost trade-off between RL and reward-guided search?

A3: We further compare the inference costs, reporting generation time, generated tokens, rendering and scoring time per prompt. As shown in the table and supported by the results in Q2, BoN improves generation quality at higher inference cost without updating the generation policy. In contrast, RL learns a better policy through a one-time additional offline training stage, yielding higher quality than BoN at lower inference cost. Furthermore, RL supports adaptation to categories unseen during SFT using a small number of training samples and steps, without requiring additional annotations.

MethodNGeneration Time (s/prompt)Generated Tokens (tokens/prompt)Rendering Time (s/prompt)Scoring Time (s/prompt)
RL1200-4002,400-3,50010-200
BoN82,200-3,60020,000-24,00040-705-10

Q4: Evaluate the categories that are absent from both SFT and RL.

A4: We understand the reviewer’s concern about zero-shot category generalization. Our contribution is adaptation to SFT-unseen categories through annotation-free RL. We also explore generalization to categories unseen during both SFT and RL. Specifically, we select four such categories, yielding 40 test samples, and compare the baseline, Stage-1 SFT, and Stage-1 SFT+Stage-2 RL models.

As shown in the table, Stage-1 SFT improves HTML Valid and Structure Score, indicating that it helps the model acquire transferable text-to-primitive assembly capabilities. However, as SFT relies on imitation learning, its generalization remains constrained by limited training data. Stage-2 RL further improves all metrics and surpasses the baseline, suggesting that reward-driven RL can complement limited supervision and improve primitive assembly policies. These results also provide preliminary evidence that RL may promote generalization to categories unseen during both SFT and RL.

MethodHTML Valid(%) ↑VQA Score ↑Structure Score ↑
Baseline87.500.70303.043
+Stage-197.500.68123.253
+Stage-1+Stage-2100.000.76833.478
Reviewer_jSN1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PrimLLM combines component planning, SFT of a 9B language model, and GRPO to generate executable 3D primitive programs. Experiments examine generation quality and adaptation to categories unseen during SFT but included in RL.

Critical strengths
  1. The task is useful, and primitive programs provide an explicit, editable representation.

  2. Ablations support the training stages. The model reports 100% program validity, and the visual reward has some human validation.

Critical weaknesses
  1. Reproducibility and comparison fairness are insufficiently supported. No accessible code repository is provided, and key reward configurations and baseline prompting, decoding, and retry/postprocessing settings are missing. These omissions prevent independent verification of the implementation and equivalent treatment across methods. The VQA margin over Gemini 3.8 Flash is only 0.0007, without uncertainty estimates or human validation of the final ranking.
  2. The contribution of the reward-based training design is not established. The pipeline combines established planning, SFT, GRPO, and rewards, but lacks a matched-budget SFT plus reward-based selection control. Its physical reward also has a concrete limitation: two non-overlapping primitives touching far above ground avoid the floating penalty in Equation (6) and can receive the maximum physical reward under Equation (7). Whether visual feedback reliably compensates is not evaluated, leaving the effectiveness of this reward design insufficiently validated.
Questions for authors
  1. [Clarification] Can you provide anonymized code, complete reward settings, and the exact baseline protocol, including prompts, fixed prefixes, generation budgets, and retry/postprocessing rules?
  2. [Additional validation] Can paired confidence intervals and a blinded human comparison confirm the advantage over Gemini 3.8 Flash?
  3. [Additional validation] Can a matched-budget SFT plus reward-based selection control and targeted unsupported-cluster tests establish the training benefit and the reliability of visual compensation?
1 comment

Thank you for the valuable comments. Below we respond to the questions. All of them will be incorporated in the revised paper.

Q1: Provide anonymized code, reward settings, and the baseline protocol.

A1: We provide anonymized code in the supplementary material, including reward settings, prompts, post-processing, rendering, and baseline inference code. Model weights and datasets will also be publicly released upon acceptance to facilitate reproducibility.

Q2: Report paired confidence intervals and human evaluation.

A2: Our research aims to enhance open-source models’ capabilities in primitive assembly and explore their potential to compete with frontier proprietary models. To provide a more comprehensive evaluation of PrimLLM against Gemini 3.8 Flash, we conduct additional paired confidence interval analysis and human evaluation.

1. Paired confidence intervals: We compute the per-sample metric differences between the two methods on the same test descriptions and estimate the 95% confidence intervals (CI) of the mean differences, as shown in the table. PrimLLM achieves higher mean scores on all three metrics. Although all intervals include zero, these results demonstrate PrimLLM’s competitive performance against a strong closed-source model.

MetricGemini 3.8 FlashPrimLLM (Ours)CI
CLIP Score ↑27.4227.45+0.03 [-0.005, +0.054]
VQA Score ↑0.79440.7951+0.0007 [−0.0091, +0.0105]
Structure Score ↑4.0214.109+0.088 [-0.010, +0.186]

2. Human evaluation: We randomly select generation results from 12 categories, covering both in-domain and out-of-domain categories, and invite 10 participants to conduct two evaluations: (1) Dimension-wise scoring: Participants rate the generated results on a five-point scale for semantic consistency and structural plausibility. (2) Pairwise preference: Results from PrimLLM and Gemini 3.8 Flash for the same text description are presented in pairs, and participants choose the result they prefer overall. As shown below, PrimLLM achieves higher scores on both dimensions and receives a higher preference rate in pairwise comparisons. This further supports PrimLLM’s generation quality and competitiveness.

MethodSemantic Score ↑Structure Score ↑Win Rate vs.Ours(%)
Gemini 3.8 Flash4.0084.02540.83
PrimLLM(Ours)4.3334.225—

Q3: Advantages of RL training over the matched-budget SFT plus reward-based selection control.

A3: To clarify the benefits of RL training over reward-based best-of-n (BoN) selection, we evaluate three configurations starting from the same Stage-1 SFT checkpoint: Stage-1 with BoN (N=8), Stage-1+RL with a single output (N=1), and Stage-1+RL with BoN (N=8). BoN scores candidates using the same reward models and weights as RL and selects the highest-reward candidate as the final output. The results are summarized below:

1. Stage-1+BoN vs. Stage-1+RL: RL outperforms inference-time reward-based selection. This indicates that the performance gains stem not only from reward-based selection but also from incorporating reward feedback into model parameter updates, enabling progressive improvements in generation policy.

2. Stage-1+BoN vs. Stage-1+RL+BoN: With the same candidate count and reward-scoring procedure, Stage-1+RL+BoN still performs better. This further supports the benefits of RL under matched online candidate-generation and reward-evaluation budgets.

MethodNHTML Valid(%) ↑VQA Score ↑Structure Score ↑
Stage-1197.000.74683.790
+ BoN898.000.78433.877
+ RL1100.000.79514.109
+ RL + BoN8100.000.81954.187

Q4: Can visual feedback reliably compensate for unsupported clusters?

A4: We thank the reviewer for this insightful question. Visual feedback and physical reward can play complementary roles in mitigating unsupported-clusters. First, the physical reward penalizes individual unsupported primitives, encouraging more plausible structures and thereby indirectly reducing unsupported-clusters. Second, visual feedback can identify unsupported-clusters. Such clusters often exhibit noticeable visual anomalies, providing clear evidence for VLM assessment. To validate this observation, we constructed 30 sample pairs, each with a normal output and one with an unsupported cluster, and scored both using the VLM reward. The normal output scores higher in 80% of pairs, supporting the effectiveness of visual feedback as a complementary signal. The two rewards combine geometric constraints with holistic visual assessment for more plausible outputs.

RewardNormal score (mean ± SD) ↑Unsupported-cluster score (mean ± SD)Correct preference rate ↑
VLM Reward0.873 ± 0.2030.307 ± 0.42680%
Reviewer_eXuUOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose PrimLLM, a post-training framework for LLMs capable of generating primitive abstractions of real-world 3D shapes taking as input a text prompt and a set of primitives. The authors use HTML/Three.js programs as a medium to generate the 3D shapes, perform SFT on captions-programs paired in a first stage and GRPO using structural and program validity rewards in a second stage. The authors decent qualitative results, and outperform all open-weight models they compare to. They also show competitive results against proprietary models - although performance against frontier models (e.g. GPT-6 Astra, Opus 5 or Opus 5.5, Gemini Pro variants) is not reported anywhere in the paper.

Critical strengths

The task the authors are focusing on is important and will become increasingly key in future shape generation paradigms as frontier LLMs keep getting stronger at 3D structure understanding. Paradigms involving the combination of coarse generated structures with LLMs and diffusion models for geometry refinement are going to be a strong alternative to holistic methods based on voxels are implicit fields. The fact that this method outperforms all open-weight methods and is competitive against some proprietary LLMs is encouraging for open-weight research in this direction. The approach is sound and meaningful, and somewhat novel (although structural rewards are not completely novel, I do believe applying them in a GRPO setup to finetune an LLM is quite novel).

Critical weaknesses

This paper has some minor weaknesses.

  • Human evaluation would make the paper much stronger and convincing, although GPT evaluation should be good enough for this task. The authors do use human evaluators to verify the relevance of their visual quality reward, but a general human evaluation of the end-to-end task would also be a plus.
  • A quantitative metric measuring generated sample quality would also be valuable. The authors only measure semantic alignment, but FID based on rendered views of the samples would be an easy way to strengthen the claims - especially considering they already have ground truth primitive arrangements for each of their dense captions.
  • The authors show competitive results against proprietary models - but never compare to frontier models (e.g. GPT-6 Astra, Opus 5 or Opus 5.5, Gemini Pro variants). I don't think that it would be fair to the authors to ask for competitive results against these frontier models - especially considering the small parameter size of their specialized LLM. Nonetheless, showing that a gap still exists would still be valuable for future research on this direction.
Questions for authors
  • Have you considered using image conditioning as an alternative input to your dense captions? The image would be a rendering of a realistic 3D asset, or even a real world 3D object. Even if you don't have pairings with programs and realistic 3D renders (by realistic, I mean not primitive abstractions) on real world images, it may be that your model can already achieve this using only the structural rewards combined with a visual similarity reward with the input object. Since you are finetuning Qwen3.5-9B, the model already has native visual-language alignment. Datasets like 3DCoMPaT++[1] or Objaverse [2] already have pairings with renders of 3D shapes and 3D geometry that you may use there. You may have too little time to run this experiment but the alternative mode of control could be really quite valuable to other fields (e.g. people doing scene understanding, scene reconstruction, etc.).
  • Can you add comparison to stronger frontier models? (GPT-6 Astra, Opus 5 or Opus 5.5, Gemini Pro). We don't expect this method to outperform these models but the comparison would still be interesting.

[1] Habib Slim, Xiang Li, Yuchen Li, Mahmoud Ahmed, Mohamed Ayman, Ujjwal Upadhyay, Ahmed Abdelreheem, Arpit Prajapati, Suhail Pothigara, Peter Wonka, Mohamed Elhoseiny. 3DCoMPaT++: An Improved Large-Scale 3D Vision Dataset for Compositional Recognition. TPAMI, 2025.

[2] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, Ali Farhadi. Objaverse: A Universe of Annotated 3D Objects. CVPR, 2023.

1 comment

Thank you for the valuable feedback. Below, we respond to the questions. All of them will be incorporated in the revised paper.

Q1: Adding human evaluation.

A1: To further validate the reliability of our evaluation, we conducted a human evaluation comparing PrimLLM with three proprietary models: GPT-5.5, Claude Sonnet 5, and Gemini 3.8 Flash. We randomly sampled outputs from 12 categories, including in-domain and out-of-domain categories, and invited 10 participants to complete two tasks: (1) Dimension-wise scoring: Each participant rated outputs from all four models on a five-point scale for semantic consistency and structural plausibility. (2) Pairwise preference: Each participant compared PrimLLM’s outputs with those of each model and selected the output they preferred overall.

As shown in the table, PrimLLM achieves the highest scores on both dimensions. All three baselines have win rates below 50% against PrimLLM, indicating an overall preference for PrimLLM’s outputs. These findings are consistent with the results reported in Table 1 of the main paper.

MethodSemantic Score ↑Structure Score ↑Win Rate vs.Ours(%)
GPT-5.53.8583.62530.83
Claude Sonnet 53.8923.66734.17
Gemini 3.8 Flash4.0084.02540.83
PrimLLM(Ours)4.3334.225—

Q2: Adding FID as an quantitative metric.

A2: We additionally report FID alongside existing metrics on the in-domain test set to assess generation quality of SFT and RL. FID measures distributional differences between generated and real samples but does not directly assess semantic alignment or structural plausibility, so we use it as a complementary measure.

As shown in the table, SFT reduces FID, indicating that paired supervision helps the model learn the distribution of real primitive assemblies. RL maintains a comparable FID while further improving program validity, semantic consistency, and structural plausibility. This suggests that RL further optimizes primitive assemblies based on the structural priors learned during SFT, thereby improving generation quality.

MethodHTML Valid(%) ↑VQA Score↑Structure Score ↑FID ↓
baseline72.000.73823.100114.05
baseline+Stage1 SFT99.000.79854.02150.37
baseline+Stage1 SFT+Stage2 RL100.000.82484.29052.97

Q3: Comparison to stronger frontier models.

A3: To evaluate PrimLLM against stronger frontier models, we conduct comparisons with GPT-6 Astra, Claude Opus 5.5, and Gemini 3.1 Pro on the same test set as in the main paper. As shown in the table, PrimLLM achieves the highest HTML Valid, demonstrating its strength in generating executable primitive programs that conform to the predefined format. It also slightly outperforms Gemini 3.1 Pro in CLIP Score and Structure Score, indicating competitive performance in semantic alignment and structural plausibility. Although some frontier models lead on several metrics, PrimLLM enables a 9B open-source model to achieve competitive performance on primitive assembly through structure-aware SFT and annotation-free RL. This highlights the framework’s potential to enhance the structured generation capabilities of open-source models.

MethodHTML Valid(%) ↑CLIP Score ↑VQA Score ↑Structure Score ↑
GPT-6 Astra99.0027.830.81124.211
Claude Opus 5.598.0027.420.80424.290
Gemini 3.1 pro99.5027.310.80544.072
PrimLLM(Ours)100.0027.450.79514.109

Q4: Have you considered using image conditioning?

A4: We thank the reviewer for this valuable suggestion. We have considered using realistic object images as input conditions, but two main challenges remain:

1. Lack of paired data for effective initialization. Datasets such as 3DCoMPaT++ and Objaverse provide 3D geometries and rendered images but lack corresponding primitive structures and parameterized programs, limiting their direct use for image-to-program SFT. Without effective SFT initialization, the initial policy may struggle to generate syntactically valid programs, making exploration and optimization during RL more difficult.

2. Challenges in reward design. A simple visual similarity reward may not reliably reflect primitive assembly quality, as input images contain textures, lighting, and backgrounds, whereas primitive representations capture abstract object structure. Therefore, a reliable reward model should assess structural consistency between input images and rendered primitive assemblies, while accounting for reasonable geometric simplifications and occlusion ambiguities. Designing such a reward model requires further investigation.

In future work, we will explore image-conditioned primitive assembly and corresponding reward designs to broaden the applicability of our framework.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.