acceptodds
Under review as a conference paper at ICLR 2027

PartSynth: Generative Sparse Part Structures for Compositional 3D Generation with PBR Materials

10 views

Abstract

High-fidelity compositional 3D assets require separable parts, detailed geometry, and physically based rendering materials. Existing methods either constrain part counts or incur inference costs that grow with the number of parts. Monolithic material generators also lack explicit part awareness. We introduce PartSynth, an image-conditioned framework for compositional 3D geometry and native part-aware PBR generation. Its core representation, the Sparse Part Structure (SPS), uses reusable set-indexed latent slots to represent multi-part structure, jointly generating each slot's activity and feature. Multiple active slots at the same spatial location represent overlapping components, while reusing set indices across disconnected components supports flexible part counts. Connected components of the decoded set-indexed volumes define individual parts. The SPS-guided geometry and PBR stages adapt pretrained monolithic generators to produce detailed multi-part geometry with PBR materials. Experiments show improved part-level geometry and PBR material quality over part-generation baselines, while maintaining competitive whole-shape fidelity.

Primary area:
generative models
Keywords:
Compositional 3D generation, PBR generation, Sparse Part Structures

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_F8MLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PartSynth, a four-stage image-to-3D framework built upon TRELLIS.2 that jointly generates compositional multi-part geometry and part-aware PBR materials. Its core representation, Sparse Part Structure (SPS), decomposes a coarse 64364^3 whole-shape scaffold into up to K=8K=8 set-indexed volumes via graph coloring on the part-contact graph, compressed onto a shared 16316^3 sparse latent support Ω\Omega.

Critical strengths
  1. This work makes a contribution to pushing native 3D compositional generation forward. By instantiating K=8K=8 set-indexed slots only on the active 16316^3 sparse scaffold support (LSPS=KSL_{\mathrm{SPS}} = KS with S≪163S \ll 16^3), it provides a clean and scalable paradigm for expanding the capacity of PartCrafter/PartPacker-style generators without suffering from dense volumetric memory explosion. This formulation handles complex non-bipartite contact graphs and overlapping parts in a single fast forward pass (~10s). Although the current implementation has limitations in per-part resolution and controllability, the underlying SPS representation offers valuable insights for the 3D generation community and justifies acceptance.

  2. Extending pretrained monolithic geometry and PBR flow models with 4D rotary positional encoding (k,x,y,z)(k, x, y, z) enables global information exchange across all parts while keeping set-level surface decoding separated. As verified by the PartSynth-PerSet ablation and the proposed PI-ME metric, this design effectively reduces cross-part material bleeding while maintaining whole-shape coherence.

Critical weaknesses
  1. The geometric quality of individual parts in the supplementary material and normalized part galleries (Fig. 3) is noticeably coarse and low-resolution. The pipeline stops at the 323 slat→5123 mesh32^3 \text{ slat} \to 512^3 \text{ mesh} stage and omits the second-stage high-resolution refinement of TRELLIS.2 (643 voxel→643 slat→10243 mesh64^3 \text{ voxel} \to 64^3 \text{ slat} \to 1024^3 \text{ mesh}), likely due to the sequence-length overhead of concatenating up to K=8K=8 sets in joint self-attention. Because all parts must share a single global 5123512^3 grid rather than being synthesized in per-part full-resolution grids (as in FullPart or OmniPart), small components receive very few active voxels. Moreover, part topology is permanently locked on the 64364^3 sub-scaffolds (after a 643→163→64364^3 \to 16^3 \to 64^3 pass through VAEsparse\mathrm{VAE}_{\mathrm{sparse}}), making the pipeline highly sensitive to 64364^3 quantization errors that cause thin parts to fragment or nearby parts in the same set to merge irreversibly.

  2. Following an unguided end-to-end paradigm like PartCrafter rather than a 2D-controllable framework like OmniPart leaves the part decomposition process as an uncontrollable black box. Conditioned only on a single RGB image, the model provides no mechanism for users to specify part boundaries, guide where splits should occur, or even control the decomposition granularity (e.g., whether to split an object into a few coarse assemblies or dozens of fine sub-components).

Questions for authors
  1. Can the 4D (k,x,y,z)(k, x, y, z) joint-attention architecture be extended to TRELLIS.2's second-stage cascade refinement (643 voxel→643 slat→10243 mesh64^3 \text{ voxel} \to 64^3 \text{ slat} \to 1024^3 \text{ mesh}) to improve per-part geometric sharpness, or does the KK-fold token expansion make 64364^3 structured latents computationally prohibitive?

  2. Since SPS currently operates purely from a single image without structural guidance, how could user-specified 2D part masks (as in OmniPart) or granularity controls be injected into the (k,x,y,z)(k, x, y, z) slot generation process so that users can control how parts are partitioned?

Reviewer_H5p4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The goal of this paper is to generate high-quality 3D shapes with separate parts, each having its own geometry and PBR materials. To achieve this, they first generate a coarse whole-shape scaffold using TRELLIS.2. Then, they generate a Sparse Part Structure (SPS) using a flow model that predicts the activity and features of each slot. To construct the SPS training representation, they first build a part-contact graph, where nodes represent parts and edges connect parts that co-occur in a voxel or occupy face-adjacent voxels. A graph-coloring algorithm then assigns contacting parts to different sets within a fixed budget , while disconnected parts can share the same set. These sets are encoded into the SPS representation, where each slot contains a set index, spatial location, activity indicator, and latent features. After the flow model generates the SPS, set-wise decoding reconstructs the sparse volumes, and connected components are extracted as individual parts. Finally, detailed geometry and part-aware PBR materials are generated using adapted TRELLIS.2 models. The main contributions are the Sparse Part Structure, which supports flexible part counts and overlapping parts without sequential generation, and part-aware geometry and PBR generation, which improves part quality and reduces material bleeding between parts.

Critical strengths

The strengths are:

  1. The paper is well written and easy to understand, with clear figures that effectively explain the proposed method. The qualitative results are also presented clearly.
  2. The paper introduces a novel representation for compositional 3D generation.
  3. The paper demonstrates strong experimental results, outperforming other part-generation baselines on part-level geometry and material quality.
Critical weaknesses

The weakness are:

  1. One of the paper's claims is inference efficiency, but there is no direct end-to-end inference-time comparison between the proposed method, TRELLIS.2, and other baselines.
  2. The paper does not clearly discuss the additional training cost required to adapt TRELLIS.2 to the proposed framework.
  3. The paper does not directly evaluate the correctness of the generated part structure, such as part connectivity, contact relationships, or part counts.
  4. One limitation of the SPS representation is that not all part-contact configurations can be represented with a fixed number of sets (K=8). Increasing the number of sets also increases computational cost.
Questions for authors

My questions are:

  1. Can the authors provide an end-to-end inference-time comparison between the proposed method and the baselines using the same hardware?
  2. Can the authors discuss the potential overlap between the training and evaluation datasets? Were any steps taken to identify and remove overlapping assets?
  3. Could the authors provide an analysis of how increasing K affects generation quality, inference time, and memory usage?
2 comments

Thanks for the detailed rebuttal. My concerns about inference time and training cost are resolved. One small request: please reconcile the rebuttal timings with the per-stage times in Section 4.1.

Remaining concerns:

  • Fragmentation: PartSynth's connectivity (1.83 vs. 1.20 for GT) is worse than OmniPart's (1.52), which points to a part-fragmentation issue.
  • Choice of K: No models were trained with other K values. A one-line explanation of why (e.g., compute cost) would help.
  • Dataset overlap: 109 of the 200 test assets appear in the training data, but the revised PDF doesn't mention this, and the Table 1 numbers include those assets. Please disclose the overlap and report the seen/unseen split in the final version.

I am still borderline.

Thank you for your follow-up comments and suggestions.

We have now incorporated the relevant rebuttal results, including the dataset overlap analysis, into the revised paper. We had not realized that these revisions were expected to be incorporated into the manuscript during the rebuttal period.

1. Fragmentation

We would like to clarify that OmniPart is trained on human-annotated part decompositions, which are closer in granularity to the manually annotated ground truth in PartObjaverse-Tiny. In contrast, our method is trained on mesh components without semantic part annotations and tends to produce finer-grained decompositions.

As discussed in our response to Reviewer H9mr, evaluating part-generation quality against a particular ground-truth decomposition also requires care.

Our evaluation uses GT geometry and part decomposition as references. However, differences from a particular reference do not necessarily imply that additional or missing parts are generation errors. Unlike reconstruction from 3D inputs, compositional 3D generation from a single image is inherently ambiguous: the image does not uniquely determine the underlying geometry or part decomposition.

For example, one designer may model a sphere as a single shell, while another may use two nested shells with different materials. Both can represent the same visible object. An additional shell in the generated model should therefore not automatically be considered an error solely because it does not correspond to a GT part.

We acknowledge that our global connectivity score is worse than OmniPart's. However, this assembly-level metric should not be interpreted as a direct measure of fragmentation within individual parts.

2. Choice of K

A larger K allows the SPS representation to handle more complex part-contact configurations, increasing its representational capacity and dataset coverage. Simpler contact configurations can be represented with fewer sets.

Importantly, K=8 specifies the maximum number of available sets, not a fixed number of active sets for every generated asset. The model can use fewer sets depending on the input and the stochastic sampling process. Fig. 1 and Fig. 10 show examples with k=7 and k=3, respectively.

Using the same Seen (109 assets) / Unseen (91 assets) split, we report the number of nonempty slots (active sets) in our generated outputs and in the GT representations produced by our coloring algorithm. Each percentage denotes the proportion of assets with exactly k nonempty slots, except for the k>8 column, which groups all counts above eight. The denominator is the sample count in each row. Percentages may not sum to 100% because of rounding.

Seen (109 assets)

DataSamplesMean slot countk=2k=3k=4k=5k=6k=7k=8k>8
Ours1094.41312.8%15.6%28.4%16.5%18.3%3.7%4.6%0.0%
GT: all1094.5784.6%23.9%33.9%11.0%9.2%9.2%7.3%0.9%

Unseen (91 assets)

DataSamplesMean slot countk=2k=3k=4k=5k=6k=7k=8k>8
Ours914.7365.5%18.7%19.8%26.4%19.8%2.2%7.7%0.0%
GT: all915.4295.5%16.5%19.8%20.9%16.5%4.4%6.6%9.9%

Both generated outputs and GT predominantly use two to six slots. Generated assets use an average of 4.413 and 4.736 active slots on the Seen and Unseen splits, respectively, compared with 4.578 and 5.429 for GT. Only 4.6% and 7.7% of generated assets use all eight slots, confirming that the model does not simply fill the maximum budget.

We did not train separate models with different maximum K values because of the additional training cost. The statistics above characterize slot usage and coverage under our coloring procedure, rather than generation quality under different maximum budgets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.