acceptodds
Under review as a conference paper at ICLR 2027

GrowFlow: Unified 3D Structure and Feature Generation via Expanding Structural Flow Matching

Abstract

Structured latent representations have been widely adopted for efficient 3D shape modeling and generation in recent years, due to their favorable trade-off between geometric fidelity and computational efficiency. However, most existing genera- tive approaches decompose the generation process into two stages: first predicting sparse voxel structure, and then generating latent features based on that struc- ture. Although effective, this staged design introduces extra parameters to predict the voxel structure. While the voxel structure does not require many parameters compared to the detailed geometry, because of the need for generalization, it is necessary to build a huge structure prediction network. To address this, we pro- pose GrowFlow, the first 3D generative framework to unify structure and feature generation across resolutions within a single flow-matching process. Specifically, we provide a principled mapping strategy that enables consistent flow matching across different resolutions and structural states, laying the foundation for multi- scale structural modeling. To avoid structure-feature decomposition, we integrate a dedicated structure generation module into the flow matching paradigm, which jointly models sparse voxel structure and fine-grained geometric features in an end-to-end manner. Using a Diffusion Transformer (DiT) as the backbone, our framework achieves generation quality comparable to the state-of-the-art while using fewer parameters than multi-stage pipelines and reducing the computation required. Extensive experiments on benchmark datasets show that GrowFlow in- fers 3D occupancy progressively from coarse to fine resolution, matching the ge- ometric fidelity of state-of-the-art methods while improving generation efficiency. Notably, our model enables resolution-adaptive, progressive generation, letting users preview low-resolution results, allowing early termination or full-generation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Generation, Object-Level Generation, Diffusion, Flow-Matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_utREOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes GrowFlow, a single-image 3D generator that produces sparse voxel structure and SLAT features jointly within one piecewise flow-matching process over an octree. Generation goes from coarse to fine. On Toys4k, the method matches or exceeds multi-stage SLAT methods on ULIP and Uni3D with about half the parameters of Trellis2 and faster inference, and it can stop at intermediate levels to give lower-resolution outputs.

Critical strengths

1 The paper addresses a real limitation of current SLAT-based generators: structure is fixed in a separate first stage and cannot be corrected later. Extending piecewise multi-resolution flow matching to an octree with a variable branching factor is a natural and non-trivial step. The proposed expansion operator is simple and handles the variable number of children per node without timestep rescheduling. 2 The method is practically appealing. With 1.53B parameters, it reaches quality comparable to Trellis2 (3.11B) while running about 1.7× faster, and a single model supports multi-resolution readout and early termination. The qualitative results on thin structures, open cavities and flat plates (Figs. 4, 5, 9, 10) are convincing and support the claim that revisiting occupancy at each level helps.

Critical weaknesses

1 The quantitative evaluation is limited. For example, only image–shape embedding scores (ULIP and Uni3D) are reported, from a single run with no seed averaging, and the authors themselves treat differences of a few thousandths as ties. 2 The method still uses a separate structure decoder, so “single-stage” really means interleaved structure prediction. It is somehow overclaimed. 3 Some related work is missing.Hierarchical sparse-voxel generation (XCube, which is cited but not discussed) and octree-based diffusion (e.g., OctFusion) should be discussed

Questions for authors

See weakness.

1 comment

Thank you for your constructive feedback. We address the three weaknesses below and will revise the final version accordingly, including ensuring consistent formatting throughout the paper.

  1. Critical Weakness 1 — Quantitative evaluation.

Our reported scores are averaged over approximately 4,000 samples. In addition, during structure generation, we exploit the low computational cost at coarse resolutions to generate multiple occupancy predictions in parallel and use voting to improve stability. We will also revise the discussion to clarify the interpretation of differences on the order of 0.001 between models.

Evaluating multiple random seeds requires more time than the two-week rebuttal period allows. These experiments are ongoing, and we aim to report the mean scores before the final decision.

Besides, to further assess geometric quality, we have added an evaluation based on CLIP scores averaged over multi-view normal renderings. The scores are:

GrowFlow: 0.923 Trellis.2: 0.915 Direct3D-S2: 0.920 Hunyuan3D: 0.915 UniLat: 0.911 Hi3DGen: 0.902

We will include these results in Table 1 in the final version.

  1. Critical Weakness 2 — The meaning of “single-stage.”

In generative pipelines, the DiT accounts for most of the inference cost, whereas decoding is comparatively fast. We therefore count stages according to the distinct DiT-based generative processes, each of which may include a decoder. For example, Trellis.2 uses a DiT and a decoder to generate the 3D structure in its first stage, followed by another DiT and decoder to generate the 3D model conditioned on that structure. Direct3D-S2 uses two DiT-and-decoder stages for structure generation, followed by a third for generating the 3D model.

GrowFlow uses a single shared DiT with VAE decoding to jointly generate structure and features; the structure VAE is a very small module. Our use of “single-stage” refers to this shared generative process. We will clarify this definition in the final version to make the role of the structure decoder explicit.

  1. Critical Weakness 3 — Missing related work.

Thank you for pointing out these omissions. In the final version, we will add a comparison with XCube to Table 1, including metrics: 1. ULIP:0.1475, 2. Uni3D: 0.2595 3.CLIP: 0.875 and discuss OctFusion in the related work section.

Reviewer_vB63OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GrowFlow jointly generates occupancy and geometric features on an expanding sparse octree for single-image 3D generation. It introduces feature rescaling and noise correction across resolutions to handle varying numbers of children per node. This allows one generative process to grow the structure and produce shapes at multiple resolutions.

Critical strengths
  1. The transition between resolutions addresses a concrete problem in sparse generation. Variable branching means that mean downsampling and directly copying parent features alter noise variance and correlations between children. Local rescaling and a correction to the child noise allow the resolution to change at the same flow time. This provides a principled way to expand sparse structure during generation.

  2. The results support a useful trade-off between generation quality and computation. In Table 1, the method achieves semantic agreement comparable to Trellis2 with fewer parameters and shorter inference time. The same trajectory also provides a coarse preview that can be refined further as needed. Together with the multi-resolution results in Table 2, this establishes practical benefits in efficiency and output flexibility.

Critical weaknesses

More reliable geometry from progressive generation is not yet convincingly established. Because the method updates occupancy during generation, preserving thin connections and keeping cavities open are important tests of the design. ULIP and Uni3D in Tables 1–2 mainly measure semantic agreement and are not sensitive enough to these errors. The examples show promise, but do not establish how consistently the structural advantage holds across the test set.

Questions for authors

On test shapes with thin connections or open cavities, does the method consistently reduce broken connections and incorrectly filled cavities compared with the baselines? Such a comparison would directly affect the assessment of progressive structure generation.

2 comments

Thank you for raising the concern about structural reliability, particularly for thin connections and open cavities. We provide further details on our generation procedure and additional evaluation results below. We will also revise the final version to ensure consistent formatting throughout the paper.

Regarding the preservation of thin connections and open cavities, current image-to-3D generation pipelines commonly use signed distance field (SDF) representations trained with watertight supervision. Under this representation, thin connections and the thin walls surrounding open cavities are represented by two closely spaced surfaces enclosing a watertight volume. Preserving these closely spaced surfaces is therefore important for faithfully reconstructing such structures. The first two rows of Figure 5, the last row of Figure 10, and several examples in Figure 11 illustrate our model's ability to generate these challenging structures. Below, we further explain the generation procedure and provide additional evaluation results.

  1. Structural stability during progressive generation

Our progressive generation procedure helps preserve connectivity through repeated occupancy refinement. At resolutions of 16 and 32, computation is relatively inexpensive (the cost of a single inference is approximately one-eighth of that at resolution 64). We therefore run inference in parallel from three different noise initializations, obtain three occupancy predictions, and use voting to improve prediction stability.

At resolution 64, we continue inference from the two samples that receive the most support in the voting process. We then take the union of their predicted occupancies as the starting structure for prediction at resolution 128. Finally, redundant tokens are pruned after the entire denoising process is complete.

This procedure improves the stability and completeness of structure prediction through the 16-to-32-to-64 transitions and the subsequent transition from 64 to 128. In contrast, applying a similar procedure to the VAE in the conventional two-stage Trellis.2 pipeline is difficult. Its second stage, at resolution 128, also does not perform pruning. Our progressive occupancy refinement therefore provides additional opportunities to stabilize the generated structure.

  1. Additional evaluation using multi-view normal renderings

To complement the semantic metrics, we additionally evaluate CLIP scores using multi-view normal renderings. For the methods compared in Table 1, the scores are:

GrowFlow: 0.923 Trellis.2: 0.915 Direct3D-S2: 0.920 Hunyuan3D: 0.915 UniLat: 0.911 Hi3DGen: 0.902

Normal renderings are particularly sensitive to instability in thin, sheet-like structures: geometric errors can produce pronounced rendering artifacts and substantially lower the score. These results provide additional evidence of the geometric quality of our outputs.

We also evaluate the resolution ablation in Table 2 using the same metric. At resolutions of 128, 64, 32, and 16, our CLIP scores are 0.920, 0.909, 0.887, and 0.869, respectively.

Thank you for the clarification and additional evaluation. The thin-structure and open-cavity examples are relevant to my concern. However, I'm wondering if the the global CLIP similarity of normal renders can directly establish whether broken connections and filled cavities occur less often?

The response also introduces a multi-initialization voting and occupancy-union procedure. I don't know if I miss anything, but are the reported geometry comparisons in the main paper obtained with this procedure, and do the end-to-end runtimes in Table 1 include its full cost? This is important for interpreting the paper's quality–efficiency advantage, since the response attributes improved connectivity partly to this procedure.

I will remain my score based on the joint structure–feature generation approach, with I'll prefer the above clarification is still needed.

Reviewer_HoVgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GrowFlow, a method for generating 3D geometry from a single image. Its core idea is to replace the traditional two-stage pipeline, which first generates a sparse voxel structure and then generates geometric features, with progressive flow matching based on an octree. It introduces a somewhat novel sparse multi-scale generation scheme and validates its effectiveness across multiple metrics and baselines.

Critical strengths
  1. Existing sparse 3D generation methods typically predict the structure first and then generate the corresponding geometric features. GrowFlow continues to adjust the structure as the spatial resolution progressively increases, rather than fully fixing the occupancy configuration at the initial stage.

  2. It attempts to construct a multi-resolution generation process suitable for sparse hierarchies with variable branching structures.

Critical weaknesses
  1. Insufficient evaluation metrics: ULIP and Uni3D use image–shape embedding similarity. These metrics can reflect semantic or joint embedding-space consistency between the generated results and the input image, but cannot directly measure surface geometry quality.

  2. The figures are not formatted consistently, as shown in Figures 4 and 5.

  3. The visualizations are not sufficiently convincing. Although the metrics are comparable to those of Trellis2, the visual quality is much worse.

  4. The paper does not adequately describe the decoder's network architecture, inputs and outputs, training objectives, or how it interfaces with the progressive octree generation process.

Questions for authors
  1. Why is there such a large discrepancy between the quantitative metrics and visual quality when compared with Trellis2?

  2. What is the decoder process? The paper does not provide a relevant description.

  3. How should geometric quality be evaluated?

1 comment

Thank you very much for recognizing the novelty of our work. In response to the weaknesses and questions you raised, we provide the following clarifications and corrections, which will be incorporated into the final version of the paper.

  1. Critical Weakness 1 — Use metrics to measure geometric quality.

In the field of 3D generation, there is currently no metric that can adequately measure geometric quality across objects of varying complexity. Complex objects require rich geometric detail, whereas simple objects are better represented by geometry that is clean and uncluttered.

To better compare geometric quality, we conducted an additional evaluation using multi-view normal renderings. We use the RGB image as the input to each model. For evaluation, we render both the ground-truth geometry and the generated geometry from four fixed viewpoints and compute the cosine similarity between the corresponding renderings using CLIP ("Learning Transferable Visual Models from Natural Language Supervision"). Because geometric and normal errors can introduce visible rendering artifacts, this evaluation provides complementary evidence of geometric quality. Our method achieved a CLIP score of 0.923, compared with 0.915 for Trellis.2. We will include these results in the final version of the paper. The scores for the methods in Table 1 are:

GrowFlow: 0.923 Trellis.2: 0.915 Direct3D-S2: 0.920 Hunyuan3D: 0.915 UniLat: 0.911 Hi3DGen: 0.902

  1. Critical Weakness 2 — Inconsistent formatting.

Thank you for pointing this out. We will thoroughly revise the final version to ensure consistent formatting throughout the paper.

  1. Critical Weakness 3 — The visualizations are not sufficiently convincing.

We acknowledge that Trellis.2 can produce richer local details, which may make some of its outputs visually more appealing. However, this does not necessarily translate into better average performance across the test set, as some outputs contain substantial spurious geometry. For example, the first row of Fig. 5 shows large extraneous block-like structures beside the character in the Trellis.2 output, whereas our result avoids these artifacts. Such failures provide a possible explanation for why richer details in some examples do not translate into higher aggregate scores. Nevertheless, we agree that ULIP and Uni3D do not fully capture local geometric fidelity, and we will clarify this distinction in the revised paper.

  1. Critical Weakness 4 — The decoder network architecture is not adequately described.

As stated in Section 4.1, we reuse the SparseFlex architecture and pretrained weights for the feature VAE, including its decoder. The encoder takes a point cloud uniformly sampled from the mesh surface and processes it using PointNet and Transformer layers to predict the mean and variance of the latent distribution. A latent representation is then sampled from this distribution and passed to the decoder, which uses a series of Transformer layers followed by a final linear layer to predict signed distance field (SDF) values. A mesh can then be extracted from the resulting SDF. During generation, the decoder converts the generated structured 3D latent features into the output mesh. For supervision, a mesh is extracted from the predicted SDF using FlexiCubes and rendered from randomly sampled camera viewpoints to compute a rendering-based supervision loss.

The structure decoder uses a decoder-only Transformer architecture. For the DiT backbone, we use adaLN-Zero conditioning: the timestep t and resolution level r are encoded into a global conditioning vector, which modulates features through adaptive layer normalization within each Transformer block, rather than being directly concatenated with or added to the patch tokens. The structure decoder's input consists of the noisy occupancy values of eight child nodes multiplied by an identity full-rank matrix, and its output is the denoised result. Multiplying by an identity full-rank matrix allows the feature outputs to cross-check one another, preventing an error in a single occupancy value from propagating into the result. The structure decoder is supervised using ground-truth occupancy labels with a binary cross-entropy (BCE) loss.

  1. Question 1 — Why is there a large discrepancy between the quantitative metrics and the visual quality?

This issue is addressed in our response to Critical Weakness 3 above.

  1. Question 2 — What is the decoder process?

This issue is addressed in our response to Critical Weakness 4 above.

  1. Question 3 — How should geometric quality be evaluated?

This issue is addressed in our response to Critical Weakness 1 above.

Comments on the paper

1 comment

Hi reviewers, as we are approaching the end of the author rebuttal phase, could you please take a look at the authors' responses and provide your feedback asap? Thx, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.