acceptodds
Withdrawn from ICLR 2027

VoxScene: Voxel Representation for 3D Scene Arrangement

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Current data-driven layout generation techniques typically encode objects using bounding-box proxies or implicit features. While computationally efficient, such representations abstract away volumetric occupancy, obscuring object boundaries and weakening direct geometric constraints on spatial arrangement. To address these limitations, we introduce VoxScene, which shifts the 3D scene arrangement paradigm to an explicit voxel representations, enabling direct modeling of fine-grained spatial relations. Our method factors the generation process into two stages: First, we generate box-level anchors to specify the coarse layout. Second, an object-centric diffusion network sequentially synthesizes instance-labeled volumetric occupancies conditioned on these anchors. Specifically, each voxel is assigned to at most one instance, preventing inter-instance overlap at the voxel level. These generated voxels further serve as geometric queries for asset retrieval. Extensive experiments on 3D-FRONT demonstrate that our method outperforms existing methods in physical plausibility while maintaining visual quality. Furthermore, results on M3D-Shelf show that VoxScene can generate high-quality arrangement for fine-grained objects under complex structural constraints, where previous methods struggle.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D scene generation, voxel diffusion, volumetric representation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_BUoxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VoxScene, which moves 3D scene arrangement toward an explicit voxel representation so that fine-grained spatial relations can be modeled directly. Generation happens in two stages. First, box-level anchors are generated to set the coarse layout. Then an object-centric diffusion network sequentially synthesizes instance-labeled volumetric occupancies conditioned on those anchors.

Critical strengths

(1) The paper introduces an explicit, instance-labeled voxel representation for 3D scene arrangement. This is an interesting departure from the box- and token-based representations common in prior work.

(2) The experiments show that VoxScene improves physical plausibility while keeping layouts realistic.

Critical weaknesses

(1) The main contribution seems to be the overall pipeline, specifically the coarse-to-fine two-stage decomposition, rather than new network designs.

(2) The paper has no figure illustrating the anchor generation network or the object-centric voxel generation network.

(3) On 3D-FRONT, the evaluation uses FID on top-down semantic renders, overlap, and CD. Prior work such as DiffuScene and PhyScene also reports KID.

(4) The set of baselines feels somewhat limited. Comparisons with more recent scene-synthesis methods (for example, InstructScene)

(5) The paper states that “each voxel belongs to at most one instance, preventing overlap.” However, Appendix C.1 shows that disabling free-space write-back increases OR by only 0.048, about 1% of the roughly 4.8 baseline. This suggests the exclusivity constraint accounts for only a small part of the measured collision reduction.

(6) It would help to include ablations justifying the specific choices for the anchor generation and object generation networks. For example, alternative architectures or a single-stage variant would show how much each design decision contributes.

(7) The experiments use only the living-room split, the smallest at 813 scenes. Results on the bedroom split, the largest, would give stronger evidence that the method generalizes and scales.

Questions for authors

(1) Convolution over slot order is not permutation-equivariant, even with random permutation during training. Could the authors explain why they chose a convolutional design over a transformer, which handles unordered sets naturally?

(2) A codebook with 32,768 entries and embedding dimension 1 is effectively scalar quantization. Could the authors explain the motivation for using VQ here, and why the model samples continuously and then quantizes rather than working in the discrete or continuous space throughout?

Reviewer_Qjw1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VoxScene, a two-stage generative framework for 3D scene arrangement that uses an explicit, instance-labeled voxel representation instead of bounding boxes or latent shape codes. In the first stage, a 1D U-Net diffusion model generates box-level anchors (position, heading, size, category), optionally conditioned on a floorplan, text, or known objects. In the second stage, an object-centric 3D latent diffusion model (on top of a VQ-VAE) sequentially generates each object's occupancy inside a local 64^3 block, conditioned on already-generated voxels, the target anchor, and the remaining pending anchors. Each newly generated object is written back only into free space of a global grid, so voxels are mutually exclusive across instances. Training uses stochastic context masking and anchor shifting. Final scenes are obtained by retrieving CAD assets with a soft Chamfer score on voxels, with optional filtering of poorly matched objects (τ_ret) and style clustering.

Experiments on 3D-FRONT (bedroom, living room, dining room) report lower intersection (Ip) and overlap (OR) than DiffuScene, PhyScene, Planner3D, and LayoutGPT, with comparable FID. On M3D-Shelf, a 490-scene subset of M3DLayout, VoxScene is compared with DiffuScene and SceneWeaver and shows much lower out-of-bounds, shelf-collision, and overlap rates, plus higher GPT-5.4 and user-study scores. The appendix includes ablations, threshold sensitivity, runtime, scene editing, and a small robotic data-augmentation experiment.

Critical strengths
  1. The shelf-level setting is a good test case and the paper shows a clear improvement there. Placing many small items inside a structure with thin boards and tight clearances is exactly where box-based or latent-code-based layout generators tend to fail, and the qualitative results (Fig. 5, Fig. 11) show far fewer objects penetrating shelf boards or ending up outside the shelf. Treating structures as voxelized context in a scene-completion setup is a natural way to handle this, and I think this setting is a useful addition to the usual room-level benchmarks.

  2. The idea is simple and the pipeline is easy to follow. Generating per-object occupancy in a canonical local block with context from neighbors, and then using that occupancy for geometry-aware retrieval, is a reasonable design. It also yields more geometry-consistent retrieval than size-only matching, as shown by the "anchor boxes + size-based retrieval" ablation (OR 11.17 vs. 7.78 without filtering). The paper is clearly written and the implementation details are thorough.

Critical weaknesses
  1. The paper's main claim, that mutually exclusive voxel occupancy is what reduces collisions, is not well supported by its own results. Appendix C.1 shows that disabling free-space write-back, i.e., the mechanism that enforces exclusivity, increases OR by only 0.048 (×10^3), against a full-model OR of 4.79. The authors acknowledge that this "contributes only a limited part of the overlap reduction." Also, all metrics are computed on retrieved CAD meshes that are scaled to the generated bounding boxes, so voxel-level exclusivity does not carry over to the final scene anyway.

    The headline result, "lowest Ip and OR in each setting," relies mostly on retrieval filtering. With τ_ret = 0.75, 6.8–7.8% of generated objects are discarded (Table 8), while the baselines get no filtering at all. Without filtering, VoxScene is not the best: its OR is worse than LayoutGPT's on Bedroom (9.54 vs. 8.91) and Diningroom (12.57 vs. 7.92), and its Ip is worse than PhyScene's on floorplan-conditioned Livingroom (3.42 vs. 3.30). Removing objects lowers collision almost by construction, and the paper does not report the number of objects per scene for any method. Some of the ablation results also point the other way from the text: removing the SDF head gives lower Ip/OR (1.40/3.43) than the full model (1.52/4.79), and so does removing pending-anchor conditions (OR 4.24).

  2. The M3D-Shelf comparison is not controlled. VoxScene receives the ground-truth shelf as a voxelized structural condition. As far as I can tell, DiffuScene has no access to the shelf geometry, and SceneWeaver generates a different shelf from a room-level prompt (Fig. 5 shows "the most similar scene" from SceneWeaver). The very large gaps in Ro and Rs (e.g., 0.05 vs. 0.40) therefore mostly reflect having the structure as input, not the voxel representation itself. The benchmark is also small (about 98 test shelves, 200 generated scenes), and the user study (12 participants, 10 renderings per method) is reported without variance or significance tests.

    More generally, the room-level evaluation is limited. FID is computed only on top-down semantic renderings, differences are small, and VoxScene is worse than DiffuScene on Bedroom. Strong recent diffusion baselines that are cited (e.g., MiDiffusion, SemLayoutDiff) are not compared, and PhyScene is evaluated only on living rooms. The CD metric counts distinct CAD IDs and can exceed 1 (1.069 in Table 1). Noisy voxel queries naturally hit more distinct assets, so CD does not say much about layout diversity.

Questions for authors
  1. Could you compare against baselines given an equivalent post-hoc filter, e.g., removing the same fraction of objects with the worst collisions or retrieval scores? Please also report the average number of objects per scene for all methods with and without filtering.
  2. Given that free-space write-back changes OR by only 0.048 (Appendix C.1), which component do you consider responsible for the collision reduction? How do you interpret the ablation where removing the SDF head gives lower Ip/OR than the full model?
  3. On M3D-Shelf, can DiffuScene (or another baseline) be given the same shelf geometry as a condition, e.g., as a known object in scene completion, so that the comparison isolates the effect of the representation?
  4. How sensitive are the results to the random generation order? Variance over orders or seeds would help.

I am willing to raise my score if all the questions are addressed adequately.

Reviewer_ZJrhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method for generating 3D scene layout using box level anchors, followed by object-centric voxel diffusion. The local blocks encoded generated objects, anchors and structure context. To preserve instance id, a write-back operation is performed. The method is evaluated on 3DFront and M3D-shelf subset. User study is also provided.

Critical strengths

The motivation of having a voxel level representation rather than box-shaped proxy is a valid arugment. A lot of indoor layout needs such fine-grained info, like the chairs fit into the table, or objects put on the sofa.

The object centric diffusion perserves the instance-level information while avoiding diffusion on a global scale grid. The abliation study shows that the proposed scheme can better modeling local occupancy and neighboring geometry.

The numeric resutls and especially the shelf-collision example do illustrates the advantages of the method.

Critical weaknesses

The overall quality of the result is still on the weak side. Algthough the authors claims to solve interlocking cases, those are not very often seen (if there are any) in the room level results. The shelf example, I regard as a special case, where actually the space of the shelf can be regarded as separated rooms / spaces which do not nesseary need this voxel based solution.

The difference between the comparsion methods makes the evaluation difficult to be fair. Like how the shelf-case is applied for SceneWeaver is questionable, and might significanlty affects the resutls.

For layout generation, espeically for the bedroom case, the method is not better than the one compared with.

Questions for authors

The authors should provide more details on how and why the other method is configured for the comparsion, especially for the shelf-organizing caes.

Providing more details on how the user study is conducted. On what kind of results, in which way?

Why CD is not measured in the ablation study?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.