acceptodds
Under review as a conference paper at ICLR 2027

Tetris3D: 3D Scene Generation With Objects That Fit Together

Abstract

We propose Tetris3D, a generative framework for single-image 3D scene reconstruction that recovers objects which are physically and geometrically coherent as a scene. Existing methods often generate objects independently or couple them implicitly, providing limited guidance for ensuring fine-grained spatial compatibility between neighboring objects that interact with one another. To address this, we explicitly condition the generation of each object on the geometry of surrounding objects and their physical relationships, guiding its shape and pose to remain geometrically and physically plausible within the scene. Moreover, we introduce ComOb, a physics simulation-based dataset of 1.2M scenes featuring physical interactions across diverse object categories, with per-object meshes and pairwise physical relation annotations. Comprehensive experiments on synthetic and real-world scenes show that Tetris3D recovers coherent object shapes and poses even when interacting regions are occluded, and achieves state-of-the-art performance in both generation quality and physical stability. Our code, model weights, and dataset will be publicly released.

Primary area:
generative models
Keywords:
3D scene generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zawROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reconstructs a 3D scene from a single image as separate objects that physically fit together, even where they touch out of sight. It generates the objects one at a time, each conditioned on the shapes already placed and on how it touches them, and trains on 1.2M new physics-simulated scenes. It reports better accuracy and stability than recent methods on one synthetic and two real-world benchmarks.

Critical strengths
  1. A real problem with a simple, natural idea. When objects are generated one by one, they often sink into or float above each other, especially where they touch out of sight. The paper builds the scene one object at a time and lets each new object see the shapes already placed and how it touches them, passed in on the same grid the object is generated on. This is the main reason I see value in the work: the idea is easy to follow and only adds light conditions to an existing generator, so it should carry over to stronger generators.

  2. The gains hold on real photos. The model is trained only on simple simulated scenes, yet on two real-world benchmarks its objects are better placed than those of all baselines and, on most stability measures, clearly more stable. This matters to my rating because it suggests the idea works beyond the setting it was trained on, and the large simulated dataset could also be useful to others.

Critical weaknesses
  1. The main synthetic comparison may not be fair. The paper does not say whether the method is given the true shapes of the surrounding objects, which the baselines cannot use. This weighs most because it is the only test of generation quality.

  2. The source of the stability gains is not shown. There is no comparison with simply settling a baseline's output in the same simulator, the ablation removes neighbor shape and contact type together, and how meshes become collision shapes is not described. Since physical plausibility is one of the paper's two headline claims, this matters for judging the contribution.

Questions for authors
  1. In the synthetic test, does the method receive the true shapes of the surrounding objects, or does it generate them too?
  2. How are the generated meshes turned into collision shapes in the simulator, and is this done the same way for all methods?
  3. How many scenes are in each test set, and how much do the results vary across scenes, especially where the margins are small, such as per-object shape error in the ablation and penetration on the kitchen benchmark?
  4. How much do the neighbor shape and the contact type each contribute? The ablation removes them together, so I cannot tell which one carries the gain.

If these are answered, I will be glad to raise my score.

6 comments
AuthorsOpenReview ↗

[1/5] Author Rebuttal

Dear reviewer zawR,

We sincerely appreciate the constructive and insightful review. We also thank the reviewer for recognizing the value of our core idea and effectiveness.

Below, we address the concerns and questions noted in the review. If we have misunderstood any point or if further clarification would be helpful, please let us know. We would be happy to discuss any remaining questions in more detail.


[W1 & Q1] Evaluation protocol clarification and further evaluation for fair comparison

We would like to respond this point in two parts,

  • (1) clarification of the synthetic evaluation protocol, and
  • (2) additional comparisons under settings where no ground-truth surrounding geometry is provided for fair comparison.

1. Clarification of evaluation protocol (Table 1)

The main objective of Table 1 is to evaluate the benefit of conditioning on surrounding object geometry, which existing baselines do not consider during generation. We therefore provide the ground-truth primitive meshes as interaction context, and these primitives are not generated. We admit that this setting gives our method access to additional geometric context compared against the baselines. To address this concern in W1, we additionally evaluate fully autoregressive reconstruction without any ground-truth surrounding geometry, below. In this setting, each object is conditioned only on meshes generated in previous steps during inference. We report these results below. We add this clarification to our manuscript (Page 7, Line 350-368, Green).

AuthorsOpenReview ↗

[2/5] Author Rebuttal

2. Additional experiments for fair comparison

To provide a fair comparison without ground-truth object meshes, we conduct two additional evaluations in which all methods operate without ground-truth shapes and Tetris3D reconstructs the scenes in fully autoregressive manner. Further evaluation details and results are provided in the revised manuscript (Dataset construction-Page 21, Line 1101-1109, Green; Results-Tables 2, 7).

  • Exp 1: Fully autoregressive evauation on Toys4K-based composited scene. We construct 700 new scenes using only multiple Toys4K objects, without the primitive-based context used during training. The objects are arranged and physically settled using MuJoCo. During reconstruction, all objects are generated autoregressively using previously generated meshes as interaction context, with no ground-truth object geometry provided. Evaluation is performed over all objects in each scene.

Table R1. Toys4K evaluation under a new scene configuration. Ours generates in fully autoregressively and no GT interaction context is used.

BaselinesCD-SCD-OF1-SF1-OIoU-BICP-RotMMDCOVP-FIDUni3DULIPPen. Dep.D_meanE_peak
SAM3D8.7811.060.68750.73430.587811.261.645457.81996.02070.69760.7320.078867.273.3739
Shaper5.3813.130.80940.68640.69256.581.744657.819915.17270.58170.62970.026751.782.0404
WorldSculpt5.7011.720.79640.74540.68558.911.510960.86664.0960.66530.69870.0329544.242.0030
Amodal3R8.6313.760.72580.67860.619412.401.879558.09076.08830.67250.70980.0468659.562.7001
GENA3D12.3219.900.64710.60070.549118.292.601453.8938.14750.59070.62050.045688.362.8617
MIDI24.7126.770.50060.53760.362230.102.680453.8938.91690.5930.60533.83653710.7732.0235
SceneMaker16.2719.730.52470.56930.444925.122.994159.44512.95120.61560.6318.95335889.4461.9041
Ours2.296.010.93540.86800.84053.280.83967.23092.6080.72690.76790.0186312.641.0035

In Table R1, our method maintains strong performance under fully autoregressive generation and unseen scenes during training. We thank the reviewer for suggesting this complementary evaluation.

  • Exp 2: New Realistic Compositional Scene Evaluation. We conduct an additional experiment to assess the full autoregressive reconstruction pipeline in more complex multi-object scenes. The evaluation contains 30 newly constructed scenes with 5.67 objects per scene on average. Given a text prompt, an agent-based pipeline (GPT Astra) retrieves objects from a 3D asset pool and arranges them into a scene, which is then physically settled using MuJoCo. Further construction details are provided in the revised manuscript (Page 21, Line 1101-1109, Green).

Table R2. Quantitative comparison on new Realistic Compositional Scenes. Ours generates in fully autoregressively and no GT interaction context is used.

BaselinesCD-SCD-OF1-SF1-OIoU-BICP-RotPen. Dep.D_meanE_peak
SAM3D3.595.390.82360.82210.54309.850.0766633.371.7566
Shaper2.5214.070.90570.67240.60677.430.0733654.982.2997
WorldSculpt4.259.800.82960.77350.553413.430.0396518.721.995
Amodal3R6.0415.860.79850.70660.496815.710.157750.403.0951
GENA3D5.6118.770.80580.69120.495116.420.1348608.153.0224
MIDI16.0729.280.55810.57640.222233.364.7862689.6820.1427
SceneMaker11.2313.910.61720.65800.348021.217.88723738.6023.9906
Ours0.563.230.98090.92370.83402.700.0147129.820.3854

Under fully autoregressive setting with unseen objects, Tetris3D shows strong performance across all metrics. These results provide direct evidence that the learned interaction conditioning remains effective when each object relies only on previously generated meshes as context.

We added this experiment as an additional evaluation in the revised manuscript (Tables 2, 7).

AuthorsOpenReview ↗

[3/5] Author Rebuttal

[W2-1 & Q2] Clarification of collision shape conversion

All methods use the same simulation-ready conversion pipeline. Each generated mesh is converted into a compound collision shape using CoACD [1] with fixed parameters (concavity threshold 0.05, at most 64 parts) and the same physical parameters and MuJoCo settings are applied to Tetris3D and all baselines. Penetration is computed directly from the original generated surface geometry, rather than from the CoACD collision proxies. We add clarification this protocol in the revised paper (Page 21, Line 1082-1086, Red).

[1] Wei, X., Liu, M., Ling, Z., & Su, H. (2022). Approximate convex decomposition for 3d meshes with collision-aware concavity and tree search. ACM Transactions on Graphics (TOG), 41(4), 1-18.

[W2-2 & Q4] Ablation on condition types

We address this point with an ablation study that evaluates each condition separately. To separate the contributions of spatial context and relation type, we evaluate both conditions independently.

Table R3. Ablation of spatial context and physical relation conditioning.

Condition typesCD-SCD-OF1-SF1-OIoU-BICP-RotMMDCOVP-FIDUni3DULIPPen. Dep.D_meanE_peak
Full model3.901.280.79630.76910.76035.501.56384.301.8750.67040.70690.019239.220.1193
Spatial context only3.921.290.79590.76830.75875.431.56684.151.8790.67060.70670.019739.520.1331
Physical relation only5.251.510.76440.74550.70656.381.73481.892.0070.66180.69721.6124112.440.4943
Drop both5.291.520.76460.74430.70506.501.74582.352.1160.66100.69701.8006113.740.4702

The results show that both conditions improve the final performance, and spatial context accounts for most of the gain and physical relation remains modest. Physical relations complement spatial geometry by specifying how objects should interact, such as stacking, containment, leaning, or contact. Representative examples are provided in Figure. 10 (Page 27, Green). They also help to define the dependency graph, which determines the generation order and the neighboring geometry available to each object (Sec. 3.5).

Reviewer_Tq3eOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on single-image 3D scene reconstruction, mainly emphasizing the physical alignment between multiple objects (stacking, leaning, accommodating), and these contact areas are often exactly the ones occluded. Tetris3D, based on the TRELLIS.2 backbone, generates objects in the scene space one by one. Each object is conditioned on already generated neighbors, injecting a voxel-wise field as a condition — the condition consists of a truncated UDF to the neighbor surface, the direction and normal of the nearest surface point, and a learnable pairwise relation type embedding. There is also a 'visible→occluded' cross-attention module to handle occluded parts. During inference, object relations and generation order are predicted by a VLM. Experimentally, the authors built ComOb, a MuJoCo dataset of 1.2 million scenes featuring interactions between objects and parametric geometries. Experiments were conducted on Toys4K scenes synthesized by the ComOb generator, as well as on real-world scenes MessyKitchens and Picasso, demonstrating the superiority and generalizability of the method.

Critical strengths
  1. The idea of injecting inter-object relations as an explicit condition into the generation process is natural and meaningful. Generating each object against its already generated neighbors, instead of independently or through implicit attention, is a simple way to get objects that actually fit together.
  2. The experimental validation is thorough. The method is evaluated on both synthetic and real data, and the ablations are basically clean, which supports the effectiveness of the method.
Critical weaknesses

1.In Tab. 1 the scenes are composed by the authors' own ComOb generator — the same primitives, placement scripts, settling criteria and renderer as the training data. So everything except the target asset is in-distribution for Tetris3D and out-of-distribution for the baselines, which raises a fairness concern for the largest reported margins. It would help to compose the Toys4K scenes with an independent procedure or renderer, and/or to fine-tune one strong baseline on ComOb for a comparable budget. This matters because on the two benchmarks that are genuinely independent (Tab. 2) the object-level metrics are not clearly better than SAM-3D (e.g., CD-O 0.63 vs. 0.64 and F1-O 0.828 vs. 0.819 on MessyKitchens). It looks like the gain there mainly comes from more plausible placement and physical relations, not from better object shapes. I would like the authors to discuss this carefully.

2.The paper needs to show that the physical relation actually helps. The current ablation removes the whole c_int at once, so it cannot tell whether the gain comes from the spatial context (UDF, direction, normal) or from the relation type. Since the title, the method and the ComOb design are all built around physical relations, I would like the authors to evaluate the two parts separately.

3.Both the generation order and the relations rely heavily on the VLM. I would like the authors to discuss whether the method is robust when the VLM gives wrong relations or a wrong order, and what happens when such errors accumulate along the autoregressive generation. Tab. 5 does not really answer this: it is run on the simple ComOb-composed scenes, and the paper does not report how often the VLM relations differ from the ground truth.

Questions for authors

1.The paper's motivation is recovering occluded contact regions. Can you report shape metrics restricted to the occluded or contact regions on Tab. 2, so we can see whether shapes improve where it matters, beyond placement?

  1. Can authors add a "spatial context only" and a "relation type only" row to Tab. 3? Since the relation is already an independent dropout group during training (App. B.2), the first one should only need nulling the relation condition at inference.

  2. What is the VLM's relation accuracy on MessyKitchens/Picasso, and how often do the VLM relations differ from GT in Tab. 5?

8 comments
AuthorsOpenReview ↗

[1/5] Author Rebuttal

Dear Reviewer Tq3e,

We sincerely thank the reviewer for the careful review and the constructive comments. We also appreciate the reviewer for recognizing the value of our core idea to improve inter-object compatibility.

We address the points through,

  • (i) additional experiments beyond the original Toys4K evaluation for fair comparison,
  • (ii) ablations of spatial context and physical relation conditioning,
  • (iii) analysis of the effect of VLM-based relation and ordering predictions, and
  • (iv) quantitative evaluation of shape reconstruction in occluded regions.

We would be happy to provide further clarification or discuss any remaining questions or concerns.


[W1] Additional experiments beyond the original Toys4K evaluation for fair comparison

We thank the reviewer for suggesting these constructive experiments for a fair and solid comparison. We admit that the original Toys4K evaluation (Table 1) uses the ComOb generator and is therefore closer to the training setting. We therefore conduct additional experiments to examine two questions. First, we test whether the reported gains persist under evaluation settings that differ from the original ComOb scene construction (Exp 1, 2). Second, we test how much of the performance gap can be reduced by finetuning a strong baseline to ComOb (Exp 3). Overall, Tetris3D maintains strong performance in the newly constructed evaluation settings, while ComOb finetuning improves the baseline but does not close the gap. Further details, including dataset construction, are provided in the revised manuscript (Eval protocol-Page 7, Line 350-368, Green; Dataset construction-Page 21, Line 1092-1109, Green).


Exp 1: Toys4K with a new scene construction pipeline

We construct 700 new scenes using a procedure different from ComOb. Instead of using ComOb primitives, we build scenes using only multiple Toys4K objects, arrange them and physically settle them with the MuJoCo simulator. Rendering follows the common pipeline used in prior works [1, 2, 3].

Table R1. Toys4K evaluation under a new scene configuration.

BaselinesCD-SCD-OF1-SF1-OIoU-BICP-RotMMDCOVP-FIDUni3DULIPPen. Dep.D_meanE_peak
SAM3D8.7811.060.68750.73430.587811.261.645457.81996.02070.69760.7320.078867.273.3739
Shaper5.3813.130.80940.68640.69256.581.744657.819915.17270.58170.62970.026751.782.0404
WorldSculpt5.7011.720.79640.74540.68558.911.510960.86664.0960.66530.69870.0329544.242.0030
Amodal3R8.6313.760.72580.67860.619412.401.879558.09076.08830.67250.70980.0468659.562.7001
GENA3D12.3219.900.64710.60070.549118.292.601453.8938.14750.59070.62050.045688.362.8617
MIDI24.7126.770.50060.53760.362230.102.680453.8938.91690.5930.60533.83653710.7732.0235
SceneMaker16.2719.730.52470.56930.444925.122.994159.445,12.95120.61560.6318.95335889.4461.9041
Ours2.296.010.93540.86800.84053.280.83967.23092.6080.72690.76790.0186312.641.0035

Tetris3D maintains strong performance even when the surrounding context is composed only of Toys4K objects, rather than the primitive-based contexts used in ComOb. This result supports that the proposed interaction-aware generation remains effective under a fairer evaluation setting with fully autoregressive scene reconstruction.

AuthorsOpenReview ↗

[2/5] Author Rebuttal

Exp 2: New Realistic Compositional Scene Evaluation

We additionally conduct an experiment to evaluate performance on scenes built with a configuration not used in ComOb and generated through an independent scene construction pipeline. These scenes contains more realistic interaction with diverse objects. Example scenes are shown in the teaser (Figure 1) and (Page 27, Figure 9, Green). We construct scenes by retrieving objects from a diverse 3D asset pool and arranging them from text prompts using an LLM agent (GPT Astra), followed by simulation-based settling. The resulting scenes contain 5.67 objects per scene on average. Further construction details are provided in the revised manuscript (Page 21, Line 1101-1109, Green).

Table R2. Quantitative comparison on the new Realistic Compositional Scenes.

BaselinesCD-SCD-OF1-SF1-OIoU-BICP-RotPen. Dep.D_meanE_peak
SAM3D3.595.390.82360.82210.54309.850.0766633.371.7566
Shaper2.5214.070.90570.67240.60677.430.0733654.982.2997
WorldSculpt4.259.800.82960.77350.553413.430.0396518.721.995
Amodal3R6.0415.860.79850.70660.496815.710.157750.403.0951
GENA3D5.6118.770.80580.69120.495116.420.1348608.153.0224
MIDI16.0729.280.55810.57640.222233.364.7862689.6820.1427
SceneMaker11.2313.910.61720.65800.348021.217.88723738.6023.9906
Ours0.563.230.98090.92370.83402.700.0147129.820.3854

As shown in Table R2, Tetris3D outperforms the baselines across all reconstruction quality and physical stability metrics, further supporting its generalization to realistic multi-object scenes constructed independently of ComOb.


Exp 3: ComOb-finetuned baseline

Following the reviewer’s suggestion, we finetune WorldSculpt on ComOb for the 150k iterations and evaluate it on the original ComOb-Toys4K setting. We choose WorldSculpt because it provides competitive performance, publicly available training code, and a feasible finetuning cost.

Table R3. Comparison with ComOb-finetuned WorldSculpt.

BaselinesCD-SCD-OF1-SF1-OIoU-BICP-RotMMDCOVP-FIDUni3DULIPPen. Dep.D_meanE_peak
WorldSculpt-Finetuned10.871.740.38160.63450.485915.082.56872.973.7310.62270.64500.1327101.20.4882
WorldSculpt8.681.980.43020.59760.536917.312.97370.564.8160.54540.57760.1177103.50.4812

Finetuning WorldSculpt on ComOb improves object-level reconstruction and several generative-quality metrics, although some scene-level reconstruction and physical-stability metrics deteriorate. The finetuned model still remains far behind Ours, suggesting that exposure to the ComOb training distribution alone does not explain the performance gap.

Performance gap on MessyKitchens. Finally, we admit that the object-level performance margin on MessyKitchens is modest. We believe this is because the benchmark often contains many relatively simple kitchen objects, such as cups and bowls, and has a relatively low average occlusion rate of 7.64%. This leaves fewer cases that require recovering complex occluded geometry from neighboring objects. In contrast, we observe larger performance gaps on the Toys4K-ComOb setting and our New Realistic Compositional Scene evaluation (Exp 2), which contain more diverse objects and have higher average occlusion rates of 27.82% and 22.02%, respectively.

[1] Jianfeng Xiang, Xiaoxue Chen, Sicheng Xu, Ruicheng Wang, Zelong Lv, Yu Deng, Hongyuan Zhu, Yue Dong, Hao Zhao, Nicholas Jing Yuan, et al. Native and compact structured latents for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14419–14429, 2026.

[2] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 21469–21480. IEEE, 2025.

[3] Dong-Yang Li, Wang Zhao, Yuxin Chen, Wenbo Hu, Meng-Hao Guo, Fang-Lue Zhang, Ying Shan, and Shi-Min Hu. Pixal3d: Pixel-aligned 3d generation from images. In Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Papers, pp. 1–12, 2026a.

AuthorsOpenReview ↗

[3/5] Authors Rebuttal

[W2 & Q2] Ablation on spatial context and physical relation.

We thank the reviewer for these thoughtful comments and suggestions. To separate the contributions of spatial context and relation type, we evaluate both conditions independently.

Table R4. Ablation of spatial context and physical relation conditioning.

Condition typesCD-SCD-OF1-SF1-OIoU-BICP-RotMMDCOVP-FIDUni3DULIPPen. Dep.D_meanE_peak
Full model3.901.280.79630.76910.76035.501.56384.301.8750.67040.70690.019239.220.1193
Spatial context only3.921.290.79590.76830.75875.431.56684.151.8790.67060.70670.019739.520.1331
Physical relation only5.251.510.76440.74550.70656.381.73481.892.0070.66180.69721.6124112.440.4943
Drop both5.291.520.76460.74430.70506.501.74582.352.1160.66100.69701.8006113.740.4702

Table R4 show that both conditions improve the final performance, and spatial context accounts for most of the gain and physical relation remains modest. Physical relations complement spatial geometry by specifying how objects should interact, such as stacking, containment, leaning, or contact. Representative examples are provided in Figure 10 (Page 27). They also define the dependency graph, which determines the generation order and the neighboring geometry available to each object (Sec. 3.5). Consistent with this role, correcting the generation order in our W3 analysis (Table R6) yields a larger improvement than correcting relation labels alone. We clarified these roles in the revised manuscript (Page 9-10, Lines 482-509, Green).

Reviewer_cNmXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Tetris3D reconstructs a 3D scene from a single image by generating objects autoregressively so that each shape and pose fits its neighbors, including where contact is occluded. Existing scene generators either build objects independently and then place them, or couple them only through shared attention. Contact surfaces are usually hidden, so the result often penetrates, floats, or misses the fit. Tetris3D finetunes TRELLIS.2 (initialized from Pixal3D) and generates each object only after the objects it depends on.

Critical strengths

The interaction condition is a contribution. Tetris3D generates each object conditioned on the surfaces and physical relations of its neighbors, so contact can be recovered even when it is hidden in the image. The ablation supports this directly.

The method improves both reconstruction and physical stability, and it does so under the inputs available at test time.

Critical weaknesses

Are some evaluation numbers measured on the same kind of scene the model was trained on? ComOb builds each training scene by placing one object against simple primitives, such as boxes, bowls, and cones, and settling it in MuJoCo. The Toys4K benchmark, where the gains are largest, is built the same way.

Training never asks the model to fit two generated objects together. In training, the neighbor is a fixed primitive with a clean surface and a known relation. At test time, each new object is conditioned on meshes the model has just generated. A wrong shape, pose, or relation becomes the spatial context for every object that depends on it.

Questions for authors

On Toys4K, which meshes does Tetris3D generate, and which are given as ground truth? In particular, are the surrounding primitives ground-truth meshes during conditioning, and are they held fixed in the MuJoCo simulation while penetration and displacement are measured only on the target object?

On MessyKitchens and Picasso, how many objects are in a scene, and how long is the dependency chain? Are later objects conditioned only on meshes generated earlier in that chain, with no ground-truth geometry in the loop?

4 comments
AuthorsOpenReview ↗

[1/3] Author Rebuttal

Dear Reviewer cNmX,

We appreciate the reviewer for the time and effort devoted to reviewing our work. We also thank for recognizing the interaction condition as a key contribution.

Below, we first clarify the evaluation protocols in response to the questions raised in the review. We then address the specific points raised in the review and present additional experiments conducted during the rebuttal period. We would be happy to clarify any remaining questions or discuss any points further. If there is any misunderstanding for our response, please let us know.

[Q1 & Q2] Evaluation Protocol Clarification

[Q1] Toys4K Evaluation Protocol

Toys4K evaluation. The original Toys4K evaluation (Table 1 in manuscript) is designed to assess how effectively models leverage surrounding geometry for target reconstruction in a controlled setting. We therefore provide ground-truth primitive meshes and relation as interaction context and exclude them from reconstruction metrics. We acknowledge that this setting alone does not evaluate fully autoregressive reconstruction without ground-truth geometry. To complement this, Table 3 in the manuscript already evaluates fully autoregressive reconstruction without ground-truth geometry. Additionally, to address W1, we conduct new fully autoregressive evaluations on scenes composed only of general objects rather than training-time primitives. At inference, each object uses only previously generated meshes as interaction context, with no ground-truth surrounding geometry. We report the results below.

For the simulation-based evaluation, the generated target object is placed among the same fixed ground-truth primitives. The surrounding primitive meshes are kept fixed, while physical stability are measured only for the generated target object. The same simulation and metric protocol is applied to the outputs of all methods.

We add the clarification on our revised draft (Page 7, Line 350-368, Green).

[Q2] MessyKitchens and Picasso Evaluation Protocol

MessyKitchen and Picasso evaluation. For MessyKitchens and Picasso, since the main objective of this evaluation is to assess generalization to real-world scenes, we didn’t use any ground truth object meshes and use only previously generated meshes during inference time along the dependency chain as conditioning inputs.

MessyKitchens and Picasso dataset details. MessyKitchens contains 5.72 objects per scene on average (min:3, max:8), while Picasso contains 3.6 objects per scene on average (min:2, max:8). The longest dependency chain lengths per scene of each datasets are 1.72 and 2.5 on average, respectively. The dependency chain length is the number of dependency edges along the longest chain.

AuthorsOpenReview ↗

[2/3] Author Rebuttal

[W1] Additional Evaluation Beyond the Original ComOb Setting

We admit the reviewer’s concern and therefore conduct additional experiments to complement the evaluation. We therefore conduct two evaluations that move beyond the original ComOb setting. Both use (i) fully autoregressive reconstruction, where each object is conditioned only on previously generated meshes and ii) no ground-truth object geometry and relation is provided as interaction context. Across both settings, our method maintains strong reconstruction accuracy and physical stability. This shows that, although it is trained with primitive-based interaction contexts, it also generalizes well to scenes with general objects and previously generated meshes as context.


  • Exp 1: Evaluation on new Toys4K-Only scene configurations. We construct 700 new scenes arranging only multiple Toys4K objects, without the primitive-based context used in training setup. The objects are arranged and physically settled using MuJoCo. During reconstruction, all objects are generated autoregressively using previously generated meshes as interaction context, with no ground-truth object geometry provided.

Table R1. Toys4K-Only evaluation under a new scene configuration. Ours generates in fully autoregressively and no GT interaction context is used.

BaselinesCD-SCD-OF1-SF1-OIoU-BICP-RotMMDCOVP-FIDUni3DULIPPen. Dep.D_meanE_peak
SAM3D8.7811.060.68750.73430.587811.261.645457.81996.02070.69760.7320.078867.273.3739
Shaper5.3813.130.80940.68640.69256.581.744657.819915.17270.58170.62970.026751.782.0404
WorldSculpt5.7011.720.79640.74540.68558.911.510960.86664.0960.66530.69870.0329544.242.0030
Amodal3R8.6313.760.72580.67860.619412.401.879558.09076.08830.67250.70980.0468659.562.7001
GENA3D12.3219.900.64710.60070.549118.292.601453.8938.14750.59070.62050.045688.362.8617
MIDI24.7126.770.50060.53760.362230.102.680453.8938.91690.5930.60533.83653710.7732.0235
SceneMaker16.2719.730.52470.56930.444925.122.994159.44512.95120.61560.6318.95335889.4461.9041
Ours2.296.010.93540.86800.84053.280.83967.23092.6080.72690.76790.0186312.641.0035

Despite being trained with primitive-based interaction, Tetris3D maintains strong performance under fully autoregressive generation. This additional setting provides a more solid evaluation of the proposed interaction-aware generation framework.


  • Exp 2: New Realistic Compositional Scene Evaluation. We conduct an additional experiment to assess the full autoregressive reconstruction pipeline in more complex multi-object scenes. The evaluation contains 30 newly constructed scenes with 5.67 objects per scene on average (min: 4, max: 7) and an average longest dependency chain length of 2.57. Given a text prompt, an agent-based pipeline (GPT Astra) retrieves objects from a 3D asset pool and arranges them into a scene, which is then physically settled using MuJoCo. (Further details are available at Page 21, Line 1101-1109, Green)

Table R2. Quantitative comparison on new Realistic Compositional Scenes. Ours generates in fully autoregressively and no GT interaction context is used.

BaselinesCD-SCD-OF1-SF1-OIoU-BICP-RotPen. Dep.D_meanE_peak
SAM3D3.595.390.82360.82210.54309.850.0766633.371.7566
Shaper2.5214.070.90570.67240.60677.430.0733654.982.2997
WorldSculpt4.259.800.82960.77350.553413.430.0396518.721.995
Amodal3R6.0415.860.79850.70660.496815.710.157750.403.0951
GENA3D5.6118.770.80580.69120.495116.420.1348608.153.0224
MIDI16.0729.280.55810.57640.222233.364.7862689.6820.1427
SceneMaker11.2313.910.61720.65800.348021.217.88723738.6023.9906
Ours0.563.230.98090.92370.83402.700.0147129.820.3854

This comparison assesses Tetris3D under a scene construction setting independent to training, with autoregressive inference and no GT interaction context. In this setting, Ours shows strong performance.

We added these experiments as additional evaluations in the revised manuscript (Table 2, Table 7)

AuthorsOpenReview ↗

[3/3] Author Rebuttal

[W2] Generalization from training setting to test time inference

Thank you for providing us with the opportunity to clarify this point. We address this concern from two perspectives. First, Tetris3D is designed to use neighboring surface geometry and physical relations, rather than object identity. This makes the training scenario compatible with each autoregressive inference step. Second, we evaluate whether this behavior can generalize to fully autoregressive generation (test time).

Why the training setup aligns with test-time autoregressive inference. Tetris3D uses the local surface geometry of neighboring objects with their physical relations to guide generation, rather than the identity of the neighboring object. In both training and inference, the surrounding object therefore is fed into the model in the same form, as geometric and relational input. Thus, although training uses ground-truth primitive surfaces, the model learns to generate a target object to fit the geometry and relation of a provided neighboring surface. At test time, the same type of information is used, but the neighboring surface comes from a previously generated object.

Evaluation on test-time autoregressive inference. To verify that this behavior can generalize to autoregressive scene generation, our MessyKitchens and Picasso evaluations in Table 3 already use fully autoregressive inference. No ground-truth geometry or relation is inserted into the generation loop. We have clarified this point in the revised manuscript (Page 27, Lines 1402-1439), and we thank the reviewer for highlighting the need to make this setting explicit.

Additionally, our new Toys4K-Only evaluation (Table R1) and Realistic Compositional Scene evaluation (Table R2) further test the same setting. Across these evaluations, Tetris3D shows strong reconstruction performance and physical stability. These results support that the interaction behavior learned from clean primitive surfaces can generalize to autoregressive scene reconstruction using generated object geometry.

Dear Reviewer cNmX,

We sincerely appreciate your time and thoughtful review of our work. We have carefully addressed your comments through additional experiments and clarifications in our rebuttal and revised manuscript.

As the discussion period is coming to an end, we would be very grateful if you could let us know if you have any remaining questions or concerns. We would be pleased to address any points that may require further clarification.

Thank you again for your valuable time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary and Revisions

Dear AC and reviewers,

We sincerely thank all reviewers for taking the time and their thoughtful and constructive feedback. We appreciate the reviewers' recognition of the value of explicitly conditioning object generation on neighboring geometry and physical relations, as well as the improvements in reconstruction quality and physical stability.

In response to the reviewers' concerns, we have revised the manuscript and conducted additional experiments, particularly to address (i) evaluation fairness (cNmX, Tq3e, zawR), (ii) generalization to fully autoregressive reconstruction (cNmX, Tq3e, zawR), (iii) the individual contributions of interaction conditions (Tq3e, zawR), and (iv) further analyses, including the effects of VLM-based reasoning (Tq3e, zawR).

Changes in the revised manuscript are color-coded by reviewer: green for revisions addressing concerns shared by multiple reviewers, yellow for Reviewer cNmX, blue for Reviewer Tq3e, and red for Reviewer zawR.

Below, we summarize the main additions and revisions. We would be happy to discuss further for any remaining questions and concerns.

Summary of Added Experiments and Revisions

Added Experiments

  1. Fully autoregressive evaluation on Toys4K-only scenes (All reviewers). We added a new evaluation comprising 700 scenes constructed entirely from multiple Toys4K objects, rather than the primitives used in ComOb training. Tetris3D reconstructs all objects autoregressively, conditioning each only on previously generated meshes, without ground-truth interaction context. It maintains superior reconstruction quality and physical stability under this more challenging setting (Table 2, Sec. 4.1, App. C.2).

  2. Evaluation on independently constructed realistic compositional scenes (All reviewers). We added 30 realistic multi-object scenes with complex interaction, constructed using an independent agent-based pipeline and physics simulation. This evaluation further tests generalization beyond ComOb, with fully autoregressive reconstruction and no ground-truth object geometry. Tetris3D outperforms the baselines across all reconstruction and physical stability metrics (Table 7, Fig. 9, Apps. C.2 and D.1).

  3. Separate ablations of spatial context and physical relations (Tq3e, zawR). We expanded the ablation study to evaluate the individual contributions of neighboring-surface geometry and physical relation types. The results show that spatial context accounts for most of the reconstruction and stability improvements, while physical relations provide complementary semantic guidance. We revised the corresponding discussion to reflect these findings (Table 4, Sec. 4.2, Fig. 10).

  4. Comparison with a ComOb-finetuned baseline (Tq3e). We finetuned WorldSculpt on ComOb for 150K iterations to examine the effect of training-distribution advantages. Despite improvements in several metrics, it remains substantially behind Tetris3D (Table 10, App. D.1).

  5. Further analyses (Tq3e, zawR). We added (i) VLM relation and ordering analyses, including prediction accuracy, effects on reconstruction, comparisons with a stronger VLM, and failure cases (Tables 9 and 11, Fig. 11); (ii) occluded-region reconstruction evaluation using one-sided Chamfer distance, where Tetris3D achieves the best performance across all four benchmarks (Table 8) and (iii) comparison with simulator-based settling, showing that post-processing can improve penetration at the cost of reconstruction accuracy (Table 12, Apps. C.3 and D.1).

Other Revisions and Clarifications

  1. Evaluation protocols (All reviewers). We clarified that the original ComOb-based Toys4K evaluation uses ground-truth surrounding primitives, while the other scene-level evaluations use fully autoregressive reconstruction without ground-truth surrounding meshes. We also added details of the new evaluation datasets (Sec. 4, App. C.2).

  2. Simulation protocols (zawR). We clarified the common collision-shape conversion pipeline using CoACD, the shared simulation settings across methods, and the measurement of penetration before simulation (App. C.1).

  3. Statistical analysis and refined claims (zawR). We provided across-scene variations and paired bootstrap confidence intervals in our rebuttal. Accordingly, we refined the discussion of the modest object-level improvements on MessyKitchens and the V2I attention ablation (Secs. 4.1 and 4.2).

  4. Training-to-inference generalization and limitations (cNmX). We expanded the discussion of how interaction conditioning learned from primitive surfaces transfers to autoregressive reconstruction with generated meshes, and clarified the potential propagation of reconstruction and VLM prediction errors (App. E).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.