- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper reconstructs a 3D scene from a single image as separate objects that physically fit together, even where they touch out of sight. It generates the objects one at a time, each conditioned on the shapes already placed and on how it touches them, and trains on 1.2M new physics-simulated scenes. It reports better accuracy and stability than recent methods on one synthetic and two real-world benchmarks.
Critical strengths
-
A real problem with a simple, natural idea. When objects are generated one by one, they often sink into or float above each other, especially where they touch out of sight. The paper builds the scene one object at a time and lets each new object see the shapes already placed and how it touches them, passed in on the same grid the object is generated on. This is the main reason I see value in the work: the idea is easy to follow and only adds light conditions to an existing generator, so it should carry over to stronger generators.
-
The gains hold on real photos. The model is trained only on simple simulated scenes, yet on two real-world benchmarks its objects are better placed than those of all baselines and, on most stability measures, clearly more stable. This matters to my rating because it suggests the idea works beyond the setting it was trained on, and the large simulated dataset could also be useful to others.
Critical weaknesses
-
The main synthetic comparison may not be fair. The paper does not say whether the method is given the true shapes of the surrounding objects, which the baselines cannot use. This weighs most because it is the only test of generation quality.
-
The source of the stability gains is not shown. There is no comparison with simply settling a baseline's output in the same simulator, the ablation removes neighbor shape and contact type together, and how meshes become collision shapes is not described. Since physical plausibility is one of the paper's two headline claims, this matters for judging the contribution.
Questions for authors
- In the synthetic test, does the method receive the true shapes of the surrounding objects, or does it generate them too?
- How are the generated meshes turned into collision shapes in the simulator, and is this done the same way for all methods?
- How many scenes are in each test set, and how much do the results vary across scenes, especially where the margins are small, such as per-object shape error in the ablation and penetration on the kitchen benchmark?
- How much do the neighbor shape and the contact type each contribute? The ablation removes them together, so I cannot tell which one carries the gain.
If these are answered, I will be glad to raise my score.
6 comments
[2/5] Author Rebuttal
2. Additional experiments for fair comparison
To provide a fair comparison without ground-truth object meshes, we conduct two additional evaluations in which all methods operate without ground-truth shapes and Tetris3D reconstructs the scenes in fully autoregressive manner. Further evaluation details and results are provided in the revised manuscript (Dataset construction-Page 21, Line 1101-1109, Green; Results-Tables 2, 7).
- Exp 1: Fully autoregressive evauation on Toys4K-based composited scene. We construct 700 new scenes using only multiple Toys4K objects, without the primitive-based context used during training. The objects are arranged and physically settled using MuJoCo. During reconstruction, all objects are generated autoregressively using previously generated meshes as interaction context, with no ground-truth object geometry provided. Evaluation is performed over all objects in each scene.
Table R1. Toys4K evaluation under a new scene configuration. Ours generates in fully autoregressively and no GT interaction context is used.
| Baselines | CD-S | CD-O | F1-S | F1-O | IoU-B | ICP-Rot | MMD | COV | P-FID | Uni3D | ULIP | Pen. Dep. | D_mean | E_peak |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| SAM3D | 8.78 | 11.06 | 0.6875 | 0.7343 | 0.5878 | 11.26 | 1.6454 | 57.8199 | 6.0207 | 0.6976 | 0.732 | 0.078 | 867.27 | 3.3739 |
| Shaper | 5.38 | 13.13 | 0.8094 | 0.6864 | 0.6925 | 6.58 | 1.7446 | 57.8199 | 15.1727 | 0.5817 | 0.6297 | 0.026 | 751.78 | 2.0404 |
| WorldSculpt | 5.70 | 11.72 | 0.7964 | 0.7454 | 0.6855 | 8.91 | 1.5109 | 60.8666 | 4.096 | 0.6653 | 0.6987 | 0.0329 | 544.24 | 2.0030 |
| Amodal3R | 8.63 | 13.76 | 0.7258 | 0.6786 | 0.6194 | 12.40 | 1.8795 | 58.0907 | 6.0883 | 0.6725 | 0.7098 | 0.0468 | 659.56 | 2.7001 |
| GENA3D | 12.32 | 19.90 | 0.6471 | 0.6007 | 0.5491 | 18.29 | 2.6014 | 53.893 | 8.1475 | 0.5907 | 0.6205 | 0.045 | 688.36 | 2.8617 |
| MIDI | 24.71 | 26.77 | 0.5006 | 0.5376 | 0.3622 | 30.10 | 2.6804 | 53.893 | 8.9169 | 0.593 | 0.6053 | 3.8365 | 3710.77 | 32.0235 |
| SceneMaker | 16.27 | 19.73 | 0.5247 | 0.5693 | 0.4449 | 25.12 | 2.9941 | 59.445 | 12.9512 | 0.6156 | 0.631 | 8.9533 | 5889.44 | 61.9041 |
| Ours | 2.29 | 6.01 | 0.9354 | 0.8680 | 0.8405 | 3.28 | 0.839 | 67.2309 | 2.608 | 0.7269 | 0.7679 | 0.0186 | 312.64 | 1.0035 |
In Table R1, our method maintains strong performance under fully autoregressive generation and unseen scenes during training. We thank the reviewer for suggesting this complementary evaluation.
- Exp 2: New Realistic Compositional Scene Evaluation. We conduct an additional experiment to assess the full autoregressive reconstruction pipeline in more complex multi-object scenes. The evaluation contains 30 newly constructed scenes with 5.67 objects per scene on average. Given a text prompt, an agent-based pipeline (GPT Astra) retrieves objects from a 3D asset pool and arranges them into a scene, which is then physically settled using MuJoCo. Further construction details are provided in the revised manuscript (Page 21, Line 1101-1109, Green).
Table R2. Quantitative comparison on new Realistic Compositional Scenes. Ours generates in fully autoregressively and no GT interaction context is used.
| Baselines | CD-S | CD-O | F1-S | F1-O | IoU-B | ICP-Rot | Pen. Dep. | D_mean | E_peak |
|---|---|---|---|---|---|---|---|---|---|
| SAM3D | 3.59 | 5.39 | 0.8236 | 0.8221 | 0.5430 | 9.85 | 0.0766 | 633.37 | 1.7566 |
| Shaper | 2.52 | 14.07 | 0.9057 | 0.6724 | 0.6067 | 7.43 | 0.0733 | 654.98 | 2.2997 |
| WorldSculpt | 4.25 | 9.80 | 0.8296 | 0.7735 | 0.5534 | 13.43 | 0.0396 | 518.72 | 1.995 |
| Amodal3R | 6.04 | 15.86 | 0.7985 | 0.7066 | 0.4968 | 15.71 | 0.157 | 750.40 | 3.0951 |
| GENA3D | 5.61 | 18.77 | 0.8058 | 0.6912 | 0.4951 | 16.42 | 0.1348 | 608.15 | 3.0224 |
| MIDI | 16.07 | 29.28 | 0.5581 | 0.5764 | 0.2222 | 33.36 | 4.786 | 2689.68 | 20.1427 |
| SceneMaker | 11.23 | 13.91 | 0.6172 | 0.6580 | 0.3480 | 21.21 | 7.8872 | 3738.60 | 23.9906 |
| Ours | 0.56 | 3.23 | 0.9809 | 0.9237 | 0.8340 | 2.70 | 0.0147 | 129.82 | 0.3854 |
Under fully autoregressive setting with unseen objects, Tetris3D shows strong performance across all metrics. These results provide direct evidence that the learned interaction conditioning remains effective when each object relies only on previously generated meshes as context.
We added this experiment as an additional evaluation in the revised manuscript (Tables 2, 7).
[3/5] Author Rebuttal
[W2-1 & Q2] Clarification of collision shape conversion
All methods use the same simulation-ready conversion pipeline. Each generated mesh is converted into a compound collision shape using CoACD [1] with fixed parameters (concavity threshold 0.05, at most 64 parts) and the same physical parameters and MuJoCo settings are applied to Tetris3D and all baselines. Penetration is computed directly from the original generated surface geometry, rather than from the CoACD collision proxies. We add clarification this protocol in the revised paper (Page 21, Line 1082-1086, Red).
[1] Wei, X., Liu, M., Ling, Z., & Su, H. (2022). Approximate convex decomposition for 3d meshes with collision-aware concavity and tree search. ACM Transactions on Graphics (TOG), 41(4), 1-18.
[W2-2 & Q4] Ablation on condition types
We address this point with an ablation study that evaluates each condition separately. To separate the contributions of spatial context and relation type, we evaluate both conditions independently.
Table R3. Ablation of spatial context and physical relation conditioning.
| Condition types | CD-S | CD-O | F1-S | F1-O | IoU-B | ICP-Rot | MMD | COV | P-FID | Uni3D | ULIP | Pen. Dep. | D_mean | E_peak |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Full model | 3.90 | 1.28 | 0.7963 | 0.7691 | 0.7603 | 5.50 | 1.563 | 84.30 | 1.875 | 0.6704 | 0.7069 | 0.0192 | 39.22 | 0.1193 |
| Spatial context only | 3.92 | 1.29 | 0.7959 | 0.7683 | 0.7587 | 5.43 | 1.566 | 84.15 | 1.879 | 0.6706 | 0.7067 | 0.0197 | 39.52 | 0.1331 |
| Physical relation only | 5.25 | 1.51 | 0.7644 | 0.7455 | 0.7065 | 6.38 | 1.734 | 81.89 | 2.007 | 0.6618 | 0.6972 | 1.6124 | 112.44 | 0.4943 |
| Drop both | 5.29 | 1.52 | 0.7646 | 0.7443 | 0.7050 | 6.50 | 1.745 | 82.35 | 2.116 | 0.6610 | 0.6970 | 1.8006 | 113.74 | 0.4702 |
The results show that both conditions improve the final performance, and spatial context accounts for most of the gain and physical relation remains modest. Physical relations complement spatial geometry by specifying how objects should interact, such as stacking, containment, leaning, or contact. Representative examples are provided in Figure. 10 (Page 27, Green). They also help to define the dependency graph, which determines the generation order and the neighboring geometry available to each object (Sec. 3.5).
[1/5] Author Rebuttal
Dear reviewer zawR,
We sincerely appreciate the constructive and insightful review. We also thank the reviewer for recognizing the value of our core idea and effectiveness.
Below, we address the concerns and questions noted in the review. If we have misunderstood any point or if further clarification would be helpful, please let us know. We would be happy to discuss any remaining questions in more detail.
[W1 & Q1] Evaluation protocol clarification and further evaluation for fair comparison
We would like to respond this point in two parts,
1. Clarification of evaluation protocol (Table 1)
The main objective of Table 1 is to evaluate the benefit of conditioning on surrounding object geometry, which existing baselines do not consider during generation. We therefore provide the ground-truth primitive meshes as interaction context, and these primitives are not generated. We admit that this setting gives our method access to additional geometric context compared against the baselines. To address this concern in W1, we additionally evaluate fully autoregressive reconstruction without any ground-truth surrounding geometry, below. In this setting, each object is conditioned only on meshes generated in previous steps during inference. We report these results below. We add this clarification to our manuscript (Page 7, Line 350-368, Green).