- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents HARMONY for single-image 3D scene reconstruction. It combines VLM reasoning with object generation and geometric refinement. Objects are placed through several predefined stages. A feedback loop checks the rendered scene and corrects errors.
Critical strengths
- The pipeline is well organized.
- The evaluation includes synthetic scenes, real images, and a user study. The results look good.
Critical weaknesses
- The technical novelty is limited. This is my main concern. Appendix B.1 states that all components are existing models. The contribution mainly lies in organizing these models into a workflow. However, several central ideas already appear in prior work. The method relies heavily on GPT for reasoning and decision-making. I do not see a clear theoretical or engineering contribution beyond integrating existing tools. This area is already crowded with similar agentic pipelines. The paper should clarify its differences from recent work such as AHA-3D. 2. Sec 3.1.2 I rarely see “depth axis” used as a standard term in Manhattan-frame estimation.
- Line 274: GPT is powerful, but this does not guarantee that estimating object orientation in this way is correct.
- The paper should also report token usage, API costs, and runtime to assess the efficiency and cost-effectiveness of this GPT-based pipeline.
Questions for authors
See Weaknesses
Technical Novelty, Orientation Accuracy, and Computational Efficiency
We thank reviewer nDFi for their thoughtful feedback. We appreciate the opportunity to clarify HARMONY’s contributions, provide further quantitative evidence on orientation accuracy and computational efficiency, and refine our terminology. We address each point below.
1. Technical novelty and contribution
AHA-3D comes out on Sep 25 (ICLR deadline) in demo/blog format without much evaluation and we are not supposed to discuss it in the submission.
We would like to clarify the difference between AHA-3D and ours here due to the reviewer request: While both systems use VLM agents with 3D vision models as tools, they address different problems. AHA-3D exploits temporal and multi-view evidence for indoor reconstruction. HARMONY addresses the underconstrained single-view setting, where depth ordering, hidden object extent, and metric dimensions must be inferred from incomplete evidence. Our depth-first, dependency-aware placement targets these ambiguities.
Contribution. HARMONY's contribution is at the system level: how an agent should order and ground its decisions when reconstructing a scene from a single view. Albeit using existing models; what is new is our coordination through dependency-aware scene construction, geometry-grounded object placement, and iterative render-and-correct feedback. Rather than predicting a holistic scene layout directly from image-level reasoning, HARMONY progressively resolves spatial dependencies through geometry- and layout- grounded placement and intermediate render-and-correct feedback.
To evaluate this strategy, we compare full HARMONY with an all-at-once variant. Both use the same input images, object meshes, camera, room geometry, and VLM. The variant predicts all object placements in a single call, without sequential placement, per-object verification, or geometric refinement. Across 100 3D-FRONT scenes (409 GT furniture objects), we establish one-to-one correspondences using Hungarian assignment based on 3D center distance, requiring the same category (bed, sofa, chair, table, cabinet, or lamp) and a distance of at most 1.5 m. HARMONY matches 324 GT objects versus 304 for the variant; unmatched GT objects count as misses. HARMONY also reduces median center error from 0.47 m to 0.38 m and footprint error from 18.1% to 13.7%, while improving yaw accuracy within 15° from 75.9% to 81.7%. These gains reflect the combined strategy, not any individual component.
2. Object orientation estimation
We evaluate yaw on the 324 matched objects using each scene's camera frame and treating rectangular tables as 180° symmetric. HARMONY achieves a median yaw error of 1.5° (3.7° for chairs, 1.1° for sofas) and a mean of 14.6%. The gap reflects a long-tailed error distribution, with large errors concentrated mostly among chairs. Among matched objects, 81.7% have errors within 15° and 85.5% within 30°.
We further compare three GPT-5.5 settings. The first is a GPT-5.5-only agentic baseline implemented through the Codex harness, given the same input images as HARMONY. It directly reasons about scene and object placement without HARMONY's reconstructed geometry, hierarchical placement, or per-object verification and refinement. The second uses HARMONY's components with all-at-once placement; the third is full HARMONY. Their yaw accuracies within 15° are 22.8%, 75.9%, and 81.7%, respectively. This demonstrates the benefit of geometry-grounded reconstruction over direct image-based reasoning. The additional 5.8-point gain reflects sequential placement, per-object verification, and geometric refinement together, not depth-first ordering alone.
3. Token usage, API costs, and runtime
Runtime and API cost. We measure an average end-to-end runtime of 71 minutes per scene across 300 HARMONY300 scenes, using a single node with two NVIDIA L40 GPUs: one dedicated to Hunyuan3D-2 inference and the other running the reconstruction pipeline. The pipeline executes its stages sequentially and processes objects one at a time within each stage; approximately 18 minutes per scene are spent waiting for hosted API responses.
At list prices, the average API cost is approximately 3.65 USD per scene, comprising 2.05 USD in GPT-5.5 calls and 1.60 USD in Gemini 3 Pro Image calls for amodal inpainting and empty-room generation. Hunyuan3D-2 runs locally and incurs no API cost. The pipeline makes an average of approximately 89 hosted-model calls per scene (77.5 GPT-5.5 and 11.5 Gemini calls), consuming 145.7K input tokens and 61.4K output tokens, including approximately 42.3K reasoning tokens. We will include these efficiency details in the revised appendix.
4. Terminology We will replace "depth axis" with "camera-space depth direction" or, when referring specifically to the room coordinate system, "Manhattan-frame depth direction" for precision.
We thank the reviewer again for the helpful suggestions. We hope these clarifications and additional analyses address the reviewer's concerns.