acceptodds
Withdrawn from ICLR 2027

HARMONY: Hierarchical Agentic Reasoning for MONocular Image-to-Scene Synthesis

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Compositional 3D scene reconstruction has recently been explored from two directions: agentic reasoning that provides semantic understanding of spatial relationships but lacks precise alignment with input images; and visual geometry foundation models that predict dense point maps from input images but the reconstruction quality is limited. Therefore, recovering a complete 3D scene from a single monocular image with accurate inter-object relationships and high-fidelity reconstruction quality remains challenging. In this paper, we present HARMONY, a hierarchical chain-of-thought framework that leverages both agentic reasoning and visual geometry foundation. Given an image of an indoor scene, starting from an empty 3D floorplan, HARMONY first calibrates the camera against the reference image to establish a semantically-grounded spatial frame, then uses agentic VLM reasoning to recover the 3D room layout and an initial placement order. It then places the objects in a hierarchical order, from wall-mounted elements, free-standing furniture, to dependent decorations on top of furniture. We also use depth-first traversal for furniture so each placement conditions on previously resolved structure and a reflective feedback loop to avoid error accumulation. After each object placement by VLM, we use the point cloud estimations to perform geometry-based refinement so that the rendered image aligns better with the input. HARMONY can produce 3D scenes that are semantically consistent and perceptually aligned with the reference image, extending single-image compositional reconstruction to complex indoor scene images. Experiments on synthetic and real-world images demonstrate that HARMONY outperforms the evaluated reconstruction baselines, while qualitative comparisons with GPT-6 Astra suggest more faithful object arrangements and better preservation of scene details.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Graphics, Agentic Systems, 3D Reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_nDFiOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HARMONY for single-image 3D scene reconstruction. It combines VLM reasoning with object generation and geometric refinement. Objects are placed through several predefined stages. A feedback loop checks the rendered scene and corrects errors.

Critical strengths
  1. The pipeline is well organized.
  2. The evaluation includes synthetic scenes, real images, and a user study. The results look good.
Critical weaknesses
  1. The technical novelty is limited. This is my main concern. Appendix B.1 states that all components are existing models. The contribution mainly lies in organizing these models into a workflow. However, several central ideas already appear in prior work. The method relies heavily on GPT for reasoning and decision-making. I do not see a clear theoretical or engineering contribution beyond integrating existing tools. This area is already crowded with similar agentic pipelines. The paper should clarify its differences from recent work such as AHA-3D. 2. Sec 3.1.2 I rarely see “depth axis” used as a standard term in Manhattan-frame estimation.
  2. Line 274: GPT is powerful, but this does not guarantee that estimating object orientation in this way is correct.
  3. The paper should also report token usage, API costs, and runtime to assess the efficiency and cost-effectiveness of this GPT-based pipeline.
Questions for authors

See Weaknesses

1 comment
AuthorsOpenReview ↗

Technical Novelty, Orientation Accuracy, and Computational Efficiency

We thank reviewer nDFi for their thoughtful feedback. We appreciate the opportunity to clarify HARMONY’s contributions, provide further quantitative evidence on orientation accuracy and computational efficiency, and refine our terminology. We address each point below.

1. Technical novelty and contribution

AHA-3D comes out on Sep 25 (ICLR deadline) in demo/blog format without much evaluation and we are not supposed to discuss it in the submission.

We would like to clarify the difference between AHA-3D and ours here due to the reviewer request: While both systems use VLM agents with 3D vision models as tools, they address different problems. AHA-3D exploits temporal and multi-view evidence for indoor reconstruction. HARMONY addresses the underconstrained single-view setting, where depth ordering, hidden object extent, and metric dimensions must be inferred from incomplete evidence. Our depth-first, dependency-aware placement targets these ambiguities.

Contribution. HARMONY's contribution is at the system level: how an agent should order and ground its decisions when reconstructing a scene from a single view. Albeit using existing models; what is new is our coordination through dependency-aware scene construction, geometry-grounded object placement, and iterative render-and-correct feedback. Rather than predicting a holistic scene layout directly from image-level reasoning, HARMONY progressively resolves spatial dependencies through geometry- and layout- grounded placement and intermediate render-and-correct feedback.

To evaluate this strategy, we compare full HARMONY with an all-at-once variant. Both use the same input images, object meshes, camera, room geometry, and VLM. The variant predicts all object placements in a single call, without sequential placement, per-object verification, or geometric refinement. Across 100 3D-FRONT scenes (409 GT furniture objects), we establish one-to-one correspondences using Hungarian assignment based on 3D center distance, requiring the same category (bed, sofa, chair, table, cabinet, or lamp) and a distance of at most 1.5 m. HARMONY matches 324 GT objects versus 304 for the variant; unmatched GT objects count as misses. HARMONY also reduces median center error from 0.47 m to 0.38 m and footprint error from 18.1% to 13.7%, while improving yaw accuracy within 15° from 75.9% to 81.7%. These gains reflect the combined strategy, not any individual component.

2. Object orientation estimation

We evaluate yaw on the 324 matched objects using each scene's camera frame and treating rectangular tables as 180° symmetric. HARMONY achieves a median yaw error of 1.5° (3.7° for chairs, 1.1° for sofas) and a mean of 14.6%. The gap reflects a long-tailed error distribution, with large errors concentrated mostly among chairs. Among matched objects, 81.7% have errors within 15° and 85.5% within 30°.

We further compare three GPT-5.5 settings. The first is a GPT-5.5-only agentic baseline implemented through the Codex harness, given the same input images as HARMONY. It directly reasons about scene and object placement without HARMONY's reconstructed geometry, hierarchical placement, or per-object verification and refinement. The second uses HARMONY's components with all-at-once placement; the third is full HARMONY. Their yaw accuracies within 15° are 22.8%, 75.9%, and 81.7%, respectively. This demonstrates the benefit of geometry-grounded reconstruction over direct image-based reasoning. The additional 5.8-point gain reflects sequential placement, per-object verification, and geometric refinement together, not depth-first ordering alone.

3. Token usage, API costs, and runtime

Runtime and API cost. We measure an average end-to-end runtime of 71 minutes per scene across 300 HARMONY300 scenes, using a single node with two NVIDIA L40 GPUs: one dedicated to Hunyuan3D-2 inference and the other running the reconstruction pipeline. The pipeline executes its stages sequentially and processes objects one at a time within each stage; approximately 18 minutes per scene are spent waiting for hosted API responses.

At list prices, the average API cost is approximately 3.65 USD per scene, comprising 2.05 USD in GPT-5.5 calls and 1.60 USD in Gemini 3 Pro Image calls for amodal inpainting and empty-room generation. Hunyuan3D-2 runs locally and incurs no API cost. The pipeline makes an average of approximately 89 hosted-model calls per scene (77.5 GPT-5.5 and 11.5 Gemini calls), consuming 145.7K input tokens and 61.4K output tokens, including approximately 42.3K reasoning tokens. We will include these efficiency details in the revised appendix.

4. Terminology We will replace "depth axis" with "camera-space depth direction" or, when referring specifically to the room coordinate system, "Manhattan-frame depth direction" for precision.

We thank the reviewer again for the helpful suggestions. We hope these clarifications and additional analyses address the reviewer's concerns.

Reviewer_kEZxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents HARMONY, a training-free pipeline for reconstructing compositional 3D indoor scenes from a single image. It combines VLM-based semantic reasoning with geometry-based refinement. The method estimates room layout and camera pose, reconstructs objects, and places them hierarchically. Object placement is further refined using silhouettes and depth, with VLM feedback to check intermediate results.

The method is evaluated on HARMONY300, including 100 synthetic 3D-FRONT scenes with 3D ground truth and 200 real images. The paper reports improvements over recent baselines, together with a user study and ablation experiments.

Critical strengths
  1. Good combination of semantic reasoning and geometry. Using VLMs for relations, orientation, and grouping, while using geometry for position, depth, and scale, is reasonable and well motivated.

  2. Strong results and solid implementation. HARMONY performs well on the reported metrics, and the qualitative results are convincing. The paper also provides useful implementation details and ablations.

Critical weaknesses
  1. The comparison does not clearly isolate HARMONY's main contribution. Several baselines are evaluated with modified settings. For example, SAM3D uses HARMONY masks, some methods use HARMONY backgrounds, CAST uses an unofficial implementation, and some components of VIGA/3D-RE-GEN are replaced. Meanwhile, HARMONY uses several strong foundation models. Therefore, it is difficult to know whether the improvement mainly comes from the proposed hierarchical reasoning or from stronger components and different evaluation settings.
    A more controlled comparison would be helpful, where methods use the same detections, meshes, geometry modules, and foundation models, and only the reasoning/placement strategy is changed.

  2. The evaluation is still limited for supporting strong 3D reconstruction claims. Only 100 synthetic scenes have full 3D ground truth. Evaluation on real images mainly measures similarity from the input view, which cannot fully evaluate hidden geometry or distinguish accurate reconstruction from plausible completion.
    The ablations mainly use perceptual metrics, and some improvements from hierarchy-related components are small. The full-model numbers in the ablation table differ from the main table without clear explanation. The user study contains only 20 scenes and seems to include only scenes where all methods successfully generated renders, which may introduce selection bias.

Questions for authors
  1. Can the authors provide a controlled experiment where all methods use the same detections, reconstructed assets, camera initialization, and geometry refinement, while only changing the reasoning/placement strategy?

  2. Why are the full-model N-CLIP and PL values in the ablation table different from the main results? What subset and evaluation protocol are used?

  3. How are predictions aligned with 3D-FRONT ground truth before computing Chamfer distance and F-score? What are the units of the F-score thresholds?

  4. How were the 20 scenes in the user study selected? Does requiring successful renders from all methods introduce selection bias?

  5. Can the authors evaluate novel views, object pose/scale, support relations, collision/contact errors, or other metrics that more directly measure 3D reconstruction correctness?

3 comments
AuthorsOpenReview ↗

Controlled comparisons

We thank the reviewer kEZx for the constructive comments. We agree that controlled experiments and geometry-focused evaluations are important for establishing HARMONY's contributions. In response, we have added a controlled comparison of placement strategies, clarified our evaluation protocols, and expanded our 3D evaluation to include object-level accuracy, physical plausibility, and occluded geometry.

1. Controlled comparison of placement strategies

We clarify that modifications to the baselines were made to provide stronger and more competitive comparisons, rather than to disadvantage prior methods. Specifically, SAM3D requires pre-masked objects as input and does not provide its own masking module; we therefore provide masks to enable its object reconstruction, as described in Lines 374–376. CAST does not release official code, so we use an unofficial implementation. For VIGA and 3D-Re-GEN, we retain their respective pipelines and replace only their base models with stronger alternatives: GPT-5.5 instead of GPT-5 for VIGA, and GPT-Image-2 instead of Gemini-Image for 3D-Re-GEN (Lines 798–804).

Controlled placement comparison. To evaluate HARMONY's combined placement procedure, we hold the reconstruction inputs fixed: the same detections and masks, generated meshes, calibrated camera, room shell, and VLM (GPT-5.5). We compare HARMONY's sequential placement with per-object verification and geometric refinement against a single-call all-at-once placement baseline without per-object verification or refinement. We run each method 3 times on a subset of 30 3D-FRONT scenes:

Metric (30 scenes, 124 GT objects)HARMONY (mean ± std)All-at-once (mean ± std)
GT objects matched97 ± 093.3 ± 1.2
GT objects within 0.5 m (out of 124)46.8 ± 0.8%40.9 ± 2.3%
Median center error (matched)0.40 ± 0.00 m0.46 ± 0.02 m
Yaw within 15° (matched)78.7 ± 1.6%69.7 ± 2.4%
Median footprint error (matched)12.4 ± 0.0%15.3 ± 2.1%

The reported ± values denote standard deviations across three run-level aggregate metrics, with each run evaluated on the same 30 scenes. Matching uses per-scene one-to-one Hungarian assignment on 3D center distance, constrained to the same category (bed, sofa, chair, table, cabinet, or lamp), with pairs farther than 1.5 m between ground truth 3D and reconstructed object rejected. Distances are measured in the input camera's gravity-aligned metric frame, without rescaling or alignment. Unmatched GT objects count as misses, while unmatched predictions are not scored. Center, yaw, and footprint errors are computed over matched pairs.

The object-level improvements persist across the three runs, with relatively small run-to-run variation. On the same 30-scene subset, scene-level CD is similar between the two methods (HARMONY: 0.0532; all-at-once: 0.0534). This suggests that normalized, ICP-aligned scene-level metrics may be less sensitive to object-level placement errors, motivating our additional metric-frame object-level evaluation.

This experiment compares HARMONY's combined sequential placement, per-object verification, and geometric refinement procedure against single-call placement. It does not separately isolate the contributions of placement order, verification, and refinement.

Our random-order ablation further examines placement order while retaining the sequential placement procedure. It compares depth-first ordering against random ordering on 10 scenes, showing that depth-first ordering performs better. This experiment provides complementary evidence for the ordering strategy, although it is limited to a smaller subset due to API cost.

We additionally compare the HARMONY pipeline using GPT-5.5 against a GPT-5.5-only agentic baseline and a GPT-6 Astra baseline. These baselines use the same input images and evaluation set but are asked to generate the scene without using HARMONY's hierarchical reasoning, geometry grounding and verification pipeline.

Geometric metrics on 100 3D-FRONT scenes

MethodCD ↓[email protected] ↑[email protected] ↑[email protected] ↑
HARMONY0.04989.8219.470.130
GPT-6 Astra0.06177.4414.040.090
GPT-5.5-only0.060876.211.840.056

Perceptual metrics on 100 3D-FRONT scenes

MethodN-CLIP ↓LPIPS ↓PL ↓
HARMONY0.09290.51070.0524
GPT-6 Astra0.09540.52830.0535
GPT-5.5-only0.1800.6530.117

These comparisons show that using a stronger VLM alone does not account for HARMONY's improvements: the GPT-5.5-only and GPT-6 Astra baselines have similar geometric performance, while the HARMONY pipeline achieves better results.

AuthorsOpenReview ↗

Evaluation Protocols and User-Study Reliability

2. Difference between the ablation and main evaluation sets

The ablation study uses a randomly selected subset of 10 medium- and hard-difficulty scenes from HARMONY300, whereas the main evaluation uses the larger evaluation set. The full-model N-CLIP and PL values in the ablation table are therefore computed over these 10 scenes, rather than the full main evaluation set. We will clarify the subset and evaluation protocol in the revised paper to avoid direct comparison between these values.

3. 3D-FRONT alignment and F-score thresholds

All methods follow the same geometric evaluation pipeline, using Gen3DSR's Chamfer distance and F-score implementation with 30,000 surface samples per scene. Since the methods do not produce reconstructions in the 3D-FRONT world coordinate frame, we center and normalize both predicted and GT point clouds so that their longest bounding-box dimension equals one, and then rigidly align the prediction to the GT using ICP (30 iterations, with the best of four PCA-axis initializations). Scale is fixed during alignment.

The F-score thresholds of 0.1, 0.01, and 0.001 are therefore expressed in normalized scene units, corresponding approximately to 40 cm, 4 cm, and 4 mm, respectively, for a typical scene. This normalization removes global uniform scale differences, meaning that CD and F-scores evaluate normalized geometric fidelity rather than absolute metric-scale accuracy. We separately report median object-height and scene-scale estimation errors of 15.2% and 10.5%, respectively, to characterize metric-scale accuracy.

4. User-study scene selection

For the initial user study, we randomly shuffled and selected 20 HARMONY300 scenes for which all methods included in that study produced usable renders. We excluded cases where a method produced no output, a blank output, or background meshes that occluded the scene, since these failures prevent a meaningful visual preference comparison.

We acknowledge that this selection conditions the user study on successful generation and therefore does not measure end-to-end generation reliability. The initial study measures visual preference conditional on successful generation, rather than end-to-end system reliability. We separately report generation success rates across the complete evaluation set to characterize this limitation.

We plan to conduct an expanded user study on 20 scenes covering all evaluated baselines, including GPT-6 Astra, GPT-5.5-only, MIDI, and SceneGen. This study has not finished its new data collection, and its preference results will be included in the revised manuscript.

Here we report the success rates over all 300 HARMONY300 scenes: whether each method produced a render, and whether that render is blank. Missing-output failures occur only on real photographs (Gen3DSR: 26/200; 3D-Re-Gen: 18/200). A small number of blank renders also occur among the other methods, as reported in the table. Conditioning the user study on successful renders may favor methods with lower generation success rates, since their failed cases are excluded from the preference comparison. We therefore report generation success separately and interpret the user-study results as conditional visual preference, not end-to-end performance.

MethodProduced outputBlank renderUsable3D-FRONT (100)Real photos (200)
HARMONY3000300 (100%)100200
GPT-6 Astra3000300 (100%)100200
CAST3001299 (99.7%)99200
VIGA3001299 (99.7%)100199
AuthorsOpenReview ↗

Physical plausibility and hidden geometry evaluations

5. Additional 3D Reconstruction and Physical Plausibility Evaluations

Object-level pose and scale. We evaluate object recovery, center localization, orientation, and footprint accuracy against 3D-FRONT ground truth in the input-camera metric frame. The results are reported in the controlled comparison above.

Physical plausibility. We evaluate object support and inter-object collisions across 100 3D-FRONT scenes containing 531 objects placed by HARMONY, excluding the floor, walls, and ceiling.

To assess inter-object collisions, we voxelize each mesh at 2 cm and count two objects as colliding if their intersection exceeds 5% of the smaller object's volume. Of the 531 placed objects, 42 (7.9%) collide with at least one other object; these collisions occur in 19 of the 100 scenes and involve 23 unique object pairs. Counting any voxel contact, including objects that merely touch, 89 objects (16.8%) are involved.

We additionally evaluate object support among the reconstructed objects. An object is considered supported if its base lies within 3 cm of the floor, or within 3 cm of the top surface of another object directly beneath it, with overlapping footprints; otherwise, it is classified as floating. By this definition, 96.4% of evaluated objects are supported, while 3.6% (19 objects) are floating.

Hidden geometry. Input-view metrics cannot assess geometry the camera does not see, so we evaluate it directly against 3D ground truth on the 100 3D-FRONT scenes. The visible/hidden partition is defined on the GT surface; each predicted point inherits the label of its nearest GT point.

Visibility split. We sample 30k points on the GT object surface and project each into the input camera. A point is visible if it lies within 5 cm of the GT depth map at its pixel, and hidden otherwise (e.g., because it is occluded by other objects or lies on the back or underside of objects). On average, 41% of the object surface is visible and 59% is hidden.

Evaluation. Each prediction is first aligned to the full GT scene using the main-table protocol (per-scene normalization and rigid ICP), so alignment never uses the hidden/visible split. We then evaluate only the hidden region using two complementary measures:

  • Completeness: distances from hidden GT points to their nearest predicted points.
  • Accuracy: distances from predicted points whose nearest GT correspondences are hidden to those GT points.

Hidden CD averages the two directional distances, and hidden F-scores combine precision and recall at the main-table thresholds. This evaluation rewards predictions that recover the occluded GT surface while penalizing geometrically inaccurate predicted surfaces corresponding to hidden GT regions.

All methods are evaluated on foreground object surfaces only, excluding room-shell geometry, to ensure a consistent comparison across methods that do not reconstruct the room. Hidden CD and F-scores use the same normalized scene units and thresholds as the main geometric evaluation.

MethodHidden CD ↓[email protected] ↑[email protected] ↑[email protected] ↑
HARMONY0.038491.8624.200.177
GPT-6 Astra0.040190.8123.160.138
CAST0.040889.7623.670.175
SceneGen0.043888.4620.180.113
Gen3DSR0.047882.9918.360.073
3D-Re-Gen0.048185.1318.930.138
SAM3D0.048784.8121.230.127
MIDI0.049285.5117.690.113

HARMONY achieves the best hidden-geometry CD and F-scores among the evaluated methods. This provides a direct 3D-ground-truth-based assessment of occluded geometry, complementing the input-view perceptual evaluation.

We thank the reviewer again for these suggestions. We hope the rebuttal adderss the reviewer's questions and we are happy to answer further questions.

Reviewer_LWfVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HARMONY, a training free framework for reconstructing a compositional 3D indoor scene from a single image. The authors claim two main contributions. First, they frame the problem as a hierarchical chain-of-thought process in which a VLM grounds the scene spatially and then places objects in stages, with VLM feedback loops to limit error accumulation. Second, they combine this VLM reasoning about semantic and spatial relations with visual geometry models, using silhouette and VGGT cues to refine each object’s position and scale.

Critical strengths

S1. The method combines VLM semantic reasoning with metric geometry refinement in a concrete, well-motivated way, and achieves the best geometric results on Front3D, where ground truth is available.

S2. The paper is unusually transparent and reproducible for a system paper, releasing prompts, hyperparameters, runtime, failure cases, and a benchmark.

Critical weaknesses

W1. GPT-6 Astra is presented as a key comparison in the abstract and contributions, but the paper gives no explanation of how its results were obtained, and it is excluded from the user study.

W2. MIDI and SceneGen, which are discussed in the related work as closely related single-image scene methods, are not compared in the experiments.

W3. It is unclear whether the gains come from the proposed hierarchical reasoning or from the strong components (image-to-3D generator, VLM). The ablation differences for placement order and depth first traversal are very small, there is no baseline that places all objects at once.

W4. Object orientation and overall metric scale depend on VLM estimates (yaw selection, room height prior), but the paper provides no quantitative analysis of their errors or of their effect on the final reconstruction.

Questions for authors

Q1 (W1). How were the Astra results obtained, and why was it excluded from the user study? Could you also provide a gpt-5.5-only baseline, the VLM used inside HARMONY?

Q2 (W2). Is there a reason MIDI and SceneGen were not compared? If feasible, could you provide a comparison on Front3D?

Q3 (W3). Could you add a baseline that uses the same meshes and VLM but places all objects at once, and report the variance across repeated runs for Table 3?

Q4 (W4). Could you report the object yaw error against ground truth on Front3D, and the sensitivity of scale and Chamfer Distance to room-height estimation error?

(Minor). LPIPS is listed as an evaluation metric in Section 4.1 but is not reported in Table 2. Could you provide these results?

4 comments
AuthorsOpenReview ↗

Baselines: Astra, GPT-5.5-only, MIDI, and SceneGen

We thank the reviewer LWfV for the constructive feedback and for highlighting the need for additional baselines and quantitative analyses. We address each concern below.

Q1. GPT-6 Astra and GPT-5.5-only baselines (W1)

We run the gpt6-Astra baseline using Codex harness with gpt6-Astra Ultra effort on the same input image as ours. We excluded gpt6-Astra from the initial user study because of the additional cost and time required to generate and evaluate its outputs. We plan to conduct an expanded user study on 20 HARMONY300 scenes, including all evaluated baselines such as gpt-6 Astra, gpt-5.5-only, MIDI, SceneGen, and the other comparison methods. This expanded study has not finished its data collection, so updated preference results will be later reported. We will include the results in the revised manuscript.

Both agentic baselines, gpt6-Astra and gpt-5.5-only, use the same Codex harness and input images, with gpt-5.5 or gpt-6 Astra as the underlying model. They do not use HARMONY's reconstructed geometry, hierarchical placement procedure, or per-object verification and refinement. We report the gpt-5.5-only baseline to compare the geometric and perceptual results against gpt6-Astra baseline and gpt-5.5 with HARMONY pipeline below:

Geometric metrics on 100 3D-FRONT scenes

MethodCD ↓[email protected] ↑[email protected] ↑[email protected] ↑
HARMONY0.04989.8219.470.130
GPT-6 Astra0.06177.4414.040.090
GPT-5.5-only0.060876.211.840.056

Perceptual metrics on 100 3D-FRONT scenes

MethodN-CLIP ↓LPIPS ↓PL ↓
HARMONY0.09290.51070.0524
GPT-6 Astra0.09540.52830.0535
GPT-5.5-only0.1800.6530.117

In the above metrics, HARMONY outperforms both baselines on geometric and perceptual measures. gpt-5.5-only and gpt6-Astra reach similar geometry (CD 0.0608 vs 0.0610), so a stronger VLM alone does not close the gap to HARMONY (0.049). This comparison separates HARMONY’s reasoning from its components not by itself, and the controlled all-at-once baseline in Q3 provides further clarifications.

Q2. MIDI & SceneGen evaluation results on Front3D

We add comparisons with MIDI and SceneGen on 100 3D-FRONT scenes. Since both methods generate foreground furniture without reconstructing the room, we use HARMONY's room geometry as the background when rendering their outputs for perceptual evaluation. The geometric comparison is only performed on the foreground furniture because the ground truth geometry from Front3D doesn't contain room-shell.

Geometric metrics

MethodCD ↓[email protected] ↑[email protected] ↑[email protected] ↑
HARMONY0.04989.8219.470.130
MIDI0.057682.111.50.068
SceneGen0.049886.5714.260.080

Perceptual metrics

MethodN-CLIP ↓LPIPS ↓PL ↓
HARMONY0.09290.51070.0524
MIDI0.1680.5920.096
SceneGen0.1590.6120.094

HARMONY achieves the best results across all reported perceptual metrics. Geometrically, it outperforms MIDI on all metrics and achieves a similar CD but higher F-scores than SceneGen. Note that MIDI and SceneGen is trained Front3D while our method is general. These results indicate better foreground geometry and visual quality, particularly in texture and structural consistency.

AuthorsOpenReview ↗

Controlled placement comparison and repeated runs

Q3. All-at-once placement baseline and repeated-run variance

We add an all-at-once baseline that uses the same meshes, camera, room, and VLM as HARMONY, but places every object in a single placement call without per-object verification or refinement.

At the scene level, normalized, ICP-aligned CD is similar (HARMONY: 0.049; all-at-once: 0.051). Since this metric is less sensitive to individual object-placement errors, we additionally evaluate object-level accuracy in each camera's metric frame, without rescaling or alignment.

Object-level evaluation on 100 3D-FRONT scenes (409 GT objects)

MetricHARMONYAll-at-once
GT objects matched within 0.5 m (out of 409)183 (44.7%)164 (40.2%)
Matched GT objects (out of 409)324304
Median center error (matched)0.38 m0.47 m
Yaw within 15° (matched)81.7%75.9%
Median footprint error (matched)13.7%18.1%

For each scene, GT and predicted furniture objects are matched one-to-one using Hungarian assignment on Euclidean 3D center distance, constrained to the same category (bed, sofa, chair, table, cabinet, or lamp). Matches with distances greater than 1.5 m are rejected. Unmatched GT objects count as misses when computing object recovery (denominator: all 409 GT objects). Unmatched predictions, including extra or spurious objects, are not penalized. Center, yaw, and footprint errors are computed over matched pairs only.

With identical reconstruction components, HARMONY achieves better object recovery and footprint accuracy. The improvement in yaw accuracy is smaller (81.7% vs. 75.9%). These results demonstrate that HARMONY's combined sequential placement, per-object verification, and geometric refinement procedure improves object recovery and footprint accuracy over single-call placement. This experiment does not isolate the contribution of traversal order alone.

Repeated-run evaluation on the same 30-scene subset (124 GT objects)

MetricHARMONY (mean ± std)All-at-once (mean ± std)
GT objects matched97 ± 093.3 ± 1.2
GT objects within 0.5 m (out of 124)46.8 ± 0.8%40.9 ± 2.3%
Median center error (matched)0.40 ± 0.00 m0.46 ± 0.02 m
Yaw within 15° (matched)78.7 ± 1.6%69.7 ± 2.4%
Median footprint error (matched)12.4 ± 0.0%15.3 ± 2.1%

We additionally conduct three independent runs of both methods on the same 30-scene subset of the 100 Front3D scenes to quantify generation variability in the object-level metrics above. The reported ± values denote standard deviations across three run-level aggregate metrics, with each run evaluated on the same 30 scenes.

AuthorsOpenReview ↗

Yaw, CD and scale sensitivity

Q4. Yaw error on 3d gt and room height estimation error

We show the comparison of error rate for HARMONY’s yaw, scene scale and object height estimations compared to the 3D ground truth from Front3D dataset below:

1. Object yaw errors. Across the 100 3D-FRONT scenes, we evaluate yaw over the 324 matched furniture objects out of 409 GT objects, using each scene's camera frame and treating rectangular tables as 180° symmetric. HARMONY achieves a median yaw error of 1.5° (3.7° for chairs and 1.1° for sofas), with a mean of 14.6°. The gap between the median and mean reflects error distribution, with a small number of large errors, mostly on chairs. Among the 324 matched objects, 81.7% have yaw errors within 15° and 85.5% within 30°. Chairs account for 21 of the 38 objects with errors of at least 45°, likely because their appearance is often approximately symmetric across multiple orientations and they are frequently occluded. Additional geometric cues, such as top-down layout renderings, may help refine yaw after initial placement.

2. Sensitivity of CD to room scale and object-height error. HARMONY's median object-height and scene-scale estimation errors are 15.2% and 10.5%, respectively, against 3D-FRONT ground truth. Our standard Chamfer distance (CD) and F-score evaluation measures normalized geometric fidelity rather than absolute metric-scale accuracy. Specifically, we center each predicted and GT point cloud, normalize it by its largest bounding-box extent, and align it to the GT using rigid ICP. Thus, the F-score thresholds (0.1, 0.01, and 0.001) are expressed in normalized units, not meters. Because global scale is removed before alignment, these metrics cannot reflect absolute scene-scale discrepancies.

We distinguish this evaluation from our metric-frame object-level evaluation. HARMONY and the all-at-once baseline produce reconstructions in the input camera's metric frame. For these methods, we transform the ground-truth geometry into the same frame using the ground-truth camera extrinsics, allowing us to evaluate metric object positions, orientations, and dimensions. The other baselines do not produce metric-scale reconstructions in the input camera frame; consequently, we evaluate their geometric fidelity using the same normalization and ICP protocol described above. We therefore do not interpret normalized CD/F-scores as evidence of accurate absolute scale.

Sensitivity to the room-height prior. We scale the VLM-predicted room height by factors of 0.8, 0.9, 1.0, 1.1, and 1.2, and rerun the pipeline from camera calibration onward on 10 3D-FRONT scenes, keeping all other inputs and settings fixed. This is a separate 10-scene subset, so its default-condition error (6.1%) is not directly comparable to the 10.5% median scene-scale error over the full 100-scene evaluation.

Room-height perturbationScene-scale error ↓Object-height error ↓Center error ↓CD ↓[email protected] ↑Matched objects
−20%17.6%17.2%0.68 m0.057382.835
−10%10.3%15.3%0.43 m0.058482.735
0% (default)6.1%9.7%0.43 m0.058283.429
+10%16.4%14.0%0.61 m0.058784.128
+20%25.3%15.8%0.97 m0.057182.825

Metric-scale accuracy degrades as the room-height prior is perturbed: scene-scale error increases from 6.1% at the default estimate to 17.6% and 25.3% under −20% and +20% perturbations, respectively. Center error increases from 0.43 m to 0.68 m and 0.97 m. The number of matched objects also decreases under positive perturbations, from 29 at the default to 25 at +20%, indicating that larger scale errors can cause additional objects to fail the 1.5 m matching threshold.

In contrast, normalized CD (0.0571–0.0587) and [email protected] (82.7–84.1) remain relatively stable. This illustrates why normalized geometric metrics alone cannot characterize metric-scale accuracy. The results also show that the pipeline does not fully compensate for an inaccurate room-height prior: errors in this prior propagate into metric scale and object placement.

In our full evaluation of 100 3D-FRONT scenes, the VLM's room-height estimates have a median relative error of 4.8%, with 86.6% of scenes within 10% of ground truth.

AuthorsOpenReview ↗

Additional result: LPIPS

We report LPIPS on all methods evaluated on the 100 3D-FRONT cases.

MethodLPIPS ↓
HARMONY0.5107
GPT-6 Astra0.5283
SceneGen0.612
MIDI0.592
VIGA0.6575
GPT-5.5-only0.653
CAST0.5355
Gen3DSR0.4783
3D-Re-GEN0.5714
SAM3D0.567

HARMONY achieves the second-best LPIPS, behind Gen3DSR. LPIPS measures perceptual feature distances between spatially aligned image patches and may favor outputs that preserve local appearance and texture even when geometric fidelity differs. Although Gen3DSR obtains the lowest LPIPS, its outputs exhibit noticeable blurring (Fig. 3), illustrating that LPIPS alone does not fully characterize 3D reconstruction quality.

We thank the reviewer again for these suggestions. We hope the rebuttal adderss the reviewer's questions and we are happy to answer further questions.

Comments on the paper

2 comments

Dear Reviewers,

Thank you again for your constructive feedback and time spent reviewing our paper.

We have posted our detailed response and updated results addressing your concerns. As the discussion period has been several days, we would greatly appreciate it if you could let us know whether our response has adequately addressed your concerns, or if there are any remaining questions we could clarify.

Best regards, The Authors

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.