acceptodds
Under review as a conference paper at ICLR 2027

Don't Ask Agents to Do Everything: Selective Agentic Reasoning for Single Image to 3D Scene Generation

Abstract

Recent agentic scene generation frameworks often rely on holistic pipelines that ask agents to resolve multiple entangled factors at once, which limits their performance in complex real-world environments. We propose a framework that generates a complete 3D scene from a single image in three stages, i.e., scene initialization, environment construction, and scene refinement. Our key idea, termed selective agentic reasoning, is to compute what can be measured geometrically and to leave agents mainly the decisions that remain ambiguous, most of which are simple accept-or-reject questions about candidates proposed from geometry. During initialization, an inspector agent segments and describes the objects in the image, and a reviewer agent decides which proposed mask merges and deletions to apply before each object is generated and posed. The scene is then brought to metric scale and enclosed by a room fitted to the observed walls, which anchors every later correction. Finally, the reviewer screens geometry-proposed corrections of the arrangement, and a group editor agent locally rearranges only the groups that remain wrong before they are merged back. The output is an complete 3D scene that stays faithful to the input image. To pose objects accurately, we further introduce a geometry-aware layout predictor supervised by the point map estimated from the input image, so it can also be trained on real-world images without 3D annotation. Experiments on synthetic and real-world benchmarks show that our framework achieves the highest perceptual quality among agent-based and generative approaches while ranking first or second on every physical metric, and that its layout predictor remains competitive in geometric accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image-to-3D Scene Generation, Multi-Agent Framework, Selective Agent Reasoning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_ESKEOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a three-stage framework, including scene initialization, environment construction, and scene refinement, for single image to 3D scene generation. At the same time, the paper proposes a geometry-aware layout predictor based on point map supervision to estimate the scale, rotation, and translation of objects.

Critical strengths
  1. The system design is quite comprehensive, integrating segmentation, 3D object generation, layout prediction, room construction, and scene refinement into an end-to-end scene generation framework.
  2. Converting the continuous geometric parameter prediction into the "geometry proposes, agent verifies" approach is relatively intuitive.
Critical weaknesses
  1. "The geometric module generates candidates + VLM makes acceptance/rejection decisions" is more akin to a reasonable system design principle rather than a new agent reasoning method, lacking innovation.
  2. The performance mainly stems from the combined effect of multiple modules such as SAM3, SAM3D, layout predictor, a large number of manually designed candidate rules, and deterministic physical finalization, etc. There is no fundamental method innovation.
Questions for authors
  1. The experiment extensively uses VLM. How can we ensure that the evaluation does not have model bias?
  2. Problems such as floating objects, collisions, and wall alignment ultimately still rely on fixed deterministic physical finalization. It is unclear whether the performance improvement comes from the agent's reasoning or from stronger geometric rules and post-processing.
  3. The method was only tested on 160 real indoor images. The main perceptual quality indicators were derived from the VLM scoring of Qwen3.5-9B and the CLIP similarity. There were no human ratings, and the reliability of the VLM scoring was unclear.
  4. The method is only applicable to indoor rooms and cannot directly handle outdoor, large city, multi-room or multi-level scenarios.
2 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely thank Reviewer ESKE for the review and for finding the "geometry proposes, agent verifies" conversion intuitive.

(W1) Our novelty is to identify the agent's weakness at 3D spatial reasoning as the core problem, and our framework facilitates this reasoning, whereas prior methods leave the agent to do it alone. Current VLMs often misjudge sizes, distances, and angles in 3D (SpatialVLM, VSI-Bench). Prior propose-then-verify methods offer no way around this weakness, as the agent still reasons in 3D when it writes the values, sets a solver's arguments, or picks among sampled layouts (Supp. Sec. G.4). Our framework instead measures everything it can from the photograph and its point map, and asks the agent only yes/no questions that measurement cannot decide, such as which masks belong to one object. Our three stages build this observed scaffold (Sec. 2), and the next response shows that the gain comes from facilitating the agent's 3D reasoning rather than from the modules.

(W2) The modules alone do not make a good scene, and neither does a VLM that writes the scene. Thank you for raising this. SAM3D alone is judged lowest in Table 1, and REST3D, which uses the same SAM3, SAM3D, and backbone with its own agent, is judged lower than ours and puts over ten times more objects through walls (Table 1). With our own modules, what matters is what the agent is asked, as writing the values or the whole plan fixes far fewer of the problems our checks flag and is judged lower.

What the agent is askedVLM↑Fixed↑
Write the values (Estimate)6.4338.6%
Plan everything at once (Propose)6.4957.5%
Accept or reject measured candidates (ours)6.7184.2%

Note that Fixed is the share of problems flagged by our geometric checks, such as floating or wall-penetrating objects, that no longer fail right after the agent's decisions, before any finalization. Applying every candidate without the agent's yes/no decisions keeps only about half of the objects (Table 5), and with our questions even the open 27B model scores above baselines run on Claude Opus 5.5 (Table S8). Details are in Table 7 and Sec. 4.2.

(Q1, Q3) Other judges and human raters also rank ours first.

160 scenesQwen↑GLM↑Sonnet↑User (%)↑
VIGA5.774.815.285.0
REST3D6.875.826.1123.4
Code-as-Room7.145.546.294.0
Ours7.416.216.5167.6

All three judges and the raters rank ours first (Table 3, Sec. 4.1). On 137 further test photographs (85 MIT-67, 52 COCO), ours is again judged highest, with the highest CLIP and the fewest floating and wall-penetrating objects (Table S15).

137 new photosQwen↑CLIP↑Float↓Coll.↓Wall↓Stable↑
REST3D6.30.67870.24.837.458.4
Code-as-Room6.79.68324.418.714.814.0
Ours7.07.73820.85.03.258.5
AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

(Q2) Even without finalization, ours already outperforms the other methods, and finalization changes our scenes only a little. Without finalization, ours is already judged above every baseline (Qwen 7.10 / 7.40 on MIT-67 / COCO in Table 5, 7.24 over all 160 scenes, against 7.14 for the best baseline in Table 3) and already ranks first or second on every physical metric against the baselines of Table 1 (Table S1). Finalization leaves about half of our objects within 0.2 cm and about two thirds within 5 cm, almost never rotates them, and raises our judge score (7.24 → 7.41). Applied to our Stage 2 instead, the same finalization lowers the score, so our advantage comes from the agent's decisions in Stage 3.

Change from finalization (ours)Value
Objects moved < 0.2 cm47%
Objects moved < 5 cm67%
Objects rotated more than 1°0.4%
Object pixels changed, median scene13%
Qwen of ours, without → with finalization7.24 → 7.41
Qwen of our Stage 2, without → with finalization7.00 → 6.79

Figure 7 and Figure S7 show our scenes without and with finalization from the camera of the photograph, where the two look similar. Furthermore, the result does not hinge on finely tuned constants, as it comes from the agent's facilitated 3D reasoning (W2). When each shared finalization constant, such as the 3 cm distance at which an object is attached to a wall, is halved or doubled, every physical metric stays within about one point of ours, and constants chosen per scene by our agent do worse (Table S7). We discuss this in Supp. Sec. G.6.

(Q4) We agree that our framework covers a single indoor room, and we now state this scope as a limitation. Outdoor, city-scale, multi-room, and multi-level scenes are outside its scope, so we scope the title and abstract to single-image indoor scenes and state these limits in Sec. 4.2 (Limitations and future work). As a direction, more views could enter our framework as further measurements, since its geometry already measures what the input shows. Figure S6 shows a first step on one SAGE scene, where two more views refine the objects without any agent call.

We thank the reviewer again for the constructive comments, all of which are now in the revision.

Reviewer_nmryOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a three-stage framework (scene initialization, environment construction, scene refinement) for single-image 3D scene generation. The central idea, selective agentic reasoning, is to let geometric procedures measure what is measurable and enumerate candidate edits, while agents are restricted to accept-or-reject judgments against the photograph rather than emitting coordinates. They additionally introduce a geometry-aware layout predictor supervised by a point-map Chamfer loss, which can be trained on real images without 3D layout annotation.

Critical strengths
  1. The core argument is well posed and empirically supported: the w/o reviewer and w/o refinement candidates ablations (Table 3) show that neither pure geometry nor a free-form agent suffices, which is exactly the claim the paper makes.

  2. Supervising the layout predictor with a point-map geometry loss is a practical way around the scarcity of scene-level 3D annotation, and the trimmed one-sided Chamfer plus pseudo-box term (Eq. 1) is a sensible treatment of the "inflate along the viewing direction" degeneracy.

Critical weaknesses
  1. The main perceptual metric is a VLM score from Qwen3.5-9B while all agents in the framework are Qwen3.8-27B (Sec. 4). A judge from the same model family as the system under test is a questionable choice for the headline metric in Table 1, especially since the margin over Code-as-Room on COCO is only 0.15 (7.4733 vs. 7.3233). Please report a judge from a different family, and ideally a small human study, with variance over repeated scoring runs.

  2. CLIP is used inconsistently. In Sec. 4.1 it is evidence that Stage 1 "aligns with the input image better than all baselines," but in Sec. 4.2 the w/o mask candidates variant obtains the highest CLIP on both datasets (0.7711 / 0.7132 vs. 0.7649 / 0.7124 for the full model) and this is explained away as CLIP "rewarding appearance regardless of the layout." A metric cannot be load-bearing when favorable and dismissed when not.

  3. Related to the above, CLIP decreases from Stage 2 to Stage 3 on both datasets (MIT-67 0.7782→0.7649, COCO 0.7240→0.7124; Table 1). The refinement stage improves physical validity but makes renderings less similar to the photograph. This trade-off is central to a method whose stated goal is faithfulness to the input image, yet it is not discussed.

  4. The layout predictor's contribution is weaker than the framing suggests. In Table 2, SAM3D is better on 3D-FUTURE mIoU (0.5904 vs. 0.5820) and on all three SAGE metrics (CD 0.0038 vs. 0.0048, F-Score 0.9643 vs. 0.9511, mIoU 0.4801 vs. 0.4029); the proposed predictor wins cleanly only on ScanNet. The abstract's "remains competitive" is accurate, but Contribution 2 reads as a stronger claim. Note also that SAM3D is simultaneously the object generator inside Stage 1, so the comparison is partly against a component of the authors' own pipeline.

Questions for authors

No

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We are grateful to Reviewer nmry for the precise and thoughtful review of our paper.

(W1) Other judges and human raters also rank ours first. Thank you for this suggestion.

160 scenesQwen↑GLM↑Sonnet↑User (%)↑
VIGA5.774.815.285.0
REST3D6.875.826.1123.4
Code-as-Room7.145.546.294.0
Ours7.416.216.5167.6

All three judges and the raters rank ours first (Table 3, Sec. 4.1). Over two more runs of our pipeline, ours stays first with a judge-score standard deviation of 0.11 to 0.17 (Table S14), and Figure S9 shows scenes from the three runs.

(W2) We now read CLIP one way throughout, as appearance agreement with the photograph, and rest faithfulness claims on mask overlap, photometric loss, and point-map agreement, where ours is best on all four (Table 4). We agree that CLIP should not support a claim in one place and be set aside in another. We removed the Stage-1 CLIP claim from Sec. 4.1 and revised the explanation in Sec. 4.2. Read as appearance agreement, the higher CLIP of the variant without mask candidates is expected. Without filtering, it keeps fragmented and duplicated masks as objects, so its renders contain more of the photographed appearance. The same objects cause about three times as many collisions (15.7 / 13.9% vs. 5.3 / 5.7%) and fewer stable objects (34.7 / 37.0% vs. 56.4 / 62.0%), and the judges score it clearly lower (Table 5, Supp. Table S1). CLIP and the other metrics thus show one trade-off, between covering the photograph's appearance and building a valid scene.

Table 4mIoU*↑PL↓CD↓F-Score↑
VIGA.0938.11.0252.793
REST3D.1766.94.0280.743
Code-as-Room.1057.38.0291.793
Ours.1946.08.0244.812

(W3) Stage 3 gives up some pixel agreement with the photo for a physically valid scene. We agree that this trade-off needed discussion and that CLIP alone cannot settle it. Every image measure prefers Stage 2, CLIP as well as mask overlap and photometric loss, because they compare only the rendered pixels with the photo and ignore whether objects float or penetrate each other. Stage 3 moves floating objects onto their supports and walls and merges wrong masks, so its renders match the photo slightly less. In exchange, floating objects fall from about three quarters to about a fifth, collisions and wall penetrations drop by about 70 to 80%, stable objects rise from about a tenth to about three fifths, and the judge score rises over both datasets together (Table 1). We added this trade-off to Supp. Sec. G.2.

160 scenesVLM↑CLIP↑mIoU*↑PL↓Float↓Coll.↓Wall↓Stable↑
Stage 26.59.753.2525.8873.318.613.410.7
Stage 36.71.740.1946.2622.05.52.759.1

Values are over MIT-67 and COCO together. VLM is the mean of the three judges of Table 3. PL covers 155 scenes and mIoU* the 70 COCO scenes with panoptic masks.

(W4) We toned down Contribution 2, but our predictor stays competitive with SAM3D. Thank you for the careful review. We agree that Contribution 2 read stronger than intended. Our predictor reuses SAM3D's point encoder and replaces only its per-object poses, so Table 2 compares two ways of placing the same shapes. Per object, its geometric accuracy is close to SAM3D's rather than better, but from four views around each scene its layouts receive the highest judge score on all three datasets of Table 2 (GT-aligned), and they are also judged higher on our out-of-distribution rooms.

Qwen↑3D-FUTURE (four views)ScanNet (four views)SAGE (four views)OOD (15 views)
SAM3D6.202.364.634.32
Ours6.543.304.855.42

In the full pipeline it matters as well, as replacing it with SAM3D's poses lowers the judge score (Table 5) and those poses are judged significantly worse from side views (Supp. Sec. G.8). On 15 views of five out-of-distribution rooms, ours is also slightly better on average in CD and F-Score, while SAM3D has the higher 3D box mIoU (Table S9, Figure S8). We reworded Contribution 2 in Sec. 1 to "a geometry-aware layout predictor that places the SAM3D objects of a photograph jointly in one floor-aligned frame, can be trained on photographs without 3D layout annotation, and remains competitive per object with the strongest layout baseline."

We thank the reviewer again for these suggestions, which made our evaluation clearer and our claims more precise, all now in the revision.

The authors have addressed my concerns with the additional evaluations and clearer discussion of the results, so I raise my score to 4.

Thank you for your follow-up, your positive assessment, and for raising your score. We appreciate your constructive feedback, which helped us strengthen the paper.

Reviewer_6fcKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a multi-agent pipeline that turns a single indoor photograph into a complete Blender scene. Its central idea, "selective agentic reasoning," is to let geometric procedures measure the image and its point map and enumerate candidate edits, while a VLM reviewer mainly accepts or rejects each candidate and never outputs coordinates.

The pipeline has three stages. In scene initialization, an inspector agent lists object classes for SAM3 and describes each mask. The reviewer then rules on geometry-proposed mask merges, deletions, and shared-mesh groups, after which SAM3D generates meshes and a new layout predictor poses all objects jointly. In environment construction, the scene is brought to metric scale and enclosed by a room fitted to walls observed in the MoGe point map. In scene refinement, the reviewer rules on candidate groups and operations (e.g., attach to support or wall, unify scales), a group editor locally rearranges groups that still fail a geometric check, and a deterministic physical finalization resolves remaining collisions and floating objects.

The layout predictor regresses per-object rotation, translation, and scale plus a shared floor rotation. It is trained with a trimmed one-sided Chamfer loss and a pseudo-box loss against the estimated point map, so it can also use unannotated real images (COCO).

On 160 real images (MIT-67, COCO), the system obtains the best VLM score and ranks first or second on all physical metrics against VIGA, REST3D, Code-as-Room, and SAM3D. The layout predictor is competitive with SAM3D and better than joint generators on 3D-FUTURE, ScanNet, and SAGE. Ablations remove the mask candidates, the layout predictor, the refinement candidates, and the reviewer.

Critical strengths
  1. The design principle is clear and the ablations mostly support it. Asking the agent only to accept, reject, or redirect geometry-proposed edits, instead of producing coordinates, is a sensible division of labor given how unreliable VLMs are at metric reasoning. Both directions are tested: letting the agent write the refinement operations itself leaves more objects floating (27.4% vs. 20.2% on MIT-67), and applying all candidates without the reviewer drops the VLM score substantially (by 1.72 on COCO) and removes about half of the objects. Together these show that both halves of the design matter, which is a useful takeaway for agent-based 3D generation beyond this particular system.

  2. The system works on real, cluttered photographs and produces complete, usable scenes. It handles full rooms with walls, materials, and lighting, not just object sets, and it achieves physical validity on par with REST3D (which uses a physics simulator) while staying much closer to the input image. Room penetration is reduced to 2–3.5% vs. at least 12.6% for the baselines. Training the layout predictor with a point-map loss, so that it can use unannotated real images, is a practical idea. The paper is also well written and easy to follow.

Critical weaknesses
  1. The evaluation of complete scene generation is thin and relies mostly on automatic, potentially biased metrics. The main metric is a 0–10 score from Qwen3.5-9B, while all agents in the framework are Qwen3.8-27B, so self-preference within the same model family cannot be ruled out. There is no user study. The test set is small (160 images, with 75 on COCO), results come from a single run, and no variance is reported. Some margins are small, e.g., 7.47 vs. 7.35 for REST3D on COCO.

    More importantly, nothing directly measures geometric faithfulness to the input image at the scene level, which is the paper's main claim against the baselines. The only image-alignment metric, CLIP, actually decreases after Stage 3 refinement (MIT-67: 0.7782 to 0.7649; COCO: 0.7240 to 0.7124), and the variant without mask candidates has a higher CLIP than the full model. A metric such as error against the estimated point map, or reprojected mask IoU, would make the faithfulness claim much more convincing.

    On the physical side, part of the Stage 3 gain comes from deterministic physical finalization (about a third of the stability gain), which the baselines do not receive. Even after it, 20–24% of objects still float. Finally, the agent baselines were all run on Qwen3.8-27B, which may be weaker than the models they were designed for.

  2. The layout predictor, presented as the second main contribution, shows only marginal gains over SAM3D, on which it partly builds (it uses the SAM3D point encoder). It is worse than SAM3D on all three SAGE metrics and on 3D-FUTURE mIoU (0.5820 vs. 0.5904), and only slightly better on ScanNet. It is also trained on 3D-FRONT, ScanNet, and SAGE and then evaluated on the same datasets, so the comparison favors it. The ablation in Table 4 shows small effects: the geometry loss reduces ScanNet CD from 0.0557 to 0.0530, the floor-rotation head lowers F-Score, and adding unannotated COCO images reduces CD from 0.0519 to 0.0489. This is weak evidence for the claim that point-map supervision on real images is what makes the predictor accurate.

    More broadly, propose-then-verify with VLMs is already well established (FirePlace, PIVOT, LLM-Modulo), and the system relies on many hand-designed candidate rules and thresholds, so it is unclear how robust it is outside the tested settings. Runtime and cost are not reported. The related work also does not discuss monocular holistic 3D scene understanding (e.g., Total3DUnderstanding, Im3D), which jointly estimates camera/floor orientation, room layout, and object poses from a single image and is closely related to the floor-aligned layout predictor and the measure-then-decide design. More broadly, propose-then-verify with VLMs is already well established (FirePlace, PIVOT, LLM-Modulo), and the system relies on many hand-designed candidate rules and thresholds, so it is unclear how robust it is outside the tested settings. Runtime and cost are not reported. The related work also misses some closely related lines of work. Monocular holistic 3D scene understanding (e.g., Total3DUnderstanding, Im3D) jointly estimates camera/floor orientation, room layout, and object poses from a single image, which is directly relevant to the floor-aligned layout predictor. In addition, [1] shares a very similar idea of first perceiving the scene geometrically and then making layout decisions on top of it, also for single-image 3D layout. It should be cited and discussed, and the paper should clarify how selective agentic reasoning differs from it.

Perceive-then-Plan: Layout-as-Policy for Monocular 3D Scene Layout Estimation (NeurIPS 2026)

Questions for authors
  1. Could you report a metric of geometric faithfulness to the input image for the full scenes (e.g., error against the point map or reprojected mask IoU), and explain why CLIP decreases after Stage 3? A small user study, or a judge from a different model family than the agents, would also address possible self-preference bias.
  2. How do the baselines perform if the same deterministic physical finalization is applied to their outputs?
  3. For Table 2, are the test scenes disjoint from the training scenes of the layout predictor? How does it compare with SAM3D when both are evaluated zero-shot on a dataset neither was trained on?
  4. What are the runtime and number of agent calls per scene, compared with the baselines?
4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We sincerely thank Reviewer 6fcK for the careful and constructive review.

(W1-1) Other judges and human raters also rank ours first, and our results vary little across runs.

160 scenesQwen↑GLM↑Sonnet↑User (%)↑
VIGA5.774.815.285.0
REST3D6.875.826.1123.4
Code-as-Room7.145.546.294.0
Ours7.416.216.5167.6

All three judges and the raters rank ours first (Table 3, Sec. 4.1), and over two more runs ours stays first with a judge-score standard deviation of 0.11 to 0.17 (Table S14). Figure S9 shows scenes where all three runs score above both baselines.

(W1-2, Q1) Our scenes agree best with the photograph on all four faithfulness measures. We compared every method with the COCO panoptic masks, the photo pixels, and the photo's point map, from each method's own camera or a photo-aligned one (Sec. 4.1). Figure 5 shows examples.

mIoU*↑PL↓CD↓F-Score↑
VIGA.0938.11.0252.793
REST3D.1766.94.0280.743
Code-as-Room.1057.38.0291.793
Ours.1946.08.0244.812

We agree that CLIP alone is not a reliable measure of faithfulness, as it rewards covering more of the photograph. Without mask candidates, the scene covers more of the photo and CLIP rises, but its judge score drops, in Table 5. Stage 3 moves objects onto supports and walls and merges wrong masks, so it gives up some pixel agreement with the photo for a physically correct scene, and the judges prefer it over Stage 2 on average (Table 1).

(W1-3, Q2) Our lead does not come from finalization. Finalization improves the physical metrics of every method, but ours already scores above every baseline under both judges before it, and finalization lowers the judge scores of REST3D and VIGA. With the same finalization for every method, ours alone ranks first or second on all four physical metrics and scores highest under both judges (Table S6, Supp. Sec. G.5). Our judge score rises the most with finalization, and finalization moves the fewest of our objects, so our final scenes keep the layout the agent decided.

MethodQwen↑ before → afterMoved > 0.2 cmMoved > 2 cmMoved > 30 cm
VIGA5.77→4.7273.0%68.0%38.8%
REST3D6.87→6.6292.6%56.6%26.8%
Code-as-Room7.14→7.1963.8%40.7%16.9%
Ours (Stage 2)7.00→6.7996.1%92.3%53.7%
Ours7.24→7.4153.3%39.5%13.9%

Moves are over all objects that finalization keeps in the 160 scenes, and Ours (Stage 2) applies the same finalization before the agent refines the scene. The Float metric also flags resting objects with rounded bottoms, and fewer than 1% of ours truly hover (Supp. Sec. G.5).

(W1-4) Stronger backbones do not change the ranking. With Claude Opus 5.5 or Sonnet 5.5 in all three pipelines, ours stays first, and on the open 27B model it already scores above both baselines run on Opus (Table S8, Sec. 4.2).

Qwen↑ (20 scenes)REST3DCode-as-RoomOurs
Open 27B5.166.216.60
Claude Opus 5.55.546.106.50
Claude Sonnet 5.54.986.106.48

The per-scene cost, tokens, and GLM and physical scores are in Table S8.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

(W2-1) Our predictor is competitive with SAM3D per object and keeps objects upright on one shared floor. On the same meshes, its layouts are judged significantly better than SAM3D's on MIT-67 and COCO (Supp. Sec. G.8) and from four views around each scene they receive the higher judge score on all three Table 2 datasets, clearly on ScanNet (Table 2, Table S10). Against Table 6's 'w/o COCO' checkpoint, our predictor agrees better with the photo in almost every scene, most clearly on unusual rooms such as the game room in Figure S4, though the two also differ in training length. We reworded Contribution 2 in Sec. 1 to "a geometry-aware layout predictor that places the SAM3D objects of a photograph jointly in one floor-aligned frame, can be trained on photographs without 3D layout annotation, and remains competitive per object with the strongest layout baseline."

Same SAM3D meshesSAM3D posesOur predictor
Qwen↑ (four views), 3D-FUTURE / ScanNet / SAGE (Table 2)6.20 / 2.36 / 4.636.54 / 3.30 / 4.85

(Q3) Yes, the Table 2 test scenes are disjoint from training, and on rooms neither model was trained on, ours places objects about as accurately as SAM3D and its layouts are judged better. No 3D-FUTURE, ScanNet, or SAGE evaluation scene was used in training (Supp. Sec. D.3). For a zero-shot comparison, we built five out-of-distribution rooms from unseen Meshy assets and turned their renders into photoreal images, and evaluated both on 15 views with the ground-truth masks (Sec. 4.1).

OOD rooms, 15 viewsSAM3D posesOur predictor
Qwen↑4.325.42
Mean CD↓ / F-Score↑ / 3D box mIoU↑.0046 / .958 / .551.0038 / .964 / .522

The two are close in geometry, and Qwen scores our layouts higher in most views. Details are in Table S9, and Figure S8 shows selected views.

(W2-2, W2-4) Unlike prior methods, our framework facilitates the agent's 3D spatial reasoning, and we added the suggested related work. Thank you for pointing us to these works. We differentiate our framework from propose-then-verify methods and Perceive-then-Plan in terms of how the agent's 3D reasoning is facilitated. In prior methods, the model still writes the values or plans the layout itself (Supp. Sec. G.4). Ours computes every candidate with geometry and asks the agent only to accept or reject it, and asking for values or the whole plan instead fixes far fewer flagged problems (Table 7), while our shared constants barely change the results when halved or doubled (Table S7). Total3D and Im3D estimate camera, layout, and object poses with full 3D supervision, and our floor rotation plays the role of their camera pitch and roll without it (Sec. 2).

What the agent is askedVLM↑Fixed↑
Write the values (Estimate)6.4338.6%
Plan everything at once (Propose)6.4957.5%
Accept or reject measured candidates (ours)6.7184.2%

Note that Fixed is the share of problems flagged by our geometric checks that no longer fail right after the agent's decisions (Table 7).

(W2-3, Q4) Ours writes the fewest output tokens, since its agent answers yes/no with short reasoning, and it keeps its quality on a cheaper backbone.

MethodQwen↑Output tokens↓Minutes↓
REST3D6.8717.8k47
Code-as-Room7.14118.7k74
Ours7.4110.1k66

Qwen is over the 160 test scenes (Table 3), and tokens and minutes are per-scene means (Table S8), with the baselines' tokens from a metered re-run on 20 of these scenes. On Sonnet, ours still scores above both baselines run on Opus at less than half their cost (Table S8).

We thank the reviewer again for these valuable suggestions, all now in the revision.

Reviewer_6fcKOpenReview ↗

Response to authors

Thank you for the responses and additional experiments, which address several of my evaluation concerns. However, the methodological novelty remains limited, and the layout predictor’s gains over SAM3D are still mixed. The COCO ablation is also confounded by different training lengths, while Stage 3 improves physical validity at the cost of input faithfulness. I therefore maintain my original rating of 3 (Weak acceptance).

We sincerely thank Reviewer 6fcK for the follow-up and the positive assessment. We briefly clarify the remaining points.

(Novelty) As our first reply showed, our framework is a harness that facilitates the agent's 3D reasoning, which the role-of-agent comparison isolates (Table 7).

(Layout predictor) Per object our predictor is competitive with SAM3D, as stated before, and its contribution is at the scene level. Removing it lowers the judge score of the full pipeline from 7.35 / 7.47 to 6.98 / 6.99 on MIT-67 / COCO (Table 5).

(COCO ablation) Both runs use the same number of epochs, so the difference in training length is the added data itself.

(Stage 3) Stage 3 loses a little pixel overlap, but the scene becomes physically valid and more realistic.

160 scenesRealism↑Float↓
Stage 26.6673.3
Stage 37.1022.0

Realism is the mean of the two judges that scored every stage of our pipeline, Qwen and GLM. Ours remains the most faithful method after Stage 3 (Table 4).

We thank the reviewer again for the constructive suggestions.

Reviewer_oE49OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed an agentic scene generation framework consists of three stages. In scene initialization, an inspector agent identifies the objects and their attributes, the reviewer applies rules on the proposed mask candidates. Environment construction then scales the scene to real-world units and fits the room to the walls observed in the point map. In scene refinement, the reviewer rules on corrections proposed from the layout, the room, and the object attributes, and only the groups that still fail a geometric re-check are rearranged by a group editor agent before physical finalization. The authors argue that agents should mainly decide what still remains ambiguous after geometric verification.

Critical strengths
  1. The division of labor between geometric rules and vlm judge is sound where geometry enumerates candidate corrections and a vlm resolves the remaining semantic ambiguity by accepting or rejecting.
  2. The supplementary material provides extensive implementation details about the method and experimental setup
  3. Experiments show that the proposed framework achieves the highest perceptual quality among agent-based and generative approaches while ranking first or second on every physical metric, and that its layout predictor remains competitive in geometric accuracy
Critical weaknesses
  1. The main weakness is the limited novelty and the limited significance of the methodological contribution. Each component in the method overlaps substantially with existing propose-then-verify agents. The key idea of using geometric computation for precise measurements and reserving VLMs for semantic ambiguity is well motivated by the respective capabilities of these components. The paper presents a useful integration of them but does not sufficiently establish what new insight it adds.
  2. The experiments are not convincing enough. The results demonstrate the effectiveness of the complete pipeline, but provide a less clear explanation of when and why the proposed restriction of agent reasoning is preferable. Comparison to coding agents only includes one coding agent and two visual examples without quantitative evaluation.
  3. A large portion of the framework is rule-based and seems brittle, e.g. deterministic leveling, support completion, collision resolution, lowering hovering objects, and wall alignment. These are useful heuristics but provide little scientific insight.
  4. It is unclear how generalizable the proposed framework is. As acknowledged by the author, the method targets indoor scenes with a single room with flat walls on one floor. Considering its largely rule-based nature, it may not generalize to more diverse cases. There are counterexamples for the rules used in the framework. For example, tilted objects are leveled, but there exist deliberately tilted objects such as a mirror leaning against a wall; a hovering object is lowered onto the first support, but it could be a hanging lamp in the input image.
  5. The title and paper framing suggests a general principle about agent reasoning, while the actual implementation is a carefully engineered indoor-scene pipeline
Questions for authors

Are the rules hand-designed? Would it make sense to make agents come up with scene-specific rules?

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/2)

We thank Reviewer oE49 for the review.

(W1) Our novelty is to identify the agent's weakness at 3D spatial reasoning as the core problem, and our framework facilitates this reasoning, whereas prior methods leave the agent to do it alone. Current VLMs often misjudge sizes, distances, and angles in 3D (SpatialVLM, VSI-Bench). Prior propose-then-verify methods offer no way around this weakness, as the agent still reasons in 3D when it writes the values, sets a solver's arguments, or picks among sampled layouts (Supp. Sec. G.4). Our framework instead measures everything it can from the photograph and its point map, and asks the agent only yes/no questions that measurement cannot decide, such as which object supports another. Our three stages build this observed scaffold, i.e., clean masks, a floor-aligned layout, and a room fitted to the observed walls. We added this discussion to Sec. 2, and the experiments in the next response show what happens when the agent must do the 3D reasoning itself.

(W2-1) We explain when and why facilitating the agent is preferable. It is preferable when the agent would otherwise have to write numbers or plan the whole scene, because it judges measured candidates far more reliably than it estimates values. We kept our pipeline and changed only what the agent is asked. Note that Fixed is the share of problems flagged by our geometric checks, such as floating or wall-penetrating objects, that no longer fail right after the agent's decisions, before any finalization.

What the agent is askedVLM↑Fixed↑
Write the values (Estimate)6.4338.6%
Plan everything at once (Propose)6.4957.5%
Accept or reject measured candidates (ours)6.7184.2%

With our questions, even the open 27B model scores above both baselines run on Claude Opus 5.5 (Table S8). Details are in Table 7 and Sec. 4.2 (Role of the agent).

(W2-2) A frontier coding agent ties our open 27B pipeline in judge score but leaves its wall fixtures loose. Claude Opus 5.5 builds windows and lights as free objects that penetrate walls and fall, whereas ours fits them to the room. When it edits our scene instead, it scores highest, but the objects it adds again float or penetrate walls, so the two are complementary. The coding agents VIGA and Code-as-Room score below ours in Table 1.

Five scenesVLM↑Float↓Wall↓Stable↑
Opus from scratch7.3437.321.432.5
Ours7.4026.44.247.2
Opus editing ours7.6546.322.133.7

Details are in Table 8, Table S12, and Sec. 4.2.

AuthorsOpenReview ↗

Rebuttal by Authors (2/2)

(W3) Our scientific insight is that facilitating the agent's 3D reasoning with measured candidates makes it decide far more accurately than when it predicts the values itself. Thank you for this important point. We agree that each rule on its own is a simple heuristic. The rules supply the measurements, and the same agent fixes more than twice as many problems when it judges them than when it writes the values (W2-1). The rules also need the agent, as applying every candidate without it keeps only about half of the objects (Table 5). A rule therefore only has to propose a plausible correction, which is why the same rules serve all room and object types (W4-1) and a failing rule can be met with one more question (W4-2). Finally, the rules are not brittle, as halving or doubling each shared constant barely changes any physical metric (Table S7).

Table 5 (Qwen↑)MIT-67COCO
w/o reviewer (rules only)6.585.76
w/o mask cand.6.306.82
w/o ref. cand.6.657.19
Ours7.357.47
Table S7Float↓Coll.↓Wall↓Stable↑
Shared constants (ours)22.05.52.759.1
Each constant ×0.5 or ×221.5–22.74.7–5.92.3–3.458.2–60.1

(W4-1) The rules generalize across indoor scenes and object types. The same constants serve all 160 test scenes, and ours has the highest judge score in 12 of 17 room types and the lowest wall penetration in all 17 (Table S11). The rules alone are right for 101 of the 116 objects most likely to break them, one more question raises this to 107, and all 41 hanging lights stay in place, as the table in W4-2 lists.

(W4-2) Counterexamples are few, and one selective question fixes almost all hanging ones without harming correct cases. We checked 116 objects in the photographs, 9 tilted, 24 hanging, and 83 upright. Our rules failed on 15 of them, all tilted or hanging. Asking our agent post hoc one more yes/no question, whether such an object hangs from above, is mounted on the wall, or leans against it, turned 5 of the 6 hanging failures and the leaning mirror into successes, where we stand the mirror on the floor since its table was missed in Stage 1, and changed none of the correctly handled objects we checked (Figure S3). The remaining failures are mostly books leaning on other books, which we list as a limitation in Sec. 4.2.

Object typeCheckedOurs failedFixed by one question
Tilted991
Hanging2465
Upright830–
All116156

All 41 hanging lights also stay in place.

(W5) We agree that our target is single-image indoor scenes, so we changed the subtitle to "Selective Agentic Reasoning for Single-Image 3D Indoor Scene Generation" and scoped the abstract, the introduction, and the limitations in Sec. 4.2 to this setting.

(Q1) Yes, the rules are hand-designed, and the agent decides where they apply. Room-specific rules are a promising next step. Thank you for the question and the suggestion. We tried this for the finalization constants, and constants chosen per scene by our agent do worse than the shared ones (Table S7). As in W4-2, one case-specific question already fixes almost all hanging failures, and adding such questions, or rules for rooms with special layouts or objects, is our next step.

We thank the reviewer again for these comments, all addressed in the revision.

Thank you again for your review. We would greatly appreciate it if you could let us know whether our responses have addressed your concerns.

In brief, we (1) clarified our novelty as facilitating the agent's 3D spatial reasoning, (2) added a role-of-agent comparison showing that accept-or-reject decisions fix 84.2% of flagged problems, against 38.6% and 57.5% when the agent writes values or plans the whole scene (Table 7), along with a quantitative comparison with a frontier coding agent (Table 8), (3) showed that the rules are robust to their constants (Table S7) and that one selective question fixes most counterexamples such as hanging objects (Figure S3), and (4) rescoped the title and paper to single-image indoor scenes.

If any concerns remain, we would be glad to address them.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General Response by Authors

We appreciate the reviewers' valuable comments. We added experiments and analyses to the revised manuscript, where all changes are marked in blue. The major updates are below.

Novelty. We clarified that our framework facilitates the agent's 3D spatial reasoning. With the same pipeline, accepting or rejecting measured candidates fixes 84.2% of flagged problems, against 38.6% for writing values and 57.5% for planning the whole scene (Table 7).

Evaluation. We added two judges from other model families, a user study, run-to-run variance. Ours ranks first in all of them.

Image faithfulness. We measured agreement with the photograph by mask overlap, photometric loss, and Chamfer distance and F-Score against its point map. Ours is best on all four (Table 4).

Detailed responses to each reviewer are below.

Dear Reviewers,

The authors have submitted their rebuttal to address your concerns. As the discussion deadline approaches, please review their response and share any remaining concerns or questions soon so the authors have enough time to respond.

Thank you, -AC

AuthorsOpenReview ↗

Summary of the Discussion Stage

Summary of the Discussion Stage

We would like to summarize this discussion stage so that it can be followed easily. We thank you and the reviewers for the discussion, and below we list the main concerns, from the first reviews to the follow-ups, with our answer to each.

Concern 1 (oE49, 6fcK, ESKE): The novelty over propose-then-verify methods is limited.

Answer: Propose-then-verify methods still ask the VLM to infer specific values by itself, although VLMs find it hard to understand the whole scene and its spatial information. We instead turn each hard reasoning step into a question and ask the VLM only for an easy yes/no answer. Note that for a VLM that struggles with 3D reasoning, this easy form of reasoning raises decision accuracy substantially, from 67.9 to 86.2 for the same agent (Supp. G.4).

Concern 2 (6fcK, nmry, ESKE): The evaluation uses a same-family judge, no user study, a small test set, and a single run.

Answer: Two judges from other model families, a user study, and two more runs all rank ours first (Tables 3 and S14).

160 scenesQwenGLMSonnetUsers (%)
Best baseline7.145.826.2923.4
Ours7.416.216.5167.6

Concern 3 (6fcK, nmry): Faithfulness to the photograph is not measured, and CLIP drops after Stage 3.

Answer: Ours is the best on mask overlap, photometric loss, and Chamfer distance and F-Score against the point map (Table 4), so Stage 3 trades only a little pixel agreement for a physically valid scene.

Concern 4 (6fcK, nmry): The layout predictor gains little over SAM3D.

Answer: We reworded Contribution 2 to competitive per object, and removing the predictor lowers the full pipeline on both test sets (Table 5), so its contribution is at the scene level.

Concern 5 (6fcK, ESKE): The gain may come from the physical finalization.

Answer: Before finalization ours already scores above every baseline's final output, and with the same finalization for all methods it stays the highest (Table S6).

Concern 6 (oE49, ESKE): The gain may come from the geometry or the modules, not from the agent.

Answer: Part of the gain does come from the geometry, but the geometry alone is not enough. When the agent is given the measurements and asked to work out the answer by itself, it often fails to reach the correct one. Its accuracy rises only when it receives the candidates together with the measurements and rules on them one at a time.

Decision accuracyPlans aloneRules on candidates
w/o measurements67.979.3
w/ measurements73.386.2

We also answered smaller concerns on the robustness of the hand-set constants (Table S7), stronger backbones and a frontier coding agent (Tables S8 and 8), and the single-room scope, which the title, abstract, and limitations now state.

We thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.