Don't Ask Agents to Do Everything: Selective Agentic Reasoning for Single Image to 3D Scene Generation

Recent agentic scene generation frameworks often rely on holistic pipelines that ask agents to resolve multiple entangled factors at once, which limits their performance in complex real-world environments. We propose a framework that generates a complete 3D scene from a single image in three stages, i.e., scene initialization, environment construction, and scene refinement. Our key idea, termed selective agentic reasoning, is to compute what can be measured geometrically and to leave agents mainly the decisions that remain ambiguous, most of which are simple accept-or-reject questions about candidates proposed from geometry. During initialization, an inspector agent segments and describes the objects in the image, and a reviewer agent decides which proposed mask merges and deletions to apply before each object is generated and posed. The scene is then brought to metric scale and enclosed by a room fitted to the observed walls, which anchors every later correction. Finally, the reviewer screens geometry-proposed corrections of the arrangement, and a group editor agent locally rearranges only the groups that remain wrong before they are merged back. The output is an complete 3D scene that stays faithful to the input image. To pose objects accurately, we further introduce a geometry-aware layout predictor supervised by the point map estimated from the input image, so it can also be trained on real-world images without 3D annotation. Experiments on synthetic and real-world benchmarks show that our framework achieves the highest perceptual quality among agent-based and generative approaches while ranking first or second on every physical metric, and that its layout predictor remains competitive in geometric accuracy.