acceptodds
Under review as a conference paper at ICLR 2027

Lived-In by Design: Behavior-Grounded 3D Scene Generation via Functional-Group Assembly

Abstract

We present Lived-In by Design, a framework for behavior-grounded agentic 3D scene generation by carrying everyday-activity context from prompt interpretation to room-scale layout. Unlike existing methods that construct scenes primarily from object inventories and pairwise spatial relations, our method models the functional dependencies and human-use constraints that organize objects in inhabited environments. Specifically, we first design a behavior-to-asset stage to convert plausible activities generated by agent into object requirements, quantities, shared resources, and functional relations. To preserve these relations during construction, we further introduce functional-group assembly, which organizes related objects into behavior-dependent local configurations. Finally, we design a lived-in scene layout stage to place these groups jointly at room scale while considering architectural constraints, accessibility, and human usage space. Evaluations on 179 room-level prompts using automatic metrics, qualitative comparisons, controlled ablations, and human judgments demonstrate improved functional organization and lived-in plausibility.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Agentic Scene Generation, Behavior-Grounded Generation, Scene Layout

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jhbTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Lived-In by Design, a framework for text-conditioned 3D indoor scene generation that carries inferred everyday activities through scene specification, local assembly, and room-level layout. A behavior-to-asset stage converts plausible activities into object requirements and functional relations. Related objects are assembled into functional groups with behavior-dependent orientations and approximate human-use regions. A group-level search then places these assemblies while considering physical constraints, architectural relations, and access, with constrained local revision when necessary. The system builds on SceneSmith for room construction, asset acquisition, rendering, and physical post-processing. On 179 room-level prompts, the method reports higher attribute and object–object relation scores than the compared baselines, while trailing SceneSmith on count satisfaction, object–architecture relations, and collisions. A blinded five-way study with 25 participants on five selected scenes reports 82.4% preference for lived-in plausibility. The appendix also reports two component removals, although these lack a full-method control evaluated on the same subset.

Critical strengths
  1. A coherent representation connecting activities, local organization, and room layout. The method addresses a relevant gap between placing plausible objects and arranging them for use. Its activity-derived requirements, persistent functional groups, support bindings, and required/preferred use regions form a concrete construction pipeline (Sections 3.2–3.4). Preserving local organization while allowing constrained revision is a sensible way to reconcile functional arrangements with room geometry.

  2. Promising evidence for improved semantic organization and perceived plausibility. The 179-prompt comparison includes a strong underlying baseline, SceneSmith. ATR increases from 74.4 to 88.9 and OOR from 67.6 to 75.2, despite a lower average object count. The blinded human study also shows a substantial preference for the proposed method on all five evaluated scenes. These are meaningful positive results, even though they do not establish improvements across every dimension or across the full prompt distribution.

Critical weaknesses
  1. The experiments do not isolate the contribution of the central mechanisms. Appendix A.6/Table 3 compares two component removals on an unspecified subset with full-method results from all 179 prompts; the caption explicitly states that this is not a matched-subset control. Consequently, the reported OOR gap cannot be interpreted as a component effect. Neither behavior-to-asset grounding nor functional-group assembly is independently removed. Since the system builds on SceneSmith and related work already studies functional groups and human-aware constraints, controlled evidence for the particular integration is important. Missing model configurations and search budgets further limit attribution of the gains to the proposed design rather than execution differences.

  2. Evidence for broadly improved functional usability remains indirect and narrowly sampled. Table 1 reports semantic and general spatial metrics, but no object-accessibility score or aggregate validation of the proposed use-space/access diagnostics. NAV alone does not establish that the intended interaction sides are usable. Relative to SceneSmith, CNT drops from 82.9 to 66.8, OAR from 80.6 to 72.9, and COL rises from 1.2 to 7.0; these trade-offs matter for the intended application. The human study is positive and statistically analyzed, but covers only five selected scenes with an unspecified selection rule. Its participant-level intervals do not quantify generalization to new scenes, and common rendering settings across methods are not established. Thus, the current evidence supports perceived improvement on the selected examples more clearly than a general improvement in functional usability.

Questions for authors

1.How many prompts are in the ablation subset, which prompts were selected, and how does the full method perform on the same subset? Can you separately evaluate the contributions of behavior-to-asset grounding and functional-group assembly while keeping the backbone, asset-acquisition configuration, and computational budgets fixed?

2.Beyond NAV, what evidence shows that objects are easier to approach and use? Could you evaluate the existing scenes with SceneEval ACC or independently check whether the intended use regions are obstructed or inaccessible? Why does COL increase while CNT and OAR decrease relative to SceneSmith, and do the additional collisions mainly involve interactive furniture or small supported objects?

3.How were the five human-study scenes selected from the 179 prompts, and were they the same outputs used in the automated evaluation? Were rendering and scene-selection conditions comparable across methods, including the renderer, lighting, and resolution? Given the lower CNT and OAR scores in Table 1, what supports extending the strong human preference for prompt faithfulness beyond these five scenes?

4.Which reasoning and vision models and versions were used, and what were the prompt configurations and search/refinement budgets? How many scenes contributed to each metric, and how were generation or evaluation failures handled? Were baseline outputs regenerated or reused, and were all outputs assessed with the same evaluator version and configuration?

11 comments
AuthorsOpenReview ↗

Reply to Reviewer jhbT

Thank you for recognizing the connection between activities, local object arrangements, and room layout.

In response to your comments, we added a component comparison on the same 30 prompts and evaluated object accessibility and opening clearance. We also analyzed the trade-offs among count, relation, and collision scores. We clarify the differences in models and budgets, the evaluation rules, and the sources of the baseline outputs. The revision will report the user study results on 20 new scenes. We address the weaknesses and questions in turn below. We hope our responses address your concerns and look forward to your feedback.

W1. The experiments do not isolate the contributions of the core mechanisms. Differences in models and budgets also affect attribution.

We agree that the core components should be assessed in controlled experiments with clearly specified models and budgets.

We compared four configurations on the same 30 scenes. We separately removed behavior planning (NoBehavior) and functional-group assembly (NoGroupAssembly). Full scored 16.83 percentage points higher than NoBehavior on OAR. It scored 2.42 points higher on STB and 6.91 points higher on OPEN than NoGroupAssembly. These results are consistent with the respective roles of behavior planning in spatial organization and group assembly in preserving within-group structure.

Q1 gives the full results and sampling rules. Q4 gives the models and budgets. Table 3 in the revision will present these matched-prompt comparisons. The visualizations show how each component contributes to the resulting layouts.

W2. Evidence of functional usability is indirect, and a five-scene user study provides limited support for broad generalization.

We agree that functional usability and generalization across scenes require more direct evidence.

To assess local conditions for use directly, we added ACC and OPEN. Full scored 5.13 percentage points higher than NoHumanUse on ACC and 24.63 points higher on OPEN, as shown in Q2. These results support improved object accessibility and greater clearance at architectural openings.

We selected 20 additional scenes for the extended user study. Q3 gives the selection rules and rendering settings. The revision will report preference rates and confidence intervals that account for variation across scenes. Q2 also explains the trade-offs in CNT, OAR, and COL in the original table.

Q1. How many prompts were used for the ablations, and how were they selected? Can the core components be compared independently using the same models, assets, and budgets?

Thank you for this suggestion.

We compared Full and three ablations on the same 30 prompts. NoBehavior removes activity-derived planning. NoGroupAssembly removes local assembly followed by placement of each group as a unit. NoHumanUse removes feedback on human-use space.

We drew a stratified sample from SceneSmith's 179 room-level prompts using the difficulty levels and source categories in the original list. The random seed was 20261107. We selected 10 prompts at each of the three difficulty levels, covering 19 room types. The supplementary material gives the full prompt list, sampling quotas, and adjustments.

All four configurations use gpt-5.6-sol for both language and vision, Qwen-Image-2.1-Turbo to generate reference images, and SAM3D to generate 3D assets. The base budget for whole-room review is 3 rounds for Full and 5 for each ablation. The extra rounds allow the ablations to complete layout repair and export the scene after a component is removed.

The table summarizes the current results. All scores are percentages.

MethodCNT↑ATR↑OOR↑OAR↑ACC↑NAV↑STB↑COL↓SUP↑OOB↓OPEN↑
NoBehavior85.358̲1̲.̲7̲0̲79.3372.7081.5194.5092.2910.579̲0̲.̲5̲9̲0.5276.64
NoGroupAssembly8̲8̲.̲8̲4̲83.688̲2̲.̲6̲2̲85.8379.169̲4̲.̲9̲3̲9̲6̲.̲5̲3̲5.8789.030.468̲6̲.̲7̲1̲
NoHumanUse88.4878.8185.328̲7̲.̲9̲4̲74.7795.9591.175̲.̲8̲3̲84.860̲.̲4̲3̲68.99
Full89.5281.3978.9589.527̲9̲.̲9̲0̲94.1398.965.4491.340.2893.62

Behavior planning translates activity requirements into object choices and layout constraints. Full scores 16.83 percentage points higher than NoBehavior on OAR, supporting the role of behavior planning in guiding object–architecture relations. Full scores 2.42 points higher on STB and 6.91 points higher on OPEN than NoGroupAssembly. This supports the benefit of establishing local structure before placing each group as a unit, both for stability and for clearance at room openings. Full scores 5.13 points higher on ACC and 24.63 points higher on OPEN than NoHumanUse. Although NoHumanUse has higher NAV, its object accessibility and opening clearance are lower. This shows that human-use feedback adds local constraints beyond the floor-connectivity check.

The revision includes the corresponding ablation visualizations.

Reviewer_F8F9OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an agentic 3D indoor scene generation framework named Lived-In by Design that improves lived-in plausibility and functional organization by incorporating activity context. Experimentally, Lived-In by Design achieves comparable results in the automated comparison and the highest preference rates in the user study.

Critical strengths
  • Lived-in plausibility and usability are important for digital 3D scenes. This paper explicitly takes these aspects into consideration, helping generate scenes that better support everyday use and are closer to real-world scenes.
  • This paper is well written. The figures and explanations are clear and easy to understand.
Critical weaknesses
  • The paper provides insufficient implementation details. In particular, it is unclear how scene descriptions are used to infer persona-based activities and how these activities are converted into the behavior–asset graph and functional groups (Sec. 3.2). The paper also does not clearly explain how relative object positions within each functional group are determined and adjusted, or which model is used for the spatial profiler (Sec. 3.3 and Appendix A.3).
  • The user study includes only five representative scenes, which may not adequately reflect the framework’s performance across diverse scenes.
Questions for authors

Could you please provide more implementation details, including which models are used in the agentic framework, what the spatial profiler is and which model or algorithm it uses, and how object positions, orientations, and spatial relations are proposed and adjusted?

5 comments
AuthorsOpenReview ↗

Reply to Reviewer F8F9

Thank you for your positive assessment of the paper's motivation, figures, and presentation. In response to your suggestions, the revision traces a bookstore from prompt to layout and reports the expanded 20-room user study. We hope our responses address your concerns and look forward to your feedback.

W1: From the room description to the group layout

We agree that the generation steps should be explicit. Figure 10(a–c) (revised manuscript, p. 26) traces the bookstore example.

  1. Prompt and activities. The gpt-5.6-sol planner extracts objects, counts, and relations, then infers a persona and ordered actions. For the bookstore, it infers a curator who prepares displays and restocks.
  2. Graph and functional groups. The graph records explicit requirements and activity-derived additions, such as a placard on the display table. The planner proposes each template's anchor, member counts, and repetitions. The program expands the templates and checks the counts (Section 3.2 and Appendices A.2 and A.12, entries 1–4).
  3. Spatial profiler. A gpt-5.6-sol VLM identifies candidate interaction sides from five asset views. Activity relations guide side selection (Figure 3(b) (revised manuscript, p. 5) and Appendix A.3).
  4. Relative poses and refinement. Given asset dimensions and local relations, the planner proposes poses relative to the fixed anchor. Geometry checks collisions and support, and the visual critic guides orientation and spacing corrections (Section 3.3 and Appendix A.12, entries 7–9).

W2: User-study coverage across room types

We agree that the original five rooms covered a limited range of layouts. The expanded study uses 20 rooms across 19 types drawn from the 30 ablation prompts (7 easy, 6 medium, 7 hard). All 25 participants compared five anonymized layouts per prompt in randomized order and selected one per criterion, yielding 500 votes per criterion.

Table 2 (revised manuscript, p. 9) gives the five-way preference rates (%) and 95% bootstrap intervals from resampling participants and rooms.

MethodPrompt FaithfulnessRealismLived-in Plausibility
HSM5.0 [1.2, 10.4]3.4 [0.6, 7.2]4.6 [1.2, 9.6]
Holodeck5.8 [1.8, 10.8]7.4 [2.8, 13.2]8.4 [3.2, 14.8]
LayoutVLM (Obj.)3.0 [0.2, 7.6]2.0 [0.0, 5.0]2.0 [0.0, 5.6]
SceneSmith34.2 [22.8, 46.8]31.0 [20.4, 42.8]31.0 [20.2, 43.2]
Ours52.0 [38.0, 66.0]56.2 [43.0, 69.4]54.0 [40.4, 67.6]

Our method received the most votes on all three criteria. For Lived-in Plausibility, it ranks first outright in 14 rooms and ties for first in two, as reported in Table 6 (revised manuscript, pp. 23–24). Figures 11–12 (revised manuscript, pp. 27–28) show the study rooms, and Appendix A.8 explains their selection.

Reviewer_gAorOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a behavior grounded pipeline for text conditioned 3D indoor scenen generation. The method is built on Scenesmith and infers plausible activities and converts them onto objects requirements and functional relations. Then, assemble them onto local groups and places them in the scene considering human use space and accessibility.

Critical strengths
  1. This method, compared to previous 3D scene generation methods, taking consideration of everday activities and accesibility into the layour generation, beyond collision free only.
  2. Functional groups preserve local arrangements while allowing coherent room-level placement.
Critical weaknesses
  1. The ablations lack a matched control: component removals use an unspecified prompt subset, while the full-method row uses all 179 prompts. They therefore do not establish component contributions.
  2. Functional usability is not directly tested. Static clearance boxes and floor connectivity cannot establish that people can perform the inferred activities.
  3. Key configuration details are missing, including reasoning-model versions and search budgets. Rendering equivalence across methods also needs clarification.
Questions for authors
  1. Can you report the full method and all ablations on exactly the same prompts, including removal of behavior derived planning and functional grouping?
5 comments
AuthorsOpenReview ↗

Reply to Reviewer gAor

Thank you for your careful review and for highlighting the value of everyday-use context and functional groups. The revision adds matched ablations and accessibility measurements, and documents the generation, search, and rendering settings. We hope our responses address your concerns and look forward to your feedback.

W1: The ablations need a matched full-method control

We agree that Full and the component removals should use the same prompts. Table 4 (revised manuscript, p. 10) compares all four variants on 30 shared prompts, ten per difficulty level. Each metric uses the scenes with valid scores for all four variants. Appendices A.5–A.6 (pp. 18–19) give the matching and sampling rules, and Tables 9–11 (revised manuscript, pp. 29–30) list the prompts.

Full scored 16.83 percentage points higher than NoBehavior on OAR, 2.42 points higher than NoGroupAssembly on STB, and 5.13 points higher than NoHumanUse on ACC. Q1 below gives the full results and specifies what each ablation removes. Table 3 in the revision will present these comparisons. The revision also includes visualizations of the layouts and human-use spaces.

W2: Evidence for functional usability

We agree that a direct test would have people perform the inferred activities in the generated rooms. Our current evidence is indirect.

  1. Visual evidence. The bunk-bed ladder in Figure 5 (revised manuscript, p. 10) faces a wall in NoHumanUse and open floor in Full. Our layout leaves room to stand at the foot of the ladder before climbing.
  2. Human judgments. In a blinded five-way study of 20 rooms, 25 participants gave ours 54.0% of 500 votes for lived-in plausibility and SceneSmith 31.0% (Table 2 (revised manuscript, p. 9)). Participants thus saw our layouts as a better fit for everyday routines.
  3. Accessibility and support. On 30 matched prompts, both variants exceed 94% on floor navigability. Usage-space feedback improves functional-side access (ACC, 74.8% to 79.9%) and door and window clearance (OPEN, 69.0% to 93.6%). Geometric support (SUP) is 91.3% for Full and 70.0% for SceneSmith (Table 4 (revised manuscript, p. 10)). Our layouts thus leave room to approach furniture and support items on shelves and tabletops.

Taken together, we think these findings provide indirect support for the functional usability of the scenes our method generates. As a next step, we plan to use NVIDIA Isaac Sim to test whether a simulated robot can complete tasks in these scenes, such as approaching a shelf and retrieving an item from it.

W3: Models, search budgets, and rendering

Thank you for asking us to clarify the settings. We agree that the models, search budgets, and rendering settings should be specified.

  1. Models. We use gpt-5.6-sol for both the LLM and the VLM throughout generation. The LLM proposes activities and layouts. The VLM provides visual feedback and performs spatial analysis. Automated evaluation uses deepseek-flash. Full and the ablations use Qwen-Image-2.1-Turbo to generate asset reference images and SAM3D to generate 3D geometry.
  2. Search budgets. Beam search defaults to 24 poses per ordinary group, width 12, and 600 state evaluations. Its adaptive caps are width 24 and 12,000 evaluations. In the ablation study, each variant can propose up to three room layouts per round for up to four rounds. Full receives three base whole-room reviews, and the removals receive five for repair and export. Appendix A.1 (pp. 13–14) specifies these settings.
  3. Rendering settings. All five methods are displayed at 1280 × 1280. Ours, SceneSmith, HSM, and Holodeck share four diagonal orthographic views, a top view, and a framing rule. Ours, SceneSmith, and HSM share Blender/Cycles and lighting settings. We use Holodeck's native Unity/AI2-THOR renderer to preserve its material and texture appearance, with neutral illumination at the common light positions. LayoutVLM uses its official Eevee perspective views (Appendix A.1, p. 14).

We provide details on additional key configuration settings in the appendix to our revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.