acceptodds
Withdrawn from ICLR 2027

LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstruction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present LiteReality-Agent, an agentic system for reconstructing real indoor environments as realistic, articulated, and simulation-ready 3D scenes from RGB-D scans. At its core, LiteReality-Agent formulates 3D reconstruction as a coding problem, in which a coding agent gathers evidence using specialised tools and iteratively edits a Python script, Room.py, which can be executed to produce a 3D digital twin of the room. Based on this formulation, we develop a robust observe-edit-verify harness that supports evidence gathering, measurement, verification, layout optimisation, simulation readiness, and quality control throughout the reconstruction process. LiteReality-Agent produces high-quality reconstructions suitable for simulation and downstream embodied AI tasks. Furthermore, as agent capabilities continue to improve rapidly, the system introduced by LiteReality-Agent provides a robust orchestration framework for future agents by equipping them with specialised tools, structured workflows, and reliable verification mechanisms that substantially improve reconstruction quality and reliability. We demonstrate that LiteReality-Agent produces reconstructions that are more geometrically accurate, visually realistic, and simulation-compatible than those generated by recent frontier models, such as Astra and Fable. We therefore view LiteReality-Agent as a practical and important building block for robust real-to-sim systems. Both the source code and the data-capture application are publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D reconstruction, embodied AI, RGB-D scanning, digital twins, agentic systems, real-to-sim, simulation-ready environments, indoor scene reconstruction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sTqmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an agentic system for reconstructing real-captured indoor environments as sim-ready 3D scene. The input to the system is the 3D scan of a room and the corresponding RGB frames. The overall pipeline is divided into two stages, where the first stage reconstructs objects and an initial layout, whereas the second stage refines the scene through iterative rendering and comparison. The authors test their system on 10 rooms and compare it with frontier models, such as Astra and Fable, showing better results.

Critical strengths
  1. The quality of the reconstrution looks pretty impressive, and are better than vanilla coding agents. The comparisons with GPT Astra and Claude Fable are informative and make the position of this paper clearer, although the implementation details of those baselines are not very clear.
  2. The agentic design for this task makes sense, as the scene reconstruction is a long-horizon task requiring many steps of reasoning and tool use (e.g., object reconstruction / collision check), and a dedicated harness can be useful.
Critical weaknesses
  1. My main concern with the paper is the evaluation: The authors only mention that they use 10 scenes, but it is unclear which dataset the scenes are from, or whether they were collected by the authors. The authors should either properly cite the data source or provide details on how the data are captured. Moreover, 10 scens are very limited in scale. Finally, the authors should test on the same scenes are the baseline LiteReality for an apple-to-apple comparsion, espeically since LiteReality seems to totally fail in the scenes selected by the authors as shown Fig 3.
  2. The authors should provide the exact prompts used for the agents in the proposed system, as well as the agent-based baselines, so that the results are reproducible and it is possible to judge if the baseline implementations are best-effort.
  3. The writing of the paper needs improvement, especially the methods section. There are some undeclared terms such as "Geometric checks", "Deterministic repairs", etc (L203-209), making it very hard to understand how the layout agent work. The 3 levels in section 3.3.1 are also confusing, and I think it would be helpful to explain with some diagrams or example figures.
  4. This is not a particular weakness of this paper, because papers have been published in top venues on the same task. But I kind of don't understand why the scene level real to sim is a useful task, especially compared to the text-to-3D-scene task. Note the approcah is not very scalable, as it requires carefully scan and reconstriction of the scene. The reconstruction is also far from faithful, compared to 3D reconstrtion method such as VGGT or gaussian splatting. From my opioion, both of these places this real-to-sim task in an awkward position, where the reconstrructed scenes are neither diverse nor faithful.
Questions for authors

N/A

2 comments

We thank the reviewer for their detailed feedback and thoughtful questions. We would like to first address Weakness 4 regarding the usefulness and significance of the task.

  • First, this task produces 3D environments that are directly compatible with traditional graphics and simulation pipelines, making the results immediately useful for robotic simulation, training, and industrial digital twins. Crucially, robotics requires interactive, articulated, and physically grounded environments for training and evaluation, whereas other visually faithful 3D representations, such as Gaussian splatting, image-to-3D reconstructions, and point clouds, cannot be directly used for these purposes without further processing. Our pipeline therefore produces practical, reusable assets that can be directly integrated into a wide range of commercial and robotic applications.

  • Second, we believe this approach offers a promising path toward scalable reconstruction. As agent capabilities continue to advance, increasingly complex and diverse environments can be reconstructed with less manual intervention. The improvements observed from LiteReality to LiteReality-Agent demonstrate this potential. Prior to LiteReality-Agent, there are no publicly available tools capable of performing this task at all. A central motivation of our work is to make this challenging reconstruction process significantly more accessible, streamlined, and practical for the broader community.

We hope this clarifies both the usefulness of the task and the practical value of our contribution, and we respectfully encourage the reviewer to reconsider this aspect of their assessment. We address the remaining concerns below.

CW1: Dataset and comparison with the original LiteReality paper

The ten scans were captured in real environments, including offices, kitchens, bedrooms, meeting rooms, and hallways, covering a diverse set of scenes. We also provide a comparison on the original LiteReality paper’s scenes below, including its other baselines.

MethodRMSE ↓SSIM ↑LPIPS ↓
Phone2Proc0.36040.55120.7338
Digital Cousin0.36530.55310.7364
DC + Sem&Vis0.32260.54250.6717
DC + MIR0.30460.54920.6648
LiteReality0.26640.58180.6522
LiteReality-Agent0.23950.60010.5846

LiteReality-Agent upgrades the original deterministic pipeline to an agentic approach, providing substantial improvements in reconstruction quality.

CW2: Prompt fairness and additional comparisons

We agree and will provide the exact text prompts. We use the same task prompt and input information for the corresponding comparisons. We also provide more detailed comparisons covering:

  1. Multiple agents with and without the proposed harness.
  2. The same LLM at four reasoning-effort levels.

1. Multiple agents with and without the proposed harness

MethodSSIM ↑LPIPS ↓RMSE ↓
LiteReality0.59410.65020.2537
Astra0.62050.67400.2598
Astra + LiteReality-Agent0.66780.50540.1928
Fable 5.10.55650.71120.3084
Fable 5.1 + LiteReality-Agent0.65100.54090.2129

2. The same LLM at four reasoning-effort levels (Astra as base model)

Reasoning EffortMethodPSNR ↑SSIM ↑LPIPS ↓
LowBaseline14.0840.66870.5938
+ LiteReality-Agent14.3160.67870.5344
MediumBaseline13.5930.64620.6006
+ LiteReality-Agent13.6340.67500.5443
HighBaseline13.7300.64440.5552
+ LiteReality-Agent14.7140.68390.5040
XHighBaseline14.5570.64920.5362
+ LiteReality-Agent14.8820.66890.4869

LiteReality-Agent achieves better scores than the baseline at each effort level in these results.

CW3: Writing and methodological details

We included additional methodological details in the supplementary material and open-sourced the entire codebase to support reproducibility. We will improve the main paper’s explanations and add diagrams to make the workflow easier to understand.

Area_Chair_Lj3pOpenReview ↗

Response needed

Dear Reviewer, As the discussion phase deadline is approaching, could you please review the authors' response and let us know whether your concerns have been adequately addressed, and if you have any further questions? Thank you very much for your time.

Reviewer_TXuVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a method for using agents to reconstruct indoor scenes from RGBD observations of a room. The method involves an initial scene generated using prior work, LiteReality. Then an agentic harness can use this plus RGB and 3d information to refine the scene. The agent operates on a python script representation of the scene that can render it in different ways such that the agent can assess the scene reconstruction.

The paper presents validation in terms of visual accuracy, 3D accuracy, and overall opinion of accuracy (from AI agents and human evaluators)

The major contributions are on the methodological side. The paper presents a way to incorporate 3D information into the agent to generate and refine the scenes. This differentiates the method from other VLMs performing the task. They confirm that their method which uses this

Critical strengths

The strongest point I can tell is the results are very good looking. I can see that the method is able to reconstruct scenes with great accuracy. Even with very custom objects, there is a lot of correspondence from the original to the rendered scenes. This seems to be attributed to using code as a representation and agents as a method for generating these scenes. While the harness that the paper presents improves upon just simple LLMs generating the scenes, the largest improvements over prior work like LiteReality come simply from switching to a code representation and agentic editor.

The paper is very clear and well written. I appreciate that the authors organized the paper well and edited the writing so that it was free of errors and very concise.

The work is opensource which I think is beneficial to many people since the results are very high quality.

Critical weaknesses

It is not clear how the presented harness is different from other 3D agentic generators. There is a nice section in the related works about other methods. I can see that they often don't focus on indoor scenes but in terms of method, the paper needs to clearly show why this harness is different or similar to others. For example, SceneSmith and SceneWeaver seem to have very similar methods. In the introduction the paper states: "What remains missing is a system that can orchestrate such agents while grounding their outputs in realworld observations and ensuring that the resulting scenes are consistently high quality and suitable for their intended applications." These prior works seem to also orchestrate agents, right? I can see that some of these prior works focus on text-to-scene generation rather than conditioning on the observations, but if you swap the conditioning modality from text to rgbd it would be very similar methods, right?

I do not see any surprising finding that a reader could take with them as new knowledge. The method presented does have very nice results, but from what I can tell, this is because the method is using code as a representation with agents as a method. We can see from many prior works that this is a good representation and coding agents are great at editing them. So to me, it is not a conclusion that I would find novel or useful for this paper.

Small weaknesses to help future iterations of the paper:

Section 3.2: The overview of the scene initialization is vague. It mostly just says that it does what LiteReality does. More details here would be nice to make the paper more self-contained. Distinguishing the new details from LiteReality would be good here and in Figure 2 so readers know what this paper is contributing and what is taken from the prior work.

In table 3 it could be nice to see some error bars on the MAE. I imagine there is some decent variance that would help understand the significance in difference between the methods' results.

The paper does not address the limitations very well. There is only a short section with two sentences. I would like to see more content on what types of errors are typical for the method.

Questions for authors

I want to make sure that I understand the differences between the LiteRealtiy Agent and the other agentic baselines with scan inputs. My understanding is that LiteReality Agent has some specific tools and evaluation criteria baked in as a harness. One question is, the other baselines could potentially develop their own tools and criteria right? For example, Astra could create its own harness internally and operate similar to the presented method, right? If so, do you see this happening or is there some limitation to the current LLMs for creating a nice harness for themselves?

How detailed are the observations? Are the depth maps and point clouds perfect? In real world scenarios these may be captured with inaccurate sensors, such as RealSense cameras, and I am wondering if your method would work with such sparse depth data.

2 comments

We thank the reviewer for their thoughtful feedback and for acknowledging the strong reconstruction quality, clear presentation, and potential benefits of our open-source system.

CW1: On the novelty and primary contribution of LiteReality-Agent

We would like to emphasise that the primary objective of LiteReality-Agent is to provide a practical, reliable, and readily usable system for reconstructing simulation-ready 3D environments from real-world observations. To our knowledge, no comparable open-source tool existed prior to LiteReality-Agent for this task. There is substantial demand for such tools in digital twin reconstruction, Real2Sim, and environment mapping. Our objective is to address this need by developing carefully engineered tools and workflows and making them openly available to benefit the broader research community.

We agree that code-based scene representations and agentic editing are not novel concepts, particularly with the emergence of frontier coding agents such as Astra. However, the contribution of LiteReality-Agent lies not simply in adopting these components, but in identifying and addressing non-trivial challenges in real-world Real2Sim and digital twin reconstruction, developing robust solutions, and integrating them into a practical, streamlined system. We would like to highlight several designs:

  1. Code-based scene representation: We introduce scene.py as the final scene representation, enabling agents to directly interact with, modify, and iteratively refine structured 3D environments through code.

  2. Agentic layout refinement: We address the longstanding challenge of refining noisy but observation-grounded 3D detections into coherent and physics-grounded scene layouts. Our agentic approach enables iterative optimisation of spatial arrangements while maintaining consistency with real-world observations.

  3. Specialised authoring and verification tools: We develop a collection of domain-specific tools that support scene authoring, refinement, and verification, improving the robustness and reliability of the reconstruction process. The ablation results for each component are presented below:

ExpWorkflowLiteReality ToolsQCPSNR ↑SSIM ↑LPIPS ↓RMSE ↓
I———13.25540.64280.59730.2268
II✓——14.04620.64300.52840.2048
III✓✓—14.68220.66050.52350.1912
IV✓✓✓14.71390.68390.50400.1899

CW2: On whether LLMs could independently develop their own harness

LLMs are increasingly capable of autonomously developing tools, workflows, and evaluation mechanisms. However, there remains an important distinction between the potential capabilities of a general-purpose LLM and a readily deployable, reliable system designed to solve a specific real-world problem. LiteReality-Agent incorporates domain-specific expertise through carefully designed tools, orchestration workflows, and quality-control mechanisms, with the goal of providing a robust, accessible, and immediately usable solution for simulation-ready reconstruction. We believe these are hard to achieve through improvements in agent capabilities alone.

Importantly, LiteReality-Agent can also benefit directly from advances in underlying LLM capabilities, while its domain-specific tools and verification mechanisms remain important for ensuring reliable, observation-grounded reconstructions. We therefore view LiteReality-Agent as a complementary system that translates improvements in general-purpose agent capabilities into practical advances in Real2Sim reconstruction. The results of our experiments so far are presented below:

MethodSSIM ↑LPIPS ↓RMSE ↓
LiteReality0.59410.65020.2537
Astra0.62050.67400.2598
Astra + LiteReality-Agent0.66780.50540.1928
Fable 5.10.55650.71120.3084
Fable 5.1 + LiteReality-Agent0.65100.54090.2129

CW3: On limitations and typical failure cases

We thank the reviewer for this suggestion and agree that the limitations deserve further discussion. We identify three main limitations:

  • Physical properties: Accurately estimating object-level properties, such as mass and friction, remains challenging.
  • Asset diversity: The current system primarily supports rigid and articulated objects, with limited support for deformable assets.
  • Reconstruction efficiency: Complex scenes can require substantial time for iterative refinement and verification.

We are actively addressing these challenges in the next version of LiteReality-Agent and will expand the limitations discussion in the revised manuscript.

Area_Chair_Lj3pOpenReview ↗

Response needed

Dear reviewer, As the discussion phase deadline is approaching, could you please review the authors' response and let us know whether your concerns have been adequately addressed, and if you have any further questions? Thank you very much for your time.

Reviewer_mVwVOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides an agentic system of using RGB-D scans to reconstruct real indoor simulation-ready 3D scenes in the form of coding problem. To be more detailed, the agent iteratively edits Room.py and this file can be executed to produce the 3D digital twin. What's more, the framework of their LiteReality-Agent is a general harness for agents to use tools, improve workflows and achieve robust verification.

As shown in the paper, LiteReality-Agent has better qualitative and quantitative results than latest frontier models like GPT-6 Astra and Fable 5.1 across the 10 room cases they present in Table 2,3 and Figure A1, A2. The paper also reports human and agent-based perceptual evaluations (Table 4), and demonstrates the editability and simulation-readiness of the reconstructed scenes (Figures 5–6).

Critical strengths
  1. Building on LiteReality and its full scan input, the method achieves better quantitative visual metrics than general-purpose foundation-model baselines. The qualitative results also look better, and the reconstructed scenes demonstrate simulation-ready capabilities.

  2. The ten evaluated rooms cover a relatively diverse range of indoor environments, providing evidence of the pipeline's generality.

Critical weaknesses
  1. The paper claims that the harness can continue to benefit from improvements in agent capabilities. But the experiments do not compare different agents under the same harness, nor do they provide ablations of the harness components. Since the pipeline also builds heavily on the existing LiteReality system, it is quite difficult to tell how much the proposed harness contributes to the performance.

  2. The paper compares LiteReality-Agent with GPT-6 Astra via Codex and Fable 5.1 via Claude Code, but both of them are already capable coding-agent systems. The experimental setup for these baselines is not sufficiently clear. Without details on how the scan inputs were provided and what prompts and tools were used, it is difficult to judge whether the comparison is fair.

Questions for authors
  1. Have the authors tested different agents under the same harness or ablated harness components?
  2. How much of the improvement comes from the RoomPlan-provided layout information (walls, floors, objects) versus the proposed harness?
  3. Are there more specific asset-routing criteria and reports about the proportion of assets from code vs. image-to-3D models?
  4. For the baselines running in Codex/Claude Code, what prompts, input files, tools and execution budgets were used?
  5. Could the authors report the end-to-end reconstruction time and computational/API cost per scene?
3 comments

We thank the reviewer for the detailed feedback and constructive suggestions for improving our evaluation and clarifying our claims. We address the concerns and questions below.

CW1: Comparison of different agents under the same harness and the contribution of LiteReality-Agent

Reconstructing a detailed indoor scene as a simulation-ready environment is a long-horizon task. LiteReality-Agent provides an orchestration layer that guides agents through this task using a dedicated workflow, task-specific tools, and quality-control checks to improve output consistency and robustness. To evaluate whether these benefits generalize across agents, we test Astra and Fable 5.1 with and without the same LiteReality-Agent harness. As shown in Table 1, the harness consistently improves both agents across all three metrics.

Table 1: Performance of different agents with and without the LiteReality-Agent harness, alongside the original LiteReality baseline.

MethodSSIM ↑LPIPS ↓RMSE ↓
LiteReality0.59410.65020.2537
Astra0.62050.67400.2598
Astra + LiteReality-Agent0.66780.50540.1928
Fable 5.10.55650.71120.3084
Fable 5.1 + LiteReality-Agent0.65100.54090.2129

Ablation of harness components. We conduct a progressive ablation study by adding three components in sequence: (1) the dedicated workflow, (2) the LiteReality task-specific tools, and (3) the quality-control (QC) layer for final verification. The results below show that each addition improves performance across all four metrics, with the complete harness achieving the best results. These findings demonstrate the incremental benefit of each component within the tested configuration.

ExpWorkflowLiteReality toolsQCPSNR ↑SSIM ↑LPIPS ↓RMSE ↓
I———13.25540.64280.59730.2268
II✓——14.04620.64300.52840.2048
III✓✓—14.68220.66050.52350.1912
IV✓✓✓14.71390.68390.50400.1899

The original LiteReality pipeline follows a deterministic workflow and achieves lower performance in our comparisons in Tables 2 and 3. LiteReality-Agent extends this pipeline with iterative scene editing, rendering, comparison, and verification. Together, the cross-agent comparisons and progressive ablation results support the contribution of the proposed harness to reconstruction quality.

CW2: Clarity and fairness of the coding-agent baselines

For comparisons with off-the-shelf coding agents, we provide the same input information and detailed prompts consistent with the initialization prompts used by LiteReality-Agent. The main difference is whether the agents have access to our harness, including the dedicated tools designed, verification layers, and workflow.

Q1: Have the authors tested different agents under the same harness or ablated harness components?

We have tested different agents under the same harness. Please see the comparison below and the ablation table above.

MethodSSIM ↑LPIPS ↓RMSE ↓
LiteReality0.59410.65020.2537
Astra0.62050.67400.2598
Fable 5.10.55650.71120.3084
With LiteReality-Agent
Astra0.66780.50540.1928
Sol 6.10.65400.52190.2092
Opus 5.50.63190.51830.2227
Fable 5.10.65100.54090.2129

Q2: How much of the improvement comes from the RoomPlan-provided layout information versus the proposed harness?

We provide the same scan information to the off-the-shelf coding agents evaluated through Codex and Claude Code. The “+Scan” results in Table 2 correspond to these scan-aware baselines. A more directly comparison of layout info + proposed harness can be fined below:

MethodSSIM ↑LPIPS ↓RMSE ↓
GPT-6, images only0.62050.67400.2598
GPT-6 +Scan0.64280.59730.2268
Full LiteReality-Agent0.68390.50400.1899

Q3: Are there more specific asset-routing criteria and reports on the proportion of assets generated through code versus image-to-3D models?

We use a VLM to route objects according to their geometric complexity. The routing criteria are:

  • Code generation: Simple, regular geometry that can be constructed from boxes, cylinders, extrusions, or revolved profiles—for example, tables, cabinets, shelves, and box-shaped appliances.
  • Image-to-3D generation using TRELLIS: Curved, upholstered, organic, or visually complex geometry—for example, sofas, ergonomic chairs, plants, and decorative objects. There are also explicit overrides: chair clusters, categories containing chair, sofa, couch, armchair, plant, or person, and objects classified as organic are routed to TRELLIS.

Q4: For the baselines running in Codex and Claude Code, what prompts, input files, tools, and execution budgets were used?

The Codex and Claude Code baselines receive the same prompts and input files. For reproducibility, we will include these in the final revision. The input files include RGB-D images, the layout from RoomPlan, and the point cloud. The agents can use whichever tools they find useful. We also give the baselines the same maximum budget of 200 iterations as LiteReality-Agent, although they may finish earlier. The results above demonstrate the improvements achieved with the proposed harness. we showcase the budget comparison below.

Q5: Could the authors report the end-to-end reconstruction time and computational/API cost per scene?

The estimated API costs for each scene are provided below, including those of the baseline models.

SceneArea (m²)Input objectsOutput scene entries¹Time (min)Estimated API cost
Airbnb-Cam9.8675824.70$10.27
Airbnb-Room15.1849631.06$11.63
MIL-Meeting22.66209730.55$12.98
Office14.95714731.75$12.90
Psi_2_meeting17.321111550.79$23.57
Psi_2_seminar33.62188431.05$11.89
Fallside Kitchen13.24117947.00$20.70
Fallside Office11.1267336.24$15.23
Area_Chair_Lj3pOpenReview ↗

Response needed

Dear reviewer,

As the discussion phase deadline is approaching, could you please review the authors' response and let us know whether your concerns have been adequately addressed, and if you have any further questions? Thank you very much for your time.

Comments on the paper

1 comment

Dear Reviewers,

Thank you very much for your time and effort in reviewing our paper. We have conducted additional experiments and ablation studies to address the concerns you raised. We believe these results demonstrate the practical and tangible value that the proposed agentic system brings to this important task. please let us know whether it addresses your concerns and happy to answer more questions.

Litereality-Agent Team

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.