acceptodds
Under review as a conference paper at ICLR 2027

Measuring Frontier Code Agents at Digital-Twin Scene Construction

Abstract

Simulation-based robot learning is gated by a manual step that repeats at every deployment site: constructing a physical digital twin of the scene. This paper measures how far unmodified, off-the-shelf code agents can automate that step, a setting referred to as Frontier Agent as Scene Constructor (FASC). Frontier code agents such as Claude Code and Codex, each used as shipped, rebuild eight BEHAVIOR-1K rooms from one photograph, an asset catalog and the simulator. Each reconstruction is scored against the room's known 3D layout on ACDC's sim-to-sim protocol, with memorisation and file-access contamination controlled. The strongest agent on the rooms, Codex with GPT-6 Astra, retrieves the exact asset model for 66% of the scored objects and places objects within 14 cm of ground truth across repeated runs. Claude Code with Opus 5, released a little over one month earlier, retrieves the exact asset model for 52% of the scored objects, against 6% for the same agent with Opus 4.6, released under six months earlier, and places objects within 39 cm against 153 cm, a 4× reduction in placement error over that predecessor. A gap to ACDC's curated pipeline, at 6 cm, remains. Next, the two strongest agents on the rooms both extend to outdoor farm scenes, where no calibrated camera or 3D ground truth exists, using a procedural plant model and common asset libraries. Validated against their photographs in DINOv2 embedding space, 24 of the 46 GPT-6 Astra twins and 22 of the 46 Opus 5 twins rank their own photograph first among the 46 reference photographs in cosine similarity. Together with prior results on trajectory generation, these findings indicate that a generic code-agent harness covers a growing share of embodied tasks, and that its capabilities improve with each frontier-model generation while requiring no embodiment-specific harness engineering.

Primary area:
applications to robotics, autonomy, planning
Keywords:
code agents, LLM agents, digital twins, scene reconstruction, simulation, robot learning, sim-to-real, benchmark, BEHAVIOR-1K, Isaac Sim, procedural plant modeling, agriculture

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_wV8rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies Frontier Agent as Scene Constructor (FASC), an evaluation setting in which general-purpose code agents reconstruct simulation scenes from a single image using an asset catalog and a renderer. Seven agent configurations are evaluated on eight BEHAVIOR-1K rooms, with two runs per room. The strongest configuration achieves 66% exact-asset recovery and a median-run placement error of 14 cm over matched objects. Two configurations are additionally evaluated on 46 farm photographs each using image-embedding similarity. The study addresses the practical cost of creating environments for robot simulation, although it evaluates reconstruction rather than downstream robot learning or transfer.

Critical strengths
  1. The empirical question is timely and practically relevant. Measuring the capabilities of general-purpose agents can help researchers decide which parts of scene construction still require specialized systems. The comparison includes seven configurations, repeated room runs, and both performance and resource measurements. The substantial differences between successive models provide useful observations, even though they do not isolate the causes of improvement.
  2. The contamination controls and reporting are valuable. The study restricts filesystem access, disables web tools, audits agent and sub-agent transcripts, and performs recitation probes for several models. The appendices disclose access attempts, exceptions, scoring choices, and differences from ACDC. These details improve the interpretability of an evaluation involving agents with broad tool access and opaque pretraining data.
Critical weaknesses
  1. The protocol limits the central comparison and the interpretation of reconstruction accuracy. ACDC is quoted on its own four rooms rather than rerun on the eight rooms used here; Appendix A also documents differences in camera information, asset access, matching, and orientation scoring. Consequently, the reported gap to ACDC is not a controlled estimate of the advantage of specialized reconstruction. Within the agent benchmark, unmatched reference objects are excluded from placement error and extra predictions are unpenalized. GPT-6 Astra places 123 objects against 150 reference instances across the repeated runs, so its 14 cm median-run error must be interpreted alongside incomplete recovery. Finally, the “as shipped” framing is imprecise: Appendix C describes empty system prompts for the Claude SDK arms and vendor prompts with high reasoning effort for Codex. These results characterize particular configurations rather than a clean model-only comparison.
  2. The evaluation does not establish that the reconstructed scenes function as useful robot environments. The indoor test contains eight scenes curated to retain large furniture, with known camera pose and floor height. The farm extension evaluates similarity to the input photograph without metric 3D ground truth. Neither evaluation validates contact geometry, articulation, physical parameters, or policy performance. In particular, image-embedding retrieval cannot establish the geometry needed for navigation, reaching, or harvesting. There are no real-robot experiments or downstream policy studies connecting these reconstructions to the motivating deployment bottleneck. A measurement paper need not introduce a new algorithm, but the present study needs stronger functional evidence to support its broader conclusions about embodied capability. I therefore lean toward weak rejection.
Questions for authors
  1. How do the agents compare under object precision/recall and a scene-level metric that penalizes both omissions and extra predictions? Can you report uncertainty over rooms and repeated runs for these completeness-aware results?
  2. Can you evaluate a specialized reconstruction system on the same scenes, with matched camera information and asset access, to establish the remaining gap under a common protocol?
  3. Is there evidence that the reconstructions support valid collisions, articulation, or a representative robot task? How robust are the capability conclusions to stock system prompts and matched inference budgets?
Reviewer_KW2LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work evaluates coding agents ability to reconstruct a 3D scene from a single image. The authors compare 7 agent configurations on 8 indoor scenes for asset and layout accuracy, and also test reconstruction on real farm photos. The main finding is that newer agents perform better on asset recovery and placement.

Critical strengths

The indoor evaluation gives concrete measurements of asset and layout correctness, rather than only showing rendered images. The cost and runtime results are also useful for understanding what current coding agents can do in practice

Critical weaknesses
  1. the related work is incomplete

Works such as VIGA, Articraft and BlenderGym are not cited, although all were available more than three months before ICLR submission, and all of them are representative works on "agentic for 3D" to my knowledge. Their tasks are not identical, but they are relevant to agent workflows and evaluation. The authors should discuss these works and explain the differences.

VIGA, Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning. Yin et al. arXiv 2026

Articraft, An Agentic System for Scalable Articulated 3D Asset Generation. NeurIPS 2026.

BlenderGym, Benchmarking Foundational Model Systems for Graphics Editing. CVPR 2025.

  1. the contribution over existing work is not clear enough

My main concern is what we learn beyond newer agents doing better on a few scenes. VIGA and Articraft already explore dedicated agent workflows and tools, while BlenderGym provides a detailed benchmark and analysis. The tasks differ, but I would like to understand what new insight this study adds beyond these works, rather than mainly testing newer agents in a different setup.

  1. the benchmark is limited

Since the main contribution is an empirical evaluation, I would appreciate a new evaluation perspective or broader scene coverage. The main 3D evaluation uses only 8 simplified synthetic rooms. The real farm photos are useful, but cover a specific domain and are evaluated through image similarity rather than 3D accuracy. There is some behavior analysis, but I still find the insight limited. Broader real scene coverage and clearer analysis of failure cases would help

Questions for authors

What is the main new insight compared with VIGA, Articraft, BlenderGym, beyond evaluating newer agents on a different scenes?

What evidence supports the broader scene reconstruction capability outside the simplified synthetic rooms or farm domain?

Reviewer_bkasOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether existing LLM-based coding agents can reconstruct robotics simulation environments from a single input image. It introduces FASC, a pipeline in which agents generate simulation scenes, render them, and sub-agent iteratively refine them based on visual feedback. The paper mainly evaluates the single image 3D scene reconstruction capabilities of different existing frontier models, with experiments on 8 indoor rooms and 46 agricultural scenes.

Critical strengths

This paper investigates whether general purpose coding agents can automatically reconstruct simulation environments from visual observations. The proposed framework provides an end-to-end pipeline for 3D scene reconstruction from a single image. This could potentially reduce the manual effort required to reconstruct simulation environments for robotics applications.

The authors provides a quantitative comparison of multiple frontier coding agents, covering both indoor and outdoor environments. The evaluation includes geometric reconstruction accuracy for indoor scenes, visual similarity for outdoor scenes, and runtime and cost measurements. These results provide a useful reference for us to understand the current capabilities and limitations of existing agents on this task.

Critical weaknesses
  1. The paper mainly introduces a pipeline combining existing agent capabilities and evaluates different models. I do not find the narrower capability-measurement contribution sufficiently novel or informative to warrant acceptance. The results seem more appropriate for a technical report.

  2. The indoor evaluation contains only 8 rooms and 46 outdoor scenes limited to agricultural distribution. I do not think this is sufficient to faithfully reflect the performance of different models.

  3. The task is not purely image-to-3D reconstruction. In addition to the reference image, agents rely on a predefined asset catalog and, in the indoor experiments, known camera parameters. This makes the setting more constrained and difficult to extend to realworld scenes.

  4. I do not see sufficient comparisons against established Real2Sim reconstruction methods, such as HoloScene [1] and DISCOVERSE [2]. This makes it difficult to assess the proposed pipeline relative to existing approaches. Also the ACDC comparison is unfair since it's a 4 rooms evaluated results

  5. One of the main claim of this paper is that building simulation environments efficiently can address the environment scalability challenges of robotics training (or sim-to-real gap). However, there is no robotics training experiment in this paper to verify that claim.


[1] Xia, H., Lin, C.-H., Hsu, H.-Y., Leboutet, Q., Gao, K., Paulitsch, M., Ummenhofer, B., & Wang, S. (2025). HoloScene: Simulation-ready interactive 3D worlds from a single video. Advances in Neural Information Processing Systems, 38. https://doi.org/10.52202/085713-1090 NeurIPS Proceedings

[2] Jia, Y., Wang, G., Dong, Y., Wu, J., Zeng, Y., Lin, H., Wang, Z., Ge, H., Gu, W., Ding, K., Yan, Z., Cheng, Y., Li, Y., Wang, Z., Li, C., Sui, W., Shi, L., Tian, G., Huang, R., & Zhou, G. (2025). DISCOVERSE: Efficient robot simulation in complex high-fidelity environments. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 6272–6279). IEEE. https://doi.org/10.1109/IROS60139.2025.11247559

Questions for authors
  1. How much does the method depend on the provided asset catalog and known camera information?
  2. Why are there no direct comparisons against existing Real2Sim approaches?
  3. How are runtime and computational costs compared between different approaches?

I think the work could be useful as a technical report of current model capabilities, but I do not consider this a sufficiently innovative academic contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.