acceptodds
Under review as a conference paper at ICLR 2027

LEGO-Anything: Coding Agents for 3D Scene Reconstruction

11 views

Abstract

A 3D scene reconstructed from a single image is most useful when represented not as a rendering or a fixed 3D output, but as an explicit scene program whose execution yields a scene that can be inspected, edited, and queried. We present **LEGO-Anything**, an *Image-to-Code* framework in which a coding agent builds such a program by iteratively writing and executing Blender code, inspecting the evolving scene and its renderings, and revising the program. To evaluate how well such agents recover scenes end to end, we introduce **LEGO-Bench**, a simulator-grounded benchmark with 208 images from 104 diverse scenes spanning indoor and outdoor environments. LEGO-Bench separately scores artifact validity, visible-surface geometry, and rendered appearance, and its simulator-grounded design makes the benchmark naturally extensible while retaining precise automatic evaluation. Across the evaluated coding agents, GPT-6-astra achieves the strongest overall results, with 53.4% indoor and 39.6% outdoor scores, yet substantial gaps remain between delivering valid scene artifacts and faithfully recovering scene geometry and appearance. To understand these failures, we analyze agent construction trajectories and identify three recurring issues: weak scene initialization, regressive edits during iteration, and unreliable self-evaluation. These findings motivate **LEGO-Plugin**, a training-free harness plugin for more controlled iterative scene construction, which improves all six evaluated models, with relative gains of up to 62.7% in overall score. Finally, we ask whether reconstructed scenes are faithful enough to represent natural images and support vision tasks. In **LEGO-World**, we take scenes reconstructed by GPT-6-astra and derive object detections, instance masks, and relative depth as deterministic queries on each scene. These readouts show non-trivial performance across all three tasks but fall well short of specialized vision models, suggesting that program-constructed scenes from current coding agents are a promising but not yet sufficiently precise representation of natural images.

Primary area:
datasets and benchmarks
Keywords:
Coding Agent, Agent Harnesses, 3D Scene Reconstruction, Image-to-Code

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_Hyx8OpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on 3D scene reconstruction via an image-to-code approach. The approach iteratively writes and improves the program by rendering out the blender code the agent writes. The authors propose a benchmark, LEGO-Bench which enables evaluation of their method and similar baselines. The propose LEGO-Plugin, a harness that improves various LLMs in this task. They evaluate in settings comparable to classical reconstruction via LEGO-World. The authors present findings based on the results.

Critical strengths
  1. The authors study the problem comprehensively and shares findings such as the effect of reasoning on different model tiers, these findings are useful to the community.
  2. The harness design includes analysis, such as judgement correctness evaluation. This makes the design motivated rather than purely trial-and-error.
  3. The authors make image-to-code comparable with classical reconstruction methods on known datasets, and highlight their current limitations, which motivates future development.
Critical weaknesses

I like the paper and the results. The main weakness I find is about the wording of the contribution, which slightly over-claims.

  1. In terms of contribution wording of "LEGO-Anything framework"- the iterative code writing+review loop is quite common, and I do not consider LEGO-Anything a different framing from any existing agentic work. I do appreciate the method, and it might be better to just say this paper shares analysis and LEGO-Plugin, a harness
  2. Naming LEGO-World as an evaluation setting contribution. I really appreciate the LEGO-World evaluation and feel it's important to put this on the same datasets as traditional methods. Maybe just say you make it comparable to vision methods, rather than setting it up as a new setting with a new name.
Questions for authors
  1. Could you compare with AHA3D (https://kevinxu02.github.io/real2sim-indoor-site/), which is a similar blender scene generation harness?
  2. Would appreciate toning down the novelty claim on framework and evaluation setting. The paper provides clear and informative analysis and a useful harness, which in my opinion is sufficient contributions, and the additional contribution claiming feels untrue and overdone.
4 comments
AuthorsOpenReview ↗

Response to R1 Hyx8

We sincerely thank you for recognizing our comprehensive empirical findings, LEGO-Plugin harness design and evaluation settings with classical CV methods! We hope the following responses address your feedbacks. Please let us know If you have any further inquiries!

AuthorsOpenReview ↗

Response to R1 Hyx8 (1/1)

[Weaknesses 1 & 2, Question 2 - Toning Down Contribution Wording]

“I like the paper and the results. The main weakness I find is about the wording of the contribution, which slightly over-claims.

  1. In terms of contribution wording of "LEGO-Anything framework"- the iterative code writing+review loop is quite common, and I do not consider LEGO-Anything a different framing from any existing agentic work. I do appreciate the method, and it might be better to just say this paper shares analysis and LEGO-Plugin, a harness
  2. Naming LEGO-World as an evaluation setting contribution. I really appreciate the LEGO-World evaluation and feel it's important to put this on the same datasets as traditional methods. Maybe just say you make it comparable to vision methods, rather than setting it up as a new setting with a new name.”

“Would appreciate toning down the novelty claim on framework and evaluation setting. The paper provides clear and informative analysis and a useful harness, which in my opinion is sufficient contributions, and the additional contribution claiming feels untrue and overdone.”

[Response]

We agree and have revised the Abstract, Introduction, Section 3, and Section 6 to remove any over-claiming:

  1. Reframing LEGO-Anything: Rather than claiming the iterative code–render–inspect loop as a novel framework, we now present it strictly as our problem formulation and study testbed, centering our core contributions on LEGO-Bench, our empirical trajectory diagnostics, and the LEGO-Plugin harness.
  2. De-branding LEGO-World: Following your suggestion, we no longer frame LEGO-World as a newly named evaluation setting contribution. Instead, Section 6 is presented directly as a downstream vision readout analysis that makes reconstructed scenes directly comparable to classical perception models on standard benchmarks (COCO, LVIS, and ETH3D).

[Question 1 - Comparison with AHA3D]

“Could you compare with AHA3D (https://kevinxu02.github.io/real2sim-indoor-site/), which is a similar blender scene generation harness?”

[Response]

Thank you for pointing us to AHA3D! AHA3D is a concurrent study. To provide our readers a better understanding of the field, we have cited and discussed AHA3D in Appendix A.4 and added a direct comparison between our proposed LEGO-Plugin in Appendix B.7.

Table 1. Comparison between LEGO-Plugin and AHA3D on an indoor subset (%).

ModelSettingsValidityReconstructionAppearanceFinal Score
GPT-5.6-SolVanilla100.0016.2814.9115.59
with AHA3D100.0016.5120.4618.49
with LEGO Plugin100.0034.2735.9935.13
GPT-6-SolVanilla100.0022.9342.2732.60
with AHA3D100.0027.1835.8231.50
with LEGO Plugin100.0034.4239.3136.86

Thank you! I will maintain my score.

We are glad to see we could improve the quality of the manuscript with your review. Thanks for your time and efforts!

Reviewer_HLk7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether general-purpose coding agents can reconstruct 3D scenes from a single image by writing, running, and repeatedly revising Blender code. The authors introduce LEGO-Bench, which contains 208 images from 104 scenes across indoor and outdoor environments and different difficulty levels. It separately evaluates whether the submitted files are usable, whether visible geometry is accurate, and whether the rendered appearance matches the reference. By examining intermediate results, the authors study problems such as poor initialization, later edits that undo earlier progress, and difficulty judging whether results have improved. They propose LEGO-Plugin to support initialization, validation, and rollback. Finally, LEGO-World explores whether detection boxes, instance masks, and relative depth can be read directly from reconstructed scenes.

Critical strengths

1.The benchmark covers different environments and difficulty levels and helps identify different types of failure. Besides indoor and outdoor scenes, the authors provide matched scenes with progressively more objects to study the effects of increasing difficulty. Evaluating file usability, geometry, and appearance separately helps clarify where the agents struggle. 2.The paper examines problems during reconstruction and proposes ways to address them. It measures when agents produce their first usable scene, which edits reduce quality, and whether the final result is worse than an earlier version. These observations show that further editing does not always improve the result. The plugin’s initialization, validation, and edit-protection features are clearly motivated by these findings.

Critical weaknesses

1.The main metrics focus on geometry and appearance from the reference view. Direct tests of whether objects are properly grouped and easy to edit individually are limited. Additional tests, or a clearer statement of what the current evaluation supports, would help. 2.The plugin shows improvements, but its evaluation is limited to the Office subset, and separate ablations of its components are not reported. These ablations and tests in other environments would help clarify which components drive the gains and how broadly the approach applies.

Questions for authors

1.Are separate component ablations or preliminary results beyond the Office subset available? 2.Are there tests specifically evaluating object grouping or edits to individual objects?

3 comments
AuthorsOpenReview ↗

Response to R2 HLk7

We thank you for recognizing our benchmark's design, analysis and well-motivated harness plugin! We hope the following responses address your concerns. Please let us know If you have any further inquiries!

AuthorsOpenReview ↗

Response to R2 HLk7 (1/1)

[Weakness 1 & Question 2 - Object Grouping and Individual Editability]
“The main metrics focus on geometry and appearance from the reference view. Direct tests of whether objects are properly grouped and easy to edit individually are limited. Additional tests, or a clearer statement of what the current evaluation supports, would help.”
“Are there tests specifically evaluating object grouping or edits to individual objects?”
[Response]
Thank you for this great question. Unlike prior tabletop benchmarks with simple isolated object sets, LEGO-Bench spans complex, multi-scale indoor and outdoor scenes. While we explored object- and part-level grouping analysis during development, we deliberately kept this analysis out of the headline metric SS to avoid introducing item matching noise:

  1. Benchmark Support & Pilot Exploration: LEGO-Bench natively provides instance- and part-level annotations across all 443 items (Figures 15–17), and we conducted a pilot object/part-grouping and articulation study in Appendix B.4 & E.6.
  2. Why Scene-Wide Item Matching Is Excluded from Headline SS: As documented in Appendix E.2 (L1401–1416), our Reconstruction metric intentionally partitions surfaces via projected reference masks rather than predicted mesh grouping or Hungarian/IoU object matching. Because coding agents construct complex scenes at varying procedural granularities, enforcing scene-wide object-to-GT bipartite matching is noisy and compounds correspondence errors into the final score.

To make this more clear, we have clarified the scope of our headline metrics in Appendix E.2 to explicitly emphasize this trade-off against noisy item matching.

[Weakness 2 & Question 1 - LEGO-Plugin Ablations and Results Beyond Office Subset]

“The plugin shows improvements, but its evaluation is limited to the Office subset, and separate ablations of its components are not reported. These ablations and tests in other environments would help clarify which components drive the gains and how broadly the approach applies.”

“Are separate component ablations or preliminary results beyond the Office subset available?”

[Response]

Thanks for your suggestion on the ablation study. We provide corresponding ablation on (1) baseline providing the same perceptual outputs to the vanilla agent and (2) component-wise baselines in following Tables and in our Appendix B.6.

Table 1. Ablation Study of LEGO-Plugin on an indoor subset (%).

ModelSettingsValidityReconstructionAppearanceFinal Score
GPT-5.6-SolVanilla100.0016.2814.9115.59
GPT-5.6-SolVanilla with tool outputs100.0016.8926.7821.84
GPT-5.6-SolLEGO-Plugin without Initialization100.0009.4119.7214.57
GPT-5.6-SolLEGO-Plugin without Grounded Refinement100.0029.1331.6030.37
GPT-5.6-SolLEGO-Plugin without Version Control100.0029.7636.7133.24
GPT-5.6-SolLEGO-Plugin100.0034.2735.9935.13

Table 2. Performance of LEGO-Plugin on the whole dataset (%).

ModelSettingsValidityReconstructionAppearanceFinal Score
GPT-5.6-Solw/o Plugin98.2411.5219.1115.32
with Plugin100.0018.2830.0924.19
GPT-6-Solw/o Plugin100.0021.1536.0128.58
with Plugin99.5226.9335.5631.24

We are very happy to see that our response addressed your concern. Thank you for your time and effort in reviewing our paper!

Reviewer_YouVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents LEGO-Anything, a framework in which coding agents reconstruct single images as executable Blender scenes through iterative code generation, rendering, and refinement. It introduces LEGO-Bench to evaluate artifact validity, geometric reconstruction, and rendered appearance. Based on trajectory analysis, the authors propose LEGO-Plugin, which combines initialization, reference-grounded refinement, and version control. The paper also explores detection, segmentation, and depth estimation through readouts from reconstructed scenes.

Critical strengths
  1. A useful evaluation platform for executable scene reconstruction.

The paper constructs a simulator-grounded benchmark with indoor and outdoor scenes and controlled changes in scene complexity. The analysis of intermediate trajectories further identifies failures that final-output evaluation alone would overlook.

Critical weaknesses

1. Insufficiently established methodological contribution.

The benchmark primarily evaluates final reconstruction quality through artifact checks, surface agreement, and image similarity. These measures are relevant to scene reconstruction, but the paper does not sufficiently explain how its benchmark design diagnoses coding-agent capabilities beyond those of general reconstruction systems. Although trajectory analysis provides some insight, the connection between the benchmark’s design and capabilities such as code generation, debugging, and iterative revision remains unclear.

The plugin largely integrates existing perceptual models, including VGGT [1], SAM 3 [2], and Depth Anything V2 [3], with initialization, validation, and rollback mechanisms. The reported plugin-on/off comparison does not distinguish improvements from these additional perceptual priors from improvements attributable to the proposed control mechanisms. A baseline providing the same perceptual outputs to the vanilla agent, together with component-wise ablations, is needed to establish the plugin’s independent contribution.

2. Reported results are inconsistent with the stated aggregation protocol.

Section 4.2 defines the overall score as (S_i=V_i(R_i+A_i)/2), assigns zero reconstruction and appearance scores to invalid submissions, and averages over all attempted cases. Under this definition, the reported means must satisfy (\overline S=(\overline R+\overline A)/2), provided the columns use the same samples and aggregation weights.

However, Table 3 reports indoor Gen3DSR results of R = 65.4, A = 0.0, and S = 30.1, whereas the stated protocol implies S = 32.7. Similarly, its outdoor results of R = 35.0 and A = 0.0 imply S = 17.5, rather than the reported 14.0. The authors should clarify each column’s result and reconcile the definitions with the reported results.

References:

[1] Wang et al. VGGT: Visual Geometry Grounded Transformer.

[2] Carion et al. SAM 3: Segment Anything with Concepts.

[3] Yang et al. Depth Anything V2.

Questions for authors
  1. What coding-agent capabilities does the benchmark specifically diagnose, and how much of the plugin’s improvement remains when the baseline receives the same pretrained-model outputs?

  2. Are the R/A columns conditional on valid submissions? Please reconcile Table 3 with the stated aggregation protocol and provide per-case results or evaluation code to verify the comparisons.

6 comments
AuthorsOpenReview ↗

Response to R3 YouV

Thanks for your review of our paper and for recognizing our benchmark design and analysis! We hope the following responses address your concerns. Please let us know If you have any further inquiries!

AuthorsOpenReview ↗

Response to R3 YouV (1/2)

[Weakness 1 (a) & Question 1 (a) - LEGO-Bench Diagnosing Coding-Agent Capabilities]

“The benchmark primarily evaluates final reconstruction quality through artifact checks, surface agreement, and image similarity. These measures are relevant to scene reconstruction, but the paper does not sufficiently explain how its benchmark design diagnoses coding-agent capabilities beyond those of general reconstruction systems. Although trajectory analysis provides some insight, the connection between the benchmark’s design and capabilities such as code generation, debugging, and iterative revision remains unclear.”

“What coding-agent capabilities does the benchmark specifically diagnose, …”

[Response]

Thank you for this constructive feedback. Unlike specialist 3D reconstruction pipelines that map images to fixed outputs in a narrow domains, coding agents solve reconstruction as an iterative loop. Accordingly, LEGO-Bench is specifically designed around three agent-centric requirements:

  1. Open-Domain Generalization & Scaling: The LEGO-Bench spans realistic indoor and outdoor scenes with nested complexity to reflect real-world user workflows, while our automated ingestion pipeline allows users to seamlessly convert their own Unreal/LychSim assets into new benchmark cases.
  2. Unified 0–100% Metric Scale: The LEGO-Bench deliberately adopts a bounded 0–100% scale, separating code executability (Validity) from geometric and visual fidelity (Reconstruction & Appearance), to enable clean, linear comparisons across both models and iterative revision steps.
  3. Standardized Agent Infrastructure: Extending Harbor [1], our containerized runtime natively supports diverse models and harnesses under a unified execution environment for reproducible code generation and debugging evaluation.

We have updated (1) Section 2 Related Work in the main text and (2) Appendix I to explicitly highlight these agent-centric design choices.

[1] Harbor Framework Team. Harbor: A framework for evaluating and optimizing agents and models in container environments, 2026.

[Weakness 1 (b) & Question 1 (b) - Ablation on LEGO-Plugin]

“The plugin largely integrates existing perceptual models, including VGGT [1], SAM 3 [2], and Depth Anything V2 [3], with initialization, validation, and rollback mechanisms. The reported plugin-on/off comparison does not distinguish improvements from these additional perceptual priors from improvements attributable to the proposed control mechanisms. A baseline providing the same perceptual outputs to the vanilla agent, together with component-wise ablations, is needed to establish the plugin’s independent contribution.”

“… how much of the plugin’s improvement remains when the baseline receives the same pretrained-model outputs?”

[Response]

Thanks for your suggestion on the ablation study. We provide corresponding ablation on (1) baseline providing the same perceptual outputs to the vanilla agent and (2) component-wise baselines in following Table 1 and in our Appendix B.6.

Table 1. Ablation Study of LEGO-Plugin on an indoor subset (%)

ModelSettingsValidityReconstructionAppearanceFinal Score
GPT-5.6-SolVanilla10016.2814.9115.59
GPT-5.6-SolVanilla with tool outputs10016.8926.7821.84
GPT-5.6-SolLEGO-Plugin without Initialization10009.4119.7214.57
GPT-5.6-SolLEGO-Plugin without Grounded Refinement10029.1331.6030.37
GPT-5.6-SolLEGO-Plugin without Version Control10029.7636.7133.24
GPT-5.6-SolLEGO-Plugin10034.2735.9935.13
AuthorsOpenReview ↗

Response to R3 YouV (2/2)

[Weakness 2 & Question 2 - Table 3 Aggregation Protocol]

“Section 4.2 defines the overall score as (S_i=V_i(R_i+A_i)/2), assigns zero reconstruction and appearance scores to invalid submissions, and averages over all attempted cases. Under this definition, the reported means must satisfy (\overline S=(\overline R+\overline A)/2), provided the columns use the same samples and aggregation weights.

However, Table 3 reports indoor Gen3DSR results of R = 65.4, A = 0.0, and S = 30.1, whereas the stated protocol implies S = 32.7. Similarly, its outdoor results of R = 35.0 and A = 0.0 imply S = 17.5, rather than the reported 14.0. The authors should clarify each column’s result and reconcile the definitions with the reported results.”

“Are the R/A columns conditional on valid submissions? Please reconcile Table 3 with the stated aggregation protocol and provide per-case results or evaluation code to verify the comparisons.”

[Response]

Thank you for catching this. While SS already followed Section 4.2 across all attempted cases, several original Table 3 R/AR/A entries (including Gen3DSR) aggregated over valid-only trials rather than all attempts. We have updated all Table 3 R/AR/A columns to strictly follow Section 4.2 (scoring invalid trials as zero across all attempted cases) so S‾=(R‾+A‾)/2\overline{S} = (\overline{R} + \overline{A})/2 holds throughout. All SS means, standard deviations, and rankings remain completely unchanged.

Reviewer_3S7ZOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This is primarily a benchmark paper that studies how to convert images to 3D via agentic coding, AKA agentic real-to-scene. The main pipeline starts from a single RGB image: the coding agent writes Blender code, renders it, compares it with the input image, and iterates. Finally, it produces an editable and executable scene program. To support the pipeline the paper has a harness system coupled with several specialist vision tools for initialization. Though the method is straight forward, the authors also present a benchmark to evaluate the agentic real2sim at a small sacle (only 208 images and 104 scenes). It compares with prior art like VIGA and benchmarks GPT family.

Critical strengths
  • The author present the analysis of failures in the paper, which may be useful for junior readers.
  • The benchmark and paper also notices the outdoor scene, which is much harder for agentic systems than simple indoor scene.
  • Though very common and straightforward, the paper tries to save the traces of the agent rollout and use them as feedback to improve the harness.
Critical weaknesses
  • The only new stuff this paper introduce is a tiny dataset with 100-200 level instances, beyond that the findings are commonsense in the area and the paper lacks a significant and novel enough technical method.
  • The results quality is not impressive enough, it looks like a standard astra real2sim result.
  • The paper writing has many obvious AI slops, and does not look like a scientific paper, for example in L229 and L248.
Questions for authors
  • This paper mostly is like a CVPR finding paper, though the findings are also trivial, the primary question is: please reclaim the significance of this submission and why it can be a ICLR paper? what is a real contribution?
  • Since the benchmark source data are synthetic, why the scale is only at hundurds level?
  • How are the performance for opensource models (that are really controllable and is reproducible), and what API of GPT the experiments are using? how to make sure there is no hidden quality loss behind the API of these closed source models?
4 comments
AuthorsOpenReview ↗

Response to R4 3S7Z

We thank you for recognizing the value of our benchmark design, our analysis and our failure-motivated harness plugin design. We've provided detailed responses below, and we welcome any additional questions or feedback you may have:

AuthorsOpenReview ↗

Response to R4 3S7Z (1/2)

[Weakness 1 & Question 1 - Significance, Core Contributions, and Fit for ICLR]

“The only new stuff this paper introduce is a tiny dataset with 100-200 level instances, beyond that the findings are commonsense in the area and the paper lacks a significant and novel enough technical method.”

“This paper mostly is like a CVPR finding paper, though the findings are also trivial, the primary question is: please reclaim the significance of this submission and why it can be a ICLR paper? what is a real contribution?”

[Response]

Thank you for your thoughtful feedback regarding the contribution and significance of our work. Rather than proposing a task-specific vision model, we study a core ICLR problem: single-image 3D reconstruction as iterative Image-to-Code synthesis by general-purpose coding agents, focusing on three questions: (1) how faithfully coding agents reconstruct executable 3D scenes, (2) what bottlenecks limit their iterative action–refinement loop, and (3) whether the resulting scene programs are precise enough to represent natural images. Crucially, this capability only became viable around 20 days before submission: while GPT-5.6 agents stalled at 14–15% scores, the newly released GPT-6 jumped to 53.4% indoor and 39.6% outdoor. Simply put, "commonsense" cannot be established within a 20-day window of an emergent capability and our findings actively refute standard intuitions. Each research question directly drives one contribution:

  1. LEGO-Bench targets open-domain agent generalization: Unlike specialist 3D baselines restricted to narrow indoor settings or code benchmarks limited to isolated objects and text prompts, coding agents generalize across open domains. Reflecting practical real-world usage, LEGO-Bench evaluates realistic indoor and outdoor scenes.
  2. LEGO-Plugin turns failure diagnosis into a harness-agnostic method: Directly addressing bottlenecks uncovered in our trajectory analysis, LEGO-Plugin strengthens the ReAct loop while remaining training-free and harness-agnostic.
  3. LEGO-World evaluates scene code as a visual representation: Treating executable programs as structured image representations, we evaluate whether a single reconstructed scene can zero-shot support downstream detection, segmentation, and depth via deterministic readouts.

[Weakness 2 - Performance of Coding Agents]

“The results quality is not impressive enough, it looks like a standard astra real2sim result.”

[Response]

Thanks for pointing out coding agent’s performance on our task. This observation aligns directly with our central finding rather than reflecting a limitation of our work. In LEGO-Bench, even the strongest GPT-6 reaches only 53.4% indoor and 39.6% outdoor scores. Our benchmark is specifically designed to expose bottlenecks in the loop and serve as a diagnostic testbed for future models and harnesses.

[Weakness 3 - AI Slops]

“The paper writing has many obvious AI slops, and does not look like a scientific paper, for example in L229 and L248.”

[Reference in manuscript]

L229: “scene.blend must open and contain renderable geometry and an active camera, scene.glb must be a wellformed, non-empty export, and final.png must be a parseable, non-degenerate image”

L248: “Rather than scoring the agent’s own final.png, the evaluator re-renders … ”

[Response]

Thank you for flagging L229 and L248. We would like to clarify a misunderstanding regarding those lines: the inline filenames (scene.blend, scene.glb, and final.png) are not LLM artifacts, but expected human-written filenames (also explicitly illustrated in Figure 2 Agent Workspace). Specifying concrete target filenames is standard practice in execution-based coding agent benchmarks (e.g., SWE-bench [1], Terminal Bench [2], Cap-X [3], Articarft [4]). That said, we appreciate that inline filenames can read abruptly; we have revised Section 4.2 to introduce our metrics more smoothly.

[1] Jimenez, Carlos E., et al. "Swe-bench: Can language models resolve real-world github issues?." International Conference on Learning Representations. Vol. 2024. 2024.

[2] Merrill, Mike, et al. "Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Fu, Letian, et al. "CaP-X: A framework for benchmarking and improving coding agents for robot manipulation." arXiv preprint arXiv:2603.22435 (2026).

[4] Zhou, Matt, et al. "Articraft: An agentic system for scalable articulated 3d asset generation." arXiv preprint arXiv:2605.15187 (2026).

AuthorsOpenReview ↗

Response to R4 3S7Z (2/2)

[Question 2 - Benchmark Scale]

“Since the benchmark source data are synthetic, why the scale is only at hundurds level?”

[Response]

Thank you for raising this question. LEGO-Bench is intentionally sized at the scale of agent benchmarks for three reasons:

  1. Agentic Rollout Cost & Accessibility: Unlike single-pass vision inference, each LEGO-Bench trial rollouts a multi-turn agentic trajectory. Keeping the benchmark at the ~200-instance scale ensures it remains financially accessible to the research community: even with 208 instances, completing our main-table evaluations required 128 hours and $5,993.76 with Amazon Bedrock service on an AWS P4 EC2.
  2. Human-Verified Curation, Not Random Generation: Although simulator-rendered for exact 3D ground truth, our scenes are not randomly generated. Each scene family enforces (1) strictly nested complexity, (2) verified through physics stability checks, (3) human visual review (Figure 3), and (4) 15 rounds of human asset annotation (Figure 17).
  3. Open-Source Extensibility: We will open-source the entire benchmark generation pipeline (Figure 3) so users can seamlessly plug in their own Unreal/LychSim scenes and custom assets to generate new automatically scored test cases.

We have added a dedicated discussion clarifying these design trade-offs in Appendix D.3.

[Question 3 - Open-source Models]

“How are the performance for opensource models (that are really controllable and is reproducible), and what API of GPT the experiments are using? how to make sure there is no hidden quality loss behind the API of these closed source models?”

[Response]

Thanks for raising this. We have added the two open-source model evaluations in Table 1 to Appendix B.5. Both configurations have low artifact validity under the tested runtime. This finding concerns these models and configurations and does not establish a general limitation of open-source models.

Table 1. Open-Source Model Performance on LEGO-Bench (%).

ModelValidity - IndoorReconstruction - IndoorAppearance - IndoorValidity - OutdoorReconstruction - OutdoorAppearance - Outdoor
DeepSeek V4.1 Flash3.701.471.403.001.130.80
MiniMax M3.1 Flash6.480.682.145.000.481.46

Besides, we provide more detailed documentation of API settings in Appendix I.

AuthorsOpenReview ↗

We would like to hear back from Reviewer 3S7Z

Dear reviewer 3S7Z,

Thank you for your time and efforts in reviewing our paper and responses. As the discussion DDL is approaching, we would like to follow up to see if the response addresses your concerns. We would greatly appreciate the opportunity to discuss with you.

Thank you again for your time and valuable suggestions.

Sincerely, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common reply to all reviewers

Dear reviewers,

We sincerely thank you all for your time and effort in reviewing our paper on 3D scene reconstruction with coding agents. Your insightful comments and constructive feedback have been invaluable in helping us improve our work. We are particularly encouraged by the positive feedback we received on our simulator-grounded benchmark design (R1 Hyx8, R2 HLk7, R3 YouV, R4 3S7Z), clear and easy-to-follow presentation (R1 Hyx8, R2 HLk7), informative trajectory and failure-mode diagnostics (R1 Hyx8, R2 HLk7, R3 YouV, R4 3S7Z), diagnostic-motivated LEGO-Plugin harness design (R1 Hyx8, R2 HLk7, R4 3S7Z ).

We also greatly value the constructive concerns raised, particularly regarding (1) contribution framing and scope, (2) LEGO-Plugin component ablations and broader evaluation. These points have significantly helped refine our paper. Below, we summarize our clarifications to major cross-cutting concerns and outline the adjustments made to the manuscript.

[On agent-centric scope of benchmark design]

We understand the reviewers' concerns regarding the core significance of studying general-purpose coding agents on this task (R3 YouV, R4 3S7Z). We must emphasize that we study a newly emergent capability, where single-image executable 3D scene synthesis only became viable with recent frontier models. We design the benchmark that targets end-to-end scene from diverse, realistic image inputs, reflecting real-world agent usage. We also provide systematic diagnostics and a practical harness layer tailored to coding agent. To make our contribution clearer, we have revised our manuscript to highlight our contribution.

[On LEGO-Plugin ablations]

To verify that LEGO-Plugin's gains stem from its three diagnostic-driven control mechanism (R2 HLk7, R3 YouV), we conducted additional controlled experiments in Table 1 and revised our manuscript in Appendix.

Table 1. Ablation Study of LEGO-Plugin on an indoor subset (%)

ModelSettingsValidityReconstructionAppearanceFinal Score
GPT-5.6-SolVanilla10016.2814.9115.59
GPT-5.6-SolVanilla with tool outputs10016.8926.7821.84
GPT-5.6-SolLEGO-Plugin without Initialization10009.4119.7214.57
GPT-5.6-SolLEGO-Plugin without Grounded Refinement10029.1331.6030.37
GPT-5.6-SolLEGO-Plugin without Version Control10029.7636.7133.24
GPT-5.6-SolLEGO-Plugin10034.2735.9935.13

Below are the highlighted changes in the revised manuscript (marked in blue):

  1. [TEXT] + Revised Abstract, Introduction, Section 3, and Section 6 to tone down novelty framing around LEGO-Anything and LEGO-World (R1 Hyx8)
  2. [TEXT] + Updated Table 1 caption, and Appendix I to clarify agent-centric benchmark design (R3 YouV, R4 3S7Z)
  3. [TEXT] + Expanded Appendix D.3 on benchmark scale/cost trade-offs (R4 3S7Z)
  4. [TEXT] + Expanded Appendix E.2 on avoiding noisy scene-wide item matching in headline metrics (R2 HLk7)
  5. [TEXT] + Unified Table 3's R/A columns to average over all attempts with invalid and failed evaluations scored as zero; clarified Section 4.2 and the caption, and added detailed unconditional aggregates and valid-only diagnostics in Appendix F.3 (R3 YouV)
  6. [EXP] + Added open-weight model evaluations in Appendix B.5 (R4 3S7Z )
  7. [EXP] + Added LEGO-Plugin raw-tool baseline, component-wise ablations, and full-dataset evaluation results in Appendix B.6 (R2 HLk7, R3 YouV)
  8. [EXP] + Added citation, discussion (Appendix A.4), and a comparison against concurrent work AHA3D [1] (Appendix B.7) (R1 Hyx8)

We hope our responses and revisions can help you in finalizing the ratings of our paper. If you have any other questions, please feel free to reply back, and we will answer them asap!

Sincerely,

The Authors of Submission 9679

[1] AHA3D: Real-to-Sim Indoor Scene Generation (https://kevinxu02.github.io/real2sim-indoor-site/).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.