acceptodds
Under review as a conference paper at ICLR 2027

Code4Scene: Benchmarking Coding Agents for Constructing and Editing Interactive 3D Scenes

Abstract

Frontier coding agents can now write and execute code that authors 3D environments, but whether they reliably understand 3D structure and precisely control scene state remains unclear. The resulting 3D scene is a persistent, executable artifact: a convincing render can hide incorrect spatial relations, intersecting objects, or unintended modifications. We introduce Code4Scene, a benchmark of 190 Unreal Engine cases built from human-assembled scenes that evaluates coding agents on two complementary settings under a shared execution interface. Construction tests scene-level spatial reasoning from open-ended language specifications, where many realizations are valid; editing tests precise control of scene state, where the agent must recover the target scene from reference images while preserving everything else. Rather than scoring code or rendered views, Code4Scene evaluates the engine-native scene produced by the agent for task fulfillment, artifact integrity, and static physical validity, with edits additionally compared against withheld ground truth. Across 14 coding-agent configurations on the 95-case public set, construction and editing performance are strongly correlated but not interchangeable (Spearman ): Claude Fable 5.1 leads construction, Gemini 3.8 Flash leads editing, and GPT-6 Astra narrowly leads overall. Spatial Composition is the weakest construction category for every agent, while editing remains imprecise: the best Repair F1 is only 0.527, and 35.8% of edits that fully recover the target still introduce unintended changes elsewhere in the scene. These results expose a gap between plausible 3D generation and reliable spatial reasoning and state control.

Primary area:
datasets and benchmarks
Keywords:
Coding Agent, 3D Construction, 3D Scene Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uJ9MOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a benchmark dataset that can test coding agent's ability to construct and edit 3D scenes. The paper collects and curates unreal engine scenes to construction the text prompts, the asset catalogue, and scene edit pairs needed to assess output artifacts

Critical strengths

Overall, the need for benchmarking agent's ability to create 3D scenes is well motivated. The overall concept seems plausible.

The image-to-scene editing task seems cleverly constructed and provides a nice way to have ground truth for these tasks.

Critical weaknesses

The paper is impossible to review since all relevant details about the method is in the appendix which is around 30 pages long.

For example, the paper introduces the concept of Semantic verifier in ln 281 as "Semantic alignment is evaluated by the Semantic Verifier at two complementary levels". This is the only mention of Semantic verifier outside of the appendix. This makes the appendix mandatory to understand even the core of the benchmark being introduced. Other examples of this is "Overview alignment", "Ground-Truth-Based Verifier" etc. None of these concepts are explained with any sort of technical rigor and it is not clear how these are reported as a numeric score.

The paper also introduces a whole host of scores, metrics, and other ways to evaluate the benchmark. But there is little in the paper to motivate and analyze the validity of these metrics. For example Figure 5 attempts to evaluate human/benchmark agreement. However, it is never explained what adopted fit, PL fit, or individual fit is. Agreement with Human judgements section overall does not have any details on what user study was done, how it is analyzed, or even interpretation of the metrics provided (does agreeing with 86.90% of Plackett-Luce consensus pairs good? Does achieving Kendall’s τb = 0.738 a good enough alignment with human judgement? What number what you expect if it isn't good enough?)

To summarize, the paper is not written to be undertandable within the given page limit and offers very little analysis of its method or design decisions outside of simply reporting metrics. The insights drawn by evaluating methods against each other on this benchmark might be interesting, but given that it is not possible to understand the benchmark by just reading the main paper, the validity of these conclusions in questionable.

Questions for authors

Please focus the main paper on validating the design of the benchmark and demonstrating the soundness of it. Make sure the paper stands on it's own without the appendix e.g. explain concepts introduced in the main text.

3 comments
AuthorsOpenReview ↗

Response to Reviewer uJ9M

Thank you for recognizing the motivation and the ground-truth editing design.

1. W1 - The main text explains the verifier design; we will make numerical aggregation explicit.

The claim that “all relevant details” are in the appendix overlooks Figure 3 and §2.3 (L279-296), which describe Detailed/Overview Alignment and ground-truth repair, and §2.4 (L306-339), which gives their evidence sources and rationale. The Semantic Verifier also appears at L142-143, contrary to the assertion that L281 is its only mention. We will consolidate the essential decision rules and these equations in §2.4:

T2S = 0.2 Detailed + 0.6 Overview + 0.2 Physics; I2S = 0.8 Repair F1 + 0.2 Physics; Overall = 0.5 mean(T2S) + 0.5 mean(I2S).

App. C.6 gives these weights and the overall aggregation rule (L1389-1395 and Eq. 7). Table 8 lists the scheduled public cases scored zero. App. C.4 defines the clause and family aggregation for Detailed Alignment and the modifier and cap for Overview Alignment. We will include these rules alongside the equations in §2.4.


2. W2  - Human agreement has an empirical reference.

Yes. Across 28 generated T2S scenes, the evaluator agreed with the human consensus on 73 of 84 scene pairs (86.90%, Kendall’s τb = 0.738). We consider this strong agreement with human rankings. Section 3.3 (L413-425), Fig. 5 and Table 16A report these holdout results. Random ordering would agree on 50% of pairs on average (expected τb = 0).

On the same pairs, §3.3 (L419-425) and App. D.7 report 79.37% evaluator–individual agreement versus 80.95% between annotators, a gap of 1.58 percentage points. We would have less confidence if evaluator–individual agreement fell well below that human reference. We did not set a numerical cutoff in advance. As §3.3 explains, earlier weight exploration included the holdout prompts, so this is an internal validation. We will clarify Fig. 5’s labels using App. C.9 and Tables 9 and 16B: adopted is the policy used for reporting, individual-fit fits individual preferences, and PL-fit fits the Plackett–Luce consensus.

A second study on generated scenes points the same way. Six annotators ranked 96 scenes (12 prompts × 8 configurations; 266 triplet rankings), and the evaluator weights stayed frozen. The evaluator’s ranking of the eight configurations closely matches the human ranking (Kendall τb ≈ 0.84), with the same top two and bottom three. At the pair level, evaluator–individual agreement is 72.2% versus 79.4% between annotators. On pairs judged by at least two annotators, the gap is −6.8 pp (95% CI [−14.0, +0.3]), which includes zero. The evaluator also agrees with the human consensus on 73.2% of pairs, well above the 50% expected from random ordering.


3. W3 - The submitted paper already reports the preservation analysis, and the scores are stable across runs.

The main text reports this finding in §3.4 (L467-474) and Fig. 6c. Across 14 public configurations, 76 of 212 submissions (35.8%) repaired every target but also introduced unintended changes. Checking preservation changes the comparison on indoor tasks: Astra and Gemini fully repaired 23 and 19 submissions, respectively. Requiring the rest of the scene to be preserved changes those counts to 16 and 17.

Table 13B reports the sensitivity check: the share with unintended changes was 36.4% at strict tolerance and 35.6% at relaxed tolerance. Reviewer eUoX described the preservation analysis as “convincing.”

We also measured run-to-run variability (see our reply to mvra, W2). For three configurations spanning Table 2, two additional runs on all 95 public tasks changed the Overall score by at most 0.029, every 95% confidence interval included zero, and the three configurations kept their order.

Dear Reviewer,

As the discussion period is now have 2 days left, we wanted to kindly follow up and ask whether our rebuttal has addressed your concerns. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

The rebuttal has not address my concerns.

"The Semantic Verifier also appears at L142-143, contrary to the assertion that L281 is its only mention." Since the usage quoted is "These demands motivate complementary checks of prompt fulfillment, physical plausibility, and artifact validity, captured by the Semantic Verifier, Physical Safety, and Candidate Integrity, respectively (Sections 2.3 and 2.4)", it does not make what a semantic verifier is any clearer.

It also makes me question if the rebuttal is AI generated without human in the loop, as it completely missed the intent of my question while superficially addressing it. I also did not question the preservation analysis - so I am not sure why the rebuttal brought it up.

None of the points deal at all with my concern that the main paper itself does not work without appendix. As such, I will stay with clear rejection

Reviewer_mvraOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Code4Scene, a benchmark for evaluating whether coding agents can reliably construct and edit persistent, executable 3D scenes in Unreal Engine. Rather than focusing on whether the generated code runs successfully or whether the rendered output merely “looks plausible,” the benchmark evaluates whether the underlying scene state within the engine actually satisfies the task requirements, including whether spatial relationships are correct, whether objects exhibit collisions or unintended floating, and whether the editing process inadvertently alters content that should remain unchanged.

Critical strengths
  • The presentation is clear and well organized.
  • The conclusions are insightful and informative.
Critical weaknesses
  • GPT-6 Astra, Grok 4.6, Claude Opus 5, and Claude Fable 5.1 are evaluated only on the 95 public cases. Thus, the claim that GPT-6 Astra performs best is not validated on the full 190-case suite or hidden tasks.
  • Results are mainly based on single or limited runs, without multiple random seeds or variance estimates. It is therefore unclear whether performance differences are robust to generation and tool-execution randomness.
  • The paper does not clearly report the number of annotators, judgments per annotator, participant background, or recruitment procedure. The small holdout (four prompts) and lack of Image-to-Scene human evaluation further limit the reliability of the study.
  • Code4Scene focuses on a fixed Unreal Engine environment with predefined assets and tools, without evaluating asset generation, cross-engine generalization, dynamic physical interaction, or long-horizon planning. Thus, it captures only a limited subset of general 3D capabilities.

I would be willing to raise my score if the authors can adequately address my concerns during the rebuttal.

Questions for authors

pls see the weakness part.

2 comments
AuthorsOpenReview ↗

Response to Reviewer mvra

Thank you for describing the presentation as “clear and well organized” and the conclusions as “insightful and informative.”

1. W1 - Do public-set rankings reflect the full suite?

Table 2 reports 14 configurations on the public set and ten on both sets. For those ten, §2.2 (L247-255) reports r = 0.998 between public and full-suite overall scores (App. A.3, Table 5). We also compared the order of all 45 configuration pairs.

ComparisonPairs in the same order
Public vs. full suite44/45
Public vs. private44/45

The pair that reversed between the public and full-suite rankings had a full-suite gap of 0.004. Public and private overall scores correlate at r = 0.988. We limit ranking claims for Astra and the other three public-only configurations to the public set. After the material-artifact correction, Astra leads Gemini on public construction (0.724 vs. 0.657), Gemini leads on public editing (0.587 vs. 0.524), and their public overall scores are close (0.624 vs. 0.622).


2. W2 - How sensitive are results to run variability?

The reported intervals measure variation across source scenes (App. D.1). To measure run variability, we reran each of three configurations twice on all 95 public tasks. We selected Gemini 3.8 Flash, GPT-5.6 SOL and DeepSeek V4.1 Flash from the top, middle and bottom of Table 2 and used the same model names, reasoning effort, task limits and scoring as in the submission.

The submitted runs were made in September, about two months before the reruns. Hosted API providers can change the checkpoint, serving stack or routing under the same model name, so we cannot verify that the September runs used the same model state. We therefore compare the two new reruns directly. We interleaved them by task in the same campaign, using identical harness code, agent-client versions and model providers. This measures run-to-run variability within one campaign and reduces the influence of serving changes over time.

ConfigurationOverall gap, rerun 2 - rerun 195% CI
Gemini 3.8 Flash−0.023[−0.070, +0.021]
GPT-5.6 SOL−0.029[−0.091, +0.027]
DeepSeek V4.1 Flash+0.011[−0.087, +0.111]

All three absolute gaps are below 0.03, and every 95% CI includes zero (paired bootstrap over tasks, 10,000 resamples). The gaps differ in sign and show no clear evidence of systematic drift between reruns. Both reruns preserve the Table 2 order, and the smallest gap between configurations (0.077) exceeds every absolute rerun gap.

Astra outscored Gemini on 18 of 20 public construction tasks. Dropping each task in turn leaves a mean construction lead of 0.058–0.108. We report construction and editing separately and do not treat the 0.002 overall gap as a reliable lead.

Repeating all 2,280 configuration–task evaluations would cost about USD 3,500 in tokens (Fig. 1b, App. D.5), so we reran three configurations spanning Table 2. Our suite covers 190 tasks, versus 50 scenes in 4DBuildBench and 65 scenarios in CutsceneBench (§2.2).


3. W3 + W4 - Human study, ground truth and long-horizon planning

W3. Seven graduate and undergraduate students provided 336 T2S triplet rankings (48 each on average) across 16 prompts and seven configurations, with three annotators per triplet (App. C.9). On the holdout (28 scenes, 84 pairs), evaluator–individual agreement was 79.37%, versus 80.95% between annotators (§3.3, Fig. 5, App. D.7). We will add recruitment and workload details to §3.3.

In a second study, six annotators provided 266 triplet rankings over 96 scenes (12 prompts × 8 configurations). The evaluator weights stayed frozen. The evaluator’s ranking closely matches the human ranking (Kendall τb ≈ 0.84), with the same top two and bottom three in the same order. Pair-level agreement was 72.2% between the evaluator and individual annotators versus 79.4% between annotators. On pairs judged by at least two annotators, the gap was −6.8 pp (95% CI [−14.0, +0.3] pp). Each annotator agreed with the others’ consensus on 79–83% of pairs. We will add this study to §3.3 and App. D.7.

For I2S, reviewer uJ9M called the editing setup “a nice way to have ground truth for these tasks.” We compare the saved scene with the original object by object, checking both what was repaired and what should stay untouched (§2.3, Fig. 3).

W4. Fig. 4 shows how Fable planned zones, roads and a water channel before building, found 48% of actors colliding at step 74, merged structures at step 96 (3,598 → 1,055 actors), and saved a 1,138-actor scene at step 122 before its final view at step 125. Fable averaged 85.2 tool calls per public construction run (Table 14). As stated in Limitations, Code4Scene evaluates static scene construction and editing in Unreal Engine. Asset generation, cross-engine transfer and dynamic interaction are outside its scope.

Dear Reviewer,

As the discussion period is now have 2 days left, we wanted to kindly follow up and ask whether our rebuttal has addressed your concerns. We would be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Reviewer_eUoXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Code4Scene benchmarks coding agents on constructing and editing Unreal Engine scenes. It contains 190 cases covering text-guided construction and image-guided repair. The evaluation checks saved scene integrity, semantic alignment, static physical validity, and actor-level repair accuracy, including unintended changes to unrelated content. The authors evaluate 14 agent configurations on the public set and ten on the private set, with additional human evaluation, scoring-repeatability tests, and sensitivity analyses. The main finding is that agents often fail to preserve the surrounding scene even when they successfully recover all repair targets.

Critical strengths
  1. The benchmark addresses a meaningful evaluation gap. It evaluates persistent Unreal scenes, including dependencies, physical defects, and unintended edits. These checks capture failures that successful code execution or plausible screenshots can miss. The distinction between open-ended construction and exact scene repair is also well motivated.

  2. The preservation analysis is convincing. Among 212 public submissions that recover all targets, 76 still introduce unintended changes. Table 13 shows that this finding persists under different repair tolerances, exclusion of asset/material-only changes, and private-task evaluation. This is the paper’s strongest result: completing the requested repair does not establish control over the rest of the scene.

Critical weaknesses
  1. The official results retain a known scoring error. Appendix D.5 reports that re-saving the laboratory-loft scene changes a runtime material-instance name, causing an untouched actor to be counted as modified. The authors remove this false positive from the failure analysis but retain it in the official Precision/F1 results in Tables 2 and 12. This affects 41 public submissions. Since some tasks contain few repair targets, one spurious false positive can noticeably change their scores. The official results should be corrected, with the effect on rankings reported.

  2. The evaluator validation is limited. The adopted weights were explored using all 16 human-study prompts, so the four-prompt retrospective holdout is not independent. Human annotators also see the same overview images used by Overview Alignment, which contributes 60% of the construction score. This provides limited validation of fine-grained requirement judgments and hidden scene-state checks. The repeatability study fixes evidence and routing and repeats Detailed Alignment only. These limitations are disclosed, but they remain consequential for a paper whose central contribution is an evaluation protocol.

Questions for authors

Q1. After consistently removing the runtime-material false positive from official scoring, how do Tables 2 and 12 change? Does the correction affect any main comparison or ranking?

Q2. What independent checks establish the accuracy of the automatically generated requirement graphs and the resulting spatial judgments? Can the authors provide audit results and family-level Known Coverage? This would help distinguish actual spatial failures from unresolved evaluation evidence, particularly since the conditional spatial analysis retains only 8.3% of the audited spatial predicates.

2 comments
AuthorsOpenReview ↗

Response to Reviewer eUoX

Thank you for recognizing the evaluation gap and the robustness of the preservation analysis.

1. W1 + Q1 - Effect of the scoring correction.

App. D.5 (L1880-1885) identifies the scoring discrepancy in the submission. Re-saving the laboratory-loft scene in Unreal renames a material even when the agent leaves the object untouched. Table 13 excluded this false positive from the preservation analysis. Tables 2 and 12 counted it as an unintended edit.

We removed the false positive from all 88 affected submissions (41 public, 47 private) and recomputed the scores. Repair F1 increased in 34 submissions (16 public, 18 private). The table shows how this affected the public scores for Astra and Gemini.

Public metricAstra: before → afterGemini: before → after
Repair F10.4445 → 0.45610.5273 → 0.5344
I2S score0.5149 → 0.52410.5811 → 0.5867
Overall0.6193 → 0.62400.6190 → 0.6218

The public I2S and overall rankings and the private overall ranking are unchanged. Table 13 already excluded the false positive, so its result remains 76/212 (35.8%).


2. W2 + Q2 - Known Coverage, the 8.3% filter, and an audit of requirements and verdicts.

App. C.4 (L1237-1241) defines Known Coverage as the share of requirements with enough evidence for a definite decision, aggregated using the applicable family weights. For this, we computed it from the per-requirement records of the 268 valid public T2S runs.

Requirement familyKnown Coverage
Identity / environment98.6%
Content / quantity91.2%
Attributes / materials87.2%
Spatial composition95.6%
Family-weighted overall93.2%

Fig. 6a and App. C.4 (L1242-1250) address a narrower question: how often is a spatial relation wrong when its objects are confirmed present? Only 369/4,455 predicates (8.3%) met that filter. In the underlying records, 98.4% of all 4,455 received a direct Match/Mismatch verdict.

Because a definite verdict can still be wrong, we audited both stages of Detailed Alignment on a stratified sample of public T2S runs across all 14 configurations:

AuditSampleResult
Requirement extraction: is each generated requirement faithful to its prompt sentence?275 requirements from 30 prompt sentences275/275 correct; 2 sentences missed some content
Verdicts: does the requirement hold in the frames the evaluator used?80 verdicts (30 spatial, 20 content, 20 attribute, 10 identity; half Match, half Mismatch)74/79 agree (93.7%); 1 undecidable from the frames
- Spatial composition only3027/29 agree (93.1%)

The audit supports the reported spatial failures as genuine errors rather than unresolved evidence.


3. W2 - State the scope of human validation precisely.

Section 3.3 (L413–425), Fig. 5 and App. D.7 (Table 16) report agreement across 28 generated scenes (84 within-prompt pairs) in the T2S holdout: 79.37% between the evaluator and individual annotators, versus 80.95% between annotators. Earlier weight exploration used all 16 prompts, and annotators judged overview images (App. C.9). Fig. 18 tests Detailed Alignment repeatability with the evidence held fixed. For editing, Tables 13B and 13C separately report the tolerance, asset/material and private-task checks.

To test beyond the holdout prompts, we kept the evaluator weights frozen and compared them against a second study: six annotators ranked 96 generated scenes from eight configurations. The evaluator’s model ranking closely matches the human one (τb ≈ 0.84), and its pair-level agreement with individual annotators is not significantly below the agreement between annotators.

Thank you for the detailed clarifications and additional results. My concerns have been addressed, and I have no further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.