- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes a benchmark dataset that can test coding agent's ability to construct and edit 3D scenes. The paper collects and curates unreal engine scenes to construction the text prompts, the asset catalogue, and scene edit pairs needed to assess output artifacts
Critical strengths
Overall, the need for benchmarking agent's ability to create 3D scenes is well motivated. The overall concept seems plausible.
The image-to-scene editing task seems cleverly constructed and provides a nice way to have ground truth for these tasks.
Critical weaknesses
The paper is impossible to review since all relevant details about the method is in the appendix which is around 30 pages long.
For example, the paper introduces the concept of Semantic verifier in ln 281 as "Semantic alignment is evaluated by the Semantic Verifier at two complementary levels". This is the only mention of Semantic verifier outside of the appendix. This makes the appendix mandatory to understand even the core of the benchmark being introduced. Other examples of this is "Overview alignment", "Ground-Truth-Based Verifier" etc. None of these concepts are explained with any sort of technical rigor and it is not clear how these are reported as a numeric score.
The paper also introduces a whole host of scores, metrics, and other ways to evaluate the benchmark. But there is little in the paper to motivate and analyze the validity of these metrics. For example Figure 5 attempts to evaluate human/benchmark agreement. However, it is never explained what adopted fit, PL fit, or individual fit is. Agreement with Human judgements section overall does not have any details on what user study was done, how it is analyzed, or even interpretation of the metrics provided (does agreeing with 86.90% of Plackett-Luce consensus pairs good? Does achieving Kendall’s τb = 0.738 a good enough alignment with human judgement? What number what you expect if it isn't good enough?)
To summarize, the paper is not written to be undertandable within the given page limit and offers very little analysis of its method or design decisions outside of simply reporting metrics. The insights drawn by evaluating methods against each other on this benchmark might be interesting, but given that it is not possible to understand the benchmark by just reading the main paper, the validity of these conclusions in questionable.
Questions for authors
Please focus the main paper on validating the design of the benchmark and demonstrating the soundness of it. Make sure the paper stands on it's own without the appendix e.g. explain concepts introduced in the main text.
3 comments
Dear Reviewer,
As the discussion period is now have 2 days left, we wanted to kindly follow up and ask whether our rebuttal has addressed your concerns. We would be happy to clarify any remaining questions.
Thank you again for your time and consideration.
The rebuttal has not address my concerns.
"The Semantic Verifier also appears at L142-143, contrary to the assertion that L281 is its only mention." Since the usage quoted is "These demands motivate complementary checks of prompt fulfillment, physical plausibility, and artifact validity, captured by the Semantic Verifier, Physical Safety, and Candidate Integrity, respectively (Sections 2.3 and 2.4)", it does not make what a semantic verifier is any clearer.
It also makes me question if the rebuttal is AI generated without human in the loop, as it completely missed the intent of my question while superficially addressing it. I also did not question the preservation analysis - so I am not sure why the rebuttal brought it up.
None of the points deal at all with my concern that the main paper itself does not work without appendix. As such, I will stay with clear rejection
Response to Reviewer uJ9M
Thank you for recognizing the motivation and the ground-truth editing design.
1. W1 - The main text explains the verifier design; we will make numerical aggregation explicit.
The claim that “all relevant details” are in the appendix overlooks Figure 3 and §2.3 (L279-296), which describe Detailed/Overview Alignment and ground-truth repair, and §2.4 (L306-339), which gives their evidence sources and rationale. The Semantic Verifier also appears at L142-143, contrary to the assertion that L281 is its only mention. We will consolidate the essential decision rules and these equations in §2.4:
T2S = 0.2 Detailed + 0.6 Overview + 0.2 Physics; I2S = 0.8 Repair F1 + 0.2 Physics; Overall = 0.5 mean(T2S) + 0.5 mean(I2S).
App. C.6 gives these weights and the overall aggregation rule (L1389-1395 and Eq. 7). Table 8 lists the scheduled public cases scored zero. App. C.4 defines the clause and family aggregation for Detailed Alignment and the modifier and cap for Overview Alignment. We will include these rules alongside the equations in §2.4.
2. W2 - Human agreement has an empirical reference.
Yes. Across 28 generated T2S scenes, the evaluator agreed with the human consensus on 73 of 84 scene pairs (86.90%, Kendall’s τb = 0.738). We consider this strong agreement with human rankings. Section 3.3 (L413-425), Fig. 5 and Table 16A report these holdout results. Random ordering would agree on 50% of pairs on average (expected τb = 0).
On the same pairs, §3.3 (L419-425) and App. D.7 report 79.37% evaluator–individual agreement versus 80.95% between annotators, a gap of 1.58 percentage points. We would have less confidence if evaluator–individual agreement fell well below that human reference. We did not set a numerical cutoff in advance. As §3.3 explains, earlier weight exploration included the holdout prompts, so this is an internal validation. We will clarify Fig. 5’s labels using App. C.9 and Tables 9 and 16B: adopted is the policy used for reporting, individual-fit fits individual preferences, and PL-fit fits the Plackett–Luce consensus.
A second study on generated scenes points the same way. Six annotators ranked 96 scenes (12 prompts × 8 configurations; 266 triplet rankings), and the evaluator weights stayed frozen. The evaluator’s ranking of the eight configurations closely matches the human ranking (Kendall τb ≈ 0.84), with the same top two and bottom three. At the pair level, evaluator–individual agreement is 72.2% versus 79.4% between annotators. On pairs judged by at least two annotators, the gap is −6.8 pp (95% CI [−14.0, +0.3]), which includes zero. The evaluator also agrees with the human consensus on 73.2% of pairs, well above the 50% expected from random ordering.
3. W3 - The submitted paper already reports the preservation analysis, and the scores are stable across runs.
The main text reports this finding in §3.4 (L467-474) and Fig. 6c. Across 14 public configurations, 76 of 212 submissions (35.8%) repaired every target but also introduced unintended changes. Checking preservation changes the comparison on indoor tasks: Astra and Gemini fully repaired 23 and 19 submissions, respectively. Requiring the rest of the scene to be preserved changes those counts to 16 and 17.
Table 13B reports the sensitivity check: the share with unintended changes was 36.4% at strict tolerance and 35.6% at relaxed tolerance. Reviewer eUoX described the preservation analysis as “convincing.”
We also measured run-to-run variability (see our reply to mvra, W2). For three configurations spanning Table 2, two additional runs on all 95 public tasks changed the Overall score by at most 0.029, every 95% confidence interval included zero, and the three configurations kept their order.