acceptodds
Withdrawn from ICLR 2027

SceneActBench: Can Agents Act on the 3D Scenes They See?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language model (VLM) agents increasingly use tools to act on 3D scenes rather than only describe them. Existing 3D benchmarks score textual responses or single-object operations, leaving agent action on complete multi-object 3D scenes under-evaluated. We present **SceneActBench**, a benchmark for visually conditioned action across five 3D tasks under a unified agent–environment loop. Given PNG images or sampled video frames and, where applicable, supplied 3D assets, an agent acts on a 3D environment. We evaluate each final output against hidden ground truth with task-specific geometric metrics. SceneActBench comprises five tasks built from 210 source instances, yielding 520 task cases including paired input conditions. Every task runs through one fixed agent loop to keep the comparison fair. Across eleven proprietary VLM configurations, Overall scores span 38.6–50.2, and none performs consistently well across tasks. We further analyse where and how failures manifest.

Primary area:
datasets and benchmarks
Keywords:
VLM, Agent, 3D, Blender

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8eGuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce SceneActBench, a benchmark for testing the ability of foundation models to reconstruct objects in 3D scenes from single inputs. The authors evaluate their benchmark on a suite of foundation models.

Critical strengths

The evaluation is thorough (both in main paper and in the Supplementary).

Critical weaknesses

Novelty with respect to related works: The paper is entirely missing discussion of how the benchmark compares to related works. The only mention of related works is three small paragraphs in the Appendix and Table 1 in the main paper. This table, features simple checks and crosses, and there is no mention of the table in the text. As the paper is currently presented, I cannot infer the novelty or utility of this benchmark with respect to related works. If this benchmark is simply a unification of skills that are previously tested in other benchmarks - then it should be stated as such (though the questions of utility remain as these are tasks that are well-covered in other single-task benchmarks).

Lack of Analysis/qualitative examples: As I understand it, the goal of the benchmark is to probe the ability of foundation models to model 3D objects based on visual inputs. If this is the case, then the paper desperately needs some qualitative examples with corresponding analysis. Currently, the analyses are purely restricted to comparing values from a table (and a few isolated qualitative examples in the appendix) - but this gives the community very little insight to the current issues these models encounter. A more thorough analysis, with many qualitative examples, would be far more valuable than many of the details that are currently in the main paper (e.g. the file types of the benchmark).

Significant AI-generated content: The text throughout the paper reads in a very AI-generated manner. The figures in the whole paper have the very typical "AI-generated" style. The textual details in the paper are typical of AI-writing - there is an emphasis on file types and formatting, as opposed to the scientific and technical questions the dataset is attempting to address. The current presentation makes the paper and subsequent benchmark hard to appreciate.

Questions for authors

Please see weaknesses.

Reviewer_9CSZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

SceneActBench evaluates visually conditioned 3D action through five tasks: Layout, Camera, Articulated, Reconstruction, and Dynamic. Agents operate Blender through a shared interface, with outputs scored against hidden geometric ground truth. The benchmark comprises 210 source instances yielding 520 task cases. Eleven VLM configurations achieve reported Overall scores of 38.6–50.2, with consistently low Reconstruction scores and substantial differences in motion recovery.

Critical strengths
  • Evaluating executable 3D outputs against hidden geometry provides a useful complement to answer-based spatial reasoning benchmarks.
  • The shared harness, asset preparation, detailed evaluators, paired input conditions, and failure-stage diagnostics reflect substantial engineering and experimental effort.
Critical weaknesses
  1. The reported Dynamic score contradicts the scoring specification. For Sonnet on low-poly Dynamic, Tables 6 and 8 report no matched movers, MME = 1 in every case, mean LE = 1.330, and a task score of 49.9. Under Equation 17, MME = 1 gives zero motion credit, limiting each case to 50 points. Since mean LE exceeds 1, at least one of the ten cases must also receive zero layout credit and therefore score zero. The mean score consequently cannot exceed 45, contradicting 49.9. This raises concerns about the affected aggregate results.

  2. Model coverage is insufficiently current. The evaluation uses GPT-5.4 and Opus 4.6 but omits representative newer models, including GPT-6 Astra, available before submission. This limits the relevance of the reported capability limitations and failure patterns to current agents.

  3. Important comparisons lack stability validation. Each configuration–case pair is run once; case bootstrap does not measure repeated-run variability. This is consequential for small input-condition differences and closely spaced rankings. Dynamic contains only ten scenes but contributes 20% of Overall, with two scenes driving most of Doubao’s lead over Opus.

  4. Primary scores do not clearly reflect task completion. Layout excludes unmatched targets from its primary error average, potentially rewarding accurate but incomplete outputs. Articulated scores are also similar for Doubao and Opus (59.6 versus 63.7), despite only 13/391 versus 255/391 parts moving. Auxiliary diagnostics expose these differences, but the implications for the primary scores and Overall remain unresolved.

  5. The analysis provides limited explanations of its main observations. It identifies the two scenes driving the leading score gap without sufficiently explaining the behavioral differences responsible. Likewise, the consistently low Reconstruction scores and the effects of multi-object complexity remain underexplored. The paper needs clearer insights into why agents fail and which scene properties expose those failures.

Minor issues include an abstract and takeaway statements that do not clearly communicate concrete findings, repetitive qualification statements, and an overly broad novelty claim that needs more precise distinctions from BlenderGym and GameDevBench.

Questions for authors
  1. Please provide Sonnet’s ten per-scene MME, LE, and normalized scores, reconcile them with Equation 17, and verify the affected Overall scores and comparisons.

  2. Can representative newer models and repeated runs confirm the principal ranking and input-condition conclusions?

  3. Please report the Articulated score obtained by keeping the input object unchanged in every frame. How much do the evaluated models improve over this baseline?

  4. Why does the primary Layout score exclude missing objects? Please check whether penalizing these omissions changes the model ranking.

  5. Using the existing outputs and traces, can you explain the behavioral differences on raceloop and factory, the common Reconstruction failures, and how these patterns vary with object count or scene complexity?

Reviewer_WiDcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SceneActBench proposes an executable benchmark for vision-language model agents acting on 3D scenes through a shared headless-Blender/MCP interface. It contains five tasks: Layout, Camera, Articulated, Reconstruction, and Dynamic. The benchmark is constructed from 210 source instances and 520 task cases, with final GLB or JSON outputs evaluated against hidden 3D ground truth using task-specific geometric metrics. The authors evaluate eleven proprietary VLM configurations and provide failure-stage diagnostics, input-condition comparisons, and interaction-budget analyses.

Critical strengths
  1. Clear executable formulation of 3D action. The benchmark requires agents to transform visual evidence into explicit geometric or animated artifacts, rather than merely producing textual answers. The use of hidden geometric evaluation is appropriate for this goal.

  2. Careful implementation and diagnostic analysis. The paper specifies the task interfaces, output formats, evaluators, budgets, and normalization rules in considerable detail. The failure-stage analysis and paired input-condition experiments are useful for understanding why agents fail, beyond reporting a single aggregate score.

Critical weaknesses
  1. The benchmark’s novelty is largely compositional and insufficiently distinguished from existing work. Most task families are established problems or directly reuse existing resources: Layout and Reconstruction use 3D-FRONT/M3DLayout, Articulated uses samples from the S2O ACD dataset, Camera is a conventional visual camera relocalization problem related to PoseNet and InLoc, and Reconstruction overlaps with Total3DUnderstanding and Pix3D. Dynamic tracking and motion recovery also have close precedents in Kubric/MOVi, HOI4D, and HOT3D. Moreover, BlenderGym and 3DCodeBench already study agentic interaction with Blender or procedural 3D modeling. The main contribution is therefore the packaging of several tasks under one agent loop, rather than a new task or a substantially new data regime. The paper does not provide specialist or non-agent baselines to show that this unification measures a capability absent from existing benchmarks.

  2. The fixed harness makes the leaderboard conditional on a particular agent stack. A common interface controls implementation variance, but it does not isolate the underlying VLM capability. The paper’s own Claude Sonnet comparison shows large stack dependence: the official CLI scores 34.6 overall versus 39.5 under the shared harness, with differences of −20.4 points on multi-view Layout and −22.9 on Dynamic, but +3.4 on Camera and +2.5 on Articulated. Since the two runs also differ in system prompts, context policies, and image handling, this experiment cannot identify the causal source of the gap. Without multiple harnesses, model-specific optimized settings, or a minimal interface baseline, the ranking may reflect compatibility with the authors’ inspect–act–render–compare loop as much as 3D reasoning ability.

  3. The primary scoring and aggregate ranking do not fully measure the stated objectives. Layout ADD-S averages only matched objects and excludes unmatched targets and extra predictions. Reconstruction applies global ICP and then recenters each matched prediction before computing F@5%, removing much of the placement error from the primary score. Dynamic corrects a global translation and ignores extra predicted movers in MME. Count-sensitive and placement diagnostics exist, but they do not contribute to Overall. The aggregate score also depends on arbitrary cutoffs and equal task weights, while the same 100 indoor rooms recur across several tasks and Dynamic contains only ten scenes. Each configuration–case pair is run once, and the top score gaps have bootstrap intervals that include zero; the Doubao lead is concentrated in two Dynamic scenes. Consequently, the leaderboard should be interpreted as a conditional, composition-sensitive result rather than a robust ordering of agents.

Questions for authors
  1. What specific research questions become answerable only through SceneActBench? Beyond consolidating five existing task families, can the authors identify a concrete capability or failure pattern that could not be meaningfully studied using existing benchmarks independently? A clearer demonstration of the scientific value of cross-task evaluation would help establish the benchmark’s contribution.

  2. How should the Overall score be interpreted when the five tasks have substantially different difficulty levels and metric characteristics? Have the authors examined whether the principal conclusions and relative model rankings remain stable under alternative reasonable normalization constants or task-weighting schemes?

  3. Can the authors provide human or task-specialist baselines for at least a representative subset of tasks? Such comparisons would help establish whether the benchmark is appropriately calibrated, whether the tasks are solvable under the available observations and interaction budgets, and how far current VLM agents remain from established task-specific solutions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.