- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduce a Blender-Videobench, a benchmark that tests the agent's ability to reconstruct real-world videos as animated blender scenes. The authors conducted extensive experiments with 51 configurations from 10 model families. Overall, this paper is well-motivated.
The most critical weakness lies in its evaluation protocol. BVB measures only rendered 2D videos rather than directly validating the underlying 3D geometry inside generated .blend scenes. Models can achieve high Latent Similarity by producing visually plausible renders while retaining fundamentally wrong object poses, sizes and camera trajectories in the 3D scene. This creates a reward mismatch: the benchmark optimizes for visual appearance instead of faithful 3D spatio-temporal understanding, contradicting its core claim.
Second, the benchmark uses a per-scene monetary spend cap as the only constraint instead of a fixed token or step budget. Since model pricing varies, identical dollar limits do not guarantee equal computational resources across models, hurting fair comparison. Third, any non-renderable Blender output receives zero scores for all metrics. This rule treats code-generation failures and video-understanding failures identically, conflating Blender coding bugs with genuine scene understanding errors.
Critical strengths
The motivation is very interesting.
Critical weaknesses
The most critical weakness lies in its evaluation protocol. BVB measures only rendered 2D videos rather than directly validating the underlying 3D geometry inside generated .blend scenes. Models can achieve high Latent Similarity by producing visually plausible renders while retaining fundamentally wrong object poses, sizes and camera trajectories in the 3D scene. This creates a reward mismatch: the benchmark optimizes for visual appearance instead of faithful 3D spatio-temporal understanding, contradicting its core claim.
Second, the benchmark uses a per-scene monetary spend cap as the only constraint instead of a fixed token or step budget. Since model pricing varies, identical dollar limits do not guarantee equal computational resources across models, hurting fair comparison. Third, any non-renderable Blender output receives zero scores for all metrics. This rule treats code-generation failures and video-understanding failures identically, conflating Blender coding bugs with genuine scene understanding errors.
Questions for authors
none
2 comments
I appreciate the authors' efforts in addressing my concerns and conducting additional experiments. The newly introduced Scene Test provides useful evidence of the consistency between rendered-video evaluation and underlying 3D scene fidelity. The additional analyses also reasonably address my concerns regarding the monetary spend cap and non-renderable outputs. Nevertheless, Scene Test remains a diagnostic evaluation rather than an integral component of the benchmark, leaving the original limitation in directly assessing 3D scene fidelity. Overall, the revision has substantially alleviated my concerns, and I maintain my Weak Accept recommendation.
We thank the reviewer for finding the motivation interesting and for three concrete concerns. Each is addressed below, with changes in red in the revised PDF. We also rewrote the introduction to state more precisely what BVB measures: the joint ability to understand a video and express it as an executable scene.
1. Only rendered video is evaluated, not the 3D scene
Concern. A model could produce plausible renders while the 3D scene has wrong poses, sizes or camera paths, so the benchmark might reward appearance over understanding.
What we added. Scene Test (new Appendix K) opens each submitted .blend file and checks the 5,130 VSI-Bench facts directly on the scene: object dimensions, world-space bounding boxes, the animated camera trajectory and per-frame ray-cast visibility. Nothing is rendered. A language model only maps each noun in a question to a named object, and pass or fail is computed by fixed code with explicit tolerances (exact match for counts, directions, routes and order; 10% for sizes and room area; 15% for distances).
What it shows. If BVB rewarded appearance at the expense of geometry, configurations with higher Latent Similarity (LS) would not hold more correct facts in the 3D scene. They do. In the GPT-5.6 family (12 configurations, three model tiers, several reasoning efforts), Scene Test correlates with LS at Spearman 0.92, with Dual VQA (DV) at 0.88 and with Overall at 0.94. Across the 43 configurations with complete results, it correlates with DV at 0.40 (p = 0.008).
DV is also not an appearance score: it asks the same spatial and temporal questions on the source and the render. A new human audit (Appendix J) shows that when DV marks a fact as not retained, people reading the same frames find the correct answer in only about one case in five; otherwise the render shows wrong content or no evidence.
Limit. Scene Test finds objects through the names agents give them, which differ across model families, so we report it as a diagnostic rather than a leaderboard metric (Appendix K).
2. Monetary spend cap instead of a token or step budget
Concern. Dollar limits do not give equal compute across models with different prices.
Response. In practice the cap almost never binds, so rankings are not driven by agents being cut off. Mean spend per scene ranges from $0.024 to $2.157, below the $3 cap, and only 10 of the 14,687 scene runs (0.07%) were stopped by the cap, nine of them from a single configuration. Per-scene tokens, turns, frames inspected and runtime are released, so the leaderboard can be re-analysed under a token or step budget. Section 2.1 now states this and notes that prices may change over time.
3. Zero score for non-renderable output conflates coding and understanding
Concern. A run that fails to render scores zero, mixing code failures with understanding failures.
Response. The two are reported separately. Table 3 gives the valid-render rate (OK%) for every configuration: its median is 99.0%, and 41 of the 51 configurations are at or above 97%, so non-renderable output is rare and cannot explain the differences between configurations. Scene Test confirms this: in the GPT-5.6 family, only 0.6% of failed checks come from scenes that could not be evaluated; the rest come from scenes that load normally but whose content is wrong. We keep the zero-score rule because excluding failed scenes would let a configuration raise its average by skipping hard ones, and Section 2.1 now states that OK% separates execution failures from content failures.