acceptodds
Under review as a conference paper at ICLR 2027

BVB: Benchmarking Agentic Video Understanding via Programmatic Reconstruction in Blender

147 views

Abstract

Multimodal agents can create complex videos in software such as Blender by writing code instead of using diffusion models. Yet video understanding benchmarks still evaluate models mainly through question answering. *If an agent truly understands a video, it can reconstruct it programmatically.* We introduce **BVB**, Blender-VideoBench, a benchmark that tests this ability by asking agents to reconstruct real-world videos as animated Blender scenes. To ensure fair comparison, each agent programs the reconstruction through a lightweight harness, **Mini-BVB**, in an identical sandbox under a shared cost limit. The benchmark renders each reconstruction from its animated camera and evaluates it on two axes: (1) **Dual VQA** measures how many spatiotemporal facts the reconstruction preserves. (2) **Latent Similarity** measures how closely the reconstruction matches the source video perceptually. Our overall score, a square-root mean, favors balanced performance. We evaluate 51 configurations from 10 model families and analyze semantic retention, perceptual similarity, reasoning effort, and cost. The best model reaches 88.6 Latent Similarity but retains only 53.7% of the spatiotemporal facts from the source video. Additional reasoning improves perceptual similarity but does not close this gap. In a blind study with 15 raters and five configurations, Latent Similarity correlates strongly with human preference. These results show that programmatic reconstruction is a viable test of agentic video understanding, and that semantic retention remains the main challenge.

Primary area:
datasets and benchmarks
Keywords:
Agentic Video Understanding, Multimodal Agents, Programmatic Reconstruction, Video Understanding Benchmarks, Spatiotemporal Reasoning, Blender

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LrJjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduce a Blender-Videobench, a benchmark that tests the agent's ability to reconstruct real-world videos as animated blender scenes. The authors conducted extensive experiments with 51 configurations from 10 model families. Overall, this paper is well-motivated.

The most critical weakness lies in its evaluation protocol. BVB measures only rendered 2D videos rather than directly validating the underlying 3D geometry inside generated .blend scenes. Models can achieve high Latent Similarity by producing visually plausible renders while retaining fundamentally wrong object poses, sizes and camera trajectories in the 3D scene. This creates a reward mismatch: the benchmark optimizes for visual appearance instead of faithful 3D spatio-temporal understanding, contradicting its core claim.

Second, the benchmark uses a per-scene monetary spend cap as the only constraint instead of a fixed token or step budget. Since model pricing varies, identical dollar limits do not guarantee equal computational resources across models, hurting fair comparison. Third, any non-renderable Blender output receives zero scores for all metrics. This rule treats code-generation failures and video-understanding failures identically, conflating Blender coding bugs with genuine scene understanding errors.

Critical strengths

The motivation is very interesting.

Critical weaknesses

The most critical weakness lies in its evaluation protocol. BVB measures only rendered 2D videos rather than directly validating the underlying 3D geometry inside generated .blend scenes. Models can achieve high Latent Similarity by producing visually plausible renders while retaining fundamentally wrong object poses, sizes and camera trajectories in the 3D scene. This creates a reward mismatch: the benchmark optimizes for visual appearance instead of faithful 3D spatio-temporal understanding, contradicting its core claim.

Second, the benchmark uses a per-scene monetary spend cap as the only constraint instead of a fixed token or step budget. Since model pricing varies, identical dollar limits do not guarantee equal computational resources across models, hurting fair comparison. Third, any non-renderable Blender output receives zero scores for all metrics. This rule treats code-generation failures and video-understanding failures identically, conflating Blender coding bugs with genuine scene understanding errors.

Questions for authors

none

2 comments

We thank the reviewer for finding the motivation interesting and for three concrete concerns. Each is addressed below, with changes in red in the revised PDF. We also rewrote the introduction to state more precisely what BVB measures: the joint ability to understand a video and express it as an executable scene.

1. Only rendered video is evaluated, not the 3D scene

Concern. A model could produce plausible renders while the 3D scene has wrong poses, sizes or camera paths, so the benchmark might reward appearance over understanding.

What we added. Scene Test (new Appendix K) opens each submitted .blend file and checks the 5,130 VSI-Bench facts directly on the scene: object dimensions, world-space bounding boxes, the animated camera trajectory and per-frame ray-cast visibility. Nothing is rendered. A language model only maps each noun in a question to a named object, and pass or fail is computed by fixed code with explicit tolerances (exact match for counts, directions, routes and order; 10% for sizes and room area; 15% for distances).

What it shows. If BVB rewarded appearance at the expense of geometry, configurations with higher Latent Similarity (LS) would not hold more correct facts in the 3D scene. They do. In the GPT-5.6 family (12 configurations, three model tiers, several reasoning efforts), Scene Test correlates with LS at Spearman 0.92, with Dual VQA (DV) at 0.88 and with Overall at 0.94. Across the 43 configurations with complete results, it correlates with DV at 0.40 (p = 0.008).

DV is also not an appearance score: it asks the same spatial and temporal questions on the source and the render. A new human audit (Appendix J) shows that when DV marks a fact as not retained, people reading the same frames find the correct answer in only about one case in five; otherwise the render shows wrong content or no evidence.

Limit. Scene Test finds objects through the names agents give them, which differ across model families, so we report it as a diagnostic rather than a leaderboard metric (Appendix K).

2. Monetary spend cap instead of a token or step budget

Concern. Dollar limits do not give equal compute across models with different prices.

Response. In practice the cap almost never binds, so rankings are not driven by agents being cut off. Mean spend per scene ranges from $0.024 to $2.157, below the $3 cap, and only 10 of the 14,687 scene runs (0.07%) were stopped by the cap, nine of them from a single configuration. Per-scene tokens, turns, frames inspected and runtime are released, so the leaderboard can be re-analysed under a token or step budget. Section 2.1 now states this and notes that prices may change over time.

3. Zero score for non-renderable output conflates coding and understanding

Concern. A run that fails to render scores zero, mixing code failures with understanding failures.

Response. The two are reported separately. Table 3 gives the valid-render rate (OK%) for every configuration: its median is 99.0%, and 41 of the 51 configurations are at or above 97%, so non-renderable output is rare and cannot explain the differences between configurations. Scene Test confirms this: in the GPT-5.6 family, only 0.6% of failed checks come from scenes that could not be evaluated; the rest come from scenes that load normally but whose content is wrong. We keep the zero-score rule because excluding failed scenes would let a configuration raise its average by skipping hard ones, and Section 2.1 now states that OK% separates execution failures from content failures.

I appreciate the authors' efforts in addressing my concerns and conducting additional experiments. The newly introduced Scene Test provides useful evidence of the consistency between rendered-video evaluation and underlying 3D scene fidelity. The additional analyses also reasonably address my concerns regarding the monetary spend cap and non-renderable outputs. Nevertheless, Scene Test remains a diagnostic evaluation rather than an integral component of the benchmark, leaving the original limitation in directly assessing 3D scene fidelity. Overall, the revision has substantially alleviated my concerns, and I maintain my Weak Accept recommendation.

Reviewer_3D1cOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces BVB, a benchmark for programmatic video reconstruction in Blender. Instead of relying solely on conventional video question answering, BVB requires agents to reconstruct real-world videos as executable 3D scenes. The benchmark evaluates reconstruction quality using two metrics: Dual VQA, which measures the preservation of spatiotemporal information, and Latent Similarity, which assesses perceptual similarity. The authors evaluate 51 model configurations across 10 model families under a standardized, cost-controlled harness. The results reveal a substantial gap between perceptual similarity and semantic retention, highlighting the limitations of current multimodal agents in reconstructing video content faithfully.

Critical strengths
  1. The paper investigates a relatively novel and interesting task, offering a different perspective on evaluating agentic video understanding.
  2. Evaluating reconstruction quality from both VQA and latent similarity perspectives is reasonable, as these metrics capture complementary aspects of video understanding. The inclusion of computational costs in the analysis is also valuable.
  3. The paper is clearly presented. The illustrations are informative, and the key findings are generally easy to understand.
  4. The paper provides comprehensive coverage of relevant literature.
Critical weaknesses
  1. The current evaluation relies on a single vision encoder (V-JEPA 2.1). It would be interesting to investigate whether combining multiple complementary vision encoders could provide a more robust perceptual similarity metric, similar to recent approaches in visual generation evaluation that use multiple feature representations rather than relying solely on traditional metrics such as FID.
  2. The paper seems to lack a detailed analysis of individual agent trajectories. For example, what types of reasoning processes, action sequences, or model behaviors tend to produce higher BVB scores? Such an analysis could provide additional insights into what makes an agent successful at video reconstruction, although I acknowledge that access to reasoning traces may be limited by the use of hidden CoT in some models.
  3. How much would the results change if different agent harnesses were used? In particular, how sensitive are the BVB scores and model rankings to the choice of harness?
Questions for authors
  1. What are the inherent limitations of reconstructing real-world videos using Blender? Is there a practical upper bound on reconstruction fidelity imposed by this approach, and for which aspects?
  2. Table 2 shows that BVB rankings agree with human preferences for models with relatively large differences in their overall rankings. However, how well does BVB correlate with human preferences among closely ranked frontier models (e.g., models ranked within the top 10)? It would be valuable to investigate whether the strong correlation observed in the current study still holds for most frontier models.
2 comments

We thank the reviewer for the positive assessment and constructive suggestions. Changes are in red in the revised PDF.

W1: A single vision encoder for Latent Similarity

What we did. We recomputed perceptual similarity on the existing renders of eight configurations spanning the leaderboard, from GPT-6-Astra-high to Qwen3.5-27B, with two image encoders trained with different objectives: DINOv2 ViT-L/14 (self-supervised) and SigLIP2 (language-supervised). Both use the same 64 frames per video and the same zero score for failed renders (new table in Appendix F).

Result. The ranking is stable across encoders. Spearman correlation with V-JEPA 2.1 is 0.98 for DINOv2 (p < 0.001) and 0.88 for SigLIP2 (p = 0.004), and GPT-6-Astra-high ranks first under all three. On the five configurations of the human study, DINOv2 reproduces the human order exactly, as V-JEPA does; SigLIP2 swaps only two adjacent configurations whose scores differ by 0.4 points.

Why we keep V-JEPA 2.1. Absolute values are not comparable across feature spaces. V-JEPA is a video encoder whose spatial feature maps also capture layout and motion, whereas the image encoders compare global descriptors averaged over time. Combining several encoders into one score remains future work, as the limitations now note.

W2: Analysis of agent trajectories

As the reviewer notes, several models hide their reasoning, so it cannot be analysed uniformly. The harness records, for every scene, the turns, frames inspected, commands run, tokens, cost and runtime, and these logs are released. Section 4.4 and Appendix H use them to show that more reasoning effort raises LS more reliably than DV, and that longer runtime does not mean a higher Overall. A finer analysis of action sequences is a promising direction for future work.

W3: Sensitivity to the harness

Mini-BVB deliberately exposes only two actions, viewing frames and running code, so that differences reflect the model rather than the scaffolding. We have not evaluated other harnesses. The limitations now state that rankings are specific to Mini-BVB, and the harness is released so that others can test alternatives under the same scoring.

Q1: Limits of reconstructing real videos in Blender

The main limit is appearance. Agents may use only untextured primitives, so textures, materials and fine shapes cannot be reproduced, which caps Latent Similarity below a photorealistic reconstruction. The facts that DV tests, such as counts, relative positions, routes and appearance order, can be expressed with primitives. The new human audit (Appendix J) supports this: on the judge's own frames, people confirmed 68% of GPT-6-Astra-high's DV passes, and a judge misreading a correct primitive render explains only about one in five DV failures. We have not measured an explicit upper bound, such as a reconstruction built from ground-truth 3D annotations, and the limitations now say so.

Q2: Agreement with people among frontier models

We agree that the preference study (ranks 2, 16, 21, 31 and 46) supports comparisons between configurations that differ substantially, not among closely ranked frontier models, and Section 4.3 now says so. Three results bear on the top of the leaderboard:

  • The human audit includes the two top-ranked configurations at the level of individual questions. For GPT-6-Astra-high, people confirmed 68% of its DV passes and 59% of its DV failures, and across the audit the judge agrees with a person about as well as two people agree with each other (Cohen's κ 0.36 versus 0.37).
  • All three encoders rank GPT-6-Astra-high above GPT-5.6-Sol-xhigh.
  • The paired bootstrap in Appendix C shows that the gap between these two configurations is significant for Overall but not for DV.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to revise the original paper and address some of my concerns, particularly the new evaluation with DINOv2 and SigLIP2. The clarifications regarding the representational limitations of Blender and the additional human audit are also helpful.

My remaining concerns mainly relate to the sensitivity of model rankings to the agent harness and the agreement between BVB rankings and human preferences among closely ranked frontier models. I understand that evaluating alternative harnesses may be beyond the scope of the current work.

Overall, I find the rebuttal constructive and believe it strengthens the paper. I am inclined to maintain my positive assessment.

Reviewer_zxyzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BVB evaluates multimodal agents by asking them to reconstruct real indoor videos as executable, editable Blender scenes. Mini-BVB supplies frame inspection and command execution under a common $3 per-scene budget, with non-renderable outputs scored zero. Dual VQA (DV) measures the fraction of questions a fixed judge answers correctly on the source that it also answers correctly on the reconstruction. Latent Similarity (LS) compares frozen V-JEPA video features, and a square-root mean combines the two metrics into Overall. The study covers 288 scenes, 5,130 questions, and 51 configurations from 10 model families. Validation includes task, effort and cost analyses, paired bootstrap intervals for the leading pair, and a 15-rater blind study of five configurations that supports LS's relative perceptual ordering.

Critical strengths
  1. An interesting task with practical value in its own right. Reconstructing a video as an executable, editable Blender scene is a useful goal for programmatic graphics and multimodal agents, independently of whether it serves as a test of genuine video understanding. The inspectable output, common harness, asset restrictions and spend cap provide a concrete basis for comparing agents (§2.1, Figure 2 and Appendix A). This practical contribution is the main reason I find the work interesting.

  2. Initial evidence that the benchmark supports useful comparisons. Table 3 provides a substantial baseline across 51 configurations. The blind study reports a Spearman correlation of 0.83 between LS and human preference across 24 scenes and five configurations, supporting its relative perceptual use within that sample (Appendix I). The paired scene bootstrap also quantifies uncertainty for the leading pair and explicitly avoids claiming a significant DV advantage (Appendix C, Table 4). These results give the proposed evaluation practical comparative value.

Critical weaknesses
  1. The link between reconstruction performance and video understanding is not established. I am not convinced by the introductory premise that an agent that truly understands a video can reconstruct it programmatically (p. 2, lines 60-65). Understanding alone does not guarantee successful implementation under a fixed budget and primitive-only constraints. Appendix K acknowledges dependence on coding ability and Blender familiarity, but the main conclusions still attribute failures to video understanding. The same appendix explicitly states that both metrics evaluate rendered video rather than underlying 3D geometry. Thus, the benchmark measures a useful joint task, but the reported evidence does not establish scene correctness beyond the evaluated renders or isolate understanding from implementation ability. This limits the claimed diagnosis of which understanding abilities remain unsolved.

  2. Metric validation does not establish factual or geometric reconstruction correctness. DV=53.7% means 981/1,827 source-correct questions remain correctly answered by the same judge; that subset covers 35.6% of all questions (§3.1, §4.1). Source conditioning does not distinguish an omitted fact from a preserved fact misread in a primitive render. Appendix E attributes lower accuracy to information loss without independent question-level adjudication; Appendix G's recovery analysis uses the same judge. LS has positive evidence for perceptual ranking, but the human study covers only 24 of 288 scenes and five of 51 configurations, at leaderboard ranks 2, 16, 21, 31 and 46 (Table 2). This supports broad quality comparisons within the sample, while leaving generalization and discrimination among closely performing leaders untested. Appendix I appropriately treats the 120 scene-model observations as clustered and descriptive, but reports neither inter-rater agreement nor uncertainty for the human correlations/rankings. Moreover, preference agreement does not calibrate absolute factual or geometric accuracy, and 88.6 LS and 53.7 DV do not share a calibrated correctness scale. The current evidence therefore does not justify treating their numerical gap as a capability gap or the DV deficit as a measured fraction of lost scene facts.

Questions for authors
  1. What evidence separates video-understanding failures from coding or Blender limitations, supporting the interpretation of BVB as a test of understanding (Appendix K)?

  2. Beyond the preference study covering 24 scenes and five configurations, what validation supports DV's factual-retention interpretation and LS's reliability across the benchmark?

If my concerns are adequately addressed, I would be willing to raise my score.

2 comments

We thank the reviewer for a careful and specific review. We agree with both central points: the submission attributed reconstruction failures to video understanding more strongly than the evidence allowed, and it read the Dual VQA (DV) deficit as a measured share of lost scene facts. The revision (changes in red) responds in three ways:

  1. Narrower claim. BVB is now presented as a joint test of video understanding and its programmatic expression, and every sentence that attributed failures to understanding alone has been revised.
  2. Human validation of DV on the judge's own frames (new Appendix J).
  3. Checks beyond the renders: facts verified directly in the submitted 3D scene (Scene Test, new Appendix K), and LS recomputed with two other encoders (Appendix F).

W1 / Q1: Understanding versus implementation

What we changed. The introduction no longer claims that an agent that truly understands a video can reconstruct it. It now says that reconstruction requires one scene consistent with everything observed, and that success also depends on coding ability and familiarity with Blender, a dependence previously stated only in the limitations. In the contributions and conclusion, "which video understanding abilities remain unsolved" becomes "which spatiotemporal facts agents fail to carry into a reconstruction".

What the evidence can and cannot separate. We cannot fully separate misperceiving a video from building it wrongly, but the revision rules out the two other explanations for a low score:

  • Code that does not run. The valid-render rate (OK% in Table 3) has a median of 99.0%, and 41 of 51 configurations are at or above 97%. In the GPT-5.6 family, only 0.6% of failed Scene Test checks come from scenes that could not be evaluated.
  • Renders the judge cannot read. When people could read an answer from a failed render, they chose the judge's own wrong option in 23 of 25 cases, against about 9 expected by chance. The scene itself is wrong.

Whether a wrong scene comes from perception or construction remains open, and Appendix J states this limit.

Beyond the renders. Scene Test (Appendix K) opens each submitted .blend file and checks the 5,130 facts on object dimensions, world-space bounding boxes, the animated camera trajectory and per-frame ray-cast visibility, with fixed tolerances and without rendering. A language model only maps each noun to a named object. Because this mapping depends on how agents name objects, we use Scene Test as a configuration-level diagnostic, not a leaderboard metric. It agrees with DV across 43 configurations (Spearman 0.40, p = 0.008). Within the GPT-5.6 family, whose 12 configurations share naming conventions, it agrees with Overall, DV and LS at 0.94, 0.88 and 0.92. Configurations that score higher on the render-based metrics therefore also hold more correct facts in the 3D scene.

W2 / Q2: What DV and LS measure

Omitted fact or misread render? (new Appendix J). People answered the same questions on the exact 16 frames the judge saw, without knowing the configuration, the judge's answer or the DV outcome. We sampled 164 (configuration, question) units from 27 scenes, covering three configurations (ranks 1, 2 and 46) and the five question types people can answer by eye, stratified by configuration, task and DV outcome. Two raters answered each unit, first on the reconstruction and then on the source, and could answer "cannot tell"; only ratings where the rater answered the source correctly are analysed. For each DV failure, what people read from the render tells which case applies:

What people read from the failed renderMeaningShare of DV failures
The correct answerJudge misread a correct scene21%
A wrong answer (the judge's own option in 23 of 25)Scene shows wrong content38%
NothingEvidence absent from the render41%

Judge misreading therefore explains about one in five failures. Raters agree moderately (Fleiss' κ 0.41), and the judge agrees with a rater about as well as two raters agree with each other (Cohen's κ 0.36 versus 0.37).

DV is not a fraction of lost facts. We agree. DV is now described as the share of judge-recoverable facts that remain answerable from the render, 1 − DV is no longer called facts lost, and Appendix E no longer says that lower accuracy "reflects information lost". The audit also shows where DV over-credits: people could not answer about 85% of items on degenerate renders (Qwen3.5-397B), including most DV passes, which the forced-choice judge reached by chance. Removing the 27.7% of questions that a text-only judge answers without any frame lowers every configuration by 5.5 to 9.9 points but leaves the ranking nearly unchanged (Spearman 0.97, same top two).

Validation independent of the judge. Unlike the recovery analysis in Appendix G, the audit uses no judge and Scene Test uses no visual judge, and both agree with DV.

LS and DV are not on one scale. We agree. The numerical "gap" between 88.6 LS and 53.7 DV is removed from the abstract, the introduction and Section 4.1. The finding is now stated on comparable quantities: renders of different configurations look alike (V-JEPA similarity 0.88) but share far fewer retained facts (Jaccard 0.45), and more reasoning effort raises LS but not DV.

LS reliability. Recomputed with DINOv2 and SigLIP2 on eight configurations spanning the leaderboard, LS gives nearly the same ranking (Spearman 0.98 and 0.88 with V-JEPA; Appendix F), and Appendix C reports bootstrap intervals for every configuration. We agree that the preference study (five configurations, ranks 2 to 46) supports broad comparisons rather than discrimination among closely ranked leaders; Section 4.3 and the limitations now say so.

We hope these changes address the reviewer's concerns: the claims now match what the benchmark measures, and two new checks, one by people on the judge's own frames and one on the 3D scene itself, support what the render-based metrics report.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.