acceptodds
Under review as a conference paper at ICLR 2027

4DCodeBench: Benchmarking Agents on Inverse Graphics of Dynamic Scenes

39 views

Abstract

We introduce 4DCodeBench, a benchmark for 4D inverse graphics through code generation, in which agents reconstruct dynamic scenes from video as executable graphics programs. To accomplish this, agents must translate video observations into compact representations of scene structure and dynamics by implementing abstractions such as physical simulations to reproduce complex behavior. To evaluate this capability, we curate a set of real-world videos and construct synthetic scenes spanning diverse physical phenomena, including deformation, fluid flow, and fracture. We perform extensive benchmarking of frontier models, finding that strong static reconstruction capabilities do not yet translate into reliable reconstruction of complex dynamics. 4DCodeBench provides a testbed for tracking progress toward agents that can interpret the dynamics of the world through code. Anonymous project page: https://anon-4d-code-bench-iclr27.site/.

Primary area:
datasets and benchmarks
Keywords:
4d, inverse graphics, VLM, reconstruction, agentic, physical simulation, dynamics

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 4 reviews, above 99% of the others.
Reviewer_UiYNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a task in which an input video is reconstructed as executable graphics code and a time-varying 3D scene. It collects 100 real-world videos and 100 synthetic scenes with ground-truth 4D states, and evaluates 17 multimodal coding models on Perceptual, 2D Dynamics, 2.5D Geometry, 3D Geometry, and 3D Dynamics metrics. It also reports agreement between human pairwise comparisons and automated evaluation.

Critical strengths
  • Requiring executable code and frame-by-frame geometric states, rather than only a final render, is useful. It allows analysis of how a model implements scene dynamics.
  • The benchmark includes both real scenes and synthetic scenes with ground-truth geometry and motion, spanning rigid, deformable, cloth, rope, and fluid phenomena.
Critical weaknesses
  • The role of agent-provided priors in 4D reconstruction remains unclear. The paper does not sufficiently analyze which components benefit from the coding agent—for example, object shape, material properties, object motion, or camera awareness. A component-level evaluation would clarify where agents provide useful information and where they fail.

  • Insufficient evaluation of 3D shape reconstruction. For a benchmark claiming to assess 4D scene reconstruction, accurate geometry matters more than a plausible render. The paper does not sufficiently establish whether agents recover complete object shapes and their changes over time.

  • Limited analysis of failures and dataset selection. The paper should show which specific scenes cause agents to fail. It should also report how many candidate scenes were excluded, why they were excluded, and representative examples. Without this, it is difficult to judge the benchmark’s scope and difficulty.

  • Observed-video fitting versus genuine 4D reconstruction. Agents can inspect every frame of the video on which they are evaluated. A convincing reconstruction from the observed camera does not necessarily imply consistent geometry and motion from unseen viewpoints. Evaluation from held-out viewpoints on synthetic scenes would help distinguish the two.

Questions for authors
  • Does the benchmark evaluate whether the objects needed to render the scene are reconstructed accurately? I think this may be more important than how plausible the video looks.

  • Which scenes do agents handle well, and which do they struggle with? Were any specific scenarios excluded from the dataset because agents failed on them? If so, I would like to see those examples.

  • Do the reconstructions actually fit the observed views accurately, or do they merely look like plausible 4D scenes?

  • Are agents given the camera poses for the input video?

3 comments

We thank the reviewer for their thoughtful comments and suggestions.

Held-Out Viewpoints

The reviewer writes:

A convincing reconstruction from the observed camera does not necessarily imply consistent geometry and motion from unseen viewpoints. Evaluation from held-out viewpoints on synthetic scenes would help distinguish the two.

Following the reviewer's suggestion, we scored all 17 models on the 100 synthetic scenes from 21 unseen viewpoints: 8 small shifts of the input camera, 9 orbital views up to 180∘180^\circ around the scene, and 4 elevations (20∘20^\circ, 45∘45^\circ, 70∘70^\circ, and 90∘90^\circ, looking straight down). At 10 frames spread over each clip, we compare the visible moving surfaces in the ground-truth and reconstruction renders using Chamfer distance, normalized by the size of the moving objects and scored as in Scene 3D.

Models retain on average 92% of their input-view score across unseen viewpoints (over 95% for the top-5 models, with the best model scoring 0.82 from the input view and 0.79 across the unseen ones), with nearly identical rankings (Spearman ρ=0.998\rho = 0.998). Removing surfaces not visible from the input camera leaves the input-view score unchanged but increases the mean score loss from directly behind the scene from 8% to 22%, with a larger loss for all 17 models. These results distinguish the two, as the reconstructions capture geometry on surfaces that the input camera never saw, and their geometry and motion remain consistent from unseen viewpoints. We will add this evaluation to Sec. 4 [E].

Object Shape and Its Change Over Time

The reviewer asks:

Does the benchmark evaluate whether the objects needed to render the scene are reconstructed accurately?

We clarify that the benchmark does evaluate the accuracy of the reconstructed objects. Scene 3D compares the surfaces visible from the input camera at frame 0, while 3D Dynamics evaluates the 3D paths of all moving matter, including parts the camera never sees. To evaluate "complete object shapes and their changes over time," we additionally computed, at each frame, the symmetric Chamfer distance between all surfaces of the moving objects in the reconstruction and those in the ground truth (Tab. R2).

As a robustness check, we also refit the registration using only the surfaces visible in the initial frame. The entries in Tab. R2 change by at most 0.03σ0.03\sigma, and shape error remains stable or decreases over time in every matter class. For the top-5 models, the mean score drop across held-out viewpoints changes by at most 0.005. We will add this analysis to the paper [E].

Table R2: Shape Over Time. Symmetric Chamfer distance over all surfaces of the moving objects, in scene radii σ\sigma; top-5 models, over the runs which produced a scene; lower is better

MatterFirst 10% of framesLast 10%
Rigid0.090.08
Deformable0.150.14
Cloth, rope0.100.08
Flowing (fluids, sand)0.160.10

Component-Level Analysis

The reviewer writes:

A component-level evaluation would clarify where agents provide useful information and where they fail.

Following the reviewer's suggestion, we scored the whole scene, the shape of the moving objects, and their motion separately against the synthetic ground truth, and report the results in Tab. R3. Rigid alignment at each frame raises the moving-object shape score from 0.76 to 0.88, nearly matching the whole-scene score of 0.89. This suggests that much of the error lies in the objects' positions and orientations rather than their shapes. Motion remains the weakest component (3D Dynamics: 0.67). We will add this analysis to Sec. 4 [E].

Table R3: Components. Scores in [0, 1], failed runs scored as 0; higher is better

ComponentTop 5
Whole scene (Scene 3D at every frame)0.89
Shape of the moving objects (Chamfer score over all of their surfaces at every frame)0.76
Shape of the moving objects, aligned rigidly to the ground truth at every frame0.88
Motion (3D Dynamics)0.67

Dataset Selection

The reviewer asks:

Were any specific scenarios excluded from the dataset because agents failed on them?

We clarify that no scene was selected or removed based on how agents performed on it.

The reviewer also writes:

It should also report how many candidate scenes were excluded, why they were excluded, and representative examples

For the real split, we selected 100 clips from more than 5,000 candidates from physics datasets, robotics datasets, and the web. We excluded clips showing people or prominent hands, clips with strong camera motion or cuts, and near duplicates within our sources, and searched for additional clips to cover missing matter types and interactions. The synthetic scenes were designed by the authors (Sec. 2.2). We will include examples of clips excluded for each reason in the appendix [E].

Failure Examples

The reviewer asks:

The paper should show which specific scenes cause agents to fail.

Fig. 7 shows which scene categories agents struggle with, and the supplementary gallery linked in the abstract of our initial submission already provides the reconstructions of all scenes beside the ground truth. We will additionally add concrete examples of failure scenes to the appendix [E].

Camera Poses

The reviewer asks:

Are agents given the camera poses for the input video?

We clarify that agents receive only the RGB video and the task prompt.

Dear reviewer,

With the discussion period closing soon, we wanted to kindly check if our response and new experiments address your concerns. We would be happy to clarify anything further if needed. Thank you!

Reviewer_ry2FOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper focuses on evaluating whether coding agents can reconstruct dynamic 3D scenes from videos using executable graphics code. It introduces 4DCodeBench, a benchmark with 100 real-world and 100 synthetic scenes, covering different materials and physical interactions. Experiments on 17 models, together with human evaluations, show that current agents perform relatively well on appearance and static geometry but still struggle with reconstructing complex dynamics.

Critical strengths

S1. The paper studies an important and timely problem. I think evaluating whether coding agents can reconstruct dynamic 3D scenes from videos is a meaningful research direction.

S2. Building this benchmark requires considerable effort and resources. I appreciate the work put into collecting real-world videos, creating synthetic scenes, and evaluating 17 different models.

S3. The evaluation is quite solid, with multiple metrics and a large human study to support the results. The paper is also well organized and easy to follow.

Critical weaknesses

W1. The benchmark allows agents to reconstruct motion using simple analytic functions or fitted trajectories, without necessarily modeling the underlying physics. Since 71% of solutions use analytic motion, I am not sure how much the results reflect physical understanding rather than fitting the observed videos.

W2. One main finding is that models perform much better on static geometry than dynamics. However, 3D geometry is evaluated only at the first frame, while dynamics is evaluated over the entire video. The different evaluation settings and scoring methods may partly explain this gap, making the conclusion less convincing.

Questions for authors

Q1. Since most solutions use analytic motion rather than physical simulation, how does your benchmark distinguish physical understanding from simply fitting the observed motion?

Q2. Your 3D geometry evaluation only considers the first frame, while dynamics is evaluated over the whole video. Could you discuss whether this difference affects the observed gap between static and dynamic reconstruction?

2 comments

We thank the reviewer for their thoughtful comments and suggestions.

Physical Understanding vs. Fitting

The reviewer asks:

how does your benchmark distinguish physical understanding from simply fitting the observed motion?

We clarify that the benchmark measures physical understanding through the fidelity of the reconstruction, whether its motion is analytic or simulated. The agent observes a single video, whereas we score the 4D scene itself against ground truth (Sec. 2.4), which the video alone does not determine, so fitting the observed motion is not sufficient. The world is high-dimensional, and modeling its dynamics as (low-dimensional) graphics code requires physical abstractions (Sec. 1). Reconstructing the dynamics of a scene as code is therefore what we mean by understanding them [C].

Static--Dynamic Gap

The reviewer asks:

Your 3D geometry evaluation only considers the first frame, while dynamics is evaluated over the whole video. Could you discuss whether this difference affects the observed gap between static and dynamic reconstruction?

To evaluate whether these differences explain the gap, we rescored the shape of the moving objects at every frame (Tabs. R2 and R3 in our response to UiYN) and compared it with Trajectory DTW under matching distance caps. Shape remains higher for all 17 models at both caps, with the top-5 models scoring 0.76 vs. 0.61 under 0.5σ0.5\sigma and 0.85 vs. 0.76 under 1σ1\sigma, and EMD step, which requires no path matching, scores lower still (Fig. A19). We thank the reviewer for the suggestion and will report this comparison in Sec. 4.2 [C,E].

Thank you for the detailed clarifications and additional experiments. Your responses have satisfactorily addressed my questions and further strengthened my positive assessment of the paper. I continue to support its acceptance and will maintain my original score.

Reviewer_yGJcOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces 4DCodeBench, a benchmark for 4D inverse graphics through code generation. Given a video, an agent must write an executable graphics program that reconstructs the scene's 3D geometry over time and renders it in Blender; the program must rebuild the world from scratch without access to the reference video, and must expose per-frame geometry plus a frame-to-frame material correspondence, which is what allows direct 3D/4D evaluation rather than pixel comparison. The benchmark has 200 scenes (100 real videos, 100 synthetic with full 4D ground truth) spanning rigid/articulated, deformable, co-dimensional and flowing matter. Seventeen models are evaluated once per scene in a common container; the metric stack has five families (perceptual, 2D dynamics, 2.5D geometry, 3D geometry, 3D dynamics) plus scene-level VQA and a VLM pairwise Elo, validated against 3,587 human pairwise judgments from 76 participants. The headline findings are that dynamics reconstruction lags appearance and static geometry (best model: 0.91 on static families vs 0.67 on dynamic), and that the automated Elo tracks humans closely (rho = 0.975).

Critical strengths

The task design is the strongest part. Requiring regeneration without the reference video blocks trivial copying, and forcing the submission to expose explicit geometry and correspondence is exactly what makes 3D ground-truth evaluation possible on the synthetic split. Leaving the dynamics representation open (analytic, keyframe, Blender physics, custom simulator) and then census-ing what models actually choose (Table A4: 71% analytic; Opus 51% custom simulation vs Terra 100% analytic) turns the benchmark into an analysis instrument, not just a leaderboard.

Validation of the automated metrics against humans is more serious than what I usually see in benchmark papers: VLM vs human Elo correlate at 0.975, on 916 shared comparisons human-VLM agreement is 89.3% (kappa 0.761) against 92.1% (kappa 0.816) between humans, and the authors even quantify where the judge fails (near chance within 50 Elo). The failure handling also shows care: gated VQA questions are scored wrong so no model gains by rendering fewer scenes, failed runs get worst-case scores, and the trimmed ICP registration prevents weak models from failing 3D metrics purely because of camera-export conventions (App. D.4).

The analyses beyond the ranking are useful: category effects with an explicit consistency criterion (Fig. 7), the static-dynamic gap (Fig. 8), the reasoning-effort sweep within Astra, the cost-quality plot, and the observation that token use does not predict quality across models but does within Astra.

Critical weaknesses

Three issues need answers before I fully trust the headline numbers; the rest I would class as lesser concerns.

  1. One run per model per scene. Coding agents are stochastic, and the top of the leaderboard is tight — Astra High at Overall 0.77 against Fable at 0.74 — yet there is no run-variance estimate anywhere in the paper. The 95% intervals shown are over scenes, which answers whether rankings are stable across scenes, not whether they survive a re-run. For a paper whose main deliverable is a leaderboard, this is the gap I care most about.

  2. The VQA metric has almost no head. In Figure 4 the always-Yes baseline sits at 87.4%, essentially tied with the best model at 87.6%, and every other model is below it. After gating, the question set is evidently heavily Yes-skewed, so VQA accuracy mostly measures gate failures at the bottom and hugs the majority class at the top. As it stands the metric is not measuring what it is described as measuring (understanding of physical events), and the text does not engage with this — the dashed line is right there in the paper's own figure.

  3. The judge is also a contestant. Gemini 3.8 Flash is both the pairwise judge and the VQA judge, and simultaneously one of the 17 evaluated models. The very strong human agreement (rho = 0.975) mitigates the concern but does not address it: there is no analysis of judgments involving Gemini's own reconstructions, and no second judge.

Lesser concerns, listed briefly: the harness split (proprietary models in their native CLI, open-weight in Stirrup) is an unexamined confound for the proprietary-vs-open gap; the real clips come from public datasets and CC sources with no contamination check, and the synthetic scenes use very common assets; some ontology cells are small (synthetic co-dimensional: 19 scenes) and the best model already sits at 0.91 Perceptual, so a holdout or refresh policy matters for the benchmark's lifetime; Overall is an unweighted mean of five families; and two bookkeeping slips — Table A4 omits GLM-5.3 without explanation and accounts for 3,191 of 3,200 solutions, and there is a duplicated sentence at the end of Sec. 4.2.

Questions for authors
  1. How stable are the rankings to re-running? Even 2-3 repeats of the top-5 models on a scene subsample would calibrate run noise for the gaps at the top.
  2. What is the Yes/No balance of the VQA questions after gating? Please report a majority-class-corrected number (or balanced accuracy); as plotted, all models sit at or below always-Yes.
  3. Is there any self-preference in the judge's judgments on Gemini's own reconstructions? Would a second VLM judge change the Elo?
  4. The static-dynamic gap could partly reflect error compounding rather than a failure to infer dynamics: the static families are point measurements, while the dynamic families score the whole trajectory. Can you report how the dynamic metrics degrade with frame index — for example, performance extrapolated to early frames — to separate "cannot infer dynamics" from "small errors compound over the rollout"?
  5. How much of the proprietary-vs-open gap survives a cross-harness check for at least one strong open model?
  6. Did you check for contamination of the real clips (e.g., source recognition)? Is a freshly collected holdout planned?
  7. Roughly what does it cost (GPU-hours plus API) to evaluate one new model on all 200 scenes? This matters for community adoption.
  8. On real scenes, how much of the 2D-dynamics penalty could come from RAFT/CoTracker estimation noise rather than from the models themselves?
5 comments

We thank the reviewer for their thoughtful comments and suggestions.

Run Variance

The reviewer asks:

How stable are the rankings to re-running? Even 2-3 repeats of the top-5 models on a scene subsample would calibrate run noise for the gaps at the top.

Following the reviewer's suggestion, we ran each of the top-5 models three times on a stratified subset of 20 scenes (10 real and 10 synthetic) and scored failed runs as in the paper. Using the run-to-run variance on these scenes, we estimate the standard deviation of each pairwise difference in the 200-scene Overall score under re-running. We find that re-running is unlikely to change the order of Astra Max and Astra High or their leads over Fable, Astra Low, and Opus 5: the leaderboard gaps between Astra Max and Astra High and between Astra High and Fable are 7.1 and 2.7 times the standard deviations of the respective pairwise differences under re-running. We estimate that a repeat keeps Fable ahead of Astra Low and Astra Low ahead of Opus 5 in 94% and 96% of cases (leaderboard gaps of 1.54 and 1.80 standard deviations of the respective pairwise differences), and Fable ahead of Opus 5 in more than 99% of cases (2.6 standard deviations). The ranking is thus largely stable under re-running, even among the closely matched top-5 models [E].

VQA Balance

The reviewer asks:

What is the Yes/No balance of the VQA questions after gating? Please report a majority-class-corrected number (or balanced accuracy)

On average, 87% of a scene's questions have the answer Yes. Following the reviewer's suggestion, we will report balanced accuracy in the updated paper. Under this metric, always answering Yes scores 0.50, a reconstruction whose first frame is correct, but which never moves, scores 0.53, while the top-5 models score 0.80--0.90, and the reference video scores 0.94 [C,E].

Judge

The reviewer asks:

Is there any self-preference in the judge's judgments on Gemini's own reconstructions?

We asked six human raters -- who saw neither the model identity nor the judge's answers -- to answer the VQA questions on 96 renders, half from Gemini and half from its four ranking neighbors. For a given model, the judge's net credit is the share of VQA questions on its reconstructions for which it incorrectly awards credit minus the share for which it incorrectly withholds credit, using human answers as the reference. Net credit is 0.7 percentage points lower for Gemini than for its neighbors (95% CI: −5.3-5.3 to +3.8+3.8), providing no evidence that the judge favors Gemini [E].

The reviewer also asks:

Would a second VLM judge change the Elo?

We re-judged the 916 human-labeled pairwise comparisons with GPT-6.1 Sol, which is not among the 17 evaluated models. Its Elo ratings agree with those of the paper (produced by Gemini), with a Spearman correlation of 0.99, and it places Gemini and every GPT model exactly where the paper's VLM Elo does (Fig. 5). This suggests that the Elo ranking is stable across judges [E].

Static--Dynamic Gap

The reviewer asks:

Can you report how the dynamic metrics degrade with frame index [...] to separate "cannot infer dynamics" from "small errors compound over the rollout"?

Following the reviewer's suggestion, we scored Trajectory DTW on disjoint windows of each clip. Scores do not drop from the first 10% of frames to the last, with no later window scoring below the first. In every window, the agents outperform a baseline that holds each reconstruction fixed at its initial frame. To check that time warping does not mask error accumulation, we also measured reconstruction error across all surfaces of moving objects at matched frames under a single fixed registration. This error does not grow from the first 10% of frames to the last in any matter class (Tab. R2 in our response to UiYN). We will include both analyses in the updated paper. These results suggest that errors compounding over the rollout are not the main source of the static--dynamic gap [E].

Harness

The reviewer asks:

How much of the proprietary-vs-open gap survives a cross-harness check for at least one strong open model?

Following the reviewer's suggestion, we reevaluated the strongest open model, DeepSeek V4.1, using Codex CLI instead of Stirrup on the same 20-scene subset used for the run-variance experiment. Its Overall score decreases by 0.02. By comparison, the proprietary models lead the open ones by 0.33 on average on these scenes. This suggests that the proprietary--open gap is not primarily explained by the choice of harness [E].

Contamination

The reviewer asks:

Did you check for contamination of the real clips (e.g., source recognition)? Is a freshly collected holdout planned?

The real clips are scored against the same video that the agent receives as input, and no 4D ground truth exists for them, so a model that has seen a clip before gains nothing that the task withholds. The 4D ground truth of the synthetic scenes is our own and was unpublished when the agents ran, and the models rank nearly identically on them and on the public real clips, with no model moving up by more than one place on the public clips. Regarding a holdout, we plan to add newly rendered synthetic scenes as a private split for future versions of the leaderboard [C,D].

Misc.

The reviewer asks:

Roughly what does it cost [...] to evaluate one new model on all 200 scenes?

An agent run takes a median of about 70 minutes and costs up to about $15.

The reviewer also asks:

On real scenes, how much of the 2D-dynamics penalty could come from RAFT/CoTracker estimation noise rather than from the models themselves?

Of the five dynamics metrics, only Flow and Track2D use estimated references, and only on real videos. The static--dynamic gap does not rest on them, as on the synthetic scenes it is measured against exact 3D ground truth, where the best-performing model scores 0.92 on 3D Geometry and 0.73 on 3D Dynamics (Fig. 5).

The reviewer also writes:

Table A4 omits GLM-5.3 without explanation and accounts for 3,191 of 3,200 solutions, and there is a duplicated sentence at the end of Sec. 4.2.

We thank the reviewer for their attention to detail and will correct Tab. A4 and remove the duplicated sentence in Sec. 4.2 [C].

Really appreciate the author's new experiments and feedback, rather than just words. In general, experiments address my major concerns including VQA balance, judge, etc.

Two follow-up questions for your considerations:

  1. please report each model's own re-run SD, not only the pairwise ratios?
  2. Please confirm the variance estimates include the tails from failed runs?
Reviewer_ihhDOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper provides an analysis/evaluation of the ability of different coding agents on the task of 4D monocular reconstruction by reconstructing dynamic scenes via code on a proposed set of 200 videos (real and synth) by aggregating a set of real videos in addition to authoring a set of synthetic videos of various dynamics. The paper proposes using this set of videos for a benchmark measuring the quality of the reconstructed 4D geometry and the visual fidelity of the reconstructed scenes.

Critical strengths
  • The paper benchmarks a comprehensive list of models. Using agents to reconstruct 4D scenes is an important and interesting task to study and having a study comparing their performance is useful to the community.
  • The accompanying videos provide a nice visual insight into the performance of these models on 4D reconstruction.
Critical weaknesses

The biggest challenge I find with this paper is the deferring of important information to the appendix, many sections of the main paper are challenging to parse without having to constantly go back and forth to the appendix. The idea is simple and interesting but the reading experience makes it challenging, here are 2 examples of that:

  • The paper’s experiment section is very hard to read because of the following:

    • Vague metric names that are not properly defined in the main paper.
    • Metrics are normalised rather than being presented with their units which makes it harder to interpret and compare with other methods.
  • The related work section is unnecessarily split between the main paper and the appendix, which makes reading it hard.

Questions for authors
  • The proposal of a new dataset sampled from existing datasets is not motivated. Why not just evaluate on full existing datasets that can contain different scene types?
  • Why is there an asymmetry in the synthetic vs real video evaluations? Disparity vs chamfer in real vs synthetic eval for example.
  • What is the motivation for having VQA? Isn’t geometry and appearance sufficient to evaluate the quality of the reconstruction?

Minor comment:

  • Figure reference is missing on line 1237.
3 comments

We thank the reviewer for their thoughtful comments and suggestions.

Text Placement

The reviewer writes:

The biggest challenge I find with this paper is the deferring of important information to the appendix

We appreciate the reviewer's suggestion and will consolidate the metric definitions of Sec. 2.4 and App. D into a single table in Sec. 2.4 (Tab. R1), merge the related work of App. B into Sec. 5, and fix the noted figure reference [C].

Table R1: Metric Definitions. σ\sigma: RMS radius of the ground-truth surfaces visible at frame 0

FamilyMetricScenesBefore normalizationScore in [0, 1]
PerceptualDINOv3allcosine similarity cc of image features(1+c)/2(1 + c)/2
2D DynamicsDynamic IoUalloverlap of moving-object masksthe IoU
Flowrealper motion step, sliced Wasserstein distance dd between the two sets of flow vectors, divided by the sum of their RMS magnitudes1−1 - mean over steps of min⁡(d,1)\min(d, 1)
Track2DrealDTW distance dd between matched 2D point tracks, in % of the image diagonal1−1 - mean over tracks of min⁡(d,3%)/3%\min(d, 3\%)/3\%
2.5D GeometryDepthrealmean absolute difference ee of disparities, each median-shifted and scaled by its mean absolute deviation as in MiDaS, largest 1% of residuals dropped1−e/21 - e/2
3D GeometryScene 3DsyntheticChamfer distance dd between the surfaces visible at frame 0, in σ\sigma1−min⁡(d,0.5σ)/0.5σ1 - \min(d, 0.5\sigma)/0.5\sigma
3D DynamicsTrajectory DTWsyntheticdynamic-time-warping distance dd between one-to-one matched 3D paths of moving matter, in σ\sigma1−1 - mean over paths of min⁡(d,1σ)/1σ\min(d, 1\sigma)/1\sigma
EMD stepsyntheticsliced Wasserstein distance dd between frame-to-frame 3D displacements, divided by the total motionmax⁡(0,1−d)\max(0, 1 - d)

Metric Units

The reviewer writes:

Metrics are normalised rather than being presented with their units which makes it harder to interpret and compare with other methods.

Following the reviewer’s suggestion, we will report metrics in their original units next to Fig. A19, alongside normalized scores in [0, 1] that account for differences in reconstruction and scene scale and allow averaging into family scores and Overall [C].

Synthetic vs. Real-World Evaluation

The reviewer asks:

Why is there an asymmetry in the synthetic vs real video evaluations?

We clarify that, unlike our synthetic scenes, the real-world videos we evaluate on have no 4D ground truth [C].

Dataset

The reviewer asks:

Why not just evaluate on full existing datasets that can contain different scene types?

Evaluating on the full datasets we draw from would be prohibitively expensive, as WISA-80K alone contains 80,000 clips. We instead select real-world clips that span diverse matter and interactions while limiting redundancy (Sec. 2.2).

Our synthetic scenes, each designed to pair this diversity with 4D ground truth, have no existing counterpart [C].

VQA

The reviewer asks:

What is the motivation for having VQA? Isn't geometry and appearance sufficient to evaluate the quality of the reconstruction?

VQA was initially motivated for real-world evaluation, which does not have 4D ground truth. We also find that it complements geometric metrics on synthetic scenes where averaging can hide key details [C].

Thanks for the clarifications, my concerns have been mostly addressed. Given that and the comprehensiveness of the evaluation setup, I will be raising my score.

I strongly encourage the authors to run the real scenes metrics on the synthetic scenes as well, such that apples to apples comparison can be established on the capability of these agents in reconstructing real vs synthetic scenes and hence inferences about the real world from the synthetic scenes evals can be a bit more calibrated.

Thank you for your continued engagement and support. We will certainly look into adding these results as we further refine the paper.

Comments on the paper

1 comment

We thank each of the reviewers for their constructive comments and suggestions, which we will use to improve the quality of our paper. They find our work "useful" (ihhD, yGJc, UiYN) and our task "important" (ihhD, ry2F), "timely" (ry2F), and "interesting" (ihhD). They describe our evaluation as "comprehensive" (ihhD) and "quite solid" (ry2F), and our human validation as "more serious than what I usually see in benchmark papers" (yGJc). We respond to each reviewer below and will update the paper accordingly, adding clarifications [C] and experiments [E] to the text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.