- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a task in which an input video is reconstructed as executable graphics code and a time-varying 3D scene. It collects 100 real-world videos and 100 synthetic scenes with ground-truth 4D states, and evaluates 17 multimodal coding models on Perceptual, 2D Dynamics, 2.5D Geometry, 3D Geometry, and 3D Dynamics metrics. It also reports agreement between human pairwise comparisons and automated evaluation.
Critical strengths
- Requiring executable code and frame-by-frame geometric states, rather than only a final render, is useful. It allows analysis of how a model implements scene dynamics.
- The benchmark includes both real scenes and synthetic scenes with ground-truth geometry and motion, spanning rigid, deformable, cloth, rope, and fluid phenomena.
Critical weaknesses
-
The role of agent-provided priors in 4D reconstruction remains unclear. The paper does not sufficiently analyze which components benefit from the coding agent—for example, object shape, material properties, object motion, or camera awareness. A component-level evaluation would clarify where agents provide useful information and where they fail.
-
Insufficient evaluation of 3D shape reconstruction. For a benchmark claiming to assess 4D scene reconstruction, accurate geometry matters more than a plausible render. The paper does not sufficiently establish whether agents recover complete object shapes and their changes over time.
-
Limited analysis of failures and dataset selection. The paper should show which specific scenes cause agents to fail. It should also report how many candidate scenes were excluded, why they were excluded, and representative examples. Without this, it is difficult to judge the benchmark’s scope and difficulty.
-
Observed-video fitting versus genuine 4D reconstruction. Agents can inspect every frame of the video on which they are evaluated. A convincing reconstruction from the observed camera does not necessarily imply consistent geometry and motion from unseen viewpoints. Evaluation from held-out viewpoints on synthetic scenes would help distinguish the two.
Questions for authors
-
Does the benchmark evaluate whether the objects needed to render the scene are reconstructed accurately? I think this may be more important than how plausible the video looks.
-
Which scenes do agents handle well, and which do they struggle with? Were any specific scenarios excluded from the dataset because agents failed on them? If so, I would like to see those examples.
-
Do the reconstructions actually fit the observed views accurately, or do they merely look like plausible 4D scenes?
-
Are agents given the camera poses for the input video?
3 comments
Failure Examples
The reviewer asks:
The paper should show which specific scenes cause agents to fail.
Fig. 7 shows which scene categories agents struggle with, and the supplementary gallery linked in the abstract of our initial submission already provides the reconstructions of all scenes beside the ground truth. We will additionally add concrete examples of failure scenes to the appendix [E].
Camera Poses
The reviewer asks:
Are agents given the camera poses for the input video?
We clarify that agents receive only the RGB video and the task prompt.
Dear reviewer,
With the discussion period closing soon, we wanted to kindly check if our response and new experiments address your concerns. We would be happy to clarify anything further if needed. Thank you!
We thank the reviewer for their thoughtful comments and suggestions.
Held-Out Viewpoints
The reviewer writes:
Following the reviewer's suggestion, we scored all 17 models on the 100 synthetic scenes from 21 unseen viewpoints: 8 small shifts of the input camera, 9 orbital views up to 180∘ around the scene, and 4 elevations (20∘, 45∘, 70∘, and 90∘, looking straight down). At 10 frames spread over each clip, we compare the visible moving surfaces in the ground-truth and reconstruction renders using Chamfer distance, normalized by the size of the moving objects and scored as in Scene 3D.
Models retain on average 92% of their input-view score across unseen viewpoints (over 95% for the top-5 models, with the best model scoring 0.82 from the input view and 0.79 across the unseen ones), with nearly identical rankings (Spearman ρ=0.998). Removing surfaces not visible from the input camera leaves the input-view score unchanged but increases the mean score loss from directly behind the scene from 8% to 22%, with a larger loss for all 17 models. These results distinguish the two, as the reconstructions capture geometry on surfaces that the input camera never saw, and their geometry and motion remain consistent from unseen viewpoints. We will add this evaluation to Sec. 4 [E].
Object Shape and Its Change Over Time
The reviewer asks:
We clarify that the benchmark does evaluate the accuracy of the reconstructed objects. Scene 3D compares the surfaces visible from the input camera at frame 0, while 3D Dynamics evaluates the 3D paths of all moving matter, including parts the camera never sees. To evaluate "complete object shapes and their changes over time," we additionally computed, at each frame, the symmetric Chamfer distance between all surfaces of the moving objects in the reconstruction and those in the ground truth (Tab. R2).
As a robustness check, we also refit the registration using only the surfaces visible in the initial frame. The entries in Tab. R2 change by at most 0.03σ, and shape error remains stable or decreases over time in every matter class. For the top-5 models, the mean score drop across held-out viewpoints changes by at most 0.005. We will add this analysis to the paper [E].
Table R2: Shape Over Time. Symmetric Chamfer distance over all surfaces of the moving objects, in scene radii σ; top-5 models, over the runs which produced a scene; lower is better
Component-Level Analysis
The reviewer writes:
Following the reviewer's suggestion, we scored the whole scene, the shape of the moving objects, and their motion separately against the synthetic ground truth, and report the results in Tab. R3. Rigid alignment at each frame raises the moving-object shape score from 0.76 to 0.88, nearly matching the whole-scene score of 0.89. This suggests that much of the error lies in the objects' positions and orientations rather than their shapes. Motion remains the weakest component (3D Dynamics: 0.67). We will add this analysis to Sec. 4 [E].
Table R3: Components. Scores in [0, 1], failed runs scored as 0; higher is better
Dataset Selection
The reviewer asks:
We clarify that no scene was selected or removed based on how agents performed on it.
The reviewer also writes:
For the real split, we selected 100 clips from more than 5,000 candidates from physics datasets, robotics datasets, and the web. We excluded clips showing people or prominent hands, clips with strong camera motion or cuts, and near duplicates within our sources, and searched for additional clips to cover missing matter types and interactions. The synthetic scenes were designed by the authors (Sec. 2.2). We will include examples of clips excluded for each reason in the appendix [E].