- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces text-driven camera-trajectory generation grounded in a given 3D point cloud. The authors propose to encode the scene with Concerto point features concatenated with coordinates expressed in the first camera's frame, encode text with UMT5, and run a DiT in the latent space of a convolutional Camera-VAE that represents 49-frame trajectories relative to the first camera. The authors also build WorldTraj: ~136K triplets over ~90K scenes, from DL3DV-10K and DynamicVerse. Captions come from rule-based motion tags, rewritten by an LLM and enriched by a VLM. A new metric, VP-IoU, measures overlap of points visible from generated vs ground-truth trajectories; a collision rate measures physical plausibility. WorldDirector beats Director3D and GenDoP on WorldTraj, TartanAir, ScanNet and ARKitScenes.
Critical strengths
- The task is well motivated: world-consistent video generation pipelines need camera paths, and most prior generators ignore the scene or see only a single RGB-D view.
- The method is simple and cheap: 65M trainable parameters, 2.2 s per trajectory.
- Ablations cover the main design choices, and the paper honestly explains why the no-VAE variant's better VP-IoU and collision numbers are artifacts of jitter.
Critical weaknesses
- The evaluation may leak the answer. Each point cloud is reconstructed from the very video whose camera path is the ground truth, so point density and coverage encode where the camera went. A model can score well on VP-IoU and collision by "going where the points are".
- Baselines are handicapped by design: Director3D sees only text and GenDoP a single RGB-D frame, while WorldDirector sees the whole scene. There is no baseline with the same inputs (e.g., GenDoP fed a rendered panorama or point tokens, a nearest-neighbour retrieval from WorldTraj, or an LLM/VLM planner given the scene).
- Text–camera metrics are partly circular. The F1 score uses the same rule-based tagger that produced the training captions, and the CLaTr embedding used for FID and CLIP scores is trained on WorldTraj itself.
Questions for authors
- Reconstruction-coverage leakage. How much of VP-IoU can be obtained by a trivial policy that follows regions of high point density or the reconstruction's coverage frustum? Please evaluate on worlds whose geometry is independent of the GT path: e.g., ScanNet/ARKitScenes meshes with a different held-out trajectory, synthetic scenes with complete geometry, or point clouds with density equalized and unobserved regions filled.
- Is collision-avoidance guidance used in the main results? The Limitations section speaks of "our collision-avoidance guidance", but guidance is only described for the autoregressive extension in B.2. If Tables 5–6 use guidance, the baselines should get it too.