acceptodds
Withdrawn from ICLR 2027

World-Grounded Camera Trajectory Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Driven by advancements in 3D world reconstruction, the demand for world-aware video generation and navigation is rapidly growing, where generating controllable camera trajectories is the prerequisite for ensuring 3D consistency. While existing methods show promise, they heavily rely on character-driven motions or single RGB-D images, leaving text-driven trajectory generation in unconstrained 3D point clouds largely unexplored. In this paper, we present WorldDirector, a novel framework that generates world-grounded camera trajectories directly from 3D point clouds and text prompts. To achieve precise spatial grounding, we propose camera-aware world tokens that explicitly encode the observer’s initial perspective by concatenating point-wise geometry features with 3D coordinates transformed into the starting camera’s local coordinate system. This representation allows a Diffusion Transformer (DiT) to effectively map semantic intent onto geometric constraints through cross-attention layers, ensuring that the generated trajectories are both physically plausible and strictly consistent with the 3D scene. To support this task, we introduce WorldTraj, an extensive dataset consisting of roughly 136K triplets of 3D worlds, camera trajectories, and text prompts spanning 90K diverse environments. Extensive experiments demonstrate that WorldDirector successfully generates camera trajectories that are highly consistent with both the 3D world geometry and the semantic instructions in the text prompts. Code and models will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Grounding, Camera Trajectory Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NWHkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces text-driven camera-trajectory generation grounded in a given 3D point cloud. The authors propose to encode the scene with Concerto point features concatenated with coordinates expressed in the first camera's frame, encode text with UMT5, and run a DiT in the latent space of a convolutional Camera-VAE that represents 49-frame trajectories relative to the first camera. The authors also build WorldTraj: ~136K triplets over ~90K scenes, from DL3DV-10K and DynamicVerse. Captions come from rule-based motion tags, rewritten by an LLM and enriched by a VLM. A new metric, VP-IoU, measures overlap of points visible from generated vs ground-truth trajectories; a collision rate measures physical plausibility. WorldDirector beats Director3D and GenDoP on WorldTraj, TartanAir, ScanNet and ARKitScenes.

Critical strengths
  • The task is well motivated: world-consistent video generation pipelines need camera paths, and most prior generators ignore the scene or see only a single RGB-D view.
  • The method is simple and cheap: 65M trainable parameters, 2.2 s per trajectory.
  • Ablations cover the main design choices, and the paper honestly explains why the no-VAE variant's better VP-IoU and collision numbers are artifacts of jitter.
Critical weaknesses
  • The evaluation may leak the answer. Each point cloud is reconstructed from the very video whose camera path is the ground truth, so point density and coverage encode where the camera went. A model can score well on VP-IoU and collision by "going where the points are".
  • Baselines are handicapped by design: Director3D sees only text and GenDoP a single RGB-D frame, while WorldDirector sees the whole scene. There is no baseline with the same inputs (e.g., GenDoP fed a rendered panorama or point tokens, a nearest-neighbour retrieval from WorldTraj, or an LLM/VLM planner given the scene).
  • Text–camera metrics are partly circular. The F1 score uses the same rule-based tagger that produced the training captions, and the CLaTr embedding used for FID and CLIP scores is trained on WorldTraj itself.
Questions for authors
  1. Reconstruction-coverage leakage. How much of VP-IoU can be obtained by a trivial policy that follows regions of high point density or the reconstruction's coverage frustum? Please evaluate on worlds whose geometry is independent of the GT path: e.g., ScanNet/ARKitScenes meshes with a different held-out trajectory, synthetic scenes with complete geometry, or point clouds with density equalized and unobserved regions filled.
  2. Is collision-avoidance guidance used in the main results? The Limitations section speaks of "our collision-avoidance guidance", but guidance is only described for the autoregressive extension in B.2. If Tables 5–6 use guidance, the baselines should get it too.
Reviewer_BDczOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies text-driven camera trajectory generation inside a given 3D scene. The proposed WorldDirector takes a point cloud, a text prompt and the first camera, and generates a camera path with a diffusion model. The key design is "camera-aware world tokens", which express the scene points in the first camera's coordinate frame. The authors also build WorldTraj, a large dataset of point clouds, trajectories and captions built from DL3DV and DynamicVerse. Experiments compare against Director3D and GenDoP on WorldTraj and three unseen datasets, using CLaTr-based metrics and two new metrics (VP-IoU and collision rate).

Critical strengths
  • The task is useful. Camera-controlled video models all need a camera path, and generating it directly from the 3D scene and a text prompt fills a real gap. The WorldTraj dataset is large and could be useful for the community if it is released.
  • The model is small and fast, and it shows consistent gains on all four test sets. The ablations remove one component at a time, and the supplementary videos show smooth paths that follow the prompt and stay inside the scene.
Critical weaknesses
  • The experiments do not clearly show "world grounding". The baselines do not see the full 3D scene, and the paper does not explain how their outputs are scaled and placed into the scene. In the supplementary videos, the baseline paths look much shorter, and even the first frame differs across methods. Also, most worlds are reconstructed from the same clip that gives the ground-truth path, so the point cloud itself already tells the model where the camera went, and VP-IoU then mostly rewards covering the reconstructed region. It is also not clear if the train/test split is done by scene. A fairer test would use worlds larger than the ground-truth segment, a scene-level split, and at least one baseline that also sees the scene.
  • The method contribution is modest, and the dataset is not well validated. Expressing the points in the first camera's frame is a natural design choice (the baselines already express trajectories in this frame), and the ablation only shows that some start-pose information is needed. So the paper relies a lot on WorldTraj. But DynamicVerse, its main source, already provides camera poses, depth and camera captions and is missing from the dataset comparison, the new annotation is not clearly better than DynamicVerse's own, and caption quality is only checked by an LLM judge using the same tags that generated the captions.
Questions for authors
  1. How are the baseline outputs scaled and aligned to the scene? Why does the first frame differ across methods in the supplementary videos?
  2. Is the train/test split done by scene? Could the authors report results separately for DL3DV and DynamicVerse worlds, and on worlds much larger than the ground-truth segment?
  3. Could the authors add a baseline that also sees the scene (e.g., the text-only Cam-DiT with collision guidance), and a human check of the captions?
  4. Will WorldTraj be released, and under what license?

Overall: The task is useful and the results look good in the videos. However, the main claim of world grounding is not convincingly isolated, and the dataset, which carries much of the contribution, needs better validation. I currently lean toward a weak rejection, but I am open to raising my score if the authors clarify the baseline alignment and the split, and show a fairer comparison as suggested above.

Reviewer_ZKgTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper generates a 49-frame camera trajectory, including position, orientation, and focal length at each frame, from an existing 3D point-cloud scene, a text instruction, and an initial camera. The authors first express the scene points in the initial camera's coordinate system and concatenate these coordinates with pretrained scene features. A diffusion model then uses the text and scene to generate the full motion in the trajectory representation learned by Cam-VAE. The authors also construct WorldTraj, with approximately 136K samples and 90K scenes, and conduct ablations, evaluations on external datasets, and human studies.

Text-controlled trajectories and coordinate systems referenced to the first camera have already appeared in prior work: GenDoP uses text and first-frame RGB-D, while Director3D also generates trajectories from text, along with the scene itself. The text annotation pipeline is also quite similar to DataDoP's. I therefore understand the main novelty to be incorporating a full scene map and its relationship to the initial camera's viewpoint into trajectory generation, together with a dataset that includes scenes. GenDoP/DataDoP, Director3D

CinemaTraj, made public in July 2026, also addresses "planning camera movements based on a scene," but uses scene structure and optimization rather than generating directly from point clouds as this paper does. It would be useful to discuss it in the related work. CinemaTraj

Critical strengths

(1) Substantially outperforms a retrained baseline.

The authors retrain GenDoP on their dataset (denoted GenDoP*) before comparing it with their method, which makes this a reasonably fair comparison. Their method is substantially better in CLaTr-FID, VP-IoU, and collision rate, and the advantages remain on TartanAir, which was not used for training.

(2) The initial-camera viewpoint design is supported by a direct controlled comparison.

With all other conditions held fixed, adding "coordinates relative to the initial camera" improves trajectory quality, overlap with the regions seen along the reference trajectory, and collision rate. The authors also compare three ways of incorporating this information. This shows that the design is useful.

Critical weaknesses

(1) The methodological contribution is limited, and the central claims lack direct evidence.

The large gaps in the main tables mainly come from the complete system and the more comprehensive inputs. The only gains that can be attributed to the method in isolation come from the controlled comparison above, and their magnitude is modest. More importantly, the two things the paper most wants to establish have not been measured directly:

  • Does it film the objects specified in the text? Specifically, the existing metrics either check whether the movements are correct, whether the text and trajectory are similar, or which regions are seen over the whole trajectory. None answers "whether the camera points at the specified object, how long it views it, or whether it keeps it centered." The human evaluation (14 experts, 8 cases, 83% choosing this method) is a good step, but does not measure this separately or clarify which version of GenDoP was used. (pp. 16–17, 20)
  • Does it actually respect the scene geometry and reduce wall collisions? Here, the evidence mainly relies on two metrics introduced by the authors, and the collision criteria are not specified. The paper itself provides a counterexample: after removing Cam-VAE, the trajectories are already very poor, yet these two scores are the best in the table. (pp. 7–8)

(2) The dataset's most novel aspect has not been validated, and its release has not been specified.

What WorldTraj adds over prior datasets is scenes paired with trajectories and text. However, this aspect has not been fully checked:

  • The text-quality check only evaluates whether the motion descriptions are accurate. It does not check whether the objects mentioned actually exist in the scene or are filmed. The earlier DataDoP dataset used expert spot checks of alignment.
  • The GenDoP retraining experiment is used to show that "the data are useful," but GenDoP does not read point clouds at all. It therefore only demonstrates the usefulness of the trajectory and text components.
  • The paper does not say whether the dataset will be released or whether the training/test split is by trajectory or by video. An average of 1.5 trajectories is extracted from each video, which makes this detail important.

In addition, Concerto, the pretrained scene encoder used by the authors, was exposed to ScanNet and ARKitScenes scenes during pretraining. These two external evaluations therefore cannot be described as testing on "completely unseen environments."

Questions for authors

1. Will the data be released, and how are training and test data split? (This could change my rating.) Please clarify whether WorldTraj will be released, including point clouds, trajectories, text, and split lists; whether the 9:1 split is by trajectory or by video, and how many test trajectories share a source video with the training set; which data were used by each component and scoring model; and which Concerto checkpoint was used.

If the data will be released, the split is by video, and the advantages remain on independent tests, I would view the data contribution much more favorably.

2. Could you spot-check whether the scenes and descriptions match? (This could also change my rating.) For example, check a small subset of samples to see whether the objects mentioned in the text actually exist in the point cloud and are visible along the reference trajectory. Alternatively, change only the target object in the text for the same scene and check whether the camera switches to filming the new target.

3. A few questions about the experimental settings. Do the main tables use the collision-avoidance guidance from the appendix? Does the no-Cam-VAE variant use the same coordinate and scale processing as the full model? What radius and point-count threshold are used to determine collisions? Does the human evaluation compare against the original GenDoP or the retrained GenDoP*?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.