acceptodds
Under review as a conference paper at ICLR 2027

Distilling Dense Trajectory Teachers for Large-Motion Video World Models

Abstract

Video world models are expected to generate videos that accurately follow specified camera trajectories while maintaining consistent 3D structures. However, existing models are predominantly trained on densely sampled videos with small inter-frame camera motions. When applied to sparse, large-step trajectories, they often produce frozen frames, inaccurate camera motion, and severe geometric drift. We attribute this degradation to a mismatch between the gradual viewpoint transitions seen during training and the large viewpoint changes required at inference time. To address this problem, we propose Dense Trajectory Teacher Distillation (DTTD), a post-training framework that transfers a model’s generation capability from dense trajectories to large camera motions. Given a sparse trajectory, DTTD interpolates additional camera poses to construct a dense trajectory covering the same path. The frozen base model then generates reliable teacher videos along this familiar dense trajectory, and pose-aligned teacher observations are used to evaluate candidate videos generated under the original sparse trajectory. In this way, the model serves as its own generation teacher, requiring neither additional video data nor a stronger external video generator. We further combine dense-teacher supervision with geometry-aware rewards that assess cross-frame consistency and camera-trajectory following, and use the resulting preferences for offline preference optimization. To stably expand the model’s motion range, we introduce a cumulative curriculum that progressively incorporates larger motion steps while retaining simpler trajectories to reduce capability forgetting. Experiments demonstrate that DTTD effectively alleviates frozen-frame degradation, improves trajectory following and cross-frame geometric consistency under large-step camera motions, and preserves the visual quality of the base model

Primary area:
generative models
Keywords:
Video Generation, World Model, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_haCfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a post-training framework called DTTD (Dense Trajectory Teacher Distillation), designed to address specific limitations in video world models—namely, static imagery, inaccurate camera motion, and significant geometric drift—that arise when these models encounter sparse, large-stride camera movements.

Critical strengths
  1. This paper proposes a "Self-Teacher" approach, in which a foundation model acts as its own teacher; by leveraging the model's inherent familiarity with dense trajectories—facilitated through interpolation—it eliminates the need for additional data or external models.
  2. Experimental results demonstrate significant improvements in 3D consistency and trajectory-following capability, particularly in scenarios involving large-stride camera movements.
  3. A training strategy analogous to curriculum learning has been designed, with its effectiveness validated through experiments.
Critical weaknesses
  1. The method was only evaluated on ScanNet; its performance on other datasets remains unknown. Given that DL3DV features more unstable camera trajectories with a wider range of motion, why was it not tested on this dataset?
  2. Some of the examples produced by this method, as shown in Figure 4, are not of particularly high quality; for instance, the cabinet in the fifth row exhibits some deformation and distortion, while the canvas in the sixth row appears somewhat visually blurred.
  3. During the training phase, it appears that multiple candidate videos must be generated for each student scenario, with several "teacher" videos sampled for each dense trajectory. Furthermore, depth estimation and geometric re-projection evaluations must also be performed. This could lead to excessively high computational overhead and slow training speeds; however, the paper does not seem to report these metrics, nor does it provide comparisons with other methods in this regard.
Questions for authors

See Weaknesses

Reviewer_proTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DTTD improves camera-conditioned video generation under large inter-frame camera motions. It uses the frozen base model to generate dense-trajectory videos, then evaluates sparse-trajectory candidates against pose-aligned teacher frames. These preferences are combined with geometry-aware rewards and a cumulative motion curriculum for Flow-DPO training.

Critical strengths

1 The dense-trajectory teacher is an intuitive, task-specific extension to geometry-based preference alignment, such as VideoGPA. It exploits the base model’s existing capabilities without requiring an external video generator.

2 The reported improvements are consistent across motion settings, with baselines trained on the same data and number of steps. At s=1s=1, DTTD improves 3DCS from 0.6534 to 0.5825 and CamMC from 0.6852 to 0.6597 over VideoGPA. The appendix provides useful ablations of teacher rewards, geometric rewards, and curriculum strategies.

Critical weaknesses

1 The benefit over simpler uses of the dense teacher remains unclear. There is no quantitative comparison against generating a dense video and subsampling it at the requested poses, or supervised fine-tuning on these subsampled teacher videos. These comparisons would establish the quality–cost tradeoff and whether preference-based transfer offers an advantage over direct distillation.

2 The trajectory protocol is insufficiently specified. Equation (2) changes sequence length with sampling interval, but actual teacher/student frame counts and any windowing or truncation are not reported. This makes it difficult to separate motion-spacing effects from sequence-length effects and assess generation cost. Evaluation sample counts, reward weights, and the preference-gap threshold are also missing.

Questions for authors

1 How does DTTD compare with dense generation followed by subsampling, and with supervised fine-tuning on subsampled teacher outputs?

2 What are the exact frame counts at each sampling interval? How are the same path endpoints preserved when handling different sequence lengths?

3 Please report evaluation sample counts, uncertainty estimates, and the missing reward settings and generation costs.

Reviewer_c7U9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the degradation of camera-controlled video models under sparse trajectories with large inter-frame motion, where generations tend to freeze, drift, or ignore the camera path. It proposes DTTD, which interpolates each sparse trajectory into a dense one and uses the frozen base model’s own dense-trajectory videos as pose-aligned teachers to score candidate videos generated under the sparse trajectory. These teacher scores are combined with geometry-based rewards for reprojection consistency and trajectory following to build preference pairs. The model is then fine-tuned with Flow-DPO and LoRA under a cumulative curriculum over increasing motion intervals. Experiments on ScanNet with a 1.3B Wan camera-control model compare against Epipolar-DPO and VideoGPA using visual quality, camera-following and 3D consistency metrics.

Critical strengths

The central idea is simple and appealing. The model’s own generations under a familiar dense trajectory serve as pose-aligned references for the same path sampled sparsely, which yields a reward without extra video data or a stronger external generator. This matters because it offers a cheap, self-contained way to extend the motion range of existing camera-controlled models, and Table 3 shows consistent reductions of about 16 percent in 3DCS across all motion intervals.

Critical weaknesses

The evaluation does not establish that DTTD is preferable to simpler alternatives. The most direct baseline is missing, namely generating along the interpolated dense trajectory at inference and subsampling the frames. So is supervised fine-tuning on real sparse ScanNet frames, even though ScanNet provides real videos. In addition, 3DCS, MVCS and CamMC are computed with the same kind of VGGT-based geometry estimation used in the rewards. On the independent SIFT-based Epipolar metric, DTTD is behind Epipolar-DPO, and no ground-truth metrics such as PSNR or LPIPS are reported. The claim of alleviating frozen frames is also not supported by Table 1, where DTTD has a lower Dynamic Degree than the base model at s equal to 0.5 and 1. These issues are fixable, but without them the benefit of the method remains unclear.

The method description and results contain gaps and inconsistencies that weaken confidence in the findings. The paper does not explain how a fixed-length base model generates a teacher video along a trajectory that is eight times denser. The DTTD imaging quality at s equal to 0.125 is 0.4498 in Table 1 but 0.4398 in Table 4. In Tables 5 and 6, single components outperform the combined rewards in CamMC, contrary to the text. All experiments use one indoor dataset and one 1.3B model, never test motion beyond the largest trained interval, and report no variance over seeds.

Questions for authors

1.Report a baseline that generates along the interpolated dense trajectory at inference and subsamples the frames at the sparse poses, together with its inference cost. This would show whether post-training is needed at all.

2.Explain how the teacher videos are generated when the dense trajectory has many more frames than the base model’s native length. Please also state whether the geometry model used for evaluation is the same as the one used for the rewards.

3.Report ground-truth based metrics against the real ScanNet frames, and results for motion intervals larger than s equal to 1. Please also explain why the Dynamic Degree of DTTD is lower than that of the base model at large intervals, given the claim of reducing frozen frames.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.