- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes MultiLive, a framework for generating synchronized multi-view videos from text and camera poses without synchronized multi-view video training data. It distills a video teacher and a camera-conditioned multi-view teacher through distribution matching on complementary slices of the same student-generated sample. A geometry-aware adapter enables interaction across time and views, while a shared fake-score model uses separate forward passes for the two objectives. The multi-view teacher is trained on causal VAE representations of repeated static images to support supervision at different latent-time positions. Experiments demonstrate the effectiveness of the proposed method.
Critical strengths
- The proposed method applys both teachers to the same generated sample is a useful approach to learning without synchronized multi-view video data, and the comparisons with mixed-data SFT, single-teacher distillation, and sequential distillation support this design. This is interesting and novel to me.
- Experiments demonstrate the proposed method outperforms other baselines and achieves better results.
- The paper is well-written and easy to understand.
Critical weaknesses
- The performance of the multi-view teacher (trained by the authers) is not reported, making it difficult to assess the quality of the geometric prior and how well it is transferred to the student. I would recommend that the authors evaluate the teacher’s visual quality, camera accuracy, and cross-view consistency, and compare these with the corresponding student outputs under matched condition.
- The proposed method involves two teachers, which also makes the training to cost more time and computational resources. As discussed in the paper, the proposed method is limited to four 81-frame videos currently. I would recommend that the authors clarify the maximum supported sequence length and provide ablations with different frame counts and numbers of views, reporting generation quality, inference time, and peak GPU memory usage.
Questions for authors
N.A.
2 comments
Response to Reviewer zfcK (2/2)
W2: The two teachers make training cost more time and computational resources, and the method is currently limited to four 81-frame videos. Please clarify the maximum supported sequence length and provide ablations with different frame counts and numbers of views, reporting generation quality, inference time, and peak GPU memory usage.
Thank you for raising this practical concern. Below, we report the training and inference costs, clarify the supported sequence length, and provide the requested ablations over frame counts and numbers of views.
Cost of two teachers. Both teachers are frozen and only run forward passes. They need no gradients or optimizer states, so they add little memory. In the default setting (four views × 81 frames), a training step takes 25.37 s with 42.0 GiB peak memory, and distillation converges in only 2,000 iterations. More importantly, the teachers are used only during training. At inference, only the student runs, in 4 steps without CFG. Four views take 89.65 s with 22.8 GiB peak memory, and two views take 38.12 s with 18.6 GiB, over 10× faster than SynCamMaster (400.31 s, Table 1).
Maximum sequence length. The joint attention and both readouts are defined for any number of views and latent times . In fact, the model trained with four views directly generates eight views without retraining (last row of the table below). In practice, the sequence length is set by the video teacher, whose scores are most reliable within its pretraining length. We therefore use 81 frames to match Wan. This is not a cost bottleneck: peak training memory is only 42.0 GiB, about half of a single 80 GB GPU. With a video teacher pretrained on longer videos, MultiLive can directly distill longer sequences.
Different numbers of views and frames. We train MultiLive with 2 or 4 views and 33 or 81 frames. We also apply the default four-view model to eight views at inference. Latency and peak memory are measured for generating all views at inference:
| Views | Frames | FID ↓ | FVD ↓ | CLIP-T ↑ | CLIP-F ↑ | RotErr ↓ | TransErr ↓ | Mat. Pix. (K) ↑ | FVD-V ↓ | CLIP-V ↑ | Latency (s) ↓ | Peak Mem. (GiB) ↓ |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2 | 33 | 114.64 | 113.52 | 29.80 | 99.71 | 14.25 | 19.23 | 370.75 | 32.02 | 96.34 | 10.62 | 18.4 |
| 2 | 81 | 112.33 | 109.61 | 30.47 | 98.87 | 16.67 | 16.94 | 387.54 | 36.49 | 97.02 | 38.12 | 18.6 |
| 4 | 33 | 117.56 | 114.63 | 30.68 | 99.78 | 14.26 | 16.55 | 429.91 | 25.62 | 97.96 | 26.96 | 18.5 |
| 4 | 81 (default) | 114.67 | 111.32 | 30.67 | 99.69 | 13.18 | 18.21 | 443.26 | 24.79 | 97.91 | 89.65 | 22.8 |
| 8 (trained on 4) | 81 | 121.93 | 131.64 | 29.31 | 98.87 | 12.33 | 15.71 | 467.02 | 22.27 | 96.98 | 348.14 | 31.5 |
MultiLive works robustly across the trained settings, with similar visual quality and camera accuracy. Notably, cross-view consistency improves with more views: Mat. Pix. rises from two to four to eight views, and FVD-V drops. Since all views are generated jointly, more views constrain each other more, which shows the benefit of joint multi-view generation. The eight-view result also shows that MultiLive generalizes beyond the number of views seen in training.
Response to Reviewer zfcK (1/2)
Thank you for the helpful suggestions and encouraging feedback!
Thank you for the suggestion. We evaluate the multi-view teacher's visual quality, camera accuracy, and cross-view consistency, and compare them with the corresponding student outputs under matched conditions. When training the multi-view teacher, we update only the adapter and keep the Wan backbone frozen, so the teacher can also generate multi-view videos directly. We therefore compare both images and videos:
The multi-view teacher achieves low camera error and high cross-view consistency, providing a strong geometric prior. This prior is well transferred to the student: on images, the student matches the teacher in cross-view consistency, and on videos, it clearly surpasses the teacher. The teacher only learns cross-view consistency within a single time step, so its consistency degrades when it generates whole videos. During distillation, however, it only needs to score the student one time step at a time, which is exactly what it does well.
The multi-view teacher offers more precise camera control, but this comes at the price of more denoising steps and less dynamic motion. Bringing such camera precision to few-step generation is a promising direction for future work.