- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies camera-controlled video re-rendering: generating a frame-synchronized target video along a specified camera trajectory while preserving the source scene’s appearance and dynamics. Track2View conditions a pretrained video diffusion transformer on paired 3D point tracks. Its dual-view conditioner transfers source-video features through deterministic bilinear sampling and scattering, combined with learned temporal aggregation and view-specific depth encoding. The paper also introduces a reverse-and-concatenate pipeline for constructing paired track annotations. Evaluation includes a 400-video real-world benchmark and additional controlled experiments on Kubric-4D.
Critical strengths
- Paired 3D tracks provide explicit geometric feature transfer and temporal aggregation for synchronized camera-controlled video generation.
- Cross-dataset evaluation provides convincing evidence of generalization: the model trained on synthetic MultiCamVideo is tested on real videos from RealEstate10K and MiraData.
Critical weaknesses
- Table 2 does not consistently favor the ALL (1152) configuration: Compared with First only (576), the ALL (1152) configuration shows only small gains and even increases TransErr.
- Each ablation comparison is evaluated on only one benchmark. Repeating the ablations on both RealCam-Vid and Kubric-4D Moving-20 would better establish the robustness of the design choices across real and synthetic settings.
Questions for authors
Could you compare with a monocular-compatible 4D Gaussian Splatting baseline [1] that reconstructs a dynamic scene from source frames and poses, then renders the target poses? Comparing training costs and evaluation metrics with explicit reconstruction methods would better clarify the benefits of generative re-rendering.
[1] Wu G, Yi T, Fang J, et al. 4d gaussian splatting for real-time dynamic scene rendering[C]//2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024: 20310-20320.
4 comments
Response to Reviewe GZ8b (Part2)
Q1 — 4D Gaussian Splatting baseline run on Moving-20 with ground-truth poses and point-cloud initialisation (Table Q1): 19.10 vs. 22.16 dB PSNR, 90.7% vs. 100% coverage, 503 s of per-scene optimisation vs. none.
We ran the comparison with the official 4D Gaussian Splatting code [1] on the Kubric-4D Moving-20 benchmark of Tab. 6, where both the input poses and the ground-truth target views exist. For each of the 20 scenes, 4DGS is optimised from scratch with its default D-NeRF configuration on the 81 source frames, with ground-truth camera poses and ground-truth point-cloud initialisation, and then renders the target views of Tab. 6; Track2View is evaluated as in Tab. 6. Because 4DGS can only reconstruct what the source trajectory observed, we also report its coverage and its PSNR restricted to the covered pixels (Table Q1).
4DGS reaches 19.10 / 0.560 / 0.314 (PSNR / SSIM / LPIPS) with 90.7% coverage and 21.88 dB on the covered pixels, after 503 s of optimisation per scene on one H100; Track2View reaches 22.16 / 0.675 / 0.196 with 100% coverage, 22.76 dB on the same pixels and no per-scene optimisation. The gap grows with scene dynamics: splitting the 20 scenes at the median number of dynamic objects, 4DGS obtains 20.84 vs. 22.88 dB on the less dynamic half and 17.35 vs. 21.44 dB on the more dynamic half. 4DGS is ahead only on scenes with small camera and object motion and falls to 12–14 dB under large motion; since its train-view PSNR is 43 dB, the failure is extrapolation beyond what the source trajectory observed, not under-fitting. Generative re-rendering therefore brings three things a per-scene reconstruction cannot: no optimisation at test time, generation of the regions the source never observed, and stable dynamic content. We will add Table Q1 to the appendix.
Table Q1 — 4D Gaussian Splatting vs. Track2View on Kubric-4D Moving-20 (20 scenes, target frames of Tab. 6)
| Method | Per-scene optimisation | PSNR↑ | SSIM↑ | LPIPS↓ | Coverage | PSNR on covered pixels |
|---|---|---|---|---|---|---|
| 4DGS [1] | 503 s / scene (H100) | 19.10 | 0.560 | 0.314 | 90.7% | 21.88 |
| Track2View (ours) | none | 22.16 | 0.675 | 0.196 | 100% | 22.76 |
PSNR by scene dynamics (the 20 scenes split at the median number of dynamic objects)
| Scenes | 4DGS [1] | Track2View (ours) |
|---|---|---|
| Fewer dynamic objects | 20.84 | 22.88 |
| More dynamic objects | 17.35 | 21.44 |
References [1] 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering. CVPR 2024.
We hope these results address the reviewer’s concerns and would be glad to clarify any remaining point during the discussion period.
The authors’ response addressed my concerns. I consider this a solid paper with strong theoretical and empirical support. I have a minor reservation about the paper’s novelty. My overall score remains 3.
We sincerely thank the reviewer for the positive feedback and for confirming that our response addressed the concerns.
Response to Reviewe GZ8b (Part1)
We thank the reviewer for the careful reading, valuable feedback, and the constructive suggestions. Following them, we added experiments and analysis. Our point-by-point responses follow.
W1 — Table 2 does not favor All. With the 8-layer model on both benchmarks (Table W1), the result is driven by the accuracy and the coverage of the tracks, not by their number.
We repeated the ablation with the final 8-layer model on RealCam-Vid and report the same comparison on Kubric-4D Moving-20 with ground-truth tracks (Table W1). We do not claim that using all tracks is best. Our view, which the two benchmarks support, is that generation quality depends on the accuracy and the coverage of the conditioning tracks rather than on their number.
Coverage: First only has as many tracks as Rand. ½ but places all queries in a single frame, and it is the weakest setting in camera accuracy on RealCam-Vid and in PSNR on Kubric-4D, whereas halving or quartering the tracks at random while keeping both query frames costs little (within 0.06 CLIP-V on RealCam-Vid and 0.35 dB on Kubric-4D). Accuracy: App. E shows that a confidence-selected subset is more accurate than the dense pool and that, with estimated tracks, adding more tracks lowers PSNR. All is our default because it covers both query frames without a selection heuristic. We will replace Table 2 by Table W1 and rewrite the corresponding text in terms of accuracy and coverage rather than track density.
Table W1 — Ablation on the query strategy, 8-layer model (RealCam-Vid and Kubric-4D Moving-20)
W2 — Both ablations are now reported on both benchmarks: temporal depth (Table W2) and query strategy (Table W1); the trends agree.
We agree and have repeated the temporal-depth ablation of Table 3 on the second benchmark. Table W2 reports the number of temporal aggregation layers on RealCam-Vid (Table 3) and on Kubric-4D Moving-20 side by side, with all query tracks in both cases. The trend is the same on the two datasets: two and four layers are close, and the gain comes from eight layers. Table W1 above does the same for the query-strategy ablation of Table 2: the two benchmarks agree that the differences between strategies are small and that the weakest setting is the one with the poorest coverage (First only), not the one with the fewest tracks. We will add both tables to the appendix.
Table W2 — Ablation on temporal aggregation depth (All queries)