- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes ViSPaC, a method for camera-controlled video diffusion. Instead of doing scene reconstruction or using camera pose embeddings, this work projects synthetic 3D landmarks into 2D colored points to represent camera motion as an image sequence. The video and landmark images are then trained together in a diffusion transformer. The authors also use a spatial codebook to change the landmark layout for different camera paths, and test on DL3DV and Tanks and Temples.
Critical strengths
- Interesting representation idea: Using synthetic 3D landmarks to turn camera poses into visual point tracks is an interesting idea and avoids doing explicit scene reconstruction.
Critical weaknesses
- No video results at all: This is a video generation submission, but there are zero video results. There is no supplementary website, no link, and no zip file with videos. The paper only shows a few static image frames in the pdf. But video generation papers need videos to check temporal consistency, motion smoothness, camera precision, and artifacts like flickering. You cannot evaluate a video paper from static frames. Having no videos is enough to reject the paper right away.
- Only static scenes: The paper only evaluates on DL3DV and Tanks and Temples, which are just static scenes. Real videos have moving objects, but it is not clear at all how this landmark projection handles dynamic scenes or if it completely breaks down when objects move.
- Decoding seems very fragile: Decoding camera poses needs detecting tiny colored dots, reading their RGB values, and running PnP. As shown in Table 5, VAE compression creates a lot of color mismatch errors. Putting layout settings into tiny corner dots also seems very easy to break if the video has blur or artifacts.
- Method is overly complicated: The pipeline has too many hand-crafted parts like discrete color bins, corner marker palettes, layout codebooks, and disk size rules. It is not clear why we need such a complex pipeline when other camera conditioning methods are much simpler.
Questions for authors
The idea is interesting, but having no video results for a video generation submission is not acceptable. I would like authors to address following questions:
- Why are there no videos or supplementary website included? How can reviewers judge the video quality and camera control without seeing any videos?
- How does the method handle dynamic scenes with moving objects?
- How robust is the pose decoding when the generated frames have blur or compression artifacts?
- Why do we need such a complex landmark codec when simpler camera conditioning methods already work well?
I am negative and vote for clear rejection because of the missing videos, but I am happy to see what authors say in the rebuttal.
9 comments
Response to Reviewer 2xX7 – Part 2/2
W4 / Q4: Necessity and complexity of the landmark codec
Methods based on scene geometry provide explicit visual guidance, but reconstruction errors can introduce inaccurate or inconsistent constraints that the generator must accommodate. Dynamic scenes further complicate maintaining consistent geometry and correspondences across frames. In the supplementary video demonstrations, Lyra-2 and Gen3C show comparatively limited object motion. Uni3C can produce natural object motion, yet its static-scene results still deviate from the prescribed camera path despite point-cloud guidance. Pose- and ray-based conditioning encodes geometric relationships, but the generator must still learn how these relationships translate into image changes. The higher camera errors of these baselines relative to Lyra-2 illustrate the remaining gap in control accuracy.
ViSPaC exposes camera-induced visual motion through projections of a known synthetic reference, without depending on scene reconstruction. Its identifiable point trajectories express motion through the spatial patterns and temporal changes that video models are trained to process, while preserving geometric correspondences for pose recovery. For this representation to remain useful, the points need to be distinguishable after visual reconstruction and distributed so that the camera continues to observe enough of them along its path.
As discussed in Q3, maintaining color separation for reliable readout limits the number of landmark identities. With a fixed identity budget, spreading points over a larger region reduces their density, and a fixed layout may provide poor coverage for some trajectories. The spatial codebook therefore adapts scale and layout to the trajectory while preserving color identities. Configuration markers specify the mapping used for decoding, since the same color identity can refer to different 3D positions across configurations. Rendering landmarks as disks provides an interior region for color estimation, with disk size balancing this support against overlap.
Under the shared joint framework in Table 3, ViSPaC reduces ATE from 0.1161 for raw pose and 0.0922 for Raxel to 0.0227, while also improving pose readout. Removing spatial adaptation or increasing the identity count to 1,404 raises ATE to 0.0285 and 0.0398, respectively; both variants also degrade pose readout. These results support both the visual-pose representation and the contribution of its capacity and coverage choices to camera control and geometric recovery.
Thanks for the rebuttal. The dynamic video results are not very clear. What is the target camera trajectory? The camera is barely moving? Ideally also you show multiple different camera trajectories per dynamic video.
Response to Reviewer 2xX7 – Part 1/2
We thank the reviewer for recognizing our representation of camera motion without scene reconstruction. Below, we provide video evidence and address dynamic scenes, decoding robustness, and the codec's design rationale.
We apologize for the missing video evidence. The accompanying supplement (ViSPaC_Supplementary_Videos.html) provides synchronized side-by-side comparisons on TNT, DL3DV, and dynamic scenes.
The fixed landmarks form a synthetic 3D reference for camera motion and are not attached to scene objects. Their projections depend only on the prescribed trajectory and intrinsics, so fixing this reference does not require the generated scene to remain static.
We test this with a separate exploratory evaluation using 30 images selected from the VBench-I2V Image Suite, each paired with a prompt describing scene motion and a prescribed camera trajectory. The table reports camera-following errors, Dynamic Degree (DD), Motion Smoothness (MS), Imaging Quality (IQ), and Overall Consistency (OC).
Table R7. Camera control and generation quality on dynamic scenes.
Notes: Best results are bold; second-best are underlined. VBench scores are multiplied by 100. DD includes camera-induced motion; OC measures video–text consistency.
ViSPaC achieves the highest DD, the second-lowest ATE and RPE-t, and the second-highest MS and OC. In the supplementary comparisons, ViSPaC generates object motion alongside camera motion, while Lyra-2 and Gen3C show comparatively limited object motion. Compared with RealCam-I2V, Uni3C, and SCoPE, which also generate scene dynamics, ViSPaC has lower ATE and RPE-t. Together, these results support camera control with dynamic scene content.
The codec is designed to tolerate visual reconstruction errors through discrete color encoding with separated codewords. Both landmark identities and configuration choices are decoded by matching observed colors to discrete codes. Color perturbations that remain within the accepted region of the same code leave the recovered identity or configuration unchanged, so exact RGB reconstruction is unnecessary.
This tolerance can be quantified analytically. With RGB normalized to [0,1], correct color decoding is preserved when the absolute error in each channel of the estimated interior color is below the following bounds:
Table R8. Analytic error bounds for color readout.
For landmarks, we limit the number of identities to retain sufficient color separation. Table 5 directly examines identity recovery after VAE compression: mismatch among matched landmark detections is 0.19%/2.98% for the 210/432 identities used by ViSPaC, compared with 18.53% at 1,404 identities. This motivates the chosen identity budgets. We also estimate landmark colors from disk interiors, reducing the influence of color mixing at blurred boundaries.
The corner configuration markers follow the same discrete-coding principle, with only a few choices per marker—for example, eight layouts—represented by widely separated palette colors. Each marker occupies a fixed location in a reserved region and is rendered as a radius-9-pixel disk. We read the median color from its central 9×9 region and select the nearest palette entry. The separated palette preserves configuration decoding for color errors within these margins, while interior median aggregation reduces the influence of blurred boundaries.