- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a new method, namely, SyncVid, to generate synchronized video from a single source video. The main idea is to replace standard Gaussian noise with a structured noise in canonical 3D space. In particular, the proposed method first reconstructs a dynamic 4D Gaussian representation from the source video. Given Gaussian primitives, fixed noise vectors are assigned to the 3D Gaussian primitives and rasterized into target camera views. A view-consistent whitening is further introduced to reduce local correlations while preserving cross-view correspondence. The structured noise feature, source, and rendered video embeddings are concatenated as the input of a video diffusion model, where the video diffusion model is fine-tuned with LoRA. Experiments show that the proposed method outperforms state-of-the-art approaches in overall view Synchronization, temporal consistency, and camera accuracy, as well as in human preference evaluations.
Critical strengths
-
The paper is well motivated and interesting.
-
The paper provides a theoretical analysis of how whitening reduces local correlations
-
Extensive experiments and the demo show that the proposed method outperforms the state-of-the-art methods.
Critical weaknesses
-
The main contribution is to replace typical standard Gaussian noise with the noise that preserves correspondence across views. However, as shown in Table 4, canonical 3D noise without whitening reduces FVD-V from 518.77 to 468.39. It is unclear whether the improvements mainly come from cross-view noise correspondence. It would be better to provide an additional ablation that removes cross-view noise correspondence. For example, first independently sample Gaussian noise for different target views and combine it with the whitening procedure.
-
The paper introduces the whitening procedure, which is claimed one of the main contributions. However, the ablation study results show that whitening only reduces FVD-V from 468.39 to 461.44. In addition, the RFF frequency scale plays an important role in the whitening procedure. Yet, it seems that the paper does not provide the details of how this parameter is determined.
Questions for authors
How are the RFF frequency vectors and scene-dependent frequency scale determined? How sensitive is the whitening procedure to these parameters?
3 comments
Dear Reviewer yDkL,
Thank you once again for your valuable and positive comments and the time you have invested in reviewing our paper. As the discussion period is nearing its end, we would greatly appreciate hearing whether our responses have helped clarify and address the concerns outlined in your review.
Thank the authors for the detailed response and additional experiments. The new ablation studies show the benefits of noise correspondence and whitening, and the additional analysis of the RFF frequency scale addresses most of my concerns. I will maintain my positive rating.
We thank the reviewer for the positive assessment and encouraging recommendation. We are pleased that the reviewer found the paper well motivated, the theoretical analysis useful, and the experiments convincing. We address the two weaknesses and the two questions below; the new results have been added to the revision (in blue).
W1. Do the gains come from cross-view noise correspondence?
We agree that this should be isolated directly. We removed the correspondence where it acts: the per-primitive noise vectors zi are resampled independently at every frame, so the noise keeps the same rasterized structure and is whitened identically, but no longer follows the same scene content, neither over time nor between views:
With the correspondence removed, whitening alone brings no benefit: the variant performs on par with, or slightly below, standard i.i.d. noise on all four metrics, whereas the same whitening applied to shared canonical 3D noise yields the full gain. The synchronization improvement therefore comes from binding the noise to persistent scene content, and whitening is what lets the pretrained model exploit it. We note that the existing Table 4 already gives partial evidence: shared 3D noise without whitening reduces FVD-V from 518.77 to 468.39, i.e. most of the FVD-V improvement is obtained by the correspondence alone, before any whitening is applied.
W2/Q1/Q2. Effect of whitening and how the RFF frequency scale is set.
Effect of whitening. FVD-V alone understates the role of whitening. Its role is not mainly to further improve distribution-level similarity, which the correspondence already provides, but to remove the local spatial correlations introduced by overlapping splat footprints (Eq. 12), which push the noise away from the i.i.d. distribution the pretrained model expects. Without whitening, the structured noise improves FVD-V but degrades Mat. Pix. (222→208), CLIP-V and CLIP-F relative to standard noise. Whitening reverses this: it raises Mat. Pix. by 28% (208→267), CLIP-V by 2.5 points (88.53→91.05) and CLIP-F by 0.7, and the full method is the only variant that improves all four metrics over standard noise. We also measured this decorrelation directly (App. A.8, Fig. 9): the lag-1 correlation between neighbouring noise values drops from 0.18 before whitening to 0.09 at the reference scale s0 and to 0.02 at 16s0.
Q1. How the RFF frequency vectors and the frequency scale s are set. The fixed frequency vectors of Eq. 14 are random unit directions scaled by dyadic octaves, ωa=2ℓara with ℓa∈{0,…,L−1} and L=12, so the projection spans frequencies from s to 2L−1s. Because the feed-forward reconstruction has an arbitrary metric scale, we tie s to the correlation length of the rasterized noise, ℓc=(2σˉ)2+dˉ2, from the median footprint radius σˉ and the median 3-NN distance dˉ of the visible Gaussians, and set s0=1/(2⋅2L−1ℓc), i.e. the highest frequency sits at the Nyquist limit of ℓc (Eq. 35, App. A.8). Higher scales alias and become sensitive to small errors in Xp; lower scales leave neighbouring bases nearly identical. s0 is computed in closed form, with no tuning.
Q2. Sensitivity. Sweeping s=Ms0 (Wan2.2-5B, +both; Table 9 of the revision):
The results are sensitive to s and degrade on both sides of M=1, which is best on all four metrics, as predicted by Remark 1 (App. A.3); every swept scale still improves FVD-V over i.i.d. noise. At the noise level (Fig. 9), increasing M removes more of the neighbour correlation (6%, 50%, 93% at M=161,1,16) but keeps less of the cross-frame agreement (98%, 64%, 19%); the untuned, closed-form s0 lands at the balance point.
We thank the reviewer again for the constructive suggestions, which we believe strengthen the paper's central claim. If there are any remaining questions, concerns, or aspects that would benefit from further clarification, we would greatly appreciate your feedback. Your insights are invaluable to us, and we would be grateful for any additional suggestions that could help strengthen the paper.