- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This manuscript presents a method for converting a static 3DGS scene into a 3D cinemagraph with diverse dynamics, without relying on masks or physical simulation. Given a reference image rendered from the 3DGS, the method first uses a proprietary LLM and video generation model to produce an approximately looping reference video. This video is then lifted and projected to multi-views, and inpainted as multi-view videos. The original 3DGS renderings, the reference video, and the inpainted multi-view videos are used to train the proposed Inconsistency-Robust Periodic 4DGS, where the Periodic Deformation Field is used to capture periodic motion, the Grounded Drift Field is designed to absorb inconsistencies among the generated videos during training. When optimizing the reference view, the Drift Field is disabled to encourage the Periodic Deformation Field to explain the desired periodic motion. This manuscript also introduces a perceptually-aligned ground-truth-free evaluation framework for 3D cinemagraph, under which the proposed approach shows a clear advantage over the alternatives.
Critical strengths
- This work explores an interesting problem, and the generated videos are overall satisfactory.
Critical weaknesses
- Grounding the drift field sounds reasonable in principle. But in many scenes, anchoring a single view is insufficient to determine the its 3D and motions, so it is unclear whether this mechanism can work as expected. So in addition to the quantitative evaluation, it is suggested to include some qualitative ablation about the grounding mechanism, preferably in video. Moreover, if the goal is to enlarge the motion amplitude modeled by Periodic Deformation Field, an appropriate regularization term might already be sufficient. It would be useful to include such baselines.
- The merit and necessity of the mask-free need further justification. Based on the ablation videos in the supplementary material, the improvement brought by the drift field seems to be happened mainly in static regions. Similar improvements could potentially be achieved at lower cost by just using a mask during optimization.
Questions for authors
- Why not directly use Fourier series to model the periodic attributes instead of the features?
3 comments
I have reviewed the rebuttal and supplementary materials and been satisfied with the newly provided results and discussions. For W2, while I agree that the authors' argument sounds overall reasonable, it cannot directly address my original concern stemminng from an observation of the video ablation in supplementary material, i.e., the improvement in the presented case happened mainly in static regions. In this case, we don’t really need an “accurate 3D mask”, a coarse and conservative mask would be already enough to eliminate artifacts in regions far from the dynamic areas. Overall, I decide to maintain my original score.
Follow-up on W2: Drift Field also matters in dynamic regions
We thank the reviewer for the constructive feedback. We observe that the effect of removing the Grounded Drift Field is not limited to static regions: dynamic regions also degrade, as the Marble/world scene of the supplementary ablation shows:
- Within the motion. The rotating galaxy is noticeably blurrier w/o Drift. The inconsistency lies in the moving content itself, so masking cannot help.
- Along the motion path. As the light sweeps across the stones, w/o Drift averages them out until they nearly vanish. A coarse mask over the lighting change will covers these stones, so they cannot be frozen as static, and the mask cannot prevent them from vanishing.
To quantify this, we measure sharpness (VoL) only within the dynamic region of each video, defined as the union of the pixels flagged by the motion and illumination terms of our Vividness metric (App. E).
| Dynamic-region VoL ↑ | |
|---|---|
| w/o Drift | 938.10 (−1.43%) |
| Ours | 951.72 |
In some scenes the drop is much larger: World from Marble (−26%), City Entrance from Marble (−25%), Great Wall from Marble (−17%) and Snowy Riverbank from Lyra (-9%).
A mask can only suppress artifacts away from the motion, and cannot isolate boundaryless effects such as smoke, fog, and lighting changes. A reasonable coarse mask is also hard to obtain: the motion region is unknown before generation, and the dilation margin needs per-scene tuning. The Drift Field handles artifacts within, along, and away from the motion, without any mask.
We sincerely thank the reviewer for pushing the discussion on W2 further, which helped us clarify the strengths and limitations of our approach relative to mask-based methods.
Response to Reviewer Gxtg
We thank the reviewer for recognizing (1) an interesting problem and (2) satisfying generated videos. We address both concerns and the question below.
W1 — Whether grounding to a single view is enough.
What grounding does. A per-view Drift Field absorbs the disagreement between generated views during training and is discarded at inference. Grounding removes the drift at the reference view, so the drift cannot take over the scene motion.
Qualitative ablation. We added w/o Grounding vs. Ours videos at the bottom of the supplementary index.html. Without grounding, the drift absorbs scene structure and details (City Explosion) or the intended motion, such as the lake ripples in Mountain Lake, instead of only cross-view disagreement.
Limitation. We agree grounding anchors the drift at one view only. If some training views differ from it by a very large angle, it may not keep the motion out of their drift. We will state this in the limitations.
Drift-magnitude regularizer instead of grounding. As suggested, we replace grounding with a penalty λ‖Δ‖² on the drift (λ = 10³), applied to all views including the reference; all else unchanged. D/M measures drift size relative to the deformation, Vividness how much the scene moves, and VoL sharpness. All 39 scenes, Table 2 protocol:
*D/M: drift translation / deformation translation, averaged over views. Percentages relative to Ours.
W2 — Why mask-free, instead of a motion mask during optimization.
index.html; Fig. 12 in the revision). Too wide, and static regions are included, bringing the artifacts back.Q1 — Why put the Fourier series on triplane features rather than per-Gaussian attributes?
Following 4DGS, which models motion with a HexPlane, we place the Fourier coefficients on a triplane of the same form. Compared with per-Gaussian coefficients, this has three advantages:
We thank the reviewer again, and are glad to answer further within the remaining discussion time.
Sincerely, The Authors