- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents VoxelMem, a method that uses 4D voxel as proxy to guide the video generation process, resulting in fine grained control and consistent scene. The main contribution of the paper is the presented hybrid voxel memory. The paper also extend the method to streaming generation. A data pipeline for both real world and synthetic scenes is proposed to support the network training.
Critical strengths
- The method leverages 4D voxel as a genereation reference for video models, performing as a unified representation for both fine-grained control and long-term memory.
- The design of the hybrid memory system is novel and effective. The ablation study shows the effectiveness of this mechanism.
- The paper adapts the proposed memory mechanism into a streaming mode, enhancing the model's usability.
Critical weaknesses
- Although performing better than the baseline methods, the input condition of this method, the 4D voxel, is much harder to obtain compared to conditions like point cloud or trajectory, limiting its scope of application。
- Since dataset contruction is an important contribution of the paper, putting its core pipeline in the main paper is needed.
- The state of the voxel is static and updated according to a temporal interval, meaning that when the video model drifts, the local memory can still be affected. Meanwhile, when the appearance of the object changes (reflection or dynamic effects, for example), such static voxel memory can also restrict these geometry-unrelated dynamics.
- The evaluation remians in short video clips (500 frames), more experiments on longer-term scalability are needed to evaluate its streaming generation ability.
Questions for authors
- How is the scale of the voxel (resolution) decided? There is no clear number in the paper.
- Although the scale of each voxel in 3D is equal, when rendered in 2D screen, the scales may differ a lot. When the camera is close to the voxel, will the generation be affected? Since all the local memory and the control condition latent will be similar in the entire frame. Will this affect the prior knowledge of the video model?
Dear Reviewers,
This is a friendly reminder that the rebuttal discussion period is approaching its deadline.
Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.
As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.
Thank you for your time and effort.
Best regards,
Area Chair