acceptodds
Under review as a conference paper at ICLR 2027

VoxelMem: video generation with 4D voxel control and memory

Abstract

Long video generation remains limited by weak controllability, spatial drift, and memory loss when objects leave and re-enter the view. We propose VoxelMem, a framework that uses 4D voxels as both controllable primitives and memory carriers. Each voxel stores time-dependent geometry, visibility, segmentation identity, and local appearance, enabling multi-granularity control from object-level layout to part-level motion. To maintain long-term consistency, VoxelMem combines local appearance propagated with voxel motion and global history frames retrieved by future voxel coverage, providing both fine-grained spatial correspondence and holistic scene references. We further extend VoxelMem to streaming generation by distilling the bidirectional model into a causal generator. Finally, we construct 4D video–voxel training data from synthetic rendered scenes and real-world videos to support multi-granularity control and memory learning.

Primary area:
generative models
Keywords:
Diffusion

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GZByOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents VoxelMem, a method that uses 4D voxel as proxy to guide the video generation process, resulting in fine grained control and consistent scene. The main contribution of the paper is the presented hybrid voxel memory. The paper also extend the method to streaming generation. A data pipeline for both real world and synthetic scenes is proposed to support the network training.

Critical strengths
  1. The method leverages 4D voxel as a genereation reference for video models, performing as a unified representation for both fine-grained control and long-term memory.
  2. The design of the hybrid memory system is novel and effective. The ablation study shows the effectiveness of this mechanism.
  3. The paper adapts the proposed memory mechanism into a streaming mode, enhancing the model's usability.
Critical weaknesses
  1. Although performing better than the baseline methods, the input condition of this method, the 4D voxel, is much harder to obtain compared to conditions like point cloud or trajectory, limiting its scope of application。
  2. Since dataset contruction is an important contribution of the paper, putting its core pipeline in the main paper is needed.
  3. The state of the voxel is static and updated according to a temporal interval, meaning that when the video model drifts, the local memory can still be affected. Meanwhile, when the appearance of the object changes (reflection or dynamic effects, for example), such static voxel memory can also restrict these geometry-unrelated dynamics.
  4. The evaluation remians in short video clips (500 frames), more experiments on longer-term scalability are needed to evaluate its streaming generation ability.
Questions for authors
  1. How is the scale of the voxel (resolution) decided? There is no clear number in the paper.
  2. Although the scale of each voxel in 3D is equal, when rendered in 2D screen, the scales may differ a lot. When the camera is close to the voxel, will the generation be affected? Since all the local memory and the control condition latent will be similar in the entire frame. Will this affect the prior knowledge of the video model?
1 comment

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards,

Area Chair

Reviewer_cckwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VoxelMem, a long video generation framework that uses 4D voxels as a single representation for control and memory. Rendered voxel canvases guide layout and motion, while image patches carried by voxels and history frames retrieved via the voxel field serve as local and global memory for revisits and moving objects. The model is also distilled into a causal streaming generator and trained on video–voxel pairs from synthetic renders and real videos.

Critical strengths
  • Voxels offer good controllability, and the argument that they connect easily to physics or game engines is reasonable.
  • Overall, the approach increases the controllability of the video generation model.
Critical weaknesses
  • The full 4D voxel sequence must be prepared externally, which limits practicality outside settings where an engine provides the voxels. As noted in the appendix, voxels can be estimated from a source video with an external tracker, but this is difficult when only an input image is given. Existing methods discuss building their representation (e.g., point clouds) in a closed loop from generated frames for the next chunk. In VoxelMem, generated frames are used only to update the appearance stored in each voxel, while the voxel geometry must be given for every frame.
  • The advantages over existing representations claimed in the introduction are not fully convincing. For example, the noise attributed to point clouds mainly comes from the estimation process rather than the representation itself, and voxels estimated by an external model are not free from it either. Point-cloud-conditioned models could also learn to reduce such artifacts when reference frames are additionally provided.
  • Much of the 3D structure and motion central to video generation is delegated to the external voxel input. The method thus appears less about improving the memory of a video generator that must also produce 3D structure and motion itself, and more about using the video model as a renderer on top of given geometry and motion.
  • The main comparison may not be fully fair. FlexTraj and VerseCrafter have no memory, and Spmem and LiveWorld receive point clouds ray-cast from VoxelMem's voxels, which differ from the point clouds these models are designed for and may fall outside their training distribution.
Questions for authors
  • When only an input image is given (no source video or engine), how would the voxel sequence be obtained? Could the voxel geometry be built and updated in a closed loop from generated frames?
  • How robust is the method to noisy voxels, such as those estimated by an external tracker from real videos?

--- Update ----

Thank you for the detailed response. I appreciate the clarifications regarding the scope of this work and the role of the memory module. Also the rebuttal resolved my concerns about the contribution of voxels over point clouds.

However, within this defined scope, my concerns regarding the novelty of the proposed method remain. While the method would be useful as a diffusion-based renderer within external engines or LLM-based agentic workflows, there are already many alternatives for this purpose (e.g., general video editing models such as H3), and the overall approach feels straightforward, offering limited additional insights to the research field. Compared to those alternatives, the main distinction of this paper lies in adopting a voxel representation and providing an in-depth analysis of it.

Nevertheless, I agree that the qualitative results are strong and that the work is practically useful. Although my overall assessment is more positive than before the rebuttal, I will maintain my current score given the granularity of the rating scale.

1 comment

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards,

Area Chair

Reviewer_9sypOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

4D voxels serve as both control primitive and memory carrier. Control is a color-coded voxel canvas channel-concatenated to the latents; memory is hybrid, including history frames retrieved by voxel coverage plus per-voxel RGBA patches warped along voxel trajectories, injected by token concatenation. A distilled causal variant with "interleave-forcing" supports streaming. This work is build on Wan2.2 5B/14B; evaluated on a new revisit benchmark VM-4D and VBench-I2V.

Critical strengths
  1. I like the idea of 4D voxels for control and memory. This work is claimed can handle dynamic situation, which is difficult for traditional 3D based method.

  2. The motivation is good and the general pipeline makes sense.

Critical weaknesses
  1. This work is only ealuated on two benchmarks with one benchmark proposed by themselves.

  2. The key claim is consistency of revist dynamic objects but there seems not specific metric evaluate this issue.

  3. No FPS or latency for the streaming model despite the "real-time web game" application.

  4. Key details missing from the main text: K retrieved frames, voxel counts, patch resolution

Questions for authors
  1. More public benchmarks are recommended.

  2. Please address the issues raised above

2 comments

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards,

Area Chair

Thank you for your time and consideration. We would greatly appreciate it if you could take a look at our responses. Please let us know if you have any further questions or concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.