acceptodds
Withdrawn from ICLR 2027

4LC: 4D Latent Cache for Efficient Conditioning in Dynamic World Modeling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

For video generation in dynamic world modeling, observed static and dynamic content should be efficiently reused to provide conditions for repeated generation at different times and viewpoints. We introduce 4LC, a 4D latent cache with composite visual representations and a flexible spatiotemporal access interface. As a latent cache, 4LC directly reuses visual representations of static and dynamic content to provide generation conditions, avoiding per-request RGB rendering and re-encoding. As a composite cache, 4LC binds visual representations from multiple visual encoders (e.g., VAE, DINO, V-JEPA, VGGT) to shared geometry and motion, allowing the composite representations to describe the dynamic world with rich information. Besides, we design a flexible spatiotemporal interface, which directly reads the cache at independently specified times and viewpoints and combines readouts across observations, providing a common conditioning format for diverse downstream tasks. Building on 4LC, we develop 4LC-Video to address multiple video generation tasks through different configurations of source observations, spatiotemporal requests, and historical reuse, including camera control, temporal control, novel-view synthesis, scene revisits, and long-term generation. Our experiments show more efficient condition acquisition, effective spatiotemporal controllability, and scene consistency across tasks using a single 4LC-Video model without task-specific fine-tuning.

Primary area:
generative models
Keywords:
4D World Model, Video Generation, Latent Cache

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Xsx3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper targets efficient conditioning for dynamic world video generation. Existing pipelines often reconstruct a 3D/4D scene, then RGB-render and re-encode conditions for every new request. The authors propose 4LC, a 4D latent cache that binds multi-encoder visual features (e.g., VAE, DINO, V-JEPA, VGGT) to shared geometry and motion, then reads them directly at independently specified times and viewpoints without per-request RGB rendering/re-encoding. Composition fills uncovered regions across cache entries. Built on this, 4LC-Video adapts cache readouts to a frozen Wan diffusion backbone and handles five tasks with one checkpoint—camera control, temporal control, novel-view synthesis, scene revisits, and long-term generation—by changing request configuration rather than fine-tuning. Experiments report large gains in condition-acquisition latency/memory and competitive or better control/consistency metrics, with ablations on direct readout vs. re-encoding and composite representations.

Critical strengths
  1. The efficiency problem is well motivated and practically important: condition acquisition can be a recurring bottleneck in multi-request / long-horizon generation.
  2. The cache design is coherent: shared geometry/motion + composite latents + time/view readout + composition is a clean interface for multi-task conditioning.
  3. Efficiency results are strong and carefully measured.
  4. One model covering five task configurations without task-specific fine-tuning is a meaningful systems contribution.
  5. Ablations are useful: FLUX.2 vs Wan VAE readout quality, and complementary gains from V-JEPA / composite encoders.
Critical weaknesses
  1. “SOTA across five tasks” is too broad. Performance looks strong overall, but claims should be broken down by task/metric rather than a blanket SOTA statement. Some comparisons also involve heterogeneous pipelines and coverage assumptions.
  2. Dynamic reuse is incomplete. For long-term generation, later segments reuse only static units from earlier entries; dynamic historical content is not reused across segments (Conclusion). This weakens the “dynamic 4D cache” story relative to the title/intro.
  3. Heavy dependence on upstream reconstruction. Geometry/motion come from a feed-forward reconstructor (NeoVerse). Estimation errors can propagate into readout; robustness under reconstruction failure is not deeply stress-tested.
  4. System complexity / compositional novelty. The method stacks many pretrained components (NeoVerse, Wan, VACE, FLUX/DINO/VGGT/V-JEPA). The contribution is mainly the cache interface and conditioning path; sharper isolation against simpler latent caches (e.g., Mirage-style) would help.
Questions for authors

Please see weaknesses.

Reviewer_4SufOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the efficiency of long-video generation under repeated generation at different times and viewpoints. Specifically, it proposes a 4D latent cache with composite visual representations and a flexible spatiotemporal access interface.

Critical strengths
  1. This work is well-written, and the readers can understand the proposed points.
  2. The construction-readout-composition is easy to follow.
Critical weaknesses
  1. Although 4LC accelerates condition acquisition by 68.5×, this stage accounts for only 14.9% of the baseline runtime, resulting in merely a 1.14× end-to-end speedup (from 18.47s to 16.19s).
  2. The reported timing excludes multi-encoder feature extraction, cache construction, and cache updates, making it impossible to determine the actual time and memory breakdown. And I think it is important for this paper.
  3. The advantage of 4LC relies on the same scenario being queried multiple times. How many repeated queries are typically required for different types of tasks? What are the performance and time/memory costs of 4LC when the scene is queried only a few times?
  4. During long-video generation, generated segments are reconstructed and added back to the cache. This may turn previous generation or reconstruction errors into persistent conditions for subsequent segments. The paper should provide more evidence for robustness and failure modes.
Questions for authors

I have no further questions. Please focus on the issues mentioned in the weakness.

Reviewer_c5diOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes 4LC, a 4D latent cache for conditioning video generation in dynamic scenes. Instead of rendering RGB views and re-encoding them for every conditioning, features from multiple visual encoders are cached and read out directly at the input time and viewpoint. Built on this, a proposed adaptor conditions a pretrained video diffusion model and handles several tasks (camera/temporal control, novel-view synthesis, long-term generation).

Critical strengths
  • The motivation makes sense: naive pixel-based warping and re-encoding hurt the efficiency of video generation.
  • Caching and reprojecting in latent space rather than pixel space is intuitive.
  • The method section gives good technical details.
  • The ablation in Table 5 (condition-acquisition paths and composite representations) is useful.
Critical weaknesses
  • It is hard to grasp the full contributions and their effects. Is the rich multi-encoder representation meant to be a contribution? If so, why it helps is not clear early on (e.g., in the intro).
  • While intuitive, latent-space caching and reprojection does not feel like a novel enough insight for the community. The architecture is also a fairly standard setup: a thin adapter and concatenation followed by a Wan VACE control branch.
  • The method section is hard to follow at a high level. Making the key contributions and points clearer there would help.
  • In Table 5, the gain from combining multiple representations over direct readout does not look significant. Qualitative comparisons would help.
  • The construction cost is not discussed enough. For example, if we want to apply this to autoregressive setup, VAE decoding, the reconstructor, and the extra visual encoders would keep adding cost, which limits the applicability of the method.
Questions for authors
  • What is the construction cost, and how does the overall efficiency hold up in an autoregressive / long-horizon setting?
  • It would be great if more qualitative comparisons can be added (e.g., video results in the suppl.).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.