acceptodds
Under review as a conference paper at ICLR 2027

WorldKV: Efficient World Memory with World Retrieval and Compression

22 views

Abstract

Autoregressive video diffusion models have enabled real-time, action-conditioned world generation. However, sustaining a persistent world, where revisiting a previously seen viewpoint yields consistent content, remains an open problem. Full KV cache attention may preserve this consistency but breaks real-time constraints: memory footprint and attention cost grow linearly with rollout length. Sliding window inference restores throughput but discards long-term memory. We observe that, even without memory-specific training, attention over historical KV caches follows viewpoint correspondence, concentrating on chunks that overlap with the current view. Motivated by this observation, we propose WorldKV, a training-free framework for persistent world modeling with two components: World Retrieval selectively retrieves scene-relevant chunks via camera/action correspondence, inserting them back into the native attention window without re-encoding. World Compression prunes redundant tokens within each chunk via key-key similarity to an anchor frame, halving per-chunk storage to fit 2 more history under a fixed budget. On Matrix-Game-2.0 and LingBot-World-Fast, WorldKV matches or exceeds the fidelity achieved with Full KV memory, at roughly 2 the throughput.

Primary area:
generative models
Keywords:
Efficiency, KV cache Management, Generative World Models, Video World Models, Autoregressive Video Diffusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uj7VOpenReview ↗
Rating
2
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper addresses the problem of long-horizon memory in autoregressive video world models. The main idea is based on the observation that KV cache follows viewpoint correspondence, and the authors propose world KV based on this. The claimed contribution of World Compression is interesting. But several concerns exist.

Critical strengths

The writing is clear to read. And the figures are in high quality. And The idea itself is relatively interesting. The emergent-memory observation on a short-context model is the paper's contribution.

Critical weaknesses

One major concern is that the claimed contribution of World Compression is contradicted by the authors' own ablation. In table 2, the uncompressed 3-3 setting is world retrieval only, achieves best across four metrics, LPIPS 0.443, PSNR 14.630, SSIM 0.433, FID 93.536, this is against the adopted 3-1.5 setting, 0.462 / 14.101 / 0.405 / 93.561.

Another concern is why 3-1.5 is adopted as the default setting, and what evidence supports compression improving fidelity rather than only saving storage.

Questions for authors

The first question is: how much fidelity does compression cost at equal retrieval coverage?

Next, What is the per-component latency breakdown for Full KV inference at the same rollout length

5 comments

We thank the reviewer for raising these important points. We acknowledge that our presentation of Table 2 may not have clearly conveyed the budget setting of each row, and we clarify it below. For clarity, we have highlighted the revisions addressing your comments in blue in the revised paper.

W1, Q1. Role of World Compression and its cost at equal coverage

We first clarify that the intra-chunk 3→3.0 row indeed corresponds to World Retrieval (WR) without compression, and the reviewer is correct that it outperforms 3→1.5 on Matrix-Game-2.0. Each chunk consists of 3 frames, and by default the attention window holds 9 retrieved frames, i.e., 3 uncompressed chunks. Since the intra-chunk ablation keeps the number of retrieved chunks fixed at 6, 3→3.0 attends to 18 retrieved frames, 2× the attention budget of our default 3→1.5. Table 2 thus contains two WR-only rows under different budgets, as its two parts vary different quantities:

  • Table 2, top (intra-chunk): the number of retrieved chunks is fixed at 6, and the frames kept per chunk vary. The total number of retrieved frames, and thus the budget, grows with retention. Comparing 3→3.0 and 3→1.5 therefore measures the cost of compression at equal coverage (Q1), not at equal budget.
  • Table 2, bottom (inter-chunk): the total is fixed at 9 retrieved frames (3 chunks), and the number of chunks compressed into it varies. Here 3→3 is WR only under the same budget as ours. This is the budget-matched comparison our contribution refers to.
SettingRetrieved chunksFrames per chunkRetrieved frames (budget)LingBot-World-Fast LPIPS↓ / PSNR↑ / SSIM↑ / FID↓Matrix-Game-2.0 LPIPS↓ / PSNR↑ / SSIM↑ / FID↓
Intra 3→3.0 frames6318 (2×)0.459 / 15.594 / 0.467 / 75.0050.443 / 14.630 / 0.433 / 93.536
Intra 3→1.5 frames (ours)61.59 (1×)0.455 / 15.660 / 0.463 / 75.6440.462 / 14.101 / 0.405 / 93.561
Inter 3→3 chunks339 (1×)0.468 / 15.436 / 0.454 / 91.3890.496 / 13.416 / 0.369 / 105.988
Inter 6→3 chunks (ours)61.59 (1×)0.455 / 15.660 / 0.463 / 75.6440.462 / 14.101 / 0.405 / 93.561
Inter 9→3 chunks919 (1×)0.482 / 14.982 / 0.430 / 101.2260.499 / 13.403 / 0.360 / 108.460

Intra 3→1.5 and Inter 6→3 denote the same configuration (our default), viewed from the two perspectives.

Our contribution concerns the bottom part of Table 2. By “2× more retrieval coverage attainable within a fixed attention budget” and “yielding consistently improving revisit fidelity,” we meant that within the same budget of 9 retrieved frames (3 chunks), compressing each chunk so that 6 chunks fit into this budget (6→3) improves revisit fidelity over uncompressed retrieval (3→3) on all four metrics for both backbones. Once the budget of each row is taken into account, the ablation is consistent with this claim. In contrast, aggressive compression (9→3) retains only anchor frames and reverses the gain (Sec. 5.4), so the benefit holds at our default 2× compression rather than at any compression ratio.

Answer to Q1. At equal coverage (6 retrieved chunks), compression halves both the per-chunk storage and the attention cost of the retrieved region. This costs almost nothing on LingBot-World-Fast: LPIPS and PSNR slightly improve, while SSIM and FID are nearly unchanged. On Matrix-Game-2.0, it incurs a modest cost in LPIPS, PSNR, and SSIM, with FID nearly unchanged. Under a fixed budget, this cost is outweighed by the 2× coverage it enables.

We acknowledge that the original wording may read as a claim that holds regardless of budget, and that the caption of Table 2 does not specify the number of retrieved chunks in the intra-chunk ablation. We have revised the paper to make this clearer: (i) we restate the contribution as “enabling 2× more retrieval coverage under a fixed attention budget and consistently improving revisit fidelity over uncompressed retrieval at the same budget,” and (ii) we specify in the caption of Table 2 that the intra-chunk rows use 6 retrieved chunks.

W2. Why 3→1.5 is the default, and evidence beyond storage savings

Our central goal is efficient world memory, since storing the full history is prohibitive (Fig. 2a). We selected 2× compression because it yielded the best results under a fixed attention budget: among retrieving 3, 6, and 9 chunks within the same budget (3→3, 6→3, and 9→3 in Table 2, bottom), 6→3 performs best on all metrics for both backbones. Within the same chunks (Table 2, top), retaining more than 1.5 frames gives limited gains on LingBot-World-Fast and only gradual gains on Matrix-Game-2.0, while increasing the storage cost by up to 2×. 2× compression also fits exactly 6 compressed chunks into the 9-frame (3-chunk) retrieval region.

Beyond storage savings, the budget-matched comparison above directly shows the fidelity gain, and Appendix D shows that fidelity generally improves as more chunks are retrieved. Compression makes this broader coverage possible within a fixed attention budget.

Q2. Per-component latency of Full KV

We measured Full KV under the same setting as Table 6 (LingBot-World-Fast, 4×B200, FlashAttention-4, at the same rollout length).

MeasurementWorldKV (GPU-resident)Full KV
Retrieval search1.963 ms–
CPU↔GPU KV movement0 ms–
Store-time compression31.076 ms–
FA4 self-attention179.813 ms1813.049 ms
Total chunk latency1556.698 ms3144.907 ms
Throughput7.71 FPS3.83 FPS

The gap comes from self-attention, which WorldKV reduces by about 10× (1813 → 180 ms). The overhead of retrieval and compression is only about 2% of the attention cost it saves. Together with Table 1, this shows that WorldKV achieves about 2× throughput while maintaining revisit fidelity comparable to Full KV. We add the Full KV column to Table 6.

We hope this clarifies the role of World Compression. If our responses address your concerns, we would appreciate it if you could reconsider your assessment. We would be happy to address any further questions.

Dear Reviewer uj7V,

As the discussion period is closing, we wanted to check whether our response above addressed the concern that Table 2 contradicts our contribution.

To restate the key point briefly: the intra-chunk 3→3.0 row attends to 18 retrieved frames, twice the budget of our default. Under the same 9-frame budget, our default (6→3) improves over uncompressed retrieval (3→3) on all four metrics for both backbones. We have revised the contribution statement and the Table 2 caption to avoid this confusion.

We would be happy to clarify any remaining point.

Reviewer_GxkiOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets persistent world memory in autoregressive video world models: full-KV attention retains revisit consistency but violates real-time constraints, while sliding windows are fast but forget. Noting that attention already aligns with viewpoint correspondence even without long-term training, the authors introduce WorldKV, a training-free method with World Retrieval (storing and re-inserting view-relevant KV chunks by camera/action state) and World Compression (pruning redundant tokens via key similarity, roughly 2× storage reduction). On Matrix-Game-2.0 and LingBot-World-Fast, WorldKV matches or exceeds full-KV revisit fidelity at ~2× throughput; on short-horizon Matrix-Game-2.0 it surpasses full KV on all metrics.

Critical strengths
  1. The paper is easy to read and follow.

  2. Interesting empirical insight. View-correspondent attention over historical KV caches emerges even in models never trained for long-term memory: a finding that is interesting on its own and directly motivates the method.

  3. Practical and effective. Training-free, no architecture changes or re-encoding, yet matches or exceeds Full-KV revisit fidelity at ~2× throughput and beats Full KV on the short-horizon-trained base.

  4. Thorough evaluation. Two backbones at different scales (plus a third in the appendix), careful ablations including a same-backbone FOV-retrieval baseline and a usery study.

Critical weaknesses
  1. Self-constructed benchmark only. The proposed benchmark is a valuable contribution in its own right (and I encourage the authors to release it publicly for future research), but the central claim of revisit consistency is currently supported only by this in-house protocol. The authors should additionally evaluate on existing memory benchmarks such as MBench to enable comparison with prior and future work.

  2. No compression baselines. World Compression is ablated only against its own retention ratios. It is never benchmarked against established KV-compression schemes (e.g., SnapKV, TempCache in FAST-AR, Forcing-KV, among others), so it remains unclear whether key-key similarity is actually a better criterion than existing alternatives.

  3. Pose/action dependence. The default retrieval assumes accurate camera/action indexing at inference time. The query-based alternative is notably weaker (Appendix C), and the authors themselves acknowledge failure modes in dynamic scenes, where action similarity does not imply visual relevance.

  4. Unbounded memory growth. Since all evicted chunks must be stored for potential retrieval, memory still grows linearly with rollout length, even after 2× compression. Could the authors discuss strategies for keeping long-term storage bounded over time (e.g., hierarchical compression of older chunks, or eviction of chunks unlikely to be revisited)?

Questions for authors

While the paper offers a practical training-free method built on an interesting empirical insight, its central claim rests solely on a self-constructed benchmark, the compression component lacks comparison to existing KV-compression baselines, and memory growth remains unbounded.

5 comments

We sincerely thank the reviewer for the thorough evaluation and for recognizing the insight and effectiveness of our method. We address each point below. For clarity, we have highlighted the revisions addressing your comments in orange in the revised paper.

W1. Evaluation on an existing benchmark. At the time of submission, no memory benchmark for action-conditioned world models had been widely adopted, so we designed our own benchmark, which will be released publicly. Following the suggestion, we additionally evaluated on MBench-A, the action-conditioned track of MBench [1]:

MethodHuman Identity↑Human Appearance↑Epipolar↑Reprojection↑Style↑Lighting↑Action↑
Matrix-Game-2.0
– Sliding Window1.231.620.831.8574.0687.1644.54
– Full KV8.5910.742.941.9683.4890.5934.49
– WorldKV12.0515.333.142.3886.7591.3158.24
LingBot-World-Fast
– Sliding Window13.3236.630.302.0685.8189.2359.39
– Full KV17.6142.4412.5017.0391.4193.2460.26
– WorldKV19.1741.9318.7625.1692.4494.8161.50

Consistent with our benchmark, WorldKV substantially improves over sliding-window inference and matches or exceeds Full KV on both backbones, achieving the best score on all metrics except Human Appearance on LingBot-World-Fast, where it is comparable to Full KV. This follows the trend in Sec. 5.2: restricting attention to view-relevant chunks becomes more beneficial when a larger fraction of the history is irrelevant to the current viewpoint (Appendix A). We add these results to Sec. 5.5.

W2. Comparison with KV compression baselines. We did not include these comparisons in the submission because these methods were not specifically designed for world memory in action-conditioned world models. SnapKV [5] compresses an LLM prompt by keeping the tokens that receive the most attention from the last queries of the prompt, and we did not consider it directly applicable to autoregressive video diffusion. For similar methods in autoregressive video diffusion, Deep Forcing [6] and MemRoPE [2] manage the KV cache to sustain quality in long video generation. Participative Compression in Deep Forcing similarly keeps tokens attended by recent queries, and MemRoPE compresses past keys into memory tokens via exponential moving averages. TempCache [3] and Forcing-KV [4] compress the KV cache to improve inference speed and GPU memory efficiency. TempCache merges keys corresponding to the same content across frames into a single representative, and Forcing-KV splits attention heads into static and dynamic heads, keeping only the sink frames, the most recent frame, and the current chunk for static heads and pruning by segment-wise similarity for dynamic heads.

Notably, none of these methods evaluates revisit consistency. WorldKV instead targets efficient world memory: our compression must preserve content needed for future revisits rather than tokens relevant to the current queries. As the methods do not explicitly target our main task, we did not initially treat these methods as direct baselines.

We agree, however, that these methods can still be compared with ours, and we evaluate them under the same setting, compressing each stored chunk to the same 1.5-frame budget under the 6→3 configuration. We substitute two of the suggested methods as follows:

  • SnapKV → Participative Compression: instead of directly applying an LLM method, we use Participative Compression, a similar query-based pruning method designed for autoregressive video diffusion.
  • TempCache → MemRoPE: as TempCache has no official code release, we adopt MemRoPE’s key aggregation instead, applying it within each chunk by merging the non-anchor tokens most similar to the anchor into the anchor and keeping the rest.
MethodLPIPS↓PSNR↑SSIM↑FID↓
LingBot-World-Fast
– Participative Compression0.45615.5830.45378.789
– MemRoPE0.46615.4590.45480.597
– Forcing-KV0.48614.7570.45394.117
– World Compression (Ours)0.45515.6600.46375.644
Matrix-Game-2.0
– Participative Compression0.47713.5190.36297.273
– MemRoPE0.46513.6980.39796.352
– Forcing-KV0.49513.4900.354104.684
– World Compression (Ours)0.46214.1010.40593.561

World Compression achieves the best results on all metrics for both backbones. We attribute this to the difference in what each criterion preserves. Participative Compression keeps tokens attended by the queries at storage time, which may not be the tokens needed at a later revisit. MemRoPE’s aggregation merges the highly similar non-anchor tokens into the anchor, which might blur view-specific details. Forcing-KV’s head-wise split drops distant history in static heads, which may not be suitable for memory consistency. In contrast, our query-independent key-key criterion retains content not captured by the anchor, such as newly revealed regions, which is what a later revisit needs. These results indicate that compression for world memory calls for a different criterion from compression for long-video quality or efficiency. We add these results to Appendix N.

W3. Pose/action dependence. We would like to clarify that our scope is action-conditioned world models (Sec. 3), where the camera/action input is given by the user at every step and is therefore always available at inference. Retrieval indexes chunks by this input rather than by an estimated pose. This choice is grounded in our attention analysis (Fig. 3), where the model itself attends to chunks whose input viewpoints match the current one. That said, we would like to emphasize that World Retrieval does not hinge on pose: although weaker than pose-based retrieval, query-based retrieval still substantially outperforms sliding window inference (Appendix C, Table 4). We acknowledge that in highly dynamic scenes, action similarity may not imply visual relevance, and developing hybrid retrieval that combines query- and camera/action-based signals is a promising direction.

W4. Bounding long-term storage. We agree that total storage still grows linearly with rollout length. WorldKV aims to reduce the stored KV and bound the active attention window rather than to bound total storage. One practical strategy to bound total storage is spatial deduplication. Since retrieval is indexed by camera/action state, chunks with nearly identical states can be merged or replaced, so that storage scales with the explored area rather than with time. For instance, a rollout with a static camera would need only a single chunk. We did not adopt this in the paper because its savings depend strongly on the trajectory, large for revisiting trajectories and small for wide exploration, which makes a fair efficiency comparison difficult. We add this discussion to the Limitations & Future Work section.

We hope these additional results address your concerns, and we would be happy to answer any further questions.

[1]: Zhang et al., “MBench: A Comprehensive Benchmark on Memory Capability for Video World Models”, arxiv 2026

[2]: Kim et al., “MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens”, ECCV 2026

[3]: Samuel et al., “FAST-AR: Fast Autoregressive Video Diffusion and World Models with Temporal Cache Compression and Sparse Attention”, ICML 2026

[4]: Ji et al., “Forcing-KV: Hybrid KV Cache Compression for Efficient Autoregressive Video Diffusion Models”, arxiv 2026

[5]: Li et al., “SnapKV: LLM Knows What You are Looking for Before Generation”, NeurIPS 2024

[6]: Yi et al., “Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression”, ICML 2026

Reviewer_s4hqOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WorldKV explores how training-free KV-cache methods can maintain scene consistency with a limited GPU memory budget in autoregressive world models such as LingBot v1 and Matrix-Game 2.0. These models are built on video generators and do not have dedicated memory modules.

The authors observe that, under FullKV, both models pay more attention to earlier chunks with similar camera poses when generating new chunks as the camera rotates. Building on this observation, they introduce two complementary components—World Compression and World Retrieval—to store and retrieve historical KV caches within a fixed memory budget.

World Compression keeps the full KV cache of an anchor frame in each chunk and compresses the KV entries of the following two latent frames based on key–key similarity. The resulting cache is stored on the GPU. During subsequent generation, World Retrieval selects the top-k historical chunks based on camera pose and brings their cached KV entries back into the attention context, alongside the sink tokens, recent chunks, and the chunk currently being denoised.

Together, these components provide a training-free form of implicit state memory through the KV cache. By combining within-chunk compression with camera-pose-based retrieval, WorldKV achieves scene consistency comparable to FullKV while keeping GPU memory usage within a limited budget.

The method is evaluated on LingBot v1, which uses FullKV for inference, and Matrix-Game 2.0, which uses a sliding window. On both models, it improves consistency when revisiting previously seen views compared with sliding-window inference. Its revisit performance is broadly comparable to FullKV on LingBot v1 and better than FullKV on Matrix-Game 2.0.

Critical strengths
  1. The paper offers a new perspective on maintaining long-term scene consistency in world models. Using LingBot v1 and Matrix-Game 2.0 as base models, it shows that training-free KV compression and recaching can achieve consistency comparable to FullKV without additional memory-specific training. The empirical study is a particular strength: the finding that revisiting a view during inference increases attention to earlier chunks with matching camera poses is both interesting and useful. Compression and recaching reduce the memory budget required by FullKV while keeping generation responsive.

  2. The experimental design and evaluation are fairly comprehensive. Results on Matrix-Game 2.0 and LingBot v1 suggest that caching and retrieving chunks from similar viewpoints can maintain a consistent spatial memory comparable to FullKV, while retaining some of the responsiveness of sliding-window inference. Comparisons with WorldPlay and Yume 1.5 on the same tasks provide useful reference points.

Critical weaknesses
  1. The description of key–key similarity in the compression step lacks detail. It is unclear whether similarity is computed before or after RoPE. The paper also does not explain why the anchor frame’s token KV entries are retained in full at every DiT layer rather than compressed. How positional encodings are handled when retrieved chunks are reinserted into the context is also insufficiently explained.

  2. The benchmark describes the trajectories and evaluation principles, but does not adequately explain how actual viewpoint alignment is verified or how control deviations and loop-closure failures are handled. Both models may generate camera motion that deviates from the input controls. When the actual camera pose differs from the planned trajectory, how is this mismatch accounted for in chunk retrieval and benchmark scoring? The paper also lacks a stability analysis broken down by revisit interval or trajectory length.

Questions for authors
  1. Could the authors explain the intuition behind token compression in more detail and provide a more precise description of how it is implemented?

  2. Could the authors clarify how RoPE is handled in chunk compression and retrieval? Is key similarity computed using pre-RoPE or post-RoPE keys? When a chunk is retrieved, are its tokens remapped to positions before the recent window?

  3. How could the evaluation be designed to better assess long-term stability when revisiting previously seen views?

7 comments

We thank the reviewer for raising these questions. We address each point below. For clarity, we have highlighted the revisions addressing your comments in red in the revised paper.

W1, Q1, Q2. Intuition, implementation, and RoPE handling of World Compression

Intuition. Consecutive frames within a chunk share most of their content, so their KV caches are largely redundant. Since keys in video diffusion transformers encode spatiotemporal correspondence [1], a non-anchor key similar to the anchor keys mostly encodes content the anchor already holds, while a dissimilar key corresponds to newly revealed regions or moving objects (Appendix B, Fig. 7). We therefore drop the non-anchor tokens most similar to the anchor and keep the most dissimilar ones.

Why keep the anchor in full? At every layer, the anchor is the reference that defines redundancy: non-anchor tokens are dropped because the anchor already encodes their content, so compressing the anchor would remove that content from the chunk and leave no complete view of the scene. To verify this, we compare two allocations under the same 1.25-frame budget. The full-anchor variant is World Compression with a lower retention ratio (3→1.25 in Table 2), which keeps the full anchor and 0.25 frames of non-anchor tokens. The compressed-anchor variant first applies our default 3→1.5 compression, which keeps 0.5 frames of non-anchor tokens, and then drops 25% of anchor tokens with the highest similarity to the mean key of the anchor frame.

Anchor / Non-anchorLingBot-World-FastMatrix-Game-2.0
LPIPS↓PSNR↑SSIM↑FID↓LPIPS↓PSNR↑SSIM↑FID↓
0.75 / 0.5 (anchor compressed)0.47815.4370.45593.5340.48113.9830.39396.651
1.0 / 0.25 (anchor in full, 3→1.25 in Table 2)0.46315.5990.46790.1240.46314.1580.39795.376

Keeping the anchor in full yields better results on all metrics for both backbones, even though the compressed-anchor variant retains twice as many non-anchor tokens. We therefore keep the anchor in full and prune only the remaining frames.

Implementation. The KV of each chunk is obtained by running the model once more on its clean latent x0x_0 at t=0t=0 after denoising. When the chunk is evicted from the attention window, this KV is compressed once at storage time and stored in the KV bank. Compression operates independently at each layer:

  1. All TT tokens of the anchor (first) frame are kept.
  2. Each of the 2T2T non-anchor keys is scored by its average cosine similarity to all TT anchor keys (Eq. 5), computed on the concatenation of all heads' keys.
  3. The 25% least similar non-anchor tokens (0.5T0.5T) are retained. Keys and values share the same indices.
  4. The 1.5T1.5T kept tokens are stored.

Since keys differ across layers, the kept tokens differ across layers, while all heads within a layer share the same kept tokens.

Pre- or post-RoPE? Following Deep Forcing [2], we operate on post-RoPE keys and, when a chunk is re-inserted, adjust only the temporal RoPE of its tokens to their new positions. Key similarity is therefore computed on post-RoPE keys at their original positions. Since this adjustment rotates all tokens of a chunk identically, it does not change the similarity between them, so compressing at storage time selects the same tokens as compressing after the adjustment.

Re-insertion. Our attention window consists of sink (3 frames), retrieved (9 frames), recent (3 frames), and denoising (3 frames) regions. Recent and denoising tokens keep their positions, and the temporal RoPE of the retrieved chunks is adjusted to the temporal slots immediately preceding the recent window. This adjustment only rotates the temporal RoPE channels of each retrieved chunk by its offset Δ\Delta. Spatial channels and values are unchanged, and no per-token position metadata is needed. Each kept token preserves its relative offset within the chunk and its spatial position.

We add this more detailed description to Appendix O.

[1]: Yuan et al., “Denoise to track: Harnessing video diffusion priors for robust correspondence.”, arxiv 2025

[2]: Yi et al., “Deep Forcing: Training-Free Long Video Generation with Deep Sink and Participative Compression”, ICML 2026

W2, Q3. Viewpoint alignment, control deviation, and long-term stability

We agree that implicit video world models may not precisely follow the input controls. Camera-control accuracy varies across models, analogous to text alignment in text-to-video generation, and remains a broader open problem in the field. It is a property of the base models rather than of WorldKV, which is training-free and leaves the control conditioning unchanged. We therefore designed our evaluation to measure memory as independently of control accuracy as possible, as detailed below.

Control deviation over time. We initially measured the camera-pose error on LingBot-World-Fast with ViPE[3]:

Method1st (8.7 s)2nd (15.2 s)3rd (23.7 s)
Full KV8.9°16.0°27.3°
WorldKV8.1°16.7°27.0°

Drift exists in the base model and accumulates over time, but comparably for both, indicating that WorldKV does not degrade control adherence relative to Full KV, the native inference mode of LingBot-World-Fast.

Scoring. To account for such deviation, we do not rely on exact input pose alignment. Concretely, the revisit and first-visit frames compared in Sec. 5.1 are matched as follows. For each revisit, we take the nnbn_{\text{nb}} frames around the first-visit view and the nnbn_{\text{nb}} frames around the revisit view. PSNR, SSIM, and LPIPS are computed over all pairs between the two sets and the best-scoring pair is used, while FID is computed between the two sets. The same protocol is applied to all methods. We add this detail to Appendix P.

Verifying viewpoint alignment. We estimated the viewpoint difference of the frame pairs selected for scoring with ViPE on LingBot-World-Fast:

MethodViewpoint difference
Full KV2.5°
WorldKV2.6°

Although the realized camera drifts up to ~27° from the planned trajectory, the scored pairs are aligned within 2.6°. For sliding-window inference, estimates vary widely across estimators, since the revisited content often changes drastically.

Stability over trajectory length. To examine stability as the rollout grows longer, we use the Retracing trajectories, which visit the same viewpoints several times, and score the 2nd and 3rd visits against the first visit, with the same best-match criterion as above. We report how much each score changes from the 2nd (15.2 s) to the 3rd visit (23.7 s). Positive LPIPS changes and negative PSNR/SSIM changes indicate degradation, so values closer to zero mean more stable memory:

ModelMethodLPIPS change (3rd - 2nd) ↓PSNR change (3rd - 2nd) ↑SSIM change (3rd - 2nd) ↑
LingBot-World-FastSliding Window+0.055−1.11−0.038
Full KV+0.026−0.71−0.028
WorldKV+0.024−0.64−0.026
Matrix-Game-2.0Sliding Window+0.029−0.42−0.015
Full KV+0.046−0.49−0.021
WorldKV+0.015−0.16−0.010

WorldKV degrades comparably to or less than Full KV across repeated visits: it is on par with Full KV on LingBot-World-Fast and degrades the least on Matrix-Game-2.0. Sliding-window inference degrades less than Full KV on Matrix-Game-2.0 only because it is already far worse at both visits.

Chunk retrieval. The realized pose can deviate from the input, and our World Retrieval does not explicitly model this: it retrieves chunks by the input camera trajectory. A practical World Retrieval variant is to index the memory bank with poses estimated from the already-generated frames, using an off-the-shelf estimator such as ViPE[3] or VGGT-Ω[4]. The query would then be formed from the latest estimated pose plus the commanded relative motion. We did not adopt this for two reasons. Running a pose estimator at every chunk would add latency, whereas WorldKV is designed to keep inference lightweight. Moreover, input-pose retrieval does not appear to hurt WorldKV in our evaluation, as its revisit consistency is comparable to Full KV in Table 1 and remains as stable as or more stable than Full KV across repeated revisits. However, we expect this external pose module to improve retrieval accuracy further, and leave it for future work.

Q3. Evaluating long-term stability at revisits. We believe long-term stability could be better assessed by geometric consistency at revisits, in addition to appearance. Geometric consistency asks whether the first visit and its revisit are explained by one consistent 3D scene, which directly tests whether the world has persisted. MBench [5] provides such metrics: Epipolar Geometry and Reprojection Consistency pair each frame with the revisit frame closest in estimated camera pose and check this consistency. On MBench-A, the action-conditioned track, we obtain the following results:

MethodLingBot Epipolar↑LingBot Reprojection↑Matrix-Game Epipolar↑Matrix-Game Reprojection↑
Sliding Window0.302.060.831.85
Full KV12.5017.032.941.96
WorldKV18.7625.163.142.38

WorldKV substantially improves over sliding-window inference and achieves better results than Full KV on both metrics for both backbones. Since these metrics rely on estimated pose and depth, we recommend using them alongside appearance metrics and reporting them across repeated revisits, as in our response to W2. We add these results to Sec. 5.5 and Appendix P.

We hope these additional results address your concerns, and we would be happy to answer any further questions.

[3]: Huang et al., “ViPE: Video Pose Engine for 3D Geometric Perception”, arxiv 2025

[4]: Wang et al., “VGGT-Ω”, CVPR 2026

[5]: Zhang et al., “MBench: A Comprehensive Benchmark on Memory Capability for Video World Models”, arxiv 2026

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions

We thank all reviewers for their thoughtful and constructive feedback. We are encouraged that the reviewers found the observation of view-correspondent attention over historical KV caches interesting and useful (s4hq, Gxki, uj7V), the training-free method practical and effective, reaching revisit consistency comparable to Full KV without memory-specific training (s4hq, Gxki), the evaluation comprehensive (s4hq, Gxki), and the paper clear and easy to follow (Gxki, uj7V).

The main concerns were:

  1. the implementation details of World Compression, including RoPE handling and the role of the anchor frame (s4hq),
  2. the benefit of World Compression under a fixed attention budget and its comparison with existing KV compression methods (uj7V, Gxki),
  3. evaluation beyond our benchmark, and robustness to control deviation and long-term stability (Gxki, s4hq), and
  4. the efficiency of WorldKV relative to Full KV, including latency and long-term storage (uj7V, Gxki).

We summarize the main additions in the revised paper below. Changes are marked in red, orange, and blue for reviewers s4hq, Gxki, and uj7V, respectively, and we address each question in the individual responses. Section, table, and appendix numbers refer to the revised paper.

AdditionLocationReviewers
Compression procedure, post-RoPE key similarity, and RoPE adjustment at re-insertionAppendix Os4hq
Ablation on keeping the anchor frame in full versus compressing it under the same budgetAppendix O (Table 11)s4hq
Contribution restated as a budget-matched comparison with uncompressed retrieval, and the number of retrieved chunks specified in the Table 2 captionSec. 1, Table 2uj7V
Comparison with KV compression baselines (Participative Compression, MemRoPE, Forcing-KV) under the same 1.5-frame budgetAppendix N (Table 10)Gxki
Evaluation on MBench-A on both backbones, compared with sliding-window inference and Full KVSec. 5.5 (Table 3)Gxki, s4hq
Camera-pose drift over time, viewpoint alignment of scored frame pairs, the frame-matching protocol for scoring, and chunk retrieval under control deviationAppendix P (Tables 12, 13)s4hq
Change in revisit fidelity from the 2nd to the 3rd visitAppendix P (Table 14)s4hq
Per-component latency and memory of Full KV under the same setting as WorldKVAppendix F (Table 6)uj7V
Discussion on bounding long-term storage via spatial deduplicationAppendix QGxki

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.