- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper addresses the problem of long-horizon memory in autoregressive video world models. The main idea is based on the observation that KV cache follows viewpoint correspondence, and the authors propose world KV based on this. The claimed contribution of World Compression is interesting. But several concerns exist.
Critical strengths
The writing is clear to read. And the figures are in high quality. And The idea itself is relatively interesting. The emergent-memory observation on a short-context model is the paper's contribution.
Critical weaknesses
One major concern is that the claimed contribution of World Compression is contradicted by the authors' own ablation. In table 2, the uncompressed 3-3 setting is world retrieval only, achieves best across four metrics, LPIPS 0.443, PSNR 14.630, SSIM 0.433, FID 93.536, this is against the adopted 3-1.5 setting, 0.462 / 14.101 / 0.405 / 93.561.
Another concern is why 3-1.5 is adopted as the default setting, and what evidence supports compression improving fidelity rather than only saving storage.
Questions for authors
The first question is: how much fidelity does compression cost at equal retrieval coverage?
Next, What is the per-component latency breakdown for Full KV inference at the same rollout length
5 comments
W2. Why 3→1.5 is the default, and evidence beyond storage savings
Our central goal is efficient world memory, since storing the full history is prohibitive (Fig. 2a). We selected 2× compression because it yielded the best results under a fixed attention budget: among retrieving 3, 6, and 9 chunks within the same budget (3→3, 6→3, and 9→3 in Table 2, bottom), 6→3 performs best on all metrics for both backbones. Within the same chunks (Table 2, top), retaining more than 1.5 frames gives limited gains on LingBot-World-Fast and only gradual gains on Matrix-Game-2.0, while increasing the storage cost by up to 2×. 2× compression also fits exactly 6 compressed chunks into the 9-frame (3-chunk) retrieval region.
Beyond storage savings, the budget-matched comparison above directly shows the fidelity gain, and Appendix D shows that fidelity generally improves as more chunks are retrieved. Compression makes this broader coverage possible within a fixed attention budget.
Q2. Per-component latency of Full KV
We measured Full KV under the same setting as Table 6 (LingBot-World-Fast, 4×B200, FlashAttention-4, at the same rollout length).
| Measurement | WorldKV (GPU-resident) | Full KV |
|---|---|---|
| Retrieval search | 1.963 ms | – |
| CPU↔GPU KV movement | 0 ms | – |
| Store-time compression | 31.076 ms | – |
| FA4 self-attention | 179.813 ms | 1813.049 ms |
| Total chunk latency | 1556.698 ms | 3144.907 ms |
| Throughput | 7.71 FPS | 3.83 FPS |
The gap comes from self-attention, which WorldKV reduces by about 10× (1813 → 180 ms). The overhead of retrieval and compression is only about 2% of the attention cost it saves. Together with Table 1, this shows that WorldKV achieves about 2× throughput while maintaining revisit fidelity comparable to Full KV. We add the Full KV column to Table 6.
We hope this clarifies the role of World Compression. If our responses address your concerns, we would appreciate it if you could reconsider your assessment. We would be happy to address any further questions.
Dear Reviewer uj7V,
As the discussion period is closing, we wanted to check whether our response above addressed the concern that Table 2 contradicts our contribution.
To restate the key point briefly: the intra-chunk 3→3.0 row attends to 18 retrieved frames, twice the budget of our default. Under the same 9-frame budget, our default (6→3) improves over uncompressed retrieval (3→3) on all four metrics for both backbones. We have revised the contribution statement and the Table 2 caption to avoid this confusion.
We would be happy to clarify any remaining point.
We thank the reviewer for raising these important points. We acknowledge that our presentation of Table 2 may not have clearly conveyed the budget setting of each row, and we clarify it below. For clarity, we have highlighted the revisions addressing your comments in blue in the revised paper.
W1, Q1. Role of World Compression and its cost at equal coverage
We first clarify that the intra-chunk 3→3.0 row indeed corresponds to World Retrieval (WR) without compression, and the reviewer is correct that it outperforms 3→1.5 on Matrix-Game-2.0. Each chunk consists of 3 frames, and by default the attention window holds 9 retrieved frames, i.e., 3 uncompressed chunks. Since the intra-chunk ablation keeps the number of retrieved chunks fixed at 6, 3→3.0 attends to 18 retrieved frames, 2× the attention budget of our default 3→1.5. Table 2 thus contains two WR-only rows under different budgets, as its two parts vary different quantities:
Intra 3→1.5 and Inter 6→3 denote the same configuration (our default), viewed from the two perspectives.
Our contribution concerns the bottom part of Table 2. By “2× more retrieval coverage attainable within a fixed attention budget” and “yielding consistently improving revisit fidelity,” we meant that within the same budget of 9 retrieved frames (3 chunks), compressing each chunk so that 6 chunks fit into this budget (6→3) improves revisit fidelity over uncompressed retrieval (3→3) on all four metrics for both backbones. Once the budget of each row is taken into account, the ablation is consistent with this claim. In contrast, aggressive compression (9→3) retains only anchor frames and reverses the gain (Sec. 5.4), so the benefit holds at our default 2× compression rather than at any compression ratio.
Answer to Q1. At equal coverage (6 retrieved chunks), compression halves both the per-chunk storage and the attention cost of the retrieved region. This costs almost nothing on LingBot-World-Fast: LPIPS and PSNR slightly improve, while SSIM and FID are nearly unchanged. On Matrix-Game-2.0, it incurs a modest cost in LPIPS, PSNR, and SSIM, with FID nearly unchanged. Under a fixed budget, this cost is outweighed by the 2× coverage it enables.
We acknowledge that the original wording may read as a claim that holds regardless of budget, and that the caption of Table 2 does not specify the number of retrieved chunks in the intra-chunk ablation. We have revised the paper to make this clearer: (i) we restate the contribution as “enabling 2× more retrieval coverage under a fixed attention budget and consistently improving revisit fidelity over uncompressed retrieval at the same budget,” and (ii) we specify in the caption of Table 2 that the intra-chunk rows use 6 retrieved chunks.