acceptodds
Withdrawn from ICLR 2027

ActCache: Action-Guided Cross-Chunk KV Cache Reuse for Interactive World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive world models generate continuous visual environments chunk by chunk while maintaining historical KV caches for temporal coherence. Although few-step distillation reduces denoising iterations, dense computation within each new chunk remains a major barrier to real-time interaction, while conventional step-wise caching offers limited reuse opportunities. We observe that adjacent chunks preserve substantial visual content and that explicit controls constrain its motion, making historical KV representations a structured source of reusable computation. Based on this insight, we introduce **ActCache**, an action-guided framework for fine-grained cross-chunk KV reuse. ActCache converts motion-related controls into coarse displacement priors and restricts correspondence search to layer-adaptive neighborhoods. Since these priors cannot fully capture parallax, object motion, occlusion, or newly revealed content, ActCache uses the initial clean estimate from the first denoising step as a self-generated probe to verify token-level correspondences. Verified tokens reuse historical self-attention representations, while unmatched regions are recomputed. A shift-equivariant calibration mechanism applies relative 3D-RoPE rotations to align reused keys with their current coordinates. Without additional training, ActCache delivers up to speedup across three interactive world models while preserving world-memory consistency. Combined with complementary acceleration strategies, it further achieves up to speedup.

Primary area:
generative models
Keywords:
Interactive World Models, Autoregressive Video Generation, KV Cache Reuse

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2HJVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ActCache is a training-free method for accelerating interactive world models by reusing KV representations across video chunks. It uses action-guided local search to identify candidate correspondences, verifies them using the initial clean prediction from the first denoising step, and applies relative RoPE calibration to align reused keys with their new positions. Experiments on three world models show consistent speedups with limited quality degradation, while maintaining world-memory consistency and achieving further gains when combined with existing acceleration methods.

Critical strengths

1. ActCache addresses a useful problem by exploiting cross-chunk redundancy to accelerate interactive world models, offering a practical direction for improving inference efficiency.

2. The experimental evaluation is comprehensive, covering three world models, comparisons with existing acceleration methods, compatibility experiments, and ablation studies to validate the proposed design.

Critical weaknesses

1. The technical novelty appears limited, as the proposed method largely combines existing ideas in caching and feature matching. Its distinction from prior cross-chunk caching methods is not sufficiently clear.

2. Similarity between the initial clean prediction and historical clean latents does not necessarily imply that their self-attention outputs are sufficiently similar for reuse. The paper provides limited evidence that latent similarity is a reliable criterion for identifying reusable tokens.

Questions for authors

1. Could the authors clarify the key technical novelty of ActCache over existing caching and feature-matching approaches?

2. Could the authors provide an experiment to examine how well latent similarity predicts the error introduced by reusing historical self-attention outputs?

Reviewer_mM4uOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ActCache, a training-free method to accelerate chunk-wise autoregressive interactive world models. The key observation is that adjacent chunks share a lot of visual content, and the motion between them is roughly decided by the user controls. Based on this, the authors 1) convert the controls (e.g., WASD keys or camera poses) into a coarse displacement prior, which defines a local search window in the previous chunk; 2) use the clean estimate from the first denoising step as a probe to verify the token correspondences; 3) reuse the cached self-attention features for the verified tokens and recompute the others, with a relative 3D-RoPE rotation to align the reused keys. Overall, the idea is simple and reasonable, but some details of the method and the evaluation are not very clear.

Critical strengths

Training-free and tested on several models. The method needs no extra training and works on three backbones with different sizes (1.3B to 14B) and different control interfaces.

Critical weaknesses
  1. The method description is not well justified. Reusing clean features in noisy steps. The cached K/V and attention outputs come from the clean memory-append pass of the previous chunk, while the recomputed tokens come from the noisy intermediate steps. It is not obvious to me why these features can be mixed directly, and there is no analysis across the denoising steps.

    Unclear displacement prior. The main text describes a position-dependent displacement field, but Appendix A.1.1 says each key is mapped to one image-plane direction without modeling depth. How does this work for forward motion (W), which is a radial expansion rather than a shift? And how are the camera poses of LingBot-World and InSpatio-World converted?

  2. Lack of solid experiments. Longer video results are required. All results are on 5-second videos, which cannot show whether the reuse errors accumulate over long rollouts. For world models this is quite important.

    Missing cost. The extra GPU memory for storing the per-layer attention outputs is not reported either. Besides, the WorldKV results are reproduced from its original paper, so it is unclear whether they are tested under the same setting.

Questions for authors

na

Reviewer_V6viOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ActCache, a training-free method for accelerating chunk-wise autoregressive interactive world models. It reuses KV at the token level across adjacent chunks. Control signals are first converted into a displacement prior, and a search window is set per layer. The x0 obtained from the first denoising step is then matched against the clean latent of the previous chunk by cosine similarity to verify correspondences. Verified positions directly copy the previous chunk's K/V and self-attention output, keys are calibrated with a relative RoPE rotation, and the memory-appending pass is still fully computed. The paper reports 1.41×/1.43×/1.39× speedups on LingBot-World-Fast, MatrixGame2, and Inspatio-World, with WBench Avg dropping by 0.3–0.9. Combined with WorldKV, the speedup reaches up to 1.75×.

Critical strengths
  1. The problem is well identified. Few-step models leave little room for step-wise caching (Fig.2, p2–3), and treating the previous chunk's KV as a source of reusable computation is a natural and practical direction. Table 3 shows that it can be stacked with TeaCache and WorldKV.
  2. Several design choices are sensible: using the first-step prediction as a probe with a verification threshold; fully computing the append pass so that reuse errors are not written into the cache; and RoPE calibration that exploits the composability of rotations, so the attention kernel does not need to be modified. The component-wise ablation in Table 4 shows that each component contributes (Avg 31.5 for Direct reuse vs. 68.5 for ActCache).
  3. The evaluation covers three backbones on a multi-turn interaction benchmark, and includes a sensitivity analysis of the threshold and window (Fig.5).
Critical weaknesses
  1. The speedups do not match the amount of computation described in the text. According to p4 L193 and Fig.3, each chunk has S denoising steps plus 1 append pass. The first step must be fully computed to obtain the probe (p5 L269), the append pass is also fully computed (p7 L366), and the FFN, cross-attn, and camera injection all remain dense (p6 L301). Even if all computation in the remaining S−1 steps were skippable self-attention, at the 43.5% reuse rate (Table 4) the speedup upper bound is (S+1)/(2+(S−1)(1−0.435)): 1.28× for S=3 and 1.35× for S=4. The official Matrix-Game 2.0 configuration uses 3 steps, yet the paper reports 1.43×. Table 4 is also internally inconsistent: extrapolating from Local reuse (71.4%, 21.2 FPS) and ActCache (43.5%, 19.3 FPS) gives about 16.9 FPS at zero reuse, whereas Full computation is 13.5 FPS. This suggests a speedup of about 1.25× that is unrelated to reuse. In addition, p7 L377 states "720p", but the official MatrixGame2 resolution is 352×640, and warm-up, variance, and batch size are not reported. Please provide a per-pass, per-module latency breakdown, and confirm that native and ActCache use the same implementation.
  2. The evidence in Table 2 is problematic. In the original WorldKV paper, the FID of WorldKV on LingBot is 75.644, but Table 2 lists 85.644. With the original value, ActCache (85.281) does not have the best FID on LingBot, so the distributional consistency claim in p8 L401–402 does not hold. The Native rows are digit-for-digit identical to those in the WorldKV paper (which were measured on 4×H200), but the caption only states that the WorldKV rows are cited. Furthermore, ActCache only references the (t−1)-th chunk (Eq.3), yet on MatrixGame2 it exceeds the sliding-window native by 3.19 dB PSNR and even surpasses the Full KV result of 13.748 reported in the WorldKV paper. This gain cannot be explained by the mechanism. The authors need to state which KV configuration ActCache uses here.
  3. Baseline operating points and configurations are opaque. Each baseline in Table 1 has a single operating point (1.12–1.34×), and the paper does not give their thresholds or budgets. In the original WorldKV paper, WorldKV is slower than the sliding-window native on MatrixGame2 (16.25 vs 18.87 FPS), yet Table 1 shows a 1.34× speedup, so the KV configuration of native (labeled as sliding window in Table 2) needs to be clarified. Without speed-matched comparisons or Pareto curves, the claim of being both faster and better depends on operating points chosen by the authors. On quality, Avg hides the Interaction dimension, which matters most for interactive models: it drops from 79.4 to 77.9 on LingBot and from 72.8 to 71.5 on Inspatio, and no seed variance is reported.
  4. The "action-guided" part of the title is underspecified. Sec 3.1.1 does not give how δt is computed from the control signal, nor how the 6-DoF camera poses used by LingBot and Inspatio produce a position-dependent displacement field (Appendix A.1.1 only gives directions for discrete keys). In Table 5, Global search without the action prior already reaches 16.7 FPS and Avg 67.4, which suggests that the action prior mainly reduces matching cost. The paper lacks an action-agnostic baseline of "zero displacement + local window with the same radius", and all ablations are run only on MatrixGame2.
Questions for authors
  1. Eq.5/7 in the main text state that Q is not computed at reused positions and the output is copied directly, but line 21 of Alg.1 in the appendix says "Compute queries at all current positions". Which one does the actual implementation do?
  2. The reused K/V and O come from the append pass at timestep 0, while the current chunk is at a noisy step. Why does this noise-level mismatch not cause problems?
  3. When a chunk contains multiple latent frames, which frame of the previous chunk is a current frame matched against?
  4. Are the "held-out validation trajectories" used for radius selection (p5 L229) separate from the WBench test set?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.