acceptodds
Withdrawn from ICLR 2027

ActionCache: Action-Aware Feature Caching for Interactive Video World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive video world models require both low-latency generation and prompt responses to user control. Existing training-free caches reuse along different axes while deciding validity from signals read inside the current model computation after the forward pass has begun. This internal gating has two limitations: (1) feature redundancy along the denoising trajectory collapses once models are distilled to three or four steps, and (2) the exogenous user action that invalidates the cache arrives before the forward pass, yet is folded into a model-side quantity and so reaches the decision only after that pass has begun. We present ActionCache, which reads the action before each forward pass and uses it as the governing signal for cache reuse: it recomputes when the control changes or the cached update grows too old, and reuses it otherwise. Because the criterion comes from the control stream, reuse survives few-step distillation. ActionCache operates directly on pretrained generators and integrates into existing interactive video world models. On three such models, spanning long diffusion sampling and distilled short-step streaming generation, ActionCache attains up to speedup while the action-response-latency increase stays within frame on every backbone. These results point to a caching principle for interactive generation: the action stream as an ahead-of-time prior over cache validity.

Primary area:
generative models
Keywords:
world models, interactive video generation, feature caching, training-free acceleration, diffusion models, action-conditioned generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6NFVOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ActionCache, a training free feature caching controller designed for interactive video world models. Unlike existing visual probe caching approaches that evaluate reuse validity internally after the forward pass has begun, ActionCache utilizes the user control stream arriving prior to model computation. By defining threshold based boundaries over action variations alongside a periodic age bound and forced recomputation conditions, the method avoids forward pass probing overhead and mitigates control latency during action transitions. Experiments across multi step diffusion models such as Vid2World and few step distilled models such as Matrix Game and Oasis report noticeable inference speedups while restricting the increase in action response latency.

Critical strengths
  • The paper identifies a useful systems opportunity: user actions are available before the computation they condition, so they can inform cache decisions without adding a model-side probe to the forward-pass path.
  • The evaluation covers both a long-sampler model and few-step streaming models. The authors also examine control fidelity with blind human ratings, a frozen inverse-dynamics model, and ground-truth-anchored measurements on Vid2World.
  • ActionCache is training-free and lightweight. The paper reports negligible action-check overhead and demonstrates that the controller can be layered on existing cache implementations.
Critical weaknesses
  • The core algorithm relies on a straightforward heuristic combining threshold-based input differences with a periodic refresh, which makes framing the approach as a causal cache-validity tube feel somewhat overstated in terms of technical novelty.
  • The foundational premise that action stability across a stable span implies feature redundancy is fragile. In interactive environments, holding a constant input such as continuous walking or steering still induces dramatic visual changes due to background evolution, physics, and dynamic entities. The paper's own diagnostic shows that the action signal is an incomplete predictor of residual drift, with its predictive effect concentrated mainly in near-static action classes. The system therefore relies heavily on the rigid periodic age bound rather than genuine feature-level redundancy.
  • The acceleration comes at a measurable cost to reported visual-fidelity metrics. LPIPS and FVD worsen relative to the uncached baselines in the settings where they are reported. These changes raise a quality concern, although their perceptual significance is not established by the results.
  • The evaluation of control latency raises a circularity concern. The primary metric ARL is defined by the authors and evaluated specifically on transition frames. Because ActionCache is designed to force a full recomputation whenever a control transition occurs, its ARL advantage over other caching methods is partly built into the policy. The paper also reports independent human, inverse-dynamics, and ground-truth checks that reproduce the broad ordering, which partially addresses this concern.
Questions for authors
  • When a user sustains a constant input in an environment undergoing rapid dynamic changes, how does the system prevent ghosting or semantic drift before the periodic refresh threshold is reached?
  • Could you provide visual qualitative examples or breakdown analyses specifically illustrating failure cases under static actions with high visual motion?
  • If evaluated under standard, method-agnostic control-accuracy benchmarks that do not disproportionately sample transition frames, how much of the latency and control advantage of ActionCache remains?
Reviewer_8xYrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Action cache, a training-free cache scheduler for interactive video world models. Before each generator forward pass, it checks for keyboard or mouse-control changes and cache age; it recomputes when a boundary is crossed or a forced-refresh condition applies, and otherwise reuses cached fused CFG outputs or transformer-block residuals. The authors evaluate it on Vid2World, Matrix-Game 2.0, and an independently implemented Oasis-500M, and also test stacking it with existing caches. They report up to 3.25× speedup on Vid2World, with action-response latency increases of at most 0.2 frame across the three backbones. The evaluation includes component ablations, stress tests, and human and inverse-dynamics validation of control-fidelity metrics. The main competitor comparison ports decision rules to shared cache substrates rather than comparing all methods as complete native systems.

Critical strengths
  1. The controller uses action and cache metadata before the expensive forward pass, requires no training, and is applicable to both long-sampler and few-step settings. This makes the method a potentially useful inference optimization, rather than a backbone-specific modeling change.
  2. he paper reports an untouched 500-episode Vid2World test split, a 90-rollout Matrix-Game confirmation, ablations, continuous-control stress tests, and independent human and inverse-dynamics checks. The reported pooled Matrix-Game speedup is 1.44× with a +0.10-frame ARL change; the independent checks strengthen the case that the measured control-fidelity changes are not solely artifacts of the authors’ metrics.
Critical weaknesses
  1. The drift bound depends on local regularity and bounded-context assumptions whose constants are not estimated or checked by the controller. The paper also reports that action–drift separation is concentrated in two rare action classes, while most drift variance is associated with unobserved context; 3.17% of same-action pairs exceed the stated τ = 1.10 diagnostic tolerance. The empirical results are useful, but the theoretical framing does not establish general false-skip safety.
  2. The main baselines are decision-rule ports to shared cache substrates, with some release-specific components omitted, so the results do not establish superiority over the complete native methods. In addition, actions are latched at chunk start; a mid-chunk input takes effect only at the next chunk boundary. The reported ARL comparison therefore does not demonstrate prompt response to arbitrary mid-chunk user inputs. Matrix-Game also has no held-out corpus, while the Oasis result compares against only the authors’ uncached implementation.
Questions for authors
  1. Given that actions are latched for a chunk and mid-chunk events take effect at the next boundary, what are the typical and worst-case response delays for inputs arriving mid-chunk? Can the authors report performance under asynchronous or otherwise finer-grained control updates?
  2. For each competitor port, which components were retained or omitted, and how might those omissions affect the shared-substrate comparison? Would full native comparisons at matched speed–quality or speed–ARL operating points change the relative conclusions?
  3. Can the authors provide transition-detector and age-refresh ablations on Matrix-Game and Oasis, not just Vid2World? This would help determine whether action gating contributes consistently in the few-step regimes central to the paper’s motivation.
Reviewer_3rVaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ActionCache, a training-free feature-caching controller for interactive video world models. Unlike existing methods that require partial model computation to assess cache validity, ActionCache makes reuse decisions before expensive model computation by monitoring changes in the action inputs and cache age. It refreshes cached features when the action changes or the cache expires, accelerating generation without modifying the pretrained backbone.

Critical strengths
  1. The motivation is reasonable: user actions are available before model execution and can provide an inexpensive signal for scheduling cache reuse without model-side probing.
  2. The reported experiments demonstrate useful speedups across multiple backbones, including few-step distilled models.
Critical weaknesses
  1. The reliability of cache reuse under unchanged actions remains insufficiently established. As acknowledged in lines 73-74, scenes can evolve even when the control signal remains constant. A fixed expiry limits reuse duration but cannot detect rapid scene changes within that interval, for example, moving through a doorway while continuously holding the same forward action. Although the paper includes periodic-refresh ablations and a sweep over KK, these results do not specifically establish robustness to such events. Targeted experiments are needed to determine whether the expiry mechanism adequately handles them.
  2. Evaluation on more recent open interactive world models is missing. Results on LingBot-World-Infinity or Matrix-Game 3.0 would provide stronger evidence that the method generalizes to current generation systems.
  3. The qualitative evidence presented in the paper is limited. For a video-generation acceleration method, static frame comparisons are insufficient to assess temporal artifacts, delayed scene updates, and visual drift. The authors should provide video comparisons covering representative cases and potential failures.
  4. The largest reported speedup, 3.25×, comes from Vid2World with a 50-step DDIM sampler. However, mainstream interactive world-model backbones are now typically distilled to use only a few sampling steps. Thus, although Vid2World achieves the largest speedup, this result has relatively limited relevance to current practice.
  5. The contribution appears incremental, primarily consisting of a lightweight action- and age-based scheduler added to existing caching mechanisms. Its broader significance is further limited by the unresolved concern about cache validity when visual content changes substantially under unchanged control inputs.
Questions for authors
  1. Can the authors provide results on newer open interactive world models, particularly LingBot-World-Infinity and Matrix-Game 3.0, using their released fast or distilled configurations?
  2. Can the authors evaluate cases where the action remains unchanged but the scene changes rapidly, such as passing through a doorway while continuously moving forward? Comparisons against uncached generation would help establish whether the fixed expiry adequately handles these cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.