acceptodds
Withdrawn from ICLR 2027

ConsistKV: Long-Term KV Memory for Consistent Interactive World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive world models can generate controllable visual environments in real time, yet they often fail to preserve previously observed scenes over long interactions. We identify this failure as revisit collapse: once historical observations leave the bounded KV context, the model loses direct access to their visual states and may reconstruct different content when revisiting the same location. We propose ConsistKV, a lightweight external KV memory for frozen interactive world models that maintains compressed distant history without extending the active attention window. For retrieval, our Action–Content Encoder (ACE) jointly models visual content and action-derived relative geometry, combining appearance correspondence with complementary geometric localization. We further introduce a training-free Value-Absorbing KV Compression strategy that reduces historical KV storage while preserving information from discarded positions through shared self-Gram-guided selection and value absorption. Experiments on Matrix-Game 2.0 and ABot-World show improved revisit consistency over sliding-window and fixed-budget memory baselines, with clearer gains over longer interaction horizons.

Primary area:
generative models
Keywords:
Interactive World Models, Long-Term Memory, KV Cache Compression

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YTJYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ConsistKV aims to maintain visual consistency in interactive video world models within a fixed active KV budget. It learns an action–content retriever from generated mirror-retracing trajectories, compresses historical KV chunks through key selection and value absorption, and places selected chunks alongside an initial anchor and recent states. Experiments on Matrix-Game 2.0 and ABot-World show improved consistency on scored revisits over fixed-budget baselines. The quantitative evidence chiefly concerns returns to the same or nearby action-derived poses.

Critical strengths
  1. The method reuses the backbone’s own KV states and adapts their positions when recalling them, without retraining the generator or enlarging its active attention window (Equation 12; Appendix A.1). Its gains on both backbones (Table 1) make this a useful inference-time system design.
  2. Value absorption leaves retained keys addressable while carrying information from discarded positions. Its fixed-retrieval ablation improves revisit fidelity over selection alone (Table 2), supporting the practical value of this compression choice.
Critical weaknesses
  1. The need for ACE’s learned fusion is not established. With compression fixed, ACE improves average PSNR over action retrieval by only 0.05 dB on the main MG benchmark and is slightly worse on Complex (Table 2). Its gains on cross-path and 12-lap tests are encouraging, so this is not a claim that ACE has no benefit. The missing comparison is with simple action–content score fusion and a learned action-only encoder under matched conditions. The current comparisons cannot establish whether visual content supplies the claimed correspondence, or whether ACE mainly learns a better geometric score. Moreover, Its visual query describes chunk t−1t-1 but is assigned zero relative geometry at destination tt (Equation 5), a potentially misleading approximation after substantial motion.
  2. The specific benefit of shared-Gram value absorption over simpler transfer rules remains unclear. Absorption improves the no-absorption ablation, but adapted PyramidKV nearly matches its average PSNR (14.49 versus 14.54) and has better LPIPS at the same retained-position budget (Table 15). Moreover, Equations 8–11 favor keys representative of their chunk and mix discarded values into retained addresses without conditioning on the future query. Distinctive but rare scene details may therefore be lost or blurred. The average ablation demonstrates a benefit, but does not establish preservation of query-relevant memory.
  3. The evaluated form of long-term memory is narrower than reuse of a scene from a new viewpoint. ACE is trained on mirror-retracing loops from eight initial images (Appendix A.2). The main tests repeat or retrace paths; the non-retraced paths in Appendix E still score visits near an earlier action-integrated pose. These experiments support consistency on same- or near-pose revisits, but do not establish whether ACE can retrieve useful history when an old scene is seen from a substantially different viewpoint.
  4. The effect of always-on retrieval when history is irrelevant remains unknown. Memory delivery reserves space for six historical chunks while retaining only one or two recent native chunks (Equation 12). This may reduce the context available for current motion on trajectories without a useful historical match. Figure 6 provides qualitative non-revisit examples, but a same-budget quantitative comparison with recent-context or adaptive retrieval is needed to establish this trade-off.
Questions for authors
  1. Under matched training data, retrieval size, compression, and inference budget, how does ACE compare with a same-capacity learned geometry-only encoder and calibrated action-plus-content score fusion? Please report paired results by trajectory type, including cases where the cues disagree and cases with large turns. If feasible, it would be helpful to include a few ACE retrieval visualizations, particularly cases where action-derived geometry and visual appearance favor different historical chunks.
  2. With the retained key positions fixed, how does Equation 11 compare with nearest-key or normalized-average value transfer?
  3. Can you evaluate re-observation of a known scene from a new viewpoint when no near-pose historical frame exists? Please use verified visual overlap rather than unaligned pixel similarity, and show the query, ranked retrieved frames, and generated result for representative successes and failures.
  4. On trajectories without near-pose revisits, how does always-on retrieval compare with a same-budget recent-context baseline or an adaptive retrieval policy in action adherence, scene continuity, and runtime?
Reviewer_SW53OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes ConsistKV, an external KV memory for improving long-term revisit consistency in interactive world models. It combines similarity-based KV selection and value absorption with learned retrieval using visual content and action-derived geometry. While the problem is relevant, I find the methodological novelty and qualitative evidence insufficient to support acceptance.

Critical strengths
  • This world addresses an important limitation of interactive world models while keeping the backbone frozen and maintaining a fixed active attention budget.
  • It evaluates two backbones and provides component ablations showing improvements in revisit consistency.
Critical weaknesses
  • My major concern is the limited novelty in compression. Selecting representative KV positions through key similarity appears incremental relative to existing similarity-based compression methods. The justification for absorbing discarded values into retained positions is also insufficient: discarded positions may contain distinct information, and key similarity alone does not establish that modifying retained values preserves their original meaning.
  • Only quantitative evaluation is not engouth.. The displayed results still show noticeable distortions and loss of scene detail. More critically, no video demonstrations were available in the materials I reviewed. Selected frame comparisons and revisit metrics are insufficient to assess temporal coherence, transitions during retrieval, and persistent scene memory throughout an interaction. This is my another main concern.
  • Small metric gains. Several improvements over the strongest baseline are small, and their statistical and perceptual significance remains unclear. A video-based user study focused on memory fidelity and revisit consistency is needed to establish whether these gains translate into a meaningful perceptual improvement.
Questions for authors

my concerns lie on the novelty and evaluations.

Reviewer_BkSFOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

ConsistKV introduces an external KV memory framework for improving long-term revisit consistency in frozen interactive world models. It combines an Action–Content Encoder (ACE) for historical retrieval with training-free Value-Absorbing KV Compression, reinserting retrieved states into the native attention window under a fixed active budget. Experiments on Matrix-Game 2.0 and ABot-World demonstrate improved revisit consistency, particularly over longer interaction horizons.

Critical strengths
  • Training-free and a practical toppings for consistent world modeling. ConsistKV preserves the backbone’s native attention budget and requires training only a lightweight retrieval module. Its evaluation on two backbones supports the applicability of the framework.

  • Comprehensive results. The system improves long-horizon revisit fidelity over WorldKV and LongLive-RAG. The compression ablation provides direct evidence for value absorption.

Critical weaknesses

Major

  • ConsistKV primarily integrates existing ideas in memory retrieval and KV cache compression, ie WorldKV. Although the overall framework is practical, the individual components appear incremental. Preserving retained keys while merging discarded values also has precedents, including WeightedKV. The distinctive contributions therefore lie primarily in ACE and the shared self-Gram formulation (which is just a widely used similarity), rather than the general memory architecture. However, experimental results indicates the marginal performance gain. ACE improves average PSNR over WorldKV by only 0.05 dB (14.54 versus 14.49; Table 2) and 0.03 on SSIM, but sacrificing the efficiency (30.2 vs 26.9 FPS).

  • Although Table 15 includes relevant merging baselines, the comparisons do not isolate the benefit of sharing the same Gram structure between selection and absorption. Moreover, the compression method does not uniformly outperform existing alternatives: PyramidKV achieves comparable average PSNR and better LPIPS, while KVMerger achieves better SSIM. Controlled comparisons that fix the retained positions and vary only the absorption rule would better establish the contribution of the proposed formulation.

  • I think traditional indicators, like PSNR, SSIM are insufficient to reflect the "consistency" in world modeling. The cross-path and geometric-noise experiments are useful additions, but they are not enough. Evaluation using verified camera poses or visibility correspondences, together with separate action-following metrics, would strengthen the conclusions. Broader return trajectories would also help establish generalization beyond the scripted revisit settings.

Minors

  • Lacking of a related work [1], which already discussed the similar KV compression, in reasoning model.

References

[1] Cai et al., R-KV: Redundancy-aware KV Cache Compression for Reasoning Models, NeurIPS 2025.

Questions for authors
  1. How does ACE compare with simple geometric–visual score fusion or geometry-based filtering followed by visual reranking, using the same compression and memory budgets?

  2. Does value absorption remain beneficial when compared with norm-matched value scaling or normalized merging at identical retained positions? Since Eq. (11) changes both value content and potentially its magnitude, these controls would help distinguish information preservation from stronger memory weighting.

  3. In Section 4.2, the authors modify retained values by absorbing discarded values based on key similarity. However, it is unclear why a frozen world model can effectively utilize these modified values, as key similarity does not necessarily guarantee preservation of the original attention outputs. Could the authors provide further justification or analysis to support this design?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.