acceptodds
Under review as a conference paper at ICLR 2027

TokenWeave: Consolidating Temporal KV History into Persistent Scene Memory for Video World Models

Abstract

Streaming autoregressive video world models naturally accumulate history as a temporal sequence of key–value (KV) states. However, long interactions repeatedly observe the same scene content across distant times, making such temporally organized history increasingly redundant and eventually forcing old observations to be discarded. We introduce TokenWeave, a training-free framework that reorganizes temporal KV history into persistent scene memory by associating and consolidating repeated patch-level observations. Rather than managing history as a collection of separate observations, we reorganize repeated observations around recurring scene content. It operates entirely on the frozen model’s internal KV states, grouping observations that likely correspond to the same scene content and integrating them into persistent memory entries. These shared entries are updated with position- and multiplicity-aware fusion, while genuinely novel content is retained as new memory. Memory therefore grows mainly with new or unmatched content rather than with time. % The resulting memory therefore scales with scene novelty rather than rollout duration. Experiments across multiple domains and rollout lengths demonstrate substantial improvements in long-horizon revisit consistency without memory-specific training. Additional videos are available at https://tokenweave-iclr.github.io/.

Primary area:
generative models
Keywords:
Memory, Video World Model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TqrdOpenReview ↗
Rating
3(+1)
Confidence
4(+2)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TokenWeave proposes a training-free memory mechanism for autoregressive video world models. It uses camera geometry and native-key similarity to associate historical patch tokens with persistent memory slots, then updates these slots through positional alignment and weighted averaging. A count-based correction adjusts their contribution to attention. Results on two LingBot-World backbones show improved revisit consistency and lower cache memory than full-history attention.

Critical strengths
  1. The approach is practical. It works directly with the frozen backbone’s KV states and requires no memory-specific training. Keeping the recent window unchanged is also a sensible choice for preserving short-term temporal behavior;
  2. The paired comparisons with the original backbones are a strong part of the paper. Both LingBot-World versions improve across the three rollout durations, with the checkpoint, input, trajectory, and sampling settings held fixed within each comparison;
  3. Tab. 2 provides useful evidence for the compression benefit. TokenWeave achieves similar warped consistency to full-history attention while using substantially less cache memory.
Critical weaknesses
  1. I think key similarity is a reasonable starting point for matching observations, but the distinction between “the same scene region” and “similar-looking regions” needs more evidence. Removing RoPE addresses explicit positional encoding; it does not ensure that two similar keys refer to the same physical content. The local fallback can also admit candidates outside the pose-guided region. For example, two similar windows may have similar keys but should remain separate in memory. Showing the observations assigned to a few slots over an entire trajectory, including incorrect associations, would make the scene-centric interpretation more convincing;
  2. The paper already explains that the count correction is exact for identical keys and approximate otherwise. My concern is how this approximation behaves once the averaging weight reaches its cap. At that point, the slot increasingly reflects recent observations, while its count still includes the entire history. It is not obvious that the resulting slot preserves the contribution those historical observations would have made to attention. I think a direct comparison of attention outputs before and after consolidation, especially after many revisits, would help justify this choice. It would also clarify how much information is retained when the matched observations differ in viewpoint or appearance;
  3. Appendix B.3 states that pairs failing registration are excluded from the metric averages. I think the frequency of these failures is important for assessing revisit consistency. If a method changes the scene substantially, its revisit may fail to register and disappear from the average. It would help to report the valid-pair counts for every method and duration, alongside results on a common valid subset. The fitted homography introduces another uncertainty because it can absorb some geometric changes, as acknowledged in Appendix B.4. This matters particularly for Tab. 3, where EMR improves the warped metrics but slightly reduces the direct scores;
  4. Tab. 2 shows clear savings relative to full-history attention. However, at 60 seconds, TokenWeave still uses 77.76 GiB of cache memory, compared with 30.72 GiB for the original window. I would like to see an extended window or historical-token retrieval evaluated on the same backbone under the same cache budget. That would help establish the benefit of consolidation beyond simply retaining more history. Longer-running behavior also deserves attention: persistent slots have no eviction policy, and capacity is allocated to fit the evaluated rollout. Repeated loops and continued exploration would help show how slot count, memory usage, and throughput evolve in practice.
Questions for authors
  1. Can the authors show what individual memory slots accumulate over time, and how often observations from different scene regions are incorrectly merged?
  2. How closely does the consolidated cache reproduce the original attention outputs after the averaging weight reaches its cap?
  3. What are the registration success rates for each method and duration? Do the main conclusions hold on a common valid subset?
  4. How does TokenWeave compare with retrieval or an extended window under the same cache budget, and what happens when the slot capacity is reached?

Above all, if the author can solve all of my questions. I would like to consider increase my scores.

7 comments
AuthorsOpenReview ↗

Response to Reviewer Tqrd (Part 1/2)

We thank the reviewer for the detailed review. Short answers first, details below.

Summary. (Q1) Slots stay on the same physical content; <2% of merges are wrong. (Q2) EMR's attention outputs closely match Full-KV outputs, and capping the averaging weight barely changes them (new Fig. 7). (Q3) No pair was ever excluded in the reported metric; registration success counts are added. (Q4) Under a matched budget TokenWeave beats extended sliding window and retrieval; memory plateaus when the camera is still.

Q1. Same scene region vs. similar-looking regions; local fallback; incorrect merges

What slots accumulate (App. C.2, Fig. 6). We followed slots across many complete trajectories. Fig. 6 shows three examples: every observation merged into a slot lands on the same physical content, e.g. brickwork below a window (27 observations), a window joint (24), a fork in a tree trunk (64), despite large viewpoint changes. Slots also bridge long gaps: the window-joint slot absorbs observations from 5.38–13.88 s and again from 49.44–55.81 s, after the region had left the recent context for >35 s. This cross-visit association is exactly what a sliding window cannot provide and what TokenWeave is built for.

Incorrect merges are rare and benign. Across >200 manually inspected merge events, <2% were incorrect, and all of those fell into a neighboring slot with near-identical texture (adjacent patches on weakly textured walls), where a mistake barely changes the rendered content. Two separate similar objects (the reviewer's two windows) are usually kept apart: they project to different image locations, so one falls outside the other's capsule, and a merge also needs the RoPE-free similarity to exceed τ and be the highest among candidates (Eq. 4).

Local fallback. It is a safeguard for imperfect camera control, not a main source of associations: entering the candidate set does not imply a merge, only ~4.5% of accepted merges involve a fallback-only slot, and removing it changes results only marginally (Table C.2, benchmark subset):

Candidate setMerge ratePSNRSSIMwPSNRwSSIM
Capsule ∪ local fallback (default)70.13%12.1330.303214.7970.4463
Capsule only69.67%12.1570.306614.5730.4494

Q2. Attention outputs after consolidation

We agree that direct evidence is needed and provide it at two levels.

(i) Attention outputs, with and without the cap (App. C.3, Fig. 7).

Fig. 7. Attention-output comparison of Full KV, uncapped EMR, and EMR with cap=16. The left and middle columns overlay each EMR variant with Full KV, while the right column enlarges a fixed range of output coordinates. Curves show the mean readout across four selected heads for the same query. All settings share queries and historical observations; the two EMR variants additionally share merge assignments and KV budgets. Cap=16 limits the cumulative weights used to update representative K/V, while both variants retain the true-count log(size) correction.

Why cap. Without a cap, the nn-th observation gets weight 1/(n+1)1/(n+1), so frequently observed slots freeze and cannot follow the evolving scene. Capping wjw_j at 16 keeps each update weight ≥1/17\geq 1/17, while the uncapped njn_j still sets the slot's attention mass. We acknowledge that a capped slot's K/V is then weighted toward recent observations. However, all merged observations passed the similarity test and depict the same region despite viewpoint changes (Fig. 6), so this reweighting barely moves the representative.

Result. EMR closely reproduces Full-KV attention outputs, and the cap barely changes them. Across the selected views (four-head average), uncapped and cap=16 EMR reach mean relative L2 errors of 7.18% and 7.18% against Full KV (cosine 0.9973 for both). Over all 144 captured blocks (all eight heads), the capped output differs from uncapped EMR by only 0.20% relative L2 (cosine 0.99999).

(ii) End-to-end comparison with exact attention (Table 2). At 45 s and 60 s, when many slots have accumulated long histories, TokenWeave matches or exceeds the full-history cache in warped scores (16.47 vs. 16.43 dB wPSNR at 45 s; 15.18 vs. 15.16 dB wPSNR and 0.4880 vs. 0.4649 wSSIM at 60 s) while using 60.5–68.2% less KV memory. Removing the count correction lowers wPSNR by 0.98 dB (Table 3).

AuthorsOpenReview ↗

Response to Reviewer Tqrd (Part 2/2)

Q3. Registration failures, valid-pair counts, homography

Correction to the appendix text. : Appendix B.3 in the submission described another protocol that we used during development, not the one used for the reported numbers. No reported result excludes any pair. The actual protocol (now in App. B.3/B.4) is as below:

  • Registration succeeds → revisit frame chosen by Eq. B.3; warped scores use the fitted homography.
  • Registration fails → fixed frame index on the return sweep from the commanded trajectory; no warp, so warped = direct scores.
  • All 165 pairs per duration enter the averages for every method.

Registration success (out of 165 pairs; new Table B.2):

Method25 s45 s60 s
LingBot-World-v2794550
+ TokenWeave140132122
LingBot-World-v1824851
+ WorldKV816350
+ TokenWeave134109104
SolarWM-H3916961
+ TokenWeave142135132
Infinite-World148142139
HY-WorldPlay145140142
  • Success rate is itself a consistency measure independent of warped scores: it falls with duration without memory (79 → 50) and TokenWeave raises it on every backbone (+72 / +53 / +71 pairs at 60 s).
  • This protocol removes the selection-effect concern: a method that changes the scene fails registration and gets a poor unaligned score, rather than being dropped. It is also conservative for TokenWeave, since LingBot-World's larger camera-control error causes registration failures even when content is preserved.
  • Homography. A single global 8-DoF planar transform, applied identically to every method. Gains do not depend on warping: on SolarWM-H3, direct PSNR improves by 4.46–4.96 dB. For Table 3, EMR changes direct PSNR by only −0.17 dB but improves wPSNR by +0.98 dB, indicating better scene content under similar camera error.

Q4. Matched cache budget; behavior at capacity

Matched budget (App. C.1, Table C.1; LingBot-World-v1, averaged over 25/45/60 s; full table in our response to Reviewer E9Dn**).**

MethodwPSNRwSSIMKV budget ↓
Backbone10.0410.3058–
+ Extended sliding window11.3430.291935%
+ Retrieval (WorldKV)10.4390.284950%
+ TokenWeave14.6490.429732.13%

The gain does not come from keeping more history. The baselines retain as much or more context at an equal or larger budget, yet gain far less; what differs is how TokenWeave organizes it, consolidating observations by scene content via pose-guided association. Three training-free KV-compression methods tell the same story (see Reviewer uY4Q).

Growth and capacity (App. C.4, Fig. 8). On four trajectories mixing motion with long stationary segments, memory grows only while new observations arrive and plateaus when the camera is stationary (+0.23% over the final hold of a 66 s run; no noticeable growth under micro-motion). Memory scales with unmatched observations, not elapsed time. Our experiments currently impose no KV Cache Memory budget; under a hard budget, oldest-first eviction is compatible with our design, and the budget will be reached much later than with a full temporal cache.

We hope these results address the reviewer's concerns and are happy to answer any follow-up.

Thanks for running the extra experiments and detailed responses. The cache comparisons and registration results helped clear up several of my core concerns, and I’ve decided to raise my score to 3.

I do have one follow-up question. If we look only at pairs where registration succeeds for both the backbone and TokenWeave, do we still see similar gains? It would be helpful to see the direct and warped scores for those pairs, given that failed pairs are evaluated without alignment.

Reviewer_E9DnOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TokenWeave, a training-free framework for improving long-term revisit consistency in autoregressive video world models. It consolidates historical key–value states into persistent, patch-level memory slots using pose-guided candidate selection and native-key similarity. Matched observations are fused after aligning their rotary positional embeddings, while a multiplicity-aware attention correction accounts for the observations represented by each slot. Experiments on two frozen LingBot-World backbones across multiple domains and rollout durations report consistent improvements over sliding-window inference. Compared with full-history caching, TokenWeave achieves comparable revisit quality with substantially lower KV-cache memory usage and higher throughput.

Critical strengths
  • Improving long-term revisit consistency without additional training has practical value for existing video world models. Processing only states leaving the recent window also makes the method straightforward to integrate into streaming inference.
  • Separating fixed geometric anchors from mutable appearance timestamps preserves spatial references while allowing memory updates. Removing RoPE before matching and aligning positional encodings before fusion also addresses the challenges of merging observations across viewpoints and time.
  • The paired experiments show improvements across both backbones, all three rollout durations, and all four metrics, with wPSNR gains of 4.17–7.82 dB. Using identical checkpoints, inputs, trajectories, and random seeds strengthens the within-backbone comparisons.
  • The full-history comparison demonstrates a useful quality–efficiency trade-off. At 60 seconds, TokenWeave achieves nearly identical wPSNR and higher wSSIM while reducing KV-cache memory from 233.20 to 77.76 GiB and increasing throughput from 1.25 to 1.75 FPS.
Critical weaknesses
  • At 60 seconds, TokenWeave increases KV-cache memory from 30.72 to 77.76 GiB while reducing throughput from 2.12 to 1.75 FPS relative to the sliding-window baseline. The quality gains therefore also reflect access to a larger historical context.
  • The full-history comparison demonstrates compression benefits but does not establish whether simpler history retention, sampling, or retrieval strategies could achieve comparable quality under the same KV-cache budget.
Questions for authors
  • How much of the improvement over the sliding-window baseline comes from access to a larger historical context, and how does TokenWeave perform when the KV-cache budgets are matched?
  • Could simpler history retention, sampling, or retrieval strategies achieve comparable revisit quality under the same KV-cache budget as TokenWeave?
3 comments
AuthorsOpenReview ↗

Response to Reviewer E9Dn

We thank the reviewer for the positive assessment and precise questions.

Summary. Under matched budgets, TokenWeave beats extended sliding window, random retention, and retrieval on all four metrics with the smallest budget. Among the simple retention strategies tested, the strongest achieves only about half of TokenWeave's wPSNR gain.

Q1. How much of the gain comes from more history? Matched KV budget

To separate "more history" from "how it is organized", we compare retention strategies on LingBot-World-v1 with identical trajectories and seeds, each with a budget no smaller than ours (App. C.1, Table C.1; averaged over 25/45/60 s; budget = fraction of full-history cache):

MethodPSNRSSIMwPSNRwSSIMKV budget ↓
LingBot-World-v19.9570.262510.0410.3058–
+ Extended sliding window11.0450.278711.3430.291935%
+ Random retention11.0370.292612.3190.327635%
+ Retrieval (WorldKV)10.1190.213710.4390.284950%
+ TokenWeave12.0960.312914.6490.429732.13%

TokenWeave gains +4.61 dB wPSNR; with slightly more memory, extended sliding window and random retention gain only 1.30 and 2.28 dB. Among the simple retention strategies tested, the strongest (random retention) achieves only about half of TokenWeave's wPSNR gain. This comparison supports the benefit of scene-centric memory organization beyond these simple ways of retaining more history.

Q2. Can simpler retention, sampling, or retrieval match TokenWeave at the same budget?

No. TokenWeave is best on all four metrics at the smallest budget, ahead of retention (extended sliding window), sampling (random retention), and retrieval (WorldKV). Under the same budget, the difference is how tokens are organized, not how many are kept, and the gap is large: +2.3 dB wPSNR over the best simple baseline (random retention). The strongest ported KV-compression method, Forcing-KV, reaches 13.15 dB wPSNR at 32.85%, below TokenWeave's 14.65 dB at 32.13% (see our response to Reviewer uY4Q).

Throughput (2.12 → 1.75 FPS vs. the sliding window at 60 s) is the cost of attending to persistent memory. TokenWeave remains faster than full history (1.25 FPS) with matching or better quality, and memory growth stops when the camera is stationary (App. C.4, Fig. 8).

Additional result. On the new SolarWM-H3 backbone, TokenWeave improves all four metrics at all durations (wPSNR +5.12 / +5.78 / +5.86 dB) and is the best entry in every column of Table 1.

We hope these results address the reviewer's questions.

Thanks for your reply and your updated manuscript. My concerns have been addressed. I'll maintain my score and increase my confidence. Good luck with your rebuttal.

Thank you for your thoughtful review and prompt feedback. We’re glad that our revisions have addressed your concerns, and we sincerely appreciate your time and effort in evaluating our work!

Reviewer_uY4QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TOKENWEAVE, a training-free framework to reorganize temporal KV history in streaming autoregressive video world models into a persistent, content-centric scene memory. Rather than accumulating past observations chronologically or retrieving coarse historical frame chunks, it operates at patch granularity on tokens exiting the active sliding window. It combines a scale-free epipolar geometric gate (Pose-Guided Candidate Selection) with cosine-similarity matching over RoPE-unrotated native keys to associate repeated observations of the same scene region across time. Matched tokens are consolidated into shared persistent slots using decoupled geometric registration and temporal appearance clocks alongside capped feature averaging. Then, Evidence-Aware Memory Reading (EMR) applies a multiplicity-based logit correction to preserve softmax attention mass. Evaluated across 165 scenes and three rollout horizons (25s, 45s, and 60s) on two frozen 14B-parameter backbones (LingBot-World-v1 and LingBot-World-v2), TOKENWEAVE substantially improves long-horizon revisit consistency (PSNR, SSIM, and homography-warped variants) over sliding-window baselines. It reduces KV-cache memory by over 60% compared to full-history caching, and achieves competitive or superior aligned structural consistency relative to memory-trained systems.

Critical strengths

I found the proposed method is very effective in reducing the KV cache memory. Across multiple domains up to 60 seconds, the proposed method consistently improves revisit metrics on two 14B frozen backbones. It outperform uncompressed full-history attention at 60 seconds while reducing KV-cache memory by 60.5–68.2%.

Critical weaknesses

The major weakness of this paper is the authors did not evaluate any baseline training-free KV-cache compression, pruning, or retrieval baseline method. Several baseline methods are mentioned in the related work section. Please consider including them in the experiment baselines.

[1] Yi, Jung, et al. "WorldKV: Efficient World Memory with World Retrieval and Compression." arXiv preprint arXiv:2605.22718 (2026). [2] Ji, Yicheng, et al. "Forcing-KV: Hybrid KV Cache Compression for Efficient Autoregressive Video Diffusion Models." arXiv preprint arXiv:2605.09681 (2026). Ji, Y., Zhong, Z., Zhang, J., Ya [3] Yi, Jung, et al. "Deep forcing: Training-free long video generation with deep sink and participative compression." arXiv preprint arXiv:2512.05081 (2025). [4] Ma, Yuexiao, et al. "Flow caching for autoregressive video generation." arXiv preprint arXiv:2602.10825 (2026).

Questions for authors

My main question to the authors is regarding the baseline methods. It would be helpful to see how they perform in comparison to the proposed method.

2 comments
AuthorsOpenReview ↗

Response to Reviewer uY4Q

We thank the reviewer for recognizing the efficiency and consistency of our method, and for pointing us to these baselines.

Summary. (1) WorldKV is already in Table 1: +0.40 dB wPSNR vs. TokenWeave's +4.61 dB on the same backbone, with a smaller cache. (2) We ported Forcing-KV, Deep Forcing, and FlowCache to the same backbone; TokenWeave beats all three at the lowest budget.

1. WorldKV is already evaluated

It is the row "+WorldKV" under LingBot-World-v1 in the "Without Memory Training" block of Table 1. We use its official implementation, built on LingBot-World-v1, so it shares a backbone with our "LingBot-World-v1 + TokenWeave" row. We add a "Δ vs. base" row for WorldKV. Averaged over durations, WorldKV gains +0.40 dB wPSNR, TokenWeave +4.61 dB, with a smaller cache (32.13% vs. 50%).

2. Forcing-KV, Deep Forcing, FlowCache

These methods target autoregressive video generation and do not accept the camera-control signal of a world model, so they cannot run directly on our camera-controlled benchmark. We therefore ported the core KV-cache operation of each to LingBot-World-v1. Budgets are set so TokenWeave's is no larger than any baseline's (App. C.1, Table C.1; averaged over 25/45/60 s):

Method (all on LingBot-World-v1)PSNRSSIMwPSNRwSSIMKV budget ↓
Backbone9.9570.262510.0410.3058–
+ WorldKV10.1190.213710.4390.284950%
+ Forcing-KV11.1740.289613.1530.384632.85%
+ Deep Forcing10.6320.237011.7390.326732.19%
+ FlowCache10.5040.241912.0940.367032.13%
+ TokenWeave12.0960.312914.6490.429732.13%

All ported baselines improve warped consistency, supporting the reviewer's point that they are relevant, but TokenWeave gives the largest gain on every metric at the lowest budget. We attribute this to camera pose, a signal world models provide and generic video KV compression ignores: pose-guided association merges observations of the same scene region across distant times, whereas the baselines rank or prune tokens by importance or redundancy within a limited temporal neighborhood. We also added the missing Deep Forcing citation.

3. Additional result

On the new SolarWM-H3 backbone, TokenWeave improves all four metrics at all durations (wPSNR +5.12 / +5.78 / +5.86 dB) and is the best entry in every column of Table 1, including against memory-trained Infinite-World and HY-WorldPlay.

We hope these comparisons address the reviewer's main concern.

AuthorsOpenReview ↗

A Brief Follow-up

Dear Reviewer,

For your convenience, we would like to briefly summarize our response to your main concern about training-free KV-cache baselines.

We clarified the existing WorldKV comparison in Table 1 and added adaptations of Forcing-KV, Deep Forcing, and FlowCache to the same LingBot-World-v1 backbone. TokenWeave achieves the best results across all four metrics at an equal or smaller KV budget, including a 1.50 dB average wPSNR advantage over the strongest baseline, Forcing-KV.

We also validated TokenWeave on SolarWM-H3, with gains across all metrics and durations, and added analyses such as slot associations, attention fidelity, and memory growth in response to reviewer feedback (App. C.2–C.4).

As the discussion period is about to close, we would appreciate your thoughts on whether these comparisons address your concern and welcome any further questions. Thank you again for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.