- Rating
- 3(+1)
- Confidence
- 4(+2)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TokenWeave proposes a training-free memory mechanism for autoregressive video world models. It uses camera geometry and native-key similarity to associate historical patch tokens with persistent memory slots, then updates these slots through positional alignment and weighted averaging. A count-based correction adjusts their contribution to attention. Results on two LingBot-World backbones show improved revisit consistency and lower cache memory than full-history attention.
Critical strengths
- The approach is practical. It works directly with the frozen backbone’s KV states and requires no memory-specific training. Keeping the recent window unchanged is also a sensible choice for preserving short-term temporal behavior;
- The paired comparisons with the original backbones are a strong part of the paper. Both LingBot-World versions improve across the three rollout durations, with the checkpoint, input, trajectory, and sampling settings held fixed within each comparison;
- Tab. 2 provides useful evidence for the compression benefit. TokenWeave achieves similar warped consistency to full-history attention while using substantially less cache memory.
Critical weaknesses
- I think key similarity is a reasonable starting point for matching observations, but the distinction between “the same scene region” and “similar-looking regions” needs more evidence. Removing RoPE addresses explicit positional encoding; it does not ensure that two similar keys refer to the same physical content. The local fallback can also admit candidates outside the pose-guided region. For example, two similar windows may have similar keys but should remain separate in memory. Showing the observations assigned to a few slots over an entire trajectory, including incorrect associations, would make the scene-centric interpretation more convincing;
- The paper already explains that the count correction is exact for identical keys and approximate otherwise. My concern is how this approximation behaves once the averaging weight reaches its cap. At that point, the slot increasingly reflects recent observations, while its count still includes the entire history. It is not obvious that the resulting slot preserves the contribution those historical observations would have made to attention. I think a direct comparison of attention outputs before and after consolidation, especially after many revisits, would help justify this choice. It would also clarify how much information is retained when the matched observations differ in viewpoint or appearance;
- Appendix B.3 states that pairs failing registration are excluded from the metric averages. I think the frequency of these failures is important for assessing revisit consistency. If a method changes the scene substantially, its revisit may fail to register and disappear from the average. It would help to report the valid-pair counts for every method and duration, alongside results on a common valid subset. The fitted homography introduces another uncertainty because it can absorb some geometric changes, as acknowledged in Appendix B.4. This matters particularly for Tab. 3, where EMR improves the warped metrics but slightly reduces the direct scores;
- Tab. 2 shows clear savings relative to full-history attention. However, at 60 seconds, TokenWeave still uses 77.76 GiB of cache memory, compared with 30.72 GiB for the original window. I would like to see an extended window or historical-token retrieval evaluated on the same backbone under the same cache budget. That would help establish the benefit of consolidation beyond simply retaining more history. Longer-running behavior also deserves attention: persistent slots have no eviction policy, and capacity is allocated to fit the evaluated rollout. Repeated loops and continued exploration would help show how slot count, memory usage, and throughput evolve in practice.
Questions for authors
- Can the authors show what individual memory slots accumulate over time, and how often observations from different scene regions are incorrectly merged?
- How closely does the consolidated cache reproduce the original attention outputs after the averaging weight reaches its cap?
- What are the registration success rates for each method and duration? Do the main conclusions hold on a common valid subset?
- How does TokenWeave compare with retrieval or an extended window under the same cache budget, and what happens when the slot capacity is reached?
Above all, if the author can solve all of my questions. I would like to consider increase my scores.
7 comments
Response to Reviewer Tqrd (Part 2/2)
Q3. Registration failures, valid-pair counts, homography
Correction to the appendix text. : Appendix B.3 in the submission described another protocol that we used during development, not the one used for the reported numbers. No reported result excludes any pair. The actual protocol (now in App. B.3/B.4) is as below:
- Registration succeeds → revisit frame chosen by Eq. B.3; warped scores use the fitted homography.
- Registration fails → fixed frame index on the return sweep from the commanded trajectory; no warp, so warped = direct scores.
- All 165 pairs per duration enter the averages for every method.
Registration success (out of 165 pairs; new Table B.2):
| Method | 25 s | 45 s | 60 s |
|---|---|---|---|
| LingBot-World-v2 | 79 | 45 | 50 |
| + TokenWeave | 140 | 132 | 122 |
| LingBot-World-v1 | 82 | 48 | 51 |
| + WorldKV | 81 | 63 | 50 |
| + TokenWeave | 134 | 109 | 104 |
| SolarWM-H3 | 91 | 69 | 61 |
| + TokenWeave | 142 | 135 | 132 |
| Infinite-World | 148 | 142 | 139 |
| HY-WorldPlay | 145 | 140 | 142 |
- Success rate is itself a consistency measure independent of warped scores: it falls with duration without memory (79 → 50) and TokenWeave raises it on every backbone (+72 / +53 / +71 pairs at 60 s).
- This protocol removes the selection-effect concern: a method that changes the scene fails registration and gets a poor unaligned score, rather than being dropped. It is also conservative for TokenWeave, since LingBot-World's larger camera-control error causes registration failures even when content is preserved.
- Homography. A single global 8-DoF planar transform, applied identically to every method. Gains do not depend on warping: on SolarWM-H3, direct PSNR improves by 4.46–4.96 dB. For Table 3, EMR changes direct PSNR by only −0.17 dB but improves wPSNR by +0.98 dB, indicating better scene content under similar camera error.
Q4. Matched cache budget; behavior at capacity
Matched budget (App. C.1, Table C.1; LingBot-World-v1, averaged over 25/45/60 s; full table in our response to Reviewer E9Dn**).**
| Method | wPSNR | wSSIM | KV budget ↓ |
|---|---|---|---|
| Backbone | 10.041 | 0.3058 | – |
| + Extended sliding window | 11.343 | 0.2919 | 35% |
| + Retrieval (WorldKV) | 10.439 | 0.2849 | 50% |
| + TokenWeave | 14.649 | 0.4297 | 32.13% |
The gain does not come from keeping more history. The baselines retain as much or more context at an equal or larger budget, yet gain far less; what differs is how TokenWeave organizes it, consolidating observations by scene content via pose-guided association. Three training-free KV-compression methods tell the same story (see Reviewer uY4Q).
Growth and capacity (App. C.4, Fig. 8). On four trajectories mixing motion with long stationary segments, memory grows only while new observations arrive and plateaus when the camera is stationary (+0.23% over the final hold of a 66 s run; no noticeable growth under micro-motion). Memory scales with unmatched observations, not elapsed time. Our experiments currently impose no KV Cache Memory budget; under a hard budget, oldest-first eviction is compatible with our design, and the budget will be reached much later than with a full temporal cache.
We hope these results address the reviewer's concerns and are happy to answer any follow-up.
Thanks for running the extra experiments and detailed responses. The cache comparisons and registration results helped clear up several of my core concerns, and I’ve decided to raise my score to 3.
I do have one follow-up question. If we look only at pairs where registration succeeds for both the backbone and TokenWeave, do we still see similar gains? It would be helpful to see the direct and warped scores for those pairs, given that failed pairs are evaluated without alignment.
Response to Reviewer Tqrd (Part 1/2)
We thank the reviewer for the detailed review. Short answers first, details below.
Q1. Same scene region vs. similar-looking regions; local fallback; incorrect merges
What slots accumulate (App. C.2, Fig. 6). We followed slots across many complete trajectories. Fig. 6 shows three examples: every observation merged into a slot lands on the same physical content, e.g. brickwork below a window (27 observations), a window joint (24), a fork in a tree trunk (64), despite large viewpoint changes. Slots also bridge long gaps: the window-joint slot absorbs observations from 5.38–13.88 s and again from 49.44–55.81 s, after the region had left the recent context for >35 s. This cross-visit association is exactly what a sliding window cannot provide and what TokenWeave is built for.
Incorrect merges are rare and benign. Across >200 manually inspected merge events, <2% were incorrect, and all of those fell into a neighboring slot with near-identical texture (adjacent patches on weakly textured walls), where a mistake barely changes the rendered content. Two separate similar objects (the reviewer's two windows) are usually kept apart: they project to different image locations, so one falls outside the other's capsule, and a merge also needs the RoPE-free similarity to exceed τ and be the highest among candidates (Eq. 4).
Local fallback. It is a safeguard for imperfect camera control, not a main source of associations: entering the candidate set does not imply a merge, only ~4.5% of accepted merges involve a fallback-only slot, and removing it changes results only marginally (Table C.2, benchmark subset):
Q2. Attention outputs after consolidation
We agree that direct evidence is needed and provide it at two levels.
(i) Attention outputs, with and without the cap (App. C.3, Fig. 7).
Fig. 7. Attention-output comparison of Full KV, uncapped EMR, and EMR with cap=16. The left and middle columns overlay each EMR variant with Full KV, while the right column enlarges a fixed range of output coordinates. Curves show the mean readout across four selected heads for the same query. All settings share queries and historical observations; the two EMR variants additionally share merge assignments and KV budgets. Cap=16 limits the cumulative weights used to update representative K/V, while both variants retain the true-count log(size) correction.
Why cap. Without a cap, the n-th observation gets weight 1/(n+1), so frequently observed slots freeze and cannot follow the evolving scene. Capping wj at 16 keeps each update weight ≥1/17, while the uncapped nj still sets the slot's attention mass. We acknowledge that a capped slot's K/V is then weighted toward recent observations. However, all merged observations passed the similarity test and depict the same region despite viewpoint changes (Fig. 6), so this reweighting barely moves the representative.
Result. EMR closely reproduces Full-KV attention outputs, and the cap barely changes them. Across the selected views (four-head average), uncapped and cap=16 EMR reach mean relative L2 errors of 7.18% and 7.18% against Full KV (cosine 0.9973 for both). Over all 144 captured blocks (all eight heads), the capped output differs from uncapped EMR by only 0.20% relative L2 (cosine 0.99999).
(ii) End-to-end comparison with exact attention (Table 2). At 45 s and 60 s, when many slots have accumulated long histories, TokenWeave matches or exceeds the full-history cache in warped scores (16.47 vs. 16.43 dB wPSNR at 45 s; 15.18 vs. 15.16 dB wPSNR and 0.4880 vs. 0.4649 wSSIM at 60 s) while using 60.5–68.2% less KV memory. Removing the count correction lowers wPSNR by 0.98 dB (Table 3).