acceptodds
Under review as a conference paper at ICLR 2027

WorldRetain: Predictive Retention and Spatial Compression for Efficient and Consistent Video World Model Inference

15 views

Abstract

Autoregressive video world models must preserve scene identity across viewpoint revisits while limiting the cost of historical context. Sliding-window inference can discard evidence needed for later returns, and without an external memory bank, this eviction is irreversible. We propose WorldRetain, an inference-time cache policy that combines predictive KV retention with spatial compression. A lightweight predictor learns from future attention and viewpoint reuse to rank historical chunks, using only observed camera geometry and resident cache features at inference. Selected history is compressed into salient tokens and pooled summaries, maintaining a compact native cache without an external bank or changes to generator weights. On 100 RealCam-Vid camera-loop trajectories per backbone, WorldRetain achieves the best scene-identity and appearance consistency among the evaluated methods on LingBot-World-Fast and Matrix-Game 2.0. Compared with sliding-window inference, same-place scores increase from 1.77 to 2.29 and from 1.06 to 2.71, respectively, alongside 1.37–1.43× throughput speedups and lower peak GPU memory

Primary area:
generative models
Keywords:
Video World Models, Autoregressive Video Generation, KV Cache Compression

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ubPCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies KV cache management for camera-controlled video world models. It uses a small learned scorer to retain historical blocks, and then compresses the history through salient tokens and pooled summaries. The scorer is trained offline with future attention and poses, while only observed information is used online and the generator weights remain frozen. The authors evaluate revisit consistency, speed, and GPU memory on LingBot-World-Fast and Matrix-Game 2.0 using camera loop trajectories.

Critical strengths

On LingBot, same-place increases from 1.77 to 2.29, and throughput improves by about 37%. On Matrix, consistency, speed, and GPU memory are also improved together. The same-place gains over stronger baselines such as WorldKV and Relax Forcing are also supported by per-clip statistical analysis.

The method only trains a small scorer, without fine-tuning the generator or relying on an external history bank. Packed summaries and gists are both counted in the cache budget, and the inference cost also includes the scoring and compression overhead.

Critical weaknesses

The No packing variant keeps the full spatial tokens. Appendix B also says that the three variants use the same archive members and gist rules, changing only the L1 representation, but their average KV counts are all 15.1k. The paper does not explain clearly how different representations give the same token count, so the compression gains in this comparison are still not easy to interpret.

The revisit-pair selection is also not fully clear. The initial context is always retained, but the paper does not say how many evaluated returns concern other previously visited views. This makes the evidence for retaining non-initial history harder to judge.

Questions for authors

In Table 2(b), the compression methods are different, but the average KV counts are all 15.1k. How is this number counted?

How are the eight revisit pairs per clip selected? Do they also include views first visited after the protected initial frames?

Appendix E gives r and k_min for LingBot. What values are used for Matrix-Game 2.0 in the archive-capacity rule?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ubPC

We sincerely thank you for your careful reading of our paper and your constructive comments. We have conducted additional experiments and checked the relevant experimental settings and evaluation details. We hope that the results and clarifications below address your questions.

W1 / Q1: KV counts in Table 2(b)

Thank you for pointing out the issue with the KV counts. KV denotes the mean number of tokens actually resident in the cache during evaluation. It includes protected initial and recent context, selected original tokens, pooled summaries, and coarse gists. Full-resolution chunks contribute all spatial tokens; packed chunks contribute selected tokens plus summaries; gist-only chunks contribute their gist tokens.

The revised Table 2(b) reports the 60-clip L1-only controls below, also presented in Table 6, Appendix F.3. Retention settings are fixed and L2 gists are disabled; the packed variants retain L1 summaries. KV counts are measured from the actual resident cache.

RepresentationSP ↑PSNR (dB) ↑LPIPS ↓FID ↓FPS ↑Mean KV (k) ↓Peak GB ↓
Full tokens (no packing)1.7710.340.68039.21.8021.541.2
Uniform + summaries1.8510.450.67737.81.8317.540.0
Saliency + summaries1.8710.410.67638.51.8317.540.0

Both packed variants use fewer resident KV tokens than the full-token baseline. Table 2(b) separately lists the 100-clip full-method reference with L1 packing and L2 gists: SP 2.29 and mean resident KV 15.1k tokens.

W2 / Q2: Revisit-pair selection and non-initial history

For each clip, we first identify all candidate pairs with a pose distance of at most 0.15 and a separation of at least 30 decoded RGB frames. We uniformly select eight pairs from the candidates ordered along the timeline, without filtering based on the protected initial context, and use the same frame pairs for all methods. Across the 100-clip LingBot evaluation, 741 of the 800 selected pairs (92.625%) have their first-visit frame after the initial chunk, which covers RGB frames 0–8. Separately, 592 of the 800 pairs (74%) have a revisit pose whose distance from the initial pose exceeds 0.15.

The sampling procedure and these statistics are provided in Appendix A. We additionally evaluate two trajectory types targeting non-initial history (Table 5(a,b), Appendix F.1). Double lookback contains two departure-and-return cycles to intermediate viewpoints first observed after the initial conditioning frames and uses the original frozen predictor. Long departure-and-return uses a trajectory-specific predictor checkpoint, with attention settings of 40k tokens for Window and 47k for WorldRetain. SP is evaluated with Qwen3-VL-32B.

TrajectoryMethodSP ↑PSNR (dB) ↑LPIPS ↓Mean KV (k) ↓FPS ↑
Double lookbackWindow2.17215.3860.37626.181.55
Double lookbackWorldRetain2.56314.4660.42314.851.83
Long departure-and-returnWindow2.23415.2810.37536.071.44
Long departure-and-returnWorldRetain2.50016.3200.34314.671.88

WorldRetain achieves higher SP with fewer resident tokens and faster generation in both configurations. Figure 6 shows the intermediate reference viewpoint and its later revisit.

Q3: Matrix-Game 2.0 Configuration

Both backbones initially select half of the candidate history, rounded up, with a preliminary minimum of two historical chunks when available. Matrix-Game 2.0 protects two initial frames and the most recent chunk. The selected history then undergoes spatial compression and a final check against the 47,000-token ceiling used by both backbones. KV is measured from the resulting resident cache. These settings are specified in Appendix E.

Thank you again for your feedback. We would appreciate it if you could let us know whether our response has addressed your concerns. If there are any remaining questions or concerns, we would be happy to discuss them further and provide additional clarification.

Thanks for the clarifications and additional experiments. The revised ablation resolves my question about KV counts, and the new results show that the method can help on revisits to non-initial views. I will keep my rating.

Reviewer_s7fROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents WorldRetain, an inference time KV caching policy for autoregressive video world models. WorldRetain introduces a MLP to compute retention scores that decide which historical chunks to keep for consistent revisits. Each retained chunk keeps its high-saliency tokens, and the remaining tokens are pooled into summary tokens.

Critical strengths
  1. Bounded native cache with measured efficiency. The method keeps history inside the native KV cache without an external memory bank, resulting in improved consistency and speed.
  2. Applicability across backbones. The same retention algorithm, predictor architecture and compression procedure are applied to two frozen generators that differ in scale and native cache configuration.
Critical weaknesses
  1. Possible reliance on a trajectory prior. In an interactive world model, the camera trajectory is not fixed in advance but depends on user controls that the predictor never observes. The training rollouts appear to follow the same RealCam-Vid camera-loop protocol as the test set, so training and evaluation share the same loop structure. The gains may then come from a learned “the camera comes back” prior rather than a transferable estimate of future usefulness, and may not generalize to other trajectories. The authors should evaluate the same predictor on trajectories that do not loop back, such as forward-only paths, random-walk paths, or clips from another dataset.
  2. Irreversible retention without failure analysis. Because dropped chunks cannot be recovered, errors in the predictor’s ranking may accumulate over long-horizon generation. Could the authors show or analyze cases where a chunk needed at a later revisit has already been dropped?
  3. Unexplained gain from spatial compression. In Table 2(b), Proposed compresses each archived chunk while No packing keeps full chunks, yet the compressed variant achieves better performance. Could the authors analyze why fewer tokens lead to better consistency? Could they also add variants such as key-similarity pruning (as in WorldKV) or saliency-based selection without summary tokens?
  4. Memory may be limited to the sink. All qualitative examples return to the initial frame, which stays in the protected sink, and the supplementary videos suggest that places outside the sink are poorly preserved. Could the authors evaluate trajectories where the camera revisits a place other than its starting view? For example, the camera could turn right to observe a new place, move on to another location, and then return to the previously visited place on the right. This would show whether the gains come from the retention policy itself rather than from the sink.
Questions for authors
  1. Training trajectories. How were the 48 training trajectories sampled or constructed, and how do they differ from the evaluation loops?
  2. Saliency definition. Figure 2(b) keeps Frame 1 at full resolution and shows saliency only for Frames 2 and 3, while Eq. (8) defines saliency between adjacent chunks. Could the authors clarify whether saliency is computed between adjacent chunks or between frames within a chunk, and whether the first frame of each archived chunk is kept at full resolution?
  3. Unusually low PSNR. In Table 1, PSNR on LingBot ranges from 10 to 11 dB for all methods, a level typical of unrelated frames. This raises concern about whether the metric is measured properly. Could the authors explain these values?
  4. Human evaluation. Revisit consistency is judged only by VLMs and by pixel and distribution metrics. Could the authors conduct a human preference study to confirm that these results agree with human judgment?
6 comments
AuthorsOpenReview ↗

Response to Reviewer s7fR

We sincerely thank you for your careful reading and constructive comments. We have conducted supplementary experiments and checked evaluation details. We hope these results address your questions.

Weaknesses

W1: Trajectory prior

On eight open-world trajectories with LingBot-World, WorldRetain uses the original frozen predictor; both methods retain their respective main-experiment generation settings and cache budgets. Qwen3-VL-32B finds the same overall visual quality in both presentation orders (Appendix F.6). Double lookback contains two departure-and-return cycles to previously observed, non-initial viewpoints. With the same frozen predictor, WorldRetain achieves SP 2.563 versus Window's 2.172, mean resident KV 14.85k versus 26.18k tokens, and 1.83 versus 1.55 FPS (Table 5(a), Appendix F.1). These tests show comparable open-world visual quality and higher SP and speed on repeated revisits.

W2: Archive exclusion

Excluding an older archived segment near the reference viewpoint and replacing it with the next-ranked candidate lowers SP from 3 to 2 in a single pair separated by 414 RGB frames. The runs share settings and seed (Appendix F.2). This case illustrates the role of retained history in later revisits.

W3: Spatial compression

The 60-clip L1-only controls fix retention settings and disable L2 gists; packed variants retain L1 summaries. SP and efficiency results are summarized below (Table 2(b), Table 6, Appendix F.3).

RepresentationSP ↑Mean KV (k) ↓FPS ↑
Full tokens1.7721.51.80
Uniform + summaries1.8517.51.83
Saliency + summaries1.8717.51.83

Saliency + summaries raises SP with 18.6% fewer KV tokens than full tokens. The separate 100-clip full-method reference uses L1 + L2: SP 2.29, mean KV 15.1k. On eight diagnostic clips selected for observed saliency-over-full SP gains, SP is 2.11 with summaries, versus 1.66 without summaries and 1.47 for our summary-free key-similarity/novelty control (Table 7, Appendix F.3). These controls support combining spatial detail with pooled context.

W4: Non-initial revisits

Pairs satisfy pose distance ≤0.15 and separation ≥30 RGB frames; eight are uniformly selected in time order without filtering by protected initial context. All methods share the pairs. Of 800 pairs across 100 LingBot clips, 741 have first visits after the initial chunk (RGB 0–8); separately, 592 return to poses more than 0.15 from the initial pose (Appendix A).

The long departure-and-return test uses a trajectory-specific predictor checkpoint and attention settings of 40k tokens for Window and 47k for WorldRetain. WorldRetain achieves SP 2.500, higher than Window's 2.234, with less resident context and faster generation (Table 5(b), Appendix F.1; Figure 6).

Questions

Q1: Training trajectories

Each backbone's predictor trains on full-history rollouts from 48 trajectories disjoint from evaluation. Future attention and poses supervise offline training; online inputs are observed camera and cache features. Both backbones preselect half the eligible history, rounded up (at least two chunks when available), with a shared 47,000-token ceiling (Appendix E).

Q2: Saliency

Eq. (8) pools channel-averaged absolute residuals between corresponding latent frames of adjacent denoised chunks over 2×2 spatial regions. All three archive frames, including the first, undergo token selection and pooling; only protected initial and recent context stays full-resolution. L1 has at most 64 summary tokens per chunk; L2 gists are summary-only (Figure 2, Appendix F.3).

Q3: PSNR

PSNR measures pixel-wise agreement and is sensitive to camera-pose differences between approximately matched revisit frames. Viewpoint changes can shift corresponding pixels even when scene layout and landmarks remain consistent. PSNR may therefore be less suitable as a standalone metric for coarse-grained revisit evaluation. We interpret it alongside Same-Place and perceptual metrics (Appendix F.5).

Q4: Human evaluation

Ten raters evaluate sixteen items (eight per backbone) with anonymized method labels. They follow the Same-Place VLM criteria and 0–3 rubric: matching first-visit and revisit layout and landmarks while ignoring mild lighting or compression differences (Table 8, Appendix F.4). Each method receives 80 ratings per backbone.

BackboneWorldRetainWindowRelax Forcing
LingBot1.8001.3881.563
Matrix-Game 2.02.8130.0631.775
Pooled2.3060.7251.669

WorldRetain ranks highest on both backbones. Agreement between human rankings and the main-paper Same-Place rankings supports the reliability of our revisit-consistency evaluation.

Reviewer_s7fROpenReview ↗

Remaining concerns

Thank you for the response and the additional experiments. The following concerns remain, and the first is the major one.

  1. W1: Generalization to new trajectories (major concern).

    • Trajectory-specific checkpoint. Table 5 uses a “trajectory-specific predictor checkpoint”. How does it differ from the original predictor, how was it trained, and what does the original frozen predictor score on this protocol? Needing a new checkpoint for a new trajectory type strengthens my initial concern that WorldRetain does not generalize beyond the trajectories it was trained on.

    • Non-returning trajectories. The predictor is still not tested on trajectories that do not loop back, such as forward-only or random-walk paths.

    • Figure 6. The revisited view B shows the same castle as the initial frame A, so the protected sink could explain the result. Please show cases where B is clearly different from A, with no shared content.

    • Selective bolding. In Table 5, only the metrics where WorldRetain wins are bolded, and the response does not mention these losses.

  2. Q1: Training trajectories. The response and Appendix E say only that the 48 training trajectories are disjoint from evaluation. Please state which dataset they come from, how they were constructed, and how they differ from the evaluation loops. If they follow the same camera-loop protocol as the test set, the predictor may simply be fitting the training trajectories. Since the predictor is the paper’s core contribution, every table would then be in question.

  3. Q3: PSNR. 
If the low PSNR comes from pixel misalignment between imperfectly matched camera poses, please evaluate over a small temporal window around the revisit frame and use the frame whose camera pose best matches the first visit. This would show whether misalignment is really the cause.

  4. Q4: Human evaluation. The study is too small to support the claim, with only ten raters on eight items per backbone. WorldKV, the closest baseline on LingBot, is also not included. In its current form it should not be presented as evidence, so please expand it or remove it.

Overall. My critical concern is unchanged. The trajectory predictor may be overfit to the loop structure of its training set, and how the training trajectories relate to the evaluation trajectories remains unexplained. Until this is clarified, it is difficult to assess the core contribution of the paper.

AuthorsOpenReview ↗

Response to Remaining Concerns

Thank you for your thoughtful follow-up. We appreciate the opportunity to discuss these points further and share the additional results and clarifications below.

W1 / Overall: Generalization to new trajectories

Table 5(b) in our earlier response used a trajectory-specific predictor trained on an ABCB pool from 48 RealCam-Vid clips, with the main-experiment predictor's data-preparation procedure and an ABCB revisit schedule. We then retested ABCB and double lookback with the main-experiment predictor kept frozen. Test sets and settings were unchanged.

ABCB. Across eight clips, the camera moves from the starting view A through B to C, then looks back toward B.

MethodSame-Place ↑PSNR (dB) ↑LPIPS ↓FID ↓Mean KV (k) ↓Peak memory (GB) ↓
Window2.23415.2810.37527.0136.154.6
WorldRetain (trajectory-adapted)2.50016.3200.34325.0014.739.1
WorldRetain (main-experiment predictor)2.53116.8750.33223.8414.439.1

The frozen predictor improves all four consistency metrics over Window, with lower resident KV and peak memory.

Double lookback. Across eight scenes, the camera moves forward with two brief look-backs to previously observed, non-initial viewpoints.

MethodSame-Place ↑PSNR (dB) ↑LPIPS ↓FID ↓Mean KV (k) ↓Peak memory (GB) ↓FPS ↑
Window2.17215.3860.37636.1026.253.61.55
WorldRetain (main-experiment predictor)2.56314.4660.42330.3214.938.11.83

Window has higher PSNR and lower LPIPS; WorldRetain has higher Same-Place and FPS, with lower FID, resident KV, and peak memory.

Both protocols differ from the main experiment's return-to-start loops. These results provide partial evidence of predictor transfer beyond a single closed-loop structure. We will report the main-experiment predictor as primary and the adapted predictor as an additional comparison.

Open-world trajectories. Appendix F.6 evaluates the same frozen predictor on eight open-world trajectories without planned revisits, using each method's main-experiment generation conditions and cache budgets. Qwen3-VL-32B rates the overall visual quality as comparable in both presentation orders. This tests open-world visual quality, while the look-back protocols test revisit consistency.

Figure 6. We have added an Urban street example to Figure 6 (top). The initial view A shows a warehouse façade, whereas reference view B shows a street with a distinctive transmission tower that is not visible in A. On revisiting B, WorldRetain better preserves the reference-view landmarks and layout than Window. This case illustrates preservation of intermediate-view landmarks that are not visible in the initial view.

Appendix table annotations. We have also corrected the boldface annotations in the appendix tables to mark the best value in each metric consistently, including ties and values favoring Window.

Q1: Sampling and construction of training trajectories

The original predictor is trained on 48 RealCam-Vid clips: 26 from RE10K, 11 from DL3DV, and 11 from Mira. They are sampled with a fixed seed outside the evaluation pools, with clip IDs disjoint from the 60- and 100-clip loop evaluation sets. Poses are derived from annotated RealCam-Vid camera paths by replay or interpolation. Each clip is expanded to 481 frames using a multi-pass revisit schedule, and dense teacher rollouts provide offline future-use labels for ranking training.

Q3: PSNR and temporal pose matching

PSNR measures pixel-wise agreement between generated first-visit and revisit frames. All methods use the same pose-matched pairs, with a pose-distance threshold of 0.15 and a separation of at least 30 RGB frames.

On the main loop set, residual pose distances are approximately 0.006–0.01. Following your suggestion, we searched within ±5 and ±15 frames around each revisit and selected the frame whose pose best matches the first visit. Mean PSNR changes by less than 0.05 dB under this refinement, with main-loop values remaining around 10–11 dB.

The main loops measure longer-horizon returns after a full tour, whereas the look-back protocols above evaluate shorter-horizon revisits and yield PSNR values of approximately 14.5–16.9 dB. We will describe the pairing procedure and revisit horizon alongside each protocol's PSNR values and interpret them together with Same-Place and perceptual metrics.

Q4: Human evaluation

Thank you for the suggestions concerning study size and baseline coverage. We will describe the current sixteen-item study as a small-scale human assessment and plan an expanded study with at least 50 volunteers in a later version. The expanded human evaluation will include results from all strong baselines, including WorldKV.

Reviewer_PsbwOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes WorldRetain, an inference-time cache policy that combines predictive KV retention with spatial compression. A small MLP predicts future-use scores from 11 features, including camera geometry, revisit history, and KV summaries. It improves revisit consistency while reducing KV usage and GPU memory and increasing throughput.

Critical strengths
  • Very strong results. WorldRetain achieves huge improvements over several recent baselines on LingBot and Matrix-Game. This is useful because cache compression usually trades memory and speed for long-term consistency. The gain is robust across two different world model backbones.
  • Elegant solution. The paper decomposes the problem in a very sophisticated way. WorldRetain can control importance and representation cost separately. The ablation study supports both parts.
Critical weaknesses
  • The evaluation is somewhat biased, because the experiments use datasets with camera-loop trajectories, which are not common in ordinary datasets or natural scenes. This is not data leakage, but it uses the same revisit prior that the method exploits, so the performance may be overestimated. Open-world rollouts, noisy poses, and other interventions are needed, as the authors admit.
  • Future coverage is only a proxy for actual memory utility and does not estimate the marginal generation loss caused by removing a chunk. An intervention-based analysis is needed to strengthen the results.
Questions for authors
  • Is there any evidence that the learned retention policy also works on open-world trajectories with fewer or unpredictable revisits?
  • Have you evaluated whether the predicted ranking correlates with the causal effect of removing chunks?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Psbw

We sincerely thank you for your careful reading of our paper and your constructive comments. We have conducted additional experiments and checked the relevant experimental settings and evaluation details. We hope that the results and clarifications below address your questions.

W1 / Q1: Trajectory prior and open-world evaluation

Appendix F.6 evaluates WorldRetain on eight open-world trajectories using the LingBot-World backbone. WorldRetain uses the original frozen retention predictor, and both methods follow their respective main-experiment generation conditions and cache budgets. Qwen3-VL-32B rates both methods as having the same overall visual quality on all eight trajectories in both presentation orders. We also evaluate the same frozen predictor on double-lookback trajectories with two departure-and-return cycles to intermediate viewpoints first observed after the initial conditioning frames. The results from Table 5(a), Appendix F.1, are reproduced below. SP is evaluated with Qwen3-VL-32B; KV denotes mean resident cache tokens.

MethodSP ↑PSNR (dB) ↑LPIPS ↓Mean KV (k) ↓FPS ↑
Window2.17215.3860.37626.181.55
WorldRetain2.56314.4660.42314.851.83

WorldRetain achieves a higher SP score with fewer resident tokens and faster generation on double-lookback trajectories. Together, these tests show that WorldRetain maintains visual quality comparable to Window during open-world exploration and improves same-place consistency and generation speed on trajectories with multiple revisits.

W2 / Q2: Future Coverage and chunk-removal effects

Future Coverage combines future attention and geometric reuse to construct offline labels for anticipated-use ranking. Online retention computes scores from observed camera and resident-cache features.

Appendix F.2 reports a delayed-revisit intervention. An older archived history segment near the reference viewpoint is excluded, and the next-ranked segment takes its place. Both runs use the same configuration, a 28,000-token attention setting, and random seed 42. For the evaluated revisit pair, separated by 414 RGB frames, WorldRetain achieves a same-place score of 3 with the history segment retained, higher than 2 when it is excluded and replaced. This case illustrates the change in later revisit consistency associated with the retained-history choice.

Thank you again for your feedback. We would appreciate it if you could let us know whether our response has addressed your concerns. If there are any remaining questions or concerns, we would be happy to discuss them further and provide additional clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.