acceptodds
Under review as a conference paper at ICLR 2027

Keepsake: Selective Spatial Memory for Long-Horizon Video Generation

Abstract

Long-horizon camera-controlled video generation relies on persistent memory to maintain scene consistency beyond the model's immediate temporal context. Existing systems follow two strategies to achieve this consistency. Full-history approaches retain all generated observations, causing unbounded storage and retrieval costs to grow while allowing low-fidelity memories to persist. Selective-construction approaches reduce redundancy, but make one-time retention decisions that are never revisited, even as an observation’s value changes with the evolving memory bank. Both strategies leave a shared question unresolved: as the generated history evolves, which stored observations should still remain in memory? Our key insight is that the value of a stored observation is not fixed, but relational: it depends on the alternatives currently available in the memory bank. A view supported by many geometrically and visually similar substitutes can be relinquished with little loss of coverage, whereas an observation with few viable alternatives should remain regardless of age. We introduce Keepsake, an online, training-free controller for fixed-capacity spatial memory. At each update, Keepsake constructs a pose–appearance graph over retained and newly generated observations, combining camera-pose proximity with visual similarity. A retention priority jointly captures the number of strong substitutes and the similarity of the closest alternative, allowing Keepsake to continually reassess memory value, preserve observations with little alternative support, and evict highly replaceable ones under a fixed budget. The controller modifies only the persistent-memory update; the host generator, denoising schedule, and retrieval rule remain unchanged. Across MemCam and WorldMem, Keepsake improves FVD and LPIPS under a fixed memory budget; on 180-second MemCam trajectories, it retains only 32 of 5,397 frames while reducing FVD by 35.1%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Generation, Memory, World Model, Long Horizon Video Generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_g2FtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method to select which frames should be stored in the memory bank for autoregressive video generation. After the base method retrieves relevant candidate observations, the proposed method update the memory bank by filtering out the frames which are more redundant given the current memory bank and new candidate observations. The redundancy is decided based on the view coverage, which is approximated by the closeness in viewing position and direction and visual resemblance. The results show that it can improve long-horizon generation quality by only maintaining a small memory bank.

Critical strengths
  1. The idea is well-motivated, and the paper is clearly written.
  2. The results show that the proposed method can improve the baselines in terms of the long horizon generation quality and memory footprint.
Critical weaknesses

I don't have major concerns over the paper, but I want to know how efficient the proposed method is. Since this method requires an extra step after the memory bank is retrieved, then how much overhead does it create? Is it more straightforward to move this view-diversity design to the retrieval stage?

Questions for authors
  1. Can the authors compare the proposed method with [1]? Since [1] is focus on retrieving the missing information, does it mean if the retrieval process itself is coverage-aware, a separate selection process after retrieval is not necessary?
  2. Can the authors report the computation overhead like extra time over the baselines?

[1] Joo, Minseok, et al. "Retrieve What's Missing: Coverage-Maximizing Retrieval for Consistent Long Video Generation."

4 comments
AuthorsOpenReview ↗

Overhead And Comparison With CovRAG

Thank you for the positive assessment, and for the kind words about the motivation and clarity of the paper. Both of your questions get at how KEEPSAKE fits into a full memory pipeline, and answering them helped us sharpen that part of the paper.

Q1. Comparison with coverage-aware retrieval (CovRAG)

Thank you for pointing us to CovRAG; it is closely related, and whether coverage-aware retrieval makes retention unnecessary is exactly the right question to ask. One clarification first: KEEPSAKE does not run after retrieval. It decides which frames stay in the memory bank, once per memory update; the host's retrieval rule then reads from that bank unchanged.

This is why the two are complementary rather than alternatives. CovRAG decides what to read for each new view, but the archive it reads from still keeps every generated frame: storage keeps growing, every lookup searches the growing archive, and drifted frames remain retrieval candidates. KEEPSAKE decides what to keep, which addresses all three, and a coverage-aware retriever can run directly on its bounded bank with only 32 candidates to score. Its code is not yet public, so we could not run the combination, but we will discuss it in the revised related work.

Q2. Computation overhead

KEEPSAKE's own cost is small. It runs once per memory update, building the substitution graph over the bank and new frames, scoring them, and releasing evicted entries; on every host it takes under 1% of generation time. Its net effect is a saving, because every later lookup searches 32 frames instead of the full, growing history:

HostKEEPSAKE update costRetrievalEnd-to-end time per video
MemCam (60s, 15 traj.)1,524 ms/update (0.93% of rollout time)1344.61 → 43.77 ms/query (30.7× faster)106.5 → 65.5 min (−38.5%)
WorldMem (60s, 15 traj.)1.58 ms/update (0.135% of generation)77.92 → 4.54 ms/query (17.2× faster)12.3 → 11.9 min (−3.7%)
VMem (60s, 5 traj.)12.29 ms/update (0.041% of pipeline time)14,903.45 → 5,354.94 ms/call (2.78× faster)8.4–14.1 h → 1.5–1.8 h (5.1–8.0× faster)

The size of the end-to-end saving depends on how much of each host's runtime scales with memory. On WorldMem, generation dominates and retrieval was already cheap, so the gain is small. On VMem, 3D reconstruction scales with memory size, so a one-minute video drops from over 8 hours to under 2; stored frame data also falls by 96% and peak GPU allocation by about 70%.

MemCam and WorldMem timings use the same setup as the paper (Appendix A.2): one NVIDIA H100 PCIe GPU, 16 CPUs, and 64 GiB of host memory per generation job. VMem timings are recorded on one NVIDIA RTX PRO 6000 Blackwell GPU.

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards,

Area Chair

Reviewer_g2FtOpenReview ↗

Reply to the rebuttal

Thank the authors for providing the rebuttal. I will keep my score as Weak Accept since I don't have other major concerns. While I tend to believe that the overall scope and technical contribution remain moderate, I think the paper has its merits as a lightweight pipeline / component.

AuthorsOpenReview ↗

Thank you for the discussion

Dear Reviewer,

Thank you for reading our rebuttal and for your continued support. We appreciate your questions on overhead and coverage-aware retrieval; they helped us clarify how KEEPSAKE fits into a full memory pipeline, and we will include both discussions in the revised paper. We are glad the lightweight, plug-in nature of the method came through, as it is central to how we hope it will be used.

Thank you again for your time and constructive feedback.

Reviewer_ZQVpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies persistent spatial memory for camera-controlled long-horizon video generation. Existing systems either keep the full history (unbounded cost, drifted memories persist) or make one-shot selective write decisions that are never revisited. The authors argue that an observation’s value is relational: it depends on currently available geometric/visual substitutes. The host generator and retrieval rule stay unchanged. On MemCam and WorldMem, KEEPSAKE improves FVD/LPIPS over full retention and matched-budget FIFO/K-center/MCE; on 180s MemCam it keeps 32 of 5,397 frames and cuts FVD by 35.1%, with large retrieval/runtime savings.

Critical strengths
  1. The problem framing is sharp and useful: not only “how to retrieve,” but “which memories should still remain as the bank evolves”.
  2. The method is simple, online, and plug-and-play—easy to drop into existing memory-augmented generators without retraining.
  3. Empirical gains are convincing under matched generators/seeds: vs. full retention and vs. budget-matched FIFO/K-center/MCE on both MemCam and WorldMem; large FVD drop on 180s rollouts.
  4. Efficiency claims are concrete.
Critical weaknesses
  1. Technical novelty is moderate. The core is a redundancy score (degree + max affinity) for eviction. Relative to K-center/MCE, the MemCam 60s gap is small (FVD 690.5 vs. K-center 712.0 in Table 1). The contribution is a well-motivated policy + analysis more than a deep new learning principle.

  2. Mechanism interpretation is not fully locked down. Budget sweeps often favor smaller B (Table 3), and Appendix Table 9 shows the fidelity edge shrinks toward zero under FOV-matched restrictions. A large part of the gain may be “evict drifted junk,” not carefully preserving irreplaceable coverage. The higher view-mismatch / lower corruption tradeoff (Table 7) needs clearer causal analysis.

  3. Evaluation scope is still narrow (moderate). Fifteen matched trajectories per suite; WorldMem is Minecraft at 128×128 and capped at 60s; 180s only on MemCam. Broader hosts (e.g., other spatial-memory generators cited in Sec. 2) would strengthen generality.

  4. Baseline coverage outside custom policies is limited. Related selective/adaptive memory works (MemFlow, Surprise Forcing, etc.) are discussed but not compared head-to-head under the same host interface.

  5. Within-update eviction is one-shot ranked, not iteratively recomputed. Priorities are frozen during removal. This is fine for speed, but leaves open whether iterative re-scoring would matter.

Questions for authors

Please see weaknesses.

6 comments
AuthorsOpenReview ↗

On Novelty and Gap With K-Center

Thank you for the detailed and thoughtful review, and for recognizing the problem framing and the plug-and-play design.

W1. Novelty and the gap to K-center

As you note, the core of the paper is the question of which memories should still remain as the bank evolves. We see two contributions behind this framing.

1. Unbounded memory can hurt, and we fix this at the level of retrieval memory. Memory-augmented world models typically keep every generated frame, assuming more history can only help. We show this assumption fails: keeping everything lets drifted frames accumulate and degrade later generation, and bounding memory can either help or hurt depending on the retention criterion. A matched-budget FIFO is worse than keeping everything on both hosts, while KEEPSAKE beats full retention at a fraction of the cost. Prior work bounds memory at other levels, such as evicting or compressing KV caches, sparse attention over cached tokens, or learning compressed memory inside a trained model. To our knowledge, KEEPSAKE is the first to continually re-evaluate which stored frames remain available in explicit frame-retrieval memory, rather than deciding once at admission.

2. Plug in, plug out. KEEPSAKE is training-free and modifies only the memory update; the generator, denoising schedule, and retrieval rule are unchanged. It can be added to or removed from an existing memory-augmented generator without retraining, which is why we could apply it to three different hosts, including VMem's 3D-reconstruction memory. The scoring rule is deliberately simple to keep this property.

On the gap to K-center, we extended both hosts from 15 to 50 trajectories. At this scale the gap is clearer: on MemCam, KEEPSAKE lowers FVD by 8.9% relative to K-center (up from about 3% on 15 trajectories) and lowers LPIPS significantly (paired 95% interval [-0.0170, -0.0074]); on WorldMem it lowers FVD by 18.7%, with K-center and KEEPSAKE nearly tied on rFID and VBench.

MethodMemCam FVD ↓MemCam LPIPS ↓WorldMem FVD ↓WorldMem LPIPS ↓
Full retention344.40.58492643.90.6417
FIFO B32421.00.62203091.10.6852
MCE B32326.40.58731113.80.5764
K-center B32331.80.5856611.30.5370
KEEPSAKE B32302.20.5736496.90.5315

FVD is computed per set, so these 50-trajectory values are not comparable to Table 1. We will update Table 1 with these 50-trajectory results, and add the full WorldMem results (including rFID and VBench) to the appendix of the revised paper.

The difference from K-center is in what each rule protects. K-center maximizes spread: it keeps the frames farthest from the rest, whether or not they are useful. KEEPSAKE asks whether each frame can be replaced by what remains.

AuthorsOpenReview ↗

Mechanism, Broader Evaluation, More Baselines, One-shot Rank Update

W2. Mechanism

Both effects are real, and they show up in different measurements.

Drifted-frame removal, tested causally. To address the Table 7 trade-off directly, we took later chunks of the MemCam videos where the retrieved memory was heavily corrupted, replaced the retrieved generated frames with the ground-truth frames at the same indices, and regenerated the next chunk with the memory selection unchanged. All four cases improved (mean LPIPS -0.168, DINO distance -0.234). Corrupted memory content causally degrades later generation. Tables 3 and 9 measure per-frame fidelity, which largely reflects this effect: smaller banks hold fewer corrupted candidates.

However, Coverage, measured where it matters. Per-frame fidelity does not capture whether a revisited place looks the same as before, which is the purpose of memory. Figure 1 illustrates this goal, and Appendix E shows qualitative revisit examples where KEEPSAKE preserves the earlier view. To quantify it, we added a budget ablation measuring revisit consistency on 180-second MemCam videos: for each location the camera returns to (1,764 revisit clusters across 15 videos), the worst-pair pixel RMSE between visits, raw and relative to the change in ground truth.

BudgetSelf-consistency error ↓GT-adjusted change error ↓
B1667.0095.46
B3263.3491.67
B6457.8987.17
B12857.2187.21

Consistency improves as the bank grows and levels off around B64, the opposite of the per-frame trend in Table 3. A budget therefore trades corruption against coverage: shrinking memory removes drifted frames but eventually drops views needed for revisits. KEEPSAKE is built for this trade-off, dropping frames whose views are already covered and keeping those the bank cannot replace. The revised paper will present both effects explicitly.

W3. Evaluation scope

Trajectories. We extended both hosts from 15 to 50 trajectories, and the ranking holds on both (see W1).

More hosts. We added a third host, VMem, a spatial-memory generator from Sec. 2 that works on real scenes and builds 3D reconstructions from memory. We evaluated 7 matched comparisons across 3 scenes: 5 trajectories of about 60 seconds on one scene, and one 30-second trajectory each on two further scenes. On the first frame's trajectories, capping memory at 32 frames cut stored frame data by 96% (1,397.5 MiB to 57.0 MiB), peak GPU allocation by about 70%, and end-to-end generation time by 5.1 to 8.0×. The quality cost is small on average but depends on the trajectory: on the two additional scenes, every VBench-Long dimension stays within 0.01 of full retention; on the first scene, the mean drop is largest in aesthetic quality (0.575 to 0.554), with most of it coming from two of the five trajectories, while another improves on every dimension. On MemCam and WorldMem the budget improves quality as well as cost; on VMem, which reconstructs geometry from memory, the main benefit is efficiency.

We are extending the VMem evaluation to 5 scenes with 3 trajectories each (total 15 videos), including the K-center, FIFO and MCE baselines, reporting VBench-Long and efficiency metrics; these results will be included in the appendix of the final version.

WorldMem resolution and horizon. We use WorldMem at its released resolution and benchmark, since the controller does not change the host. Its own generation degrades beyond 60 seconds under its native configuration, so longer WorldMem rollouts would mostly measure the host's failure rather than the memory policy. On MemCam, which stays stable longer, we evaluate up to 180 seconds: there, KEEPSAKE keeps 32 of 5,397 frames while reducing FVD by 35.1%.

W5. One-shot ranking within an update

Thank you for this question. One-shot ranking is a deliberate choice to keep each update fast; iterative re-scoring is a natural variant, and we tested how much it changes the decisions. Priorities are recomputed at every memory update, but within a single update we rank once and remove the lowest-priority frames together. We replayed 360 memory updates across 15 trajectories with iterative re-scoring after each removal. Iterative re-scoring changed the evicted set in every update, on average altering about 12 of the 32 retained slots, so within-update re-scoring materially changes which frames are kept. (The replay is reconstructed from saved rollouts and approximates the original online descriptors, matching the logged one-shot decisions in 72% of updates.)

Different decisions do not by themselves imply better generation, and iterative re-scoring adds one rescoring step per removed frame to each memory update; it does not affect retrieval latency. We are measuring both generation quality and update time now, will post them here, and will add a discussion of this trade-off to the revised paper.

W4. Published baselines

Surprise Forcing. Surprise Forcing manages memory inside the attention cache of a streaming text-to-video model, while our hosts retrieve stored frames by camera geometry, so the full systems are not directly comparable. Its eviction rule, however, can be compared under one interface. Since its code is not yet public, we adapted its memory components (surprise scoring, adaptive admission, priority eviction, and content-based routing) to MemCam at the same budget (10 trajectories, 30s, B=32). It improved FVD over full retention (755.5 vs 880.6) but made LPIPS worse (0.5308 vs 0.5156). KEEPSAKE on the same trajectories is running for a direct comparison, and we will add a 50-video comparison to the main table alongside the existing baselines.

The key difference: Surprise Forcing scores a frame once, by how surprising it was when written, and a drifted frame also looks surprising. KEEPSAKE re-scores every stored frame at each update against what the rest of the bank covers, keeping views that are hard to replace and dropping those that are not.

MemFlow. MemFlow selects memory by relevance to the text prompt of each upcoming chunk, and its memory modules are learned during fine-tuning. Our camera-controlled hosts have no per-chunk prompts and are used frozen, so its selection rule has no input to operate on in this setting. We will state this distinction explicitly in Sec. 2 of the revised paper.

Reviewer_VcwcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free spatial memory management method. It constructs a graph based on camera poses and frame similarities, prioritizing the retention of historical frames that lack alternative views while removing redundant ones, thereby maintaining a fixed capacity for the memory bank. Evaluations on MemCam and WorldMem demonstrate that this approach significantly reduces storage and retrieval overhead while improving generation quality and scene consistency.

Critical strengths
  • The research addresses a highly practical issue; in long-video generation, historical memory tends to accumulate continuously, and managing it within a fixed capacity helps reduce long-term operational costs.
  • The method requires no training and does not modify the generator and it is effective across different models, demonstrating general applicability.
  • The experiments evaluate generation quality, memory capacity, and retrieval time. They also distinguish between information lost through deletion and information retained but not correctly retrieved. The analysis is detailed and informative.
Critical weaknesses
  • The experiments include a general retention strategy with a matching budget, but they lack comparisons with existing adaptive memory methods, such as "Surprise Forcing," which also employs training-free admission and priority-based eviction. Given the differences in memory representation and generative systems, the authors should clarify whether their eviction strategy can be adapted for such comparisons and highlight the specific advantages of their proposed design.
  • The paper does not separately verify the contributions of pose, appearance similarity, and the two redundant scoring terms; consequently, it remains unclear whether each design element is necessary, and the source of the full method's advantage over simpler alternatives is not known.
Questions for authors

please see the weakness

3 comments
AuthorsOpenReview ↗

Ablation and Surprise Forcing

Thank you for the careful review and the concrete suggestions; both points led to new experiments below.

W1. Comparison with Surprise Forcing

Surprise Forcing manages memory inside the attention cache of a streaming text-to-video model, while our hosts retrieve stored frames by camera geometry, so the full systems are not directly comparable. Its eviction rule, however, can be compared under one interface. Since its code is not yet public, we adapted its memory components to MemCam at the same budget (10 trajectories, 30s, B=32): it improved FVD over full retention (755.5 vs 880.6) but made LPIPS worse (0.5308 vs 0.5156). KEEPSAKE on the same trajectories is running for a direct comparison, and we will add a 50-video comparison to the main table alongside the existing baselines.

The key difference: Surprise Forcing scores a frame once, by how surprising it was when written, and a drifted frame also looks surprising. KEEPSAKE re-scores every stored frame at each update against what the rest of the bank covers, keeping views that are hard to replace and dropping those that are not.

W2. Component ablation

We ablated pose, appearance, and the two redundancy terms on both hosts (30 MemCam and 30 WorldMem trajectories):

VariantMemCam LPIPS ↓MemCam FVD ↓WorldMem LPIPS ↓WorldMem FVD ↓
Full KEEPSAKE0.5891411.70.5086601.4
Pose-only (no appearance)0.5902416.20.5211733.8
Appearance-only (no pose)0.5930449.60.5412863.9
Degree-only (no closest term)0.5901422.50.5314890.4
Closest-only (no degree term)0.5911452.70.5157750.3

The ablation uses fresh 60-second runs on 30 trajectories per host. FVD is computed per set, so these values are not comparable to the 50-trajectory main results.

Each element is necessary: full KEEPSAKE is best on every metric on both hosts, and removing any single element worsens both LPIPS and FVD. The effect is largest on WorldMem, where removing a component costs 0.007 to 0.033 in LPIPS and 132 to 289 in FVD, and where pose and the closest-substitute term contribute most. On MemCam the differences are smaller, and removing pose or the degree term costs the most FVD. Appearance contributes least to aggregate metrics, since poses are exact on these hosts, but it improves the decisions it targets: it changes the evicted frame in 29% of updates, and in those cases full KEEPSAKE discards the frame with the closer ground-truth substitute 72% of the time.

Dear Reviewers,

This is a friendly reminder that the rebuttal discussion period is approaching its deadline.

Please take some time to review the authors' rebuttal and actively participate in the discussion. If you have any remaining questions or concerns, please raise them and share your feedback with the authors and other reviewers.

As the discussion period is about to close, we kindly ask you to complete your discussion and provide your updated assessment before the deadline.

Thank you for your time and effort.

Best regards,

Area Chair

AuthorsOpenReview ↗

Experiment Update

Dear Reviewer,

Thank you again for your review. Since our first response, the experiments for both of your points have finished, and we summarize the final results here.

1. Comparison with Surprise Forcing

We adapted Surprise Forcing's retention rule to both hosts and added it to the main comparison under the same budget and trajectories (50 per host):

MethodMemCam FVD ↓MemCam LPIPS ↓WorldMem FVD ↓WorldMem LPIPS ↓WorldMem rFID ↓
Full retention344.40.58492643.90.6417134.96
FIFO B32421.00.62203091.10.6852154.66
Surprise Forcing B32439.30.5916699.00.542651.14
KEEPSAKE B32302.20.5736496.90.531556.53

On MemCam, Surprise Forcing is worse than keeping every frame on both metrics, just like FIFO, while KEEPSAKE improves on both. On WorldMem, KEEPSAKE achieves clearly better LPIPS and 29% lower FVD, while Surprise Forcing has a slightly better rFID. This matches the difference in design: Surprise Forcing scores a frame once, by how surprising it was when written, and a drifted frame also looks surprising; KEEPSAKE re-scores every stored frame at each update against what the rest of the bank covers.

2. Component ablation

On 30 trajectories per host, full KEEPSAKE is best on every metric on both hosts, and removing any single component worsens both LPIPS and FVD:

VariantMemCam LPIPS ↓MemCam FVD ↓WorldMem LPIPS ↓WorldMem FVD ↓
Full KEEPSAKE0.5891411.70.5086601.4
Pose-only (no appearance)0.5902416.20.5211733.8
Appearance-only (no pose)0.5930449.60.5412863.9
Degree-only (no closest term)0.5901422.50.5314890.4
Closest-only (no degree term)0.5911452.70.5157750.3

Pose and the closest-substitute term contribute most. Appearance contributes least to aggregate metrics, but improves the decisions it targets: when it changes which frame is evicted, full KEEPSAKE discards the frame with the closer ground-truth substitute 72% of the time.

Both results will be in the revised paper. If these address your concerns, we would be grateful if you would consider updating your score, and we are happy to answer any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.