acceptodds
Withdrawn from ICLR 2027

GraphMem: A Training-free Graph Retrieval and Memory Caching Framework for Consistent World Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present GraphMem, a training-free, graph-structured visual memory framework for long-horizon autoregressive video world models. Rolling KV caches enable efficient chunk-wise generation but discard earlier scene evidence once it leaves the recent window, causing inconsistency when the camera revisits a previously observed region. Full-history conditioning retains this evidence but makes the active context and inference cost grow with video length. Our key idea is to dynamically retrieve relevant history based on the current state. GraphMem organizes pixel-space history into a Vision Memory Graph. Conditioned on the current generation state, dynamic graph retrieval activates a compact set of relevant and complementary historical views. The retrieved history is encoded with timestep-aligned noise and prefilled through the frozen backbone, producing layer-wise history KV caches in its native attention space. Dynamic retrieval keeps the active context bounded, while strided prefill amortizes the additional computation. Across diverse backbones, long-horizon memory performance improves by about 2% on average and shows better revisit consistency.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Training-free, Retrieval Enhancement, Memory, World Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RG6uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes GraphMem, a training-free memory framework for autoregressive video world models. It organizes historical keyframes into a graph, retrieves relevant views, and converts them into native history KV caches through a frozen diffusion backbone.

Critical strengths
  1. The method addresses a clear limitation of rolling context windows: recovering previously observed scenes after long temporal gaps smartly.
  2. Integration across five frozen backbones showcases the approach’s applicability and generalization.
  3. The paired comparisons show consistent improvements on the three reported memory metrics, and the qualitative examples illustrate the intended behavior.
Critical weaknesses
  1. Tab. 2 compares against queue and temporal/spatial banks, but their retrieval procedures and budget matching are not sufficiently discussed in main paper. A particularly important baseline is flat retrieval using the same pose filtering, visual descriptors, diversity selection, adaptive budget, and KV prefill, etc. w/o this comparison, it is difficult to tell the gains come from graph rather than better candidate selection and additional historical context.
  2. In Tab.1, the reported results lack uncertainty est. or per-traj. distributions, making their robustness difficult to assess. The main content provides limited definitions of three memory metrics. Their sensitivity, e.g., pose mismatch, appearance similarity, and genuine scene reconstruction, etc., needs clarification.
  3. The efficiency claim is only partially supported in the main paper. Although the active attention context is useful, the total inference cost also includes history prefill, descriptor extraction, retrieval, and graph maintenance. The external memory continues to grow. E2E latency, memory consumption, and scaling with rollout length are necessary to assess the practical advantage.
  4. quality trade-offs require clearer discussion. Although memory scores improve, subject consistency and dynamic degree decrease for several backbones, and I2V scores show mixed changes. This suggests a trade-off between historical consistency and other aspects of generation that should be analyzed carefully.
Questions for authors
  1. How does GraphMem compare with flat pose-and-appearance retrieval using identical selection, context budgets, and KV prefill , etc.?
  2. Are the gains robust across trajectories and generation seeds?
  3. What are the e2e latency and memory costs at matched context budgets?
Reviewer_mcqBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GraphMem, a training-free, graph-structured visual memory framework for autoregressive video world models. By organizing generated keyframes into a Vision Memory Graph connected by temporal, spatial, and revisit edges, and by prefilling keyframes retrieved based on the current generation state through the frozen DiT to construct a History KV Cache, the paper reports improved long-horizon consistency under camera revisits across five backbones.

Critical strengths
  • The method requires no parameter updates or memory-specific modules and achieves consistent gains when applied to five different backbones. It is also practical because the retrieval budget bounds the active context, so per-chunk attention cost does not increase with video length.
  • The authors verify through attention analysis whether the retrieved history is actually used by the frozen backbone, thereby providing justification for the design choice of native KV-cache prefill. The paper also includes ablation studies that remove individual graph components and separates comparisons with external baselines from controlled comparisons on the same pretrained backbones, making the scope of its claims easy to follow.
Critical weaknesses
  • The three metrics used to evaluate memory are all based on the cosine similarity of DINO and CLIP features between frames or chunks of the generated video. The Long-Horizon Score compares the video's head chunks with its tail chunks, regardless of camera pose. Since this metric yields a high score whenever the overall appearance is similar, models in which the camera barely moves and the scene shows limited evolution have an advantage. The Memory Score is also a weighted combination of the Long-Horizon Score, the FarGap term, and the Loop Closure Score, so it does not constitute an independent metric. Although the revisit trajectories from DL3DV and Context-as-Memory come with ground-truth videos, the authors do not report results comparing the generated videos against the ground truth using PSNR or LPIPS, nor do they report camera-trajectory accuracy. Furthermore, the gains are small, and because the authors report neither the number of seeds nor confidence intervals, it is impossible to determine whether the gains fall within the margin of error.
  • This paper identifies flat similarity-based retrieval and a fixed retrieval budget as the two key limitations of existing training-free methods, and claims to address them through the Vision Memory Graph and demand-adaptive retrieval that dynamically adjusts the number of activated keyframes within a bounded budget. However, there are no controlled comparisons with existing training-free methods such as WorldKV applied to the same backbone, nor are there comparisons with simple retrieval baselines that select keyframes based on field-of-view overlap or camera-pose proximity. The ablation study also does not explain the settings for the Queue and Bank baselines. Furthermore, since there is no comparison with a fixed retrieval budget or analysis of how the number of selected keyframes K_n actually varies during generation, it is unclear whether the demand-adaptive retrieval is effective. The method should be compared with simple retrieval baselines and a fixed retrieval budget on the same backbone.
Questions for authors
  • My understanding is that this paper centers on "native history KV prefill," which constructs a layer-wise History KV Cache by passing retrieved keyframes through the frozen DiT. In this process, the clean latents of retrieved keyframes are perturbed with noise aligned to the current denoising timestep. However, the necessity of this timestep alignment is not experimentally validated. While the authors argue that it makes the historical representations compatible with the current denoising state, adding noise may also obscure fine-grained visual information needed to reconstruct previously visited regions. Moreover, the paper states that the sink and recent caches retain their original treatment, raising the question of why the retrieved history specifically requires timestep-aligned noise. Could the authors compare the proposed approach against prefilling clean latents without added noise (t=0), under otherwise identical conditions? Such an ablation would clarify whether timestep alignment actually improves memory utilization and revisit consistency, or whether clean historical evidence provides equally effective or better conditioning.
Reviewer_xNm4OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GraphMem, a training-free memory framework for autoregressive video world models. It organizes historical frames into a graph using camera poses and visual features, retrieves relevant views, and converts them into KV caches through a frozen DiT. Experiments across five backbones demonstrate improvements in long-horizon revisit consistency.

Critical strengths

The paper provides useful empirical results across different video world models. The controlled comparisons and ablations support the effectiveness of graph-based memory retrieval.

Critical weaknesses

W1. Limitations of heuristic memory retrieval. The retrieval strategy largely relies on handcrafted rules involving camera poses, DINO similarity, graph relations, and Top-K/MMR selection (Sec. 3.1). However, spatial proximity and visual similarity do not necessarily indicate which historical observations are most useful for the current generation. The paper lacks a systematic analysis of retrieval reliability, particularly under viewpoint changes, ambiguous appearances, or inaccurate camera poses.

W2. Limited consideration of generation stability in memory retrieval. The retrieval strategy primarily targets scene consistency during revisits, without explicitly considering how memory selection affects long-horizon autoregressive stability. Although the paper reports temporal flickering and motion smoothness, these metrics do not establish whether retrieved history mitigates accumulated generation errors. Moreover, retrieving outdated or inaccurate historical observations may introduce additional errors. Further experiments on long-horizon generation stability would strengthen the work.

Questions for authors

Q1. How robust is the heuristic retrieval strategy to inaccurate camera poses and visually similar but spatially different regions? Could the authors provide additional retrieval ablations or robustness evaluations?

Q2. How does memory retrieval affect error accumulation during long-horizon autoregressive generation? Could the authors evaluate generation stability across increasing rollout lengths, including trajectories without revisits and cases with imperfect historical observations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.