acceptodds
Withdrawn from ICLR 2027

ReWorld: An Interactive World Model with Long-Horizon Memory

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive world models must follow actions, recall past views, and stream in real time. Control needs a short attention horizon; memory needs a long one. Jointly learning these abilities can strengthen action following while weakening revisit consistency. ReWorld separates these during training through mixed per-head attention windows and random head routing. Routing exposes each head to both horizons, allowing all heads to share a bounded cache at deployment without additional model parameters. Chunk-drop training prepares the model for sparse histories, while a bounded KV cache and pose-indexed landmark bank support long-horizon recall at inference. The bank preserves selected past chunks and retrieves them by camera pose, retaining spatially relevant history beyond the recent window. A metric-aligned data pipeline unifies eight synthetic, game, and real-world sources, and LoRA-confined distillation enables both high-fidelity and real-time 704 × 1280 generation from one backbone. Against six recent interactive world models, ReWorld achieves the lowest rotation error (11.95◦), best camera-motion consistency, and highest mean score across the seven evaluated VBench dimensions. On 64 s out-and-back rollouts (384 latents), its fixed 12-chunk cache still regenerates the starting view.

Primary area:
generative models
Keywords:
Video World Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5nzrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

ReWorld is an interactive world model that combines mixed attention windows with random head routing, chunk-drop training, and a pose-indexed landmark memory bank. It aims to improve camera control and long-horizon revisit consistency under a bounded KV-cache budget.

Critical strengths
  • The paper addresses a relevant problem: balancing responsive camera control with persistent scene memory.
  • The system integrates several practical components and includes comparisons and ablations on camera control, memory, and video quality.
Critical weaknesses
  • The methodological novelty is limited. The memory mechanism largely follows a context-as-memory approach, retrieving previously generated chunks by camera pose. The paper does not sufficiently establish a conceptual advance beyond existing context retrieval and memory-bank methods.
  • Chunk-drop training is a straightforward application of context dropping to sparse inference histories. Although useful, it offers limited novelty. Mixed attention windows and random head routing provide an incremental training refinement rather than a substantial methodological contribution.
  • The demonstrated visual consistency remains unconvincing. In the supplementary videos I inspected, scene content changes inconsistently even under simple camera movements within approximately 16 seconds. This undermines the central claim of persistent world memory and suggests that the reported metrics do not fully capture perceptual consistency.
Questions for authors

My major concern is the novelty. I did not find new techniques in this paper. The results is not that promising.

Reviewer_TDEUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose ReWorld, a interactive world model with long horizon memory. The paper identfies that learning action only require small window, while learning memory/keeping fidelity require long window. Base on this, this paper proposes a head rotation training paradigm as well as sink+landmark+recent KV cache management. A data process pipeline and LoRA training recipe are also provided.

Critical strengths
  • The paper focus on a critical problem of current world models: memory and consistency
  • The paper separate the different need for context window of action generation and memorization.
  • The paper propose a valid method to maintain long horizon memroy
Critical weaknesses

Major weakness

W1: design choices about training head rotation

The paper claims that to solve action/memroy tradeoff, a mixed per-head attention is used. 18 heads are used for short-window, and 6 heads are used for long window. And heads are cycle through a pool P = 12 random sets. During inference, all heads can only attend to a fix budget of KV cache.

Why here heads are separated as 18/6? What about other ratios? What if heads are randomly selected to be short/long heads? Why only 12 sets are used, insteal of all possible C(24,6) sets?

In this setting, all heads focus on short/long context together. Another design is to let some heads focus on long context and others focus on short context. Here I propose a simple baseline: short heads only focus on short context, base on recent few frames to predict future (in the paper's claim, this head is specialized in modeling action on current frame). Other long heads access chunk-dropped full history during training. During inference, sink+landmark+recent chunks are used, and short heads can only see recent chunks, while long heads can see all chunks. These heads are fixed are not rotated.

W2: Questions on Palindromic revisit benchmark

The authors compare revisit consistency in Table 2 and Table5. Why different world model have different path(px)? Some are 3x longer than shortest (HW-worldplay). What is the effect of different path length? SSIM and LPIPS are used to compare generate frames, but these metric are vulnerable to perturbations like shift/scale, and these perturbations are common in world models[1,2]. Therefore it cannot faithfull reflect revisit consistency. Could you provide qualitative result or report other metrics like Dreamsim[1] or SGCS[2]?

Minor weakness

  • Font size in Figure 2 3 4 too small, need to zoom in to see clearly. Recommend to be as big as main text
  • Recommend using vector formats (e.g., PDF) for figures to avoid loss of quality when zoomed in. It was frustrating trying to zoom in, only find everything blurry
  • Pipeline a little bit complicated. Maybe its better to put irrelevant/not novel stuff (LoRA, MRoPE) into appendix and highlight ideas that action and memory required different horizon to model(can add more analyse), and this naturally introduce different horizon head training.

[1] DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data

[2] LoopNav: Benchmarking Spatial Consistency in World Models

Questions for authors

See weakness

Reviewer_4oDwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReWorld, a camera-controllable streaming world model built on Wan2.2-TI2V-5B. The paper brings a new finding: control and memory are not getting better in coupled framework, which should be treated separately. Nevertheless, ReWorld gives evidence about this finding and proposes its solution by combining mixed local/global attention windows with random head routing. Also, ReWorld introduces a pipeline during training and inference: chunk-drop training prepares the model for sparse histories, while a bounded, pose-indexed landmark KV bank retrieves relevant past observations during generation. Meanwhile, the paper mentions about their eight-source training corpus and LoRA-based distillation for four-step sampling. Experiments evaluate camera control, revisit consistency, and video quality against six baselines, with additional memory ablations extending to 64-second rollouts.

Critical strengths
  1. I think the most valuable point of the paper is brining the finding as I mentioned in Summary. To the best of my knowledge, this is novel to the area. It also aligns with my own experience and highlights a practically important challenge that deserves further investigation.
  2. The proposed mixed-window attention and random head routing provide a simple, lightweight way to mitigate the control–memory trade-off, without introducing additional model parameters or auxiliary losses. It's a legit way to address the proposed challenge.
  3. The work presents an intuitive and coherent design across training and inference. Chunk-drop training prepares the model for sparse histories, while the bounded landmark cache provides practical memory retention and retrieval at deployment. These components complement each other and form a cohesive engineering solution.
Critical weaknesses
  1. While the empirical finding is insightful, the methodological novelty appears more incremental. Several components have close precedents: ReMind combines camera-aware RoPE with training for sparse, non-contiguous memories (Though mentioned in method part, no direct comparison is made.); WorldScape[1] uses a hierarchical KV cache with sink anchors, local windows, and geometry-guided memory retrieval; and Memorize When Needed[2] introduces history dropout and synthetic revisit loops. These connections do not diminish the value of the paper’s specific empirical observation, but they narrow the methodological contribution. The paper would benefit from clearer positioning against these approaches and controlled comparisons that establish the additional value of its random head routing.
  2. The second biggest concern is about evaluation. The authors only evaluate on a self-constructed benchmark, makes it less convincing as to quantitative metrics. WBench[3] Nav split and SANA-WM-Bench[4] should be concluded to test the performance of ReWorld. The former is a comprehensive benchmark for world models, evaluating from five different dimensions and 22 metrics. The latter is especially designed to test the memory consistency (revisit) and camera control capabilities.

[1] WorldScape: An Efficient Real-Time Interactive World Model with Universal Control. Manifold AI. [2] Memorize When Needed: Decoupled Memory Control for Spatially Consistent Long-Horizon Video Generation. Guo et al. [3] WBench: A Comprehensive Multi-turn Benchmark for Interactive Video World Model Evaluation. Ying et al. [4] SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformer. Zhu et al.

Questions for authors
  1. Regarding Weakness 1, could the authors clarify the methodological differences from the related works discussed above? And direct comparison between ReWord and these works are welcomed.
  2. Regarding Weakness 2, could the authors evaluate ReWorld on WBench’s navigation subset and SANA-WM-Bench? Results under these shared protocols would provide stronger evidence that the reported improvements generalize beyond the authors’ own benchmark.
  3. Could the authors clarify whether the “base” configuration in Table 4 uses a fixed local/global head partition? A matched comparison between fixed and randomly routed partitions would help isolate the contribution of random head routing.
  4. I also want to know more about implementation details of real-time arch. Please report on which hardware, how much FPS can ReWorld achieve?

I would raise my rating if all questions are well addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.