acceptodds
Withdrawn from ICLR 2027

Remembering What Matters for Long-Horizon Robot World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Autoregressive robot video world models based on discrete visual tokens remain unreliable over long rollout horizons. Retaining the entire generated history does not necessarily improve long-horizon prediction quality. We investigate a possible memory-role conflict in Full Context decoding: persistent scene information, recent visual–action states, and stale generated history compete for attention despite their different predictive roles. This perspective motivates (Receding Context with Anchor Preservation), a training-free context-scheduling method for long-horizon robot video world models. keeps the pretrained tokenizer, world model, and visual decoder frozen. At inference time, it preserves the initial scene anchor and a contiguous window of recent complete visual–action blocks while evicting older generated history. The schedule preserves within-block structure and temporal order, requires no additional modules or parameter updates, and bounds active-context growth independently of rollout length. We evaluate on RT-1, BridgeV2, LIBERO-90, and CALVIN using frozen autoregressive world models. Across the evaluated environments and rollout horizons, improves visual fidelity, action-conditioned consistency, and robot structure preservation relative to the corresponding baselines. On BridgeV2, it reduces LPIPS by . In the system benchmark at , it reduces the logical KV footprint by and p50 latency by . Attention analyses show increased attention to retained recent states, while history interventions associate larger amounts of stale context with higher prediction error. These observations are consistent with the memory-role-conflict hypothesis, although they do not isolate it from context-length and position-handling effects. Our results support inference-time context scheduling as a practical approach to improving long-horizon robot video prediction and computational efficiency without retraining.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TieqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ReCAP, a training-free context scheduling method for long-horizon autoregressive robot video world models. ReCAP preserves an initial scene anchor and a fixed window of recent visual–action blocks while evicting older generated history. The method is evaluated across four robotics datasets and three world-model backbones, reporting improvements in visual fidelity, trajectory consistency, semantic evaluation, and inference efficiency.

Critical strengths
  1. The method requires no additional training, and it bounds inference context and memory consumption independently of rollout length.

  2. The experiments show improvements across multiple datasets, model backbones, and evaluation metrics, including substantial gains in visual prediction quality and concurrent inference throughput.

Critical weaknesses
  1. Apart from scheduling complete visual–action blocks rather than individual tokens, the core method closely resembles the attention-sink-plus-recent-window strategy of StreamingLLM. More importantly, Figure 6(a) shows that ReCAP and Recent Only achieve statistically indistinguishable LPIPS-AUC results. Although Appendix E reports small benefits from longer scene prefixes, the necessity of anchor preservation remains insufficiently established.

  2. While the paper points to potential applications in planning and policy evaluation, its experiments are limited to offline action-conditioned video prediction. The foreground-based trajectory metrics are indirect proxies for robot motion. The VLM judgments, whose prompt explicitly penalizes static arms and discourages tied scores, and the limited internal human evaluation cannot establish physical consistency or task success.

Questions for authors
  1. Given the negligible difference between ReCAP and Recent Only in Figure 6(a), what evidence supports the importance of anchor preservation? Can the authors report the Recent Only comparison across all backbones, datasets, horizons, and metrics in Table 1, rather than only LPIPS-AUC for RLVR-World?

  2. The 1931-token context exactly matches the adaptation training sequence length of the adapted models, whereas Full Context grows far beyond it. The improvements may therefore primarily reflect avoiding out-of-distribution context lengths rather than resolving the proposed memory-role conflict. How can the authors distinguish the proposed memory-role conflict from training-context-length mismatch and positional effects? Could length-matched or position-controlled comparisons clarify the underlying mechanism?

  3. Can the authors explain the unusually uniform improvements across different datasets in Table 1 (for iVideoGPT and RoboAlign-R1, particularly in LPIPS, nDTW, and VLM scores), and the per-episode LPIPS and SSIM 100% win rates in Table 2? Could the authors provide more evaluation details, such as the evaluation sample sizes, exact horizons, and per-episode results.

Reviewer_UK1MOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses action-conditioned autoregressive robotic video prediction and proposes an inference-time history truncation strategy called ReCAP: it keeps the initial scene and initial state/action, retains the complete image-action information from the most recent 6 steps, and deletes earlier history. This method does not require updating model parameters. The authors report improvements in video prediction quality and cache efficiency on four robotic manipulation datasets, and explain the reasons through ablations and attention analysis.

Critical strengths
  1. The method is simple and easy to use. It does not require modifying the model architecture or additional training, and it can limit the growth of context and cache.
  2. There are evidences supporting its empirical effectiveness. Paired comparisons were conducted across multiple models and datasets, and component ablations, confidence intervals, and sensitivity analyses of the evaluation protocol were provided.
Critical weaknesses
  1. Contribution is limited. The core strategy is similar to a combination of a fixed prefix and a sliding window. Compared with Recent Only, LPIPS AUC in the main ablation shows no significant improvement, so the necessity of the core design of retaining the initial prefix has not yet been fully supported. The paper also does not systematically analyze window length, so it cannot explain why the most recent 6 steps were chosen.
  2. Under the influence of multiple factors, the value of history selection is unclear. ReCAP simultaneously shortens the context and rebuilds the cache and renumbers positions. Input budget is also close the sequence length used in adaptation training. Therefore, the improvements may partly come from reducing the discrepancy between training and inference rather than from the old-history interference.
  3. The scope of applicability and downstream value is unclear. The existing experiments focus on robotic manipulation and do not validate mobile manipulation or navigation. When viewpoints and scenes change continuously, the initial observation may no longer be suitable as a fixed reference. Also, improvements in PSNR, LPIPS, foreground trajectory metrics, and subjective ratings do not directly indicate that the model is better suited for policy learning or reinforcement learning.
Questions for authors
  1. Given that Recent Only essentially matches ReCAP, what evidence shows that a fixed initial prefix is necessary? How were the most recent 6 steps chosen, and is the method sensitive to window length?
  2. Can the effects of history content, position numbering, and cache updates be separated?
Reviewer_kKeJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies long-horizon degradation in autoregressive robot video world models. It introduces ReCAP, a training-free context scheduler that retains the initial scene anchor and recent complete visual–action blocks. Older generated blocks are removed from the active context. Experiments on RT-1, BridgeV2, LIBERO-90, and CALVIN report improvements in visual fidelity, trajectory consistency, and semantic evaluation. The paper also reports lower memory requirements and faster inference under its systems benchmark.

Critical strengths
  1. ReCAP is practical and straightforward to apply to the evaluated models without parameter updates. The combination of improved prediction quality and a reduction in p50 generation latency gives it clear deployment value under the reported benchmark.

  2. I find the paired comparisons and confidence intervals useful evidence for the quality gains across datasets. The detailed metric definitions, judging prompts, and sensitivity checks also make the evaluation easier to assess.

Critical weaknesses
  1. I think the distinct contribution beyond existing prefix-preserving sliding windows remains unclear, especially given StreamingLLM and the robot-video context-refreshing method SWR. Moreover, some of the results are not strong enough to convince me. For example, Section 5.4 reports LPIPS AUC gains over "Recent Only" within ±0.0005, leaving limited evidence for the importance of anchor preservation in that comparison.
  2. Another concern of mine is that the experiments do not yet isolate interference from stale history as the reason for the gains. ReCAP's budget matches the adaptation sequence length, and the BridgeV2 reference implementation also rebuilds KV states and rebases positions after eviction.
Questions for authors
  1. What distinguishes ReCAP from blockwise StreamingLLM with identical retained tokens, positions, and cache updates? How should we interpret the role of the scene anchor given the Recent Only ablation?

  2. Can matched controls separate stale-history content from context length, positional handling, and cache rebuilding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.