acceptodds
Under review as a conference paper at ICLR 2027

Simple Video Context Scaling of Robotic Policies

Abstract

Intelligent robotic agents should exhibit strong memory capabilities over very long temporal horizons. Memory can be facilitated by directly feeding video history as context to the policy, but doing so comes with a significant challenge. Video memory spanning more than even just a few seconds quickly becomes computationally difficult: simple token math mandates tokens of context just for minute of fps video memory with current standard methods. To address this, in this work we show that by adopting an early-fusion-like model architecture, even very long videos can fit completely and token-efficiently in transformer contexts, changing the token math substantially. On simulated memory benchmark tasks our method, Scaled Robotic Memory via Early-Fusion (SRoME), greatly surpasses the token efficiency of history representations compared to baselines, while also being more performant. We also show that train step times and inference times are quick, with very favorable scaling behavior when memory context length scales. In short by tackling the main challenge that plagues video memory policies today, video context length, SRoME makes long video memory policy training easy.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Video Memory, Long-Context, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pfwcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SRoME, which compresses video history using a pretrained video VAE and large-patch tokenization, then feeds the resulting tokens into VLA or WAM policies with block-causal attention and KV caching. Validation includes RoboMME task evaluations, a patch-size ablation, and synthetic training/inference scaling measurements demonstrating reduced history-token counts and computational costs.

Critical strengths

The representation is simple and token-efficient: video compression followed by large-patch packing substantially reduces history-token counts, with timing measurements supporting computational savings over the implemented MEM baseline.

Critical weaknesses

Limited methodological novelty and attribution. The method combines existing video compression, large-patch packing, and causal caching. Without a matched no-history baseline, the experiments do not establish whether gains arise from the proposed memory representation or the WAM’s additional capacity and future-image supervision.

Long-horizon capability is not demonstrated. Extended-context results measure computation on synthetic inputs with history already cached, excluding prefill and cache updates. Task evaluation remains confined to RoboMME, with only 24.25% overall success. These results do not substantiate reliable long-duration memory or practical end-to-end deployment.

Questions for authors

NA

Reviewer_AXVfOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the cost of giving robot policies long video memory. Image encoders turn each past frame into hundreds of tokens so history quickly becomes too expensive to use. SRoME from this paper first compresses short video clips with a pretrained video tokenizer AToken, then splits the compressed latents into very large patches, each as large as the transformer's hidden size allows. Each patch becomes one token. A block-causal attention mask lets past clips be cached, so the policy does not re-encode history at every step. The authors plug this history into two policy types built on BAGEL: a vision-language-action model (VLA), and a world-action model (WAM) that predicts a future image and passes it to a separate action model.

Critical strengths
  1. The idea is simple and sound and it addresses a real problem i.e. longer video histories and policies having no memory.

  2. There is a significant efficiency gain.

Critical weaknesses
  1. The authors say that they show that SRoME enables more performant policies in both VLA and WAM model families. But they never showcase this in the case of WAM. Table 1 has WAM results, but both WAM columns use SRoME. But they never test against WAM without SRoME.

  2. In the case of VLA the gain is within noise. The paper describes the same result differently in different sections. Section 4.2 states that the SRoME VLA is "slightly better than BAGEL-MEM on average," but the abstract presents SRoME as "more performant".

Questions for authors
  1. The introduction states that SRoME "enables more performant policies in both VLA and WAM model families." Table 1 includes two WAM variants, but both use SRoME and differ only in patch size. Did the authors train a WAM without SRoME, for example with MEM-style history or with no history at all? Without such a comparison, what supports the claim for the WAM family?

  2. For the VLA, Section 4.2 describes the SRoME VLA as "slightly better than BAGEL-MEM on average". The abstract, however, describes SRoME as "more performant." Is this difference statistically significant? Could the authors report results over multiple training seeds, or confidence intervals?

  3. How does each policy perform without any video history using only the current observation? The authors need to show this baseline as this would show whether the models actually use the history SRoME provides.

Reviewer_kWmiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on efficient encoding of context, in particular video history as an input to robot policies. A pretrained video VAE is reused for video compression, and the latent-space representation is then converted into coarse patches. The history is fed via a block-causal mask, allowing for KV caching. The method, SRoME, is instantiated on two policies based on BAGEL: a VLA and a WAM.The evaluation is on RoboMME, and SRoME is shown to outperform a reimplemented BAGEL-MEM approach.

Critical strengths

Memory compression is an important, unsolved problem not only in robotics. The method is simple. The results where MEM-style training OOMs at 128 s, while SRoME trains with 30 min of history, is convincing.

Critical weaknesses

Novelty:

  • The core contribution is applying an existing VAE to a new task. The patch compression is a known mechanism, used for instance in Qwen2-VL's 2×2 MLP merger or SmolVLM. Patchification of VAE latents for long-video context is essentially what FramePack does (Zhang & Agrawala 2025 (https://arxiv.org/abs/2504.12626) (the authors do acknowledge this). This is fine, but it is an application paper and the bar on achieved results is higher to me.
  • related work on memory for robotics is under-cited and many baselines are missing. The only baseline actually compared against is MEM. Some examples: ContextVLA (Jang et al. 2025, arXiv 2510.04246 (https://arxiv.org/abs/2510.04246)), HAMLET (Koo et al., ICLR 2026, arXiv 2510.00695 (https://arxiv.org/abs/2510.00695)), NativeMEM (Wang et al., arXiv 2607.06678, Jul 2026 (https://arxiv.org/abs/2607.06678))

Method, results:

  • The results from RovoMME itself are ignored (TokenDrop and FrameSamp), while they constitute natural baselines that needs to be compared against.
  • The paper reports a single run, while the standard from prior methods is 3 runs or even 9 (RoboMME) with different ckpts and seeds. With 50 episodes/task, most results do not seem to be significant
  • it would be nice to show alternative compressors to justify the “Any reasonable video VAE can be used” statement. Also ablations such as raw-pixel large patches without a VAE or pixel-unshuffled SigLIP features at a matched token budget would help
  • the WAM results do not isolate the SRoME contribution, that would need a WAM without history or WAM-with-MEM results. Currently it compares a 14B model with a generative expert against the VLA at 7B, understanding-only.
  • there is no baseline without any history at all, while the results underperform RoboMME's published memoryless π0.5
  • The BagelMEM baseline is not well-specified, the code or hparams should be given. Currently the reproducibility seems rather limited and code is not promised.
  • Proprioception seems to enter the VLA in multiple places, which is not motivated anywhere
  • current frame uses siglip tokens, it would be nice to see an ablation where it uses the SRoME repre. for current frame as well.
  • Inference excludes building the cache and appending clips (VAE encode + LLM prefill), and it is not clear whether MEM also uses caching
  • In fig5b, ie the actual evgaluation, MEM is actually faster than SRoME WAM.
  • context scaling is never actually evaluated, there is no history length x success rate result, all the evaluated task do not need more than 1 minute

Presentation and framing:

  • WHat is described as “early fusion” is nto really early fusion. Early fusion typically describes an architecture where all modalities are projected into one shared token space from the start and a transformer is trained from scratch end to end. What the paper does seems lik late-ish fusion with a different, more compressive encoder.
  • the figs have very smal larrow-heads and tight block spacing, decreasing readability
  • Figs. 3 and 4 are really ahrd to understand standalone, many coimponents are not defined in the caption or the figure anywhere
  • Eqs. 2–3 are KI (Driess et al. 2025 (https://arxiv.org/abs/2505.23705)) verbatim, which should be stated to make it clear there is nothing new, and maybe moved int background
  • Undefined or vague terms: "spatial grid" and the divisibility constraint on p (p must divide H_z, W_z; for a 32×16 latent, p ≤ 16).
  • Writing:
    • tautological: "compresses video … effectively performing compression"
    • minor but check for typos and LaTeX quotes: ”clip”
Questions for authors

The novelty is limited and the method’s performance is currently not well-supported by the experiments, it is not properly contextualized within prior work. The paper also has presentation problems. What could change my opinion the most are:

  1. Provide performance without any memory to show whether it is actually used.
  2. provide comparison against published baselines.
  3. Provide implementation details required for reproducibility
  4. Provide experiments attributing the WAM gain properly

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.