acceptodds
Under review as a conference paper at ICLR 2027

LongContextVLA: Reasoning over Hundreds of Images for Robot Control

Abstract

Vision-language-action (VLA) models typically condition on only the current observation or a small set of past keyframes. This limits their ability to solve long-horizon tasks that require persistent visual memory. We develop a VLA that conditions on more than 160 images of visual history, represented with an average of 72 tokens per image, while maintaining real-time inference speed. Our approach builds on Qwen3.5-4B, a multimodal model with a 256K-token context window, which we fine-tune for robotic control. We find that long visual context alone is insufficient for robust generalization, and train the VLA to reason over and narrate events in its visual history, using parallel decoding to avoid the latency cost of autoregressive reasoning. By caching past visual tokens and processing only newly arriving images, our streaming implementation achieves per-step latency close to that of a single-image VLA. We evaluate our approach on a long-horizon shell game and object retrieval task, achieving over 80% success in unseen generalization scenarios, as well as competitive performance on the RoboMME benchmark. Finally, we demonstrate memory-based prompting, where the robot observes a human demonstration and reproduces the demonstrated behavior, with the ability to generalize to unseen scenarios. Together, our results show that scalable visual memory, learned reasoning, and streaming inference enable real-time, long-horizon robot control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, memory, manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_U8BkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates long-term visual memory for vision-language-action (VLA) models by directly incorporating hundreds of historical images into robot action prediction. Built on Qwen3.5-4B, the proposed LongContextVLA combines streaming cache, parallel narration decoding, and grouped action decoding to reduce inference latency. The model is also trained to narrate historical events through latent reasoning tokens, encouraging it to extract task-relevant information from long visual histories without autoregressive reasoning at inference time. Experiments on RoboMME and three real-world robotic manipulation tasks demonstrate strong performance on memory-dependent tasks and generalization to unseen scenarios.

Critical strengths
  1. Important research problem. This paper addresses the important problem of long-term visual memory in VLAs and explores the advantages of retaining dense visual histories over memory compression and keyframe selection.

  2. Efficient inference with strong empirical results. The proposed framework combines Streaming Cache, Parallel Narration Decoding, and OAT-based Grouped Action Decoding to reduce inference latency. It further uses historical event narration to supervise latent reasoning, improving the model's ability to utilize historical information without autoregressive text generation. The method achieves an average success rate of 84.9% across 16 RoboMME tasks and demonstrates generalization to unseen event sequences, object assignments, and task scales in real-world experiments.

Critical weaknesses
  1. Limited benchmark coverage and incomplete SOTA comparisons. The simulation evaluation is limited to RoboMME, and additional experiments on RoboMemArena [1], RMBench [2], or MIKASA-Robo [3] would help establish the method's effectiveness across more diverse memory-intensive tasks. More importantly, the paper reports only the ablation variants of SimpleMemVLA [4], while omitting its full model, which reports 88.3% success on RoboMME compared with 84.9% in this paper. A fair comparison with the complete method is necessary to support the claimed improvement over existing approaches.

  2. Insufficient analysis of history scaling and inference efficiency. The paper does not systematically examine how task success changes with the number of historical frames, making it unclear how many additional frames are needed to achieve meaningful improvements and when the performance gains begin to diminish. Furthermore, while Parallel Narration and Grouped Action Decoding increase computational parallelism, they do not fundamentally reduce the computational complexity of the corresponding decoding operations. The efficiency claims would be more convincing with end-to-end latency comparisons against optimized single-frame and few-frame VLA baselines under the same computational resources.

[1] Lei et al., 2026. RoboMemArena: A Comprehensive and Challenging Robotic Memory Benchmark.

[2] Chen et al., 2026. RMBench: Memory-Dependent Robotic Manipulation Benchmark with Insights into Policy Design.

[3] Cherepanov et al., 2026. Memory, Benchmark & Robots: A Benchmark for Solving Complex Tasks with Reinforcement Learning. ICLR 2026.

[4] Yin et al., 2026. SimpleMemVLA: A Simple but Effective Native-Video Memory for Vision-Language-Action Models.

Questions for authors
  1. Benchmark coverage: Could the authors provide additional results on memory-intensive benchmarks such as RoboMemArena [1], RMBench [2], or MIKASA-Robo [3] to demonstrate the generality of the proposed approach?

  2. SOTA comparison: Why is the full SimpleMemVLA [4] model, which reports 88.3% on RoboMME, omitted from Table 1 while its ablation variants are included? Could the authors provide a fair comparison under consistent evaluation settings?

  3. History length scaling: Could the authors evaluate task success rates with different numbers of historical frames? This would help determine how many additional frames yield meaningful performance improvements and at what history length further gains begin to diminish.

  4. Inference efficiency: Could the authors compare the end-to-end inference latency of LongContextVLA with optimized single-frame and few-frame VLA baselines under the same computational resources? This would help quantify the additional inference cost of maintaining long visual histories.

  5. Understanding the decision process: The paper does not sufficiently analyze how historical visual information and latent reasoning affect action decisions. Could the authors investigate the relationships between historical visual tokens, latent reasoning tokens, and action tokens, for example through token-level attention or attribution analysis, to clarify their respective roles in action prediction?

2 comments

Q1. Benchmark coverage: Could the authors provide additional results on memory-intensive benchmarks such as RoboMemArena [1], RMBench [2], or MIKASA-Robo [3] to demonstrate the generality of the proposed approach?

Thank you for suggesting these benchmarks. Our current evaluation covers the sixteen RoboMME tasks, which span four memory categories, together with three real-robot tasks. During the rebuttal period we prioritized the core experimental results, including a new real-robot drawer task and larger real-robot sample sizes, and could not complete evaluations on additional benchmarks in the same window. We agree that they would further demonstrate generality, and we hope to include results on them in the final version.

Q2. SOTA comparison: Why is the full SimpleMemVLA [4] model, which reports 88.3% on RoboMME, omitted from Table 1 while its ablation variants are included? Could the authors provide a fair comparison under consistent evaluation settings?

Thank you for pointing this out. Omitting the full result was an oversight, and we have added it to Table 1 (88.3%, compared with 84.9% for our method). All SimpleMemVLA numbers in Table 1 are taken from its paper.

For a comparison under consistent settings, we trained SimpleMemVLA ourselves on our real-robot tasks, where it is the strongest baseline. LongContextVLA achieves higher success in all three settings, with 30 trials each:

SettingSimpleMemVLASubtask only (our ablation)LongContextVLA
Cup swap, unseen 5 swaps (full task)60.0%53.3%76.7%
Cup swap, unseen 6 swaps (full task)43.3%36.7%70.0%
Drawer, unseen configurations50.0%63.3%76.7%

SimpleMemVLA performs comparably to our subtask-only ablation. This is expected, because the two are closely related: both condition on the full video history and predict a subtask before acting, and neither is trained to narrate its history. Our main finding is that narration improves generalization to unseen scenarios, which is where LongContextVLA outperforms SimpleMemVLA.

For context, we would also like to respectfully note that SimpleMemVLA is concurrent work: it appeared on arXiv on September 2, 2026, two weeks before the submission deadline. The ICLR reviewer guidelines (https://iclr.cc/Conferences/2027/ReviewerGuidelines) consider a paper contemporaneous if it was published within the two months before the full paper deadline, that is, on or after July 17, 2026, and authors are not required to compare against such work. The guidelines also note that arXiv is not a peer-reviewed venue, so authors are not required to compare against papers available only on arXiv. They add that reviewers may make authors aware of such work, but that "the lack of such comparisons cannot be a basis for rejection." We are grateful that you brought the full result to our attention, we agree that the comparison is informative, and we are glad to include it. We would kindly ask that this be taken into consideration.

Q3. History length scaling: Could the authors evaluate task success rates with different numbers of historical frames? This would help determine how many additional frames yield meaningful performance improvements and at what history length further gains begin to diminish.

Thank you for this question. We expect the policy to be robust to history length and sampling density because of its backbone. The Qwen3.5 family supports video inputs of up to two hours within its 256K-token context, at varying frame rates and resolutions. The backbone family also handles sparse and irregular frame sequences: MemER (Sridhar et al., 2026) [5] obtains useful memory from a small set of disjoint keyframes with a backbone from the same family.

In our early experiments, changing the history sampling density did not visibly affect performance, and performance did not degrade as the history grew to more than 250 images at some decision steps.

During the rebuttal period we prioritized the core experimental results, and could not complete a systematic sweep in the same window. We will include a controlled sweep of history length and sampling rate in the final version.

[5] Ajay Sridhar, Jennifer Pan, Satvik Sharma, and Chelsea Finn. Scaling up memory for robotic control via experience retrieval. International Conference on Learning Representations (ICLR), 2026.

Q4: Inference efficiency: Could the authors compare the end-to-end inference latency of LongContextVLA with optimized single-frame and few-frame VLA baselines under the same computational resources? This would help quantify the additional inference cost of maintaining long visual histories.

Thank you for this suggestion. We evaluated history length scaling using our full model on 150 unseen five-swap videos and 100 unseen six-swap videos. We kept the checkpoint and evaluation protocol fixed, sampled observations at 2 Hz, and varied the maximum number of recent frames.

The table reports offline cup-location accuracy (%).

Maximum history frames5 swaps6 swaps
138.034.0
432.031.0
830.035.0
1235.338.0
1680.731.0
2083.340.0
2484.064.0
Full history84.065.0

For five swaps, accuracy increases substantially from 35.3% at 12 frames to 80.7% at 16 frames, with smaller gains thereafter, reaching 84.0% with full history. For six swaps, the largest observed improvement occurs later: accuracy increases from 40.0% at 20 frames to 64.0% at 24 frames, close to the full-history accuracy of 65.0%.

These results demonstrate the importance of retaining sufficient temporal history, with longer swap sequences requiring more historical observations. Increasing the budget beyond the full available history provides no additional benefit on these videos, because the input remains unchanged. At 2 Hz, the available histories contain 18–26 frames for five swaps and 22–32 frames for six swaps. Thus, the observed diminishing gains apply to these recordings and do not establish saturation for longer videos.

Q5: Understanding the decision process: The paper does not sufficiently analyze how historical visual information and latent reasoning affect action decisions. Could the authors investigate the relationships between historical visual tokens, latent reasoning tokens, and action tokens, for example through token-level attention or attribution analysis, to clarify their respective roles in action prediction?

Thank you for this suggestion. We examine the roles of historical visual information and latent reasoning through history-length ablations, model ablations, and real-robot evaluation.

Historical visual information affects the model’s decisions. Keeping the checkpoint and evaluation protocol fixed, we vary the number of recent observations sampled at 2 Hz. Five-swap cup-location accuracy increases from 35.3% with 12 frames to 80.7% with 16 frames, reaching 84.0% with full history. For six swaps, accuracy increases from 40.0% with 20 frames to 64.0% with 24 frames, close to the full-history accuracy of 65.0%. These results show that retaining earlier observations substantially improves location prediction, with longer swap sequences requiring more historical observations.

The latent-reasoning variant improves both location prediction and physical task completion. We compare three model variants below. Offline evaluation measures cup-location accuracy on 150 five-swap and 100 six-swap videos. Real-robot evaluation measures full-task success over 30 trials per setting, requiring both selecting the correct cup and completing the grasp.

ModelOffline: 5 swapsOffline: 6 swapsReal robot: 5 swapsReal robot: 6 swaps
OAT + subtask55.3%50.0%53.3%36.7%
OAT + subtask + narration51.3%47.0%60.0%33.3%
Full model, including latent reasoning84.0%65.0%76.7%70.0%

Adding narration alone does not consistently improve performance. The full model achieves higher accuracy and task completion than both ablations. In particular, compared with the narration-only variant, six-swap location accuracy increases from 47.0% to 65.0%, while real-robot success increases from 33.3% to 70.0%.

Together, these experiments show that access to historical observations is important for identifying the correct target, and that the full model’s improvements extend from semantic decisions to successful physical execution.

Reviewer_hacsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LongContextVLA adapts Qwen3.5-4B to robot control using a long history of visual observations. It adds supervision for narrating timestamped past events before predicting the current subtask and actions. A parallel variant predicts narration from reserved token positions and uses their hidden representations for action prediction. The system reuses historical KV and recurrent states and decodes OAT action tokens in groups. Evaluation covers three real-robot tasks, RoboMME, and inference latency at different context lengths. The method reports 84.9% success on RoboMME and improved real-robot generalization with narration supervision.

Critical strengths
  1. The real-robot tasks make a useful test of memory. The narration ablations show improvements on unseen configurations, particularly in the cup task.

  2. The streaming implementation is clearly described, including how it preserves the historical prefix across decisions in a backbone with both attention and recurrent layers. The latency measurements show a substantial practical benefit from reusing this computation.

Critical weaknesses
  1. Table 1 omits the full version of the closest competing method while including its weaker ablations. SimpleMemVLA v1, released on September 2, 2026, reports 88.3% on RoboMME in Table 2. The present paper includes its Retrieval, Token Compression, and Recurrent State variants at 31.5%, 22.6%, and 20.6%, yet leaves out the full model's result. LongContextVLA reports 84.9%. This omission materially affects the performance comparison and the claim of a roughly 40-point improvement over the strongest prior method. The full result should be included, with any evaluation differences explained. It is especially important here because both methods build on the same native-video backbone, and the additional value of narration is the main question.

  2. The methodological novelty is limited. As the paper acknowledges, SimpleMemVLA already uses Qwen3.5's native video context, predicts a subtask, and reuses history computation for efficient control. Parallel reasoning queries also appear in DualCoT-VLA, which the paper follows. Preserving a causal history cache and grouping action-token predictions are natural implementation choices. The main additional idea is timestamped history-narration supervision. The real-robot ablations suggest that it helps, but the evidence for its contribution is limited to a small set of tasks, and the RoboMME results do not isolate its benefit over the same model with subtask supervision alone.

Questions for authors
  1. Why does Table 1 include the three SimpleMemVLA v1 ablations but omit its full 88.3% result? Are there evaluation differences that prevent comparison with the full model? If so, how do those differences affect the inclusion of its ablations?

  2. In the parallel variant, actions use the hidden states at the narration positions. How closely does the decoded narration reflect the information used by the policy? For example, do incorrect accounts of earlier cup swaps coincide with incorrect cup choices? An analysis of existing rollouts would help explain what the narration supervision teaches the model.

  3. How does memory accuracy change as the gap between a decisive event and the final action grows? A short test with existing checkpoints, adding irrelevant observations while keeping the relevant event and final task state fixed, would help show whether narration improves retention under distraction. This would complement the latency scaling experiment with evidence about memory use.

1 comment

Q1. Why does Table 1 include the three SimpleMemVLA v1 ablations but omit its full 88.3% result? Are there evaluation differences that prevent comparison with the full model? If so, how do those differences affect the inclusion of its ablations?

Thank you for pointing this out. There are no evaluation differences: the full model and its ablations follow the same RoboMME protocol, and all SimpleMemVLA numbers in Table 1 are taken from its paper. Omitting the full result was an oversight, and we have added it to Table 1 (88.3%, compared with 84.9% for our method).

We have also trained SimpleMemVLA on our real-robot tasks, where it is the strongest baseline. LongContextVLA achieves higher success in all three settings, with 30 trials each:

SettingSimpleMemVLASubtask only (our ablation)LongContextVLA
Cup swap, unseen 5 swaps (full task)60.0%53.3%76.7%
Cup swap, unseen 6 swaps (full task)43.3%36.7%70.0%
Drawer, unseen configurations50.0%63.3%76.7%

SimpleMemVLA performs comparably to our subtask-only ablation. This is expected, because the two are closely related: both condition on the full video history and predict a subtask before acting, and neither is trained to narrate its history. Our main finding is that narration improves generalization to unseen scenarios, which is where LongContextVLA outperforms SimpleMemVLA.

For context, we would also like to respectfully note that SimpleMemVLA is concurrent work: it appeared on arXiv on September 2, 2026, two weeks before the submission deadline. The ICLR reviewer guidelines (https://iclr.cc/Conferences/2027/ReviewerGuidelines) consider a paper contemporaneous if it was published within the two months before the full paper deadline, that is, on or after July 17, 2026, and authors are not required to compare against such work. The guidelines also note that arXiv is not a peer-reviewed venue, so authors are not required to compare against papers available only on arXiv. They add that reviewers may make authors aware of such work, but that "the lack of such comparisons cannot be a basis for rejection." We are grateful that you brought the full result to our attention, we agree that the comparison is informative, and we are glad to include it. We would kindly ask that this be taken into consideration.

Q2. How closely does the decoded narration reflect the information used by the policy? For example, do incorrect accounts of earlier cup swaps coincide with incorrect cup choices?

Thank you for this suggestion. In the parallel variant, the narration is decoded from the same reserved positions whose hidden states the subtask and action tokens attend to, so the decoded text is a direct readout of those states. The policy is not constrained to use only what is decodable, however, so agreement between narration and action has to be measured, not assumed.

Our current evidence is indirect. On the offline cup evaluation (Table 3), narration supervision changes what the policy tracks: cup-location accuracy rises from 55.3% with subtask-only supervision to 84.0% with the full model on 150 held-out five-swap videos, and from 50.0% to 65.0% on 100 six-swap videos. We have not yet tabulated, per rollout, whether errors in the decoded swap sequence coincide with incorrect cup choices. This analysis needs no new training, and we will add it to the final version.

Q3. How does memory accuracy change as the gap between a decisive event and the final action grows?

Thank you for proposing this test. Our existing results vary this gap in two ways. In the cup swap task, the six-swap setting adds a swap that was never seen in training between the initial placement and the final choice, and the full model retains 70.0% real-robot success compared with 36.7% for subtask-only supervision. In the drawer task, the decisive drawer removals occur at the start of the episode and the requests arrive after a long sequence of unrelated pickups, with histories of up to 160 images.

We have not run the controlled version you describe, in which irrelevant observations are inserted while the decisive event and final state are held fixed. During the rebuttal period we prioritized the core experimental results, and could not complete this test in the same window. We agree it would complement the latency scaling experiment and will include it in the final version.

Reviewer_AXY8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a significant practical challenge and demonstrates promising results for memory-intensive robotic manipulation. It proposes LongContextVLA, a vision-language-action model designed to utilize hundreds of historical images for long-horizon robotic manipulation.

The novelty is somewhat incremental relative to recent long-context VLA approaches, and the experimental comparisons do not adequately isolate the contribution of the proposed components. The missing matched comparisons and insufficient analysis of context length are particularly important given the paper's claim that scaling robot memory to hundreds of images is responsible for the improvements.

Critical strengths
  1. The paper studies a practical problem: a robot often needs information that is no longer visible. The shell game, drawer put-away task, and video-prompted sorting task cover different uses of memory, including tracking an occluded object, recalling object locations, and following a demonstration.
  2. The streaming implementation is a useful contribution. Section 3.3 explains how to retain historical computation while restoring the KV cache and the convolutional/recurrent states after each decision. This treatment of Qwen3.5's hybrid architecture, together with parallel narration and grouped action prediction, makes the proposed policy more practical for robot control.
  3. The results are promising. The reported RoboMME average is 84.9% across sixteen tasks, and the average over the unseen shell-game and drawer settings is approximately 83.3%. Figure 4 also shows that narration improves performance over subtask-only prediction, particularly on unseen shuffle sequences. The appendix provides substantial detail on the architecture, optimization, and action tokenizer.
Critical weaknesses
  1. My main concern is the omission of the complete SimpleMemVLA baseline. Section 2 calls it the closest concurrent work, but Table 1 includes only its retrieval, compression, and recurrent-state variants. The complete native-video method reports 88.3% on RoboMME (Yin et al., 2026, Table 2). These cross-paper scores are not a controlled comparison, but the omission matters for the statement that LongContextVLA is “40 points higher than the best prior method.” SimpleMemVLA already combines native video history with streaming inference, and DualCoT-VLA uses parallel reasoning tokens. The paper needs to show more clearly what historical narration adds beyond these existing designs.
  2. The narration ablations leave two questions unresolved. First, Equation (5) uses reserved positions whose hidden representations condition the subsequent outputs; predicted narration words are not fed back into the policy. A comparison using the same reserved positions without the narration loss would help identify how much improvement comes from semantic supervision. Second, Figure 4 labels the final model “Subtask + narration + latent narration,” whereas Equation (5) describes parallel narration replacing autoregressive narration. It is unclear whether the reported model uses a joint objective or the replacement described in the method. Figure 3 also states that sorting produces no narration, so the sorting result does not support the conclusion's claim that narration enables generalization to new object counts.
  3. The real-robot sample sizes are small, especially for distinguishing the narration variants. Figure 4 reports fifteen shell-game trials, five drawer episodes, and ten sorting trials per setting, with no uncertainty intervals. Appendix B.1 lists a single training seed. The drawer score of 83.3% cannot be binary whole-episode success over five episodes. If it pools thirty object placements, the improvement from 80% to 83.3% is one additional correct placement, and placements within an episode are correlated. Please clarify the scoring unit, report whole-episode success alongside object-level performance, and estimate uncertainty at the episode level. Repeated training runs are needed to assess the smaller ablation differences.
  4. The million-token timing experiment is not accompanied by a task-success evaluation at that scale. Appendix B.3 gives maximum histories of thirty, 160, and 256 images for the shell game, drawer task, and RoboMME, respectively, sampled at 2 Hz. These correspond to 2,160, 11,520, and 8,192 historical visual tokens. The paper therefore demonstrates memory-dependent behavior within these budgets, but does not show how performance changes as history grows. A controlled sweep of history length and sampling rate would establish how much of the retained history is needed. The limited scenes and viewpoints, acknowledged in the conclusion, also constrain the generalist interpretation.
  5. The control budget needs to be defined more precisely. Figure 5 shows a large benefit from caching, but latency reaches about 1.8 seconds at one million tokens. Section 5 acknowledges that this exceeds its one-second budget, while the figure marks a 500-ms reference. The connection between these thresholds and the actual robot control schedule is unclear. Please specify the control frequency, the portion of each thirty-step action chunk executed before replanning, and which processing stages are included in the latency measurement. A matched single-image baseline and end-to-end latency distribution would substantiate the abstract's efficiency comparison.
  6. Several details needed to reproduce the experiments are missing from the supplied manuscript. These include the real-robot demonstration counts, exact held-out split construction, subtask annotation procedure, and training or adaptation settings for the real-robot baselines. Sorting has no configuration column in Appendix B. Table 1 should distinguish results taken from other papers from local reruns. Appendix B.7 also describes the RoboMME tokenizer as a continuation of an earlier fit without giving its complete fitting schedule.
Questions for authors
  1. Why is the complete native-video SimpleMemVLA absent from Table 1? Which baseline results were rerun, and which were taken from the cited papers?
  2. Does the final model use Equation (5) alone or a joint autoregressive/parallel training objective? Which model variant is used for each evaluation?
  3. What happens when the narration loss is removed while the number of reserved positions and the remaining training setup are held fixed?
  4. Is drawer success measured per object or per episode? What are the whole-episode results and uncertainty intervals?
  5. How does task success change when history length or sampling density is reduced?
  6. What is the replanning deadline, how are inference and action execution scheduled, and what does the reported latency include?
  7. How many demonstrations train each real-robot policy, and how were the held-out sequences and assignments selected?
7 comments

Q1. Why is the complete native-video SimpleMemVLA absent from Table 1? Which baseline results were rerun, and which were taken from the cited papers?

Thank you for pointing this out. We have added the complete SimpleMemVLA result to Table 1.

We have also trained SimpleMemVLA on our real-robot tasks, where it is the strongest baseline. LongContextVLA achieves higher success in all three settings, with 30 trials each:

SettingSimpleMemVLASubtask only (our ablation)LongContextVLA
Cup swap, unseen 5 swaps (full task)60.0%53.3%76.7%
Cup swap, unseen 6 swaps (full task)43.3%36.7%70.0%
Drawer, unseen configurations50.0%63.3%76.7%

SimpleMemVLA performs comparably to our subtask-only ablation. This is expected, because the two are closely related: both condition on the full video history and predict a subtask before acting, and neither is trained to narrate its history. The main difference is that SimpleMemVLA generates actions with an action expert, while our model predicts discrete action tokens. Our main finding is that narration improves generalization to unseen scenarios, and all three settings above test unseen scenarios, which is where LongContextVLA outperforms SimpleMemVLA.

All baseline results in Table 1 are taken from the cited papers; only the LongContextVLA row was evaluated by us. The real-robot baselines were trained and evaluated by us. We now state this in the table captions.

For context, we would also like to note that SimpleMemVLA is concurrent work: it appeared on arXiv on September 2, 2026, two weeks before the submission deadline. The ICLR reviewer guidelines (https://iclr.cc/Conferences/2027/ReviewerGuidelines) treat papers from the two months before the deadline, and papers available only on arXiv, as contemporaneous, and state that "the lack of such comparisons cannot be a basis for rejection." We nonetheless agree that the comparison is informative and are glad to include it. We would be grateful if the reviewers could take this into consideration.

Q2. Does the final model use Equation (5) alone or a joint autoregressive/parallel training objective? Which model variant is used for each evaluation?

Thank you for catching this; the manuscript was unclear. The final model uses a joint objective, not Eq. (5) alone. Each model variant is trained with the following objectives:

ModelTraining objectiveInference
SubtaskL_sub (Eq. 3) onlyxt↦(ut,At)x_t \mapsto (u_t, A_t)
Subtask + narrationL_sub or L_narr (Eqs. 3, 4), each with probability 12\tfrac{1}{2}xt↦(ut,At)x_t \mapsto (u_t, A_t); autoregressive narration is an auxiliary training loss only
Subtask + narration + latent narration (ours)L_sub, L_narr, or L_par (Eqs. 3, 4, 5), each with probability 13\tfrac{1}{3}(xt,PK)↦(rt,ut,At)(x_t, P_K) \mapsto (r_t, u_t, A_t), with rt decoded in parallel from the K reserved positions

No model generates narration autoregressively at inference. The results reported as "ours" on the cup swap task, the drawer task, RoboMME, and the latency measurements in Figure 5 all use the third model with parallel decoding.

The sorting task is the exception: it uses no narration, and the policy is trained with L_sub (Eq. 3) only. It is intended to show that the memory context can serve as a visual prompt, not as evidence for narration helping with prompting task (although that is an interesting direction for future work).

Q3. What happens when the narration loss is removed while the number of reserved positions and the remaining training setup are held fixed?

Thank you for asking this interesting question. We have not run this ablation. Without the narration loss, the K reserved positions receive no supervision and act only as extra computation before the subtask and actions. Chen et al. (2025) [1] tested this directly when studying why embodied reasoning helps VLAs: adding non-semantic "thinking tokens", matched in number to the reasoning tokens, did not improve over a standard VLA on LIBERO-90 and reduced success by 3.8 points on average, while supervised reasoning improved it. They conclude that the benefit comes from learning semantically meaningful reasoning, not from added compute. This is why we supervise the reserved positions with narration.

We acknowledge that their setting differs from ours (a smaller VLA and no long visual history), so this is supporting evidence, not a controlled comparison. During the rebuttal period we prioritized other experiments that address the main concerns raised across the reviews, and could not complete this ablation in the same window. We will discuss this point and the evidence above in the revised paper.

[1] William Chen, Suneel Belkhale, Suvir Mirchandani, Oier Mees, Danny Driess, Karl Pertsch, and Sergey Levine. Training Strategies for Efficient Embodied Reasoning. Conference on Robot Learning (CoRL), 2025. arXiv:2505.08243.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Main response from authors

General response to all reviewers

We thank all three reviewers for their careful reading and constructive feedback. The reviews raised several shared concerns, and we have revised the paper to address them. We summarize the main changes here and respond to each question individually below.

1. Comparison with SimpleMemVLA. We have added the complete SimpleMemVLA result to Table 1 (88.3% on RoboMME, compared with 84.9% for our method). We also trained SimpleMemVLA on our real-robot tasks for a comparison under consistent settings. There, LongContextVLA achieves higher success in all three unseen settings: 76.7% vs. 60.0% on five-swap cup sequences, 70.0% vs. 43.3% on six-swap cup sequences, and 76.7% vs. 50.0% on the drawer task. SimpleMemVLA performs comparably to our subtask-only ablation, which supports our main finding that narrating the history improves generalization to unseen scenarios.

We believe the RoboMME gap mainly reflects a difference in action representation, not in memory. RoboMME demonstrations are scripted, so their trajectories are clean parametric curves and straight-line paths with limited spatial coverage. Our model predicts discrete action tokens, which requires fitting a tokenizer to these action chunks, and any tokenizer has some reconstruction error. On such narrow trajectory distributions, small reconstruction errors can move the robot off the demonstrated paths, where the policy has little data to recover. SimpleMemVLA instead generates continuous actions with an action expert, which does not have this quantization error. Our per-task results are consistent with this explanation: peg insertion, the task that demands the most precision, is our lowest-scoring task at 18%, while our other fifteen tasks average 89.3%. Real-robot demonstrations are more varied, and there our model outperforms SimpleMemVLA.

We would also like to respectfully note that SimpleMemVLA is concurrent work that appeared on arXiv two weeks before the submission deadline; under the ICLR reviewer guidelines, "the lack of such comparisons cannot be a basis for rejection." We nonetheless agree that the comparison is informative and are glad to include it.

2. A harder drawer task with a clearer protocol. We replaced the original drawer evaluation with a more demanding task in which the robot must return objects identified by how often, or in what order, they were picked up / used. Each real-robot trial is one request and one object return, scored as a single binary outcome. We report 30 real-robot trials per method, together with an offline evaluation on 58 held-out demonstrations. Videos of the updated drawer task are available on our project website, which is linked in the abstract of the paper.

3. Larger sample sizes and uncertainty estimates. We doubled the real-robot cup swap evaluation from 15 to 30 trials per setting, added offline cup evaluations on 150 five-swap and 100 six-swap held-out sequences, and now report standard errors in Figure 4.

4. Clarified method and experimental details. We now specify the training objectives and the inference mode of every model variant, the control schedule and what the reported latency includes, and the number of demonstrations and the held-out splits for each real-robot task.

5. Planned additions. Several requested experiments could not be completed during the rebuttal period, because we prioritized the core results above. We plan to include them in the final version: a controlled sweep of history length and sampling rate, an ablation that keeps the reserved positions but removes the narration loss, an analysis of how the decoded narration relates to the policy's decisions, and evaluations on additional memory benchmarks.

We hope these changes address the reviewers' concerns, and we are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.