acceptodds
Under review as a conference paper at ICLR 2027

FAME-VLA: Full-history Adapter for Memory -Enhanced vision-language-action Models

31 views

Abstract

Pretrained Vision-Language-Action (VLA) models have shown strong promise for robot manipulation. However, most existing VLAs condition primarily on the current observation or a short window of recent observations, limiting their ability to resolve history-dependent ambiguities in long-horizon tasks. To address this, we introduce , a lightweight full-history adapter that equips pretrained VLAs with dual-stream memory for history-aware action generation. Our adapter combines two complementary memory branches: full-history temporal memory, which compresses the complete observation history into a compact task context, and persistent initial-scene memory, which preserves access to spatial details that may become occluded or altered during execution. These history-aware representations are injected exclusively into the action expert, making our adapter a lightweight, plug-and-play module for pretrained VLAs. To efficiently scale to full observation histories, we precompute history features once per trajectory during training. Experiments on simulation benchmarks and real-world manipulation tasks demonstrate consistent improvements across pretrained VLA backbones, achieving average success rates of 56.11% with StarVLA and 64.89% with on RMBench. Further analyses demonstrate the complementary roles of the two memory branches and the effectiveness of full-history conditioning, while history sharing provides up to a training speedup over independently encoding each observation prefix.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Full History Memory

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aGnxOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

FAME-VLA augments pretrained VLA policies with two complementary memory branches: a causal Transformer encoding the observation history and a persistent initial-scene memory queried by current visual features. Their outputs condition the action expert through zero-initialized adaptive normalization. Episode-wise history sharing avoids repeatedly encoding overlapping prefixes. Experiments on RMBench and three physical manipulation tasks show substantial improvements, supported by ablations of memory branches, spatial pooling, conditioning mechanisms, and history length.

Critical strengths
  1. The paper is clearly motivated: it equips VLA models with a lightweight yet comprehensive representation of historical context to support long-horizon robotic execution. In addition, the manuscript is well written, and the figures and tables are clearly presented and easy to follow.
  2. With π₀, success increases from 7.11% to 64.89% on RMBench and from 10.00% to 71.67% on physical tasks. Results with StarVLA further support applicability across different policy implementations.
  3. Action-expert modulation incorporates history without expanding VLM inputs. Causal episode-wise encoding provides a sensible way to share computation across supervised timesteps while preserving observation causality.
Critical weaknesses
  1. The experimental evaluation is limited in both simulated and real-world settings. The proposed method is evaluated on only nine tasks from RMBench and three real-world tasks, which does not provide sufficiently broad evidence to substantiate its effectiveness and generalizability. The authors are encouraged to conduct additional evaluations on established simulation benchmarks, such as LIBERO, CALVIN, RoboTwin, and RLBench, as well as on a more diverse set of real-world tasks.
  2. All three real-world tasks are relatively simple and short-horizon, and therefore do not adequately demonstrate the effectiveness of the proposed historical-memory mechanism, particularly for the long-horizon execution setting that motivates this work. In addition, the real-world evaluation includes an insufficient number of baseline methods, making it difficult to assess the relative effectiveness of FAME-VLA under practical deployment conditions. More comprehensive comparisons with representative VLA and memory-augmented approaches on the same real-world tasks are needed.
  3. Incorporating historical memory into VLA models is conceptually intuitive and easy to understand. However, the manuscript lacks sufficiently thorough comparative analysis in two respects. First, it does not compare the proposed method against a baseline that directly incorporates the complete sequence of historical images, in terms of both task performance and computational efficiency. Such a comparison is necessary to clarify whether the proposed memory mechanism offers meaningful advantages over simply retaining the full visual history. Second, the paper lacks an in-depth comparison with existing memory-centric VLA methods, such as MemoryVLA, OptimusVLA, and Mem-0. For example, how does FAME-VLA perform relative to these memory-based baselines across the evaluated benchmarks? More importantly, how does its memory construction differ from existing approaches, and what are the specific methodological innovations beyond prior work?
Questions for authors

As noted in the weaknesses, the reviewer recommends expanding the evaluation to include additional benchmarks and real-world tasks, incorporating a broader set of baseline methods, and providing more in-depth comparisons with existing memory-centric VLA models.

5 comments

We thank the reviewer for the careful evaluation and constructive suggestions. Below, we present additional experiments, clarify the scope of our evaluation, and provide more detailed comparisons with direct frame stacking and existing memory-augmented VLA methods.

W1:Additional Evaluation Benchmark

To extend our experiments beyond RMBench, we conducted additional evaluations on three memory-dependent tasks from RoboDojo [1]. We selected these tasks because they directly test the capability targeted by FAME: using historical information to make decisions when the current observation alone is insufficient.

For each task, we initialize FAME-VLA from π₀-base and fine-tune it on 100 demonstrations. We evaluate a fixed checkpoint over 50 rollouts per evaluation seed across three seeds, giving 150 rollouts per task. The complete-task success rates, averaged across evaluation seeds, are reported below. We also include EventVLA [3], a keyframe-based memory-augmented VLA, as an additional reference.

TaskMemory requirementπ₀π₀.₅StarVLAEventVLAπ₀ + FAME
Cover BlocksRemember occluded block colors and uncover them in the specified order.6.00%13.33%10.00%0.00%28.00%
Match And Pick From ConveyorRemember an earlier object and identify its subsequent match.6.00%14.00%4.67%11.33%12.00%
Swap BlocksTrack progress through sequential block transfers and button presses.0.00%0.00%0.00%14.67%9.33%
Average over these three tasks—4.00%9.11%4.89%8.67%16.44%

π₀ + FAME achieves nonzero success on all three tasks, with higher observed success rates than the published π₀ leaderboard results. As FAME augments an existing VLA with historical context, absolute task success also depends on the underlying policy’s perception and manipulation capabilities. The RoboDojo results provide additional evidence of FAME’s applicability to memory-dependent tasks beyond RMBench.

We will include these additional results, task descriptions, and evaluation details in the revised manuscript.

W2: Real-World Task Horizons and Evaluation Scope

We report episode-length statistics over 50 expert demonstrations per task, with durations calculated at 15 Hz:

TaskMean steps/demoStep rangeMean duration (s)
Swap Blocks876.24758–1,28558.42
Put Back Block618.08484–71041.21
Press Button382.35282–51625.49

Swap Blocks averages nearly one minute, with the longest demonstration lasting 85.67 seconds. Beyond duration, these tasks require historical information: Press Button involves tracking repetitions, Put Back Block requires recalling an earlier object location, and Swap Blocks requires tracking progress and object–location correspondence. Although the manipulation primitives are simple, visually similar observations can require different actions depending on history.

Our real-world evaluation tests whether FAME enables a pretrained VLA to use historical information to complete memory-dependent tasks on a physical robot. With the same π₀ backbone, adding FAME improves success on all three tasks, raising average success from 10.00% to 71.67% and supporting this capability under physical deployment.

Real-robot deployment code for some competing methods is not publicly available, and we were therefore unable to reproduce their deployment on our platform. We acknowledge that the current evaluation remains limited in task diversity and baseline coverage, and does not establish superiority over other memory-augmented methods on physical robots. We provide complementary quantitative comparisons on RMBench in W3_2 and will clarify this evaluation scope alongside the episode-length statistics in the revised manuscript.

[1] Chen, Tianxing, et al. "RoboDojo: A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies." arXiv preprint arXiv:2607.04434 20262026.

[2] Li, Zaijing, et al. "Global prior meets local consistency: Dual-memory augmented vision-language-action model for efficient robotic manipulation." arXiv preprint arXiv:2602.20200 20262026.

[3] Yang, Ganlin, et al. "EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies." arXiv preprint arXiv:2606.20092 (2026).

W3_1: Comparison with Direct Frame Stacking

We agree that direct history concatenation is an important baseline. We evaluate an 8-frame π₀ variant across all nine RMBench tasks and a 4-frame variant on Put Back Block, Swap T, and Cover Blocks. Both variants concatenate main-camera frames sampled at 1 s intervals, together with two current wrist-camera views, using the same pretrained initialization and action objective as vanilla π₀. All variants use 50 expert demonstrations and 100 evaluation rollouts per task.

Taskπ₀π₀ + Multi-frame (8)π₀ + FAME
Observe and Pick Up8%7%1%
Rearrange Blocks10%2%97%
Put Back Block16%13%81%
Swap Blocks8%0%98%
Swap T13%5%96%
Battery Try9%2%40%
Blocks Ranking Try0%0%91%
Cover Blocks0%3%79%
Press Button0%0%1%
M(1) average11.00%5.40%74.60%
M(n) average2.25%1.25%52.75%
Overall average7.11%3.56%64.89%
Taskπ₀ + Multi-frame (4)π₀ + Multi-frame (8)
Put Back Block13%13%
Swap T3%5%
Cover Blocks1%3%
Average5.67%7.00%

Computational cost. Increasing the input from 4 to 8 frames raises peak GPU memory from 34.41 to 48.26 GiB and nearly doubles training time per update 53˙9to106˙6s5\.39 to 10\.66 s. Given these costs, we could not evaluate direct concatenation of the complete observation history within our computational budget and therefore used bounded-history baselines.

Task performance. The 8-frame baseline achieves 3.56% average success across all nine tasks, compared with 7.11% for vanilla π₀ and 64.89% for π₀ + FAME. On the three-task subset, the 4-frame and 8-frame variants achieve 5.67% and 7.00%, respectively, compared with 9.67% for vanilla π₀. Thus, increasing the directly concatenated history incurs higher training costs without improving average success over vanilla π₀ .

FAME encodes history through a dedicated memory module without extending the VLM input sequence, enabling full-history conditioning within our training resources. The results support its effectiveness relative to the tested direct-input baselines. We will include these performance and cost measurements, the baseline configurations, and the scope of this comparison in the revised manuscript.

W3_2: Comparison with Existing Memory-Augmented VLA Methods

  • MemER uses a high-level VLM to select historical keyframes and predict subtasks for a separately fine-tuned VLA. Training requires subtask annotations and semi-automatically generated keyframe supervision.

  • Mem-0, in its hierarchical configuration, combines a planner retaining completed-subtask descriptions and boundary observations with an executor using anchor and sliding-window memory. The models are trained separately with subtask supervision and coordinated through a subtask-end classifier.

  • HAMLET augments pretrained VLAs with moment tokens and temporal memory. Moment tokens are initialized through time-contrastive learning before policy fine-tuning; in its GR00T N1.5 configuration, the VLM and moment tokens remain frozen during fine-tuning.

  • MemoryVLA retrieves historical information from a perceptual–cognitive memory bank and uses gated fusion to condition a diffusion action expert. Similar adjacent entries are merged when the bank reaches capacity.

  • OptimusVLA[2] retrieves demonstration trajectories to construct global action priors and models action history for local consistency. Training comprises VLA pretraining, prior-retrieval learning, and local-memory learning.

FAME combines three complementary properties:

  1. Full-history conditioning with persistent initial-scene memory. FAME causally encodes the complete observed history while retaining a separate branch for initial spatial details, without relying on explicit keyframe selection or subtask-boundary detection.

  2. A modular adapter for pretrained action policies. FAME retains the pretrained backbone and action expert, injecting memory through layer-wise modulation without extending the VLM input sequence or adding a separate planner. We demonstrate this adaptation on both π₀ and StarVLA. While HAMLET also supports pretrained-policy adaptation, FAME combines it with full-history and initial-scene memory learned in a single stage.

  3. Joint memory learning through the original action objective. Action supervision jointly optimizes spatial aggregation, temporal memory, and policy-adaptation parameters, without additional keyframe or subtask annotations or a separate contrastive initialization stage.

Our RMBench comparisons includes Mem-0, MemER, HAMLET, and MemoryVLA, using matched budgets of 50 demonstrations and 100 evaluation rollouts per task. FAME achieves average success rates of 64.89% with π₀ and 56.11% with StarVLA, compared with 42.00% for Mem-0 and 41.78% for MemoryVLA. Because these methods differ in backbone and pretraining, we also independently train and evaluate π₀, StarVLA, and their respective FAME variants under the same protocol, retaining the same pretrained initialization within each pair. These comparisons assess the benefit of adding FAME to each backbone. Our comparison with OptimusVLA is methodological.

We will expand the manuscript to include these methodological distinctions, supervision requirements, and baseline configurations alongside the empirical comparisons.

[1] Chen, Tianxing, et al. "RoboDojo: A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies." arXiv preprint arXiv:2607.04434 20262026.

[2] Li, Zaijing, et al. "Global prior meets local consistency: Dual-memory augmented vision-language-action model for efficient robotic manipulation." arXiv preprint arXiv:2602.20200 20262026.

Reviewer_EhXHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FAME-VLA, a lightweight plug-in adapter that equips pretrained VLAs with full-history memory. The adapter has two memory branches: (a) a full-history temporal memory, which compresses each frame through learned spatial attention pooling and encodes the complete observation history with a causal Transformer; (b) a persistent initial-scene memory, which keeps the first frame’s spatial tokens and reads them out by cross-attention with the current observation. In RMBench, FAME-VLA significantly improves success rate of StarVLA and π0, and improves real-world success on three Franka tasks.

Critical strengths

Strengthes

  • Full-history conditioning is an intuitive and convincing way to overcome the short-context conditioning of existing VLAs.
  • The method gives consistent gains on two different backbones on RMBench, and it outperforms the recent memory-augmented baselines.
Critical weaknesses

Weaknesses

[W1] Unclear evidence for long-horizon tasks. The paper motivates full-history conditioning with long-horizon tasks, but the evaluated tasks do not appear to require long horizons. For example, RMBench categorizes tasks by memory complexity, rather than by horizon length, and its tasks are single-scene tabletop manipulations. In addition, the history-length study also does not specify the unit of W, i.e., frames versus seconds, so it is hard to tell whether the gain comes from full history itself or simply from a moderately longer window.

[W2] Incomplete inference-cost analysis. Latency is reported only as P95, at a single unspecified history length. Because the causal history encoder processes a sequence that grows with the episode, its cost should depend on history length. The current measurement on short episodes may therefore not reflect deployment on longer tasks. The cached per-frame features and the memory growth over time are also not analyzed.

Questions for authors

Questions

[Q1] What are the episode lengths of the RMBench and real-world tasks, in both control steps and seconds? Please also clarify the unit of W in Table 3.

[Q2] It would be helpful to evaluate FAME-VLA on recent memory benchmarks that require substantially longer histories (e.g., RoboCerebra, RoboMemArena, or RoboDojo), and to show how performance and cost scale with episode length.

[Q3] Beyond P95, how do the average and median end-to-end latency compare to the backbone, and how do encoder latency and memory usage scale with history length?

4 comments

We thank the reviewer for the constructive feedback. Below, we clarify episode lengths and the history-window definition, provide inference-cost analysis across history lengths, and report additional results on RoboDojo.

W1 & Q1: Episode Lengths and History-Window Definition.

We agree that reporting per-task episode lengths is necessary to clarify the temporal scale of our evaluation. Below, we provide statistics over 50 expert demonstrations per task for both RMBench and our real-world tasks.

RMBench episode lengths:

TaskMean steps/demoMean duration (s)Evaluation limit (steps)Limit duration (s)
Battery Try670.2413.401,00020
Blocks Ranking Try1,474.7829.503,50070
Cover Blocks1,021.5420.431,50030
Observe and Pick Up149.462.992505
Press Button526.0410.521,50030
Put Back Block351.247.0250010
Rearrange Blocks401.708.0370014
Swap T342.666.8560012
Swap Blocks600.3412.011,00020

Real-world episode lengths:

TaskMean stepsStep rangeMean time (s)
Swap Blocks876.24758‑1,28558.42
Put Back Block618.08484‑71041.21
Press Button382.35282‑51625.49

Durations are calculated from step counts at 50 Hz for RMBench and 15 Hz for the real-world tasks. Evaluation limits specify the maximum number of steps allowed per episode.

These statistics show that RMBench covers a range of episode horizons, from shorter tasks such as Observe and Pick Up, averaging 149.46 steps 29˙9s2\.99s, to longer tasks such as Blocks Ranking Try, averaging 1,474.78 steps 295˙0s29\.50s, with an evaluation limit of 3,500 steps 70s70s. Our real-world tasks average 382–876 steps 25–58s25–58s. We will add these statistics to make this distinction and the evaluated horizons explicit.

Clarification of W in Table 3

We acknowledge that the unit of W was not clearly specified. W denotes the number of most recent consecutive observation frames provided to FAME’s temporal memory module, and this restriction applies during both training and inference. The Full setting uses all observations available from the start of the episode to the current timestep. As shown in Table 3, performance improves as the available history increases, with Full achieving the highest success rate among the evaluated settings. This ablation supports the benefit of full-history conditioning over the tested finite windows. We will clarify the definition of W and this protocol in the revised manuscript.

W2&Q3:Inference Latency and Memory Scaling

We agree that inference cost should be evaluated across explicit history lengths. We therefore profile 128–4096 observation frames on an NVIDIA A100 GPU, using the same trained Full-history checkpoint and 10 diffusion steps. At each length, P50/P95 latency is computed from 100 measurements after 20 warm-up calls, separately for the history encoder and the complete policy.

During rollout, the history encoder incrementally incorporates each new observation using cached key/value states. To match this deployment behavior, we measure the latency of a single online update at each history length, rather than re-encoding the entire history prefix.

History length (frames)History encoder Mean / P50 / P95 (ms)End-to-end policy Mean / P50 / P95 (ms)Persistent history cache (MiB)
0 (π₀ backbone)0 / 0 / 074.69 / 74.32 / 76.090
1280.82 / 0.79 / 0.8297.99 / 97.87 / 98.764.53
10241.22 / 1.21 / 1.2999.98 / 99.89 / 102.9218.03
20481.17 / 1.17 / 1.2898.58 / 98.45 / 99.5036.03
40961.34 / 1.32 / 1.48100.92 / 101.06 / 102.8572.04

Compared with vanilla π₀, mean end-to-end latency increases from 74.69 ms to 97.99–100.92 ms +312˙–351˙+31\.2–35\.1%, while median latency increases from 74.32 ms to 97.87–101.06 ms +317˙–360˙+31\.7–36\.0% across the tested history lengths. 4096 frames exceeds our benchmark’s largest evaluation limit of 3500 steps. Despite a 32-fold increase in history length, history-encoder P95 remains below 1.5 ms, while end-to-end P95 increases by only 4.1%, from 98.76 to 102.85 ms. Thus, additional history-encoding computation has a limited impact on overall policy latency across the tested range.

Online inference caches compact history tokens, key/value states, and the initial-frame anchor, without retaining all past visual feature maps. Persistent cache storage grows from 4.53 to 72.04 MiB. We will include these results in the revision, clarifying that latency reflects warmed, steady-state inference.

Q2: Additional Evaluation on RoboDojo

Thank you for suggesting additional memory benchmarks. Our primary focus is memory-dependent manipulation, where actions require information from earlier observations. Our evaluation also includes longer execution sequences: as reported in W1/Q1, Blocks Ranking Try averages 1,474.78 steps 295˙0s29\.50 s, while real-world Swap Blocks averages 876.24 steps 584˙2s58\.42 s.

To broaden our evaluation, we evaluate FAME-VLA with π₀ as the backbone of three RoboDojo memory tasks [1]. Each task-specific policy is initialized from π₀-base, fine-tuned on 100 demonstrations, and evaluated using a fixed checkpoint over 50 rollouts per seed across three evaluation seeds (150 rollouts per task).

TaskMemory requirementMean steps/demoMean duration
Cover BlocksRemember occluded block colors and uncover them in the specified order.542.9621.72 s
Match And Pick From ConveyorRemember an earlier object and identify its subsequent match.432.6617.31 s
Swap BlocksTrack progress through sequential block transfers and button presses.435.0917.40 s

Lengths are averaged over all 100 demonstrations per task, with durations calculated at 25 Hz. We also include EventVLA [2], a keyframe-based memory-augmented VLA, as an additional reference. Complete-task success rates are:

Taskπ₀ (leaderboard reference)EventVLA (keyframe-based)π₀ + FAME
Cover Blocks6.00%0.00%28.00%
Match And Pick From Conveyor6.00%11.33%12.00%
Swap Blocks0.00%0.00%9.33%

π₀ + FAME achieves higher observed success on all three tasks than the π₀ leaderboard reference. It reaches 28.00% on Cover Blocks and 9.33% on Swap Blocks, compared with 0.00% for EventVLA on both tasks, while achieving similar performance on Match And Pick From Conveyor (12.00% vs. 11.33%).

These results support FAME’s applicability to additional memory-dependent tasks beyond RMBench. Computational scaling with history length is reported separately in W2/Q3. We will include these additional results and task descriptions in the revised manuscript.

[1] Chen, Tianxing, et al. "RoboDojo: A unified sim-and-real benchmark for comprehensive evaluation of generalist robot manipulation policies." arXiv preprint arXiv:2607.04434 20262026.

[2] Yang, Ganlin, et al. "EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies." arXiv preprint arXiv:2606.20092 (2026).

Thank you for the clarifications and additional results. The cost analysis across history lengths fully addresses W2/Q3. Two points from W1/Q2 remain open.

First, the benefit of full history over a sufficiently long window is still unclear. Since W counts 50 Hz frames, the tested windows may cover only a few seconds, so Table 3 may compare a very short window against full history rather than a moderately long one. Windows defined relative to episode length (e.g., 25/50/75%), or analysis on tasks where the needed information appears early in the episode, would isolate the effect of full history.

Second, long-horizon capability is shown only in terms of cost. All performance evaluations, including the new RoboDojo tasks, remain under about one minute, and it is untested whether the history encoder generalizes to histories longer than those seen in training.

Overall, evaluating on longer tasks or clarifying the long-horizon framing in the paper is required.

Thank you for the follow-up and constructive feedback. We address the two remaining points with additional experiments and clarifications.

R1: Longer temporal windows and the benefit of full history

To resolve the concern that tested windows may be too short, we have added evaluations with window sizes proportional to each task’s mean training-demonstration length 2525%, 50%, 75%. All settings use the same full-history-trained checkpoint, with temporal memory restricted only during inference. The persistent initial-scene branch and model parameters remain unchanged across all conditions.

Window size relative to training mean lengthSwap T M(1)M(1)Put Back Block M(1)M(1)Cover Blocks M(n)M(n)Blocks Ranking Try M(n)M(n)
25%14%10%9%40%
50%84%63%67%48%
75%93%78%73%57%
Full history96%81%79%91%

The two M(1)M(1) tasks approach full-history performance with the 75% window, as their key initial spatial references remain accessible via the initial-scene branch. For M(n)M(n) tasks, the advantage of full history scales with the demand for accumulated execution context:

  • Cover Blocks benefits monotonically from longer windows, as additional temporal context supports tracking of sequential execution and color–location correspondences under occlusion.

  • The clearest gap appears on Blocks Ranking Try, where full history outperforms the 75% window by 34 percentage points. This task relies on iterative trial-and-error; truncating history erases earlier attempts and their outcomes, making it harder to avoid repeating unsuccessful arrangements.

These results show that the value of full history is task-dependent. It is modest for tasks where critical information is concentrated early in the episode, and substantial for tasks requiring cumulative trial-and-error memory across the full episode.

R2: Clarifying the Scope of Our Long-Horizon Claims:

We agree that our use of “long-horizon” needs clarification. Our experiments focus on memory-dependent manipulation and have not tested generalization to histories longer than those seen during training. In these tasks, the current observation alone does not provide enough information to choose the correct action. FAME equips pretrained VLAs that lack memory with the ability to retain and use historical information for action generation. This capability can matter even in short episodes, and the new window results show that the benefit of retaining earlier history varies across tasks.

We will revise the manuscript to make this focus clearer and distinguish the memory capabilities demonstrated in our experiments from the ability to handle substantially longer tasks or histories beyond those seen during training.

Reviewer_3XggOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FAME-VLA augments pretrained VLAs with a full-history temporal adapter and a persistent first-frame spatial memory. Compressed visual and proprioceptive features pass through a causal Transformer, while the current frame queries the initial spatial tokens. The resulting condition modulates the action expert through zero-initialized AdaLN. Experiments report large gains on RMBench with π0 and StarVLA, together with robot validation and memory-design ablations.

Critical strengths
  1. The method addresses an important limitation of short-window VLAs and reports large, consistent improvements with two backbones. The gain is sufficiently substantial to make this a useful practical contribution even though the individual components are familiar.

  2. The ablations examine both memory branches, pooling, conditioning interfaces and history length. The design avoids extending the VLM token sequence and has a reported latency overhead of about 18%, giving the empirical contribution a plausible deployment path. These considerations lead me to lean toward acceptance.

Critical weaknesses
  1. The baseline protocol needs clarification to establish the advantage over existing memory methods. Different backbones, pretraining and demonstration budgets can confound the comparison. A matched-backbone memory baseline and a strong short-window or keyframe baseline across the full task set would better isolate the adapter. The robot comparison against π0 alone also does not show an advantage over other memory-augmented approaches.

  2. Statistical reliability and the type of memory generalization remain uncertain. One training run per method does not characterize training variability; finite rollout/trial counts also leave uncertainty about task success. With few demonstrations and full history, timestep or episode-specific cues could explain part of the benefit. The persistent first-frame branch further assumes useful information is initially visible. The established nature of the components and causal-pass sharing should be reflected in the novelty and speedup framing.

Questions for authors
  1. Which baseline results were re-trained and evaluated under the same 50-demonstration protocol? Can you identify a same-backbone memory comparison and report the strong windowed baseline across the complete task set?

  2. What uncertainty estimates are available for task success, and are the large gains stable across training seeds? Please distinguish differences between tasks from variation across repeated runs.

  3. How does performance change when executions are delayed or perturbed, repetition counts change, or relevant information is absent from frame 0? These tests would help distinguish useful memory from timestep cues. Please also report long-horizon episode lengths and memory/latency growth.

6 comments

We thank the reviewer for the constructive feedback. Below, we clarify the baseline protocols and provide additional experiments on statistical reliability, memory dependence, and computational efficiency.

W1 & Q1: Baseline Protocol and Matched-Backbone Comparisons

Baseline Protocol:

We follow the RMBench protocol of 50 expert demonstrations per task and 100 evaluation rollouts per task. We also independently train and evaluate π₀, StarVLA, and their FAME variants under this protocol. For Mem-0, MemoryVLA and MemER, we use previously reported RMBench results [1][2]. We retrain HAMLET, MTIL, and CycleManip using their released implementations, with 50 expert demonstrations per task and 100 evaluation rollouts per task.

FAME is designed as a plug-in memory adapter for pretrained VLAs, retaining each backbone’s original action prediction objective. We evaluate it on both π₀ and StarVLA, keeping the pretrained initialization unchanged between each vanilla backbone and its FAME-augmented counterpart. Improvements on both backbones demonstrate the benefit of adding FAME and support its applicability across different VLA architectures.

Matched-Backbone Comparisons:

We agree that a windowed baseline with the same pretrained backbone would provide a more informative comparison. We evaluate π₀ with 4 or 8 main-camera frames sampled at 1 s intervals, alongside both current wrist-camera views. Both variants follow vanilla π₀’s pretrained initialization, action objective, and training settings, using 50 demonstrations and 100 evaluation rollouts per task. The 8-frame baseline covers all nine RMBench tasks; the 4-frame baseline covers Put Back Block, Swap T, and Cover Blocks.

Taskπ₀π₀ + Multi-frame (8)π₀ + FAME
Observe and Pick Up8%7%1%
Rearrange Blocks10%2%97%
Put Back Block16%13%81%
Swap Blocks8%0%98%
Swap T13%5%96%
Battery Try9%2%40%
Blocks Ranking Try0%0%91%
Cover Blocks0%3%79%
Press Button0%0%1%
M(1) average11.00%5.40%74.60%
M(n) average2.25%1.25%52.75%
Overall average7.11%3.56%64.89%

The 8-frame baseline achieves 3.56% overall success, compared with 7.11% for vanilla π₀ and 64.89% for π₀ + FAME. On the three-task subset below, both the 4-frame and 8-frame baselines also remain below vanilla π₀. Thus, direct frame stacking does not improve average performance under the same evaluated protocol, while FAME provides substantial gains with the same backbone.

History-Window Analysis and Clarification of WW:

The FAME ablation in Table 3 examines a different use of history: WW counts the most recent consecutive observation frames supplied to FAME’s temporal memory module during both training and inference. Full uses all observations from the episode’s start to the current timestep. Unlike the multi-frame baselines, FAME encodes history without extending the VLM input sequence. We present both sets of results together below.

SettingPut Back BlockSwap TCover BlocksAverage
π₀16%13%0%9.67%
π₀ + Multi-frame (4)13%3%1%5.67%
π₀ + Multi-frame (8)13%5%3%7.00%
π₀ + FAME, (W=8)62%53%1%38.67%
π₀ + FAME, (W=16)69%46%22%45.67%
π₀ + FAME, (W=32)81%76%25%60.67%
π₀ + FAME, Full81%96%79%85.33%

Within FAME, average success increases from 38.67% → 45.67% → 60.67% → 85.33% as history expands from W=8W=8 to W=16W=16, W=32W=32, and Full. Full achieves the best result on every task. Together, these experiments support FAME’s advantage over the tested direct frame-stacking baselines and the benefit of full-history conditioning over finite temporal windows within FAME.

We will incorporate these additional results and clarify the sampling protocols and definition of WW in the revised manuscript.

[1] Chen, Tianxing, et al. "Rmbench: Memory-dependent robotic manipulation benchmark with insights into policy design." arXiv preprint arXiv:2603.01229 20262026.

[2] Yang, Ganlin, et al. "EventVLA: Event-Driven Visual Evidence Memory for Long-Horizon Vision-Language-Action Policies." arXiv preprint arXiv:2606.20092 20262026.

W2 & Q3: Memory Dependence and Sensitivity to Perturbations

We evaluate FAME’s robustness to an initial execution delay and noisy temporal-history conditioning, and assess each memory branch’s contribution through separately retrained ablations. All experiments use the π₀ backbone, with 100 evaluation rollouts per task.

For the perturbation experiment, we add Gaussian noise to the full model’s temporal-history conditioning vector before AdaLN:

$ \widetilde{\mathbf h}_t=\mathbf h_t+\sigma\boldsymbol\epsilon_t, \qquad \boldsymbol\epsilon_t\sim\mathcal N(\mathbf0,\mathbf I). $

Fresh noise is sampled at each policy query. The initial-frame memory branch and stored history cache remain unchanged. The two ablation columns report separately retrained models with the corresponding branch removed, while the σ columns report inference-time perturbations of the full model.

For the delay evaluation, we hold the robot’s initial joint positions and gripper states for 16 control steps at the beginning of each episode, while continuing to incorporate observations into temporal memory. The policy then resumes with a freshly predicted action chunk; the evaluation budget is extended by 16 steps to preserve the available task-execution budget. All results below are success rates %.

TaskNo perturbationInitial delay: 16 stepsσ=0.10σ=0.20σ=0.30w/o initial-frame memoryw/o temporal memory
M(1) tasks
Observe and Pick Up1212225
Rearrange Blocks97949897969016
Put Back Block81967985856512
Swap Blocks98919897968524
Swap T96979295918556
M(n) tasks
Battery Try40353838333112
Blocks Ranking Try9187878368760
Cover Blocks7984807466200
Press Button1001020
M(1) average74.67673.675.27465.422.6
M(n) average52.7551.551.254941.7532.253
Overall64.8965.1163.6763.5659.6750.6713.89

Complementary contributions of the two memory branches. Removing temporal memory reduces overall success from 64.89% to 13.89%, with Blocks Ranking Try and Cover Blocks dropping from 91% and 79% to 0%, respectively. These results indicate that conditioning on current observations and initial-frame memory is insufficient for these tasks, supporting the need to retain information accumulated during execution on these RMBench tasks. Removing initial-frame memory also lowers overall success to 50.67%, further supporting the complementary roles of the two branches.

Sensitivity to temporal-history perturbations. At σ=0.30, overall success decreases from 64.89% to 59.67%, with a larger decline on Mnn tasks 527˙552\.75% → 41\.75% than on M11 tasks 746˙074\.60% → 74\.00%. The largest decreases occur in Blocks Ranking Try 9191% → 68% and Cover Blocks 7979% → 66%, which also show strong dependence on temporal memory in the ablation. This consistent pattern supports the role of temporal-history conditioning in guiding action generation.

Robustness to an initial execution delay. FAME maintains comparable overall success under the 16-step initial delay 648˙964\.89% vs\. 65\.11%, showing robustness to the tested timing offset. During the hold, temporal memory continues to advance without corresponding task progress. The preserved performance supports FAME’s ability to tolerate this misalignment between history length and execution progress.

Q3: Episode Lengths and Inference Costs

Episode lengths. We report mean episode lengths over 50 expert demonstrations per task, with durations calculated at 50 Hz for RMBench and 15 Hz for real-world tasks.

TaskMean stepsMean duration (s)
RMBench
Battery Try670.2413.4
Blocks Ranking Try1,474.7829.5
Cover Blocks1,021.5420.43
Observe and Pick Up149.462.99
Press Button526.0410.52
Put Back Block351.247.02
Rearrange Blocks401.78.03
Swap T342.666.85
Swap Blocks600.3412.01
Real-world
Swap Blocks876.2458.42
Put Back Block618.0841.21
Press Button382.3525.49

RMBench includes tasks averaging over 1,000 steps, while our real-world tasks average 25–58 s. The largest RMBench evaluation limit is 3,500 steps 70s70 s.

Memory and latency growth. We profile the same full-history checkpoint on an NVIDIA A100 with 10 diffusion steps. P95 latency is computed from 100 measurements after 20 warm-up calls. History-encoder latency measures one incremental online update using cached key/value states.

History length (frames)History encoder P95 (ms)End-to-end policy P95 (ms)Persistent history cache (MiB)
π₀ backbone reference—76.09—
1280.8298.764.53
1,0241.29102.9218.03
2,0481.2899.536.03
4,0961.48102.8572.04

From 128 to 4,096 frames, history-encoder P95 remains below 1.5 ms, and end-to-end P95 increases by 4.1%, while persistent cache storage grows from 4.53 to 72.04 MiB. The cache stores compact history representations and key/value states without retaining all past visual feature maps. These results characterize history-length scaling beyond the benchmark’s largest evaluation limit. We will include these statistics in the revision and clarify that latency reflects warmed, steady-state inference.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.