acceptodds
Under review as a conference paper at ICLR 2027

Mem4Act: Learning Effective Latent Memory for Long-Horizon Robotic Manipulation

49 views

Abstract

Long-horizon robotic manipulation fundamentally depends on memory: agents must retain information from past experiences and interactions and take correct actions. Existing approaches typically preserve historical observations, compressed visual tokens, or language summaries, but these representations either expose the action policy to increasingly long histories or risk discarding task-relevant temporal, semantic, and perceptual information. We introduce Mem4Act, a framework for learning effective latent memory as a fixed-size representation explicitly optimized for long-horizon robotic control. Mem4Act addresses three coupled challenges: how to cover long histories, what information to retain, and how to ensure the policy actually uses memory. It organizes history through multi-granularity temporal aggregation, preserves complementary semantic and perceptual information through question answering and key-frame DINO feature reconstruction, and adopts memory-first two-stage training to mitigate reactive shortcuts. The resulting representation captures both high-level task state and fine-grained perceptual evidence while maintaining a bounded interface to the action policy. Across RoboMME, RMBench, and four real-world manipulation tasks, Mem4Act achieves state-of-the-art performance in memory-dependent control, reaching success rates of 49.88%, 61.0%, and 81.50%, respectively. Ablations further demonstrate the importance of temporal structure, complementary memory supervision, and memory-first two-stage learning.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning, Vision-Language-Action Models, Long-Horizon Manipulation, Memory-Augmented Policy

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_odnQOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Mem4Act, a framework that learns fixed-size latent memory for history-dependent robot manipulation. It pretrains a memory encoder with multi-granularity temporal attention using historical QA and keyframe DINO feature reconstruction, then jointly adapts it with a pretrained π0.5\pi_\text{0.5} policy through action and auxiliary supervision, conditioning action generation through feature modulation. The method is evaluated on RoboMME, RMBench, and four real-world manipulation tasks, with additional ablations and memory-utilization analyses.

Critical strengths
  • The paper organizes memory for action policies around coverage, content, and utilization. Its corresponding components and experimental design consistently follow this structure, making the problem formulation and methodology easy to follow.
  • The paper provides detailed analyses of the components used to construct and supervise latent memory. In particular, ablations of temporal structure and experiments that replace the historical context offer valuable insights into memory organization and the policy's reliance on historical information for control.
Critical weaknesses
  • The method relies heavily on subgoal annotations and task-specific heuristics for QA and DINO supervision, including QA templates and keyframes selected at subgoal boundaries, potentially limiting scalability. Further discussion of annotation effort and automation across diverse tasks, together with an analysis of sensitivity to subgoal granularity, would clarify its practical scope.
  • The comparison of QA and DINO supervision in Table 5(a) supports their complementary benefits but largely relies on success rates. Concrete success and failure cases across the supervision variants, showing which semantic facts or visual and spatial details are retained and used, would better support their intended roles.
Questions for authors
  • How well does Mem4Act extend to other VLA backbones, and can broader Stage 1 pretraining be reused across target tasks? Such experiments could further strengthen the framework's practical value.
  • Could the authors provide matched rollout examples with the corresponding historical frames, identifying the cues needed at failure points and assessing whether FrameSamp and Mem4Act retain them? This would help distinguish information lost during sampling or compression from ineffective use of retained context.
6 comments
AuthorsOpenReview ↗

Response to Reviewer odnQ

Thank you very much for your detailed and constructive review. We sincerely appreciate your recognition of the coverage, content, and utilization framing and of our analyses of memory structure and memory use. Below, we respond to each point.

1. Annotation effort, task-specific heuristics, and subgoal granularity (Weakness 1)

We thank the reviewer for raising this concern. We agree that Mem4Act's auxiliary supervision relies on subgoal annotations during training.

Annotation effort. In simulation, subgoal labels come at almost no cost, since scripted demonstration pipelines already track each subtask and its transitions. RoboMME and RMBench both release subgoal descriptions with temporal boundaries, from which we generate all QA pairs and DINO keyframes automatically. For datasets without such labels, automatic video segmentation and captioning offer a practical way to obtain them. To understand how much labeling effort Mem4Act actually needs, we study sensitivity to subgoal granularity below.

Task-specific heuristics. Most of our questions require no task-specific judgement. Three common questions, covering what the robot has done, what it is doing now, and what comes next, are shared across all tasks and answered directly from the completed, current, and next subgoals. Task-specific design enters only through progress questions, such as how many times the cube has reached the target in StopCube. To measure how much Mem4Act relies on these, we remove all task-specific questions in model training.

Sensitivity to subgoal granularity. We ran two tests on RoboMME. First, we removed all task-specific questions, to measure how much they matter. Second, we additionally merged consecutive subgoals, to measure sensitivity to coarser subgoals. Merging two or four subgoals leaves about half or a quarter as many boundaries to annotate, and makes the QA answers and boundary keyframes correspondingly coarser.

Supervision setting (RoboMME)Success rate (%)
Original49.88
w/o task-specific QA48.50
+ 2 consecutive subgoals merged48.13
+ 4 consecutive subgoals merged47.38

Removing the task-specific questions lowers success by only 1.38 points, showing that the shared questions account for most of the benefit. Coarser labels have a similarly small effect, lowering success by a further 0.37 and 1.12 points when two or four subgoals are merged. Even in the most reduced setting, without task-specific questions and with four times coarser labels, Mem4Act reaches 47.38%, which is still above the strongest baseline, MME-VLA (FrameSamp+Modul, 44.51%). These results indicate that Mem4Act is robust to both the choice of questions and the granularity of subgoal labels.

We have added this ablation and its discussion to Supplementary Sec. D.5 (Table S8). We also state the annotation requirement in the Limitations and Future Work paragraph of the revised version, together with automatic annotation as a way to reduce manual supervision.

2. Evidence beyond success rates for the roles of QA and DINO (Weakness 2)

We thank the reviewer for this suggestion. We agree that success rates alone do not show what information the memory retains. We therefore added Figure 5 in the revised version, which compares the three supervision variants on the same StopCube episode and visualizes their memory readouts, giving two failure cases and one success case. Succeeding on this task requires two kinds of information: a semantic fact, the number of times the cube has reached the target, and a visual detail, the cube's motion, which determines when it arrives. We have also added the corresponding discussion to Sec 4.3 Ablation Studies (Do semantic and perceptual objectives complement each other?)

The two single-objective variants fail in opposite ways.

QA only. The memory keeps the count: after the 4th pass, the policy correctly prepares to stop the cube at its 5th arrival. However, without visual detail of the cube's motion, it misjudges the speed and presses too early, so the cube stops short of the target on its 5th approach.

Key-frame DINO only. The memory keeps the visual detail of the cube's motion, and its reconstructions closely match the DINOv3 targets. This lets the policy judge the speed and time the press so the cube stops on the target. However, without semantic progress, it loses count and stops the cube at its 4th arrival instead of the 5th.

QA and key-frame DINO. With both objectives, the policy presses at the right moment of the right pass, and the cube stops on the target at its 5th arrival.

This case makes concrete the complementary benefits in Table 5(a): QA supervision retains semantic progress, while DINO supervision retains the visual details needed for precise timing.

AuthorsOpenReview ↗

Response to Reviewer odnQ

3. Other VLA backbones (Question 1)

We thank the reviewer for this question. To test whether Mem4Act depends on a specific backbone, we applied it to GR00T N1.7 using the same memory architecture, QA and DINO objectives, two-stage training schedule, and conditioning through cross-attention and feature modulation. Historical observations are encoded by GR00T's frozen vision encoder and pooled to 16 tokens per frame, matching our π0.5 setup. For each backbone, the base policy and Mem4Act receive the same number of training steps and are evaluated on all 16 RoboMME tasks with 50 episodes per task.

VLA backboneBase policyWith Mem4Act
GR00T N1.713.50%35.50%
π0.517.93%49.88%

Mem4Act raises success from 13.50% to 35.50% on GR00T N1.7 and from 17.93% to 49.88% on π0.5. The relative gains are similar, at 2.6x and 2.8x of the base success rate. These consistent gains support its applicability as a memory module across both tested VLA backbones. We added this comparison to Supplementary Table S5, with discussion in Supplementary Sec. D.1 (Mem4Act on Other VLA Backbones).

4. Reusable Stage 1 pretraining (Question 1)

To test whether broader Stage 1 pretraining can be reused, we pretrained a shared memory latent on RoboMME and RMBench together, then fine-tuned on RoboMME only during Stage 2.

Stage 1 pretrainingRoboMME
Per-benchmark (RoboMME)49.88
Shared across RoboMME + RMBench46.75

The shared checkpoint achieves 46.75%, compared with 49.88% for RoboMME-specific pretraining, which supports reuse but with a modest reduction of 3.13 points. It also remains 2.24 points above the strongest baseline, MME-VLA (FrameSamp+Modul, 44.51%).

We attribute this gap mainly to benchmark heterogeneity. RoboMME uses a single arm and a third-person memory view with left and right mirrored relative to the robot. RMBench uses dual arms and a head camera without this mirroring. As a result, the same spatial description can refer to opposite sides of the image in the two memory views. Tasks and visual appearance also differ substantially. Combining only these two distinct sources may introduce conflicting supervision, limiting the benefits of shared pretraining.

We added this comparison to Supplementary Table S6, with discussion in Supplementary Sec. D.2 (Reusing Stage 1 Pretraining Across Benchmarks). We expect larger and more diverse pretraining data to improve generalization, which we also highlight as future work in the revised paper.

5. Information lost during sampling vs. ineffective use (Question 2)

We thank the reviewer for this helpful suggestion. We added two matched rollout comparisons on RoboMME, shown in Figs. S2 and S3 and discussed in Supplementary Sec. D.8 (RoboMME Failure Analysis: MME-VLA and Mem4Act). Each shows all historical frames selected by FrameSamp and supplied to Mem4Act's memory encoder at the decision point, with the critical cues highlighted. Both methods provide 512 memory tokens to the policy.

Information lost during sampling (Fig. S2). The robot must place a cube on the target that was visited third in the demonstration, but two targets later swap positions. To find the right target, the policy must see which targets moved during the swap. The episode lasts over 1,200 frames, so FrameSamp's 32 frames are about 40 frames apart, and only one lands inside the 49-frame swap. That frame shows the targets mid-swap but not where each one started or ended, so the policy picks the wrong target. Mem4Act samples every 15 frames, sees the swap from start to finish, knows the third target never moved, and succeeds.

Failure despite retained cues (Fig. S3). The robot must identify the container hiding a green cube after a swap. FrameSamp retains more observations of the uncovered cubes (six versus three) and the swap (eight versus three), yet fails while Mem4Act succeeds. Both sampled histories contain the necessary cues, suggesting that FrameSamp fails to interpret or use the available context effectively.

Together, the two cases separate the two failure sources: in the first, the needed cue is lost during sampling, and in the second, the cue is retained but not used effectively.

Thank you for the detailed response and the extensive additional experiments, which address most of my concerns (W1, W2, Q1, and the matched examples in Q2). I have one remaining question on Q2.

[Q2] The new matched examples show which cues appear in the frames sampled by each method, but both are Mem4Act successes and neither shows which cues remain accessible in its 512 latent tokens. For one or two rollouts on a counting task such as StopCube in which Mem4Act acts on the wrong count, could the authors show whether the relevant events are visible in the sampled history and whether the existing QA decoder returns the correct count at the decision point? This would help narrow down where the cue may be lost.

Overall, the rebuttal substantially strengthens the paper, and I would be glad to maintain my positive assessment once this point is also addressed.

Reviewer_6eHcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Mem4Act proposes a framework for learning effective latent memory for long-horizon robotic manipulation. It addresses the limitations of existing VLA models, which largely rely on current observations: directly incorporating the full history is computationally expensive, while keyframe selection, history compression, and language summaries can discard task-relevant information. The framework tackles three complementary challenges. First, multi-granularity temporal aggregation ensures temporal coverage under a limited memory budget. Second, semantic question answering and keyframe DINO feature reconstruction provide complementary supervision, encouraging memory to retain both task-relevant facts and fine-grained visual structure. Third, a memory-first, two-stage training strategy encourages the policy to make effective use of memory. Built on π0.5, Mem4Act injects memory into the action expert through a bounded interface of 512 latent tokens. It achieves state-of-the-art results on RoboMME, RMBench, and four real-world tasks. Ablation studies show that temporal hierarchy matters more than simply increasing memory capacity, that the two supervision objectives are complementary, and that two-stage training strengthens the policy’s reliance on rollout-specific memory.

Critical strengths

1.Hierarchical temporal aggregation constructs memory tokens that capture both local events and long-range historical context. 2.Auxiliary semantic QA and perceptual feature reconstruction improve memory tokens’ ability to retain task-relevant facts and visual details.

Critical weaknesses

The figures are overly dense and cluttered, with numerous visual elements, annotations, and small text competing for attention. In particular, the framework diagram combines architectural details, supervision objectives, and training stages without a sufficiently clear visual hierarchy, making the information flow and relationships between components difficult to follow. Simplifying the layouts and separating the main concepts from implementation details would improve readability.

Questions for authors

1.The paper does not analyze how the temporal span of history affects performance. With a fixed number of memory tokens, does extending history consistently improve performance, or does it lead to saturation or degradation? An ablation study would help clarify this. 2.Could the authors provide memory-based QA examples and DINO feature reconstruction visualizations across different supervision settings to illustrate what information each auxiliary objective helps the memory retain?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 6eHc

Thank you very much for your careful and constructive review. We sincerely appreciate your recognition of the hierarchical temporal aggregation and of the QA and DINO objectives. Below, we respond to each point.

1. Figure clarity (Weakness)

We thank the reviewer for this helpful suggestion, and we agree that the figures were too dense. Following the reviewer's advice, we have simplified both figures and separated the main concepts from implementation details. The revised Figure 1 and Figure 2 are highlighted in the revised version.

2. Effect of the history span (Question 1)

We thank the reviewer for this question. We conducted an additional ablation on RoboMME that varies the available history while keeping the memory capacity fixed at 512 tokens. The results are added as Supplementary Table S7, with discussion in the supplementary section D.3 "Effect of History Span". We restrict the memory encoder to the most recent execution history within a fixed-length window, keeping the frame-sampling interval unchanged. We evaluate windows of approximately 9, 14, and 19 seconds, corresponding to the 25th, 50th, and 75th percentiles of RoboMME episode lengths, and full history. The longest episode contains 1,411 frames, approximately 47 seconds at 30 Hz.

Execution history span≈9 s (270 frames)≈14 s (420 frames)≈19 s (570 frames)Full history
Success rate (%)39.8844.7548.6349.88

With the memory capacity fixed, extending the history improves performance at every step, and we observe no degradation. Success rises from 39.88% to 44.75% (+4.87 points) when the window grows from 9 to 14 seconds, to 48.63% (+3.88) at 19 seconds, and to 49.88% (+1.25) with full history. The gains become smaller as the window approaches the full episode, which suggests that performance begins to saturate. Overall, full history is 10 points above the shortest window at the same 512-token capacity.

This trend is consistent with RoboMME's memory requirements: many tasks involve tracking progress throughout execution or recalling cues observed near the beginning of an episode, and longer history windows preserve access to more of this evidence.

3. QA examples and DINO reconstruction visualizations across supervision settings (Question 2)

We thank the reviewer for this suggestion. We added Figure 5 in the revised version, which visualizes both memory readouts for models trained with QA only, DINO only, and both objectives, and the corresponding discussion in Sec 4.3 Ablation Studies section ("Do semantic and perceptual objectives complement each other?"). The figure uses a StopCube episode, where the cube must be stopped on the target at its 5th arrival. This needs semantic history to count the arrivals and visual history to track the cube's speed and time the press (Table S3).

What the figure shows. Each row corresponds to a past moment. For that moment, we show the DINO features of its frame reconstructed from each model's current memory (at step 395), next to the DINOv3 target, and the answers to the memory questions asked at that moment, next to the ground truth. Since each readout requires its own decoder, QA answers are shown for the QA-trained models and DINO reconstructions for the DINO-trained models. The bottom row shows how each closed-loop rollout ends.

QA only. The memory keeps the count: after the 4th pass, the policy correctly prepares to stop the cube at its 5th arrival. However, without visual detail of the cube's motion, it misjudges the speed and presses too early, so the cube stops short of the target on its 5th approach.

Key-frame DINO only. The memory keeps the visual detail of the cube's motion, and its reconstructions closely match the DINOv3 targets. This lets the policy judge the speed and time the press so the cube stops on the target. However, without semantic progress, it loses count and stops the cube at its 4th arrival instead of the 5th.

QA and key-frame DINO. With both objectives, the policy presses at the right moment of the right pass, and the cube stops on the target at its 5th arrival.

The figure therefore illustrates what each objective helps the memory retain: QA preserves task progress, DINO preserves the visual detail needed for precise timing, and StopCube needs both. This matches the complementary gains reported in Table 5(a).

Reviewer_bSn2OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Mem4Act is a method for learning a latent fixed size memory for manipulation tasks that require long-horizon memory-dependent tasks. The train with a 2-stage process. In the first stage, they group the attention pattern of their memory tokens into temporal windows of various sizes to capture coarse and fine temporal details. Then, they train their memory tokens with a DINO features reconstruction objective for keyframes and a semantic QA objective for task-relevant questions using a VLM as a decoder. In the second stage, they finetune a VLA to use the memory tokens via feature modulation. They show results on 2 simulated benchmarks and 4 real-world tasks.

Critical strengths
  1. The empirical results are strong. It beats the best baseline from the RoboMME paper, outperforms existing baselines on RMbench, and performs relatively well on 4 real-world tasks that require memory.
  2. The ablations are extensive regarding the various auxiliary loses used to shape the latent and different temporal hierarchies used for the attention pattern of the latents.
  3. The method is more compute efficient than many other SOTA methods, such as hierarchical policies that use VLM high-level policies.
Critical weaknesses
  1. The semantic QA requires subgoal labels and task-specific judgement for what questions would be useful.
  2. For the real-world task, an ablation allowing MME-VLA to use more frames would help illustrate the contributions of Mem4Act's two-stage training. This is because it is unclear whether Mem4Act worked better because it had access to more frames or that it could utilize more frames effectively.
Questions for authors
  1. Could you evaluate MME-VLA with the same number of historical frames as your method on the real-world tasks?
  2. How does inference latency and memory usage scale with history length?
2 comments
AuthorsOpenReview ↗

Response to Reviewer bSn2

Thank you very much for your careful and constructive review. We sincerely appreciate your recognition of our empirical results, extensive ablations, and computational efficiency. Below, we respond to each point.

1. Subgoal labels and task-specific QA design (Weakness 1)

We agree that semantic QA annotation cost is an important concern.

Subgoal labels. In simulation, subgoal labels come at almost no cost, since scripted data collection pipelines can track each subtask and its transitions. RoboMME and RMBench both release subgoal descriptions with temporal boundaries, from which we generate all QA pairs and keyframes automatically. For datasets without such labels, automatic video segmentation and captioning offer a practical way to obtain them. To understand how much labeling effort Mem4Act actually needs, we study label granularity by merging every two or four consecutive subgoals, which halves or quarters the number of boundaries to annotate.

Task-specific judgement. Most of our questions require no task-specific judgement. Three common questions, covering what the robot has done, what it is doing now, and what comes next, are shared across all tasks and answered directly from the completed, current, and next subgoals. Task-specific design enters only through progress questions, such as how many times the cube has reached the target in StopCube. To measure how much Mem4Act relies on these, we remove all task-specific questions in training.

Supervision setting (RoboMME)Success rate (%)
Original49.88
w/o task-specific QA48.50
+ 2 consecutive subgoals merged48.13
+ 4 consecutive subgoals merged47.38

Removing the task-specific questions lowers success by only 1.38 points, showing that the shared questions account for most of the benefit. Coarser labels have a similarly small effect, lowering success by a further 0.37 and 1.12 points when two or four subgoals are merged. Even in the most reduced setting, with no task-specific questions and four times coarser labels, Mem4Act reaches 47.38%, still above the strongest baseline, MME-VLA (FrameSamp+Modul, 44.51%). These results indicate that Mem4Act is robust to both the choice of questions and the granularity of subgoal labels.

We have added this ablation and its discussion to Supplementary Sec. D.5 (Table S8). We also state the annotation requirement in the Limitations and Future Work paragraph of the revised version, together with automatic annotation as a way to reduce manual supervision.

2. MME-VLA with the same number of historical frames (Weakness 2, Question 1)

We thank the reviewer for this suggestion. We evaluated MME-VLA (FrameSamp+Modul) on the real-world tasks with a budget of 2,048 historical tokens, matching Mem4Act's maximum history input of 128 frames sampled once per second. Mem4Act compresses this history into 512 memory tokens, whereas the expanded MME-VLA conditions on all 2,048 tokens directly. We added this result to Table 3 of the main paper and discuss it in Sec. 4.2 (Performance on real-world tasks).

MethodCover BlocksSwap BlocksRearrange BlocksPut Block BackMean
MME-VLA (128 frames)24.00%38.00%76.00%24.00%40.50%
MME-VLA (32 frames)18.00%32.00%90.00%28.00%42.00%
Mem4Act (128 frames)78.00%70.00%100.00%78.00%81.50%

These results point to two distinct reasons why MME-VLA falls behind.

Insufficient access to history. With the default 512 tokens, MME-VLA keeps only 32 frames. Real-world episodes are long, so its sampling becomes sparse and brief cues can fall between sampled frames.

Ineffective use of long raw context. With 2,048 tokens, MME-VLA covers entire episodes, since the 128-second capacity exceeds the roughly 40 seconds needed to complete our tasks. Yet its mean success drops from 42.0% to 40.5%. Without compression, a longer history means four times more raw tokens for the policy to condition on, which makes the relevant cues harder to use. Our capacity ablation shows the same trend (main paper, Table 4 and "How do memory capacity and temporal structure affect performance?"), where enlarging Mem4Act's memory from 512 to 1,024 latent tokens also lowers performance.

We also visualize both failure sources on RoboMME with the default 512 tokens in Figs. S2 and S3, with discussion in Supplementary Sec. D.8. In Fig. S2, FrameSamp's sparse sampling misses most of a brief target swap, so it cannot tell which targets moved and places the cube on the wrong one. Mem4Act samples more densely, captures how the swap unfolds, and succeeds. In Fig. S3, FrameSamp retains more frames of every cue than Mem4Act, yet still fails.

These results show that Mem4Act's advantage comes from compressing long histories into a compact memory that the policy can use effectively, not merely from access to more frames.

AuthorsOpenReview ↗

Response to Reviewer bSn2

3. Inference latency and memory usage vs. history length (Question 2)

We thank the reviewer for this question. We conducted an additional experiment showing that inference latency and memory usage do not grow with history length. The results are added as Supplementary Table S10, with discussion in Supplementary Sec. D.7. We measured a full planning cycle (encoding new frames, building the memory, predicting the action chunk) at batch size 1 on an NVIDIA A6000 GPU, averaged over 50 cycles from 10 episodes. We compare against MME-VLA (FrameSamp+Modul) and the memory-free π0.5 base policy. The longest history, about 124 s (3,728 raw frames, about 125 sampled frames), is close to Mem4Act's 128-frame capacity.

History length0 s8.5 s34 s68 s124 s
Mem4Act latency (ms)326326329328330
MME-VLA (FrameSamp + Modul) latency (ms)331332332333334
π0.5 latency (ms)129129129129129
Mem4Act peak GPU memory (GB)7.177.177.177.177.17
MME-VLA peak GPU memory (GB)7.047.047.047.047.04
π0.5 peak GPU memory (GB)6.636.636.636.636.63

Mem4Act's cost is fixed. Each visual frame is encoded once and cached. The memory encoder always processes a fixed-capacity input (2,048 tokens, with shorter histories padded and masked) and always outputs 512 tokens to the policy. Cost is therefore set by the configured capacity, not by how much history has accumulated.

Overhead over π0.5. Adding memory costs a fixed 0.54 GB of GPU memory (about 8%) and about 200 ms per planning cycle at every history length. Most of this time goes to encoding and preprocessing the newly arrived history frames, which takes about 170 ms, while the memory module itself takes only 13-15 ms. The added latency does not slow task completion. As shown in Supplementary Table S9, successful real-world rollouts take 40 s on average for Mem4Act versus 47 s for π0.5. Without memory, π0.5 stalls in 62 failed rollouts, while Mem4Act never stalls (Supplementary Table S11).

Comparison with MME-VLA. MME-VLA is also flat across history lengths: its latency is within 6 ms of Mem4Act's at every length (331–334 ms versus 326–330 ms), and its peak GPU memory is 0.13 GB lower (7.04 versus 7.17 GB). This is consistent with Table S9, where the two methods have nearly the same cycle latency (335 versus 340 ms) and peak memory (7.2 versus 7.0 GB) in deployment. However, Mem4Act performs significantly better than MME-VLA in real-world tasks (81.5% vs. 42.0%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.