acceptodds
Withdrawn from ICLR 2027

MemBodied: Recurrent Associative Memory for Vision-Language-Action Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

17 views

Abstract

Vision-Language-Action models provide a strong foundation for general-purpose robot control, yet a vast majority of policies do not preserve and leverage episode-level information beyond the current observation. This limitation is consequential in history-dependent manipulation tasks that depend on information available only in past observations. Retaining past observations in context can aid in recovering this information, but at the significant cost of ever-growing bloated context and inference latency. We thus introduce MemBodied, a fixed-size episodic memory with two complementary components: an associative state that records interactions across policy calls and an episode anchor that preserves a compact representation of the initial scene as a reference. At each policy call, the model conditions action generation on the current input and the memory components, rather than directly using past observations. Across five evaluated RMBench tasks requiring memory, MemBodied achieves the mean success rate of a stateless policy and of vanilla recurrent memory, while outperforming the strongest memory-augmented baseline by with fewer added parameters. On the fully observable LIBERO-Long suite, it reached 90.6%, a 5.4% improvement over the stateless policy. These findings support MemBodied as a practical alternative to expanding the policy context for history-dependent manipulation. Our code is available at https://anonymous.4open.science/r/MemBodied-CA84/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Embodied AI, Episodic Memory, Robotic Policy

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UnFWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MemBodied, a fixed-size episodic memory for VLA models that combines a recurrent associative state, updated through a gated delta rule, with a fixed initial-scene anchor. Memory is updated with action–observation transitions and retrieved through a dedicated memory token. The method achieves substantial improvements on memory-dependent RMBench tasks and much lower inference latency than a compressed-history baseline (NativeMEM), with additional evaluations covering a π0.5 backbone, LIBERO, and real robots.

Critical strengths

The proposed fixed-size memory avoids growing observation histories, and the transition-conditioned update provides a sensible mechanism for retaining interaction information. The method outperforms stateless, frame-stacking, and recurrent baselines on RMBench, with lower latency than NativeMEM. The failure analysis provides useful insights, showing that memory improves decisions requiring information no longer available in the current observation.

Critical weaknesses
  • Evaluation appears to be insufficient. Only 5/9 of RMBench tasks are evaluated, and the benchmark’s own Mem-0 baseline is not included. Additionally, several comparisons seem questionable: the adapted µVLA baseline achieves 0% on every task, suggesting the adaptation failed rather than providing a meaningful comparison. The stateless π0 baseline (6.4%) falls below every published RMBench policy, suggesting a weak shared training recipe. The π0.5 and LIBERO improvements are measured against published rather than locally matched baselines. The real-robot evaluation is also limited: mean success rises from 3.3% to only 26.7%. Finally, the absence of multiple training seeds limits confidence in the results.

  • The mechanism largely transfers existing associative memory (the gated delta rule, Delta-Mem) to the VLA setting. The comparison with vanilla recurrent memory changes the write representation, readout interface, and anchor simultaneously, making it difficult to attribute the improvements to specific architectural choices.

Questions for authors
  • Why were only five RMBench tasks evaluated, and why was Mem-0 omitted? Can the authors show that the improvements persist against adequately optimized baselines under matched training conditions, particularly for π0.5 and LIBERO?

  • Can the authors isolate the contribution of transition-conditioned writing by comparing it directly against hidden-state writing, with the rest of the architecture unchanged?

  • For LIBERO, why does the memory update use the full predicted 50-step action chunk when only five actions are executed? How does this discrepancy affect training–inference consistency and performance?

2 comments
AuthorsOpenReview ↗

Response to Reviewer UnFW (Part 1)

Dear Reviewer UnFW,

We appreciate your careful reading of our paper and your constructive comments. Please find our responses below.

Weakness 1/Question 1:

Evaluation scope and baselines

Response:
Task coverage and Mem-0. We chose five tasks to cover distinct recall decisions. Put Back Block, Rearrange Blocks, and Swap Blocks require recalling the original pad, block layout, and block–tray assignment, respectively. Battery Try and Block Ranking Try require tracking outcomes across attempts. In preliminary runs, two omitted tasks had near-zero success across the methods we tried. The other two were simpler or overlapped with the selected tasks. We omitted Mem-0 from this comparison because its separate planner maintains memory and feeds the actions to an execution module. Our local comparisons check how changing memory inside the action model affects the same VLA backbone.

Baselines. Our μ-VLA adaptation uses 64 recurrent tokens, sequences of eight samples at stride one, and gradient truncation every two recurrent steps, following the best configuration determined in the original paper. It achieved 0% success rate under the shared 10,000-step training budget. The original study trained OpenVLA-OFT on MIKASA-Robo for up to 150,000 steps. The local stateless π₀ shares the backbone, LoRA settings, and optimizer budget with the other local π₀ policies. Its 6.4% mean is low, but the paired rollouts help explain where the difference comes from. Both policies complete the early Put Back Block stages, and MemBodied corrects 24 wrong-pad choices with no reverse flips at the recall decision (Appendix E).

TaskFrameStackMemoryVLAFrameStack (Initial)MemBodiedFrameStack (Initial) + MemBodied
Put Back Block14.00%12.00%68.00%40.00%64.00%
Rearrange Blocks30.00%10.00%16.00%92.00%76.00%
swap blocks0.00%38.00%54.00%56.00%84.00%
Battery Try4.00%8.00%14.00%40.00%24.00%
Block Ranking Try26.00%8.00%18.00%22.00%48.00%
Average14.80%15.20%34.00%50.00%59.20%

We also evaluated two additional baselines. Our π₀-MemoryVLA adaptation reaches 15.2% mean success and succeeds on all five tasks. To retain the initial scene needed by the three M(1) tasks, FrameStack (Initial) uses the first observation alongside those at t−30, t−15, and t. This gives the baseline access to both the initial scene and recent visual evidence. It reaches 34.0% mean success. Adding MemBodied raises this to 59.2%, with gains on four of five tasks, suggesting that Membodied can complement explicit frame history.

Matched comparisons. Matching MemBodied’s sequence training raises stateless π₀ from 6.4% to 9.6%. MemBodied reaches 50.0%, a 40.4 percentage-point gain, and leads on all five tasks. The largest gaps are on Rearrange Blocks (+78%) and Swap Blocks (+54%). Matched training may improve execution, but the policy still sees only the current observation at inference. It cannot reliably make the correct action from that observation alone for history-dependent manipulation.

Taskπ₀-statelessπ₀-Stateless (Matched)Membodied
Put Back Block6.00%20.00%40.00%
Rearrange Blocks2.00%14.00%92.00%
Swap Blocks8.00%2.00%56.00%
Battery Try16.00%4.00%40.00%
Block Ranking Try0.00%8.00%22.00%
Average6.40%9.60%50.00%

On LIBERO-Long, MemBodied reaches 90.6% versus 88.0% for matched π₀. The advantage narrows from 5.4 points against published π₀ to 2.6 points against matched π₀. MemBodied also leads on Object (+0.6 points), while matched π₀ leads on Goal (+1.0) and Spatial (+2.4). The four-suite means remain nearly identical (95.10% versus 95.15%).

SuiteMemBodiedPublished π₀Matched π₀
Long90.685.288.0
Goal95.895.896.8
Object97.898.897.2
Spatial96.296.898.6
Average95.194.1595.15

π₀.₅ comparison. The 3.87× figure compares MemBodied’s 48.0% with the published π₀.₅ result of 12.4%, showing that MemBodied can be applied to a second backbone. Table 1 and the matched π₀ controls reported above provide more direct evidence for the contribution of memory. On Put Back Block, for example, the correct return pad depends on the initial scene, which the current observation no longer reveals. Matching the training recipe could improve execution, but the stateless policy would still face that information constraint.

Training seeds. Repeating fine-tuning across the full set of baselines and ablations was computationally prohibitive within our budget. We used 50 shared evaluation seeds per RMBench task and also did a qualitative comparison at specific task stages in Appendix E. This controls for variation in evaluation and helps identify where MemBodied contributes.

AuthorsOpenReview ↗

Response to Reviewer UnFW (Part 2)

Weakness 2:

The mechanism largely transfers existing associative memory (the gated delta rule, Delta-Mem) to the VLA setting. The comparison with vanilla recurrent memory changes the write representation, readout interface, and anchor simultaneously, making it difficult to attribute the improvements to specific architectural choices.

Response:
We adopt the gated delta update from Delta-Mem and adapt how memory is built for closed-loop control. Delta-Mem, designed for LLMs, writes from a text history available before prediction. Our VLA receives the current observation at each call, and the consequence of an action becomes available only after execution. We therefore build the state incrementally, writing each action chunk together with its resulting observation and training through the sequence so later action losses can shape earlier writes.

Vanilla Recurrent Memory and MemBodied, differ in the write, readout, and anchor. Table 1 includes two intermediate variants that let us examine those changes in sequence:

  1. Write representation. With attention steering and no anchor in both variants, MemBodied-AS writes the previous action and resulting observation in place of the hidden state. Success rises from 16.8% to 20.8%. This is the closest comparison for the write design, and the modest gain is suggestive.
  2. Readout. With transition-conditioned writes and no anchor, replacing attention steering with a dedicated memory token raises success from 20.8% to 37.6%. The token gives the action expert a representation of retrieved evidence that it can attend to directly.
  3. Anchor. Adding the initial-frame anchor to the memory-token variant raises success from 37.6% to 50.0%. The anchor keeps the starting scene available while the recurrent state tracks later interactions.

The largest gains in this sequence come from the readout and anchor. The write comparison measures the change under attention steering. A direct swap between hidden-state and transition-conditioned writes in the memory-token architecture would establish whether that gain carries over to the final readout.

Question 2:

Can the authors isolate the contribution of transition-conditioned writing by comparing it directly against hidden-state writing, with the rest of the architecture unchanged?

Response:
The closest comparison in Table 1 is Vanilla Recurrent Memory versus MemBodied-AS. Both use attention steering and omit the anchor. Success rises from 16.8% to 20.8% with transition-conditioned writes, which gives preliminary evidence in the attention-steering setting. Fig. 4b shows why we combine the two inputs. Across three tasks, writing the action together with its subsequent observation achieves 46.7% mean success, versus 42.7% for vision alone and 34.7% for action alone.

Question 3:

For LIBERO, why does the memory update use the full predicted 50-step action chunk when only five actions are executed? How does this discrepancy affect training–inference consistency and performance?

Response:
We replan every five environment steps in LIBERO, while the memory module was trained to update from observations spaced 50 steps apart. To preserve both cadences, we cycle through ten independent recurrent states. Each policy call uses one state.

For example, the first state is used at steps 0, 50, and 100, while the second is used at steps 5, 55, and 105. At step 50, the policy updates the first state with the new observation and uses the updated state to generate the next plan. Cycling through the ten states lets the policy replan every five steps while preserving the 50-step gap between successive observations used to update each state, matching the observation spacing used in training.

We hope our responses address your concerns. If they do, we would be grateful if you would consider updating your score. We are happy to answer any further questions.

Reviewer_VgoSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents MemBodied, a fixed-size associative memory framework for VLA policies that retains interaction history without increasing the policy context length. MemBodied retrieves relevant past information from layer-wise associative matrices using the current policy state and injects the retrieved content through a dedicated memory token, while a fixed episode-anchor memory preserves information from the initial scene. Membodied largely increases task performance on memory-dependent situations such as RMbench, without decreasing general manipulation performance on LIBERO.

Critical strengths
  • The proposed method supports long-horizon historical memory without increasing the policy context length, while the full memory mechanism is trained end-to-end through the native action objective without requiring an additional memory-specific loss.

  • The paper provides detailed ablations across multiple baseline memory designs and architectural variants, showing the contribution of each component.

Critical weaknesses
  • MemBodied substantially outperforms other history/memory baselines such as π0-μ-VLA, FrameStack, and Vanilla Recurrent Memory, but the source of this performance gap is not sufficiently analyzed. Further analysis of what enables this improvement would help clarify the specific advantages of MemBodied.

  • Several baseline results are unexpectedly weak. For example, π0-Hint degrades performance despite access to privileged information, while π0-μ-VLA scores 0 on all RMBench tasks despite incorporating memory. Further discussion of these baseline behaviors would help contextualize the reported gains.

  • The paper notes that history compression and associative memory may have task-dependent strengths. Further discussion relating task characteristics to the behavior of each memory mechanism would help clarify the benefits of the proposed method.

  • The real-robot evaluation is limited in scope, with 20 rollouts per task, and the tasks overlap with the RMBench tasks. A broader real-world evaluation would better demonstrate the generality of the method.

Questions for authors

See weaknesses also.

  • Can the authors further explain why MemBodied can outperform the other history/memory baselines, e.g., whether it better preserves key events or avoids degrading general manipulation ability when incorporating history?

  • Why is the episode anchor fixed to the first frame? Could updating the anchor throughout the episode be more beneficial for M(n) tasks?

  • MemBodied can retain long historical information with a fixed size of memory. Did the authors observe a history length beyond which information starts to diminish? What temporal horizon do the experiments in the paper cover?

2 comments
AuthorsOpenReview ↗

Response to Reviewer VgoS (Part 1)

Dear Reviewer VgoS,

Thank you for your thoughtful review. We appreciate your feedback and have addressed your questions below.

Weakness 1/Question 1:

MemBodied substantially outperforms other history/memory baselines such as π0-μ-VLA, FrameStack, and Vanilla Recurrent Memory, but the source of this performance gap is not sufficiently analyzed. Further analysis of what enables this improvement would help clarify the specific advantages of MemBodied.

Response:

The results against Vanilla Recurrent Memory show which design changes matter beyond carrying a recurrent state. Table 1 compares the changes in stages:

  • Transition-conditioned writes. With attention steering in both variants, replacing hidden-state writes with the previous action and its resulting observation (at−1,ot)(a_{t-1}, o_t) raises mean success from 16.8% to 20.8%. This suggests that storing what the robot did and the observed consequence helps.
  • Dedicated memory token. With the anchor absent, replacing attention steering with a dedicated token raises success from 20.8% to 37.6%. Attention steering uses the retrieved state to modify attention over the current tokens. The dedicated token gives the action expert a representation of the retrieved evidence that it can attend to directly. This may help when the current observation no longer shows the object or layout needed to choose the next action.
  • Initial-frame anchor. Adding the anchor raises success from 37.6% to 50.0%. It keeps the initial scene available as a stable reference while the recurrent state tracks subsequent interactions.

FrameStack’s four observations span 150 environment steps, so the initial scene can fall outside its window. MemBodied’s anchor keeps that scene available for later decisions, such as remembering the original pad in Put Back Block. In Appendix E, both MemBodied and stateless π₀ complete the early manipulation on Put Back Block, but MemBodied corrects 24 wrong-pad choices with no reverse flips on paired starts. The difference appears at the decision that requires recall of the original pad.

The zero-success μ-VLA result applies to our local adaptation under the shared 10,000-step budget. We discuss the differences from its original training setup in our response to Weakness 2.

Weakness 2:

Several baseline results are unexpectedly weak. For example, π0-Hint degrades performance despite access to privileged information, while π0-μ-VLA scores 0 on all RMBench tasks despite incorporating memory. Further discussion of these baseline behaviors would help contextualize the reported gains.

Response:
π₀-Hint. At inference time, as a hint, we added a simulator-derived progress text to a stateless policy, which is trained without those hints. Its 2.0% mean success suggests the policy couldn't leverage the provided text memory at test time.

π₀-μ-VLA. Our μ-VLA adaptation uses 64 recurrent tokens, sequences of eight samples at stride one, and gradient truncation every two recurrent steps, following the best configuration determined in the original paper. It achieved 0/250 successes under the shared 10,000-step RMBench budget. The original study trained OpenVLA-OFT on MIKASA-Robo for up to 150,000 steps.

TaskFrameStackMemoryVLAFrameStack (Initial)MemBodiedFrameStack (Initial) + MemBodied
Put Back Block14.00%12.00%68.00%40.00%64.00%
Rearrange Blocks30.00%10.00%16.00%92.00%76.00%
Swap Blocks0.00%38.00%54.00%56.00%84.00%
Battery Try4.00%8.00%14.00%40.00%24.00%
Block Ranking Try26.00%8.00%18.00%22.00%48.00%
Average14.80%15.20%34.00%50.00%59.20%

We also evaluated two additional baselines. Our π₀-MemoryVLA adaptation reaches 15.2% mean success and solves episodes on all five tasks. We also tested an initial-frame FrameStack variant. The three M(1) tasks require information from the initial scene, so we give the policy the first observation, the observations at t−30 and t−15, and the current observation at t. The offsets are in environment steps. This gives the baseline access to both the initial scene and recent visual evidence. This approach achieved an average success rate of 34%. Combining Membodied and this FrameStack variant raises success to 59.2% (+25.2%), with gains on four of five tasks. The combined result shows that recurrent memory can complement explicit frame history.

AuthorsOpenReview ↗

Response to Reviewer VgoS (Part 2)

Weakness 3:

The paper notes that history compression and associative memory may have task-dependent strengths. Further discussion relating task characteristics to the behaviour of each memory mechanism would help clarify the benefits of the proposed method.

Response:
The task results suggest where each design can help. NativeMEM leads on Put Back Block and Swap Blocks, where explicit visual history may help recover the starting pad or follow a sequence of transfers. MemBodied leads on Rearrange Blocks, where the original layout identifies the correct block after the scene becomes ambiguous. It also leads on Battery Try and Block Ranking Try, where linking an attempt to its outcome may help the policy choose what to try next. These are plausible explanations for the task-level split.

Weakness 4:

The real-robot evaluation is limited in scope, with 20 rollouts per task, and the tasks overlap with the RMBench tasks. A broader real-world evaluation would better demonstrate the generality of the method.

Response:
We reused the three RMBench tasks so that moving to a real robot would preserve the decisions that require memory. Put Back Block tests where to return a block, Rearrange Blocks tests which block to move, and Swap Blocks tests recall of the original block–tray assignment. Recreating them on real robot tests whether MemBodied still helps with the same decisions under physical sensing and execution.

We agree that a broader evaluation would indeed be better, but we have resource constraints at this moment for further tests.

Question 2:

Why is the episode anchor fixed to the first frame? Could updating the anchor throughout the episode be more beneficial for M(n) tasks?

Response:
We fix the anchor to the first frame because the initial scene supplies the reference for the single-event tasks. This lets the recurrent state track the new interactions as changes to the initial layout that stays available. For example, the anchor improves Swap Blocks from 16% to 56%, but reduces Block Ranking Try from 32% to 22%.

Refreshing the anchor could help on multi-event tasks, where the relevant reference changes. A learned keyframe selector could be used to retain a newly tested arrangement or another consequential observation while preserving the initial reference when needed.

Question 3:

MemBodied can retain long historical information with a fixed size of memory. Did the authors observe a history length beyond which information starts to diminish? What temporal horizon do the experiments in the paper cover?

Response:

RMBench evaluation episodes run for up to 3,500 environment steps, and training episodes span roughly 300–3,000 steps, depending on the task. On Block Ranking Try, the latest observed success occurred at around 2,000 steps. This gives a concrete example of how long MemBodied can sustain a memory-dependent rollout. How long it retained the information used for the final decision depends on when it acquired that information and what it preserved through subsequent updates.

That retention is likely to vary by task. MemBodied learns how much of its state to preserve or update based on what helps it predict actions during training. Because we evaluate on the same tasks used for training, our results characterise performance under those tasks’ memory demands. To determine when recall starts to decline, we would need longer-horizon tasks in which success depends on information acquired much earlier in the episode. That evaluation is beyond the scope of this work.

We hope our responses address your concerns. If they do, we would be grateful if you would consider updating your score. We are happy to answer any further questions.

Reviewer_s4jyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adds a fixed-size episodic memory to π0\pi_0 and π0.5\pi_{0.5}. The memory has two parts. The first is a set of per-layer associative matrices updated with a gated delta rule, where each write combines the executed action chunk with the observation that follows it. The second is a fixed anchor built from the first frame of the episode (Sec. 3.2-3.4). The memory is read through an extra token in the action expert and trained over sequences of policy calls (Sec. 3.3, 3.6). On five RMBench tasks, MemBodied succeeds in 125 of 250 rollouts, against 16 for the stateless policy, 42 for a vanilla recurrent memory and 96 for the authors' NativeMEM adaptation (Table 1, Fig. 2a). The paper also reports π0.5\pi_{0.5} results, a real-robot study (16/60 vs. 2/60, Fig. 3b), LIBERO results (Table 2) and a latency comparison with NativeMEM (App. B.7).

Critical strengths
  1. The paired-seed analysis in App. E is the most convincing part of the paper. The two policies behave alike until a step that needs information from earlier in the episode, and the largest differences show up there. On Put Back Block, wrong-pad returns go from 37/50 to 13/50, and 24 paired seeds flip in MemBodied's favor with none going the other way. This is good evidence that the memory is used at the right decisions and that the gains are not just better low-level control.
  2. With the backbone, LoRA settings and step budget held fixed (Sec. 4.2), the anchor-free variant clearly beats the recurrent baseline adapted from Delta-Mem, 94 vs. 42 successes out of 250 (Table 1). So the design of the memory matters, beyond just adding recurrence.
Critical weaknesses
  1. The comparison with NativeMEM is stated more strongly in the abstract and introduction than the evidence supports.
    • Latency: the introduction and Sec. 5.2 report 91.9% lower "inference latency", and Fig. 2b uses the same label. App. B.7, however, defines NativeMEM's number per policy cycle: the sum of all per-frame history encodes since the previous query plus the inference call. The paper does not say whether those encodes block the action output. If they run while the chunk is executing, 91.9% overstates the latency difference. The stateless π0\pi_0 latency is not reported either, so MemBodied's own overhead is unknown.
    • Success rate: Sec. 5.1 does acknowledge that no single method wins on every task, but the abstract and introduction give "1.3x" without that qualifier. The net gain of 29 successes (125 vs. 96) comes almost entirely from Rearrange Blocks (46 vs. 19). On the other four tasks it is 79 vs. 77 out of 200 (Fig. 2a). Table 3 lists a single training seed, so it is hard to judge how stable this gap is.
  2. The technical step beyond Delta-Mem is small, and the paper's own ablations do not show that its new ideas are what matter.
    • App. A.3 names Delta-Mem as the most directly related work. Relative to it, MemBodied changes three things: what is written (action plus next observation), where the readout enters (a dedicated token), and the added first-frame anchor.
    • In Table 1, moving the readout into a token accounts for 52 → 94 successes out of 250 (MemBodied-AS vs. anchor-free MemBodied). The anchor accounts for 94 → 125.
    • The closest test of the transition-conditioned write is MemBodied-AS vs. Vanilla Recurrent Memory, which is 52 vs. 42 out of 250 (pooled two-proportion z ≈ 1.1, p ≈ 0.25). So the benefit of the write design in Sec. 3.2.1 is not established.
    • For a method this close to existing work, the remaining evidence is also thin. The π0.5\pi_{0.5} result ("3.87x over the stateless policy", Sec. 1) is measured against published numbers from Chen et al. (2026) (Sec. 4.2), not against a stateless π0.5\pi_{0.5} trained in the same setup.
Questions for authors
  1. (Clarification) How does the π0\pi_0 NativeMEM adaptation (Table 4) differ from the original setup? The original paper reports 100% on Put Back Block and 94% on Swap Blocks with π0.5\pi_{0.5} (Wang et al., 2026, Table 1), compared with 52% and 86% here.
  2. (Clarification) Each training sample is a sequence, MemBodied seems to see more frames per step than the stateless baseline (Table 4) and the published π0\pi_0 LIBERO run (App. B.3). App. E also shows execution differences in MemBodied's favor: 6 of 14 stateless button presses miss, versus 0 of 35 for MemBodied (E.2.2). Do the gap over stateless and the 5.4-point gain on LIBERO-Long (453 vs. 426 out of 500) still hold with the memory disabled under the same training recipe?
  3. (Clarification) How were these five tasks chosen from RMBench's nine (Sec. 4.1, App. A.4)?
2 comments
AuthorsOpenReview ↗

Response to Reviewer s4jy (Part 1)

Dear Reviewer s4jy,

Thank you for taking the time to review our paper. We appreciate your feedback and address your comments below.

Weakness 1:

The comparison with NativeMEM is stated more strongly in the abstract and introduction than the evidence supports.

Response:

Latency. The 91.9% reduction is measured on the model-side compute per policy cycle in our synchronous evaluation: 129.2 ms for MemBodied versus 1,593.7 ms for NativeMEM on an A100 (Appendix B.7). The NativeMEM includes all the history encodings between queries. If that encoding process overlaps with action execution, the difference in time to the next action could indeed be smaller. We will clarify the metric in the paper. To understand MemBodied’s overhead, we measured the latency for Stateless π₀, which took approximately 95.7 ms per cycle, so MemBodied adds about 33.5 ms per cycle.

Success rate. The 1.30× ratio refers to the five-task mean (50.0% versus 38.4%). While Rearrange Blocks has 27 extra successes, performance on the remaining four tasks varies rather than being equivalent. NativeMEM leads by 12 and 30 percentage points on Put Back Block and Swap Blocks, while MemBodied leads by 24 and 22 points on Battery Try and Block Ranking Try.

Weakness 2:

The technical step beyond Delta-Mem is small, and the paper's own ablations do not show that its new ideas are what matter.

Response:

We agree that the gated delta update comes from prior works. However, the part that does not transfer directly is how Delta-Mem builds its memory. Delta-Mem was designed for LLMs and writes from a history of text tokens available before the model answers. On the other hand, in our VLA setup, each policy call receives the current observation, and earlier observations and actions are no longer in its input. Therefore, there is no accumulated context for a separate write stage to process. In our policy, we instead build the associative state incrementally during execution. When the next observation arrives, we write the preceding action chunk together with its observed consequence, then carry that state forward across calls. We train through the same sequence so later action losses can shape earlier writes.

Table 1 shows the ablations comparing the different mechanisms introduced in MemBodied. Switching from attention steering to a memory token raises success from 52/250 to 94/250. The token gives the action expert a representation of past evidence to attend to when the current observation is ambiguous, which may explain the gain. Adding the initial-scene anchor raises success to 125/250. The closest write comparison, 42/250 versus 52/250, gives weaker evidence for the transition write itself. However, the write-value ablation shows the benefits of the transition write. Removing either actions or observations from the write value degrades performance (from 46.7% to 42.7% / 34.7%), which directly tests the write content without the attention-steering bottleneck of MemBodied-AS (Fig. 4b).

π₀.₅ comparison. The 3.87× figure indeed uses the published π₀.₅ result (48.0% versus 12.4%). It shows that the method can be applied to a second backbone. However, Table 1 provides a stronger evidence supporting the contribution of the memory design through locally trained π₀ variants. On tasks such as Put Back Block, the correct return pad depends on the initial scene, which the current observation no longer reveals. A matched training recipe could improve execution, but decisions that require missing episode history would still be challenging for the model. We show the results for a matched π₀ on RMBench and Libero under our response to Question 2.

Question 1:

(Clarification) How does the NativeMEM adaptation (Table 4) differ from the original setup? The original paper reports 100% on Put Back Block and 94% on Swap Blocks with (Wang et al., 2026, Table 1), compared with 52% and 86% here.

Response:
For a fair comparison, two aspects differ from the original NativeMEM setup. We use π₀ rather than π₀.₅ and we train its tokenizer on RMBench rather than the original broader mixture of real and simulated tasks. This removes the influence of extra training data, that is used for encoder training, on the comparison. We retain the two-stage training recipe: train the tokenizer for 50,000 steps, cache its per-frame and per-view tokens, then freeze it and train the queue-conditioned policy for 20,000 steps (Appendix B). The differences in backbone and data could both contribute to the lower Put Back Block and Swap Blocks scores.

AuthorsOpenReview ↗

Response to Reviewer s4jy (Part 2)

Question 2:

(Clarification) Each training sample is a sequence, MemBodied seems to see more frames per step than the stateless baseline (Table 4) and the published LIBERO run (App. B.3). App. E also shows execution differences in MemBodied's favor: 6 of 14 stateless button presses miss, versus 0 of 35 for MemBodied (E.2.2). Do the gap over stateless and the 5.4-point gain on LIBERO-Long (453 vs. 426 out of 500) still hold with the memory disabled under the same training recipe?

Response:
Yes. The gain persists against the completed matched π₀ controls on RMBench and LIBERO-Long, though its size changes.

Taskπ₀-statelessπ₀-Stateless (Matched)Membodied
Put Back Block6.00%20.00%40.00%
Rearrange Blocks2.00%14.00%92.00%
Swap Blocks8.00%2.00%56.00%
Battery Try16.00%4.00%40.00%
Block Ranking Try0.00%8.00%22.00%
Average6.40%9.60%50.00%

Matching the training setup raises stateless π₀’s mean from 6.4% to 9.6%, with mixed effects across tasks. MemBodied’s 50.0% average success rate is a 40.4% gain over the matched baseline, and leads on all five tasks. The largest gaps are on Rearrange Blocks (+78%) and Swap Blocks (+54%).

The original stateless policy missed 6 of its 14 button presses, versus 0 of 35 for MemBodied. Appendix E.2.2 shows that the larger difference came one decision earlier. The policies completed the initial arrangement at similar rates (38/50 for MemBodied and 40/50 for stateless π₀), but the stateless policy failed to attempt a test in 26 of those 40 cases, compared with 3 of 38 for MemBodied. Thus, among stateless rollouts that reached an initial arrangement, failure to initiate the test was a more common failure mode than a missed press. The extra training exposure may help execution, but the matched policy still has only the current observation at inference. It cannot reliably make the correct action from that observation alone for history-dependent manipulation.

SuiteMemBodiedPublished π₀Matched π₀
Long90.685.288.0
Goal95.895.896.8
Object97.898.897.2
Spatial96.296.898.6
Average95.194.1595.15

On LIBERO-Long, MemBodied reaches 90.6% versus 88.0% for matched π₀. The advantage narrows from 5.4 points against published π₀ to 2.6 points against matched π₀. MemBodied also leads on Object (+0.6 points), while matched π₀ leads on Goal (+1.0) and Spatial (+2.4). The four-suite means remain nearly identical (95.10% vs 95.15%).

Question 3:

(Clarification) How were these five tasks chosen from RMBench's nine (Sec. 4.1, App. A.4)?

Response:

We chose five tasks to cover distinct recall decisions. Put Back Block, Rearrange Blocks, and Swap Blocks require recalling the original pad, block layout, and block–tray assignment, respectively. Battery Try and Block Ranking Try require tracking outcomes across attempts. In preliminary runs, two omitted tasks had near-zero success across the methods we tried. The other two were simpler or overlapped with the selected tasks.

We hope our responses address your concerns. If they do, we would be grateful if you would consider updating your score. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.