acceptodds
Under review as a conference paper at ICLR 2027

Plan First, Act Later: Streaming Reasoning for Executable Memory in Robot Control

Abstract

Memory-dependent manipulation requires a robot to retain task-relevant evidence beyond its short observation window. Existing memory-augmented policies typically retrieve relevant information from history and reason over it at execution time, placing a heavy reasoning burden on the high-level model. To address this challenge, we propose PAL (Plan first, Act Later), a hierarchical memory system designed to alleviate this reasoning burden. PAL maintains a short-term visual buffer, a semantic-event memory, and an executable subtask list. A high-level vision-language model (VLM) converts streaming visual observations into the subtask list. At execution time, the VLM selects the currently relevant subtask from the subtask list and passes it to a low-level vision-language-action (VLA) policy. To teach the model what to write to memory and when to update it, we introduce a two-stage procedure that first densifies sparse update supervision and then uses self-labeling with task-grounded verification to generate training data that better match the state distribution encountered during execution. We validate PAL on memory-dependent manipulation benchmarks and in real-world manipulation tasks. On RoboMME, PAL achieves a 56.05% success rate, compared with 44.51% for the strongest baseline, FrameSamp-Modul. In real-world experiments across four manipulation tasks, PAL reaches 60.8% overall success, outperforming MemER (45.0%) and FrameSamp-Modul (28.3%). These results demonstrate that proactively writing future-relevant subtasks into hierarchical memory improves long-horizon visual reasoning and robot control.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Memory, Manipulation, VLA system

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ozHAOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how robots can remember visual evidence that is no longer visible and use it in later actions. PAL uses a visual buffer, event memory, and a subtask list. PAL achieves an average success rate of 56.05% on RoboMME. Across four real-world tasks, the average is 60.83%, compared with 45.00% for MemER. The gains are more noticeable in long-video understanding, retaining evidence during execution, and counting repeated actions.

Critical strengths
  1. The paper addresses a specific problem: evidence about what to do later may appear before the action needs to be carried out. Saving subtasks in advance is a reasonable and easy-to-understand solution. On RoboMME, the average success rate improves by 11.54 percentage points over the strongest baseline. The long-video group reaches 81.60%, an improvement of 35.60 percentage points, while the group of tasks that obtain evidence during execution improves by 19.15 percentage points. The gains mainly occur in the scenarios the authors aim to address.
  2. The experimental design is reasonable. The sparse-label version, the first-stage-only version, and the full method achieve 43.39%, 37.35%, and 56.05%, respectively. The authors also report update frequency and final list accuracy, and include a ground-truth update gate. I find this more informative than reporting only the final success rate. The training details and failure cases in the appendix are also useful.
Critical weaknesses
  1. Key experiments and details are missing. The current ablations show that the second stage is useful, but do not separately remove the event layer, the list of future subtasks, or planning ahead while keeping the VLM, data, and training budget the same. It is therefore unclear whether the full three-layer structure is necessary or how much writing subtasks in advance contributes. The GPT baseline samples at most 24 frames from the conditioning video, whereas PAL processes the history as a stream. They therefore differ in both the amount of information they receive and how they are adapted through training. Using the same low-level VLA helps the comparison, but the full performance gap cannot yet be attributed to the memory representation.
  2. The paper emphasizes reducing the high-level reasoning cost, but provides no comparison of high-level inference time, token counts, or GPU memory usage. A fixed number of input images only limits the visual input; the subtask list itself may still grow. In the real-world experiments, the improvement over MemER mainly comes from PickXtimes, while performance remains weak on tasks requiring fast responses or route coordination. Training also relies on reference information for each task and exact text matching. Its effectiveness on new tasks and recovery from incorrect memories remains unclear. In particular, how do these costs change as the list grows?
Questions for authors
  1. Please compare versions without the event layer, with event memory only, and without writing subtasks in advance, using the same model, data, and training budget. These comparisons would help determine the contributions of the three-layer structure and planning ahead. How much improvement comes from the second stage?
  2. Are there controls with the same number of training steps, the same resampling, and the same task weighting to separate the effect of validated self-labeling from that of additional training? After failed trajectories are filtered out, how much data remains for each task, and does the task difficulty distribution change?
  3. Please clearly report the high-level VLM inference time per call and per complete task, input and output token counts, and GPU memory usage. Compare these with the baselines under the same observation and query budgets. How do these costs change as the list grows?
  4. Data sources and list modification rules. Appendix D.2 states that GPT-6-astra generated training data, evaluation trajectories, and task configurations. Which parts of the inputs, labels, and scripts does this refer to? Which records come from independently collected robot executions? The main text describes only appending to the list, whereas the appendix allows list modification. What rules are actually used? I am also concerned about the reliability of the data generated by GPT-6-astra and would like a detailed explanation from the authors.
6 comments
AuthorsOpenReview ↗

Reply to Reviewer ozHA (Part 1/3)

We thank the Reviewer of acknowledging our performance gains and experiment design. We address the questions regarding ablations on memory components, stage 2 training, inference costs and data sources as below.

W1/Q1 Ablation on memory components.

Please compare versions without the event layer, with event memory only, and without writing subtasks in advance, using the same model, data, and training budget.

Thank you for asking for controlled comparisons of the memory components and planning ahead. Keeping the other settings unchanged, we separately remove the subtask list, remove event memory, and replace the visual buffer with recent frames (we take "with event memory only" and "without writing subtasks in advance" as the same variant). All three ablations perform worse overall than PAL, supporting the contribution of each memory component. Our analysis further identifies when each component is most useful. Together, these results clarify their respective roles; please see our response to Reviewer PU7N, W2/Q1, for the full results and discussion.

How much improvement comes from the second stage?

Thank you for asking about the contribution of Stage 2. Stage 2 improves overall success from 37.35% for Stage 1 Only to 56.05% for PAL. Section 5.2 of the main text reports this comparison and analyzes how Stage 2 regulates memory updates. These results support the contribution of Stage 2 to more reliable memory updates and higher task success.

Q2 Ablation on Controlled Finetuning of Stage 2

Are there controls with the same number of training steps, the same resampling, and the same task weighting to separate the effect of validated self-labeling from that of additional training?

Thank you for asking for controls that separate verified self-labeling from additional training and data augmentation. Our controls suggest that verified self-labeling accounts for most of the Stage 2 improvement. Additional supervised training with the same training budget and augmentation remains below PAL, whereas training on verified self-labeled data without non-empty update repetition or task reweighting retains most of PAL's performance. These results support the contribution of verified self-labeling; please see our response to Reviewer 32vj, W2/Q2, for the full comparison.

After failed trajectories are filtered out, how much data remains for each task, and does the task difficulty distribution change?

Thank you for asking about data retention and changes in task difficulty after filtering. Most episodes are retained after verification, with the lowest acceptance count being 84 out of 100 for VideoRepick. The “One-pass labeling” paragraph in Appendix B.2 reports the accepted episode count for each task. Thus, filtering retains the majority of the data for every self-labeled task.

The overall difficulty distribution changes only slightly, although individual tasks show larger shifts. For VideoPlaceOrder, the proportion of Hard episodes decreases from 25.00% to 20.93%; for VideoRepick, it increases from 25.00% to 29.76%. The aggregate proportions are shown below. Thus, the overall distribution remains similar, while the effect of filtering varies by task.

DifficultyBefore filteringAfter filteringChange in share (pp)
Easy550 / 1,100 (50.00%)521 / 1,030 (50.58%)+0.58
Medium275 / 1,100 (25.00%)257 / 1,030 (24.95%)-0.05
Hard275 / 1,100 (25.00%)252 / 1,030 (24.47%)-0.53

W2 Explanation about GPT Baseline

The GPT baseline samples at most 24 frames from the conditioning video, whereas PAL processes the history as a stream.

Thank you for raising the difference in how the GPT baseline and PAL process conditioning videos. We agree that the GPT baseline could perform better with a more carefully designed inference framework. We organize its memory similarly to FrameSamp, the strongest baseline proposed in RoboMME, which is straightforward to implement and uses a hierarchical framework similar to Physical Intelligence's Mem. Designing a stronger framework for GPT is beyond the scope of this work. Our results therefore show that GPT can struggle with memory-dependent tasks under the evaluated memory representation, rather than establish a limit on its capabilities.

AuthorsOpenReview ↗

Reply to Reviewer ozHA (Part 2/3)

Q3 Experiments on Inference Costs

Please clearly report the high-level VLM inference time per call and per complete task, input and output token counts, and GPU memory usage. Compare these with the baselines under the same observation and query budgets. How do these costs change as the list grows?

Thank you for asking us to report computational costs and examine how they change as the list grows. We compare PAL with Memer, a similar hierarchical framework where VLM gives current subtasks to low-level VLA. PAL has lower VLM inference time per call and per episode, uses fewer input and output tokens per call, and requires less peak GPU memory. We profile all 16 tasks using one evaluation seed per model and report the results below.

For MemER implemented by RoboMME, the VLM receives downsampled video frames at each query without a separate video-reasoning phase. For fair comparison, we train a MemER high-level VLM with keyframe annotations in conditioning video at the timesteps where PAL has event or subtask-list annotations, enabling explicit reasoning during the video phase. We keep the training budget and other settings, the same as PAL. We also keep the same query interval, but the number of query calls each episode may differ because the episode will terminate in advance if it fails or successes. The resulting variant achieves an overall success rate of 44.88%, compared with 42.38% from original RoboMME Implementation.

MethodVLM inference time / call (s)Cumulative VLM inference time / episode (s)Input tokens / callOutput tokens / call
MemER (Explicit Video Reasoning)1.725122.412988.4524.77
PAL1.38082.059623.9121.46
MemER (Original RoboMME Implementation)2.35955.9581,230.2233.25

PAL's memory remains compact as tasks progress. We divide the evaluation rollouts into ten phases based on their relative timesteps; the reported PAL averages range from 531.99 to 675.89 input tokens per call, a difference of approximately 27%, while MemER with explicit video reasoning uses 1,137.95 input tokens in the final phase. PAL's longest subtask list contains 123 tokens, and the mean of the per-task maximum list lengths across the 16 tasks is 63 tokens. Peak GPU memory usage is also lower for PAL: 9.98 GiB, compared with 10.95 GiB for MemER with explicit video reasoning. Together, these results show that PAL's compact memory reduces inference costs when both methods explicitly reason over video.

We also evaluate MemER's original RoboMME implementation on inference computation costs. PAL still has lower per-call latency, input and output token counts per call, and peak GPU memory usage (9.98 vs. 10.36 GiB). The exception is cumulative VLM inference time per episode (82.059 vs. 55.958 s), reflecting PAL's additional video-reasoning phase. These measurements show that PAL retains its advantage on the other reported cost metrics compared with the original implementation.

AuthorsOpenReview ↗

Reply to Reviewer ozHA (Part 3/3)

Q4 Clarification on Data Source

Data sources and list modification rules. Appendix D.2 states that GPT-6-astra generated training data, evaluation trajectories, and task configurations. Which parts of the inputs, labels, and scripts does this refer to? Which records come from independently collected robot executions?

Thank you for asking us to clarify the sources of the real-world data. We use two LLMs, GPT-5.6-Sol and GPT-6-astra. We distinguish real-world data collection, data generated by GPT-5.6-Sol and data generated by GPT-6-astra as below:

  1. Real-world data collection: The input images and robot states come from real-world teleoperation.
  2. GPT-5.6-Sol: It refines the temporal boundaries of recorded trajectories and produces labels for event and executable subtask supervision. These data have been verified by human.
  3. GPT-6-astra: It generates task configurations and goals for both training and evaluation episodes. For example, in PatternLock, it specifies the starting point and the movement sequence to be demonstrated, such as “1. Move right; 2. Move left.”

In conclusion, the observations and robot states are collected from physical executions, GPT-5.6-Sol annotates the recorded trajectories, and GPT-6-astra specifies the tasks.

The main text describes only appending to the list, whereas the appendix allows list modification. What rules are actually used?

Thank you for pointing out the ambiguity in our description of the list-update rules. Our experiments focus on appending subtasks, although the model outputs the entire list even when only one new entry is added. This output format leaves room for future work on revising existing entries. We will revise the wording to distinguish the update behavior used in our experiments from this possible extension, and we plan to release the code for future development. Thus, list revision is a future direction; please also see our response to Reviewer 32vj, Q4.


We thank Reviewer ozHA again for the insightful comments. Our responses clarify the three-layer memory design, the Stage 2 controls, computational costs, and data annotation procedure. We hope these explanations address your concerns and invite you to reconsider your score in light of the additional results. We welcome further questions and discussion

Reviewer_32vjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Plan First, Act Later (PAL), a hierarchical memory framework for robotic manipulation tasks where correct action depends on historical information that is no longer visible. The central idea is to convert visual evidence into future executable subtasks once sufficient evidence is available, and then select the relevant subtasks when execution is required.

PAL maintains three memory components: a short-term visual buffer, a semantic event memory, and a list of executable subtasks. The high-level visual language model updates these components based on streaming observations and selects the current subtask for the low-level VLA policy. Training combines intensive update-time supervision with a second stage of self-labeling that is validated against reference annotations.

On RoboMME, the average success rate of PAL is 56.05%, which is an increase of 11.54 percentage points compared to FrameSamp-Modul's 44.51%. Across four real-world tasks, PAL achieved an overall success rate of 60.83%, better than MemER's 45.00% and FrameSamp-Modul's 28.33%.

Critical strengths
  1. The memory representation addresses a concrete difficulty in memory-dependent control. When a temporary cue, an object location before occlusion, or a demonstration sequence determines a future action, recording the resulting subtask provides a direct interface to subsequent execution. The separation between semantic events and executable subtasks makes the approach coherent and interpretable. This is the paper’s strongest conceptual contribution.
  2. The empirical improvements are meaningful and supported across simulation and physical experiments. PAL improves overall RoboMME success by 11.54% over FrameSamp-Modul. Its Long-video success rate reaches 81.60%, compared with 46.00% for that baseline. Real-world experiments provide additional evidence of practical value. The complete method also outperforms Sparse Label and Stage 1 Only, supporting the importance of the second training stage.
Critical weaknesses
  1. The experiments do not directly isolate the central architectural contribution. The ablations primarily compare training procedures. They do not provide a controlled comparison between event memory with execution-time planning and event memory augmented with advance subtask construction, using the same model, supervision, and training budget. Consequently, the gains cannot be attributed specifically to executable memory. Moreover, the claimed reduction in execution-time reasoning burden is not directly measured through latency, token consumption, memory usage, or scaling with history length.
  2. The second-stage training procedure and evaluation protocols introduce unresolved confounding factors. Stage 2 combines reference-based filtering, trajectory selection, repetition of non-empty updates, and task-specific reweighting. Their individual contributions remain unclear. Because the verifier suppresses incorrect updates during data generation, the resulting memory trajectories also differ from uncorrected inference trajectories. In addition, some baseline results are taken from prior work with different checkpoint and seed configurations. The real-world evaluation uses only ten scenarios per task, each repeated three times, limiting evidence for generalization and smaller performance differences.
Questions for authors

The following clarification requests concern issues that could materially affect my assessment:

  1. Under the same model, supervision, and training budget, how does PAL compare with a variant that retains semantic-event memory but generates subtasks only when they are needed? This would directly test the benefit of advance subtask construction.
  2. How much improvement remains if Stage 2 uses the same additional training budget, non-empty update repetition, and task reweighting, but excludes verified self-labeling? Can the benefits of these components be separated?
  3. Does Stage 2 advance through fixed training videos rather than environment trajectories induced by the predicted current subtasks? If so, should its distribution-matching benefit be understood primarily as matching memory-state distributions rather than full closed-loop execution distributions? Please also clarify how the verifier tracks completed reference events after event memory is cleared.
  4. Can the implemented subtask list modify or delete existing entries? How does PAL recover when an earlier plan is incorrect or later observations invalidate a stored subtask?
  5. What are PAL’s high-level inference latency and token costs relative to MemER, and how do they scale with task duration? In the real-world evaluation, how much of the aggregate improvement remains when PickXtimes is excluded?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer 32vj (Part 1/3)

We thank Reviewer 32vj for recognizing the significance of our work and its performance improvements. Below, we present the additional experiments and clarify the training procedure, list-update rules, computational costs, and real-world results to address your concerns.

W1/Q1 Ablation on Planning Ahead / Subtask List

Under the same model, supervision, and training budget, how does PAL compare with a variant that retains semantic-event memory but generates subtasks only when they are needed?

Thank you for asking about the benefit of planning ahead compared with generating subtasks only when needed. Removing the subtask list reduces overall success from PAL's 56.05% to 51.50%, with all other settings unchanged. Our analysis further identifies two conditions under which planning ahead is particularly useful: when more subtasks are required and when events must be interpreted to determine future actions. These results clarify the benefit of planning ahead; please see our response to Reviewer PU7N, W2/Q1, for the full results and discussion.

W2/Q2 Controlled Finetuning on Stage2

How much improvement remains if Stage 2 uses the same additional training budget, non-empty update repetition, and task reweighting, but excludes verified self-labeling? Can the benefits of these components be separated?

Thank you for asking how much of the Stage 2 improvement comes from verified self-labeling, additional training, and data augmentation. The two controls below suggest that verified self-labeling accounts for most of the improvement from Stage 2. Starting from the same Stage 1 model, we compare additional supervised training with the same training budget and augmentation against training on verified self-labeled data without non-empty update repetition or task reweighting. The first control reaches 45.63%, whereas the second reaches 55.38%, close to PAL's 56.05%, supporting the contribution of verified self-labeling.

  1. Controlled Fine-tuning: We continue training the Stage 1 model for the same number of additional steps, using Stage 1 data with the same non-empty update repetition and task reweighting. This variant improves overall success from 37.35% of Stage 1 model to 45.63% but remains below PAL.
  2. Verifier Only: We train the same Stage 1 model on verified self-labeled data without non-empty update repetition or task reweighting. This variant reaches 55.38%, with only a small decline from PAL's 56.05%. Retaining most of PAL's performance without these two components further supports the contribution of verified self-labeling.
TaskControlled Fine-tuningVerifier Only
BinFill60.00%68.00%
ButtonUnmask96.00%96.00%
ButtonUnmaskSwap38.00%18.00%
InsertPeg2.00%6.00%
MoveCube46.00%80.00%
PatternLock50.00%46.00%
PickHighlight78.00%82.00%
PickXtimes80.00%84.00%
RouteStick20.00%24.00%
StopCube6.00%2.00%
SwingXtimes86.00%72.00%
VideoPlaceButton24.00%78.00%
VideoPlaceOrder6.00%78.00%
VideoRepick18.00%32.00%
VideoUnmask90.00%86.00%
VideoUnmaskSwap30.00%34.00%
Overall mean45.63%55.38%

These controls support the contribution of verified self-labeling, although they do not separately isolate the effects of non-empty update repetition and task reweighting. Due to the limited rebuttal period, we have not conducted further experiments separating these components. The Verifier Only variant also retains similar performance on PatternLock and RouteStick, the two tasks upweighted during PAL's Stage 2 training, indicating that verified self-labeling provides most of the observed benefit even without non-empty update repetition or task reweighting.

AuthorsOpenReview ↗

Reply to Reviewer 32vj (Part 2/3)

Q3 Clarification on Stage 2 Mechanism

Does Stage 2 advance through fixed training videos rather than environment trajectories induced by the predicted current subtasks?

Thank you for asking us to clarify the Stage 2 rollout procedure and the scope of distribution matching. Yes, Stage 2 advances through fixed training videos, so its distribution-matching benefit concerns memory states. The observations come from these videos rather than environment trajectories induced by the model's predicted subtasks. We will revise the paper to make the scope of this distribution matching explicit.

Please also clarify how the verifier tracks completed reference events after event memory is cleared.

We thank the reviewer for catching this. The verifier does not read progress from the model's event memory. For the event memory and the subtask list separately, it keeps a pointer to the earliest reference update that has not yet been matched. A prediction that matches this reference advances the pointer; any other prediction is replaced with ∅\emptyset. Clearing the event memory after a subtask-list update changes the model's memory but not the pointer.

For example, let the reference events be A and then B. The model predicts A at t1t_1, so the pointer moves to B. At t2t_2 the model appends a new subtask, which clears its event memory. A remains recorded as matched, so the rollout is not rejected when A's deadline passes. If the model repeats A later, the prediction is compared with B and replaced with ∅\emptyset, so the self-labeled data contain no duplicate A. If the model predicts B before B's deadline, the pointer advances again. An episode is kept only if every reference update is matched before its deadline.

Algorithm 2 and Sec. 4.3 state the deadline test as membership in the model's current memory (ΔEiref∉Et\Delta\mathcal{E}^{\mathrm{ref}}_i\notin\mathcal{E}_t), which would wrongly reject such rollouts once the event memory is cleared. This is an error in our description, not in the implementation. We will revise Sec. 4.3 and Algorithm 2 to use the verifier's pointer.

Q4 Clarification about How Subtask List Work

Can the implemented subtask list modify or delete existing entries? How does PAL recover when an earlier plan is incorrect or later observations invalidate a stored subtask?

Thank you for asking about list modification and recovery from incorrect plans. Our experiments focus on appending subtasks; modifying or deleting existing entries is left for future work. The model outputs the entire subtask list even when only one entry is appended, so this output format could support revisions if they were included in the training annotations. We did not study such revisions because they are not essential to the RoboMME tasks considered here. Recovery through revising an earlier plan therefore remains a direction for future development.

W1/Q5 Experiments on Computational Resource and Explanation on Real-world Evaluation

What are PAL’s high-level inference latency and token costs relative to MemER, and how do they scale with task duration?

Thank you for asking about high-level inference costs and how they change with task duration. Compared with MemER with explicit video reasoning, PAL has lower VLM inference time per call and per episode, uses fewer input and output tokens per call, and requires less peak GPU memory. We also examine token usage over task progress and compare against MemER's original RoboMME implementation. These results support PAL's inference efficiency; please see our response to Reviewer ozHA, Q3, for the full results and discussion.

AuthorsOpenReview ↗

Reply to Review 32vj (Part 3/3)

In the real-world evaluation, how much of the aggregate improvement remains when PickXtimes is excluded?

Thank you for asking how much of the real-world improvement remains when PickXtimes is excluded. Excluding PickXtimes, PAL achieves a mean success rate of 63.33%, compared with MemER's 60.00%, so its advantage is smaller. PAL nevertheless matches or exceeds MemER on all four real-world tasks. On PickXtimes, MemER fails to track progress, repeatedly raising the box without subsequently putting it down. We have retained all real-world training data, evaluation videos, and trajectories and plan to release them to support the transparency of the comparison. These results show that PAL retains an advantage without PickXtimes, while the largest improvement comes from that task.

MethodMean success rate excluding PickXtimes
FrameSamp-Modul24.44%
MemER60.00%
PAL (ours)63.33%

We thank Reviewer 32vj again for the helpful comments. The additional experiments clarify the contributions of the memory design and Stage 2 training, while our explanations address the training procedure, list-update rules, computational costs, and real-world results. We hope these responses resolve your concerns and invite you to reconsider your score in light of the new evidence. We welcome further discussion and will incorporate the additional results into the revised paper.

Reviewer_PU7NOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a hierarchical memory system for VLA models. A high-level VLM maintains a short-term visual buffer, a semantic-event memory, and an executable subtask list, and selects the current subtask for a low-level VLA to execute. The authors also introduce a two-stage training procedure to learn memory updates. The method is evaluated on the RoboMME benchmark and four real-world manipulation tasks.

Critical strengths
  1. A reasonable hierarchical memory design. The three memory components have clear roles: retaining recent visual observations, recording relevant events, and storing executable subtasks. This organization provides a reasonable way to preserve historical information and use it for subsequent action execution.

  2. Clear presentation. The paper is generally well organized and easy to follow. The figures and algorithm descriptions help explain the memory update mechanism and the two-stage training procedure.

Critical weaknesses
  1. Limited evaluation across benchmarks. The simulation experiments are conducted only on RoboMME. Although the paper also includes real-world experiments, these tasks are adapted from their simulated counterparts. It is therefore difficult to assess how broadly the method applies beyond the evaluated task settings.

  2. Missing ablations of the three memory components. The current ablations mainly examine the training procedure while retaining the same hierarchical memory architecture. Without ablations of the visual buffer, semantic-event memory, and executable subtask list, it remains unclear how much each component contributes and which components account for the observed gains.

Questions for authors
  1. Contributions of the memory components. Could the authors provide ablations of the three memory components to clarify their individual roles and contributions to performance?

  2. Evaluation beyond RoboMME. Could the authors evaluate the method on another simulation benchmark to provide evidence of its applicability beyond RoboMME? If evaluating an entire benchmark is infeasible within the rebuttal period, a representative task from RMBench could provide useful initial evidence.

7 comments
AuthorsOpenReview ↗

Reply to Reviewer PU7N (Part 1/3)

We thank Reviewer PU7N for the positive assessment of our presentation and recognition of our hierarchical memory design. We have added experiments on two RMBench tasks and three ablations to clarify the contribution of each memory component. Below, we address the questions about evaluation beyond RoboMME and the roles of the three components.

W1/Q2 Evaluation on RMBench

Evaluation beyond RoboMME.

Thank you for asking about evaluation beyond RoboMME. PAL outperforms the reported baselines on two additional RMBench tasks, representing memory types M(1) and M(n). We train the π0.5 VLA executor for the same number of steps as Mem-0. Given the time and computational constraints, our high-level planner uses Qwen3-VL-4B, trained for eight epochs in Stage 1 and two epochs in Stage 2, whereas Mem-0 uses Qwen3-VL-8B trained for 25 epochs per task. We evaluate PAL on 100 episodes with the same layout seed as baselines. Due to time limit, we only evaluate the results on one seed per episode. The baselines results are taken from RMBench paper. Further analysis highlights the manipulation demands of these RMBench tasks: on Observe and Pick Up, PAL identifies the correct object in 74% of episodes but successfully picks it up in only 12%; on Battery Try, batteries fall out of the box in 31% of episodes, and the robot fails to recover. Thus, these results support PAL's applicability beyond RoboMME and suggest that a stronger low-level executor could further improve performance.

TaskMemory typeπ0.5MemERMem-0PAL (ours)
Observe and Pick UpM(1)9%7%4%12%
Battery TryM(n)16%27%28%37%

W2/Q1 Full Ablation on Three Memory Layer

Contributions of the memory components.

Thank you for asking us to clarify the contribution of each memory component. All three ablations have lower overall success rates than PAL (56.05%), supporting the contribution of each memory component. We separately remove the subtask list, remove event memory, and replace the visual buffer with the four most recent frames and train these variant with the same two-stage training pipeline and arguments. The combined results below and the following analysis clarify the role of each component. Due to time limit, the ablation is conducted on a single seed 42, and full results on all five seeds will be put into the final version of paper.

TaskWithout subtask listWithout event memoryFour most recent framesPAL
BinFill55.60%46.80%48.00%65.20%
ButtonUnmask77.60%90.80%85.60%90.80%
ButtonUnmaskSwap20.00%14.40%13.60%20.40%
InsertPeg4.80%2.80%6.40%4.80%
MoveCube78.00%79.20%80.80%76.40%
PatternLock44.40%46.80%32.80%50.80%
PickHighlight58.80%78.00%77.60%68.40%
PickXtimes94.00%88.40%94.80%87.20%
RouteStick18.80%26.00%18.40%22.40%
StopCube3.20%12.80%18.80%5.20%
SwingXtimes74.00%86.40%52.40%79.60%
VideoPlaceButton81.20%76.80%65.60%80.00%
VideoPlaceOrder70.00%76.40%74.80%83.20%
VideoRepick35.20%16.40%35.20%32.00%
VideoUnmask93.20%94.00%90.40%95.20%
VideoUnmaskSwap26.80%28.80%24.00%35.20%
Overall mean52.23%54.05%51.20%56.05%
AuthorsOpenReview ↗

Reply to Reviewer PU7N (Part 2/3)

Removal of the subtask list

Removing the subtask list reduces overall success from 56.05% to 51.50%. We fine-tune the same base model with the subtask lists removed from the training data, keeping all other training settings unchanged, and report results for seed 42. Because event memory and the subtask list can contain overlapping information, we further analyze when planning ahead provides the greatest benefit.

  1. The subtask list becomes more useful as the number of required subtasks increases. For BinFill, SwingXtimes, and PickXtimes in the Episode-start group defined in our paper, harder episodes require more subtasks: for example, repeating a motion five times or picking four to five cubes of different colors, compared with one to two in easier episodes. The table below reports averages across these three tasks. The mean improvement increases from -4.36 percentage points (pp) on Easy episodes to +14.44 pp on Hard episodes, suggesting that planning ahead is particularly useful for tasks requiring more steps.
DifficultyPALPAL without subtask listImprovement (pp)
Easy84.10%88.46%-4.36
Medium83.89%72.22%+11.67
Hard56.11%41.67%+14.44
  1. Planning ahead is more useful when events must be interpreted to determine future subtasks. In RoboMME's Imitation group, descriptions of the demonstrated actions closely resemble the executable subtasks, so PAL (38.60%) performs similarly to the variant without a subtask list (37.00%). For VideoPlaceOrder, VideoUnmaskSwap, and VideoPlaceButton, deriving future subtasks requires reasoning over events, and the average improvement across these tasks increases with difficulty, as shown below. These results suggest that the subtask list is especially useful when the model must convert accumulated evidence into future actions.
TaskEasy improvement (pp)Medium improvement (pp)Hard improvement (pp)
VideoPlaceOrder+3.08+16.67+40.00
VideoUnmaskSwap+3.85+25.00+5.00
VideoPlaceButton-1.54-5.00+8.33
Three-task mean+1.79+12.22+17.78
AuthorsOpenReview ↗

Reply to Review PU7N (Part 3/3)

Removal of event memory

Event memory helps when multiple observations must be accumulated before a new subtask can be determined. We fine-tune the same base model without event annotations, using the same training pipeline and settings. Across five evaluation seeds, removing event memory reduces overall success from 56.05 ± 1.38% to 54.05 ± 0.92% (mean ± sample standard deviation), a drop of 2.00 pp. Removing event memory reduces success by 6.80 pp on VideoPlaceOrder, 3.20 pp on VideoPlaceButton, and 6.40 pp on VideoUnmaskSwap. These tasks require reasoning such as identifying the third location at which a cube was placed in a demonstration. Summarizing each placement as an event makes the relevant evidence easier to retain than identifying sparse evidence in a long video. Thus, event memory complements the subtask list by preserving evidence that cannot yet be converted into an executable subtask.

We further find that event memory yields the largest average gain on Hard episodes. Across five evaluation seeds, the mean gain over VideoPlaceOrder, VideoPlaceButton, and VideoUnmaskSwap is 3.33 pp on Easy episodes, 2.22 pp on Medium episodes, and 13.33 pp on Hard episodes. This suggests that event memory can be particularly useful when more events must be retained and combined to determine future subtasks.

TaskEasy improvement (pp)Medium improvement (pp)Hard improvement (pp)
VideoPlaceOrder-6.15+10.00+31.67
VideoUnmaskSwap+13.85-8.33+5.00
VideoPlaceButton+2.31+5.00+3.33
Three-task mean+3.33+2.22+13.33

Replacement of the visual buffer with recent frames

The visual buffer helps retain the temporal context needed to track progress and infer events. Using the same model and training settings, we replace it with the four most recent frames, as in many baselines; PAL's buffer also supplies four downsampled frames. Across five evaluation seeds, this variant achieves 51.20 ± 0.95% success (mean ± sample standard deviation), 4.85 pp below PAL. It performs worse primarily on tasks requiring progress tracking or reasoning over events that span longer intervals, such as SwingXtimes (−27.2 pp), PatternLock (−18.0 pp), and BinFill (−17.2 pp). PAL samples frames from the interval since the last event-memory or subtask-list update, whereas recent frames omit earlier observations. Thus, the buffer helps preserve relevant visual evidence between memory updates.---

We thank Reviewer PU7N again for the insightful comments. The additional experiments provide evidence beyond RoboMME and clarify the contribution of each memory component. We hope these results address your concerns and invite you to reconsider your score in light of the new evidence. We welcome further discussion and suggestions.

Comments on the paper

3 comments
Area_Chair_LvBBOpenReview ↗

reviewer discussion

Dear Reviewers,

The authors have submitted their rebuttal. Please read their response carefully and engage in the discussion as soon as possible. Let us know whether the rebuttal addresses your concerns and update your final rating accordingly.

Thank you for your time and service to ICLR.

Best regards,

AC

AuthorsOpenReview ↗

General Response

We thank the reviewers and the Area Chair for their time and constructive feedback. We have uploaded a revised manuscript with changes marked in blue. The revision clarifies PAL's main contribution: it records future subtasks as relevant evidence becomes available and selects the appropriate subtask from the list at execution time. Below, we summarize the new evidence and the points we clarified in response to the reviews.

1. Planning ahead and the roles of the memory components. Across five evaluation seeds (42–46), PAL achieves 56.05% success on RoboMME, compared with 52.23% without the subtask list, 54.05% without event memory, and 51.20% when the visual buffer is replaced with the four most recent frames (Table 1 and Appendix C.1, Table 6). Each variant uses one trained checkpoint across all five evaluations. The subtask list is PAL's main contribution, and event memory complements it by retaining evidence that cannot yet be converted into an executable subtask (Sec. 5.2 and Appendix C.2). In the separate ablation on three video tasks reported in our follow-up to Reviewer PU7N (point 3), success rates dropped further when we also replaced the visual buffer with recent frames, suggesting that the buffer can partly compensate when event memory is removed.

2. What contributes to the Stage 2 gains. Continuing supervised training on Stage 1 data with the same additional training budget, update repetition, and task reweighting reaches 45.63%. Training on verified self-labeled data without repetition or reweighting reaches 55.38%, close to the full model's 56.05%. The two controls are reported in Appendix D.2 (Table 11).

3. What we mean by “reasoning burden.” The abstract and Sec. 2 now clarify that reasoning burden refers to the difficulty of finding relevant evidence in the accumulated history and using it to infer the next subtask when it is time to act. PAL addresses this by inferring and recording future subtasks as the supporting evidence becomes available, so the high-level model can later select the relevant subtask from the list. Appendix D.3 (Table 12) reports inference time separately. PAL has lower per-call latency than both MemER implementations, while its cumulative VLM inference time per episode is lower in the comparison with MemER that explicitly reasons over video.

4. Evaluation beyond RoboMME. PAL outperforms the best baselines in the original RMBench paper on both tasks we evaluate, scoring 12% versus 9% on Observe and Pick Up, which requires retaining information from one frame (M(1)), and 37% versus 28% on Battery Try, which requires retaining information from multiple frames (M(n)). On Observe and Pick Up, PAL identifies the correct object in 74% of episodes, and the failure analysis highlights how often grasping prevents successful execution. Appendix D.1 reports the evaluation (Table 10) and failure analysis. We explain the RMBench memory types in our follow-up to Reviewer PU7N (point 2).

5. Real-world data, list updates, and the GPT baseline. Real-world observations and robot states come from physical robot executions, with training trajectories collected through teleoperation. GPT-5.6-Sol refines temporal boundaries and produces event and subtask labels that humans verify (Appendix E.1). GPT-6-astra specifies task configurations and goals for training and evaluation (Appendix E.2). We also clarified that the model outputs the full subtask list at each update, while the evaluated update rule appends new entries and preserves existing ones (Appendix C.4). Editing or deleting entries was outside the evaluated setup. Finally, the GPT-6-astra baseline results apply to the tested frame-sampling and memory setup (Appendices C.4 and C.5.1).

We thank the reviewers again for helping us make the contribution and the scope of our claims clearer. The individual replies provide the full results and address the remaining questions in detail.

Dear Area Chair,

Thank you for following up with the reviewers and for coordinating the discussion. We appreciate your help and would be happy to address any remaining questions.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.