- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how robots can remember visual evidence that is no longer visible and use it in later actions. PAL uses a visual buffer, event memory, and a subtask list. PAL achieves an average success rate of 56.05% on RoboMME. Across four real-world tasks, the average is 60.83%, compared with 45.00% for MemER. The gains are more noticeable in long-video understanding, retaining evidence during execution, and counting repeated actions.
Critical strengths
- The paper addresses a specific problem: evidence about what to do later may appear before the action needs to be carried out. Saving subtasks in advance is a reasonable and easy-to-understand solution. On RoboMME, the average success rate improves by 11.54 percentage points over the strongest baseline. The long-video group reaches 81.60%, an improvement of 35.60 percentage points, while the group of tasks that obtain evidence during execution improves by 19.15 percentage points. The gains mainly occur in the scenarios the authors aim to address.
- The experimental design is reasonable. The sparse-label version, the first-stage-only version, and the full method achieve 43.39%, 37.35%, and 56.05%, respectively. The authors also report update frequency and final list accuracy, and include a ground-truth update gate. I find this more informative than reporting only the final success rate. The training details and failure cases in the appendix are also useful.
Critical weaknesses
- Key experiments and details are missing. The current ablations show that the second stage is useful, but do not separately remove the event layer, the list of future subtasks, or planning ahead while keeping the VLM, data, and training budget the same. It is therefore unclear whether the full three-layer structure is necessary or how much writing subtasks in advance contributes. The GPT baseline samples at most 24 frames from the conditioning video, whereas PAL processes the history as a stream. They therefore differ in both the amount of information they receive and how they are adapted through training. Using the same low-level VLA helps the comparison, but the full performance gap cannot yet be attributed to the memory representation.
- The paper emphasizes reducing the high-level reasoning cost, but provides no comparison of high-level inference time, token counts, or GPU memory usage. A fixed number of input images only limits the visual input; the subtask list itself may still grow. In the real-world experiments, the improvement over MemER mainly comes from PickXtimes, while performance remains weak on tasks requiring fast responses or route coordination. Training also relies on reference information for each task and exact text matching. Its effectiveness on new tasks and recovery from incorrect memories remains unclear. In particular, how do these costs change as the list grows?
Questions for authors
- Please compare versions without the event layer, with event memory only, and without writing subtasks in advance, using the same model, data, and training budget. These comparisons would help determine the contributions of the three-layer structure and planning ahead. How much improvement comes from the second stage?
- Are there controls with the same number of training steps, the same resampling, and the same task weighting to separate the effect of validated self-labeling from that of additional training? After failed trajectories are filtered out, how much data remains for each task, and does the task difficulty distribution change?
- Please clearly report the high-level VLM inference time per call and per complete task, input and output token counts, and GPU memory usage. Compare these with the baselines under the same observation and query budgets. How do these costs change as the list grows?
- Data sources and list modification rules. Appendix D.2 states that GPT-6-astra generated training data, evaluation trajectories, and task configurations. Which parts of the inputs, labels, and scripts does this refer to? Which records come from independently collected robot executions? The main text describes only appending to the list, whereas the appendix allows list modification. What rules are actually used? I am also concerned about the reliability of the data generated by GPT-6-astra and would like a detailed explanation from the authors.
6 comments
Reply to Reviewer ozHA (Part 2/3)
Q3 Experiments on Inference Costs
Please clearly report the high-level VLM inference time per call and per complete task, input and output token counts, and GPU memory usage. Compare these with the baselines under the same observation and query budgets. How do these costs change as the list grows?
Thank you for asking us to report computational costs and examine how they change as the list grows. We compare PAL with Memer, a similar hierarchical framework where VLM gives current subtasks to low-level VLA. PAL has lower VLM inference time per call and per episode, uses fewer input and output tokens per call, and requires less peak GPU memory. We profile all 16 tasks using one evaluation seed per model and report the results below.
For MemER implemented by RoboMME, the VLM receives downsampled video frames at each query without a separate video-reasoning phase. For fair comparison, we train a MemER high-level VLM with keyframe annotations in conditioning video at the timesteps where PAL has event or subtask-list annotations, enabling explicit reasoning during the video phase. We keep the training budget and other settings, the same as PAL. We also keep the same query interval, but the number of query calls each episode may differ because the episode will terminate in advance if it fails or successes. The resulting variant achieves an overall success rate of 44.88%, compared with 42.38% from original RoboMME Implementation.
| Method | VLM inference time / call (s) | Cumulative VLM inference time / episode (s) | Input tokens / call | Output tokens / call |
|---|---|---|---|---|
| MemER (Explicit Video Reasoning) | 1.725 | 122.412 | 988.45 | 24.77 |
| PAL | 1.380 | 82.059 | 623.91 | 21.46 |
| MemER (Original RoboMME Implementation) | 2.359 | 55.958 | 1,230.22 | 33.25 |
PAL's memory remains compact as tasks progress. We divide the evaluation rollouts into ten phases based on their relative timesteps; the reported PAL averages range from 531.99 to 675.89 input tokens per call, a difference of approximately 27%, while MemER with explicit video reasoning uses 1,137.95 input tokens in the final phase. PAL's longest subtask list contains 123 tokens, and the mean of the per-task maximum list lengths across the 16 tasks is 63 tokens. Peak GPU memory usage is also lower for PAL: 9.98 GiB, compared with 10.95 GiB for MemER with explicit video reasoning. Together, these results show that PAL's compact memory reduces inference costs when both methods explicitly reason over video.
We also evaluate MemER's original RoboMME implementation on inference computation costs. PAL still has lower per-call latency, input and output token counts per call, and peak GPU memory usage (9.98 vs. 10.36 GiB). The exception is cumulative VLM inference time per episode (82.059 vs. 55.958 s), reflecting PAL's additional video-reasoning phase. These measurements show that PAL retains its advantage on the other reported cost metrics compared with the original implementation.
Reply to Reviewer ozHA (Part 3/3)
Q4 Clarification on Data Source
Data sources and list modification rules. Appendix D.2 states that GPT-6-astra generated training data, evaluation trajectories, and task configurations. Which parts of the inputs, labels, and scripts does this refer to? Which records come from independently collected robot executions?
Thank you for asking us to clarify the sources of the real-world data. We use two LLMs, GPT-5.6-Sol and GPT-6-astra. We distinguish real-world data collection, data generated by GPT-5.6-Sol and data generated by GPT-6-astra as below:
- Real-world data collection: The input images and robot states come from real-world teleoperation.
- GPT-5.6-Sol: It refines the temporal boundaries of recorded trajectories and produces labels for event and executable subtask supervision. These data have been verified by human.
- GPT-6-astra: It generates task configurations and goals for both training and evaluation episodes. For example, in PatternLock, it specifies the starting point and the movement sequence to be demonstrated, such as “1. Move right; 2. Move left.”
In conclusion, the observations and robot states are collected from physical executions, GPT-5.6-Sol annotates the recorded trajectories, and GPT-6-astra specifies the tasks.
The main text describes only appending to the list, whereas the appendix allows list modification. What rules are actually used?
Thank you for pointing out the ambiguity in our description of the list-update rules. Our experiments focus on appending subtasks, although the model outputs the entire list even when only one new entry is added. This output format leaves room for future work on revising existing entries. We will revise the wording to distinguish the update behavior used in our experiments from this possible extension, and we plan to release the code for future development. Thus, list revision is a future direction; please also see our response to Reviewer 32vj, Q4.
We thank Reviewer ozHA again for the insightful comments. Our responses clarify the three-layer memory design, the Stage 2 controls, computational costs, and data annotation procedure. We hope these explanations address your concerns and invite you to reconsider your score in light of the additional results. We welcome further questions and discussion
Reply to Reviewer ozHA (Part 1/3)
We thank the Reviewer of acknowledging our performance gains and experiment design. We address the questions regarding ablations on memory components, stage 2 training, inference costs and data sources as below.
W1/Q1 Ablation on memory components.
Thank you for asking for controlled comparisons of the memory components and planning ahead. Keeping the other settings unchanged, we separately remove the subtask list, remove event memory, and replace the visual buffer with recent frames (we take "with event memory only" and "without writing subtasks in advance" as the same variant). All three ablations perform worse overall than PAL, supporting the contribution of each memory component. Our analysis further identifies when each component is most useful. Together, these results clarify their respective roles; please see our response to Reviewer PU7N, W2/Q1, for the full results and discussion.
Thank you for asking about the contribution of Stage 2. Stage 2 improves overall success from 37.35% for Stage 1 Only to 56.05% for PAL. Section 5.2 of the main text reports this comparison and analyzes how Stage 2 regulates memory updates. These results support the contribution of Stage 2 to more reliable memory updates and higher task success.
Q2 Ablation on Controlled Finetuning of Stage 2
Thank you for asking for controls that separate verified self-labeling from additional training and data augmentation. Our controls suggest that verified self-labeling accounts for most of the Stage 2 improvement. Additional supervised training with the same training budget and augmentation remains below PAL, whereas training on verified self-labeled data without non-empty update repetition or task reweighting retains most of PAL's performance. These results support the contribution of verified self-labeling; please see our response to Reviewer 32vj, W2/Q2, for the full comparison.
Thank you for asking about data retention and changes in task difficulty after filtering. Most episodes are retained after verification, with the lowest acceptance count being 84 out of 100 for VideoRepick. The “One-pass labeling” paragraph in Appendix B.2 reports the accepted episode count for each task. Thus, filtering retains the majority of the data for every self-labeled task.
The overall difficulty distribution changes only slightly, although individual tasks show larger shifts. For VideoPlaceOrder, the proportion of Hard episodes decreases from 25.00% to 20.93%; for VideoRepick, it increases from 25.00% to 29.76%. The aggregate proportions are shown below. Thus, the overall distribution remains similar, while the effect of filtering varies by task.
W2 Explanation about GPT Baseline
Thank you for raising the difference in how the GPT baseline and PAL process conditioning videos. We agree that the GPT baseline could perform better with a more carefully designed inference framework. We organize its memory similarly to FrameSamp, the strongest baseline proposed in RoboMME, which is straightforward to implement and uses a hierarchical framework similar to Physical Intelligence's Mem. Designing a stronger framework for GPT is beyond the scope of this work. Our results therefore show that GPT can struggle with memory-dependent tasks under the evaluated memory representation, rather than establish a limit on its capabilities.