acceptodds
Withdrawn from ICLR 2027

RoboMemArena: A Challenging Robotic Memory Benchmark

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Memory is a critical component of robotic intelligence, as robots must rely on past observations and actions to accomplish long-horizon tasks in partially observable environments. However, existing robotic memory benchmarks still lack multimodal annotations for memory formation, provide limited task coverage and structural complexity, and remain restricted to simulation without real-world evaluation. We address this gap with RoboMemArena, a large-scale benchmark of 26 tasks, with average trajectory lengths exceeding 1,000 steps per task and 68.9% of subtasks being memory-dependent. The generation pipeline leverages a vision-language model (VLM) to design and compose subtasks, generates full trajectories through atomic functions, and provides memory-related annotations, including subtask instructions and native keyframe annotations, while paired real-world memory tasks support physical evaluation. We further design PrediMem, a dual-system VLA in which a high-level VLM planner manages a memory bank with recent and keyframe buffers and uses a predictive coding head to improve sensitivity to task dynamics. Extensive experiments on RoboMemArena show that PrediMem outperforms all baselines and provides insights into memory management, model architecture, and scaling laws for complex memory systems.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic Manipulation, Robotic Memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_agmMOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposed RoboMemArena, a benchmark for long-horizon tasks with partial observability, and PrediMem, a hierarchical policy which selects relevant keyframes to store in its memory. Additionally, PrediMem trains with an auxiliary loss to predict the future latent frames. They evaluate on 26 simulated tasks with subtask labels and 8 real-world tasks on a bimanual platform.

Critical strengths
  1. The component-wise ablations for PrediMem are comprehensive and help illustrate the benefits of predictive coding heads and the keyframe bank.
  2. RoboMemArena is one of a few simulated benchmarks that focus on memory for long-horizon tasks. Additionally, the have procedural data generation which allows researchers to generate data for there own long-horizon memory tasks and test the effects of scaling data.
  3. The organization, writing, and figures are clear.
Critical weaknesses
  1. The justification for predictive coding being beneficial in section 4 is hard to follow. It's unclear that this auxiliary objective will help the model decide what keyframes to select.
  2. There are lack of results on established memory benchmarks like RoboMME. This would help show PrediMem's benefits generalize to other settings that require memory.
Questions for authors
  1. Could you add an experiment showing the keyframe selection precision and recall with and without using the predictive coding auxiliary objective? This can help justify the claim made in section 4 about the benefits of predictive coding. This can help complement the t-SNE visualization in figure 5.
  2. Could you results from one external memory benchmark like RoboMME for PrediMem and its ablations?
  3. What backbone was used for the high-level policy of MemER? What were the training hyperparameters used for each method and baseline?
Reviewer_TiNaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a new benchmark, RoboMemArena, featuring long-horizon tasks that require memory. It also proposes a two-stage baseline, PrediMem, in which a VLM predicts keyframes and decomposes the task into subtasks, while a VLA executes these subtasks. PrediMem achieves the best performance on the proposed benchmark.

Critical strengths

The paper introduces a benchmark featuring memory-dependent tasks that closely resemble real-world scenarios.

Critical weaknesses
  1. The authors make somewhat arbitrary choices when comparing their benchmark with existing benchmarks. For example, although RoboMemArena tasks are indeed longer in terms of the number of steps, the threshold of 1,000 steps used to define long-horizon tasks is arbitrary and lacks justification. The history-dependent subtask ratio also raises questions, as the paper does not provide a detailed explanation of how this metric was calculated for benchmarks other than RoboMemArena. Regarding task complexity, essentially all tasks are variations of the "pick-and-place" problem. Although they better reflect real-world scenarios compared to benchmarks such as RoboMME and MIKASA-Robo, their diagnostic power and diversity remain questionable.

  2. The proposed baseline, PrediMem, is evaluated only on the proposed benchmark and the corresponding real-world environments, making it difficult to draw definitive conclusions about its generalization to other tasks. First, the authors should evaluate PrediMem on other established memory-oriented benchmarks, such as RoboMME or MIKASA-Robo. Second, they should demonstrate that the proposed memory mechanism does not degrade performance on tasks that do not require memory. For this purpose, PrediMem could be evaluated on LIBERO or LIBERO-PRO [1].

  3. The authors claim that one of the advantages of the benchmark is its multimodal memory-related annotations. However, the usefulness of these annotations beyond training the proposed baseline remains unclear.

  4. Although the real-world experiments are valuable, the practical significance of the proposed paired tasks (Section 3.1) for end users of the benchmark remains unclear.

  5. Appendix I reports standard deviations only for the averaged results presented in Table 2.

[1] Zhou, Xueyang, et al. "Libero-pro: Towards robust and fair evaluation of vision-language-action models beyond memorization." arXiv preprint arXiv:2510.03827 (2025).

Questions for authors
  1. How do the authors explain why a recent-frame window size of 4 or 8 yields worse results than a window size of 5 (Figure 5a)? Were the same values of recent-frame window size and keyframe-bank capacity used across all environments in the experiments?

  2. What are the standard deviations for the individual tasks reported in Table 2?

  3. How were the other baselines trained?

Reviewer_1EA6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RoboMemArena, a benchmark for evaluating memory in long-horizon robotic manipulation, and introduces PrediMem as a baseline memory-augmented VLA. The benchmark contains 26 simulation tasks covering four types of memory requirements: transferring, occlusion, counting, and sequential execution. It includes 2,600 demonstrations, with an average trajectory length of 1,076 steps and 68.9% of subtasks classified as history-dependent. The authors also introduce five real-world tasks for physical evaluation.

PrediMem follows a dual-system design. A high-level VLM planner (System 2) maintains a recent observation buffer and a keyframe buffer, and predicts both the current subtask and whether an observation should be stored. A low-level VLA (System 1) executes the predicted subtask. The authors further introduce an auxiliary predictive coding objective to predict the next-frame visual representation, with the aim of improving the representation of task-relevant state transitions.

On the simulation benchmark, PrediMem achieves 38.5% TSR and 55.2% CSR, outperforming the evaluated baselines. It also achieves 52% average success on the five real-world tasks, compared with 40% for MemER and 20% for π0.5. The paper includes ablations on the memory components and predictive coding, as well as studies of memory capacity and planner model size.

Critical strengths
  1. The benchmark addresses a relevant and underexplored problem in robotic manipulation. Many existing manipulation benchmarks do not explicitly test whether a policy can retain information from earlier interactions. RoboMemArena includes tasks involving occlusion, counting, object transfer, and sequential execution, where the correct action can depend on events that are no longer visible. The 26 tasks, long trajectories, and subtask-level annotations provide a useful testbed for studying this problem.

  2. The evaluation covers both simulation and real-world manipulation. The authors compare several memory-augmented VLA approaches and provide ablations on the keyframe bank and predictive coding objective. The results offer some useful evidence about the role of explicit historical observations in long-horizon tasks. In particular, the improvement over MemER on the simulation benchmark is worth investigating further.

Critical weaknesses
  1. The diversity and difficulty of the benchmark need further validation. Although RoboMemArena contains 26 tasks and a high proportion of memory-dependent subtasks, many tasks are constructed from a relatively small set of predefined manipulation primitives, such as Move, Place, Pour, Open, and Close. Several tasks also share similar structures, with variations mainly in object identities or target locations. It is therefore unclear how well the benchmark measures general memory capabilities beyond the specific task patterns used in its construction. In addition, the 68.9% history-dependent subtask ratio is based on the authors' classification of task dependencies and does not directly establish the actual difficulty of these subtasks for a robotic policy.

  2. The experimental evidence for PrediMem is not yet fully convincing. The improvement over existing methods is encouraging, but the contribution of predictive coding is mainly supported by an overall ablation and a t-SNE visualization of hidden representations. More direct evidence connecting predictive coding to better keyframe selection would strengthen the claim. The real-world evaluation is also relatively small, with only 10 rollouts per task across five tasks. In particular, the 52% average success rate should be interpreted cautiously given the limited number of trials. More information about evaluation variability and the experimental protocols used for different baselines would also help establish the reliability of the comparisons.

Questions for authors
  1. How are the training and evaluation splits constructed for the 26 simulation tasks? Are the evaluation initial states, object configurations, and task variations sufficiently different from those used during training? Could the authors clarify how they prevent the model from exploiting recurring task structures or memorizing specific execution patterns?

  2. Can the authors provide a more direct evaluation of keyframe selection quality, such as precision and recall against the annotated keyframes, with and without predictive coding? This would help establish whether the improvement in task success is actually related to better keyframe selection.

  3. How are the different baselines trained or adapted to RoboMemArena? In particular, do they use the same demonstration data, training budget, and observation history? A clearer description of these settings would help assess the fairness of the comparison.

  4. For the real-world experiments, were the initial conditions and object configurations controlled across different methods? Can the authors provide success counts or variability across trials for each task? Given the relatively small number of rollouts, this information would help assess the reliability of the reported improvements.

Reviewer_9MdROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors introduce a robotic simulation benchmark for evaluating embodied memory. RoboMemArena has four categories of tasks and eight real-world tasks. Simulated data comes from a three-stage pipeline. RoboMemArena has 100 demonstrations per task (1000+ steps on avg). Authors label ~69% of subtasks as memory-dependent.

Authors also introduce a PrediMem framework for the problem. PrediMem reports ~39% task success in simulation and 51% on a real robot.

Critical strengths
  1. One of the most cleanly written papers I have reviewed at this year’s ICLR. It is easy to follow, understand the context, and the motivation behind the work.
  2. The authors have tried to deal with an excellent and relevant problem
  3. Pipeline generation is scalable. The framework produces long demos without teleoperation.
  4. Most benchmarks follow task success rate as their main eval method. For each task, RoboMemArena also has Cumulative Success Rate that records partial progress.
  5. The authors also perform some useful diagnostics.
Critical weaknesses
  1. The simulation protocol is misreported. Table 2, section 5.1, and Appendix I say, Table 2 averages three training seeds with 100 rollouts per task. In 9 of the 12 rows of Table 2, every per-category task success rate (TSR) is an exact success count from 10 rollouts per task (PrediMem: 9/40, 30/110, 32/70, 29/40). Every average TSR equals total successes divided by 260 (PrediMem: 100/260 = 38.5%, from Table 2). In the released anonymous code (Reproducibility Statement), leaderboard_data.json lists PrediMem's protocol as "trials=10" with the same task success rate values.

  2. Predictive coding head PrediMem adds 16 successes out of 260. This result is not significant at that sample size.

  3. Hyperparameters are selected on the eval tasks. For example: loss weights (Table 3), window size, and keyframe capacity (Figure 5). They are tuned on the eval tasks.

  4. Memory dependence is asserted, and it is not measured. 68.9% ratio comes from the authors' own labels, with no agreement study. Ratios for other benchmarks come from LLM-assisted first pass (app D).

  5. RoboCerebra [1] is not used for the memory comparisons. RoboCerebra is built on LIBERO, has Memory Exploration and Memory Execution tasks, and averages 2,972 steps per trajectory. The authors' README says the task files "can be used with LIBERO/RoboCerebra evaluation environments". But in the main paper, Figure 3 and Table 9 exclude this, and also exclude MIKASA-Robo-VLA's long-horizon tier.

[1] RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation [NeurIPS 2025]

Questions for authors
  1. Can the authors provide the standard deviation for each task for each method in Table 2? Explain how the Appendix I standard deviations were computed. I want to verify whether the results fall within the noise bands.
  2. How many seeds and rollouts produced each entry in Tables 2 and 3? Kindly provide per-seed results for the stated protocol
  3. Why are RoboCerebra and MIKASA-Robo-VLA excluded from Figure 3 and Table 9?
  4. Which simulator and robots are used? Do eval episodes come from the training tasks and how are those states sampled?
  5. Weakness 1
  6. Weakness 3
  7. How were the baselines trained, and which backbone does MemER use here?
  8. Where do the 500 "held-out" trajectories in Table 10 come from, given that 100 demonstrations per task are used for training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.