acceptodds
Withdrawn from ICLR 2027

RoboCasa-Mem: Benchmarking Environment and Process Memory in Long-Horizon Mobile Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In the real world, mobile manipulation robots must perform complex object interactions and make multi-step decisions, often relying on task-relevant information that is no longer observable. However, existing benchmarks provide limited evaluation of such memory-dependent decision-making in mobile manipulation. To address this gap, we introduce RoboCasa-Mem, a new benchmark comprising nine mobile manipulation tasks that require agents to use interaction history at key decision points. Specifically, the tasks are grouped into two categories of memory: One is environment memory, which captures spatial locations, object attributes, and object relationships. The other is process memory, which captures single-task progress and multi-task scheduling. Particularly, we introduce Behavior-Conditioned Evaluation. For tasks with multiple valid execution paths, instead of fixing a single target or trajectory, the evaluator derives later task constraints from the robot's earlier actions. This allows us to evaluate whether later behavior is consistent with the robot's own interaction history while reducing shortcuts based on fixed target assignments. We provide 1,800 successful demonstrations for policy training. We also develop a baseline on our benchmark, namely Hierarchical Memory-Augmented Policy (HMAP). Its high-level planner uses structured entity and process memories to generate subtasks. Its low-level VLA controller uses subgoal-conditioned queries to incorporate VGGT- geometry into action generation. Together, \benchmark and HMAP baseline support systematic evaluation of memory-dependent behavior and the development of memory-aware mobile manipulation policies.

Primary area:
datasets and benchmarks
Keywords:
Embodied AI, Benchmark, Robotics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eX5iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a benchmark for long horizon manipulation with a focus on tasks which require specific types of memory.

Critical strengths

Overall, the paper is well written and easy to follow. The fact that successful demonstrations makes this benchmark easier to adopt and. Real-world experiments are always good to see.

Critical weaknesses

The position of this paper is in a slightly odd place, if the focus is on the memory and reasoning, then there are major omissions of the literature, if the focus is on the control, then the evaluation is somewhat lacking.

For reasoning, PARTNR [1] or ALFRED [2] (which are omitted from discussion) are highly relevant. PARTNR test similar reasoning at a much larger scale (around 100k tasks). With only nine tasks, and visually hidden attributes appearing mainly in SweetCup and TempReport, it isn't clear to me what this benchmark adds beyond these. PARTNR can also be run with low-level control, so building on RoboCasa doesn't by itself set this work apart. This paper would benefit discussing how this work should be viewed in relation to these existing benchmarks.

If the focus is control, then some demonstration of sim-to-real transfer is needed to show that the simulation is high enough fidelity for benchmark scores to correlate with real-world performance. The choice to perform a real-world task which is not one of the benchmark tasks undercuts the value of the benchmark. Real-world versions of even two or three benchmark tasks, with the method ranking compared against Table 3, would make the paper stronger.

Also, the real-world results come from 10 trials per method, and ACT and DP were trained but aren't reported in Table 4. Confidence intervals following Agarwal et al. [3] would help here.

Finally, the average SR improvement over π0.5 comes almost entirely from TempReport. This almost makes it seem like this task is an outlier or pathological in some way. This paper would benefit from more tasks so the results can better differentiate methods.

Questions for authors

"let Nj denote the number of valid evaluation episodes" What exactly makes an evaluation episode invalid?

"All methods are trained on the same task demonstrations." How are ACT and DP conditioned on the language instruction?

Reviewer_jFu9OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces RoboCasa-Mem, a benchmark of nine mobile manipulation tasks designed to evaluate so called environment and process memory, covering spatial, attribute, and relational memory, task progress, and multi-task scheduling. The authors propose Behavior-Conditioned Evaluation, which determines the correctness of later actions based on the agent's earlier behavior rather than fixed target assignments, and provide 1800 successful demonstration trajectories (200 for each of nine tasks). They also introduce their own memory-augmented model, a Hierarchical Memory-Augmented Policy (HMAP), a baseline combining a Qwen3-VL-4B high-level planner with structured entity and process memories and a pi0.5 controller augmented with VGGT-Omega geometric features. HMAP improves average success from 16.2% to 24.2% over non-memory Markovian baseline pi0.5, with the largest gains in hidden-attribute recall. The authors additionally evaluate the method on a real-robot object transport task using the AgileX Cobot Magic platform.

Critical strengths
  1. Extending memory evaluation to mobile manipulation is a useful direction, which is definitely should be developed next after the tabletop manipulation. The proposed tasks include situations where historical information becomes unavailable after navigation or manipulation

  2. Reporting both final success and intermediate stage completion helps identify where long-horizon policies fail

Critical weaknesses
  1. Insufficient evidence of memory dependence. The paper defines memory-intensive tasks using observation aliasing, but does not experimentally establish this property. Some tasks may admit deterministic shortcuts. For example, in a SweetCup task, a policy could always add sugar to the same cup and later select that cup without explicitly remembering its earlier action. Behavior-Conditioned Evaluation does not necessarily eliminate such shortcuts. Counterfactual evaluations with identical current observations but different histories would make the benchmark substantially more convincing

  2. Limited novelty and coverage of the benchmark. The proposed Environment vs. Process Memory taxonomy overlaps with existing distinctions between world and process memory, while benchmarks such as MIKASA-Robo and RoboMME already cover several related memory capabilities. The benchmark contains only nine tasks, and the paper does not sufficiently justify why these specific categories were selected or why existing memory-intensive tasks could not be adapted to mobile manipulation. I would also like to see a more careful discussion of memory-dependent activities in BEHAVIOR-1K and other mobile manipulation benchmarks

  3. Lack of a difficulty scale for each task. To investigate the memory limitations of the memory-augmented models being benchmarked, it is necessary to introduce progressive difficulty scales for each task. This will help determine at what point and for what reason the memory fails (e.g., due to an extended time horizon or increased task semantic complexity)

  4. Unclear contribution of mobile manipulation. The paper provides very limited information about the actual environments, navigation layouts, and robot trajectories. There are no top-down scene maps, navigation-distance statistics, or detailed analyses of how mobility changes the memory requirements. It is therefore difficult to assess whether the benchmark introduces substantially new challenges beyond tabletop memory tasks with additional navigation

  5. The reported HMAP improvement is dominated by one task. The average SR increases from 16.2% to 24.2%, but TempReport alone improves from 22% to 86%, accounting for approximately 89% of the overall gain. Excluding TempReport, the average improvement is only 15.5% to 16.5%. On Process Memory tasks, HMAP does not improve the average SR, and neither multi-task scheduling task is solved by any method

  6. Insufficient isolation of the memory contribution (ablation study). HMAP simultaneously introduces a high-level VLM planner, structured memory, subgoal decomposition, and VGGT-Omega geometric conditioning. Although the authors provide a progressive ablation on SingleMilkMatch, they do not compare the same planner with and without persistent memory. Consequently, it is difficult to determine whether improvements come from memory, better task decomposition, or geometric features. Oracle-memory and corrupted-memory experiments would also help establish whether the policy actually uses the stored information

  7. Limited and potentially weak baseline comparisons. ACT, Diffusion Policy, and reactive pi0.5 are not modern and strong memory-intensive baselines (they are non-memory by the way). The paper would benefit from comparisons with simple observation-history, recurrent-memory, and keyframe/textual-memory variants of pi0.5. The MemER-style implementation also requires more explanation, especially given its surprisingly low performance. Differences in action chunking, execution protocols, and training settings are not sufficiently documented to assess experimental fairness. Moreover, the absence of various families of memory-intensive baselines from RoboMME also limits the generality of the evaluation

  8. Insufficient dataset-scale and error analysis. The benchmark provides 200 successful demonstrations per task, but there are no data-scaling experiments demonstrating whether this is sufficient for training large VLAs on complex mobile manipulation tasks. The absence of recovery trajectories may also limit learning of process memory under execution errors. Furthermore, SR and SCR do not adequately distinguish memory failures from navigation, planning, and manipulation failures

  9. Major reproducibility gaps. The exact simulated robot model is not specified. The paper reports 13-dimensional actions and 37-dimensional proprioception but does not provide their complete semantics, units, or normalization. Action chunk sizes and execution protocols are missing, and important training details, including optimizer settings, LoRA configurations, and some task-specific schedules, remain unspecified. The paper itself states that several reproducibility details will be provided later. For a benchmark contribution, these omissions are particularly concerning

Questions for authors
  1. Memory dependence. How do you verify that each task genuinely requires historical information rather than allowing deterministic or reactive shortcuts? Could you provide counterfactual evaluations with identical current observations but different histories?

  2. Benchmark novelty. What is the main novelty of the proposed memory taxonomy compared with MIKASA-Robo, RoboMME, and RMBench, except the navigation part? How does RoboCasa-Mem differ from potentially memory-dependent tasks in BEHAVIOR-1K, and what new memory challenges arise specifically from mobile manipulation?

  3. Mobile manipulation environments. Could you provide scene maps, top-down robot trajectories, navigation-distance statistics, and memory-horizon distributions? These would help establish the role of mobility in the proposed tasks

  4. Robot and control interface. What exact robot model is used in simulation? Please specify the complete 13D action and 37D proprioception schemas, including units and controller semantics. What are the action chunk sizes, policy-query frequencies, and execution strategies for each baseline?

  5. Dataset scale. Why were only 200 successful demonstrations collected per task? How does performance scale with dataset size, and how diverse are the navigation routes, scene configurations, and execution sequences?

  6. Baselines and memory contribution. Why are stronger memory-augmented VLA baselines, such as recurrent memory or keyframe retrieval, not included? Could you evaluate HMAP with the same high-level planner but without persistent memory, as well as with oracle or corrupted memory, to isolate its causal contribution?

  7. Experimental results. Almost 89% of the average SR improvement comes from TempReport, while HMAP does not improve average Process Memory SR and fails on both multi-task scheduling tasks. What conclusions about general memory capabilities are supported by these results? Could you also provide memory-decision accuracy separately from navigation and manipulation success?

  8. HMAP implementation and reproducibiliyt How are structured memory annotations generated, and do they rely on privileged simulator states? How are the learnable queries and VGGT-Omega features integrated into pi0.5? Could you provide the missing training hyperparameters, planner configurations, and checkpoint-selection rules?

Reviewer_LV6YOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces RoboCasa-Mem, nine simulated mobile-manipulation tasks in RoboCasa in which a later decision depends on information that is no longer visible. It motivates the benchmark by the claim that mobile robots "lose visual access to previously observed entities" as they move, and that memory "under changing viewpoints and cross-region execution" is under-evaluated (Sec. 1). Tasks are grouped as environment memory (spatial, attribute, relational) and process memory (progress, scheduling). The paper contributes 1,800 scripted demonstrations, stage-level metrics, and Behavior-Conditioned Evaluation, in which later targets are derived from the policy's own earlier choices. The baseline, HMAP, pairs a Qwen3-VL planner holding text entity/process memory with a π0.5 controller conditioned on current-frame VGGT-Ω features. It raises average success from 0.16 to 0.24 over π0.5, and one real-robot task is reported (8/10 vs. 2/10).

Critical strengths
  1. The design criterion is the right one: the same current observation must call for different actions depending on history (Eq. 1). Benchmarks scored on how long a task is cannot make this distinction, and moving it to mobile manipulation is a real gap.
  2. Stage-level predicates and the conditional rates in Appendix B show where policies fail. For example, TempReport's correct-report rate given arrival goes from 0.41 to 0.90.
Critical weaknesses

In short: the benchmark does not test the memory problem that makes mobile manipulation different from tabletop work, which is information leaving view because the base moved. Its baselines are not set up to measure how much memory each task demands.

  1. The tasks and the baseline do not address memory under base motion, which is the paper's own motivation. In mobile manipulation, what makes memory harder than on a tabletop is that the robot's movement changes what it can see. Objects leave and re-enter view, and a remembered location has to stay valid after the base moves. In 8 of the 9 tasks, however, the information is hidden by a state change that would hide it the same way on a tabletop: sugar dissolves, water temperature isn't visible, cabinets close, or the robot has to keep a count or a pending task in mind (App. A). Only MilkMatch hides information by turning the base. PutBackCup, the single "spatial memory" task, hides the location by moving the cup, not the robot. HMAP likewise has no memory tied to the robot's position. It stores locations as text, its planner sees only the current frame and the frame from 50 steps earlier, its controller uses current-frame geometry only, and no base-pose, odometry or map input is described (Sec. 4). As a result, the benchmark and baseline cannot show whether a policy can remember where something is after the robot has moved. Needed: Add tasks where the decision depends on something observed from a different base pose, such as an object seen at one counter that must be retrieved after navigating elsewhere. Report, for each task, how far the base moved and how much the view changed between seeing the information and needing it. Include a baseline whose memory is tied to the robot's pose, such as a semantic map or 3D scene memory.

  2. The benchmark does not measure how much memory each task requires, or how much each baseline can hold, so the results cannot be read as memory capacity. Every baseline has some memory. ACT and π0.5 commit to action chunks, Diffusion Policy conditions on a short observation history, MemER-style retrieves keyframes, and HMAP carries recurrent text memory plus a frame from 50 steps earlier. The meaningful question is whether a task needs information from further back than a policy can retain. The paper characterizes neither side. It does not report how long the gap is between seeing the information and needing it, it does not state each baseline's effective context, and it does not vary either. The results are consistent with tasks that fit inside short contexts: π0.5 reaches 0.68 on PutBackCup and ties HMAP on SugarCount (0.28). The average gain over π0.5 nearly disappears without TempReport (0.155 vs. 0.165). In three tasks everything fails before the memory decision (Table 7). Behavior-Conditioned Evaluation also lets a fixed rule pass SweetCup ("sugar the left cup, pick the left cup"). Needed: Follow the controls used by the benchmarks the paper compares against in Table 1:

    • a ground-truth-memory upper bound (RoboMME's oracle-subgoal setting, RoboMemArena's "Ground Truth" row);
    • an explicitly current-observation-only π0.5 (as RoboMME states for its π0.5 baseline);
    • a sweep over the memory or history budget (RoboMME varies 64–512 memory tokens; RoboMemArena varies frame-window and keyframe-bank size);
    • a per-task label of how much must be remembered (RMBench's M(0)/M(1)/M(n)).

    Also report, per task, how many steps and how much base motion separate seeing the information from using it.

Questions for authors
  1. How far does the base move, in meters and degrees of heading change, between observing the information and using it in each task? In how many tasks is the information out of view because of base motion rather than a state change?
  2. What does each task reach with a ground-truth-memory planner feeding the same controller, and how does HMAP change as its memory or history budget is reduced? Does the π0.5 baseline see only the current observation?
  3. How does a memory tied to the robot's pose (e.g., a semantic map or 3D scene memory) perform on PutBackCup, CanRecall and MilkMatch?
  4. In SweetCup and MilkMatch, are positions re-randomized after the earlier choice so that a fixed spatial rule cannot pass the behavior-conditioned check?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.