- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a structured memory framework called OCLM. This framework is designed to address the challenges of reusing interaction knowledge and tracking states in long-horizon embodied manipulation tasks. The experimental results shows the effectiveness of the proposed framework.
Critical strengths
- The introduction is logical and clear, which describes the motivation of the proposed OCLM well.
- The experimental results (especially in Table 1-3) indeed show the effectiveness of the proposed OCLM.
Critical weaknesses
-
The ablation study on the whole Object–Concept Linked Memory is missing. The ablation in Table 4 removes individual memory components but retains the overall OCLM architecture, the same LLM, and the same tool chain. While this shows which specific memory component matters most, it fails to isolate the Object–Concept Linked Memory itself. It is suggested to add an additional baseline using the same agent and the same perception tools but relying on a simple short-term or vector-based memory instead of structured memory. Without this, the large performance gap between Ours(DP) (71%) and the standalone DP (7%) cannot be definitively attributed to OCLM rather than to the strong LLM and tool chain.
-
The comparisons with SOTA VLA methods are missing. The paper compares OCLM primarily with memory-based baselines (e.g., MemER, RoboEXP) and treats rapidly evolving end-to-end VLA models merely as "memory-less" black boxes. The paper’s impact would be significantly strengthened if it demonstrated how OCLM could serve as a plug-and-play module to enhance the long-horizon capabilities of existing SOTA VLA models, or if it provided a fairer, more detailed comparison under equivalent computational budgets.
Questions for authors
- The paper claims OCLM provides a "compact" memory representation, but why are there no quantitative storage measurements (e.g., byte size or growth curve) to support this claim?
3 comments
I thank the authors for the added storage measurements and the π0.5 experiment. Two points remain.
W1. The flat-record (59%) and generic-object-record (62%) controls still retain long-term memory, differing from OCLM only in memory organization. They isolate "structured vs. flat memory," not "memory vs. no long-term memory." Please add a short-term-memory-only baseline (same LLM, tools, and execution stack, no persistent structured memory) to directly isolate OCLM's gain.
W2. As the authors note, the two π0.5 configurations use different fine-tuning demonstrations (50 vs. 10 per handle location), so the 20% vs. 80% gap cannot be attributed to OCLM. Please either re-run under an identical demonstration and compute budget, or explicitly downgrade this to a case study and report the training/inference compute costs of both settings.
Response to W1
Thank you for this suggestion. We added a short-term-only baseline with the same LLM, concept templates and perception/grounding tools. At each subtask boundary, agent context and dynamic memory are reset; structured scene information is rebuilt using only the preceding subtask segment and current observation. The task instruction is retained, but earlier history is unavailable.
| Task | SR |
|---|---|
| Rearrange Blocks | 28% |
| Put Back Block | 18% |
| Swap Blocks | 19% |
| Swap T | 34% |
| Battery Try | 14% |
| Blocks Ranking Try | 15% |
| Cover Blocks | 5% |
| Mean | 19% |
Using 100 rollouts per task, short-term-only achieves 19% versus OCLM's 71%, now reported in Table 4. This decrease is consistent with RMBench's reliance on history beyond the preceding subtask. Although short-term memory is insufficient to retain the full task history, recent spatial changes, together with the agent's understanding of the task and access to the shared tools, can still provide enough information to complete some tasks in a limited fraction of trials.
Response to W2
Because the standalone VLA and agent-based system differ in their execution workflows and fine-tuning demonstrations, the current experiment does not provide a fully controlled comparison. Following your recommendation, we present it as an integration case study. The table below summarizes the resource usage, configurations, and performance of both systems under the same training and inference hardware, with a common 60 s episode limit that includes model inference latency.
| Item | Standalone π0.5 | OCLM + π0.5 |
|---|---|---|
| Number of demonstrations | 50 | 90: ten at each of nine handle locations |
| Mean demonstration duration | 16.4 s | 6.8 s |
| Total demonstration duration | 821 s | 612 s |
| Recording frequency | 30 Hz | 30 Hz |
| Training hardware | 4 × NVIDIA H200 Tensor Core GPU | 4 × NVIDIA H200 Tensor Core GPU |
| Training epochs | 12 | 12 |
| Inference GPU | NVIDIA GeForce RTX 5090 | NVIDIA GeForce RTX 5090 |
| GPU-memory usage during inference | 15.3 GB | 22.6 GB |
| LLM inference latency per episode | Not applicable | 35.4 s, included in the 60 s limit |
| Additional high-level LLM tokens | None | 33,066 input + 336 output tokens |
| Wall-clock time limit per episode | 60 s | 60 s |
| Successful trials within 60 s | 4/20 | 16/20 |
| Success rate within 60 s | 20% | 80% |
With demonstration data of a comparable scale, this integration case study demonstrates the potential of OCLM to improve VLA performance on memory-dependent tasks with manageable additional GPU-memory and LLM-token costs (under USD 0.50).
Thank you for these valuable suggestions on isolating the contribution of memory, evaluating integration with stronger VLA executors, and quantifying storage costs. Following your comments, we have clarified the shared-stack controls and added storage measurements and analysis to the appendix.
1. Whole-memory replacement controls
[W1] Sec. 5.1 identifies Ours(DP)'s 71% versus standalone DP's 7% as the combined agent and memory effect. The targeted comparison is the flat-record baseline in Table 4: it retains the same agent, perception tools, and execution capabilities, but serializes memory facts into flat records and uses embedding retrieval instead of structured cross-memory retrieval. It achieves 59%, versus OCLM's 71%. The generic-object-record control achieves 62%. Sec. 5.4 and App. B.1 distinguish these replacement controls from component-removal ablations. They assess memory organization/retrieval and template-linked representation within the shared stack.
2. Integration with a stronger VLA executor
[W2] OCLM provides memory to the high-level agent, while a VLA serves as a policy-grounded skill; this interface is described in Secs. 4.1.2/4.4 and App. E.3. The agent uses memory to coordinate subtasks and invokes the executor for local actions. Reliable execution can require subtask-specific fine-tuning.
[W2] To examine this integration, we conducted an additional real-world search experiment with π0.5. The standalone baseline was fine-tuned on 50 teleoperated demonstrations, each comprising placing an object into a drawer and grasping the corresponding handle. It achieved 4/20 successes (20%). For OCLM integration, π0.5 was fine-tuned on handle-pulling subtasks with ten demonstrations per handle location. The agent coordinated the search through OCLM and invoked these skills, achieving 16/20 successes (80%). We report these additional results here. This experiment demonstrates that OCLM can work with a fine-tuned π0.5 executor on a memory-dependent task. Since the configurations use different fine-tuning demonstrations, the results compare the integrated systems.
3. Storage measurements and growth
[Q1] App. B.4 and Fig. 5 now distinguish measured object-record storage from projected scene-level growth. On 131 matched parameterized assets (48 laptops, 48 faucets, 35 buckets), compact Concept records occupy 85,411 bytes, versus 1,144,579 bytes for one 640×480 JPEG per object plus metadata. JPEGs are code renders at quality 90; storage is measured in serialized bytes.
[Q1] Fig. 5 compares Concept Memory (with Prior Knowledge), Concept Memory (Record Only), Video Memory, and keyframe memory. The prior-knowledge curve adds the measured local template source once: 1,060,961 bytes across 14 categories and 84 Python files in the local source collection. The projection assumes ten objects per scene, one new object every five subsequent scenes, 1 KiB scene metadata per scene, and retention of ten JPEG frames or keyframes per scene. The inset makes the initial template overhead explicit. Video Memory models retained JPEG frames. The projection includes template function source and object/scene records; Transition/Skill histories and retrieval indexes are excluded.