Structured Memory Linking Objects and Concepts for Embodied Manipulation Agents

Long-horizon embodied manipulation requires agents to reuse interaction knowledge while tracking changing objects and scenes. A past successful action may become inapplicable when an object's state changes, and its motion parameters may not transfer to another instance with different geometry. Reusing manipulation experience therefore requires memory to connect object-part geometry with action applicability and execution parameters, while tracking changing object states and scene relations. These requirements motivate a part-based representation with reusable structure, instance-specific geometric parameters, and interfaces to manipulation functions. We propose Object–Concept Linked Memory (OCLM), which uses object code representations to link grounded objects and reusable part templates to scene, transition, and skill memories. At runtime, the agent retrieves relevant objects, scene states, action outcomes, and skills to check applicability and select or parameterize template-grounded or policy-grounded execution. Observations and execution feedback update the associated memories, retaining reusable structural knowledge alongside evolving interaction history. Experiments on memory-dependent manipulation, articulated-object transfer, and real-world tasks show improvements in task success, retrieval accuracy, object re-identification, and cross-object skill transfer over the evaluated unstructured and embedding-based memory baselines.