- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This manuscript proposes ECoMEM, which decouples memory construction from memory utilization within VLA policies. The Writer selects and maintains explicit records (entities, states, events, progress, and temporal sequences) from a shared concept library based on visual and robot-state evidence. The Reader encodes this structured memory into tokens to condition the VLA alongside image and language inputs. ECoMEM achieves the highest average success rate across 16 memory-dependent tasks in the RoboMME benchmark.
Critical strengths
This paper wons a clear problem formulation. ECoMEM decouples memory construction from strategy learning and employs explicit, evidence-linked conceptual records to enhance inspectability and task relevance, thereby avoiding reliance on shortcuts associated with purely implicit memory.
ECoMEM achieves state-of-the-art performance across multiple tasks, demonstrating significant gains in real-world robotic tasks involving repetitive counting and progress tracking. Additionally, ablation studies validate the necessity of structured extraction, task-conditioned selection, and language grounding.
Critical weaknesses
Details regarding the concept library, Writer update rules, and Reader training and deployment are insufficient.
The real-world robot experiments primarily compare against the memoryless π0.5, omitting strong memory-based baselines (such as keyframe, VLM, or scene graph approaches) that utilize the same backbone. The inference latency is not reported.
Questions for authors
-
How many concepts does the current concept library contain? On average, how much human design, annotation, and validation is required to add a new concept?
-
How are the evidence accumulation threshold, event confirmation, and counting rules configured for Writer? Is it robust against noise, occlusion, false positives, and missed detections?
4 comments
Response to Reviewer 5EVQ (2/3)
Q1: How many concepts does the library contain? On average, how much human design, annotation, and validation is required to add a new concept?
The base library contains 20 executable concepts, each a recognizer with an explicit update rule. They cover all 16 RoboMME tasks and emit the 48 predicate, event, and role codes of Table 4. The real-robot tasks add one concept.
| Base library | CupSwitch | ScoopPour | |
|---|---|---|---|
| New concepts | 20 | 0 | 1 (event.scoop) |
| Coding agents | Claude Opus 5, GPT-6-astra | – | GPT-6-astra (Codex) |
| Author effort | ≈42 person-hours, 6 working days | – | supervision and feedback |
| Revisions | 5 library versions | – | 5 recognizer versions |
| Development demonstrations | – | 2 | 6 |
| Validation | – | 6 held-out demos | 33 logic tests; 18 evidence-removal replays (all 0 counts); 15/15 held-out cycles agree with visual review |
For the new concept, design consisted of identifying the missing distinction (a scoop that actually carries beans) and reusing grounding, persistence, counting, and goal tracking unchanged. Annotation required no dense manual labels: development used agent-assisted visual review of the 6 demonstrations with author feedback. Validation is the last row, run after freezing the recognizer. We will add this table to the paper.
Response to Reviewer 5EVQ (3/3)
Q2: How are the evidence accumulation threshold, event confirmation, and counting rules configured? Is the Writer robust against noise, occlusion, false positives, and missed detections?
Configuration. The Writer accumulates signed evidence in log-odds with exponential decay (Eq. 8). One setting serves all predicates; the two slowly varying states use a slower one:
| Parameter | All predicates | activated, lifted |
|---|---|---|
| Confirmation threshold (log-odds) | 4.0 | 1.2 |
| Retention threshold (log-odds) | 1.5 | 0.5 |
| Retraction threshold (log-odds) | −4.0 | −4.0 |
| Decay half-life | 2 s | 60 s |
| Log-odds bound | 8 | 8 |
| Per-frame evidence cap | 0.1 s | 0.1 s |
Robustness rules.
- Occlusion and missed detections. A missed detection is never evidence against a state. A saturated state (log-odds 8) that receives no evidence stays confirmed for ≈4.8 s (2 s × log₂(8/1.5)), then reverts to unknown, never to false. An occluded object keeps its last supported location (Section 4.2, Appendix D.2).
- Noise and false positives. A single frame contributes at most 0.1 s × its channel weight, so confirmation needs several consistent frames;
lifted, for example, needs at least three to reach 4.0. Attachment, holding, and lifting additionally require independent physical evidence such as tracked object motion, not gripper closure alone. - Repeated detections. An event is recorded once, when first confirmed; later matching evidence updates that record (Appendix F.3). A count is the number of distinct recorded events, so repeated detections cannot inflate it.
Empirical evidence. Offline, on 95 held-out videos, the Writer reaches 95–100% on entity, state, and event concepts and 88.2–100% on temporal concepts (Table 2); replacing instruction-specific entity phrases with generic queries drops target-location agreement from 98% to 20% (Appendix G.3), so the Writer follows the evidence rather than a task prior. On the real robot, under camera noise, lighting changes, and occlusion by arm and tool, ECoMEM succeeds in 86.1% of 79 trials. Two recoveries arose without any retry rule: when a scoop spilled, the wrist camera showed an empty scoop, the count did not advance, and the policy scooped again; when a pour was incomplete, no cycle was confirmed and the policy poured again (Figure 7). For the new concept, all 18 evidence-removal replays give zero counts.
We have not run a dedicated noise-injection study that drops or corrupts detections at controlled rates. We will add one to the revision, together with the parameter table above.
Thank you again for the valuable feedback. We are happy to answer further questions during the discussion period.
Thank you for the clarification. I maintain my previous assessment of the paper.
Response to Reviewer 5EVQ (1/3)
We thank the reviewer for the positive assessment and the constructive questions. We answer each below and will incorporate the details into the revision.
These are in the supplementary material: the concept library in Appendix C (complete inventory, Table 4); the Writer in D–E (perception, recognition) and F.1–F.3 (evidence merging, log-odds state estimation, event history and progress); the Reader in F.4–F.6 (record interface, architecture, causal training alignment); training and deployment in A and I.3–I.4. We agree they should be more visible: we will move the update rules (F.2–F.3) and the record interface (Table 5) into the main text, and add the parameter values and construction statistics given under Q1 and Q2 below.
We agree this is a limitation of the real-robot study, whose goal is to test library reuse and extension. Memory-based alternatives are compared on RoboMME with the same backbone: keyframe retrieval with a VLM (MemER) and visual memory banks (SAM2Act+, FrameSamp+Modul). To add the closest analogue of a textual world-state or scene-graph memory, we ran a new comparison in which the same Writer records are rendered as text and appended to the instruction, with the Writer, data, training recipe, and evaluation protocol unchanged:
Text is comparable on PatternLock, where memory is fixed once observed, and worse where memory changes during execution. It also costs ≈21 tokens per record instead of 1 and is 27% slower per inference (96.2 vs. 75.7 ms). We will add this comparison and state the scope of the real-robot study explicitly.
Thank you for pointing this out; we now report both components.