acceptodds
Under review as a conference paper at ICLR 2027

Simple Agentic Memory for Generalist Robot Policies

Abstract

Visual-memory systems commonly retain or compress past observations. Robot control additionally requires interaction-derived state that no individual frame may explicitly represent, such as persistent identity relations, accumulated progress, or ordered procedures. We introduce Simple Agentic Robot Memory (SimpleARM), a training-free memory layer for frozen generalist robot policies. From the task instruction, SimpleARM specifies what to monitor; frozen perceptual tools maintain compact typed state online; structured access retrieves that state only when a proposed subgoal depends on history; and current-view grounding resolves recalled entities before execution. We evaluate SimpleARM on RoboMME, a benchmark of memory-dependent robot manipulation tasks that require history information no longer available in the current observation. Across all 16 tasks and three policy seeds, SimpleARM achieves 67.17% mean success, compared with 44.51% for the strongest non-oracle baseline. Matched ablations show mechanism specificity: removing relation, reference, progress, or route state produces large losses where the affected state is retrieved for control, while largely sparing other tasks. These results support a state-based view of robot memory: effective memory for control is not simply retained visual history, but compact task-relevant state derived from the interaction history.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Agentic Memory, VLA, Robotics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_U9YFOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

SimpleARM adds memory for the current episode to a frozen grounded-subgoal predictor and GroundSG VLA. Before execution, an instruction-only VLM specifies candidate entities, progress variables and readers. Perception tools maintain verified identity/container relations and event counts, as well as demonstrated routes. The structural gate allows a read only for compatible subgoal fields; a remembered entity is grounded in the current view when it needs a coordinate. Across all 16 RoboMME tasks and three policy seeds, success is 67.17%, compared with 44.51% for FrameSamp. Matched component removals and failure analyses examine which states matter, and when the policy actually accesses them.

Critical strengths

I found the distinction between maintaining state and using it for control useful. In Figure 3 and Tables 8 to 11, removing route state costs 68 points on average across the two route tasks. Elsewhere, the system can maintain the same state with little effect from its removal because it is never retrieved. The identity-tool experiment makes a similar comparison between retrieved bindings and active but unused ones. The tools and state types are familiar, but these experiments say something concrete about how to connect memory to a robot policy.

The episode-matched FrameSamp comparison is also fairly convincing: 254 episodes succeed only with SimpleARM, against 81 that succeed only with FrameSamp. I appreciate the discussion of accepted but incorrect bindings, including the distinction between internal verification and ground truth, and that StopCube's 0% result remains in the evaluation. I lean weak accept for the system and mechanism analysis, though I would narrow the claim about generality.

Critical weaknesses

The dependence on RoboMME's interface still bothers me. Appendix A.1 describes lexical tests on the subgoal text, and A.3 uses calibrated colour predicates and event parsers. In Figure 12, the parser reads pure-red demonstration cues and maps directions through a fixed camera convention. Instruction rewording costs 17.8 points in Table 12. This is a separate configuration, so it needs some caution, but the size of the loss is concerning. The evidence is stronger for this particular RoboMME implementation than for carrying the instruction-conditioned memory layer to a new task family. I would find a small held-out test with unchanged parsers more useful here than simply another large benchmark.

It is also difficult to assign the full 22.66-point gain to memory, since the system adds perception and subgoal correction. As the authors note, VideoUnmask and BinFill are not very sensitive to the tested removals. Neither the PickHighlight mark reader nor current-view grounding has a separate ablation. The matched removals support several specific mechanisms, but leave part of the system-level gain unexplained. The final component removals use only policy seed 11; the separate identity-tool and no-flow comparisons omit SAM, and the matched FrameSamp runs use different hosts. The paper is careful to document these distinctions. They make me put much more weight on the large effects localized to the relevant tasks than on the small differences.

Questions for authors
  1. Can you separate failures in planner output from failures in the structural reader under instruction rewording? I would like to see a matched test with the final configuration fixed.

  2. How much improvement remains with the same perception and grounding tools but persistent memory disabled? A separate mark-reader removal would also help identify which gains come from history.

  3. For a held-out task using the same objects in a new combination of historical relations, what would need to change in the schema, event predicates or access rules?

2 comments
AuthorsOpenReview ↗

Response to Reviewer U9YF

We thank the reviewer for highlighting the distinction between maintaining state and using it for control, and for recognizing the value of the episode-matched FrameSamp comparison and the failure analysis. We address the questions about interface dependence, attribution, and transfer below. We look forward to discussing these findings with the reviewer.

W1 / Q1: Interface dependence and rewording

W1: The dependence on RoboMME's interface still bothers me. Appendix A.1 describes lexical tests on the subgoal text, and A.3 uses calibrated colour predicates and event parsers. In Figure 12, the parser reads pure-red demonstration cues and maps directions through a fixed camera convention. Instruction rewording costs 17.8 points in Table 12. This is a separate configuration, so it needs some caution, but the size of the loss is concerning. The evidence is stronger for this particular RoboMME implementation than for carrying the instruction-conditioned memory layer to a new task family. I would find a small held-out test with unchanged parsers more useful here than simply another large benchmark. Q1: Can you separate failures in planner output from failures in the structural reader under instruction rewording? I would like to see a matched test with the final configuration fixed.

Memory-side paraphrase test. Across 800 matched episodes, success drops from 523/800 (65.38%) with canonical instructions to 488/800 (61.00%) with rewording (−4.38 points, 92 canonical-only and 57 rewrite-only successes, exact McNemar p=0.0052p=0.0052).

Each of 16 tasks uses two rewrite templates (25 episodes each). We checked the manifest for preserved objects, counts, order and event references. Runtime assertions check canonical environment and composer inputs. Only specification and repetition queries receive the rewrite. Demonstration parsers retain canonical text. The retrieval compiler votes over responses to three line wrappings of one prompt.

Structural access diagnostic. We selected ten logged coordinate subgoals spanning seven historical references and three current-view actions. With coordinates and memory held fixed, canonical and syntax-compatible wording match 10/10 intended decisions. Reordered wording misses all seven historical references and correctly skips retrieval in the three current-view cases (3/10).

Relation to Table 12. The earlier pre-unified system pooled 416 matched pairs from four handwritten templates per episode on 8 tasks × 13 episodes at seed 7. Its runtime substitution rebound colours and recognised counts and could fall back to canonical text. The current test passes literal, episode-specific rewrites to the final unified system over all 16 tasks. The two effects (−17.8 and −4.38 points) come from different implementations, rewrites and episode sets.

W2 / Q2: Same-system history removal and remaining attribution

W2: It is also difficult to assign the full 22.66-point gain to memory, since the system adds perception and subgoal correction. As the authors note, VideoUnmask and BinFill are not very sensitive to the tested removals. Neither the PickHighlight mark reader nor current-view grounding has a separate ablation. The matched removals support several specific mechanisms, but leave part of the system-level gain unexplained. The final component removals use only policy seed 11; the separate identity-tool and no-flow comparisons omit SAM, and the matched FrameSamp runs use different hosts. The paper is careful to document these distinctions. They make me put much more weight on the large effects localized to the relevant tasks than on the small differences. Q2: How much improvement remains with the same perception and grounding tools but persistent memory disabled? A separate mark-reader removal would also help identify which gains come from history.

Removing SimpleARM's persistent task state and its subgoal corrections reduces success from 523/800 to 276/800 (−30.88 points), with 282 full-only and 35 removal-only successes. The frozen composer and VLA, native demonstration and subgoal context, current-view detector refinement and segment coordinate cache remain. Historical identity verification has no stored anchors in this control.

Q3: New compositions with unchanged parsers

Q3: For a held-out task using the same objects in a new combination of historical relations, what would need to change in the schema, event predicates or access rules?

For a new combination of supported relations, the instruction-derived specification would need to select the relevant entities and retrieval operations. A new relation would require a schema field and a predicate for observing it. A new subgoal format would require a compatible access rule.

AuthorsOpenReview ↗

Kind Follow-up on Our Response

Dear Reviewer U9YF,

Thank you again for your valuable effort and thoughtful review. Following your feedback, we added a full episode matched history-removal experiment to answer W2&Q2 and an instruction rewording test in response to W1&Q1. As the discussion period is coming to an end, we would appreciate hearing whether these results and clarifications address your concerns about attribution and interface dependence. Please let us know if further explanation would help.

Best,

The Authors

Reviewer_sScaOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that robot memory should not be framed as selecting or compressing past observations, but as maintaining policy-sufficient state induced by an action–observation history: persistent identity relations, accumulated progress, and ordered procedures. On this view, the relevant information often lives in no single frame, so memory budget is not the binding constraint. From this framing the authors build SimpleARM, a training-free memory layer wrapped around a frozen hierarchical robot stack.

Critical strengths

1."Robot memory is a problem of maintaining the task-relevant state induced by an action–observation history" is a clear, defensible position, and prior RoboMME evidence makes the motivation concrete rather than rhetorical. The paper is arguing against a real, currently dominant default. 2.Exactly ten LM calls per episode, all before the first action, plus 2.02 s/episode of perception tools, itemised by component. Nonparametric, episode-scoped memory with M0 reset. There is a real path to deployment here. 3.16 tasks, 3 seeds, 50 episodes per task per seed are sufficient. And, the authors retraining and evaluating their own RecentN+ModuL recent-frame baselines at matched protocol Reporting both aggregate and paired-episode views is good practice.

Critical weaknesses

1.The headline comparison is not memory-vs-no-memory within one system. SimpleARM does not merely add a memory layer: it adds an entire frozen perception stack and control interface that the comparison baselines do not possess. 2.RecentN+ModuL variants are each trained once and evaluated at three policy seeds, so the three rounds are not independent training runs and cannot estimate training variance. 3.There is no code/model/prompt availability statement anywhere I could find. 4.Table 1's "required state" labels are explicitly not mutually exclusive and are derived from task semantics; several of the largest wins (VideoUnmask 96.0 vs 32.7) have no corresponding ablation, so the biggest contributors to the headline are the least attributed.

Questions for authors
  1. Please re-run no route state and agent-decided access at 2–3 policy seeds, so the central specificity claim does not rest on a single seed. 2.How much of the Permanence gain depends on colour-based identity? A short probe with colour-neutral or colour-relabelled objects would tell the reader whether the maintained "identity" is semantic or chromatic.
2 comments
AuthorsOpenReview ↗

Response to Reviewer sSca

We thank the reviewer for recognizing the motivation for maintaining task-relevant state, the modest inference overhead, and the value of the RecentN baselines and paired comparisons. We address the questions about attribution, reproducibility, and evaluation below. We look forward to discussing the results with the reviewer.

W1 / W4: Added perception and attribution of task-level gains

W1: The headline comparison is not memory-vs-no-memory within one system. SimpleARM does not merely add a memory layer: it adds an entire frozen perception stack and control interface that the comparison baselines do not possess. W4: Table 1's "required state" labels are explicitly not mutually exclusive and are derived from task semantics; several of the largest wins (VideoUnmask 96.0 vs 32.7) have no corresponding ablation, so the biggest contributors to the headline are the least attributed.

The +22.66-point comparison against FrameSamp is a complete-system comparison. To isolate added historical state, we retained the frozen hierarchy, its demonstration inputs and current-view refinement and disabled the additional task-history records/corrections. Success changes from 523/800 (65.38%) to 276/800 (34.50%), with 282 full-only and 35 history-disabled-only successes. The within-system difference is 30.88 points.

This effect is concentrated where maintained history is useful: PatternLock loses 82 points, SwingXtimes 64, ButtonUnmask 56, VideoUnmaskSwap 52, and the reference tasks 36–46. VideoUnmask ablation was also already included in the original demonstration-binding removal (48/50→44/50) in main paper Figure 3.

W2: Evaluation seeds versus training runs

W2: RecentN+ModuL variants are each trained once and evaluated at three policy seeds, so the three rounds are not independent training runs and cannot estimate training variance.

RoboMME evaluates averages FrameSamp+ModuL variants using the last three checkpoints across three evaluation seeds. The public FrameSamp release provides checkpoint 79999, rather than the full checkpoint set used for that average. Our local re-evaluation of this released FrameSamp+ModuL checkpoint at step 79999 achieves 46.12% (369/800), close to the published 44.51% in RoboMME paper, supporting consistency of the evaluation setup and released checkpoint. Our reported standard deviation measures evaluation variability, not training variability.

W3: Code, models and prompts

W3: There is no code/model/prompt availability statement anywhere I could find

Model identities are specified in §3.4 and Appendix A.3, including RoboMME’s released GroundSG checkpoint 79999 and Qwen3-VL-4B-Instruct adapter checkpoint 1200. Appendix A.1 clarifies that memory planning reuses the same base VLM with its adapter disabled. We have expanded the reproducibility statement and added verbatim prompts, decoding settings. Model weights and simulator assets come from the upstream releases.

Q1: Multi-seed ablations

Q1: Please re-run no route state and agent-decided access at 2–3 policy seeds, so the central specificity claim does not rest on a single seed.

Both ablations use frozen weights and 800 matched episode pairs per evaluation seed. The table reports the change in overall success across all 16 tasks (ablated minus full, in percentage points). The SD measures variability across evaluation seeds for the frozen weights.

ComparisonSeed 7Seed 11Seed 23Mean ± SD
No route state−7.38−11.50−8.25−9.04 ± 2.17
Agent-decided access−10.88−11.88−9.50−10.75 ± 1.19

The appendix gives the task-level effects for both ablations.

Q2: Colour dependence

Q2: How much of the Permanence gain depends on colour-based identity? A short probe with colour-neutral or colour-relabelled objects would tell the reader whether the maintained "identity" is semantic or chromatic.

Colour is an explicit task reference in RoboMME, e.g., “the container hiding the green cube.” It identifies the cube before occlusion; the subsequent decision requires its remembered relation to a container, whose appearance no longer reveals the hidden colour. With current-view refinement retained, removing added history reduces ButtonUnmask 80% to 24% and ButtonUnmaskSwap 52% to 16%, supporting a contribution beyond current-view colour grounding. Our identity initialization nevertheless uses calibrated colour predicates. These results establish memory utility within the benchmark's appearance vocabulary, not robustness to unseen palettes or colour-neutral identity cues, which remain untested.

AuthorsOpenReview ↗

Kind Follow-up on Our Response

Dear Reviewer sSca

Thank you again for recognizing the formulation, inference efficiency, and evaluation. We completed the requested route-state and access ablations across three evaluation runs, each with 800 matched episode pairs (Q1). We also added history memory removal (W1&W4), clarified the VideoUnmask analysis(W4), and clarified the distinction between evaluation runs and independent training runs (W2). We added more reproducibility and clarification details in our updated manuscript in response to W3. We would appreciate your assessment of whether these additions address your questions. We would be glad to clarify any remaining concerns.

Best,

The Authors

Reviewer_Zi51OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SimpleARM, a training-free memory layer around a frozen grounded-subgoal predictor and VLA policy. An instruction-only stage proposes what to monitor; perception tools and event predicates maintain entity/relation, progress, and ordered-route state; structural access supplies compatible historical information to a proposed subgoal, with current-view grounding when needed. On all 16 RoboMME tasks over three policy seeds, the system obtains 67.17% success versus 44.51% for the strongest reported non-oracle baseline. Matched component removals identify large, task-specific contributions from several maintained variables and from constrained access. The contribution is an effective robot-memory system and its empirical analysis; the approximate policy-sufficiency equation is a design objective rather than a demonstrated general sufficiency result.

Critical strengths
  1. Substantial gains across the full benchmark. Evaluation covers all 16 tasks over three seeds, with 542/800 versus 369/800 successes in matched testing. Gains are strongest on Permanence and Reference, while Counting weaknesses are also reported.
  2. Informative matched ablations. Component removals cause large, task-specific losses. Replacing structural access with VLM-based decisions reduces overall success by 11.88 points, supporting the value of both maintained states and constrained access.
  3. Transparent empirical analysis. The paper distinguishes internal verification from correctness, diagnostic associations from causal evidence, and supplementary controls from final-method ablations. Visual traces clarify how memory affects actions.
Critical weaknesses

Limited evidence of generality. The implementation relies on RoboMME-specific phrase tests, parsing rules, and appearance vocabulary. Instruction rewording substantially reduces performance in a different configuration; a final-configuration paraphrase test is needed to assess robustness.

Incomplete attribution of gains. The ablations establish component usefulness but do not isolate instruction-driven planning from fixed monitoring with the same tools. A matched control disabling all historical state is also missing, so the full 22.66-point gain cannot be attributed to the memory abstraction alone.

Missing implementation details. This paper lacks complete prompts, reader schemas, access tests, and event/demo parsing rules. The configuration differences behind Table 12 also need clarification to support reproduction.

Questions for authors
  1. With the final tool and access stack held fixed, what is the gain from instruction-conditioned monitoring over a fixed or keyword-based monitoring policy? Please clarify how the Table 12 control differs from the final system; a targeted matched comparison would be particularly informative.
  2. Does the final system retain its gains under semantics-preserving instruction rewording? Can you separate sensitivity in the instruction-only memory planner from sensitivity in the frozen composer and structural phrase matching?
  3. Can you provide the complete prompts, line widths, schemas, access predicates, and demonstration/event parser specifications, together with the development data or procedure used to set these rules and thresholds?
  4. Is there a matched control that disables all cross-time state while retaining the same frozen composer/VLA and current-view tools? If unavailable, how should the gains on VideoUnmask and BinFill, which are relatively insensitive to the tested removals, be interpreted?
3 comments
AuthorsOpenReview ↗

Response to Reviewer Zi51 [1/2]

We thank the reviewer for recognizing the strengths of our work. We appreciate the reviewer’s suggestions and address the remaining concerns and questions below. We have completed additional matched controls and clarified the scope of the formulation and its RoboMME instantiation. We look forward to further discussion.

W1 / Q2: Generality and instruction rewording

W1: The implementation relies on RoboMME-specific phrase tests, parsing rules, and appearance vocabulary. Instruction rewording substantially reduces performance in a different configuration; a final-configuration paraphrase test is needed to assess robustness. Q2: Does the final system retain its gains under semantics-preserving instruction rewording? Can you separate sensitivity in the instruction-only memory planner from sensitivity in the frozen composer and structural phrase matching?

Memory-side paraphrase test. Across 800 matched episodes, success drops from 523/800 (65.38%) with canonical instructions to 488/800 (61.00%) with rewording (−4.38 points, 92 canonical-only and 57 rewrite-only successes, exact McNemar p=0.0052p=0.0052).

Each of 16 tasks uses two rewrite templates (25 episodes each). We checked the manifest for preserved objects, counts, order and event references. Runtime assertions check canonical environment and composer inputs. Only specification and repetition queries receive the rewrite. Demonstration parsers retain canonical text. The retrieval compiler votes over responses to three line wrappings of one prompt.

Structural access diagnostic. We selected ten logged coordinate subgoals spanning seven historical references and three current-view actions. With coordinates and memory held fixed, canonical and syntax-compatible wording match 10/10 intended decisions. Reordered wording misses all seven historical references and correctly skips retrieval in the three current-view cases (3/10).

Relation to Table 12. The earlier pre-unified system pooled 416 matched pairs from four handwritten templates per episode on 8 tasks × 13 episodes at seed 7. Its runtime substitution rebound colours and recognised counts and could fall back to canonical text. The current test passes literal, episode-specific rewrites to the final unified system over all 16 tasks. The two effects (−17.8 and −4.38 points) come from different implementations, rewrites and episode sets.

W2 / Q4: Attribution to historical state

W2: The ablations establish component usefulness but do not isolate instruction-driven planning from fixed monitoring with the same tools. A matched control disabling all historical state is also missing, so the full 22.66-point gain cannot be attributed to the memory abstraction alone. Q4: Is there a matched control that disables all cross-time state while retaining the same frozen composer/VLA and current-view tools? If unavailable, how should the gains on VideoUnmask and BinFill, which are relatively insensitive to the tested removals, be interpreted?

We added the corresponding control over 800 matched episodes, bypassing SimpleARM's entity, reference, mark, progress and route stores and their subgoal corrections, including historical phase corrections. It retains the frozen composer and VLA, native demonstration and subgoal context, grasp-gated detector refinement and the segment coordinate cache. Success falls from 523/800 to 276/800 (−30.88 points), with 282 full-only and 35 removal-only successes (exact McNemar p=4.1×10−49p=4.1\times10^{-49}).

W3 / Q3: Reproducibility and development procedure

W3: This paper lacks complete prompts, reader schemas, access tests, and event/demo parsing rules. The configuration differences behind Table 12 also need clarification to support reproduction. Q3: Can you provide the complete prompts, line widths, schemas, access predicates, and demonstration/event parser specifications, together with the development data or procedure used to set these rules and thresholds?

We developed the access rules from RoboMME subgoal syntax and the event parsers from task-specific visual cues in saved rollouts. The colour verifier uses RGB references calibrated offline from oracle-labelled image patches. The retained records do not identify the calibration episodes, their overlap with evaluation scenes, or a complete threshold selection procedure. The appendix specifies schemas, voting and fallback, access predicates, parser thresholds and the Table 12 protocol.

AuthorsOpenReview ↗

Response to Reviewer Zi51 [2/2]

Q1: VLM monitoring versus fixed monitoring

Q1: With the final tool and access stack held fixed, what is the gain from instruction-conditioned monitoring over a fixed or keyword-based monitoring policy? Please clarify how the Table 12 control differs from the final system; a targeted matched comparison would be particularly informative.

The VLM infers whether to record, where the evidence appears, and which entities to monitor. The fixed alternative always records, selects live/video evidence using explicit video/demonstration words, and matches named red/green/blue cubes. Retrieval and repetition specifications remain unchanged. On 200 matched Permanence episodes, fixed monitoring reaches 78.0%, versus 75.5% for VLM monitoring.

The fixed rule is restricted to predefined words and appearance categories, interpreting references outside that vocabulary requires rule extensions. The VLM instead supplies a language-based specification interface.

AuthorsOpenReview ↗

Kind Follow-up on Our Response

Dear Reviewer Zi51,

Thank you again for your valuable time and careful review. Following your feedback, we added an full episode matched history-removal experiment to answer W2&Q4 and an instruction rewording test to answer W1&Q2, and added more reproducibility and clarification details in our updated manuscript to answer Q1 and W3&Q3. We would appreciate hearing whether these additions address your main concerns. As the discussion period is coming to an end, please kindly let us know if you have any further comments, and we would be glad to address them.

Best, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We sincerely thank the reviewers and AC/SAC for their time and constructive feedback. We appreciate the recognition of our task-state formulation and its concrete motivation from RoboMME (sSca), substantial benchmark gains and comprehensive evaluation across all 16 tasks, including matched comparisons and recent-frame baselines (Zi51, sSca, U9YF), and informative component ablations showing when maintained state and structured access support control (Zi51, U9YF). Reviewers also highlighted the low inference overhead and episode-scoped, nonparametric design (sSca), alongside transparent empirical analysis, diagnostic associations, and reporting challenging tasks and visual traces (Zi51, U9YF).

To address the reviewers’ concerns, we added experiments and revised the manuscript as follows.

  • History removal and attribution. We added an 800-episode matched experiment disabling SimpleARM’s added historical memory state, keeping the rest of the added perception stack. We also clarified the VideoUnmask results and task-level attribution (Zi51 W2/Q4, sSca W1/W4, U9YF W2/Q2).
  • Instruction rewording. We tested the final memory layer with reworded instructions while keeping the composer’s input canonical, and separately examined structural-access sensitivity (Zi51 W1/Q2, U9YF W1/Q1).
  • Repeated component ablations. We completed route-state removal and access ablations across three evaluation runs, with 800 matched episode pairs per run (sSca Q1).
  • Reproducibility and clarification. We expanded the documentation of models, prompts, decoding settings, monitoring, and access rules, and clarified evaluation runs (Zi51 W3/Q3, sSca W3).

These additions are incorporated into the updated manuscript and individual responses.

We welcome further discussion and greatly appreciate your input. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.