- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SimpleARM adds memory for the current episode to a frozen grounded-subgoal predictor and GroundSG VLA. Before execution, an instruction-only VLM specifies candidate entities, progress variables and readers. Perception tools maintain verified identity/container relations and event counts, as well as demonstrated routes. The structural gate allows a read only for compatible subgoal fields; a remembered entity is grounded in the current view when it needs a coordinate. Across all 16 RoboMME tasks and three policy seeds, success is 67.17%, compared with 44.51% for FrameSamp. Matched component removals and failure analyses examine which states matter, and when the policy actually accesses them.
Critical strengths
I found the distinction between maintaining state and using it for control useful. In Figure 3 and Tables 8 to 11, removing route state costs 68 points on average across the two route tasks. Elsewhere, the system can maintain the same state with little effect from its removal because it is never retrieved. The identity-tool experiment makes a similar comparison between retrieved bindings and active but unused ones. The tools and state types are familiar, but these experiments say something concrete about how to connect memory to a robot policy.
The episode-matched FrameSamp comparison is also fairly convincing: 254 episodes succeed only with SimpleARM, against 81 that succeed only with FrameSamp. I appreciate the discussion of accepted but incorrect bindings, including the distinction between internal verification and ground truth, and that StopCube's 0% result remains in the evaluation. I lean weak accept for the system and mechanism analysis, though I would narrow the claim about generality.
Critical weaknesses
The dependence on RoboMME's interface still bothers me. Appendix A.1 describes lexical tests on the subgoal text, and A.3 uses calibrated colour predicates and event parsers. In Figure 12, the parser reads pure-red demonstration cues and maps directions through a fixed camera convention. Instruction rewording costs 17.8 points in Table 12. This is a separate configuration, so it needs some caution, but the size of the loss is concerning. The evidence is stronger for this particular RoboMME implementation than for carrying the instruction-conditioned memory layer to a new task family. I would find a small held-out test with unchanged parsers more useful here than simply another large benchmark.
It is also difficult to assign the full 22.66-point gain to memory, since the system adds perception and subgoal correction. As the authors note, VideoUnmask and BinFill are not very sensitive to the tested removals. Neither the PickHighlight mark reader nor current-view grounding has a separate ablation. The matched removals support several specific mechanisms, but leave part of the system-level gain unexplained. The final component removals use only policy seed 11; the separate identity-tool and no-flow comparisons omit SAM, and the matched FrameSamp runs use different hosts. The paper is careful to document these distinctions. They make me put much more weight on the large effects localized to the relevant tasks than on the small differences.
Questions for authors
-
Can you separate failures in planner output from failures in the structural reader under instruction rewording? I would like to see a matched test with the final configuration fixed.
-
How much improvement remains with the same perception and grounding tools but persistent memory disabled? A separate mark-reader removal would also help identify which gains come from history.
-
For a held-out task using the same objects in a new combination of historical relations, what would need to change in the schema, event predicates or access rules?
2 comments
Kind Follow-up on Our Response
Dear Reviewer U9YF,
Thank you again for your valuable effort and thoughtful review. Following your feedback, we added a full episode matched history-removal experiment to answer W2&Q2 and an instruction rewording test in response to W1&Q1. As the discussion period is coming to an end, we would appreciate hearing whether these results and clarifications address your concerns about attribution and interface dependence. Please let us know if further explanation would help.
Best,
The Authors
Response to Reviewer U9YF
We thank the reviewer for highlighting the distinction between maintaining state and using it for control, and for recognizing the value of the episode-matched FrameSamp comparison and the failure analysis. We address the questions about interface dependence, attribution, and transfer below. We look forward to discussing these findings with the reviewer.
W1 / Q1: Interface dependence and rewording
Memory-side paraphrase test. Across 800 matched episodes, success drops from 523/800 (65.38%) with canonical instructions to 488/800 (61.00%) with rewording (−4.38 points, 92 canonical-only and 57 rewrite-only successes, exact McNemar p=0.0052).
Each of 16 tasks uses two rewrite templates (25 episodes each). We checked the manifest for preserved objects, counts, order and event references. Runtime assertions check canonical environment and composer inputs. Only specification and repetition queries receive the rewrite. Demonstration parsers retain canonical text. The retrieval compiler votes over responses to three line wrappings of one prompt.
Structural access diagnostic. We selected ten logged coordinate subgoals spanning seven historical references and three current-view actions. With coordinates and memory held fixed, canonical and syntax-compatible wording match 10/10 intended decisions. Reordered wording misses all seven historical references and correctly skips retrieval in the three current-view cases (3/10).
Relation to Table 12. The earlier pre-unified system pooled 416 matched pairs from four handwritten templates per episode on 8 tasks × 13 episodes at seed 7. Its runtime substitution rebound colours and recognised counts and could fall back to canonical text. The current test passes literal, episode-specific rewrites to the final unified system over all 16 tasks. The two effects (−17.8 and −4.38 points) come from different implementations, rewrites and episode sets.
W2 / Q2: Same-system history removal and remaining attribution
Removing SimpleARM's persistent task state and its subgoal corrections reduces success from 523/800 to 276/800 (−30.88 points), with 282 full-only and 35 removal-only successes. The frozen composer and VLA, native demonstration and subgoal context, current-view detector refinement and segment coordinate cache remain. Historical identity verification has no stored anchors in this control.
Q3: New compositions with unchanged parsers
For a new combination of supported relations, the instruction-derived specification would need to select the relevant entities and retrieval operations. A new relation would require a schema field and a predicate for observing it. A new subgoal format would require a compatible access rule.