- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes GAMMA, a grounded two-agent memory system for VLA memory tasks. The key idea is to decompose the problem into episodic memory, grounded symbolic subgoal, and action, while keeping the low-level VLA executor fixed. GAMMA uses SAM-3 for spatial grounding, a VLM writer to convert each observation into grounded textual events, an append-only memory bank, and a VLM reasoner to generate the next grounded subgoal. A propose-verify harness checks agent claims and can admit, defer, correct, or reject them based on visual evidence. GAMMA reaches 0.660 average success against the oracle’s 0.841.
Critical strengths
The large gap between direct end-to-end execution and privileged grounded-subgoal execution provides strong motivation for isolating the memory-to-subgoal stage, and GAMMA improves substantially over prior non-privileged baselines. Additionally, the paper explicitly identifies where the VLM fails, including timing, grounding, and motion-direction errors, and connects these failures to individual verifier components.
Critical weaknesses
Several results rely substantially on task-specific hand engineering. Hand-built readers replace the VLM's direction judgment on PatternLock and RouteStick, while StopCube uses an arrival-period predictor that the authors explicitly describe as anticipating rather than verifying and as lying outside the evidence contract. Removing the REJECT/extractor-owned-claim mechanism drops PatternLock from 0.989 to 0.033, RouteStick from 0.567 to 0.067, and StopCube from 0.355 to 0.135. Moreover, this ablation reduces the average from 0.660 to 0.551, accounting for most of the gap between the full system and no harness. This reliance on benchmark-specific components weakens the paper's claim that a general propose-verify contract carries the memory competence. Excluding these three tasks, the harness improves over no harness by only +0.054 at 9B rather than +0.148.
Questions for authors
-
On RoboMemArena, the harness without the writer achieves 38.7/58.6, essentially identical to the full system's 38.8/59.0. The paper further explains that, because each task has a fixed subtask sequence, the predicates can raise the completion claims themselves. This suggests that the learned writer contributes very little to the reported RoboMemArena performance, while the hand-specified transition predicates may account for much of the transfer. What evidence does this experiment provide for cross-benchmark transfer of GAMMA's learned memory/reasoning architecture, rather than transfer of the verification and transition design?
-
GAMMA underperforms existing baselines on several tasks despite having access to the additional detector, symbolic bank, and verification components. For example, SwingXtimes is 68.9 versus 92.0 for FrameSamp+Modul, and MoveCube is 62.2 versus 82.7 for MemER; GAMMA also trails at least one baseline on BinFill, StopCube, and VideoUnmask. Could the authors provide a failure analysis for these cases? In particular, are the failures caused by writer omissions, verification errors, reasoner errors, or the fixed executor? Such an analysis would help establish whether the proposed design principles generalize beyond the tasks on which the harness provides its largest gains.
8 comments
Hand engineering and the size of the generic effect (weakness). Without RouteStick, PatternLock and StopCube the harness is worth +2.1 in the regenerated rows (68.9 against 66.8; +12.6 on all 16 tasks), and the REJECT ablation removes extractor-owned claims rather than generic verification. The revision reports this decomposition as such: Tables 2 and 3 carry a 13-task column, the claim that the contracts "carry most of the memory competence" is withdrawn, the grounded text memory is credited with the identity tasks, the generic verdicts with the tasks that have a checkable percept (the press-verify DEFER is the whole +2.1), and the direction readers and the arrival predictor are presented as what they are: the contract's way of letting an evidence extractor own a claim type the VLM measurably cannot read. The direction readers are task-specific code, and we state how they arose and what they address. In the on-policy failure analysis of the benchmark the writer's direction words for RouteStick and PatternLock (the stroke's side and rotation sense; its eight-way heading) came out at chance (Appendix B), and every prior VLM pipeline is at chance on these two tasks. Reading direction is a documented deficit of current VLMs, measured independently of this benchmark. Kamath et al. (EMNLP 2023, "What's up with vision-language models?") evaluate 18 models on left/right and above/below relations and find all of them poor (BLIP fine-tuned on VQAv2 at 56% against 99% for humans); the DORI orientation benchmark (Tasnim, Nichols et al., arXiv 2505.21649) finds 26 models near random on object orientation, the best at 64.2% on categorical and 42.9% on metric judgements; RotBench (Niu et al., arXiv 2508.13968) reports that GPT-5, o3 and Gemini-2.5-Pro do not reliably identify image rotation, with 90° and 270° (clockwise against counter-clockwise) the hardest cases; Ori-Bench (Wang et al., Orient Anything, arXiv 2412.18605) puts GPT-4o and Gemini-1.5-Pro near 60% on basic direction recognition. The benchmark put a clean instance of this general deficit in front of us, and the contract's response to it is the general one: a claim type the proposer cannot read is owned by an evidence extractor, and the writer's word is rejected in its favour (Table 8 of the revised PDF lists the guards). The two readers are small, apply to any memory claim that depends on a direction percept, and recover the oracle on the two tasks (PL 98.9, RS 56.7): a simple and effective remedy for a failure class that reaches past this benchmark. The arrival predictor of StopCube is the same mechanism applied to a timing percept (counted arrivals and a predicted period). Table 3 is regenerated with the released code and checkpoints under the Table 2 protocol (full system 67.9, w/o CORRECT 67.7, w/o DEFER 65.8, w/o REJECT 57.8, no harness 55.3; integer counts per task and seed in Appendix C); the full pipeline is in the supplementary material and all trained weights are shared through an anonymous link (https://huggingface.co/123123xxxx/gamma-checkpoints).
RoboMemArena (Q1). The two rows coincide because of the benchmark's structure. Every RoboMemArena task comes with a fixed subtask sequence, the benchmark's own: the released demonstrations are segmented into these subtasks and the authors' executor is prompted with the subtask text. Task 1 is an example. Its plan is "pick up the cookies, place the cookies into the basket, pick up the tomato sauce, place the tomato sauce into the basket", the same in every episode; nothing in the scene or in the history changes which subtask comes next, so the only decision left to a memory system is when the executor is handed the next subtask. In our writer-run row the writer's claim for the first subtask is logged as "completed: pick up the cookies" beside the detector reading of that tick (gripper closed on the cookies at width 0.033, end-effector lifted to z 0.59), and the proprioceptive pick predicate fires on the same evidence; on a fixed plan the writer reports the predicate's evidence back, so the hand-off happens at the same tick either way. The predicate-only configuration (38.7 / 58.6) matches the writer-run configuration (38.8 / 59.0) for this reason, and the revision states that what transfers is the verified hand-off control; the learned memory and reasoning have no decision to make on a fixed plan. On RoboMemBench the plan is not fixed: on VUS the instance to act on is fixed by what was seen before an occlusion, on SwingXtimes by the number of swings already made, so the next subgoal is a decision taken over the memory bank by the reasoner and verified by the same contract. The pipeline is a hybrid for this reason: one contract (proposer, verifier, hand-off) whose components engage according to the task's structure, the predicates alone when the plan is fixed, the writer and the reasoner when the next subgoal depends on the history, an evidence extractor when the claim is of a kind the VLM cannot read. The RoboMemArena port is the fixed-plan case of that contract and took under one engineering day (two geometry constants, detector prompts, five proprioceptive predicates audited on the released demonstrations); the writer was trained on a corpus generated from the released demonstrations by the released pipeline and runs in the reported row. Section 5.5 is retitled, the abstract wording is changed, and both rows are reported under the one protocol of Table 16 (Table 20 in the revised PDF; three serving seeds × 50 episodes per task), which the revised caption states in full; a reasoner and writer data for tasks whose plan is not fixed are the next step with more resources.
We thank the reviewer for separating the harness's generic verdicts from its task-specific extractors; the revision reports both. A revised PDF and an updated code archive are uploaded with this response; the changes in each are listed in the last part.
In brief, per point.