acceptodds
Under review as a conference paper at ICLR 2027

GAMMA: GROUNDED MULTI-AGENT MEMORY WITH MECHANICAL AUDIT FOR MEMORY-INTENSIVE MANIPULATION

Abstract

Pretrained vision–language–action (VLA) policies fail on memory tasks— behaviour that depends on events observed long before the current frame— because their training data is overwhelmingly Markov and mapping episodic memory directly to action chunks is hard. Yet the same policy, finetuned once to follow a privileged oracle’s grounded-subgoal text, nearly solves these tasks: the difficulty lies entirely in mapping memory to a grounded symbolic subgoal. Existing work entrusts this stage to a single VLM that selects keyframes and reasons over raw frames, and falls far short of the oracle—key identities and dynamics are buried in noisy pixels, the reasoning spans hundreds of frames in an unbounded context, and hallucination corrupts the memory itself. We instead present GAMMA (Grounded multi-Agent Memory with Mechanical Audit), which builds the stage as a collaborating two-agent memory system: a detector tool (SAM-3) for spatial grounding, a VLM writer that distils each replan window into one grounded text line, an append-only text memory bank, and a VLM reasoner that reads the bank and emits the next grounded subgoal at the oracle’s cadence—wrapped in a propose–verify harness that admits, defers, corrects, or rejects every agent claim against mechanical visual evidence, at training-corpus construction and at deployment alike. On a sixteen-task memory-manipulation suite GAMMA recovers a large fraction of the oracle ceiling with the policy held fixed, reaching the ceiling on several tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA, Memory, Multi-Agent System

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_z7sUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GAMMA, a grounded two-agent memory system for VLA memory tasks. The key idea is to decompose the problem into episodic memory, grounded symbolic subgoal, and action, while keeping the low-level VLA executor fixed. GAMMA uses SAM-3 for spatial grounding, a VLM writer to convert each observation into grounded textual events, an append-only memory bank, and a VLM reasoner to generate the next grounded subgoal. A propose-verify harness checks agent claims and can admit, defer, correct, or reject them based on visual evidence. GAMMA reaches 0.660 average success against the oracle’s 0.841.

Critical strengths

The large gap between direct end-to-end execution and privileged grounded-subgoal execution provides strong motivation for isolating the memory-to-subgoal stage, and GAMMA improves substantially over prior non-privileged baselines. Additionally, the paper explicitly identifies where the VLM fails, including timing, grounding, and motion-direction errors, and connects these failures to individual verifier components.

Critical weaknesses

Several results rely substantially on task-specific hand engineering. Hand-built readers replace the VLM's direction judgment on PatternLock and RouteStick, while StopCube uses an arrival-period predictor that the authors explicitly describe as anticipating rather than verifying and as lying outside the evidence contract. Removing the REJECT/extractor-owned-claim mechanism drops PatternLock from 0.989 to 0.033, RouteStick from 0.567 to 0.067, and StopCube from 0.355 to 0.135. Moreover, this ablation reduces the average from 0.660 to 0.551, accounting for most of the gap between the full system and no harness. This reliance on benchmark-specific components weakens the paper's claim that a general propose-verify contract carries the memory competence. Excluding these three tasks, the harness improves over no harness by only +0.054 at 9B rather than +0.148.

Questions for authors
  1. On RoboMemArena, the harness without the writer achieves 38.7/58.6, essentially identical to the full system's 38.8/59.0. The paper further explains that, because each task has a fixed subtask sequence, the predicates can raise the completion claims themselves. This suggests that the learned writer contributes very little to the reported RoboMemArena performance, while the hand-specified transition predicates may account for much of the transfer. What evidence does this experiment provide for cross-benchmark transfer of GAMMA's learned memory/reasoning architecture, rather than transfer of the verification and transition design?

  2. GAMMA underperforms existing baselines on several tasks despite having access to the additional detector, symbolic bank, and verification components. For example, SwingXtimes is 68.9 versus 92.0 for FrameSamp+Modul, and MoveCube is 62.2 versus 82.7 for MemER; GAMMA also trails at least one baseline on BinFill, StopCube, and VideoUnmask. Could the authors provide a failure analysis for these cases? In particular, are the failures caused by writer omissions, verification errors, reasoner errors, or the fixed executor? Such an analysis would help establish whether the proposed design principles generalize beyond the tasks on which the harness provides its largest gains.

8 comments

We thank the reviewer for separating the harness's generic verdicts from its task-specific extractors; the revision reports both. A revised PDF and an updated code archive are uploaded with this response; the changes in each are listed in the last part.

In brief, per point.

  • Weakness (hand engineering, size of the generic effect): the decomposition is reported, with a 13-task column in Tables 2 and 3, Table 3 regenerated under the Table 2 protocol, and the "most of the memory competence" claim withdrawn; the direction readers address a VLM deficit found in the failure analysis and documented by independent benchmarks.
  • Q1 (RoboMemArena): a worked example shows why a fixed plan leaves the writer nothing to decide and the two rows coincide; Section 5.5 is retitled to the transfer of verified hand-off control; both rows under one stated protocol.
  • Q2 (failure analysis): every failed episode of the five tasks is attributed from its serve trace; findings in Section 5.4 and Appendix D.
  • Code, checkpoints and serve traces are released (supplementary material and an anonymous link).

Hand engineering and the size of the generic effect (weakness). Without RouteStick, PatternLock and StopCube the harness is worth +2.1 in the regenerated rows (68.9 against 66.8; +12.6 on all 16 tasks), and the REJECT ablation removes extractor-owned claims rather than generic verification. The revision reports this decomposition as such: Tables 2 and 3 carry a 13-task column, the claim that the contracts "carry most of the memory competence" is withdrawn, the grounded text memory is credited with the identity tasks, the generic verdicts with the tasks that have a checkable percept (the press-verify DEFER is the whole +2.1), and the direction readers and the arrival predictor are presented as what they are: the contract's way of letting an evidence extractor own a claim type the VLM measurably cannot read. The direction readers are task-specific code, and we state how they arose and what they address. In the on-policy failure analysis of the benchmark the writer's direction words for RouteStick and PatternLock (the stroke's side and rotation sense; its eight-way heading) came out at chance (Appendix B), and every prior VLM pipeline is at chance on these two tasks. Reading direction is a documented deficit of current VLMs, measured independently of this benchmark. Kamath et al. (EMNLP 2023, "What's up with vision-language models?") evaluate 18 models on left/right and above/below relations and find all of them poor (BLIP fine-tuned on VQAv2 at 56% against 99% for humans); the DORI orientation benchmark (Tasnim, Nichols et al., arXiv 2505.21649) finds 26 models near random on object orientation, the best at 64.2% on categorical and 42.9% on metric judgements; RotBench (Niu et al., arXiv 2508.13968) reports that GPT-5, o3 and Gemini-2.5-Pro do not reliably identify image rotation, with 90° and 270° (clockwise against counter-clockwise) the hardest cases; Ori-Bench (Wang et al., Orient Anything, arXiv 2412.18605) puts GPT-4o and Gemini-1.5-Pro near 60% on basic direction recognition. The benchmark put a clean instance of this general deficit in front of us, and the contract's response to it is the general one: a claim type the proposer cannot read is owned by an evidence extractor, and the writer's word is rejected in its favour (Table 8 of the revised PDF lists the guards). The two readers are small, apply to any memory claim that depends on a direction percept, and recover the oracle on the two tasks (PL 98.9, RS 56.7): a simple and effective remedy for a failure class that reaches past this benchmark. The arrival predictor of StopCube is the same mechanism applied to a timing percept (counted arrivals and a predicted period). Table 3 is regenerated with the released code and checkpoints under the Table 2 protocol (full system 67.9, w/o CORRECT 67.7, w/o DEFER 65.8, w/o REJECT 57.8, no harness 55.3; integer counts per task and seed in Appendix C); the full pipeline is in the supplementary material and all trained weights are shared through an anonymous link (https://huggingface.co/123123xxxx/gamma-checkpoints).

RoboMemArena (Q1). The two rows coincide because of the benchmark's structure. Every RoboMemArena task comes with a fixed subtask sequence, the benchmark's own: the released demonstrations are segmented into these subtasks and the authors' executor is prompted with the subtask text. Task 1 is an example. Its plan is "pick up the cookies, place the cookies into the basket, pick up the tomato sauce, place the tomato sauce into the basket", the same in every episode; nothing in the scene or in the history changes which subtask comes next, so the only decision left to a memory system is when the executor is handed the next subtask. In our writer-run row the writer's claim for the first subtask is logged as "completed: pick up the cookies" beside the detector reading of that tick (gripper closed on the cookies at width 0.033, end-effector lifted to z 0.59), and the proprioceptive pick predicate fires on the same evidence; on a fixed plan the writer reports the predicate's evidence back, so the hand-off happens at the same tick either way. The predicate-only configuration (38.7 / 58.6) matches the writer-run configuration (38.8 / 59.0) for this reason, and the revision states that what transfers is the verified hand-off control; the learned memory and reasoning have no decision to make on a fixed plan. On RoboMemBench the plan is not fixed: on VUS the instance to act on is fixed by what was seen before an occlusion, on SwingXtimes by the number of swings already made, so the next subgoal is a decision taken over the memory bank by the reasoner and verified by the same contract. The pipeline is a hybrid for this reason: one contract (proposer, verifier, hand-off) whose components engage according to the task's structure, the predicates alone when the plan is fixed, the writer and the reasoner when the next subgoal depends on the history, an evidence extractor when the claim is of a kind the VLM cannot read. The RoboMemArena port is the fixed-plan case of that contract and took under one engineering day (two geometry constants, detector prompts, five proprioceptive predicates audited on the released demonstrations); the writer was trained on a corpus generated from the released demonstrations by the released pipeline and runs in the reported row. Section 5.5 is retitled, the abstract wording is changed, and both rows are reported under the one protocol of Table 16 (Table 20 in the revised PDF; three serving seeds × 50 episodes per task), which the revised caption states in full; a reasoner and writer data for tasks whose plan is not fixed are the next step with more resources.

Reviewer_E3YxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why VLA policies such as π0.5 fail on robot tasks that need memory of earlier events (counting, tracking objects through shuffles, repeating a demonstrated path). It builds on an observation from RoboMME. The same fixed policy goes from 17.9% to 84.1% average success when it is given the oracle's grounded subgoal text. From this the authors argue that the bottleneck is producing the right grounded subgoal from past observations, not low-level control.

The proposed system, GAMMA, produces that subgoal stream without privileged information:

  • SAM-3, with a fixed set of class prompts, detects objects in every frame.
  • A finetuned VLM writer adds at most one grounded text line per replan tick to an append-only memory bank.
  • A second finetuned VLM, the reasoner, reads only the instruction and the bank and outputs the next subgoal for the frozen π0.5.
  • A rule-based harness checks every claim against detector evidence and admits, defers, corrects or rejects it. The same rules are used to generate the training corpus.
  • For the two direction tasks (RouteStick, PatternLock), hand-written deterministic readers replace the VLM's output. The main result is 66.0% average success on the 16 RoboMME tasks, against 44.5% for the best prior non-privileged method and 84.1% for the oracle. A 0.8B variant reaches 57.4%. Validation also includes:
  • per-verdict and model-scale ablations
  • transfer to RoboMemArena (38.8% task success)
  • a per-tick failure analysis comparing GAMMA with MemER and GroundSG+QwenVL
Critical strengths
  1. The problem decomposition is well supported and useful. The comparison under one fixed policy (end-to-end 17.9, oracle without coordinates 49.6, oracle with coordinates 84.1) shows clearly where the difficulty lies. It also motivates producing grounded text subgoals instead of adding memory inside the policy. This framing is useful beyond this specific system.

  2. The gains are large and the analysis explains them. Even leaving out RouteStick and PatternLock, where hand-written readers are used, GAMMA averages about 64.4 on the remaining 14 tasks versus about 42.3 for FrameSamp+Modul. The biggest improvements are on the shuffle and binding tasks, roughly doubling prior methods. The tick-level diagnosis (Tables 8 to 13) shows concrete failure modes in the baselines: pre-shuffle coordinates, premature press switches, chance-level direction words. Because the memory is plain text, the authors can attribute failures, and that is a real advantage of the design.

Critical weaknesses
  1. A noticeable part of the result comes from task-specific engineering rather than the learned memory system, and the paper's framing understates this.

    • The REJECT ablation drop (66.0 to 55.1) is almost entirely from RouteStick and PatternLock (PL 98.9 to 3.3, RS 56.7 to 6.7). On those two tasks the VLM output is replaced by deterministic readers that use benchmark regularities (fixed 50-frame strokes, marker semantics).
    • StopCube uses an arrival-period predictor that the authors themselves say is outside the evidence contract.
    • The detector vocabulary and prompts are fixed per benchmark (Table 18), and the corpus generator needs a per-task role vocabulary.
    • On the 14 tasks without readers, the harness adds about 6.5 points at 9B (roughly 57.8 to 64.4), much less than the headline +14.8.
    • On RoboMemArena, the harness predicates alone with no writer give 38.7 versus 38.8 for the full system. So the transfer result shows that the rules transfer, not the two-agent memory. The method is still interesting. But claims like "general contract" and "transfers with unchanged contracts" need to be scoped more carefully.
  2. The comparison to baselines is not fully matched. All baseline and oracle numbers are copied from RoboMME, which used 50 episodes over three checkpoints and three seeds. GAMMA uses 30 episodes per task per seed, three serving seeds, and one checkpoint. No confidence intervals are given. The average margin is large enough that I believe it, but per-task claims such as "best on nine of sixteen tasks" or "reaching the oracle ceiling" could be within noise for several tasks.

Questions for authors

Clarification (could change my assessment):

  1. Can you report the main table and the harness ablation excluding RouteStick, PatternLock and StopCube? Alternatively, mark which numbers depend on non-learned components. This would show how much the writer/reasoner design contributes by itself.
  2. Can you re-run at least the strongest baselines (FrameSamp+Modul, MemER) under your evaluation protocol, or report confidence intervals for both? Otherwise per-task comparisons are hard to interpret.
  3. For RoboMemArena, what does the writer contribute, given that predicates alone match the full pipeline? Is there a task family where the writer helps?
  4. How much manual work does a new task take: detector prompts, role vocabulary, evidence predicates and their thresholds? The port to RoboMemArena changed prompts and five predicates. How long did that take, and was any of it tuned on evaluation episodes?

Clarification (presentation):

  1. Figure 2 does not show the instruction as an input, though Eq. (1) and the prompt templates include it for both agents. Please make the pipeline figure consistent, and state clearly that the detector runs a fixed class list with no task conditioning.

Future work (does not affect my score):

  1. What is the wall-clock cost per tick (one SAM-3 pass plus two 9B generations), and could it run asynchronously at real control rates?
5 comments

We thank the reviewer for the precise accounting of what the readers and the arrival predictor contribute; the revision reports the decomposition. A revised PDF and an updated code archive are uploaded with this response; the changes in each are listed at the end of Part 4/4.

In brief, per point.

  • W1 / Q1 (what the harness is worth without the extractors): Tables 2 and 3 carry a 13-task column; the headline delta is replaced by a per-component decomposition, with Table 3 regenerated under the Table 2 protocol.
  • W2 / Q2 (matched comparison, intervals): cluster-robust intervals per task, FrameSamp+Modul and MemER re-run under our protocol, a paired comparison on the authors' released runs; Section 5.2 is rewritten.
  • Q3 / Q4 (RoboMemArena): what transfers is the verified hand-off control, with a worked example of why a fixed plan leaves the writer nothing to decide; predicate-only and writer-run rows under one stated protocol; the port's cost and what a new task needs.
  • Q5 / Q6 (Figure 2, detector conditioning, serving cost): Figure 2 redrawn; the detector's fixed class list stated; serving cost measured per tick.
  • Code, checkpoints and serve traces are released (supplementary material and an anonymous link).

What the harness is worth without task-specific extractors (weakness 1, Q1). Without RouteStick, PatternLock and StopCube the 13-task averages are GAMMA 66.6, FrameSamp+Modul 42.3, MemER 49.9, GroundSG+QwenVL 39.3 and the privileged oracle 87.9; Tables 2 and 3 carry this column. On the regenerated runs the full system scores 68.9 and the no-harness configuration 66.8 on these 13 tasks (67.9 and 55.3 on all 16), so the generic verdicts account for +2.1 and the extractors for the rest; on the 14 tasks without direction readers the effect is +3.9, within the half-widths given below. The revision separates the two sources of the headline delta: the generic verdicts (press-verify DEFER, duplicate REJECT, grounding CORRECT) and the claim-type-specific extractors (the direction readers, the arrival predictor) and, at 0.8B, the additional completion gates of the generic family. The StopCube predictor stays flagged as the one anticipatory component.

What we claim is stated with one comparison behind each part. (i) The grounded text memory carries the identity tasks: on VideoUnmaskSwap, VideoRepick and VideoPlaceOrder GAMMA scores 75.6 / 65.6 / 66.7 against 38.0 / 30.4 / 36.0 for the best prior method, outside both intervals, and the regenerated no-harness configuration (writer, bank and reasoner kept, every claim admitted) scores 76.7 / 70.0 / 75.6 against 76.7 / 71.1 / 75.6 with the full harness on the same episodes. The verdicts add nothing there; the gain belongs to the memory, not to write verification. (ii) The generic verdicts matter where a claim type has a checkable percept, and the w/o DEFER row measures this directly: removing DEFER alone costs 14.5 on ButtonUnmask (86.7 to 72.2), 14.5 on SwingXtimes (77.8 to 63.3) and 2.1 on the 13-task mean (68.9 to 66.8), the whole generic effect there; removing REJECT or CORRECT alone leaves the 13-task mean at 68.8; every other task moves by less than its half-width. (iii) The extractors carry RouteStick, PatternLock and StopCube and are reported as task-specific components. The contract is then described as what the data support: the mechanism through which perception tools, learned or scripted, enter the memory, with the gain attributed per component and the +14.8 headline replaced by this decomposition.

Regenerated ablations, code and checkpoints. Table 3 is re-run with the released code and checkpoints under the Table 2 protocol (three serving seeds × 30 episodes per task, pooled counts): full system 67.9, w/o CORRECT 67.7, w/o DEFER 65.8, w/o REJECT 57.8, no harness 55.3, with integer counts per task and seed in Appendix C. The complete pipeline is in the supplementary material, and all trained weights (both agent pairs, the executor) and the serve traces are shared through an anonymous link (https://huggingface.co/123123xxxx/gamma-checkpoints).

Matched comparison and confidence intervals (weakness 2, Q2). With the demonstration as the cluster (the three serving seeds replay the same 30 demonstrations), cluster-robust 95% half-widths are 9–16 points per task and 2–4 on PickXtimes, PatternLock and InsertPeg. With binomial intervals for the RoboMME-reported baselines (on the reported run's episodes: 150 for MemER, whose row is one checkpoint × three seeds × 50; 450 assumed for the other rows, optimistic for the same reason), GAMMA's lead is outside both intervals on VideoUnmaskSwap, VideoRepick, VideoPlaceOrder and PatternLock, its deficit on SwingXtimes is outside both, and the other eleven tasks, MoveCube among them, overlap. Section 5.2 is rewritten on this classification, and "reaching the oracle ceiling" is stated only where the interval contains the oracle value. Two further comparisons. (a) FrameSamp+Modul re-run under our protocol with the released policy (episodes 0–29, seeds 7, 8 and 9) scores 46.5 over the 16 tasks (669 of 1,440; per task in Appendix E) and reproduces the benchmark authors' released run of the same checkpoint on these episodes (48.1) within sampling noise: on seed 7 the two runs agree on 429 of 480 outcomes, 238 against 227 successes. MemER re-run the same way with the released adapter checkpoint scores 43.5 over the 16 tasks (626 of 1,440; per task in Appendix E, Table 19) against 42.0 for the authors' released run of the same checkpoint on these episodes; on seed 7 the two runs agree on 371 of 480 outcomes, 210 against 201 successes; the per-episode outcomes of both re-runs are in GAMMA_baseline_reruns.zip at the checkpoint link. (b) The authors' released per-episode results give a paired comparison on the same 30 demonstrations of every task (seed 7): GAMMA succeeds where FrameSamp+Modul fails on 150 episodes and the reverse on 57 (exact binomial test on the discordant pairs, p < 1e-10), and 155 against 36 for MemER (p < 1e-18); on these episodes FrameSamp+Modul averages 48.1 and MemER 42.0 against 66.0 for GAMMA. The classification above does not depend on the re-runs: where GAMMA leads, the Wilson intervals are about 20 points apart or more (VideoUnmaskSwap 75.6 [65.8, 83.3] against MemER 38.0 [30.6, 46.0]; VideoRepick 65.6 [55.3, 74.6] against FrameSamp+Modul 30.4 [26.4, 34.8]; PatternLock 98.9 [94.0, 99.8] against 53.6 [48.9, 58.1]). Over all 50 test episodes per task under the three serving seeds (150 per task and configuration), the full system scores 66.4 and the no-harness configuration 55.3 (Table 23 of the revised PDF).

RoboMemArena (Q3, Q4). Every RoboMemArena task comes with a fixed subtask sequence, the benchmark's own (task 1: pick up the cookies, place the cookies into the basket, pick up the tomato sauce, place the tomato sauce into the basket, the same in every episode), so the only decision left to a memory system is when the executor is handed the next subtask. The writer's completion claim and the proprioceptive predicate fire on the same evidence (for the first pick of task 1 the writer logs "completed: pick up the cookies" with the gripper closed on the cookies and the end-effector lifted, which is the pick predicate's condition), so the writer-run and predicate-only rows coincide (38.8 / 59.0 against 38.7 / 58.6). The revision says so: Section 5.5 is retitled to the transfer of verified hand-off control, the predicate-only configuration is reported beside the writer-run configuration under the one protocol of Table 16 (Table 20 in the revised PDF; three serving seeds × 50 episodes per task, family entries as means of the per-task rates), and "transfers with unchanged contracts" leaves the abstract. The harness port took under one engineering day: two geometry constants, detector prompts for the new object classes, and five proprioceptive evidence predicates (pick, place, pour, open, close) with fixed geometric thresholds (lift height, pour angle, drawer stroke) and per-object grasp-width priors read from the released demonstrations, audited on the released training demonstrations (879 of 888 segments verified inside their own segment). The writer was then trained on a corpus generated from the released demonstrations by the released pipeline and runs in the reported configuration; the contract is a hybrid whose components engage according to the task's structure: the predicates alone when the plan is fixed, the writer and the reasoner when the next subgoal depends on the history (the RoboMemBench tasks), an evidence extractor when the claim is of a kind the VLM cannot read; no reasoner is trained here because the plan is fixed; nothing on the RoboMME side changed. The hand-off timing rule (admit a verified claim after it has held for two ticks) was selected on 10-episode runs of seeds 50–59, inside the evaluated range 50–99; the revision states this. A new task needs detector phrases for its object classes, one evidence predicate per completion type its subgoals use, and a reader only if its demonstrations encode a percept the VLM measurably cannot read. With more time and compute, the next step is a reasoner and writer data for tasks whose plan is not fixed.

Figure 2, the detector's class list, serving cost (Q5, Q6). Figure 2 now shows the instruction as an input to both agents, consistent with Eq. (1), and Section 4.1 states that the detector runs a fixed class list with no task conditioning (phrases in Table 22). Per replan tick of 16 control steps on one H200, medians over 36,459 ticks of the regenerated full-system runs: detector 3.2 s, writer 0.4 s (p90 2.3 s), reasoner 1.2 s, 4.9 s in total (the card was shared by two serving stacks; the detector alone measures 2.4 s on an idle card). The closed-loop runs wait for the call, so success rates do not depend on it; a deployment runs the agents asynchronously at the subgoal timescale.

Reviewer_W7atOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GAMMA addresses memory-dependent manipulation by generating grounded language subgoals for a fixed, previously fine-tuned executor. SAM-3 provides object detections; a fine-tuned VLM writer converts observation windows into an append-only event bank; and a second VLM reads the bank to predict subgoals. A programmed propose–verify mechanism admits, defers, corrects, or rejects claims using perceptual evidence. The same evidence constraints guide training-target construction, and the reasoner is trained on the writer’s predicted banks. The deployed system also uses deterministic demonstration readers and progress control for selected tasks. On 16 RoboMME tasks, the paper reports 66.0% average success versus 44.5% for the strongest listed non-privileged baseline and 84.1% for the reported grounded oracle. Validation includes verdict ablations, two model sizes, trace diagnostics, and a RoboMemArena adaptation reporting 38.8% task success.

Critical strengths
  1. A substantial and practically meaningful improvement on memory-dependent manipulation. With its executor held fixed, GAMMA reports strong results across several identity-sensitive tasks: VideoUnmaskSwap reaches 75.6% versus 38.0% for MemER, VideoRepick 65.6% versus 30.4% for FrameSamp+Modul, and VideoPlaceOrder 66.7% versus 36.0% for TokenDrop+Modul (Table 2). The gains therefore extend beyond the two tasks handled by deterministic route readers. The main 66.0% aggregate is also consistent with the per-seed outcomes in Table 14. These results support the practical value of explicit grounded event histories.

  2. Useful, inspectable diagnosis and candid component analysis. The bank and tick-level traces make errors in identity, grounding, and completion timing easier to examine (Section 5.4; Appendix D). Matching corpus-time and deployment inputs, and training the reasoner on predicted banks, are sensible measures against distribution mismatch. The paper also explicitly documents its deterministic readers, unsuccessful development variants, and the negligible writer contribution on the second benchmark. This transparency makes the system informative even where its broader claims remain unproven.

Critical weaknesses

The reliability of the main evaluation is still unclear. Appendix C reports eight development rounds, while all serving seeds reuse the same 30 demonstrations per task. The paper should clarify which final cases were untouched by method selection; changing seeds alone does not solve demonstration reuse. Some numbers also seem inconsistent with the stated episode counts. In Table 3, 95.9% and 13.1% cannot come from integer successes out of 90, even with rounding. Table 16’s 80.7% is also incompatible with the 0.5-point increments allowed by its episode count. These may be reporting errors, but they affect the ablation and transfer evidence.

The experiments do not clearly separate learned memory from programmed components. Removing REJECT also replaces reader-provided claims with VLM claims, causing large performance drops. Since the readers use task-specific perception and deterministic progress control, this ablation does not isolate general claim verification. The transfer evidence is also limited: RoboMemArena fixes the reasoner to a recorded plan, and predicates alone nearly match the writer’s results (38.7/58.6% versus 38.8/59.0% task/subtask success). This supports useful handoff control, but provides little evidence for transferring learned memory writing or reasoning. Since AGM already studies physically verified progress over frozen VLAs, a matched simpler event/binding baseline is needed to show what GAMMA adds. The current controls do not fully support the strongest claims about memory medium, multi-agent structure, and generality.

Questions for authors

pls provide the episode-index splits used for agent training, validation, prompt/predicate/reader tuning, every screening round, and final evaluation. Were the Table 7 development cases reused during method selection? What are the main and ablation results on demonstration indices never used for these decisions, with uncertainty accounting for repeated demonstrations across serving seeds?

pls provide integer successes and attempted episodes per task and seed for Tables 3 and 16, explain the averaging and any exclusions, and reconcile the examples above with the stated protocols. Also identify the three tasks excluded from the “identical evidence extractors across scales” statement in Section 5.3 and explain how that exception is consistent with the claim that only model size changed.

Under the same executor, detections, readers, training budget, and untouched evaluation cases, how does full GAMMA compare with a simpler deterministic event/binding bank or a single VLM using the same evidence? Which results establish the incremental value of the learned writer–reasoner separation, independently of the additional tools and supervision?

How is the recorded RoboMemArena plan obtained, and do the compared methods receive equivalent plan information? Can the central comparison use the same executor and layouts, given that PrediMem predicts subtasks from observations and memory [2], while GAMMA’s reasoner is pinned to a plan? Otherwise, would the authors narrow this claim to transfer of verified handoff control, consistent with Table 16’s no-writer result?

8 comments

We thank the reviewer for the careful reading, and in particular for checking the arithmetic of Tables 3 and 16. Both checks were right: Table 3 is addressed with new runs, Table 16 with the protocol and the counts behind it. A revised PDF and an updated code archive are uploaded with this response; the changes in each are listed at the end of Part 6/6.

In brief, per point.

  • W1 / Q2 (reproducibility, Table 3): the complete code is in the supplementary material, all weights and serve traces at an anonymous link; every row of Table 3 is regenerated with the released code and checkpoints under the Table 2 protocol, with integer counts per task and seed in Appendix C.
  • Q2 (Table 16): the one protocol is stated (three serving seeds × 50 episodes per task), the family total behind 80.7 is given (484 of 600), the Sequence family re-run with the released code and checkpoints is complete (per serving seed 82.5 / 68.5 / 75.0, mean 75.3; per task and seed in Part 2), and the counts for the other families are being regenerated.
  • W1 / Q1 (splits, method selection): the development loop of the harness predicates is stated with the episodes it used; the full system and the no-harness configuration were run on test episodes 30–49, which no decision used: 64.1 against 55.3 there, 66.4 against 55.3 over all 50 episodes (per-task table in Part 2).
  • Q1 (uncertainty with repeated demonstrations): cluster-robust intervals per task and a paired comparison on the benchmark authors' released runs; Section 5.2 is rewritten on them.
  • Q2 (the three tasks, 0.8B): the 0.8B guard set and its task scope are listed; the "identical extractors across scales" comparison is withdrawn and the two-scale comparison is restated on all 16 tasks.
  • W2 / Q3 (learned memory vs. programmed components, AGM): the controls that separate them are identified in Tables 2 and 3 with a 13-task column; AGM is cited and positioned.
  • Q4 (RoboMemArena): what transfers is the verified hand-off control, a consequence of the benchmark's fixed plans and the designed behaviour of the pipeline; the port's cost and the next step are stated.

Code and checkpoints (weakness 1, Q2). The complete pipeline behind the closed-loop numbers is in the supplementary material: detector prompts and thresholds, the corpus generator with its verification gates, the training scripts for both agents, the agent server, the closed-loop evaluation client with lane launchers and serving seeds, and the per-tick trace logger. All trained weights (writer and reasoner adapters at both scales, the subgoal-conditioned executor) and the per-tick serve traces of the regenerated runs are shared through an anonymous link (https://huggingface.co/123123xxxx/gamma-checkpoints), so every number can be checked independently.

Table 3 (weakness 1, Q2). The reviewer's observation is explained by how the ablation rows were assembled. Table 2 pools successes over the 90 episodes of a configuration (three serving seeds × 30 episodes; the per-seed values are in Appendix E). The ablation rows of Table 3 were compiled from the episodes each serving seed had completed at the time, fewer than 30 for some seed–task pairs, so some cells are rates over fewer than 90 episodes and do not lie on the grid. The difference concerns the ablation rows only; Table 2 and the per-seed table (Table 16 of the revised PDF) come from complete runs. We have regenerated every row of Table 3 with the released code and checkpoints under the Table 2 protocol (same checkpoints and executor, seeds 7, 8 and 9, test episodes 0–29, every cell scored over 90 completed episodes), and Appendix C lists the integer successes per task and seed. The regenerated means are: full system 67.9, w/o CORRECT 67.7, w/o DEFER 65.8, w/o REJECT 57.8, no harness 55.3 (submitted: 66.0, 65.0, 62.7, 55.1, 51.2). The picture is unchanged in kind. REJECT acts on StopCube, PatternLock and RouteStick (33.3, 98.9 and 58.9 to 4.4, 16.7 and 8.9); DEFER acts on ButtonUnmask (86.7 to 72.2) and SwingXtimes (77.8 to 63.3); no other cell moves by more than its interval half-width, and CORRECT has no measurable effect at this sample size. The switches that produce each row (WAM_HARNESS_OFF, WAM_NO_DEFER, WAM_NO_REJECT, WAM_NO_CORRECT) are named in the caption and are all exposed in the updated archive.

Table 16 (Q2; Table 20 in the revised PDF). One protocol for every row: three serving seeds × 50 episodes per task (10 episodes per task where a row is marked), so unmarked rows are scored over 150 episodes per task; the family sizes are 4 tasks for Transferring and Sequence, 7 for Counting and 11 for Occlusion, so a family entry pools 600, 600, 1,050 and 1,650 episodes. The caption gave the per-seed count without the seed multiplier, which produced the apparent 0.5-point grid. Family entries are means of the per-task rates, each rounded to one decimal before the mean is taken, so a family entry need not be a multiple of one over its episode count; Transferring 27.4 (four tasks, 600 episodes), between 164/600 = 27.3 and 165/600 = 27.5, is such a case. The Sequence entry 80.7% is 484 of 600 episodes. The per-task, per-seed records behind it, and the records of the other three families, were not retained, so the counts are being regenerated with the released code and checkpoints. RoboMemArena is 26 multi-stage long-horizon tasks at 50 episodes per task and seed (3,900 episodes per row), which is why the Sequence family comes first within the discussion period; the remaining families will be posted in this thread as the re-run completes. The Sequence re-run with the released code and checkpoints is complete under the table's protocol (layouts 50–99, serving seeds 7, 8, 9; 600 episodes). Per serving seed it scores 82.5, 68.5 and 75.0 (165, 137 and 150 of 200), mean 75.3 against 80.7 in the table, so the table's value lies inside the range the three serving seeds span; the subtask-success mean is 88.2 against 90.3. Per task, Put Cookies Sauce into Target Container in Order 28 of 150 (20, 1 and 7 of 50 under the three seeds), Put Butter Popcorn into Target Container in Order 149 of 150, Put Cream Pudding into Target Container in Order 150 of 150 and Pour Sauce Twice Put Cookies into Microwave 125 of 150 (45, 37 and 43). The seed spread sits on the first task, and it is the executor's: the plan and grounding are identical across the serving seeds and the first subtask completes in every episode, after which the executor's grasp of the tomato sauce succeeds under one sampling seed and fails under the other two; the hand-off contract behaves the same way in every seed. The other three tasks score 424 of 450 (94.2). The per-episode records of the re-run (GAMMA_rma_seq_rerun.zip) are at the checkpoint link. The code archive and the checkpoint link contain the RoboMemArena port, the writer adapter and the executor, so every count in the table can be regenerated independently. Appendix G gives the Sequence total; the other families and the predicate-only row are reported as family means in the PDF.

Episode splits and method selection (weakness 1, Q1). Both agents are trained on the benchmark's training split (100 demonstrations per task, every tenth held out for validation), and the corpus-time contracts were developed on that split. Evaluation uses test episodes 0–29 of each task under three serving seeds; the seed sets the policy's sampling and the demonstration is fixed by the episode index, so the three seeds replay the same 30 demonstrations (the Appendix E sentence "only the environment seed differs" is corrected). The harness has no trained weights: its verdicts are predicates over detector output and proprioception with a handful of thresholds. These predicates were developed on-policy from the deployed system's own closed-loop failures: the eight screening rounds of Appendix C ran test episodes 0–9 of each task under seed 7, each failed episode was attributed to a defect class from its trace, and the repairing predicate was added and re-screened. So the writer and reasoner never saw a test episode, while the serve-time predicates were selected against a fixed subset of 160 test episodes inside the reported 30; Table 7 is that record, and the revision labels those episodes. Two further uses of test episodes are stated in the same caption: the press-verify rule was set after an audit of a full 30-episode seed-7 run, after which the ButtonUnmask, ButtonUnmaskSwap and PatternLock seed-7 cells were re-run, and the 0.8B guards were iterated on 30-episode runs (PickXtimes on all three seeds). Test episodes 30–49 of each task were never run by any configuration before this rebuttal (every screening round and audit used indices inside 0–29), so they are the indices the reviewer asks for. We ran the full system and the no-harness configuration on them under the three serving seeds (60 episodes per task and configuration). On episodes 30–49 the full system scores 64.1 (615/960; Wilson 95% [61.0, 67.0]) and the no-harness configuration 55.3 (531/960; [52.2, 58.4]); over all 50 test episodes, 66.4 (1593/2400) against 55.3 (1327/2400). The table below gives the per-task rates on episodes 0–29, on episodes 30–49 with Wilson intervals, and over all 50, for both configurations; Table 23 of the revised PDF (Appendix E) is the same table. The per-verdict rows of Table 3 are re-run on episodes 0–29; on 30–49 the ablation is the full system against no harness.

TaskFull 0–29Full 30–49 [95% CI]Full all 50No harness 0–29No harness 30–49 [95% CI]No harness all 50
PickXtimes95.698.3 [91.1, 99.7]96.795.695.0 [86.3, 98.3]95.3
BinFill71.171.7 [59.2, 81.5]71.372.268.3 [55.8, 78.7]70.7
SwingXtimes77.875.0 [62.8, 84.2]76.767.871.7 [59.2, 81.5]69.3
StopCube33.330.0 [19.9, 42.5]32.05.61.7 [0.3, 8.9]4.0
VideoUnmask92.285.0 [73.9, 91.9]89.387.891.7 [81.9, 96.4]89.3
ButtonUnmask86.773.3 [61.0, 82.9]81.368.978.3 [66.4, 86.9]72.7
VideoUnmaskSwap76.780.0 [68.2, 88.2]78.076.773.3 [61.0, 82.9]75.3
ButtonUnmaskSwap36.733.3 [22.7, 45.9]35.338.931.7 [21.3, 44.2]36.0
PickHighlight67.868.3 [55.8, 78.7]68.066.770.0 [57.5, 80.1]68.0
VideoRepick71.168.3 [55.8, 78.7]70.070.073.3 [61.0, 82.9]71.3
VideoPlaceButton78.960.0 [47.4, 71.4]71.383.373.3 [61.0, 82.9]79.3
VideoPlaceOrder75.661.7 [49.0, 72.9]70.075.671.7 [59.2, 81.5]74.0
MoveCube64.460.0 [47.4, 71.4]62.761.155.0 [42.5, 66.9]58.7
InsertPeg1.15.0 [1.7, 13.7]2.74.48.3 [3.6, 18.1]6.0
PatternLock98.996.7 [88.6, 99.1]98.05.611.7 [5.8, 22.2]8.0
RouteStick58.958.3 [45.7, 69.9]58.74.410.0 [4.7, 20.1]6.7
Suite mean67.964.166.455.355.355.3
Harness gain+12.6+8.8+11.1

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to all reviewers

We thank the reviewers for their careful reading. A revised PDF and an updated supplementary code archive are uploaded with these responses, and detailed replies to every point are under the corresponding review.

Revised PDF. Table 3 is regenerated with the released code and checkpoints under the Table 2 protocol, with integer counts per task and seed in Appendix C; Tables 2 and 3 carry a 13-task column; Section 5.2 is rewritten on per-task confidence intervals and a matched comparison with the baselines (Appendix E); Section 5.5 is retitled to the transfer of verified hand-off control, with one protocol stated for every row of the RoboMemArena table and the predicate-only configuration reported; Figure 2 is redrawn; the serving cost is measured. A list of changes closes each reply.

What the revision shows. The grounded text memory, written by one agent and read by another, carries the memory-dependent tasks: on the three identity tasks under occlusion (VideoUnmaskSwap, VideoRepick, VideoPlaceOrder) GAMMA roughly doubles the best prior method, outside the cluster-robust intervals now reported. The harness is reported as two parts: generic verdicts, and evidence extractors for claim types the VLM measurably cannot read; the direction readers answer a deficit of current VLMs that independent benchmarks document (direction and orientation reading), found here in the on-policy failure analysis. The contract is one pipeline whose components engage according to the task's structure: the writer and the reasoner where the next subgoal depends on the history (RoboMemBench), the verified hand-off alone where the plan is fixed (RoboMemArena), which is why the predicate-only and writer-run rows of that benchmark coincide.

Code archive (supplementary material). The complete pipeline: detector prompts and thresholds, the corpus generator with its verification gates, the training scripts for both agents, the agent server with one switch per Table 3 row, the closed-loop evaluation client with lane launchers and serving seeds, the per-tick trace logger and the aggregation script behind the regenerated rows. CHANGELOG_rev1.md lists every file that differs from the submitted archive.

Checkpoints and serve traces. All trained weights (writer and reasoner adapters at both scales, the subgoal-conditioned executor) and the per-tick serve traces of the regenerated runs are at an anonymous link: https://huggingface.co/123123xxxx/gamma-checkpoints.

All 50 test episodes. The full system and the no-harness configuration have been run on all 50 test episodes of each task under the three serving seeds: 66.4 against 55.3 (150 episodes per task and configuration; Appendix E and Table 23 of the revised PDF). MemER re-run under our protocol with the released checkpoint scores 43.5 over the 16 tasks against 42.0 for the benchmark authors' released run of the same checkpoint on the same episodes (Appendix E, Table 19).

AuthorsOpenReview ↗

Closing summary from the authors

Summary of the discussion. We thank the reviewers for a discussion that improved the paper. The revised PDF and code archive are uploaded; all weights, serve traces and per-episode records are at the anonymous link (https://huggingface.co/123123xxxx/gamma-checkpoints), and every number in the revision can be regenerated from them.

What changed. Table 3 is regenerated with the released code and checkpoints under the Table 2 protocol, with integer counts per task and seed; Tables 2 and 3 carry a 13-task column; Section 5.2 reports cluster-robust intervals per task and a matched, paired comparison with the baselines; the full system and the no-harness configuration are reported on all 50 test episodes of every task (66.4 against 55.3); FrameSamp+Modul and MemER are re-run under our protocol and match the benchmark authors' released runs; Section 5.5 is retitled to the transfer of verified hand-off control with one protocol for every row; the claim that the contracts "carry most of the memory competence" is replaced by the per-component decomposition, and the two-scale comparison is restated on all 16 tasks with each scale's guard set listed.

What the evidence supports. (i) The grounded text memory, written by one agent and read by another, carries the memory-dependent tasks: the no-harness configuration scores 55.3 over the 16 RoboMemBench tasks against 44.5 for the best prior method, and on the three identity tasks under occlusion GAMMA roughly doubles the best prior method, outside the intervals. (ii) The harness adds +12.6 and is reported per component: generic verdicts where a claim type has a checkable percept, evidence extractors where the VLM measurably cannot read the percept (direction reading, a deficit that independent benchmarks document). (iii) The contract is one pipeline whose components engage according to the task's structure: on RoboMemBench the writer and the reasoner decide the next subgoal from the history; on RoboMemArena, whose plans are fixed, the verified hand-off alone acts, and the writer-run and predicate-only rows coincide. (iv) Every number the reviewers asked about has been regenerated from the released code and checkpoints during the discussion period, with per-episode records at the link: all of Table 3, the two baselines under our protocol, the full system on all 50 test episodes, and the RoboMemArena table family by family in the W7at thread, where the regenerated counts are posted as they complete.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.