acceptodds
Under review as a conference paper at ICLR 2027

AGM: Achievement-Grounded Memory for Closed-Loop Agents with Frozen VLA Policies

Abstract

Frozen vision-language-action (VLA) policies offer broad manipulation skills but execute open-loop action chunks without tracking task progress, so the agent cannot reliably decide whether to continue, retry, or terminate. External memory is a natural remedy, yet we find it can be harmful when attempted actions are recorded as completed progress: transient execution failures become persistent task-state errors, and such a memory can underperform no progress memory at all. We propose Achievement-Grounded Memory (AGM), a lightweight closed-loop framework for frozen VLA policies. AGM represents a task as a static subgoal sequence with a dynamic progress pointer and advances the pointer on physically verified achievement rather than on attempts. Proprioceptive gripper-load cues decide when to verify; coherent point tracking verifies grasps, and language-conditioned cross-view comparison, read by a single trained 2.43M-parameter verification head, verifies placements. The policy, tracker, and encoder remain frozen, the head is the only trained component, and deployment needs no auxiliary vision-language model. On the RoboMME Counting benchmark, AGM reaches 100.0% on PickXTimes and 84.0% on BinFill, surpassing the strongest memory-augmented baseline by 7.7 points on the four-task average, and the gains carry over to a physical robot, where AGM reaches 100.0% and 82.0%. These results suggest that reliable embodied memory depends more on disciplined state updates than on memory capacity.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Achievement-Grounded Memory, Closed-Loop Agents, VLA

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_aoA3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AGM, a lightweight memory framework for frozen VLA policies in long-horizon repetitive manipulation. Instead of advancing task progress after every attempted action, AGM updates a compact subgoal pointer only when physical achievement is verified. Grasp success is checked using gripper events and point tracking, while placement success is verified from cross-view visual features and a small learned head. The framework supports advance, retry, and rollback without modifying the VLA policy. Experiments on RoboMME Counting and a real robot show large gains on PickXTimes and BinFill.

Critical strengths

1.The paper identifies a simple but important failure mode: recording attempted actions as completed progress can permanently corrupt task state. The distinction between “attempt” and “achievement” is intuitive and well motivated.

2.The controlled comparison between verified and attempt-based memory is particularly convincing. It directly isolates the value of grounded memory writing rather than simply comparing against larger memory architectures.

3.The empirical results are strong on the supported tasks. AGM reaches 100% on PickXTimes and 84% on BinFill, and the same qualitative behavior transfers to a physical robot.

4.The system is lightweight relative to external-VLM baselines. Most perception modules remain frozen, with only a small verification head trained.

Critical weaknesses

1.The current scope is narrow. AGM only verifies subgoals tied to grasp and release events, and therefore does not improve tasks involving other interaction types such as pushing, sliding, or articulated manipulation.

2.The method relies on substantial task-specific structure, including a predefined subgoal sequence, manually specified subgoal types, recoverability rules, state-machine thresholds, and several verification heuristics. This limits the generality of the claimed memory principle.

3.The placement verifier is not uniformly reliable. In particular, the irreversible-placement classifier performs poorly on held-out failures, and the system compensates with a very permissive threshold and task-specific transition logic. This weakens the claim that progress updates are consistently grounded in reliable verification.

4.The strongest gains occur on only two of the four RoboMME tasks; on the other two, AGM simply falls back to the frozen policy. The reported four-task average therefore partially reflects unchanged performance outside the method’s supported event vocabulary.

5.The real-robot validation is useful but limited to two task families, one robot platform, and one gripper configuration. In addition, the policies and verification head are retrained for the physical platform, so the hardware results do not fully demonstrate zero-adaptation transfer.

Questions for authors
  1. How much of AGM’s performance depends on the manually specified subgoal types, recoverability rules, and transition logic, rather than on achievement verification itself?

  2. How should the poor held-out discrimination for irreversible placement be reconciled with the claim that memory updates are physically verified?

  3. Can the authors clarify how much of the four-task average improvement comes specifically from the two supported gripper-anchored tasks, given that AGM leaves the other tasks unchanged?

  4. To what extent can the same verification and transition framework be reused for tasks whose completion is not associated with grasp or release events?

2 comments
AuthorsOpenReview ↗

AGM: Component Contributions, Verification Reliability, and Cross-Platform Reuse

We thank the reviewer for the thoughtful assessment and for recognizing the distinction between attempts and achievements, the controlled comparison, and the real-robot results. Your questions helped us make the contribution of each component more explicit and extend the evaluation to a new interaction type and robot platform.

We have added a new real-robot experiment, AfternoonTeaServe, using a bimanual AgileX Nero robot to serve items such as coffee and cake in the instructed order. Each item requires five subgoals: pick up with one arm, put down to the right of a red line, push onto the target with an empty gripper, pick up from the target with the other arm, and place on the tray. The robot presses a button after serving all instructed items. Evaluation instructions contain 1–4 items, corresponding to 6–21 subgoals.

The push is non-prehensile. If the item does not reach the target, AGM retains the push subgoal and allows another push. This experiment examines verification beyond grasp and release, recovery through corrective pushing, and adaptation to a second robot platform. The task setup and results appear in the experimental section of the revised manuscript (Section 4.6), with implementation details in Appendix E.1–E.5 and adaptation details in Appendix E.6.

1. Contributions of task structure and achievement verification (W2, Q1).

Thank you for raising this attribution question. The controlled comparison answers it directly: the policy, subgoal sequence, and progress pointer are held fixed, and only grasp and placement verification is switched on or off (Appendix C.2, Table 11).

On PickXTimes, success is 32.0% without verification and 100.0% with it: with the task structure already provided, verification alone accounts for 68 percentage points. On BinFill, the results are 82.0% and 84.0%; the provided structure already yields high success there, and verification adds a smaller gain.

The structure itself does not have to be written by hand for each task. With verification retained, replacing the benchmark-provided decomposition with a parser that reads only the instruction, without environment state or task ground truth, changes success from 100.0% and 84.0% to 98.0% and 82.0%. The type labels and recovery rules are defined once per interaction type and reused across instructions and repetition counts; the revised manuscript lists them in Table 17.

2. Reliability of irreversible-placement verification (W3, Q2).

Thank you for raising this concern. We agree that the irreversible-placement head has limited failure discrimination: among 28 held-out events, only 4 are failures, and none is detected at argmax. However, AGM does not equate an interaction event with successful achievement. The event only triggers verification, while the memory update depends on additional execution evidence.

For irreversible placement, this evidence includes a preceding verified grasp, a loaded-release event, and a visual verification score (Appendix A.5). On hardware, we additionally check whether the release occurs within the bin region. The visual head acts as a low-score veto rather than a standalone success classifier. Indeed, it rejected five updates in one BinFill run. We acknowledge that these cues do not guarantee successful placement.

The permissive threshold (0.05) reflects the asymmetric cost of errors: falsely rejecting a successful irreversible placement may cause an extra object to be inserted. Using a stricter, shared threshold reduces BinFill success from 84.0% to 74.0% (Table 3). Moreover, verification adds only 2 percentage points over attempt-based progression on BinFill (82.0% vs. 84.0%), unlike the 68-point gain on PickXTimes (32.0% vs. 100.0%).

Thus, AGM is event-triggered in when to verify and evidence-dependent in whether to update. We do not claim equally reliable achievement discrimination across all interaction types. The BinFill results reflect weaker outcome evidence and a risk-sensitive update rule, rather than strong irreversible-placement verification.

3. Source of the four-task average improvement (W4, Q3).

Thank you for requesting this breakdown. The entire four-task average improvement over the frozen π0.5 entries in Table 1 comes from PickXTimes and BinFill: (57.11 + 54.00 + 0 + 0) / 4 = 27.78 percentage points. On SwingXTimes and StopCube, performance falls back to the frozen policy, as marked by † in the table.

MethodPickXTimes + BinFill averageSwingXTimes + StopCube averageFour-task average
Frozen π0.536.45%21.12%28.78%
SimpleSG + Qwen3-VL-4B86.45%2.75%44.60%
MemER68.00%29.67%48.83%
AGM92.00%21.12%56.56%

On the supported pair, AGM achieves 92.00%, exceeding the strongest baseline on that pair, SimpleSG + Qwen3-VL-4B, by 5.55 percentage points, and MemER by 24.00 percentage points.

AuthorsOpenReview ↗

AGM: Component Contributions, Verification Reliability, and Cross-Platform Reuse

4. Framework reuse beyond grasp and release and across platforms (W1, W5, Q4).

AfternoonTeaServe addresses these points together: a push completes without a grasp or a release, on a second robot and with everyday objects. AGM's memory and verification-driven transition rule are reused unchanged; the extension adds a push event, a type label, and a recovery rule.

Number of instructed items1234Overall
Full-instruction π0.540%20%10%0%17.5%
π0.5 + AGM60%60%40%20%45.0%

With both policies trained on the same demonstrations, AGM succeeds in 18/40 trials, compared with 7/40 for the full-instruction policy. Verification acts during execution: 13 of 42 push subgoals were rejected at least once, 7 subsequently passed verification after further attempts, and video inspection confirmed 5 corrective pushes.

Across platforms, we reuse the frozen vision-language encoder and point tracker, the core gripper state-machine logic, and the MLP architecture and training procedure. For each platform, we refit the policy and verification head and recalibrate gripper thresholds and scene regions; the Nero deployment additionally introduces bimanual segmentation logic and the push trigger. Beyond the existing policy demonstrations, this required approximately 2 hours of failure-data collection and 1 hour of event review (Table 20).

The physical evaluation now covers three task families, two robot platforms, two gripper configurations, and both cubes and everyday objects.

Reviewer_t2uLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a lightweight framework, Achievement-Grounded Memory (AGM), to increase long-sequence manipulation reliability. The idea is that the framework needs to verify the successful completion of a task (subgoal), and then it updates the progress. The motivation behind this is that the current task progress memory is limited, and memory updates are unreliable.

Critical strengths
  1. The structure of the paper is clear, and the paper is quite well written.
  2. The motivation is clear and straightforward.
  3. The authors have also tested real-world rollouts, and the results show performance gains on PickXTimes and BinFill.
Critical weaknesses

For the method sections

The AGM lightweight framework is designed for reliable action execution for long-term manipulation, so in a broader scope, the action requirements may not be limited to grasp and place. What about push/spin? I understand that the Achievement Verification is specifically designed for these two tasks, but real-world challenges may not be limited to these two types of tasks, or in other words, AGM at this current stage has limited generalization. One quick example I can think of at the current stage is spinning, for example, if you spin an object, the object may fall, and then the Achievement Verification module can be applied in this case to better update the memory bank.

Also, in Section 3.3, the authors state that: “Achievement-grounded updates apply only to subgoals admitted by x; other subgoals are delegated to the frozen base policy”, and “bounded timeout and retry rules act only when evidence remains unavailable or repeated rejection would otherwise deadlock the controller”. So we can see here that Achievement Verification is limited to Grasp and Place, and then the framework also suffers from repeated rejection. Thus, at the current stage, it is a weakly verified progression. Some failed cases may also pass, and memory bank updates may be insufficient as well, as introduced in the motivation.

For the experimental sections

Please refer to Table 10. It denotes that, for the BinFill task suite specifically, “Attempt-based (unverified writes)” reaches a success rate of 82.0, and AGM (verified writes) reaches 84.0, so we see a comparatively weak performance gain here. Since the Verification process is quite a significant module in AGM, the contribution of the Verification process in this task suite is quite limited.

Questions for authors

Please refer to the weaknesses section.

2 comments
AuthorsOpenReview ↗

Verification Beyond Grasp and Release: Recovery, Progression, and Task-Dependent Contributions

We thank the reviewer for the thoughtful and constructive feedback. Your comments helped us strengthen the evaluation beyond grasp and place and clarify how verification and recovery contribute to task progress.

We have added a new real-robot experiment, AfternoonTeaServe, using a bimanual AgileX Nero robot to serve items such as coffee and cake in the instructed order. Each item requires five subgoals: pick up with one arm, put down to the right of a red line, push onto the target with an empty gripper, pick up from the target with the other arm, and place on the tray. The robot presses a button after serving all instructed items. Evaluation instructions contain 1–4 items, corresponding to 6–21 subgoals.

The push is non-prehensile. If the item does not reach the target, AGM retains the push subgoal and allows another push. This experiment provides end-to-end results and online recovery examples for verification beyond grasp and release. The task setup and results appear in the experimental section of the revised manuscript (Section 4.6), with implementation details in Appendix E.1–E.5.

1. Verification beyond grasp and release (W1).

Thank you for suggesting pushing and spinning as relevant extensions. These examples highlight the importance of verifying whether an interaction achieved its intended outcome. We implemented and evaluated this extension for pushing in AfternoonTeaServe.

Gripper closure and reopening localize a push attempt, while the verifier uses frozen SigLIP visual features and the subgoal text to determine whether the item has reached the target. Acceptance advances the pointer; rejection retains the push subgoal and allows another push. We reuse AGM's memory representation and verification-driven progress updates, adding a push trigger, a type label, and a recovery rule, with the head retrained on the new platform's events.

We trained the full-instruction and subgoal-conditioned policies on the same 100 demonstrations. The complete AGM system achieves 45.0% success (18/40), compared with 17.5% (7/40) for the full-instruction policy, with improvements at every evaluated sequence length.

Number of instructed items1234Overall
Full-instruction π0.540%20%10%0%17.5%
π0.5 + AGM60%60%40%20%45.0%

The execution records show how push verification contributes. Across 23 analyzed AGM episodes, 13 of 42 push subgoals were rejected at least once, and 7 subsequently passed verification after further attempts. We confirmed 5 corrective pushes among these recoveries through video inspection. In episode-grouped repeated hold-out evaluation, the push head achieves an AUROC of 0.917, with 3.6% of successful pushes rejected.

Your spinning example suggests a corresponding extension: an event marking the end of an attempt, followed by verification of the required object pose and stability. AfternoonTeaServe demonstrates this extension process for pushing through both task-level results and online recovery records.

2. Verification-driven progression and repeated rejection (W2).

We agree that forced advancement after repeated rejection can compromise achievement-grounded memory updates. In the original implementation, the controller advances after a bounded number of rejections to avoid deadlock. In AfternoonTeaServe, we instead terminate the episode after 10 consecutive rejections, without marking the rejected subgoal as complete.

Across the 23 analyzed episodes, all 192 pointer advances were supported by verification. Six push subgoals reached the rejection limit, and their episodes terminated without an unverified advance.

The controlled simulation results further demonstrate why verification matters. With the same policy, subgoal sequence, and pointer, enabling verification improves PickXTimes success from 32.0% to 100.0%. The attempt-based controller terminates prematurely in 34/50 episodes, compared with 0/50 for AGM. Thus, verification prevents unsuccessful attempts from being recorded as completed progress, while the revised rejection handling avoids forced progression when evidence remains insufficient.

AuthorsOpenReview ↗

Verification Beyond Grasp and Release: Recovery, Progression, and Task-Dependent Contributions

3. Contribution of verification on BinFill (W3).

Thank you for pointing out the small difference between 82.0% and 84.0%. We agree that this controlled comparison shows a small additional gain from verification on BinFill. Your observation helps clarify how the contributions of AGM's components vary across tasks.

On BinFill, the attempt-based controller achieves 82.0%, and the complete AGM system achieves 84.0%, under the same policy and subgoal structure. The controller already reaches high success with the given subgoal sequence and explicit progress pointer, with verification adding two percentage points.

This behavior is consistent with BinFill's physical structure. Objects released into the bin leave the subsequent manipulation workspace, so attempts that reach release often correspond to successful placements. PickXTimes repeatedly manipulates an object that remains in the workspace, where unsuccessful attempts must be identified and repeated. Under the same verification-on/off comparison, PickXTimes improves from 32.0% to 100.0%, demonstrating verification's contribution when attempts frequently diverge from achievements. We discuss this task-dependent effect in Appendix C.2.

The threshold ablation in Table 3 further examines how verification decisions should be used. Removing the reversibility-dependent threshold distinction (flattened) reduces BinFill success from 84.0% to 74.0%. Strict acceptance can falsely reject a successful container placement, leaving the pointer on an already completed subgoal. We therefore account for the execution cost of false rejection when setting the acceptance thresholds.

AfternoonTeaServe complements these controlled results with hardware examples of corrective execution. The push recoveries reported in Point 1 show how verification keeps the controller on a subgoal that requires further action and allows subsequent attempts to complete it.

Reviewer_EbgWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Achievement-Grounded Memory (AGM) for frozen VLA policies. Its central insight is to ask what evidence justifies advancing to the next subgoal, rather than simply storing more history. AGM separates action generation, achievement verification, and progress tracking, using verified outcomes to advance, retain, or roll back a subgoal pointer. Experiments demonstrate improvements on repetitive pick-and-place tasks in simulation and on a real robot.

Critical strengths
  1. The focus on when to update memory is insightful: attempted actions should not automatically count as completed progress.
  2. The framework clearly separates how to act, whether the action succeeded, and which subgoal comes next.
  3. Controlled experiments provide strong evidence for verified updates on PickXTimes, complemented by real-robot validation.
  4. The manuscript is clearly written, with detailed implementation descriptions and explicit limitations.
Critical weaknesses
  1. Restricted task structure. A fixed subgoal sequence and a single pointer cannot naturally handle branching, flexible ordering, or failures requiring replanning. Recovery is largely limited to retrying the associated grasp–place operation.
  2. Narrow verification coverage. Event triggers and success criteria are tailored to grasp–release interactions. Their applicability to pushing, articulated-object manipulation, or continuous tool use remains unclear, and AGM provides no improvement on SwingXTimes or StopCube.
  3. Unclear adaptation cost. A small verification head does not necessarily imply easy transfer. Hardware deployment requires verifier retraining and threshold recalibration, but the data and engineering effort needed for new tasks are not quantified.
Questions for authors
  1. How could the memory representation support conditional plans or recovery beyond returning to the associated grasp?
  2. Can AGM improve a task whose progress is not marked by grasp–release events, while retaining the same memory-update framework?
  3. Which verifier components can be reused for a new task, and how much additional annotation, training, and calibration are required?
2 comments
AuthorsOpenReview ↗

Extending AGM Beyond Grasp and Release: Recovery, Verification, and Adaptation Cost

We thank the reviewer for the thoughtful assessment and the specific questions about task structure, verification coverage, and adaptation cost. These questions helped us extend the evaluation and clarify which components can be reused across tasks and platforms.

We have added a new real-robot experiment, AfternoonTeaServe, using a bimanual AgileX Nero robot to serve items such as coffee and cake in the instructed order. Each item requires five subgoals: pick up with one arm, put down to the right of a red line, push onto the target with an empty gripper, pick up from the target with the other arm, and place on the tray. The robot presses a button after serving all instructed items. Evaluation instructions contain 1–4 items, corresponding to 6–21 subgoals.

The push is non-prehensile. If the item does not reach the target, AGM retains the push subgoal and allows another push. This experiment examines verification beyond grasp and release, recovery through corrective pushing, and adaptation to a second robot platform. The task setup and results appear in the experimental section of the revised manuscript (Section 4.6), with implementation details in Appendix E.1–E.5 and adaptation details in Appendix E.6.

1. Conditional plans and recovery beyond the associated grasp (W1, Q1).

AGM selects a recovery action according to the subgoal type and its recovery rule. In the submitted version, a rejected grasp retains the current subgoal, while a rejected recoverable placement rolls back to the associated grasp. AfternoonTeaServe adds recovery through corrective pushing: when an item has not reached the target after a push, the pointer stays on the push subgoal, and the policy pushes again based on the current observation.

Across the 23 analyzed AGM episodes, 13 of 42 push subgoals were rejected at least once, and 7 subsequently passed verification after further attempts. We confirmed 5 corrective pushes among these recoveries through video inspection. The remaining 6 reached the rejection limit, mostly after overshooting the target, and the episodes terminated. These records demonstrate recovery by retaining the current subgoal and continuing the manipulation.

Your question about conditional plans also motivates a concrete extension of the memory representation. The ordered sequence can be extended to a task graph with preconditions and branches, with progress represented by the set of verified completed nodes. The controller selects an executable node based on its dependencies and the current observation, and verification determines whether that node is marked complete. A failure can activate a predefined recovery branch or invoke a planner to generate subsequent actions. The task graph selects execution and recovery paths, while the verifier confirms achieved progress. Appendix F discusses this task-graph extension as future work.

2. Verification beyond grasp and release (W2, Q2).

To address this question, we evaluated AGM on AfternoonTeaServe, where pushing an item onto the target is a verified subgoal. We trained the full-instruction and subgoal-conditioned policies on the same 100 demonstrations. The complete AGM system achieves 45.0% success (18/40), compared with 17.5% (7/40) for the full-instruction policy, with improvements at every evaluated sequence length.

Number of instructed items1234Overall
Full-instruction π0.540%20%10%0%17.5%
π0.5 + AGM60%60%40%20%45.0%

Gripper closure and reopening localize a push attempt, while the verifier uses frozen SigLIP visual features and the subgoal text to determine whether the item has reached the target. Acceptance advances the pointer; rejection retains the current subgoal and allows another push. Progress is therefore confirmed by the resulting object–target relation.

We retain AGM's memory representation and its verification-driven advance, retain, and rollback mechanism. The extension adds a push event, a type label, and a recovery rule, with the verification head retrained on the new platform's events. AfternoonTeaServe also terminates the episode and scores it as a failure after 10 consecutive rejections. Across the 23 analyzed episodes, all 192 pointer advances were supported by verification.

The end-to-end comparison demonstrates the benefit of the complete AGM system on the new task, while the recovery records in Point 1 show how push verification contributes during execution. For SwingXTimes and StopCube, the corresponding extension would define motion- or timing-based event triggers and achievement evidence for the same memory-update framework.

AuthorsOpenReview ↗

Extending AGM Beyond Grasp and Release: Recovery, Verification, and Adaptation Cost

3. Component reuse and adaptation cost (W3, Q3).

Thank you for asking us to quantify the adaptation effort. The PiPER and Nero deployments provide concrete examples of component reuse and the additional data, training, and calibration involved.

Reusable components include the frozen vision-language encoder and point tracker, the core gripper state-machine logic, and the verification head's MLP architecture and training procedure. Adaptation involves refitting the head weights, calibrating gripper thresholds and scene regions, and defining event triggers, achievement criteria, and recovery rules for new interaction types.

ComponentPiPERNero / AfternoonTeaServe
Existing policy demonstrations194 episodes100 episodes
Automatically extracted positive events526 placement events1,086 placement and push events
Additional failure data75 failure events from 20 dedicated episodes156 failure events from 18 dedicated episodes, supplemented by failed push attempts preceding retries in the demonstrations
Failure-data collection and reviewApproximately 2 hours in totalApproximately 2 hours of collection and 1 hour of event review
Verification-head trainingThe lightweight MLP head converges quickly on cached featuresFeature extraction, held-out evaluation, and head fitting take approximately 5 minutes on one GPU in total

For PiPER, the verification-head training pool contains all 75 failure events and 175 subsampled positive events. Positive events are extracted automatically from existing policy demonstrations, with additional labeling concentrated on reviewing failure events. Gripper calibration uses a scripted probe of three configurations: fully open, empty closure, and holding an object. The bin, target, and button regions are also registered, and pixel-based tracking parameters are adjusted for the camera geometry.

The Nero deployment adds bimanual demonstration-segmentation logic, a push trigger, and corresponding subgoal templates, with calibration for both grippers and the scene regions. All 100 demonstrations are segmented automatically and pass sequence-consistency checks, without manual frame-level annotation.

The subgoal definitions, recovery rules, and trained verification head are reused across instructions and sequence lengths. The adaptation-cost table in the revised manuscript summarizes the data, training, and engineering requirements of both deployments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.