- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AGM, a lightweight memory framework for frozen VLA policies in long-horizon repetitive manipulation. Instead of advancing task progress after every attempted action, AGM updates a compact subgoal pointer only when physical achievement is verified. Grasp success is checked using gripper events and point tracking, while placement success is verified from cross-view visual features and a small learned head. The framework supports advance, retry, and rollback without modifying the VLA policy. Experiments on RoboMME Counting and a real robot show large gains on PickXTimes and BinFill.
Critical strengths
1.The paper identifies a simple but important failure mode: recording attempted actions as completed progress can permanently corrupt task state. The distinction between “attempt” and “achievement” is intuitive and well motivated.
2.The controlled comparison between verified and attempt-based memory is particularly convincing. It directly isolates the value of grounded memory writing rather than simply comparing against larger memory architectures.
3.The empirical results are strong on the supported tasks. AGM reaches 100% on PickXTimes and 84% on BinFill, and the same qualitative behavior transfers to a physical robot.
4.The system is lightweight relative to external-VLM baselines. Most perception modules remain frozen, with only a small verification head trained.
Critical weaknesses
1.The current scope is narrow. AGM only verifies subgoals tied to grasp and release events, and therefore does not improve tasks involving other interaction types such as pushing, sliding, or articulated manipulation.
2.The method relies on substantial task-specific structure, including a predefined subgoal sequence, manually specified subgoal types, recoverability rules, state-machine thresholds, and several verification heuristics. This limits the generality of the claimed memory principle.
3.The placement verifier is not uniformly reliable. In particular, the irreversible-placement classifier performs poorly on held-out failures, and the system compensates with a very permissive threshold and task-specific transition logic. This weakens the claim that progress updates are consistently grounded in reliable verification.
4.The strongest gains occur on only two of the four RoboMME tasks; on the other two, AGM simply falls back to the frozen policy. The reported four-task average therefore partially reflects unchanged performance outside the method’s supported event vocabulary.
5.The real-robot validation is useful but limited to two task families, one robot platform, and one gripper configuration. In addition, the policies and verification head are retrained for the physical platform, so the hardware results do not fully demonstrate zero-adaptation transfer.
Questions for authors
-
How much of AGM’s performance depends on the manually specified subgoal types, recoverability rules, and transition logic, rather than on achievement verification itself?
-
How should the poor held-out discrimination for irreversible placement be reconciled with the claim that memory updates are physically verified?
-
Can the authors clarify how much of the four-task average improvement comes specifically from the two supported gripper-anchored tasks, given that AGM leaves the other tasks unchanged?
-
To what extent can the same verification and transition framework be reused for tasks whose completion is not associated with grasp or release events?
2 comments
AGM: Component Contributions, Verification Reliability, and Cross-Platform Reuse
4. Framework reuse beyond grasp and release and across platforms (W1, W5, Q4).
AfternoonTeaServe addresses these points together: a push completes without a grasp or a release, on a second robot and with everyday objects. AGM's memory and verification-driven transition rule are reused unchanged; the extension adds a push event, a type label, and a recovery rule.
| Number of instructed items | 1 | 2 | 3 | 4 | Overall |
|---|---|---|---|---|---|
| Full-instruction π0.5 | 40% | 20% | 10% | 0% | 17.5% |
| π0.5 + AGM | 60% | 60% | 40% | 20% | 45.0% |
With both policies trained on the same demonstrations, AGM succeeds in 18/40 trials, compared with 7/40 for the full-instruction policy. Verification acts during execution: 13 of 42 push subgoals were rejected at least once, 7 subsequently passed verification after further attempts, and video inspection confirmed 5 corrective pushes.
Across platforms, we reuse the frozen vision-language encoder and point tracker, the core gripper state-machine logic, and the MLP architecture and training procedure. For each platform, we refit the policy and verification head and recalibrate gripper thresholds and scene regions; the Nero deployment additionally introduces bimanual segmentation logic and the push trigger. Beyond the existing policy demonstrations, this required approximately 2 hours of failure-data collection and 1 hour of event review (Table 20).
The physical evaluation now covers three task families, two robot platforms, two gripper configurations, and both cubes and everyday objects.
AGM: Component Contributions, Verification Reliability, and Cross-Platform Reuse
We thank the reviewer for the thoughtful assessment and for recognizing the distinction between attempts and achievements, the controlled comparison, and the real-robot results. Your questions helped us make the contribution of each component more explicit and extend the evaluation to a new interaction type and robot platform.
We have added a new real-robot experiment, AfternoonTeaServe, using a bimanual AgileX Nero robot to serve items such as coffee and cake in the instructed order. Each item requires five subgoals: pick up with one arm, put down to the right of a red line, push onto the target with an empty gripper, pick up from the target with the other arm, and place on the tray. The robot presses a button after serving all instructed items. Evaluation instructions contain 1–4 items, corresponding to 6–21 subgoals.
The push is non-prehensile. If the item does not reach the target, AGM retains the push subgoal and allows another push. This experiment examines verification beyond grasp and release, recovery through corrective pushing, and adaptation to a second robot platform. The task setup and results appear in the experimental section of the revised manuscript (Section 4.6), with implementation details in Appendix E.1–E.5 and adaptation details in Appendix E.6.
1. Contributions of task structure and achievement verification (W2, Q1).
Thank you for raising this attribution question. The controlled comparison answers it directly: the policy, subgoal sequence, and progress pointer are held fixed, and only grasp and placement verification is switched on or off (Appendix C.2, Table 11).
On PickXTimes, success is 32.0% without verification and 100.0% with it: with the task structure already provided, verification alone accounts for 68 percentage points. On BinFill, the results are 82.0% and 84.0%; the provided structure already yields high success there, and verification adds a smaller gain.
The structure itself does not have to be written by hand for each task. With verification retained, replacing the benchmark-provided decomposition with a parser that reads only the instruction, without environment state or task ground truth, changes success from 100.0% and 84.0% to 98.0% and 82.0%. The type labels and recovery rules are defined once per interaction type and reused across instructions and repetition counts; the revised manuscript lists them in Table 17.
2. Reliability of irreversible-placement verification (W3, Q2).
Thank you for raising this concern. We agree that the irreversible-placement head has limited failure discrimination: among 28 held-out events, only 4 are failures, and none is detected at argmax. However, AGM does not equate an interaction event with successful achievement. The event only triggers verification, while the memory update depends on additional execution evidence.
For irreversible placement, this evidence includes a preceding verified grasp, a loaded-release event, and a visual verification score (Appendix A.5). On hardware, we additionally check whether the release occurs within the bin region. The visual head acts as a low-score veto rather than a standalone success classifier. Indeed, it rejected five updates in one BinFill run. We acknowledge that these cues do not guarantee successful placement.
The permissive threshold (0.05) reflects the asymmetric cost of errors: falsely rejecting a successful irreversible placement may cause an extra object to be inserted. Using a stricter, shared threshold reduces BinFill success from 84.0% to 74.0% (Table 3). Moreover, verification adds only 2 percentage points over attempt-based progression on BinFill (82.0% vs. 84.0%), unlike the 68-point gain on PickXTimes (32.0% vs. 100.0%).
Thus, AGM is event-triggered in when to verify and evidence-dependent in whether to update. We do not claim equally reliable achievement discrimination across all interaction types. The BinFill results reflect weaker outcome evidence and a risk-sensitive update rule, rather than strong irreversible-placement verification.
3. Source of the four-task average improvement (W4, Q3).
Thank you for requesting this breakdown. The entire four-task average improvement over the frozen π0.5 entries in Table 1 comes from PickXTimes and BinFill: (57.11 + 54.00 + 0 + 0) / 4 = 27.78 percentage points. On SwingXTimes and StopCube, performance falls back to the frozen policy, as marked by † in the table.
On the supported pair, AGM achieves 92.00%, exceeding the strongest baseline on that pair, SimpleSG + Qwen3-VL-4B, by 5.55 percentage points, and MemER by 24.00 percentage points.