- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a training-free method that helps existing VLA policies perform better by learning from successful past experiences without changing the policy itself. Instead of directly replaying recorded actions, RAAG retrieves useful gripper–object relationships from past demonstrations and uses them to guide the policy toward successful interactions in new scenes. It combines geometry-based retrieval, action guidance, and confidence gating to decide when and how strongly to use the retrieved experience. The authors evaluate RAAG on several simulation benchmarks, including LIBERO and ManiSkill3, as well as a real robot. The results show consistent improvements under distribution shifts. For example, RAAG increases OpenVLA-OFT’s success rate on ManiSkill3 OOD from 24% to 69.6%, and improves real-robot success on an unseen white plate from 10% to 56.7%. Overall, the paper shows that reusing successful interaction geometry can be an effective way to improve frozen VLA policies without additional training, although the efficiency gains depend on the policy architecture and hardware setup.
Critical strengths
-
Effective transfer of interaction geometry while retaining the frozen policy’s action prior. RAAG retrieves and retargets gripper–object relations rather than replaying scene-dependent actions. Table 3 shows that on ManiSkill3 OOD, full RAAG reaches 69.6%, compared with 21.7% for geometric action replay and 25.5% for visual servoing; appearance-only retrieval with guidance reaches 46.2%. These results support the practical value of combining geometric compatibility with policy-mediated execution.
-
Large improvements across policy competence levels and validation settings. Table 2 reports ManiSkill3 OOD gains from 24.0% to 69.6% for the base discrete policy, 57.5% to 80.5% for GRPO, and 75.9% to 90.0% for PPO. The paper also evaluates off-trajectory failure states and a real robot; Table 6 reports improvement from 10.0% to 56.7% on the unseen white plate using the same frozen checkpoint and demonstration collection. These results make the system-level contribution meaningful.
Critical weaknesses
The ablations do not fully isolate the contribution of online relational retrieval from the rest of the guidance pipeline. A more controlled comparison would help clarify how much the proposed retrieval mechanism itself contributes to the gains.Recent work such as Retrieve-then-Steer explores a similar idea of using successful experience to guide frozen VLA policies at inference time. RAAG differs in its use of transferable gripper–object geometry, but the paper could better explain what this adds beyond existing retrieval-based guidance methods. Some experimental details, including the predefined subtask order in LIBERO-Long and the updated simulator calibration in LIBERO-Plus, could also be clarified.
Questions for authors
-
Could you provide concise pseudocode for the full execution loop, including instruction-to-mask grounding, switching the reference from the grasped object to the receptacle, maintaining stage identity across repeated retrieval, and handling an unsuccessful close/open event? How is the manually specified LIBERO-Long subtask order used at test time, and what equivalent progress information is available to the baselines?
-
For LIBERO-Plus camera perturbations, does RAAG receive exact updated intrinsics/extrinsics from the simulator? Which baselines receive the same information? Please distinguish transfer with known recalibration from robustness when calibration is unavailable or inaccurate.
-
Can you compare full RAAG against a task- and stage-compatible fixed prototype endpoint from the same successful memory while keeping perception, stage logic, gating, guidance, and execution frequency unchanged? This would more directly test the value of online relational retrieval.
-
For the flow-policy extension, can you separate geometric reranking from the effects of extra sampling and shorter execution prefixes, e.g. with a native-policy short-horizon control and an equal-candidate-budget control on one existing setting? These questions concern material ambiguities in the current claims, rather than requests for a new research direction.
6 comments
(2/3) Response to Reviewer mPGx
Execution Loop (Q1)
1. Retrieve memory: keep successful trajectories whose task label matches the current task.
2. Read the stages from the memory's gripper events: e.g., APPROACH A with an open gripper -> CLOSE -> CARRY to B -> OPEN.
3. Instruction-to-mask grounding: Locate the task objects and receptacles with SAM 3:
use their names as text prompts on the first image.
retain the selected mask identities and track them in subsequent images.
4. For each object A and its required receptacle B, in task order:
stage <- APPROACH
repeat until this gripper events pair finishes:
read the new image, update tracked masks, and read the robot state.
if stage is APPROACH or GRASP: target <- A and its mask.
if stage is CARRY or RELEASE: target <- B and its mask.
estimate the gripper-target relation and search only same-task memories for this subtask and the current stage.
use its stage endpoint relative to the current target and guide the VLA.
if APPROACH no longer makes predicted progress toward its endpoint:
stage <- GRASP.
if GRASP executes CLOSE:
if finger opening indicates an empty grasp*: reopen; stage <- APPROACH.
otherwise: stage <- CARRY, switching the target from A to B.
if the carry-localization stage finishes without reporting a failure:
stage <- RELEASE.
if RELEASE executes OPEN:
if perceived placement fails*: stage <- APPROACH for the same object.
otherwise: complete this pair and select the next pair.
if a stage times out without its CLOSE/OPEN event:
retry approach/grasp or carry/release within the remaining budgets.
To detect failures, the empty-grasp and placement checks use measured finger opening to assess grasp status.
RAAG derives progress cues from successful memories to align retrieval with the current execution stage. Visual VLA baselines already successfully capture and handle task progress from observations, providing extra progress cues can hardly improve their performance. Our bookkeeping synchronizes memory use without supplying privileged test-time stage labels without leading to unfair comparison with baselines.
For LIBERO-Long, we previously specified the subtask order manually. The updated implementation supports selecting the nearest unfinished target using calibrated gripper–object distances, allowing execution order to adapt to the current scene.
Sampling, Reranking, and Execution Frequency (Q4)
We summarize the most challenging Cam/Rob/Lang results in LIBERO-PLUS with 240 episodes, 80 per factor below. All settings follow the official episode limits.
Horizon is the maximum number of executed actions before replanning; Samples counts candidate action chunks; Geometric selection picks the candidate predicted to finish closest to that goal.
| Arm | Horizon | Samples | Geometric rerank and selection | Success (%) |
|---|---|---|---|---|
| (a) Frozen , native execution | 10 | 1 | – | 79.6 |
| (b) Frozen , native equal-candidate-budget control | 10 | 3 (random selection) | – | 80.0 |
| (c) Frozen , native short-horizon control | 2 | 1 | – | 82.5 |
| (d) Full RAAG | 10 for lozalization, 2 for grasp | 3 | ✓ | 85.8 |
| (e) Full RAAG, short-horizon control | 2 | 3 | ✓ | 86.7 |
RAAG outperforms the short-horizon policy while retaining the efficiency. The frozen policy with horizon 2 (c) reaches 82.5%, whereas RAAG (d) achieves 85.8%, a 3.3-point improvement.
With the comparable efficiency, (d) also exceeds random selection (b) by 5.8 points, supporting geometry-based selection beyond additional sampling alone.
Further shortening RAAG’s horizon (e) adds only 0.9 points. Setting (d) therefore retains most of the performance while allowing longer action prefixes between policy queries.
(3/3) Response to Reviewer mPGx
Camera Self-Calibration (Q2)
A distinctive strength of RAAG is its ability to combine camera calibration, SAM 3 masks, and gripper proprioception into gripper–object geometry in the robot frame for retrieval and guidance. The evaluated image-based VLA baselines have no input interface for camera matrices, so supplying them alone would not change their predictions.
This capability depends on accurate calibration, while incorrect parameters can distort the estimated relations and misdirect action generation. We acknowledge this dependence in Sec. 4.4.
To mitigate this limitation, we additionally introduce robot-based self-calibration. The robot itself provides a moving reference whose position is known from its joint angles. Moving the camera does not change the gripper pose in the robot frame, so the mismatch between its predicted and observed image positions can be used to correct the camera extrinsics:
- Set anchors once offline. With a calibrated camera, fit a representative point fixed to the gripper (ManiSkill) or each visible robot part (LIBERO).
- Match what the robot knows to what the camera sees. During execution, forward kinematics gives each anchor's 3D position, while robot-part masks locate its observed position in the image. Accumulate these pairs during normal arm motion.
- Correct the camera and reuse the corrected geometry. We fit the camera pose from at least four FK/image-point pairs. We keep intrinsics fixed and update the extrinsics as new observations arrive, then use them to recompute gripper–object relations.
FK provides a camera-independent reference, allowing the calibration error to be corrected before it misleads
retrieval and guidance. No calibration board or test-time exact camera pose is required.
The code provides online self-calibration in recovery/calibration/pose.py and offline anchor fitting in
recovery/calibration/anchors.py, integrated through adapters/scene_maniskill.py and adapters/scene_libero.py.
We perturb the camera parameters used for geometry while leaving the rendered observations unchanged in ManiSkill OOD. Camera-pose error is the test camera's displacement from the original training/memory viewpoint. Object displacement is the median offline localization error for incorrect calibration. Exact calibration uses the updated test-camera parameters; stale calibration retains the original parameters; self-calibration starts from the stale parameters. All entries are success rates (%).
| Camera-pose error | Object displacement | Frozen policy | RAAG, exact calibration | RAAG, incorrect calibration | RAAG, incorrect calibration + self-calibration |
|---|---|---|---|---|---|
| 0 | 0 cm | 24.0 | 69.6 | — | 68.4 (98%) |
| 1° + 1 cm | 1.2 cm | 23.2 | 69.6 | 66.0 (95%) | 69.5 (99%) |
| 2° + 2 cm | 2.4 cm | 24.2 | 69.6 | 49.6 (71%) | 69.1 (99%) |
| 3° + 3 cm | 3.7 cm | 23.6 | 69.6 | 33.2 (48%) | 62.1 (89%) |
| 5° + 5 cm | 6.1 cm | 22.9 | 69.6 | 19.1 (27%) | 59.4 (85%) |
Accurate calibration matters, but RAAG does not require exact camera parameters at test time. At 5° + 5 cm error, self-calibration raises success from 19.1% to 59.4%, recovering 85% of exact-calibration performance (69.6%) and exceeding the frozen policy (22.9%) by 36.5 points. Robot observations can therefore correct inaccurate calibration and retain a substantial advantage over the frozen policy, which cannot use camera parameters.
(1/3) Response to Reviewer mPGx
We appreciate your valuable comments for identifying details needed to assess the contribution and have addressed your concerns as follows.
Additional Ablation for Retrieval (W1 / Q3)
To further examine which information matters for retrieval, we extend Table 3(b) with two fixed prototypes.
Relational Fixed-Prototype (e) selects a geometric medoid from the task bank, minimizing summed differences in endpoint relative positions and approach orientation. It uses relation geometry without shape matching. Shape-oriented Fixed-Prototype (f) matches the task and target object name, using object identity as a shape proxy. Both prototypes keep the selected trajectory fixed throughout execution.
(c) ignores shape and interaction geometry, risking unsuitable grasps. Conversely, (e) provides representative endpoint geometry but omits shape-dependent requirements: differences in object width or elongation can require different contact positions and grasp orientations.
(f) highlights the importance of shape compatibility, but exact object matching requires memory coverage of all test objects (for unseen objects, we use the closest available object memory). Moreover, the object may be positioned or oriented differently from the memory, so its fixed grasp may still be unsuitable. Full GR therefore combines shape and current gripper–object relations to match both the grasp pose and approach direction.
Notably, (f) still transfers the retrieved gripper–object relations into action guidance. This geometric retargeting is essential, as visual matching and hiostorical action cannot ensure that recorded motions remain valid in the current scene (Table 3(c)).
Relation to Retrieve-then-Steer (W2)
There are two key differences between Retrieve-then-Steer (RtS) and RAAG.
1. Retrieval. RtS retrieves by visual-embedding similarity, while RAAG matches shape and gripper-object relations. As shown in the Table above, (d) appearance-based selection has two limitations: similar-looking observations need not have compatible shapes or interaction geometry, while objects with familiar shapes but new colors or textures may look dissimilar despite sharing useful grasp geometry (Fig. 1(c)). The retrieval controls above motivate explicitly matching shape and relations.
2. Action guidance. The RtS-style control initializes the sampler with retrieved actions, which the VLA refines using current observations. However, the recorded motion may be unsuitable when the arm state or object pose differs from memory. RAAG instead transfers gripper–object geometry, allowing motion to adapt to the current scene while preserving the desired grasp relation.
To test whether retrieved geometry provides better generation guidance than retrieved actions, we compare RtS-style injection of its recorded actions and RAAG guidance toward its grasp/release endpoint using frozen pi0.5.
The RtS-style result uses the best tested injection strengths (t0=0.3). RAAG achieves the best success rate of 91.3%, exceeding the frozen policy by 5.6 points and the best tested RtS-style guidance by 2.7 points. This supports our hypothesis that retrieved experience provides better generation guidance through gripper–object geometric goals than through stored action priors in this setting.