RAAG: Training-Free Retrieval-Augmented Action Generation Guided by Gripper–Object Geometry from Successful Experience

Vision-language-action (VLA) policies generate actions from current observations, which can be brittle under distribution shift due to the lack of reliable guidance about future interaction outcomes. World models provide such foresight by predicting future states, but add training or test-time cost. Retrieval offers a training-free alternative, yet memory-based methods often reuse recorded actions tied to the original scene geometry, while geometry-aware alternatives typically require heavier 3D perception. We observe that successful trajectories contain two distinct signals: a scene-specific motion path and the relative gripper–object geometry that defines successful interaction. The path may become invalid after scene and position changes, while the relative geometry can remain transferable and serve as a future target even in unseen cases. Based on this insight, we propose RAAG, a training-free framework that augments frozen VLAs with future targets retrieved from successful memories. RAAG retrieves geometrically compatible experience using gripper–object relations, retargets successful relations to the current scene, and gates guidance by policy risk and retrieval reliability. The frozen VLA still determines how to reach these targets, avoiding brittle action reuse. Across five benchmarks, three simulators, and a real robot, RAAG substantially improves frozen VLAs under distribution shifts, raising OpenVLA-OFT's success rate from 24.0% to 69.6% on ManiSkill3 OOD. These gains come with modest inference overhead of 38.6% additional latency on a single GPU and only 3.0 ms on dual GPUs, where retrieval runs in parallel with VLA inference.