acceptodds
Under review as a conference paper at ICLR 2027

RAAG: Training-Free Retrieval-Augmented Action Generation Guided by Gripper–Object Geometry from Successful Experience

Abstract

Vision-language-action (VLA) policies generate actions from current observations, which can be brittle under distribution shift due to the lack of reliable guidance about future interaction outcomes. World models provide such foresight by predicting future states, but add training or test-time cost. Retrieval offers a training-free alternative, yet memory-based methods often reuse recorded actions tied to the original scene geometry, while geometry-aware alternatives typically require heavier 3D perception. We observe that successful trajectories contain two distinct signals: a scene-specific motion path and the relative gripper–object geometry that defines successful interaction. The path may become invalid after scene and position changes, while the relative geometry can remain transferable and serve as a future target even in unseen cases. Based on this insight, we propose RAAG, a training-free framework that augments frozen VLAs with future targets retrieved from successful memories. RAAG retrieves geometrically compatible experience using gripper–object relations, retargets successful relations to the current scene, and gates guidance by policy risk and retrieval reliability. The frozen VLA still determines how to reach these targets, avoiding brittle action reuse. Across five benchmarks, three simulators, and a real robot, RAAG substantially improves frozen VLAs under distribution shifts, raising OpenVLA-OFT's success rate from 24.0% to 69.6% on ManiSkill3 OOD. These gains come with modest inference overhead of 38.6% additional latency on a single GPU and only 3.0 ms on dual GPUs, where retrieval runs in parallel with VLA inference.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-language-action models, Robotic manipulation, Geometry-based retrieval, Training-free guidance

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_mPGxOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a training-free method that helps existing VLA policies perform better by learning from successful past experiences without changing the policy itself. Instead of directly replaying recorded actions, RAAG retrieves useful gripper–object relationships from past demonstrations and uses them to guide the policy toward successful interactions in new scenes. It combines geometry-based retrieval, action guidance, and confidence gating to decide when and how strongly to use the retrieved experience. The authors evaluate RAAG on several simulation benchmarks, including LIBERO and ManiSkill3, as well as a real robot. The results show consistent improvements under distribution shifts. For example, RAAG increases OpenVLA-OFT’s success rate on ManiSkill3 OOD from 24% to 69.6%, and improves real-robot success on an unseen white plate from 10% to 56.7%. Overall, the paper shows that reusing successful interaction geometry can be an effective way to improve frozen VLA policies without additional training, although the efficiency gains depend on the policy architecture and hardware setup.

Critical strengths
  1. Effective transfer of interaction geometry while retaining the frozen policy’s action prior. RAAG retrieves and retargets gripper–object relations rather than replaying scene-dependent actions. Table 3 shows that on ManiSkill3 OOD, full RAAG reaches 69.6%, compared with 21.7% for geometric action replay and 25.5% for visual servoing; appearance-only retrieval with guidance reaches 46.2%. These results support the practical value of combining geometric compatibility with policy-mediated execution.

  2. Large improvements across policy competence levels and validation settings. Table 2 reports ManiSkill3 OOD gains from 24.0% to 69.6% for the base discrete policy, 57.5% to 80.5% for GRPO, and 75.9% to 90.0% for PPO. The paper also evaluates off-trajectory failure states and a real robot; Table 6 reports improvement from 10.0% to 56.7% on the unseen white plate using the same frozen checkpoint and demonstration collection. These results make the system-level contribution meaningful.

Critical weaknesses

The ablations do not fully isolate the contribution of online relational retrieval from the rest of the guidance pipeline. A more controlled comparison would help clarify how much the proposed retrieval mechanism itself contributes to the gains.Recent work such as Retrieve-then-Steer explores a similar idea of using successful experience to guide frozen VLA policies at inference time. RAAG differs in its use of transferable gripper–object geometry, but the paper could better explain what this adds beyond existing retrieval-based guidance methods. Some experimental details, including the predefined subtask order in LIBERO-Long and the updated simulator calibration in LIBERO-Plus, could also be clarified.

Questions for authors
  1. Could you provide concise pseudocode for the full execution loop, including instruction-to-mask grounding, switching the reference from the grasped object to the receptacle, maintaining stage identity across repeated retrieval, and handling an unsuccessful close/open event? How is the manually specified LIBERO-Long subtask order used at test time, and what equivalent progress information is available to the baselines?

  2. For LIBERO-Plus camera perturbations, does RAAG receive exact updated intrinsics/extrinsics from the simulator? Which baselines receive the same information? Please distinguish transfer with known recalibration from robustness when calibration is unavailable or inaccurate.

  3. Can you compare full RAAG against a task- and stage-compatible fixed prototype endpoint from the same successful memory while keeping perception, stage logic, gating, guidance, and execution frequency unchanged? This would more directly test the value of online relational retrieval.

  4. For the flow-policy extension, can you separate geometric reranking from the effects of extra sampling and shorter execution prefixes, e.g. with a native-policy short-horizon control and an equal-candidate-budget control on one existing setting? These questions concern material ambiguities in the current claims, rather than requests for a new research direction.

6 comments
AuthorsOpenReview ↗

(1/3) Response to Reviewer mPGx

We appreciate your valuable comments for identifying details needed to assess the contribution and have addressed your concerns as follows.

Additional Ablation for Retrieval (W1 / Q3)

To further examine which information matters for retrieval, we extend Table 3(b) with two fixed prototypes.

Relational Fixed-Prototype (e) selects a geometric medoid from the task bank, minimizing summed differences in endpoint relative positions and approach orientation. It uses relation geometry without shape matching. Shape-oriented Fixed-Prototype (f) matches the task and target object name, using object identity as a shape proxy. Both prototypes keep the selected trajectory fixed throughout execution.

ExperimentMemory selectionTask / stageObject ShapeGripper-object relation matchingFailureBenchOfficial OOD
Table 3(b)(a) Full GR✓✓✓39.169.6
Table 3(b)(b) Fixed across verbs✗✗✗3.89.5
Table 3(b)(c) Random within the task bank✓✗✗15.829.9
Table 3(b)(d) Appearance only✓✓✗25.546.2
New(e) Relational Fixed-Prototype✓✗✓27.846.4
New(f) Shape-oriented Fixed-Prototype✓✓✗35.464.8

(c) ignores shape and interaction geometry, risking unsuitable grasps. Conversely, (e) provides representative endpoint geometry but omits shape-dependent requirements: differences in object width or elongation can require different contact positions and grasp orientations.

(f) highlights the importance of shape compatibility, but exact object matching requires memory coverage of all test objects (for unseen objects, we use the closest available object memory). Moreover, the object may be positioned or oriented differently from the memory, so its fixed grasp may still be unsuitable. Full GR therefore combines shape and current gripper–object relations to match both the grasp pose and approach direction.

Notably, (f) still transfers the retrieved gripper–object relations into action guidance. This geometric retargeting is essential, as visual matching and hiostorical action cannot ensure that recorded motions remain valid in the current scene (Table 3(c)).

Relation to Retrieve-then-Steer (W2)

There are two key differences between Retrieve-then-Steer (RtS) and RAAG.

1. Retrieval. RtS retrieves by visual-embedding similarity, while RAAG matches shape and gripper-object relations. As shown in the Table above, (d) appearance-based selection has two limitations: similar-looking observations need not have compatible shapes or interaction geometry, while objects with familiar shapes but new colors or textures may look dissimilar despite sharing useful grasp geometry (Fig. 1(c)). The retrieval controls above motivate explicitly matching shape and relations.

2. Action guidance. The RtS-style control initializes the sampler with retrieved actions, which the VLA refines using current observations. However, the recorded motion may be unsuitable when the arm state or object pose differs from memory. RAAG instead transfers gripper–object geometry, allowing motion to adapt to the current scene while preserving the desired grasp relation.

To test whether retrieved geometry provides better generation guidance than retrieved actions, we compare RtS-style injection of its recorded actions and RAAG guidance toward its grasp/release endpoint using frozen pi0.5.

GuidanceOverall success (%)
Original frozen policy85.7
RtS-style action guidance, best tested strength88.6
RAAG sampling91.3

The RtS-style result uses the best tested injection strengths (t0=0.3t_0=0.3). RAAG achieves the best success rate of 91.3%, exceeding the frozen policy by 5.6 points and the best tested RtS-style guidance by 2.7 points. This supports our hypothesis that retrieved experience provides better generation guidance through gripper–object geometric goals than through stored action priors in this setting.

AuthorsOpenReview ↗

(2/3) Response to Reviewer mPGx

Execution Loop (Q1)

1. Retrieve memory: keep successful trajectories whose task label matches the current task.

2. Read the stages from the memory's gripper events: e.g., APPROACH A with an open gripper -> CLOSE -> CARRY to B -> OPEN.

3. Instruction-to-mask grounding: Locate the task objects and receptacles with SAM 3:
   use their names as text prompts on the first image.
   retain the selected mask identities and track them in subsequent images.

4. For each object A and its required receptacle B, in task order:
   stage <- APPROACH
   repeat until this gripper events pair finishes:
       read the new image, update tracked masks, and read the robot state.
       if stage is APPROACH or GRASP: target <- A and its mask.
       if stage is CARRY or RELEASE: target <- B and its mask.

       estimate the gripper-target relation and search only same-task memories for this subtask and the current stage.
       use its stage endpoint relative to the current target and guide the VLA.

       if APPROACH no longer makes predicted progress toward its endpoint:
           stage <- GRASP.
       if GRASP executes CLOSE:
           if finger opening indicates an empty grasp*: reopen; stage <- APPROACH.
           otherwise: stage <- CARRY, switching the target from A to B.
       if the carry-localization stage finishes without reporting a failure:
           stage <- RELEASE.
       if RELEASE executes OPEN:
           if perceived placement fails*: stage <- APPROACH for the same object.
           otherwise: complete this pair and select the next pair.

       if a stage times out without its CLOSE/OPEN event:
           retry approach/grasp or carry/release within the remaining budgets.

To detect failures, the empty-grasp and placement checks use measured finger opening to assess grasp status.

RAAG derives progress cues from successful memories to align retrieval with the current execution stage. Visual VLA baselines already successfully capture and handle task progress from observations, providing extra progress cues can hardly improve their performance. Our bookkeeping synchronizes memory use without supplying privileged test-time stage labels without leading to unfair comparison with baselines.

For LIBERO-Long, we previously specified the subtask order manually. The updated implementation supports selecting the nearest unfinished target using calibrated gripper–object distances, allowing execution order to adapt to the current scene.

Sampling, Reranking, and Execution Frequency (Q4)

We summarize the most challenging Cam/Rob/Lang results in LIBERO-PLUS with 240 episodes, 80 per factor below. All settings follow the official episode limits.

Horizon is the maximum number of executed actions before replanning; Samples counts candidate action chunks; Geometric selection picks the candidate predicted to finish closest to that goal.

ArmHorizonSamplesGeometric rerank and selectionSuccess (%)
(a) Frozen π0.5\pi_{0.5}, native execution101–79.6
(b) Frozen π0.5\pi_{0.5}, native equal-candidate-budget control103 (random selection)–80.0
(c) Frozen π0.5\pi_{0.5}, native short-horizon control21–82.5
(d) Full RAAG10 for lozalization, 2 for grasp3✓85.8
(e) Full RAAG, short-horizon control23✓86.7

RAAG outperforms the short-horizon policy while retaining the efficiency. The frozen policy with horizon 2 (c) reaches 82.5%, whereas RAAG (d) achieves 85.8%, a 3.3-point improvement.

With the comparable efficiency, (d) also exceeds random selection (b) by 5.8 points, supporting geometry-based selection beyond additional sampling alone.

Further shortening RAAG’s horizon (e) adds only 0.9 points. Setting (d) therefore retains most of the performance while allowing longer action prefixes between policy queries.

AuthorsOpenReview ↗

(3/3) Response to Reviewer mPGx

Camera Self-Calibration (Q2)

A distinctive strength of RAAG is its ability to combine camera calibration, SAM 3 masks, and gripper proprioception into gripper–object geometry in the robot frame for retrieval and guidance. The evaluated image-based VLA baselines have no input interface for camera matrices, so supplying them alone would not change their predictions.

This capability depends on accurate calibration, while incorrect parameters can distort the estimated relations and misdirect action generation. We acknowledge this dependence in Sec. 4.4.

To mitigate this limitation, we additionally introduce robot-based self-calibration. The robot itself provides a moving reference whose position is known from its joint angles. Moving the camera does not change the gripper pose in the robot frame, so the mismatch between its predicted and observed image positions can be used to correct the camera extrinsics:

  1. Set anchors once offline. With a calibrated camera, fit a representative point fixed to the gripper (ManiSkill) or each visible robot part (LIBERO).
  2. Match what the robot knows to what the camera sees. During execution, forward kinematics gives each anchor's 3D position, while robot-part masks locate its observed position in the image. Accumulate these pairs during normal arm motion.
  3. Correct the camera and reuse the corrected geometry. We fit the camera pose from at least four FK/image-point pairs. We keep intrinsics fixed and update the extrinsics as new observations arrive, then use them to recompute gripper–object relations.

FK provides a camera-independent reference, allowing the calibration error to be corrected before it misleads retrieval and guidance. No calibration board or test-time exact camera pose is required. The code provides online self-calibration in recovery/calibration/pose.py and offline anchor fitting in recovery/calibration/anchors.py, integrated through adapters/scene_maniskill.py and adapters/scene_libero.py.

We perturb the camera parameters used for geometry while leaving the rendered observations unchanged in ManiSkill OOD. Camera-pose error is the test camera's displacement from the original training/memory viewpoint. Object displacement is the median offline localization error for incorrect calibration. Exact calibration uses the updated test-camera parameters; stale calibration retains the original parameters; self-calibration starts from the stale parameters. All entries are success rates (%).

Camera-pose errorObject displacementFrozen policyRAAG, exact calibrationRAAG, incorrect calibrationRAAG, incorrect calibration + self-calibration
00 cm24.069.6—68.4 (98%)
1° + 1 cm1.2 cm23.269.666.0 (95%)69.5 (99%)
2° + 2 cm2.4 cm24.269.649.6 (71%)69.1 (99%)
3° + 3 cm3.7 cm23.669.633.2 (48%)62.1 (89%)
5° + 5 cm6.1 cm22.969.619.1 (27%)59.4 (85%)

Accurate calibration matters, but RAAG does not require exact camera parameters at test time. At 5° + 5 cm error, self-calibration raises success from 19.1% to 59.4%, recovering 85% of exact-calibration performance (69.6%) and exceeding the frozen policy (22.9%) by 36.5 points. Robot observations can therefore correct inaccurate calibration and retain a substantial advantage over the frozen policy, which cannot use camera parameters.

Reviewer_GHmeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RAAG, a training-free framework that augments frozen vision-language-action policies with successful experience. Geometry-based Retrieval (GR) identifies compatible memories, Future-guided Generation (FG) retargets their endpoint gripper–object relations to the current scene, and Confidence Gating (CG) adjusts guidance strength. RAAG applies logit-bias guidance to discrete-token VLAs and sampling-based reranking to other policies. Experiments show substantial improvements under distribution shifts.

Critical strengths

The distinction between transferring successful interaction geometry and replaying scene-dependent actions is well motivated. RAAG provides a practical mechanism for incorporating this geometry without updating policy parameters. Its empirical evaluation is extensive, covering multiple policy backbones, distribution shifts, FailureBench, cross-policy transfer, and real-robot experiments. Additional experiments examine memory budgets, perception errors, and policy competence. The real-robot comparison also controls demonstration data and policy checkpoints, strengthening the evidence that retrieval guidance provides additional value.

Critical weaknesses
  1. The advantage over closely matched geometric guidance remains insufficiently isolated. The replay and visual-servoing baselines in Table 2 do not invoke VLA generation, whereas RAAG retains the frozen VLA. A baseline using the same perception, retrieved endpoints, and frozen backbone with simpler action fusion would better establish the specific benefit of FG.
  2. Sim-to-real transfer remains untested. The real-robot experiments use real demonstrations for both policy fine-tuning and memory construction. They demonstrate physical deployment, but do not establish whether simulation-derived gripper–object relations transfer to real observations and dynamics without collecting new real-world memories.
Questions for authors

Please see weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer GHme

We sincerely appreciate your valuable comments and have addressed your concerns as follows.

Comparison with Geometric Guidance Baselines (W1)

To further isolate the benefit of FG, we added baselines that retain the same inputs and frozen backbone, changing only how the retrieved endpoint influences localization. All rows share perception, memory, retrieval and the frozen OpenVLA-OFT checkpoint, with no extra training, learned correction, or discriminator. Here, aa is the policy translation, uu the clipped step toward the endpoint, and cc the shared retrieval-confidence weight.

Localization operatorClosest published formFailureBenchOfficial OOD
Our VLA logits guidance39.169.6
Hard switch: uu if c≥0.5c\geq0.5, otherwise aaShared-control arbitration (Dragan & Srinivasa, 2012)23.759.8
Bounded residual: a+cua+cuResidual control (Johannink et al., 2019)32.463.4
Minimal correction of insufficient endpoint-directed progressQP-based intervention (Ames et al., 2017)33.865.1
Endpoint-conditioned logit tilt, β=30\beta=30Decoding-time reweighting (Yang & Klein, 2021)37.267.1

Among the controls retaining the frozen VLA, RAAG achieves 39.1% / 69.6%, exceeding the other alternatives.

Bounded residual retains the original policy motion before adding a correction; projection preserves motion perpendicular to the endpoint direction; hard switching provides no geometric assistance below its threshold. Consequently, these methods can retain policy motion that conflicts with the endpoint, limiting recovery when the policy is unreliable, as on FailureBench. FG instead steers action selection in terms of VLM logits, toward the endpoint while retaining the learned policy prior.

Sim-to-Real Memory Transfer (W2)

We agree that the existing real-robot experiments demonstrate deployment with real-world memories, rather than transfer of simulation-derived memories.

We will conduct real-robot experiments with simulation-derived memories as soon as possible and report the results immediately upon completion.

Reviewer_GHmeOpenReview ↗

Official Comment by Reviewer GHme

Thank you for the clarification and the additional experiments, which have addressed some of my concerns. I will maintain the score.

AuthorsOpenReview ↗

Response to Reviewer GHme for Sim-to-Real Memory Transfer Experiment

Sim-to-Real Memory Transfer (W2)

Thank you for recognizing the practical value and completeness of our work. We sincerely appreciate your valuable suggestions and the time you have devoted to this review.

We have now completed additional real-robot experiments testing whether simulation-derived memories can improve real-world manipulation. OpenVLA-OFT was fine-tuned on 20 real-robot carrot-placement demonstrations and then frozen. All three conditions share this checkpoint and are evaluated for 20 trials each.

MethodSuccess (%)
VLA55
VLA + RAAG (sim mem.)70
VLA + RAAG (real mem.)75

Simulation memory improves success by 15 percentage points, approaching the performance obtained with real-world memory. These preliminary results support transferring useful interaction geometry from simulation without collecting new real-world memories or further updating the policy.

We have incorporated these results into Appx. E.1 of the updated manuscript. Fig. 5(b) shows the simulated grasp geometry used to define the real robot's grasp goal in Fig. 5(a). Although the camera viewpoints differ, calibrated camera parameters allow RAAG to express the retrieved geometry in the real robot's coordinate frame. Thus, matching viewpoints is not required for successful memory transfer in this setting.

These results support RAAG as a practical and reproducible way to improve robot generalization through accumulated experience, helping reduce the need to collect new demonstrations and retrain policies for every new scene.

Thank you once again for your thoughtful feedback and careful review. We wish you all the best.

Reviewer_um1nOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RAAG retrieves successful gripper–object relations and transfers their endpoints to guide a frozen VLA. Guidance biases action logits or reranks continuous proposals, with confidence gating intended to limit unreliable intervention. Evaluation covers standard manipulation, distribution shifts, failure recovery, and real-robot pick-and-place tasks.

Critical strengths
  1. Transferring interaction endpoints while retaining the policy’s motion prior is a useful approach to scene–memory mismatch. The reported improvements are substantial, including ManiSkill3 OOD gains from 24.0% to 69.6% for the base policy.
  2. The evaluation includes stronger policies, replay and servoing controls, real-robot trials, and analyses of gating and localization error. This breadth supports the potential usefulness of the approach.
Critical weaknesses
  1. The perception setup needs reconciliation. The paper describes SAM 3 masks, but recovery/base.py documents ground-truth segmentation in simulation and SAM on the real robot. The supplied package lacks the concrete environment adapters and configurations needed to determine what generated the reported results. This distinction directly affects the RGB-only robustness claim.
  2. The supplied guidance code does not clearly implement Eqs. 7–10. In localize_guide.py, guidance strength depends on a product of retrieval similarities; the same path can include memory-action blending and pure memory candidates. Without the evaluated configuration, it is difficult to attribute the gains to the stated endpoint guidance and policy-risk gating.
Questions for authors
  1. Which results use SAM 3 masks versus simulator segmentation? Please identify the actual perception implementation.
  2. Where are Eqs. 7–10 implemented, and which settings generated the main tables? Were memory-action candidates, blending, or direct servo corrections enabled?
6 comments
AuthorsOpenReview ↗

(1/2) Response to Reviewer um1n

We appreciate your valuable comments for pointing out the ambiguity in our perception setup.

The Utilization of SAM 3 (W1 / Q1)

For real-robot experiments, we directly use off-the-shelf SAM 3 without fine-tuning, achieving 100% recognition accuracy across manually inspected test instances. For simulation, the submitted results use GT segmentation for two reasons:

  1. Simulation visual gap. SAM 3 performs reliably on real-world images but struggles with certain simulated objects, particularly in LIBERO, where it may fail to identify the target (e.g., distinguishing ketchup from several visually similar cans).
  2. Isolating our contribution. Our focus is transferring successful gripper–object geometry to guide frozen VLAs, rather than improving segmentation. GT masks allow us to evaluate this contribution without confounding perception errors. Importantly, both simulation and real-world deployment estimate 3D geometry from masks and camera calibration, without directly accessing simulator GT object poses.

To further validate SAM 3's effectiveness in simulation and visually ambiguous scenarios, we conducted a data-efficient, lightweight fine-tuning study. We reuse RGB frames and GT masks from seven existing memory banks, splitting trajectories into training, validation, and test sets at a 7:1:2 ratio to prevent data leakage. To minimize training cost, we froze SAM 3's vision backbone and text encoder, updating only 3% of its parameters (24.8M/840.4M). The best checkpoint was selected at 1,500 training steps based on validation performance (~0.6 L40 GPU-hours). We evaluated fine-tuned SAM 3 on 1,419 object instances from 285 held-out trajectories.

Segmentation methodMean IoU ↑Recognition accuracy ↑Accuracy on LIBERO ↑
Grounding DINO + SAM 20.76879.6%--
Original SAM 30.84487.0%73.4%
Fine-tuned SAM 30.97999.9%99.8%

Fine-tuned SAM 3 achieves 99.9% accuracy and 0.979 IoU against GT masks, substantially outperforming the original SAM 3. On the challenging LIBERO benchmark, accuracy improves from 73.4% to 99.8%. These results demonstrate that lightweight fine-tuning enables SAM 3 to closely approximate simulator-provided segmentation, while off-the-shelf SAM 3 already provides reliable perception on our real robot without adaptation. Crucially, neither setting requires modifying RAAG's geometric retrieval or action-guidance mechanisms.

To further verify that high segmentation accuracy translates into task success, we are evaluating the complete RAAG pipeline using fine-tuned SAM 3 on OpenVLA-OFT-SFT, while keeping all other components unchanged.

Benchmarkw/o RAAGRAAG + GTRAAG + Fine-tuned SAM 3
ManiSkill3 OOD24.069.668.9
FailureBench13.139.138.9
LIBERO93.998.597.4

According to the table above, replacing GT segmentation with fine-tuned SAM 3 causes only a minor performance drop of 0.7–1.1 percentage points on ManiSkill3 OOD and LIBERO, and 0.2 points on FailureBench. Notably, RAAG retains 68.9% success on ManiSkill3 OOD (vs. 24.0% without RAAG) and 38.9% on FailureBench (vs. 13.1%), while achieving 97.4% on LIBERO. These results demonstrate that RAAG's substantial performance gains are preserved with RGB-based segmentation, rather than relying on oracle masks, confirming the robustness and practicality of our geometry-guided framework under realistic perception conditions.

AuthorsOpenReview ↗

(2/2) Response to Reviewer um1n

Different Confidence-Gating Implementations (W2)

We acknowledge the implementation mismatch identified by the reviewer and apologize for the ambiguity. The supplementary attachment was an earlier reduced minimal export, prepared while development continued until submission. It did not clearly distinguish the confidence measures and execution settings. We have supplied an updated implementation with explicit CG calculations, calibration utilities, environment adapters and configurations, and SAM 3 evaluation scripts.

For our previous implementation, in localize.py, the best-versus-second-best trajectory-distance margin provides a margin-based alternative estimator of retrieval confidence, serving the role of Eq. (9). In localize_guide.py, the product of the initial retrieval similarity and the current-frame similarity scales memory intervention and thereby adjusts reliance on the frozen policy, providing an alternative to Eq. (8). Both scores remain retrieval-derived heuristics; the updated implementation explicitly separates policy-risk estimation from retrieval reliability.

The updated recovery/stages/gate.py explicitly separates policy risk from retrieval reliability and combines them as β=βmax⁡criskcret\beta=\beta_{\max}c_{\mathrm{risk}}c_{\mathrm{ret}}. We agree that CG remains imperfect: calibration mitigates uncertainty but does not guarantee that the gate identifies when assistance is useful (Appx. H).

We provide the implementation comparison using single-rollout success rates (%) below.

ImplementationOfficial OODFailureBench
Earlier supplementary implementation61.729.8
Explicit Eqs. (7)–(10), logit guidance69.639.1

The updated gate checks both whether the policy needs help and whether the retrieved memory is reliable, making intervention more selective than the earlier retrieval-only heuristic.

Endpoint Guidance, Memory Actions, and Servoing (Q2)

Memory-action candidates, memory–VLA action blending, and direct servo corrections are all disabled for main results and in the configurations. These branches are retained in explicitly selected compatibility configurations to reproduce earlier experimental variants and support additional comparisons of memory reuse and endpoint guidance.

Thank you for the detailed clarification and additional experiments. The end-to-end results with fine-tuned SAM 3 substantially reduce my concern that the reported gains depend on oracle segmentation. The explanation of the earlier code export and the disabled memory-action branches also addresses an important part of my implementation concern.

Before finalizing my assessment, I would appreciate clarification on two points:

  1. Were the original manuscript’s 69.6% / 39.1% results generated using the explicit Eqs. (7)–(10) implementation, with the supplementary package containing an earlier version, or were these results obtained through reruns during rebuttal? Please identify the implementation and configuration corresponding to the original results.
  2. Please confirm that SAM 3 fine-tuning and checkpoint selection used only in-distribution memory data, excluding benchmark evaluation trajectories and OOD evaluation data. The revised paper should also clearly distinguish training-free retrieval and guidance from the additional segmentation fine-tuning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.