acceptodds
Under review as a conference paper at ICLR 2027

Retrieve to Act: Motion Primitive Graph Retrieval for Robust VLA Execution

Abstract

Vision-Language-Action (VLA) policies demonstrate remarkable semantic generalization, yet remain highly vulnerable to out-of-distribution (OOD) visual shifts during execution. While existing test-time methods attempt to suppress spurious visual cues, they fundamentally overlook the rich, uncorrupted kinematic priors embedded in the policy's own training experience. We introduce Retrieve to Act (R2A), a test-time scaling framework that formulates OOD robustness purely as a geometric motion primitive retrieval problem. R2A bypasses fragile visual features by structuring successful training demonstrations into an offline spatiotemporal motion primitive graph. During inference, a query-dependent Graph Neural Network dynamically localizes the robot's ongoing execution within this graph, retrieving structurally compatible future motion primitives. These retrieved geometric priors are then seamlessly harmonized with the frozen VLA's predicted actions via a learned, dimension-wise gating mechanism, adaptively rectifying erroneous spatial trajectories while preserving the policy's local reactivity. Extensive experiments across three state-of-the-art VLA architectures, GR00T N1.7, , and StableVLA, demonstrate that R2A consistently and significantly enhances robustness against diverse contextual perturbations. R2A achieves absolute success rate improvements of up to +12% in simulation benchmarks (LIBERO-Plus, RoboTwin2.0-Plus) and +40% in real-world deployments, without requiring any finetuning of the base policies.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Learning, Experience Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uKCEOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces "Retrieve to Act" (R2A), a method designed to enhance the robustness of Vision-Language-Action (VLA) models under OOD execution conditions. The authors organize successful demonstration segments into a spatiotemporal graph and retrieves suitable continuations using the robot's current predicted actions and recent motion history. It keeps the original VLA parameters fixed, corrects action outputs through a small fusion module, and adjusts the robot's initial posture before execution to improve performance under perturbations. Experiments across several models and benchmarks evaluate the resulting gains.

Critical strengths
  1. The problem is practically important and the method is clearly specified. By combining motion history, graph-based retrieval, and action blending, the paper offers a workable way to exploit demonstration experience at inference time.
  2. The experiments encompass various backbones, simulation platforms, and real-world robot tasks, demonstrating the effectiveness of the method.
Critical weaknesses
  1. On RoboTwin, R2A also changes the control frequency and action-processing rules: Appendix C.1, the baseline executes full 50-step chunks, while R2A requeries the policy every 10 steps after the gripper closes, and adds displacement compensation triggered by stagnation detection. Since chunk size is known to materially affect policy performance [1][2], such discrepancies inevitably lead to unfair comparisons.
  2. The contributions of initial-state correction and trajectory retrieval are not disentangled. Tab.1 and 5 suggest that most of the gain comes from initial-state correction alone: removing it drops the success rate from 87.5% to 49.9%. Without a "base VLA + identical initial-state correction" baseline, the independent contribution of trajectory retrieval cannot be isolated. The method should also be compared against simpler recovery strategies (a fixed initial state, or the initial state of the nearest demonstration) to establish that diffusion-based correction is necessary.
  3. The method depends on motions covered by existing demonstrations and lacks explicit scene geometry, limiting its handling of new obstacles or recovery requiring novel action sequences. Real-robot tests cover only two tasks and initial-state perturbations; broader real-world evaluation and cost analysis with larger demonstration libraries remain useful directions.
  4. There are unexplained discrepancies in configurations and results between the main text and the appendices (for instance, regarding connection configurations, training workflows, and success rates for RoboTwin/LIBERO-LONG) which require careful rectification.

[1] PACE: Phase-Aware Chunk Execution for Robot Policies with Action Chunking arXiv:2606.00537

[2] Mixture of Horizons in Action Chunking arXiv.2511.19433

Questions for authors

As noted above, the presence of several inconsistencies and some unfair comparisons within the paper directly affects my overall assessment. The value of the work could be enhanced if the author were to provide clarifications, make necessary corrections, and conduct more objective ablation studies.

7 comments
AuthorsOpenReview ↗

Response to Reviewer uKCE (1/2)

Thank you for the thorough review, especially your comments on the fairness of the RoboTwin comparison. We address each point below.

W1. RoboTwin comparison: chunk size and re-querying

Chunk size affects performance, and the original comparison confounded this effect with the contribution of R2A. We therefore reran RoboTwin2.0-Plus under two matched schedules. Both methods use the same chunk size throughout each rollout, and R2A no longer re-queries the policy after gripper changes; query frequency is therefore identical. R2A raises average success from 56.8% to 65.5% (+8.7 pp) with 50-step chunks, outperforming the baseline on six of seven axes, and from 59.3% to 67.1% (+7.8 pp) with 10-step chunks, outperforming it on all seven. Thus, the gains do not result from more frequent queries.

RoboTwin2.0-Plus success rates (%), same execution schedule for both methods.

ScheduleMethodCameraRobot initLanguageLightBackgroundNoiseLayoutAvg.
50-stepπ₀.₅57.534.271.755.567.060.951.156.8
50-step+ R2A59.172.671.664.068.967.754.665.5
10-stepπ₀.₅56.138.673.460.967.564.154.459.3
10-step+ R2A60.974.874.665.670.867.155.967.1

Displacement compensation is one component of R2A. Even without it, R2A improves the base policy from 57.4% to 63.1% with 50-step chunks and from 59.1% to 64.7% with 10-step chunks.

Effect of displacement compensation on RoboTwin2.0-Plus (average success rate, %). The first two rows come from one separate run, so the base differs slightly from the table above.

Method50-step10-step
π₀.₅57.459.1
+ R2A w/o displacement compensation63.164.7
+ R2A (from the table above)65.567.1

We updated Table 2, added Tables 17 and 18 to Appendix D.5, and clarified the protocol in Appendix C.1.

W2. Disentangling ISR from retrieval, and simpler recovery strategies

Our original experiments did not isolate the two contributions. We added a "base VLA + ISR" variant (R2A with ISR only). On the Robot Initial States axis, ISR accounts for most of the gain, raising GR00T N1.7 from 32.6% to 82.7%; the full method reaches 87.5%. Across all seven axes, retrieval and harmonization without ISR reach 77.2%, ISR alone reaches 70.1%, and the full method reaches 82.1%. These results indicate that the components are complementary.

Contribution of each component, GR00T N1.7 on LIBERO-10 (success rate, %).

ConfigurationRobot initAll axes
GR00T N1.732.662.9
R2A w/o ISR49.977.2
R2A only ISR82.770.1
R2A87.582.1

We also compared ISR with simpler recovery strategies. On LIBERO-10, fixed reset (81.4%), nearest-demonstration start (79.4%), ISR-MLP (82.4%), and ISR (82.7%) are within 3.3 pp; we therefore do not claim that diffusion is necessary in this setting. In Meta-World, which has two start regions, ISR reaches 90.3%, compared with 73.8% for the MLP. To isolate retrieval under the same initial-state correction, we applied an identical fixed reset across methods: GR00T N1.7 reaches 81.4%, SDN reaches 83.0%, and R2A's retrieval and harmonization reach 89.3%.

Start-state recovery (success rate, %). The two "Fixed reset +" rows apply the same fixed reset to SDN and to R2A's retrieval and harmonization.

SettingStrategySR
LIBERO-10 (GR00T N1.7)No recovery32.6
LIBERO-10 (GR00T N1.7)Fixed reset (mean start state)81.4
LIBERO-10 (GR00T N1.7)Nearest-demonstration start79.4
LIBERO-10 (GR00T N1.7)ISR-MLP82.4
LIBERO-10 (GR00T N1.7)ISR82.7
LIBERO-10 (GR00T N1.7)Fixed reset + SDN83.0
LIBERO-10 (GR00T N1.7)Fixed reset + R2A retrieval and harmonization89.3
Meta-World (Evo-1)No recovery35.6
Meta-World (Evo-1)ISR-MLP73.8
Meta-World (Evo-1)ISR90.3

These results appear in Table 5, Section 5.2(b), and Appendix D.4 (Table 16).

AuthorsOpenReview ↗

Response to Reviewer uKCE (2/2)

W3. Dependence on demonstrated motions, breadth of the real-world evaluation, and cost

We agree that R2A can retrieve only motions represented in the demonstration library and does not use explicit scene geometry. Our experiments do not establish that it can handle novel obstacles; we now state this limitation in Section 6. Our target setting is robustness to contextual perturbations while the task remains unchanged. In this setting, VLS, which builds reward functions from scene geometry, improves neither base policy, whereas R2A improves both.

R2A and VLS, which uses scene geometry, on LIBERO-10 (success rate, %).

MethodCameraRobot initLanguageLightBackgroundNoiseLayoutAvg.
π₀.₅50.673.388.895.693.475.589.180.9
+ VLS45.744.379.480.372.356.875.664.9
+ R2A55.196.991.995.695.278.091.386.3
GR00T N1.738.232.677.384.282.455.969.662.9
+ VLS38.130.966.877.973.250.264.157.3
+ R2A58.987.584.694.588.978.681.482.1

Hardware constraints limited the breadth of our real-world evaluation, so we prioritized perturbations that most strongly degrade the base policies in simulation. We also added a camera-viewpoint perturbation for Banana2Pot and evaluated a second platform, SO-101, with π₀.₅.

Real-world success rates (%), 20 trials per entry.

Platform, taskPolicyConditionBase+ R2A
ALOHA, Banana2PotGR00T N1.7No perturbation6080
ALOHA, Banana2PotGR00T N1.7Robot initial state3080
ALOHA, Banana2PotGR00T N1.7Camera viewpoint1060
ALOHA, Cube2DrawerGR00T N1.7No perturbation5585
ALOHA, Cube2DrawerGR00T N1.7Robot initial state1045
SO-101, Drawer Stackπ₀.₅No perturbation6580
SO-101, Drawer Stackπ₀.₅Robot initial state2580

To assess computational cost, we measured storage requirements and query overhead as the demonstration library grows.

Cost of the motion primitive graph as the demonstration library grows.

LibraryFramesNodesTemporal edgesSpatial edgesStorage (MB)Overhead per query (ms)
LIBERO-10138,09028,304252,040516,53619.516.0
LIBERO Goal63,72813,235101,350242,1289.012.3
LIBERO Object74,50715,444123,440274,32010.412.8
LIBERO Spatial62,25012,94898,480226,3788.812.1
LIBERO (four suites)338,57569,931575,3101,221,71047.525.2
RoboTwin 2.0552,28788,964655,3441,540,35272.038.8

These results are reported in Section 6 and Table 3, with the camera-viewpoint and SO-101 experiments in Appendix B and the cost analysis in Appendix D.6 (Table 19).

W4. Discrepancies between the main text and the appendices

We reviewed the manuscript for consistency in three areas. Method: the temporal-edge description is now consistent between the main text and appendix (each node connects to nodes Δ∈{1,2,4,8,16}\Delta \in \{1, 2, 4, 8, 16\} steps before and after it in the same demonstration), and the ISR inference procedure matches the implementation. Settings: all simulation experiments use the released checkpoints in Table 6 without fine-tuning. The real-world policies are fine-tuned on task demonstrations, and R2A is applied to the same fine-tuned policy as the baseline. Numbers: we replaced outdated results in the introduction and checked the remaining values against the tables. These revisions are reflected in Section 1, Sections 3.3 and 3.4, and Appendix C.1.

We hope these revisions address your concerns.

AuthorsOpenReview ↗

Follow-up to Reviewer uKCE

Dear Reviewer uKCE,

Thank you again for your detailed review. As the discussion period closes soon, we would like to briefly check whether our response addresses your two main concerns.

(1) RoboTwin comparison (W1): with one execution schedule for both methods, R2A improves π₀.₅ from 56.8% to 65.5% with 50-step chunks and from 59.3% to 67.1% with 10-step chunks (Tables 2 and 17).

(2) ISR vs. retrieval (W2): with ISR removed, retrieval and harmonization raise GR00T N1.7 from 62.9% to 77.2% on LIBERO-10. When every method is given the same fixed reset, R2A reaches 89.3%, SDN 83.0% and the base policy 81.4% (Tables 5 and 16).

If any concern remains, we would be happy to clarify. Thank you for your time.

Reviewer_kbBVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper points out that VLA outputs can be unreliable under OOD conditions, including changes in the robot's initial state, camera viewpoint, and lighting. The paper proposes R2A to help VLA policies operate more reliably under these conditions. It uses a diffusion model to rectify the initial state and builds a graph from action segments in demonstrations offline. During execution, it retrieves relevant motion segments and blends them with the VLA's predictions.

Critical strengths
  1. Retrieving successful demonstrations to correct potentially unreliable VLA outputs under OOD conditions is a well-motivated approach.

  2. The simulation results shared in the authors' repository were informative.

Critical weaknesses
  1. The paper does not sufficiently justify the design choices for each component.

Ex 1) ISR moves the robot back to a familiar initial state, but the motivation for using diffusion is unclear. The diffusion model generates an initial state from random noise without conditioning on the current OOD state. The paper does not establish its advantage over simpler alternatives, such as selecting a familiar state with a fixed rule and moving the robot to it, or using an MLP-based generator. Comparisons with these alternatives are needed to justify this choice.

Ex 2) For retrieval, the benefit of the graph representation is also unclear. Demonstrations are already temporally ordered sequences. The paper should explain what is gained by decomposing them into nodes and connecting these nodes as a graph, compared with directly retrieving temporally contiguous demonstration segments.

  1. The experiments do not clearly establish the contribution of each component.

Ex 1) The improvement under robot initial-state perturbations is particularly large, but it is unclear how much of this gain comes from moving the robot back to a familiar state. An ISR-only result is needed to separate this effect from the contribution of retrieval and harmonization. If ISR accounts for most of the improvement, the comparisons with simpler alternatives discussed in Weakness 1 become especially important.

Ex 2) In Table 5, using only the retrieved actions without harmonization results in a success rate of just 1.2%. The retrieved actions alone are therefore largely ineffective in this setting. This raises the question of how much useful guidance they provide when combined with the VLA outputs. A more analysis of the blending weights and the resulting action corrections would help clarify when and how retrieval contributes to successful execution.

Questions for authors
  1. Some implementation details are difficult to understand from the current description. Could the authors explain more precisely how each graph node (q_v, delta q_v, gamma_v) is constructed from a demonstration segment? (For example, how is gamma_v defined if the gripper state changes within a segment, and how is the joint displacement delta q_v calculated?) Also, how exactly is the ICL baseline implemented?

  2. Could the authors compare R2A with existing retrieval-based methods, such as RA-VLA (ICML 2026) and Retrieval-VLA (CVPR 2026 Findings)?

9 comments
AuthorsOpenReview ↗

Response to Reviewer kbBV (1/3)

Thank you for the detailed feedback on our design choices and ablations. We address each point below.

W1 (Ex. 1). Why diffusion for ISR, and comparison with simpler alternatives

We agree that ISR should be compared with simpler alternatives, and we do not claim that diffusion is necessary on the benchmarks evaluated. We also apologize for an inaccurate description in the submission: ISR does not generate a start state from pure noise. Its reverse process starts from the observed out-of-distribution configuration and denoises it toward the demonstrations' start region. The error was in the text only; the evaluated implementation was unchanged. On LIBERO-10, we compared four recovery strategies: a fixed reset to the mean start state, the nearest demonstration's start state, an MLP trained on the same noisy–clean pairs, and ISR. Because start states are unimodal in this setting, all four strategies are within 3.3 pp. In Meta-World, which has two distinct start regions (frozen Evo-1), ISR reaches 90.3%, compared with 73.8% for the MLP.

Start-state recovery only (success rate, %).

SettingRecovery strategySR
LIBERO-10 (GR00T N1.7)No recovery32.6
LIBERO-10 (GR00T N1.7)Fixed reset (mean start state)81.4
LIBERO-10 (GR00T N1.7)Nearest-demonstration start79.4
LIBERO-10 (GR00T N1.7)ISR-MLP82.4
LIBERO-10 (GR00T N1.7)ISR82.7
Meta-World (Evo-1)No recovery35.6
Meta-World (Evo-1)ISR-MLP73.8
Meta-World (Evo-1)ISR90.3

We corrected the description in Section 3.4 and added these comparisons to Section 5.2(b), Table 5, and Appendix D.4.

W1 (Ex. 2). Benefit of the graph over contiguous demonstration segments

Demonstrations are already temporally ordered, so the graph must add value beyond that ordering. It enables retrieval across demonstrations: temporal edges follow motion within a demonstration, while spatial edges connect nodes from different demonstrations that have nearby end-effector poses. On the same 2,518 paired rollouts, contiguous-segment retrieval reaches 77.7%, compared with 75.0% for the temporal-only graph, 76.5% for the spatial-only graph, and 80.3% for the full graph. These results suggest that the two edge types provide complementary benefits beyond contiguous-segment retrieval. The ablation is reported in Section 5.2(c) and Appendix D.3 (Table 15).

Retrieval and graph ablation, GR00T N1.7 on LIBERO-10, same 2,518 paired rollouts (success rate, %).

VariantTemporal edgesSpatial edgesPropagationSR
No graph (top-k similarity)✗✗✗71.9
Contiguous segments✓✗✗77.7
Temporal-only graph✓✗✓75.0
Spatial-only graph✗✓✓76.5
Full graph (R2A)✓✓✓80.3
Full graph w/o ranking loss✓✓✓72.6
AuthorsOpenReview ↗

Response to Reviewer kbBV (2/3)

W2 (Ex. 1). How much of the gain comes from ISR alone

To isolate ISR's contribution, we evaluated R2A with ISR alone (without retrieval or harmonization) and R2A without ISR on LIBERO-10 with GR00T N1.7. On the Robot Initial States axis, ISR accounts for most of the gain, raising success from 32.6% to 82.7%; the full method reaches 87.5%. Across all seven axes, retrieval and harmonization without ISR reach 77.2%, ISR alone reaches 70.1%, and the full method reaches 82.1%. These results show that ISR is particularly effective for initial-state perturbations, while the components provide complementary gains overall (Table 5 and Section 5.2(b)).

Contribution of each component, GR00T N1.7 on LIBERO-10 (success rate, %).

ConfigurationRobot initAll axes
GR00T N1.732.662.9
R2A w/o ISR49.977.2
R2A only ISR82.770.1
R2A87.582.1

W2 (Ex. 2). Retrieved actions alone reach 1.2%: when and how does retrieval help?

Retrieved actions perform poorly on their own (1.2%) because they come from past demonstrations and are not conditioned on the current observation; executing them directly is therefore open-loop. R2A uses them as references to refine the VLA prediction, not as replacements.

To understand when and how retrieval helps, we measured the effective blending weight per action dimension (the redirected share in the paper). It ranges from 0.29 to 0.54, peaks under camera and noise perturbations, and increases with task difficulty. Corrections are smaller during approach and larger during transport.

Effective blending weight (redirected share) per action dimension and cumulative correction per episode, by perturbation axis.

AxisxyzRollPitchYawCorrection
Camera0.360.430.350.460.540.4099 cm / 69°
Noise0.350.400.330.410.480.3681 cm / 59°
Layout0.330.390.310.360.430.3379 cm / 55°
Language0.350.390.310.310.410.3577 cm / 55°
Background0.320.390.310.310.420.3274 cm / 51°
Robot init0.330.380.300.320.390.3172 cm / 50°
Light0.330.390.310.300.390.2971 cm / 51°

By difficulty level, with success rates (%).

LevelxyzRollPitchYawBase SRR2A SRGain (pp)
10.320.390.300.340.410.3371.683.6+12.0
20.310.360.290.310.390.3075.490.2+14.8
30.320.370.290.310.400.3264.990.2+25.3
40.340.400.320.350.440.3356.084.0+28.0
50.380.440.370.430.530.3935.861.9+26.1

By execution phase.

PhasexyzRollPitchYawCorrection
Approach (before first grasp)0.360.470.300.440.520.352.5 cm / 2.8°
Transport (gripper closed)0.320.360.310.340.430.3338.9 cm / 23.9°
After release0.360.450.330.370.450.3638.6 cm / 29.4°

On the same 2,518 rollouts, R2A raises success from 60.4% to 80.3%. It recovers 623 of the 998 rollouts failed by the base policy (62.4%) and loses only 121 of the 1,520 rollouts the base policy solves (8.0%).

Paired outcomes of GR00T N1.7 and R2A on the same 2,518 LIBERO-10 rollouts.

GR00T N1.7 outcomeR2A succeedsR2A failsTotal
Success1,399121 (8.0%)1,520
Failure623 (62.4%)375998
Total2,0224962,518

Finally, the gate learns the blending weight from demonstrations alone, without a sweep on evaluation rollouts. Its success rate (80.3%) is close to that of the best constant weight found by sweeping the evaluation episodes (81.6%).

Learned gate and constant blending weight (pooled success rate, %).

Blending weightSR
Constant 0.2 (swept on the evaluation episodes)81.6
Learned gate (demonstrations only)80.3

This analysis is reported in Section 5.2(a) and Appendix D.2 (Tables 10–14).

AuthorsOpenReview ↗

Response to Reviewer kbBV (3/3)

Q1. Node construction and the ICL baseline

We split each demonstration into consecutive, non-overlapping segments of at most five frames. A new node starts whenever the gripper state changes, ensuring that γᵥ remains constant within each node. Each node is encoded as

hv(0)=[norm⁡(qv,δqv,γv) ∥ PE⁡(sˉv)],\mathbf{h}_v^{(0)} = [\operatorname{norm}(\mathbf{q}_v, \delta\mathbf{q}_v, \gamma_v) \,\Vert\, \operatorname{PE}(\bar{s}_v)],

Here, qv\mathbf{q}_v is the end-effector pose (x, y, z, roll, pitch, yaw) at the node's first frame; δqv=qvlast−qv\delta\mathbf{q}_v = \mathbf{q}_v^{\mathrm{last}} - \mathbf{q}_v is the displacement across the node; and PE⁡(sˉv)\operatorname{PE}(\bar{s}_v) encodes the node's relative position within its demonstration.

For the ICL baseline, Cosmos-Reason2-2B generates a textual description of each demonstration timestep, covering the observation, proprioceptive state, and action outcome. At inference, we retrieve the nearest observation by cosine similarity and append its description to the VLA's language input. R2A outperforms ICL with both policies: 86.3% vs. 77.7% with π₀.₅ and 82.1% vs. 74.5% with GR00T N1.7 (see the per-axis results under Q2). Further details appear in Section 3.3 and Appendices C.1 and C.3.

Q2. Comparison with RA-VLA and Retrieval-VLA

We added RA-VLA and RICL as retrieval-based baselines and evaluated both with π₀.₅ and GR00T N1.7 across the seven LIBERO-10 perturbation axes. R2A achieves the highest average success rate with both policies. Retrieval-VLA has no public code, and reimplementing it would require assumptions that could make the comparison unfair. We therefore discuss its differences from R2A in Related Work (Section 2). The results below are from Table 3.

LIBERO-10 success rates (%) with retrieval-based baselines.

MethodCameraRobot initLanguageLightBackgroundNoiseLayoutAvg.
π₀.₅50.673.388.895.693.475.589.180.9
+ ICL45.179.191.989.179.278.980.777.7
+ RICL44.663.684.388.681.756.682.171.6
+ RA-VLA33.444.873.481.072.341.969.559.5
+ R2A55.196.991.995.695.278.091.386.3
GR00T N1.738.232.677.384.282.455.969.662.9
+ ICL50.860.189.381.091.377.571.874.5
+ RICL6.425.467.465.957.832.349.743.6
+ RA-VLA19.821.172.673.363.039.455.849.3
+ R2A58.987.584.694.588.978.681.482.1

We hope these revisions address your concerns.

Reviewer_7cUqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Retrieve to Act (R2A), a test-time augmentation framework that corrects model outputs by retrieving motion priors from training demonstrations while keeping the VLA model parameters frozen. The method constructs an offline motion primitive graph, with nodes representing short motion segments and edges encoding temporal connections and spatial similarities between end-effector poses. At inference time, R2A constructs a query from the robot’s state history and the VLA’s current predictions. A query-conditioned graph neural network localizes execution progress and retrieves the top-K matching segments. Their subsequent actions are aggregated into a reference trajectory through weighted averaging and then fused with the VLA’s predictions through learned, dimension-wise gating. Before task execution, a conditional diffusion module generates an initial joint configuration consistent with the training distribution, and the robot is physically guided to that configuration. R2A requires no fine-tuning of the base VLA, only offline training of auxiliary modules. Its effectiveness is demonstrated on LIBERO-Plus, RoboTwin2.0-Plus, and two real-world tasks.

Critical strengths
  1. The method improves policy robustness and task success rates while keeping the base VLA parameters frozen. It trains auxiliary networks to retrieve reference motions and refine action predictions, providing a way to enhance the policy without retraining or additional fine-tuning of the base model.

  2. The paper evaluates the method across multiple VLA models, single-arm and bimanual simulation environments, and real-world robot tasks. Component ablations and inference-latency measurements provide further support for its effectiveness under the tested conditions. The report indicates modest additional inference overhead in the measured setting

  3. Organizing successful demonstrations into a motion primitive graph and using a query-conditioned GNN for retrieval is an appealing design. It allows retrieval to exploit temporal connections and geometric relationships between motion segments, providing structured motion references for current action predictions without directly using visual features in the retrieval representation.

Critical weaknesses
  1. The abstract states that R2A achieves an absolute success-rate improvement of up to 40 percentage points in real-world experiments. However, Table 4 reports an increase from 30% to 80% on Banana2Pot under initial-state perturbations, corresponding to a 50-percentage-point improvement. Furthermore, the introduction states that R2A improves π₀.₅’s success rate on RoboTwin2.0-Plus from 57.5% to 61.4%, whereas the experimental section and Table 2 report an increase in average success rate from 56.8% to 66.2%, a gain of 9.4 percentage points. The paper does not explain whether these discrepancies arise from different evaluation settings or aggregation procedures, making it difficult to reconcile the reported figures. The authors should clarify these discrepancies and ensure consistent reporting across the abstract, introduction, and experimental section.

  2. Although Appendix B.2 describes the hardware, tasks, demonstration collection, and training setup, the paper does not specify the number of real-world evaluation trials per task, method, and condition underlying Table 4. The reported success rates also lack confidence intervals or other uncertainty estimates, making it difficult to assess their statistical precision. The authors should report the evaluation trial counts and corresponding uncertainty estimates

Minor weakness

  1. Section 5 states that pi0.5 exhibits a slight performance decrease under background texture perturbations. However, Table 1 reports an improvement under background perturbations (95.6% → 97.1%) and a decrease under lighting perturbations (97.1% → 96.3%). The authors should verify the underlying experimental results and resolve this inconsistency in the manuscript.
Questions for authors
  1. Could the authors verify the discrepancies between the abstract, introduction, and experimental tables, and identify which results are correct?

  2. How to generate policy and retrieval perturbations when training the harmonization module? In particular, how does training expose it to incorrect VLA predictions when the base policy already performs well on the demonstration data?

  3. How are perturbations applied during harmonization training? Do they affect only the query, or also the VLA action predictions and retrieved references?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 7cUq (1/2)

Thank you for carefully reviewing the paper and identifying the numerical inconsistencies. We address each point below.

W1. Inconsistent numbers across the abstract, introduction, and tables

The abstract and introduction contained outdated values. Table 4 reports the correct results: under robot initial-state perturbations, Banana2Pot improves from 30% to 80% (+50 percentage points, pp). For RoboTwin2.0-Plus, the baseline and R2A originally used different action-chunk configurations, so we reran both with matched execution schedules. R2A raises average success from 56.8% to 65.5% (+8.7 pp) with 50-step chunks and from 59.3% to 67.1% (+7.8 pp) with 10-step chunks. We updated the abstract, Section 1, and Table 2, and added the 10-step results to Appendix D.5 (Table 17).

RoboTwin2.0-Plus success rates (%), same execution schedule for both methods.

ScheduleMethodCameraRobot initLanguageLightBackgroundNoiseLayoutAvg.
50-stepπ₀.₅57.534.271.755.567.060.951.156.8
50-step+ R2A59.172.671.664.068.967.754.665.5
10-stepπ₀.₅56.138.673.460.967.564.154.459.3
10-step+ R2A60.974.874.665.670.867.155.967.1

W2. Trial counts and confidence intervals

We agree that trial counts and uncertainty estimates should be reported. Each real-world result is based on 20 trials per task, method, and condition; we now report success counts with 95% Wilson confidence intervals (CIs). We provide the same information for LIBERO-Plus, where each policy is evaluated on 10,026 rollouts.

Real-world results with GR00T N1.7: successes out of 20 trials (95% Wilson CI).

TaskConditionGR00T N1.7SDNR2A
Banana2PotNo perturbation12 (38.7%–78.1%)15 (53.1%–88.8%)16 (58.4%–91.9%)
Banana2PotRobot init6 (14.5%–51.9%)11 (34.2%–74.2%)16 (58.4%–91.9%)
Cube2DrawerNo perturbation11 (34.2%–74.2%)14 (48.1%–85.5%)17 (64.0%–94.8%)
Cube2DrawerRobot init2 (2.8%–30.1%)6 (14.5%–51.9%)9 (25.8%–65.8%)

LIBERO-Plus with GR00T N1.7: success rate (successes/rollouts) and 95% Wilson CI.

AxisRolloutsBase SR (%)Base 95% CI+ R2A SR (%)+ R2A 95% CI
Camera1,59965.2 (1043/1599)62.9–67.580.5 (1287/1599)78.5–82.4
Robot init1,55045.8 (710/1550)43.3–48.394.9 (1471/1550)93.7–95.9
Language1,53783.9 (1290/1537)82.0–85.786.6 (1331/1537)84.8–88.2
Light1,13895.5 (1087/1138)94.2–96.697.1 (1105/1138)96.0–97.9
Background1,07694.3 (1015/1076)92.8–95.694.2 (1014/1076)92.7–95.5
Noise1,60183.1 (1330/1601)81.2–84.891.1 (1459/1601)89.6–92.4
Layout1,52571.8 (1095/1525)69.5–74.078.3 (1194/1525)76.2–80.3
All episodes10,02675.5 (7570/10026)74.7–76.388.4 (8861/10026)87.7–89.0

LIBERO-Plus with π₀.₅: success rate (successes/rollouts) and 95% Wilson CI.

AxisRolloutsBase SR (%)Base 95% CI+ R2A SR (%)+ R2A 95% CI
Camera1,59972.0 (1151/1599)69.7–74.178.7 (1258/1599)76.6–80.6
Robot init1,55075.3 (1167/1550)73.1–77.497.2 (1507/1550)96.3–97.9
Language1,53785.8 (1319/1537)84.0–87.586.6 (1331/1537)84.8–88.2
Light1,13897.1 (1105/1138)96.0–97.996.3 (1096/1138)95.0–97.3
Background1,07695.6 (1029/1076)94.2–96.797.1 (1045/1076)95.9–98.0
Noise1,60188.6 (1418/1601)86.9–90.089.2 (1428/1601)87.6–90.6
Layout1,52587.0 (1327/1525)85.2–88.687.7 (1337/1525)85.9–89.2
All episodes10,02684.9 (8516/10026)84.2–85.689.8 (9002/10026)89.2–90.4

The results are reported in Appendix D.1: Table 8 covers the real-world experiments, and Table 9 covers LIBERO-Plus.

AuthorsOpenReview ↗

Response to Reviewer 7cUq (2/2)

Minor. π₀.₅ under Background vs. Light

You are right: Table 1 is correct, and our sentence was inaccurate. π₀.₅ decreases slightly under Light Conditions (97.1% → 96.3%) but improves under Background Textures (95.6% → 97.1%). We corrected the sentence in Section 5.1.

Q1. Which results are correct?

The tables contain the final results, and we have updated the abstract and introduction to match them (see W1).

Q2 and Q3. Perturbations during harmonization training

Because the base policy performs well on its own demonstrations, replaying them rarely exposes incorrect predictions for the harmonization module to correct. We therefore perturb the policy's predictions in four ways: temporal mismatch (a prediction from another timestep in the same demonstration), cross-task mismatch (a prediction from another task), endpoint displacement (under- or overshooting), and pose displacement (a constant offset between the predicted chunk and the robot's current pose). We also replace some retrieved references with unreliable ones, including segments from another task or task stage and references with displaced poses. We keep the query unchanged and retain clean predictions and references. This trains the module to follow a retrieved reference only when it is more trustworthy than the policy prediction. Section 4 describes the procedure, with full details in Appendix C.4.

We hope these revisions address your concerns.

Thanks for the rebuttal, which has addressed the major concerns. I have no more questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions (page and line in the revised PDF)

We thank the AC, SAC and reviewers for their time. R2A is a 0.37M-parameter module that attaches to a frozen VLA at test time. It needs no fine-tuning of the base policy and adds 21 ms per step. On LIBERO-Plus it raises the average success of GR00T N1.7 from 77.1% to 89.0%, of StableVLA from 78.1% to 89.5% and of π₀.₅ from 85.9% to 90.4%.

Main new results

  1. With one execution schedule for both methods, R2A still improves π₀.₅ on RoboTwin2.0-Plus, from 56.8% to 65.5% with 50-step chunks and from 59.3% to 67.1% with 10-step chunks (Tables 2 and 17).
  2. Retrieval and harmonization help without ISR. With ISR removed, they raise GR00T N1.7 from 62.9% to 77.2% on LIBERO-10. When every method is given the same fixed reset, R2A reaches 89.3%, SDN 83.0% and the base policy 81.4% (Tables 5 and 16).
  3. The two retrieval baselines requested by Reviewer kbBV, RICL and RA-VLA, score 71.6% and 59.5% on LIBERO-10 with π₀.₅, below the base policy at 80.9%. R2A reaches 86.3%, which is also above VLS, SDN and ICL (Table 3).
  4. On the ALOHA robot, R2A improves Banana2Pot from 30% to 80% under initial-state perturbation and from 10% to 60% under a camera shift. On a second platform, SO-101, it improves from 25% to 80% under initial-state perturbation. Table 8 gives trial counts and 95% Wilson CIs for every entry.

Changes are in orange in the revised PDF, and line numbers are those printed in the margin.

Where each change is

ChangeWhereRaised by
Abstract and introduction report the table valuesp. 1, l. 026–028 and p. 2, l. 098–1067cUq W1, Q1
RoboTwin2.0-Plus rerun under one execution schedule for both methods, plus 10-step results and an ablation of the displacement compensationp. 8, l. 377–383 and l. 407–411. Appendix D.5, p. 23–24 (Tables 17–18)uKCE W1, 7cUq W1
ISR-only and no-ISR variants of R2A, and a comparison of ISR with simpler start-state recovery strategies on LIBERO-10 and Meta-Worldp. 9, l. 433–446 (Table 5) and l. 458–476. Appendix D.4, p. 23–24 (Table 16, Figure 7)kbBV W1 (Ex. 1), W2 (Ex. 1), uKCE W2
RICL and RA-VLA added as baselines, and the ICL baseline describedp. 8, l. 385–402 and l. 416–430. Appendix C.3, p. 19kbBV Q1, Q2
Graph ablation against contiguous demonstration segmentsp. 9, l. 477–480. Appendix D.3, p. 22–23 (Table 15)kbBV W1 (Ex. 2)
Analysis of when and how much harmonization changes the policy's actionsp. 9, l. 466–469. Appendix D.2, p. 20–22 (Tables 10–14)kbBV W2 (Ex. 2)
Trial counts and 95% Wilson CIs for the real-world and LIBERO-Plus resultsAppendix D.1, p. 20–21 (Tables 8–9)7cUq W2
Camera-viewpoint experiment and SO-101 platformp. 9, l. 447–455. Appendix B, p. 17, l. 868–895uKCE W3
Cost of the graph as the demonstration library growsp. 10, l. 489–491. Appendix D.6, p. 24 (Table 19)uKCE W3
ISR description corrected to match the implementationp. 4, l. 204–205 to p. 5, l. 217. Appendix C.1, p. 19, l. 993–996kbBV W1 (Ex. 1)
Node construction and harmonization training described preciselyp. 4, l. 180–186. p. 6, l. 313 to p. 7, l. 333. Appendix C.4, p. 20kbBV Q1, 7cUq Q2, Q3
Main text and appendix made consistent on edges, module names, checkpoints and training settingsp. 4, l. 191–197. p. 5, l. 235–237. Appendix C.1–C.2, p. 18–19 (Tables 6–7)uKCE W4
Sentence on π₀.₅ under Light and Background corrected, and a "w/o Robot" column added to Table 1p. 7, l. 343–377. p. 8, l. 403–4057cUq Minor
Limitations state the lack of explicit scene geometry and the scope of the real-world evaluationp. 10, l. 497–507uKCE W3
New versions of Figure 1, Figure 3 and Figure 5p. 2, p. 9 and p. 16general

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.