acceptodds
Under review as a conference paper at ICLR 2027

OVOGO: Offline Value-Guided On-Policy Generative Optimization for Flow-Based Vision-Language-Action Models

Abstract

Flow-based vision-language-action models can benefit from robot experience collected during deployment, but obtaining new interactions for every policy update is costly. Offline reinforcement learning enables repeated reuse of collected experience, yet applying it to generative VLAs requires a critic that can evaluate newly generated actions and an effective way to improve the generative policy from these value signals. We propose OVOGO, an offline reinforcement learning framework that decouples offline value learning from on-policy generative optimization. OVOGO introduces a Return-Anchored Ranking Critic that combines in-sample distributional value learning with guarded local ranking, keeping values grounded in observed robot experience while improving discrimination among generated actions. The frozen critic then provides group-relative value signals for selected-transition optimization of fresh actions sampled from the current flow policy. Experiments on RoboTwin 2.0 and DexJoCo show that OVOGO consistently improves the pretrained VLA and outperforms existing offline adaptation and value-guided methods. The results suggest that expensive robot experience can be repeatedly reused for value learning while inexpensive generative sampling drives policy optimization, providing an effective approach to experience-based post-training of large VLAs.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Vision-Language-Acion Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4uvXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OVOGO improves flow-based VLAs using offline robot experience. It trains a critic through value learning and synthetic action-ranking supervision, then uses the frozen critic to score newly generated actions and update the VLA action expert. Experiments on RoboTwin and DexJoCo show improvements over the initial π0.5 policy. Deployment uses the updated policy without additional critic evaluation.

Critical strengths
  • The method improves average success over the same pretrained policy on both benchmarks, using a fixed offline dataset within each training round.

  • The updated policy does not require critic evaluation or multiple-candidate selection during deployment.

Critical weaknesses
  • The reliability and contribution of the critic design are insufficiently established. The synthetic ranking labels assume that successful replay actions outperform perturbed actions and that smaller perturbations outperform larger ones. Their agreement with actual execution outcomes is not measured. The value visualizations show score differences, but do not establish that those differences reflect action quality. Since the critic remains frozen while the policy changes, its reliability on later policy-generated actions also needs examination. Furthermore, the comparison between unguarded and guarded ranking changes the temperature from 1 to 0.1. The reported improvement from 63.0% to 67.8% therefore cannot be attributed specifically to the guard.

  • The comparisons and practical validation are incomplete. LWD and Q-VGM are discussed as closely related policy-training methods but are not evaluated. Best-of-N and QGF instead modify inference, while Success BC and RECAP both reduce average performance relative to the initial policy. The reasons for these degradations and the baseline tuning procedures need clarification. All experiments are conducted in simulation, leaving effectiveness on real robot experience unverified. Although dexterous tasks are included, several remain unsuccessful or have very low success rates, and these limitations receive little analysis.

Questions for authors
  • How often do the synthetic ranking labels agree with returns measured by executing the corresponding actions? Does critic ranking remain reliable throughout policy optimization?

  • Can you compare guarded and unguarded ranking at the same temperature, and examine sensitivity to perturbation magnitude and negative-sample type?

  • Can you provide comparisons with LWD or Q-VGM, or explain why they are unsuitable? Please clarify the training schedules and checkpoint selection for Success BC and RECAP, and analyze their performance degradation.

3 comments
AuthorsOpenReview ↗

Problem discussion and supplementary experiments

We thank the reviewer for the thoughtful suggestions on critic reliability, guard ablations, and policy-training baselines. We added perturbation execution, late-training Best-of-N, same-temperature guard comparisons, and LWD experiments across three environments.

Q1. Synthetic ranking labels and reliability during training

Thank you for this suggestion. Following our negative-sample construction, we add mild and strong noise to each base-policy action chunk, using the same evaluation initial states and seeds:

ExecutionNoise scaleSuccess (%)Change (percentage points)
Base policy060.7—
Mild0.0456.13 ± 0.80−4.57
Strong0.0849.40 ± 6.24−11.30

Success decreases as perturbations increase, supporting our assumption that stronger perturbations degrade effective actions more.

Second, Best-of-N with the frozen critic improves the initial policy from 60.7% to 64.23 ± 1.98%, and the final OVOGO actor from 68.17 ± 1.19% to 70.43 ± 0.65%. These execution results support synthetic perturbation ranking and the critic’s discrimination before and after optimization. Thank you for helping us strengthen this evidence.

Q2. Independent contribution of the guard and negative-sample sensitivity

Temperature and the guard jointly control ranking pressure on a return-grounded Q scale: temperature controls gradient decay, while the guard stops pressure after sufficient separation. Following your advice, we compared them at the same temperature:

RoboTwin configurationτSuccess (%)
No guard (original ablation)1.063.17 ± 0.76
No guard (new comparison)0.166.50 ± 0.56
Guard (full RARC)0.168.17 ± 1.19

Without the guard, lowering τ from 1 to 0.1 improves success by 3.33 percentage points. At τ=0.1, the guard adds 1.67 points. Temperature plays the main role in controlling ranking pressure, while the guard provides additional protection and gains.

We also added a LIBERO ablation: τ=1 without the guard achieves 71.05 ± 3.10%, the initial policy 71.40%, and full OVOGO 78.36 ± 2.00%. The 7.31-point gain supports the design in another environment. Q1 also supports our perturbation scales.

Q3. Related methods and baseline training details

Thank you for the suggestion. We reproduced LWD using the algorithm settings and hyperparameters reported in its paper. Results are shown in the comparison table below.

OVOGO exceeds LWD by 1.04, 4.50, and 3.15 percentage points, respectively. This adds a direct critic-guided policy-training comparison.

RECAP training code was unavailable, so we followed Evo-RL. We tried learning rates 1×10⁻⁵, 2×10⁻⁵, and 5×10⁻⁵ and batch sizes 32, 64, 128, and 256, selecting learning rate 2×10⁻⁵ and batch size 64 for expert-only training. For fair critic comparisons, we use Gemma 3 270M and SigLIP2 (224×224) encoders. The original setup used expert-only training with the VLM backbone frozen.

Success BC uses batch size 256 and learning rate 1×10⁻⁵, initialized from the public clean50 model already SFT-trained on 50 tasks. Further training on fewer successful trajectories may cause overfitting or forgetting, reducing closed-loop success. All training methods run for 10,000 steps; we evaluate the best saved checkpoint.

We added RECAP full fine-tuning, updating both the VLM backbone and action expert. Its success rates on RoboTwin, DexJoCo, and LIBERO are 64.23 ± 1.06%, 29.03 ± 3.44%, and 76.11 ± 0.50%, respectively, all above the initial policies. We will use these results as the RECAP baseline in the revised paper.

W2. Practical validation and difficult tasks

We added LIBERO, extending simulation evaluation to three environments. Real-robot experiments are underway, and we will provide results as soon as they are complete. Thank you for suggesting this validation.

Trained methods use three seeds; results are mean ± standard deviation. The initial policy is the pretrained checkpoint. Success rates (%):

MethodRoboTwinDexJoCoLIBERO
Initial policy60.727.571.40
BC54.10 ± 0.3627.57 ± 1.9075.11 ± 0.45
RECAP64.23 ± 1.0629.03 ± 3.4476.11 ± 0.50
QGF63.30 ± 1.4829.47 ± 0.4572.30 ± 1.11
Best-of-N64.23 ± 1.9829.73 ± 0.2373.97 ± 1.92
LWD67.13 ± 0.8532.40 ± 0.6075.21 ± 1.42
OVOGO68.17 ± 1.1936.90 ± 0.9878.36 ± 2.00

RECAP uses full fine-tuning. OVOGO has the highest mean success in all three environments, with improvements maintained across seeds.

The official SFT model also struggles on near-zero-success DexJoCo tasks. With sparse terminal rewards, few successful rollouts limit the return signal and subsequent RL improvement. Thank you for highlighting these tasks; we will add this analysis to the revision.

Thank you for the detailed response and additional experiments. The authors have adequately addressed my main concerns, and I have increased my rating accordingly.

AuthorsOpenReview ↗

Real robot experiment

Real-robot experiment

Hardware and policy interface

The platform consists of a Universal Robots UR5e arm with six revolute joints and a five-finger INSPIRE Robots RH56DFX-series dexterous hand with six actuator channels. A fixed external Orbbec Femto Bolt camera observes the workspace, and a wrist-mounted Intel RealSense L515 provides a close-up view. The policy receives RGB images from both cameras, resized to 224 × 224 pixels, along with the language instruction and the current robot state. The policy uses RGB only.

The proprioceptive state and the executed action each have 12 dimensions. The state contains the measured positions, and the action specifies absolute position targets in the same channel order.

DimensionsComponentChannel order and units
1–6UR5e armSix arm joint positions in robot joint order, in radians
7–12INSPIRE handLittle finger, ring finger, middle finger, index finger, thumb flexion, and thumb rotation, in raw actuator counts

At each policy call, π₀.₅ predicts a 32-step action chunk. Its internal representation is 32 × 32, and the first 12 channels at each step form the 32 × 12 action chunk passed to the robot controller. The deployment code supports a configurable execution horizon and defaults to 32 actions before the next policy call.

Initial policy and rollout collection

We used 50 demonstrations of the block-stacking task to fine-tune π₀.₅ for 30,000 optimizer steps with a global batch size of 256. We used AdamW with betas = (0.9, 0.95), eps = 1e-8, and weight decay of 1e-10. We clipped the gradient norm to 1.0. The learning rate warmed up linearly for 1,000 steps to 5 × 10⁻⁵, then followed a cosine decay schedule configured for the 30,000-step run, with an end value of 5 × 10⁻⁶.

The checkpoint at step 30,000 was the initial policy for this experiment. It succeeded in 9 of 20 real-robot trials. We used that checkpoint to collect 100 additional rollout episodes for offline post-training. The results below list the demonstration-trained initial checkpoint and the post-training SFT baseline separately.

Each method was evaluated on 20 real-robot trials. We report the number of successful trials and the corresponding empirical success rate.

MethodSuccessful trialsSuccess rate
Initial policy (30k demonstration SFT)9/2045%
SFT baseline11/2055%
RECAP13/2065%
Best-of-N11/2055%
QGF10/2050%
LWD13/2065%
OVOGO16/2080%

We appreciate your careful review and your openness to our clarifications. Thank you for raising your score; your feedback has improved both the evaluation and the presentation of the paper.

Reviewer_CzfZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OVOGO trains a critic on offline robot data, then uses its scores to improve newly sampled actions from a flow-based VLA. The critic combines return learning with local action ranking. Experiments cover RoboTwin and DexJoCo.

Critical strengths

s1. Reusing offline experience without collecting new rollouts for every update is practically useful.

s2. The method improves over the initial policy on both benchmarks and requires no critic at inference time.

s3. The Best-of-N comparison provides some evidence that the critic can identify better actions.

Critical weaknesses

w1. The ranking labels assume that perturbing successful actions makes them worse. It remains unclear how well this assumption supports evaluating actions from an evolving policy.

w2. The guard ablation also changes the ranking temperature from 1 to 0.1, so the contribution of the guard is not isolated.

w3. Both BC and RECAP underperform the initial policy, without enough analysis of why. Comparisons with LWD or Q-VGM would strengthen the evaluation.

w4. Evaluation is limited to simulation, and variability across training seeds is not reported.

Questions for authors

q1. Could you execute sampled candidates in simulation and compare their actual returns with the critic’s rankings, both early and late in policy training?

q2. Could you compare guarded and unguarded ranking at the same temperature?

q3. Why do BC and RECAP degrade under this setup? Could you report their tuning details and include a direct critic-guided policy- training baseline?

q4. Could you include a real-robot evaluation to test whether the improvements carry over beyond simulation?

2 comments
AuthorsOpenReview ↗

Problem discussion and supplementary experiments

We thank the reviewer for recognizing the practical value of reusing offline experience and deploying the policy without a critic. Following your suggestions, we added perturbation, late-training Best-of-N, guard, and LWD experiments.

Q1. Perturbation labels and critic reliability

Thank you for this suggestion. Following our negative-sample construction, we added noise at scales 0.04 (mild) and 0.08 (strong) to each action chunk predicted by the base policy and executed it in the environment. We use the same initial states and evaluation seeds:

ExecutionNoise scaleSuccess (%)Change (percentage points)
Base policy060.7—
Mild0.0456.13 ± 0.80−4.57
Strong0.0849.40 ± 6.24−11.30

Success decreases as perturbation increases, supporting the premise of our negative-sample construction: perturbing effective actions reduces execution quality, and stronger perturbations produce a larger decrease.

We also evaluated the frozen critic on the final OVOGO actors. Best-of-N increases success from 68.17 ± 1.19% to 70.43 ± 0.65%. Together with the initial-policy result, this shows that the critic selects better candidates before and after policy optimization:

Candidate policyDirect execution (%)Best-of-N (%)
Initial SFT policy60.764.23 ± 1.98
Final OVOGO actor68.17 ± 1.1970.43 ± 0.65

The 2.26-point gain shows that the critic remains effective late in training. Thank you for suggesting this validation.

Q2. Guard comparison at the same temperature

Thank you for this suggestion. Temperature and the guard jointly control ranking pressure to keep Q values on a return-grounded scale: temperature controls the decay of ranking gradients, while the guard stops additional pressure once action pairs are sufficiently separated. Following your advice, we added the unguarded τ=0.1 comparison:

RoboTwin configurationτSuccess (%)
No guard1.063.17 ± 0.76
No guard0.166.50 ± 0.56
Guard0.168.17 ± 1.19

Without the guard, lowering τ from 1 to 0.1 raises success by 3.33 percentage points. At τ=0.1, adding the guard raises it by another 1.67 points. Temperature plays the main role, while the guard adds a further gain.

Q3. BC, RECAP, and direct policy-training baselines

Success BC. We use batch size 256 and learning rate 1×10⁻⁵. Training runs for 10,000 steps, and we select the best checkpoint. The initialization is the public clean50 model already SFT-trained on 50 RoboTwin tasks. Further supervised training on a smaller set of successful trajectories may move it away from a good parameter region through overfitting or forgetting, reducing closed-loop success.

RECAP. The original RECAP training code was unavailable, so we followed Evo-RL. We tried learning rates 1×10⁻⁵, 2×10⁻⁵, and 5×10⁻⁵ and batch sizes 32, 64, 128, and 256, selecting learning rate 2×10⁻⁵ and batch size 64 for expert-only training. For fair critic comparisons, we use Gemma 3 270M and SigLIP2 (224×224 input) encoders. The original setup used expert-only training with a frozen VLM backbone.

We added full fine-tuning of both the VLM backbone and action expert. RECAP achieves 64.23 ± 1.06%, 29.03 ± 3.44%, and 76.11 ± 0.50% on RoboTwin, DexJoCo, and LIBERO, respectively, exceeding the initial policies and improving on the original expert-only setup. We will use full fine-tuning as the RECAP baseline in the revised paper.

LWD. Following your suggestion, we reproduced LWD from its paper, using its reported algorithm settings and hyperparameters because the official training code was unavailable. Results are in the comparison table below.

OVOGO exceeds LWD by 1.04, 4.50, and 3.15 percentage points, respectively, supporting the effectiveness of our policy-training method.

Q4. Real-robot evaluation

Real-robot experiments are underway; we will provide the results as soon as they are complete. Thank you for your patience, and we welcome any further suggestions.

W4. Training-seed variability

Trained methods use three seeds; results are mean ± standard deviation. The initial policy is the pretrained checkpoint. Success rates (%):

MethodRoboTwinDexJoCoLIBERO
Initial policy60.727.571.40
BC54.10 ± 0.3627.57 ± 1.9075.11 ± 0.45
RECAP64.23 ± 1.0629.03 ± 3.4476.11 ± 0.50
QGF63.30 ± 1.4829.47 ± 0.4572.30 ± 1.11
Best-of-N64.23 ± 1.9829.73 ± 0.2373.97 ± 1.92
LWD67.13 ± 0.8532.40 ± 0.6075.21 ± 1.42
OVOGO68.17 ± 1.1936.90 ± 0.9878.36 ± 2.00

RECAP uses full fine-tuning. OVOGO has the highest mean success in all three environments, with improvements maintained across seeds. Real-robot results will follow.

AuthorsOpenReview ↗

Real robot experiment

Real-robot experiment

Hardware and policy interface

The platform consists of a Universal Robots UR5e arm with six revolute joints and a five-finger INSPIRE Robots RH56DFX-series dexterous hand with six actuator channels. A fixed external Orbbec Femto Bolt camera observes the workspace, and a wrist-mounted Intel RealSense L515 provides a close-up view. The policy receives RGB images from both cameras, resized to 224 × 224 pixels, along with the language instruction and the current robot state. The policy uses RGB only.

The proprioceptive state and the executed action each have 12 dimensions. The state contains the measured positions, and the action specifies absolute position targets in the same channel order.

DimensionsComponentChannel order and units
1–6UR5e armSix arm joint positions in robot joint order, in radians
7–12INSPIRE handLittle finger, ring finger, middle finger, index finger, thumb flexion, and thumb rotation, in raw actuator counts

At each policy call, π₀.₅ predicts a 32-step action chunk. Its internal representation is 32 × 32, and the first 12 channels at each step form the 32 × 12 action chunk passed to the robot controller. The deployment code supports a configurable execution horizon and defaults to 32 actions before the next policy call.

Initial policy and rollout collection

We used 50 demonstrations of the block-stacking task to fine-tune π₀.₅ for 30,000 optimizer steps with a global batch size of 256. We used AdamW with betas = (0.9, 0.95), eps = 1e-8, and weight decay of 1e-10. We clipped the gradient norm to 1.0. The learning rate warmed up linearly for 1,000 steps to 5 × 10⁻⁵, then followed a cosine decay schedule configured for the 30,000-step run, with an end value of 5 × 10⁻⁶.

The checkpoint at step 30,000 was the initial policy for this experiment. It succeeded in 9 of 20 real-robot trials. We used that checkpoint to collect 100 additional rollout episodes for offline post-training. The results below list the demonstration-trained initial checkpoint and the post-training SFT baseline separately.

Each method was evaluated on 20 real-robot trials. We report the number of successful trials and the corresponding empirical success rate.

MethodSuccessful trialsSuccess rate
Initial policy (30k demonstration SFT)9/2045%
SFT baseline11/2055%
RECAP13/2065%
Best-of-N11/2055%
QGF10/2050%
LWD13/2065%
OVOGO16/2080%
Reviewer_fUJKOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents a method for offline RL for post-training flow-based VLAs that combines offline value learning with fresh on-policy optimization of the generative process. There are two algorithmic contributions. First they train a critic (RARC) using only actual robot data, using in-sample distributional TD learning that orders synthetic perturbations around replayed actions. This keeps the critic grounded while being able to provide relative ranking between actions, even if the absolute value estimations may not be correct, specifically for actions not logged. The frozen critic then scores groups of freshly sampled action chunks, and those scores drive a GRPO-style update of the action expert.

Critical strengths

The main problem that the paper tries to solve is how to turn an offline-trained critic into policy improvement for large flow-based VLAs. That requires a critic that is still grounded in logged returns yet discriminative enough for newly generated actions.Flow-based VLAs lack tractable action likelihoods, so the update uses the likelihood of only one selected stochastic denoising transition and no critic gradients. While not explicitly stated, this seems to be the reason why standard offline RL methods (like IQL/AWR/CQL) are insufficient here. Overall, the problem the paper aims to solve is an important one and the presented approach is credible. The simulation results are promising.

Critical weaknesses

The main limitation is that the evaluation severely limited. There are only two simulation environments with ~10 tasks each. No physical experiments were conducted. The ablations themselves are limited and only performed in one of the two simulation environments. While the performance of the proposed method is better on average that the baselines, no confidence intervals were presented, so not clear how statistically significant the results are.

Questions for authors

The ablations in Table 2 seem to suggested that ranking is actually doing the improvement and the return-grounded critic is mostly at the same level as the baseline pi0.5 policy. Is that correct? In that case, what is the benefit of the return-grounded critic terms. Is there an ablation where the critic is trained with ONLY the ranking loss but not TD? Without it, the claim that the critic is “return-anchored” in a way that drives the gains isn’t shown.

Why do the baselines degrade? Especially RECAP which forms the basis of pi0.6. Are there hypeyrpmaeter/configuration differences (baselines were not tuned but th proposed method was, e.g.) that cause this anomaly.

What policy was used for the 100 rollouts? What was its success rate?

Does the critic rank new candidates correctly? For example, execute several sampled candidates from the same state and correlate the critic’s scores with actual success.

Three DexJoCo tasks stay near 0% for every method. Can OVOGO improve tasks where the logged data has almost no successes, or does it only refine behaviors the base policy can already do?

2 comments
AuthorsOpenReview ↗

Problem discussion and supplementary experiments

We thank the reviewer for recognizing the importance of offline value learning for improving flow-based VLAs, and for the specific and helpful experimental suggestions. We added experiments on return grounding, baselines, critic discrimination, and evaluation coverage. We respond to each question below.

Q1. Contribution of return-grounded value learning

Critic configurationRoboTwin success (%)
Initial policy60.7
Return-grounded critic only60.97 ± 1.38
Ranking only55.90 ± 1.54
Return-grounded critic + guarded ranking (RARC)68.17 ± 1.19

Removing TD with the same data and settings reduces success to 55.90 ± 1.54%, below the initial 60.7% and 12.27 points below RARC. Without TD, ranking loses its grounding in observed returns and policy learning deteriorates. TD alone establishes this foundation (60.97 ± 1.38%); adding guarded ranking raises success to 68.17 ± 1.19%. Ranking thus strengthens action discrimination on the return-grounded foundation.

Q2. RECAP degradation and baseline configuration

RECAP training code was unavailable, so we followed Evo-RL for our implementation. We tried learning rates of 1×10⁻⁵, 2×10⁻⁵, and 5×10⁻⁵ and batch sizes of 32, 64, 128, and 256, selecting learning rate 2×10⁻⁵ and batch size 64 for expert-only training. For a fair comparison of critic representations, the experiments use the same Gemma 3 270M and SigLIP2 (224×224 input) encoders. The original setup used expert-only training, with the VLM backbone frozen.

We also fully fine-tuned RECAP, updating the VLM backbone and action expert. Its three-seed success rates on RoboTwin, DexJoCo, and LIBERO are 64.23 ± 1.06%, 29.03 ± 3.44%, and 76.11 ± 0.50%, respectively, all above the corresponding initial policies. We will use these results as the RECAP baseline in the revised paper.

Q3. Rollout collection policy

We used public checkpoints both to initialize training and to collect the initial rollouts: the clean50 SFT model for RoboTwin, the official multitask π0.5 model for DexJoCo, and the RLinf LIBERO SFT model for LIBERO. Their initial success rates are 60.7%, 27.5%, and 71.40%, respectively.

Q4. Critic reliability on new candidates

We evaluated Best-of-N with the frozen critic on the final OVOGO actors. Success increased from 68.17 ± 1.19% to 70.43 ± 0.65%, a gain of 2.26 percentage points.

Candidate policyDirect execution (%)Best-of-N with frozen critic (%)
Initial SFT policy60.764.23 ± 1.98
Final OVOGO actor68.17 ± 1.1970.43 ± 0.65

These gains show that the frozen critic discriminates among candidates at both stages.

Q5. Tasks with very low success rates

The official SFT checkpoint also struggles on these difficult DexJoCo tasks: assembly and unlock-iPad remain at 0%, while Hanoi improves from 2% to 4%. Our setting uses sparse terminal success rewards. When successful rollouts are scarce, the data contain very little return signal indicating success, making it difficult for the critic to learn sufficiently discriminative supervision. Other RL methods face the same difficulty without expert data.

W1. Evaluation coverage and variability

Thank you for your suggestion on evaluation coverage. We added LIBERO main results and ablations, extending the ablations beyond RoboTwin. In the LIBERO ablation, the initial policy achieves 71.40%, full OVOGO achieves 78.36 ± 2.00%, and the τ=1 configuration without the guard achieves 71.05 ± 3.10%.

EnvironmentInitial policyτ=1, no guardτ=0.1, guard
RoboTwin60.763.17 ± 0.7668.17 ± 1.19
LIBERO71.4071.05 ± 3.1078.36 ± 2.00

The full configuration exceeds the τ=1 unguarded variant by 5.00 percentage points on RoboTwin and 7.31 on LIBERO, supporting the combined design across environments.

Trained methods use three seeds; results are mean ± standard deviation. The initial policy is the pretrained checkpoint. Success rates (%):

MethodRoboTwinDexJoCoLIBERO
Initial policy60.727.571.40
BC54.10 ± 0.3627.57 ± 1.9075.11 ± 0.45
RECAP64.23 ± 1.0629.03 ± 3.4476.11 ± 0.50
QGF63.30 ± 1.4829.47 ± 0.4572.30 ± 1.11
Best-of-N64.23 ± 1.9829.73 ± 0.2373.97 ± 1.92
LWD67.13 ± 0.8532.40 ± 0.6075.21 ± 1.42
OVOGO68.17 ± 1.1936.90 ± 0.9878.36 ± 2.00

OVOGO has the highest mean success across all three environments, with improvements maintained across seeds. Real-robot experiments are underway, and we will provide the results when complete. Thank you again for your thoughtful and detailed suggestions.

AuthorsOpenReview ↗

Real robot experiment

Real-robot experiment

Hardware and policy interface

The platform consists of a Universal Robots UR5e arm with six revolute joints and a five-finger INSPIRE Robots RH56DFX-series dexterous hand with six actuator channels. A fixed external Orbbec Femto Bolt camera observes the workspace, and a wrist-mounted Intel RealSense L515 provides a close-up view. The policy receives RGB images from both cameras, resized to 224 × 224 pixels, along with the language instruction and the current robot state. The policy uses RGB only.

The proprioceptive state and the executed action each have 12 dimensions. The state contains the measured positions, and the action specifies absolute position targets in the same channel order.

DimensionsComponentChannel order and units
1–6UR5e armSix arm joint positions in robot joint order, in radians
7–12INSPIRE handLittle finger, ring finger, middle finger, index finger, thumb flexion, and thumb rotation, in raw actuator counts

At each policy call, π₀.₅ predicts a 32-step action chunk. Its internal representation is 32 × 32, and the first 12 channels at each step form the 32 × 12 action chunk passed to the robot controller. The deployment code supports a configurable execution horizon and defaults to 32 actions before the next policy call.

Initial policy and rollout collection

We used 50 demonstrations of the block-stacking task to fine-tune π₀.₅ for 30,000 optimizer steps with a global batch size of 256. We used AdamW with betas = (0.9, 0.95), eps = 1e-8, and weight decay of 1e-10. We clipped the gradient norm to 1.0. The learning rate warmed up linearly for 1,000 steps to 5 × 10⁻⁵, then followed a cosine decay schedule configured for the 30,000-step run, with an end value of 5 × 10⁻⁶.

The checkpoint at step 30,000 was the initial policy for this experiment. It succeeded in 9 of 20 real-robot trials. We used that checkpoint to collect 100 additional rollout episodes for offline post-training. The results below list the demonstration-trained initial checkpoint and the post-training SFT baseline separately.

Each method was evaluated on 20 real-robot trials. We report the number of successful trials and the corresponding empirical success rate.

MethodSuccessful trialsSuccess rate
Initial policy (30k demonstration SFT)9/2045%
SFT baseline11/2055%
RECAP13/2065%
Best-of-N11/2055%
QGF10/2050%
LWD13/2065%
OVOGO16/2080%

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.