- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
OVOGO improves flow-based VLAs using offline robot experience. It trains a critic through value learning and synthetic action-ranking supervision, then uses the frozen critic to score newly generated actions and update the VLA action expert. Experiments on RoboTwin and DexJoCo show improvements over the initial π0.5 policy. Deployment uses the updated policy without additional critic evaluation.
Critical strengths
-
The method improves average success over the same pretrained policy on both benchmarks, using a fixed offline dataset within each training round.
-
The updated policy does not require critic evaluation or multiple-candidate selection during deployment.
Critical weaknesses
-
The reliability and contribution of the critic design are insufficiently established. The synthetic ranking labels assume that successful replay actions outperform perturbed actions and that smaller perturbations outperform larger ones. Their agreement with actual execution outcomes is not measured. The value visualizations show score differences, but do not establish that those differences reflect action quality. Since the critic remains frozen while the policy changes, its reliability on later policy-generated actions also needs examination. Furthermore, the comparison between unguarded and guarded ranking changes the temperature from 1 to 0.1. The reported improvement from 63.0% to 67.8% therefore cannot be attributed specifically to the guard.
-
The comparisons and practical validation are incomplete. LWD and Q-VGM are discussed as closely related policy-training methods but are not evaluated. Best-of-N and QGF instead modify inference, while Success BC and RECAP both reduce average performance relative to the initial policy. The reasons for these degradations and the baseline tuning procedures need clarification. All experiments are conducted in simulation, leaving effectiveness on real robot experience unverified. Although dexterous tasks are included, several remain unsuccessful or have very low success rates, and these limitations receive little analysis.
Questions for authors
-
How often do the synthetic ranking labels agree with returns measured by executing the corresponding actions? Does critic ranking remain reliable throughout policy optimization?
-
Can you compare guarded and unguarded ranking at the same temperature, and examine sensitivity to perturbation magnitude and negative-sample type?
-
Can you provide comparisons with LWD or Q-VGM, or explain why they are unsuitable? Please clarify the training schedules and checkpoint selection for Success BC and RECAP, and analyze their performance degradation.
3 comments
Thank you for the detailed response and additional experiments. The authors have adequately addressed my main concerns, and I have increased my rating accordingly.
Real robot experiment
Real-robot experiment
Hardware and policy interface
The platform consists of a Universal Robots UR5e arm with six revolute joints and a five-finger INSPIRE Robots RH56DFX-series dexterous hand with six actuator channels. A fixed external Orbbec Femto Bolt camera observes the workspace, and a wrist-mounted Intel RealSense L515 provides a close-up view. The policy receives RGB images from both cameras, resized to 224 × 224 pixels, along with the language instruction and the current robot state. The policy uses RGB only.
The proprioceptive state and the executed action each have 12 dimensions. The state contains the measured positions, and the action specifies absolute position targets in the same channel order.
| Dimensions | Component | Channel order and units |
|---|---|---|
| 1–6 | UR5e arm | Six arm joint positions in robot joint order, in radians |
| 7–12 | INSPIRE hand | Little finger, ring finger, middle finger, index finger, thumb flexion, and thumb rotation, in raw actuator counts |
At each policy call, π₀.₅ predicts a 32-step action chunk. Its internal representation is 32 × 32, and the first 12 channels at each step form the 32 × 12 action chunk passed to the robot controller. The deployment code supports a configurable execution horizon and defaults to 32 actions before the next policy call.
Initial policy and rollout collection
We used 50 demonstrations of the block-stacking task to fine-tune π₀.₅ for 30,000 optimizer steps with a global batch size of 256. We used AdamW with betas = (0.9, 0.95), eps = 1e-8, and weight decay of 1e-10. We clipped the gradient norm to 1.0. The learning rate warmed up linearly for 1,000 steps to 5 × 10⁻⁵, then followed a cosine decay schedule configured for the 30,000-step run, with an end value of 5 × 10⁻⁶.
The checkpoint at step 30,000 was the initial policy for this experiment. It succeeded in 9 of 20 real-robot trials. We used that checkpoint to collect 100 additional rollout episodes for offline post-training. The results below list the demonstration-trained initial checkpoint and the post-training SFT baseline separately.
Each method was evaluated on 20 real-robot trials. We report the number of successful trials and the corresponding empirical success rate.
| Method | Successful trials | Success rate |
|---|---|---|
| Initial policy (30k demonstration SFT) | 9/20 | 45% |
| SFT baseline | 11/20 | 55% |
| RECAP | 13/20 | 65% |
| Best-of-N | 11/20 | 55% |
| QGF | 10/20 | 50% |
| LWD | 13/20 | 65% |
| OVOGO | 16/20 | 80% |
We appreciate your careful review and your openness to our clarifications. Thank you for raising your score; your feedback has improved both the evaluation and the presentation of the paper.
Problem discussion and supplementary experiments
We thank the reviewer for the thoughtful suggestions on critic reliability, guard ablations, and policy-training baselines. We added perturbation execution, late-training Best-of-N, same-temperature guard comparisons, and LWD experiments across three environments.
Q1. Synthetic ranking labels and reliability during training
Thank you for this suggestion. Following our negative-sample construction, we add mild and strong noise to each base-policy action chunk, using the same evaluation initial states and seeds:
Success decreases as perturbations increase, supporting our assumption that stronger perturbations degrade effective actions more.
Second, Best-of-N with the frozen critic improves the initial policy from 60.7% to 64.23 ± 1.98%, and the final OVOGO actor from 68.17 ± 1.19% to 70.43 ± 0.65%. These execution results support synthetic perturbation ranking and the critic’s discrimination before and after optimization. Thank you for helping us strengthen this evidence.
Q2. Independent contribution of the guard and negative-sample sensitivity
Temperature and the guard jointly control ranking pressure on a return-grounded Q scale: temperature controls gradient decay, while the guard stops pressure after sufficient separation. Following your advice, we compared them at the same temperature:
Without the guard, lowering τ from 1 to 0.1 improves success by 3.33 percentage points. At τ=0.1, the guard adds 1.67 points. Temperature plays the main role in controlling ranking pressure, while the guard provides additional protection and gains.
We also added a LIBERO ablation: τ=1 without the guard achieves 71.05 ± 3.10%, the initial policy 71.40%, and full OVOGO 78.36 ± 2.00%. The 7.31-point gain supports the design in another environment. Q1 also supports our perturbation scales.
Q3. Related methods and baseline training details
Thank you for the suggestion. We reproduced LWD using the algorithm settings and hyperparameters reported in its paper. Results are shown in the comparison table below.
OVOGO exceeds LWD by 1.04, 4.50, and 3.15 percentage points, respectively. This adds a direct critic-guided policy-training comparison.
RECAP training code was unavailable, so we followed Evo-RL. We tried learning rates 1×10⁻⁵, 2×10⁻⁵, and 5×10⁻⁵ and batch sizes 32, 64, 128, and 256, selecting learning rate 2×10⁻⁵ and batch size 64 for expert-only training. For fair critic comparisons, we use Gemma 3 270M and SigLIP2 (224×224) encoders. The original setup used expert-only training with the VLM backbone frozen.
Success BC uses batch size 256 and learning rate 1×10⁻⁵, initialized from the public clean50 model already SFT-trained on 50 tasks. Further training on fewer successful trajectories may cause overfitting or forgetting, reducing closed-loop success. All training methods run for 10,000 steps; we evaluate the best saved checkpoint.
We added RECAP full fine-tuning, updating both the VLM backbone and action expert. Its success rates on RoboTwin, DexJoCo, and LIBERO are 64.23 ± 1.06%, 29.03 ± 3.44%, and 76.11 ± 0.50%, respectively, all above the initial policies. We will use these results as the RECAP baseline in the revised paper.
W2. Practical validation and difficult tasks
We added LIBERO, extending simulation evaluation to three environments. Real-robot experiments are underway, and we will provide results as soon as they are complete. Thank you for suggesting this validation.
Trained methods use three seeds; results are mean ± standard deviation. The initial policy is the pretrained checkpoint. Success rates (%):
RECAP uses full fine-tuning. OVOGO has the highest mean success in all three environments, with improvements maintained across seeds.
The official SFT model also struggles on near-zero-success DexJoCo tasks. With sparse terminal rewards, few successful rollouts limit the return signal and subsequent RL improvement. Thank you for highlighting these tasks; we will add this analysis to the revision.