- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Learning dexterous manipulation from limited visual demonstrations is a major challenge, since policies must not only understand perceptual information but also possess fine-grained control. Reinforcement learning (RL) in simulation, by contrast, can leverage privileged information and accumulate rich control behaviors through large-scale interaction. Motivated by this, the authors propose Latent RL Prior, a two-stage framework designed to turn simulation-acquired control knowledge into data-efficient visual policies. The first stage, privileged pretraining, distills large-scale RL rollouts into a flow-matching action decoder conditioned on proprioception and latent task context. The second stage, visual adaptation, keeps the decoder frozen and trains only a visual interface via action supervision, mapping images to the corresponding latent context without privileged labels or latent targets. Moreover, the same decoder can be shared across arms by learning coordinated visual contexts for bimanual manipulation.
Critical strengths
- Latent RL prior achieves better performance than the baselines with less data.
- The method is simple and effective, and the paper is clearly written and easy to understand.
- The ablation studies effectively demonstrate the benefit of the prior.
Critical weaknesses
- The additional computational resources and training time introduced by the Latent RL prior are not reported.
- There is no validation through real-robot experiments.
- There is no discussion of whether the method offers generalization, for example, in coping with environmental changes.
Questions for authors
- I think this is a good idea, but it needs to be validated on a widely recognized benchmark or through real-robot experiments.
- How much additional computational resource and training time does the introduction of the Latent RL prior incur?
- Does the method provide a certain degree of generalization?
- Are the three tasks handled by a single model or by one model per task? Is the RL prior shared across the three tasks?
- What kind of network architecture is the Q-Former?
2 comments
Thank you for your response. I think you have solved all my questions. I will raise my score to 3.
Thank you for your time and for recognizing the simplicity of the method and the clarity of the paper. We appreciate your observation that the policy performed well with fewer visual demonstrations and that the ablations showed the contribution of the RL prior.
We reused the released SimToolReal teacher, whose authors report approximately 12 days on one A6000 GPU (288 GPU-hours) for the final Isaac Gym training run (General Response 3). We fine-tuned the checkpoint in MuJoCo Warp for 500 epochs with 18,432 parallel environments. This took 2.5 hours on one RTX PRO 6000 and raised the teacher’s success rate from 57% to 80.4%.
We then collected 100,000 successful simulated state–action rollouts and trained one action decoder. All four downstream tasks share this frozen decoder, so we incurred the distillation cost once.
We agree that evaluation on one simulated platform limits our conclusions. The SimToolReal teacher commands a KUKA iiwa in joint space, and our physical robot uses a different arm. We expect the main transfer challenge to be matching the arm and hand dynamics during contact. We therefore plan to calibrate the simulator to our hardware, retrain the RL teacher, and distill a new decoder. With the decoder frozen, we would train the visual interface on real camera images using only the teleoperator’s actions as supervision (General Response 4). Hardware validation and evaluation on an independent benchmark remain future work.
We added Grasp, a fourth simulated task that reuses the same frozen decoder as the original three tasks. The robot must lift a tabletop object at least 10 cm and hold it steady for 1.5 s. We used 96 training objects and 24 held-out objects from Google Scanned Objects, YCB, and Objaverse-LVIS. Our policy achieved higher grasp success than both baselines on training and held-out objects at all four demonstration budgets.
Grasp success (%) on training / held-out objects. Ours reports mean ± SD over three training seeds. Each baseline uses one seed.
Our policy generalizes to object masses unseen during visual adaptation (Fig. 8). Each task’s demonstrations used a single reference mass. We tested pose alignment and door lifting at higher masses without further training. Door-lifting success was similar at the reference mass. At 5.0 kg, our policy retained 32.6% success with 50 demonstrations, compared with 12.9% for π₀.₅ with 500. These results suggest that the visual policy inherits robustness to mass changes from the frozen RL prior, whose large-scale pretraining included randomized object densities.
The four downstream tasks share one frozen action decoder. Each learns its own visual interface.
The BLIP-2-style Q-Former has four pre-norm blocks (width 384, six heads, feed-forward width 1,024). In each block, learned queries attend to one another, cross-attend to DINOv2 ViT-S/14 patch tokens with learned camera embeddings, and pass through a feed-forward layer.
Door lifting uses six queries. Bimanual assembly uses one joint Q-Former with 12 queries, six per arm. Self-attention lets the two arms’ queries exchange information, and every query cross-attends to the front and both wrist views. We split the output into six context tokens per arm. The same frozen single-arm decoder predicts each arm’s action from its tokens and proprioception.