- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses the limitation that existing tactile-reactive Vision-Language-Action (VLA) models are mainly trained via imitation learning on teleoperated demonstrations, which prevents them from improving beyond the quality and coverage of offline data. To overcome this, the authors propose an offline-to-online reinforcement learning (RL) approach in which a tactile-reactive actor and critic are first initialized through offline RL with teleoperation data and then continuously updated using online robot rollouts. Since the VLA is pretrained on large-scale data while RL post-trains it on limited task-specific robot experience, the VLA is frozen and the RL actor and critic are added as Mixture-of-Transformers (MoT) experts that attend to its representations. The VLA predicts action plans at low frequency, while the RL actor refines them at high frequency from tactile signals to react to contact changes, and the critic evaluates the refined actions to guide the actor's learning. Evaluated on a suite of long-horizon bimanual dexterous manipulation tasks, the method substantially outperforms baselines.
Critical strengths
- I consider it reasonable and effective to adopt the real-world residual RL approach to address the task in this paper.
- The experiments in the paper provide strong validation of the effectiveness of RL as well as tactile sensing.
- The experiments in the paper are considerably challenging, yet the proposed method still achieves a satisfactory success rate.
Critical weaknesses
- The paper proposes a real-world RL algorithm, yet all baselines are VA/VLA models.
- The description of the model architecture is not sufficiently specific.
- Real-world residual RL was not first proposed in this paper, so the work lacks sufficient novelty.
Questions for authors
- I think the method should be compared with some mainstream real-world RL approaches.
- Real-world residual RL is not proposed for the first time in this paper; it is recommended to clarify what unique designs the residual RL in this work has compared with existing methods.
- The architecture diagram of the model is relatively simple, and the description is somewhat insufficiently specific; it is suggested that the model be re-described.
- Why is a milestone-based reward design adopted instead of a progress reward?
- Would end-effector 6-D force information, rather than tactile sensing, also be helpful for accomplishing these tasks?
- Does the method involve human intervention, or does it learn purely online by itself?
- How is the gripper dimension specifically handled for RL?