acceptodds
Withdrawn from ICLR 2027

T-Rex-RL: Improving Tactile-Reactive Dexterous Manipulation with Real-World Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tactile-reactive vision-language-action (VLA) models combine fast closed-loop tactile control with the broad visuomotor priors acquired through large-scale pretraining. However, existing models are mainly trained with imitation learning on teleoperated demonstrations, limiting their ability to improve beyond the quality and coverage of offline data. In this work, we study how tactile-reactive VLAs can improve via real-world interaction using offline-to-online reinforcement learning (RL). To achieve this, we initialize a tactile-reactive actor and critic through offline RL with teleoperation data, then continuously update them using online robot rollouts. The VLA is pretrained on large-scale data, whereas RL post-trains it on limited task-specific robot experience. We therefore freeze the VLA and add the RL actor and critic as Mixture-of-Transformers (MoT) experts that attend to its representations. While the VLA predicts action plans at a low frequency, the RL actor refines them at a high frequency from tactile signals to react to contact changes. The critic evaluates the refined actions to guide the actor's learning. We evaluate our approach on a suite of long-horizon bimanual dexterous manipulation tasks, where our method substantially outperforms baselines.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Dexterous Manipulation, Tactile, Real World Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G71XOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the limitation that existing tactile-reactive Vision-Language-Action (VLA) models are mainly trained via imitation learning on teleoperated demonstrations, which prevents them from improving beyond the quality and coverage of offline data. To overcome this, the authors propose an offline-to-online reinforcement learning (RL) approach in which a tactile-reactive actor and critic are first initialized through offline RL with teleoperation data and then continuously updated using online robot rollouts. Since the VLA is pretrained on large-scale data while RL post-trains it on limited task-specific robot experience, the VLA is frozen and the RL actor and critic are added as Mixture-of-Transformers (MoT) experts that attend to its representations. The VLA predicts action plans at low frequency, while the RL actor refines them at high frequency from tactile signals to react to contact changes, and the critic evaluates the refined actions to guide the actor's learning. Evaluated on a suite of long-horizon bimanual dexterous manipulation tasks, the method substantially outperforms baselines.

Critical strengths
  1. I consider it reasonable and effective to adopt the real-world residual RL approach to address the task in this paper.
  2. The experiments in the paper provide strong validation of the effectiveness of RL as well as tactile sensing.
  3. The experiments in the paper are considerably challenging, yet the proposed method still achieves a satisfactory success rate.
Critical weaknesses
  1. The paper proposes a real-world RL algorithm, yet all baselines are VA/VLA models.
  2. The description of the model architecture is not sufficiently specific.
  3. Real-world residual RL was not first proposed in this paper, so the work lacks sufficient novelty.
Questions for authors
  1. I think the method should be compared with some mainstream real-world RL approaches.
  2. Real-world residual RL is not proposed for the first time in this paper; it is recommended to clarify what unique designs the residual RL in this work has compared with existing methods.
  3. The architecture diagram of the model is relatively simple, and the description is somewhat insufficiently specific; it is suggested that the model be re-described.
  4. Why is a milestone-based reward design adopted instead of a progress reward?
  5. Would end-effector 6-D force information, rather than tactile sensing, also be helpful for accomplishing these tasks?
  6. Does the method involve human intervention, or does it learn purely online by itself?
  7. How is the gripper dimension specifically handled for RL?
Reviewer_CLYqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed an extension to T-Rex by adding a residual RL module. In addition to the pre-training and mid-training in T-Rex, T-Rex-RL adds a online RL stage with RLPD. On real-world robots and five tasks, they show significant improvement compared to supervised baselines.

Critical strengths
  1. The paper has good empirical results. In particular, online RL on humanoids takes a lot of effort.
Critical weaknesses
  1. The paper is incremental compared to T-Rex, adding the neural module and the training stage for RL. The design decisions for RL are not well articulated and there lacks other RL baselines for comparison. The baseline, as the authors described themselves, are "supervised trained". The ablation shows having RL and tactile information is beneficial, but it doesn't say anything about whether the RL design choices are good.

  2. The method section is not well written. It lacks principled mathematical description and basic information is missing, e.g., what's the algorithm for offline RL midtraining? The design decisions are not grounded in either prior work or empirical evidence through ablation, e.g., why use PCA for tactile information? Is it better than VQ-VAE in T-Rex?

Questions for authors
  1. None of the results in Table 1, Table 2, and Figure 4 have confidence intervals. The paper states that "we evaluate each policy on 16 new episodes per task with a fixed checkpoint", which explains Table 1. But, how about the results in Table 2 and Figure 4. How many seeds are the online learning experiments averaged over?
  2. Can you articulate the rationale behind the key RL design decisions, including design of the RL module, PCA for tactile information, choice of learning algorithms?
Reviewer_Cgo3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

T-Rex-RL improves a frozen tactile-reactive VLA through offline-to-online residual RL. The VLA generates action plans at a low frequency, while a tactile-conditioned RL actor refines them more frequently to respond to contact changes. The method is evaluated on five real-world dexterous manipulation tasks.

Critical strengths
  • The experiments address several manipulation tasks, with improvements in average task progress over the evaluated imitation-learning baselines.
  • The tactile and asynchronous-refinement ablations help establish the contribution of frequent contact feedback.
Critical weaknesses
  • The improvement in task progress is good, but does not necessarily imply reliable task completion. For example, the pen-extraction score remains only 19%. Full-task success rates would make the practical significance of the gains clearer.
  • The supervised baselines are relevant for measuring improvement beyond imitation, but insufficient to establish an advantage over existing RL approaches. A comparable residual-RL baseline with the same observations and interaction budget is needed to assess the proposed architecture.
  • Results are averaged over 16 evaluation episodes per task, without uncertainty estimates or reported variation across independent training runs. This makes smaller differences, such as the seven-point gain from asynchronous refinement, difficult to assess.
Questions for authors
  • What are the full-task success counts for each method, and how much of the improvement comes from completing additional intermediate milestones rather than finishing the task?
  • How consistent are the results across independent learning runs, particularly the gain from asynchronous refinement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.