- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents TouchWorld, a hierarchical system for tactile dexterous manipulation that separates task-level reasoning from fast contact correction. At the highest-level, a subtask planner selects the current subtask and a tactile world model predicts a visual-tactile subgoal for it. A goal-conditioned VLA then generates action chunks from these goals and current observations. Finally, a Tactile Residual Transformer refines those chunks at a higher rate using recent tactile and proprioceptive history. On six real-robot tasks, TouchWorld achieves the highest task success rates.
Critical strengths
- The motivation is clear and relevant. A hierarchical system that separates high-level task planning and goal prediction from reactive contact correction addresses a real need in dexterous manipulation, where tasks require both reasoning about the intended contact outcome and responding quickly when actual contact deviates from it
- The evaluation is done on a real robot with a reasonable number of trials and task coverage
- The paper is overall well-written and easy to follow
Critical weaknesses
1. Baseline comparison
The paper argues that monolithic visuo-tactile policies are limited, but there is no monolithic baseline that shares TouchWorld’s backbone and training data. FTP-1 is the closest comparison, but it differs in several ways at once. The paper does not describe how the baselines were trained, for example whether they used the same 200 trajectories per task. Because of this, it’s hard to tell how much of the gain in Table 1 comes from the architecture and how much from the additional data and labels.
2. Tactile representation and human-to-robot transfer
I couldn’t find a concrete description of how human and robot tactile signals are represented as images; the text says only that they are “rendered or normalized into images.” The figures give some hints, but the tactile in Fig. 2 (hand-shaped maps) and Fig. 6 (patch grids) looks quite different. Since both the world model and the VLA operate on this representation, a clear description seems necessary.
Relatedly, it’s unclear how this image representation carries over between human and robot hands, which have different sensor layouts. Some explanation of how the two are aligned would help. Sec. 2.4 also states that pretraining "transfers human contact priors," but I didn’t find an experiment supporting this, such as a world model trained on robot data only.
3. Ablation design and TRT training
According to Appendix B.5, the planner ablation reuses checkpoints without retraining. The VLA was trained with subtask text and subtask-level goals but receives task-only inputs in this ablation, so part of the drop may come from the input mismatch rather than the loss of planning. The same question applies to the world-model ablation. It would help to clarify which variants were retrained, or to retrain them.
The TRT’s residual targets are computed on states from the teleoperation demos. At deployment, it operates on states the robot reaches itself, which may differ from the demos, particularly under perturbations, which is where it’s meant to help. Was any on-policy or perturbation data used during training? If not, some discussion of where the perturbation gains come from would be useful.
4. Evaluation protocol
The paper doesn’t give per-task success criteria. The human perturbations are listed by type, but their strength, timing, and consistency across methods aren’t described. No confidence intervals are reported. With 100 rollouts per task, some of the smaller differences, such as the 2.5-point world-model gap under perturbations, may be within noise. A written protocol and confidence intervals would make the results easier to trust.
Questions for authors
- Which model is the nominal policy’s vision-language backbone, and what was it pretrained on? The paper describes it only as "a vision-language backbone and a diffusion Transformer action expert." Knowing this would help reproducibility and the interpretation of results.
- How were the baselines trained? Did they use the same 200 trajectories per task and a comparable fine-tuning budget, and did any of them have access to subtask annotations, or access to the EgoTrack dataset?
- How exactly are the human and robot tactile signals rendered into images, and how are they aligned given the different hand and sensor layouts? Which representation do Fig. 2 and Fig. 6 show?
7 comments
Part 2/5: A same-backbone monolithic visuo-tactile baseline; additional supervision (W1, continued)
W1 (continued) — Does the gain come from the architecture, or from additional data and labels?
A monolithic visuo-tactile baseline with the same backbone (W1). Following your comment, we add a monolithic visuo-tactile policy that shares TouchWorld's backbone and data:
- Setup. It is initialized from π0.5 weights and takes the same current tactile image as TouchWorld as an additional input, without the Planner, predicted goals, or TRT.
- Training and evaluation. It is fine-tuned on the same 200 demonstrations per task for 30,000 steps with global batch size 32; the evaluation protocol is that of Table 1, with 100 rollouts per task per condition.
| Policy | Initialization | Tactile input | Clean success (%) | Perturbed success (%) |
|---|---|---|---|---|
| π0.5 (Table 1) | π0.5 | None | 40.7 | 27.7 |
| π0.5 + tactile image (new) | π0.5 | Current tactile image | 42.5 | 29.3 |
| FTP-1 (Table 1) | FTP-1 | Native tactile interface | 49.3 | 35.2 |
| TouchWorld | π0.5 | Current tactile image and TRT tactile history | 65.0 | 53.7 |
- TouchWorld is 22.5/24.3 percentage points above the monolithic visuo-tactile policy with the same backbone. With the same backbone, the same robot data, and the same tactile sensor, this gap comes from the hierarchy.
- Adding the tactile image to a monolithic policy brings only a limited gain. Adding the tactile image to π0.5 raises clean/perturbed success by only 1.8/1.7 percentage points. This is consistent with the submission's argument that monolithic visuo-tactile policies are limited.
- FTP-1 is a stronger monolithic reference. According to its official release, FTP-1 is pretrained on approximately 3,000 hours of tactile manipulation data. TouchWorld is 15.7/18.5 points above it.
Adding the modules one level at a time. To separate the contributions, we train one more level on top of the monolithic baseline above: it adds subtask planning, i.e., the text input of the nominal policy includes the current subtask, which the Planner provides at deployment; it uses neither predicted goals nor TRT, with the same data and training budget. Together with existing results it gives the table below, in which every level is trained in its own configuration and involves no input mismatch:
| Configuration | Clean success (%) | Perturbed success (%) | Gain over the previous level |
|---|---|---|---|
| π0.5 + tactile image (monolithic, new) | 42.5 | 29.3 | — |
| + subtask planning (new) | 51.0 | 37.8 | 8.5/8.5 |
| + world-model goals ("No TRT" in Figure 5) | 55.3 | 40.3 | 4.3/2.5 |
| + TRT (full system) | 65.0 | 53.7 | 9.7/13.3 |
- Subtask planning and TRT contribute the most, followed by the world-model goals. Subtask planning brings 8.5/8.5 percentage points, the world-model goals 4.3/2.5 points, and TRT 9.7/13.3 points.
- The effect of the subtask annotations is measured on its own. It appears at the second level and accounts for 38%/35% of the total gain of 22.5/24.3 percentage points; the rest comes from the predicted goals and the tactile residual correction.
What the subtask annotations and EgoTouch are used for. Both are used only to train TouchWorld's own Planner and world model. They are part of the method, and both are additional to the action-policy fine-tuning budget above. The three baselines are monolithic policies trained in their original form on the full task instruction, and do not involve either of them:
- Subtask annotations. The Planner is fine-tuned separately on 128,866 records from annotated teleoperation traces (LoRA rank 16, learning rate 10^-4); the text input of the nominal policy includes the current subtask.
- EgoTouch human data. Used only for pretraining the world model. The world model is then adapted on 10 hours of robot demonstrations (rank-64 LoRA, learning rate 10^-4) and is trained before the nominal policy; the terminal visual–tactile goals it generates condition the nominal policy during training and deployment.
The gain does not come only from these two sources of supervision. With either one removed, the system remains above FTP-1, the strongest baseline:
- Without EgoTouch. With a world model trained on robot data only (see Part 3), the success rate is 62.8%/52.3%, which is 13.5/17.2 percentage points above FTP-1.
- Without subtask planning. With the Planner removed, the nominal policy receives only the full task instruction. Even though this differs from the inputs it was trained with, the success rate is still 55.5%/46.2%, which is 6.2/11.0 points above FTP-1.
Part 3/5: Tactile representation and human pretraining (W2 / Q3)
W2 / Q3 — How are tactile signals represented as images? How are human and robot hands aligned? Does human pretraining help?
Figures 2 and 6. Figure 6 shows the representation the system actually uses; the hand-shaped drawing in Figure 2 is only an illustration of tactile sensing. We will state this distinction in the captions of both figures.
Rendering. Touch uses the representation format of TouchAnything (EgoTouch):
- From readings to a grid. The glove on each hand outputs 256 raw readings. After the baseline of the first frame is subtracted, the readings are placed, according to the position of each sensing unit on the hand, into a 21 × 21 hand-shaped grid; 217 cells correspond to sensing units on the thumb, the four fingers, and the palm, and the remaining positions have no sensor.
- Normalization. The grid is divided by the pressure upper bound computed for the task and clipped to [0, 1].
- Coloring. Each cell is colored by its value with a blue–cyan–yellow–red color map, without interpolation, which gives the colored patches in Figure 6; positions without sensors take the background color. The grids of the left and right hands are placed side by side in one tactile panel.
- How it enters the models. The input and output of the world model is a 384 × 224 image arranged as a 2 × 2 grid, whose four tiles are the head view, the two wrist views, and the tactile panel above. The nominal policy takes the current tactile image as a separate image, together with the three camera images (all resized to 224 × 224), rendered in the same way.
Alignment between human and robot hands. Human data (EgoTouch) and robot data use the same grid format and the same rendering pipeline, so they can enter the same model without an additional mapping. The shapes of the human and robot hands and the physical positions of the sensing units differ (the different sensor layouts noted in Sec. 2.4); this difference is not handled by a unit-to-unit correspondence but by two-stage training: the world model is first pretrained on human data and then adapted on robot data.
Effect of human pretraining. Following your suggestion, we add a world model trained on robot data only and compare it with the original model in both prediction quality and closed-loop success:
- Setup. It starts from the same Wan initialization and uses the same robot data and adaptation budget, omitting only EgoTouch pretraining.
- Prediction quality. Evaluated on held-out subgoal segments in the same way as Table 2 of the submission.
- Closed-loop evaluation. The nominal policy, the Planner, and TRT are kept unchanged, with 100 rollouts per task per condition.
Tactile prediction quality on held-out subgoal segments (%). The first two rows and the last row are from Table 2 of the submission.
| Method | Temporal contact accuracy | Terminal contact IoU | Volumetric IoU |
|---|---|---|---|
| Current tactile copy | 70.4 | 31.8 | 24.6 |
| Nearest-neighbor retrieval | 77.5 | 39.2 | 31.0 |
| World model: robot adaptation only | 82.4 | 46.8 | 37.9 |
| World model: EgoTouch pretraining + robot adaptation | 86.3 | 52.7 | 43.8 |
Closed-loop success (%). The first row is from Figure 5 of the submission.
| Configuration | Clean success (%) | Perturbed success (%) |
|---|---|---|
| No world-model goals | 60.2 | 51.2 |
| World model: robot adaptation only | 62.8 | 52.3 |
| World model: EgoTouch pretraining + robot adaptation | 65.0 | 53.7 |
- Human pretraining improves the prediction of robot subgoals. Terminal contact IoU rises from 46.8% to 52.7%, volumetric IoU from 37.9% to 43.8%, and temporal contact accuracy from 82.4% to 86.3%. This corresponds directly to the statement in Sec. 2.4 that human contact priors transfer to robot subgoal prediction.
- The change in closed-loop success is consistent with this. The world-model goals bring a total gain of 4.8/2.5 percentage points, of which 2.2/1.3 points come from human pretraining, about half (computed before rounding).
We will add both sets of results to the revision.
Part 1/5: Overview; backbone and baseline training (W1 / Q1 / Q2)
Dear Reviewer BCLp,
Thank you for recognizing that the motivation is clear and relevant, that the evaluation is done on a real robot with a reasonable number of trials and task coverage, and that the paper is well written. Thank you also for the careful and specific comments. We first summarize our answer to each point and then respond in detail.
Overview
W1 / Q1 / Q2 — Which backbone does the nominal policy use? How were the baselines trained? Does the gain come from the architecture, or from additional data and labels?
Backbone (Q1). The nominal policy uses the π0.5 architecture, i.e., PaliGemma with a Gemma-2B language backbone plus a Gemma-300M flow-matching action expert, and is initialized from the officially released π0.5 base checkpoint. According to the π0.5 paper, these weights are co-trained on manipulation data from multiple robot embodiments and on web image–text data. From this initialization, the nominal policy is fine-tuned for 30,000 steps with global batch size 32 and predicts 32-step, 120-dimensional action chunks. The Planner starts from Qwen3-VL-4B-Instruct; the world model starts from Wan2.2-TI2V-5B, is first pretrained on EgoTouch human visual–tactile data, and is then adapted on robot data.
Baseline training (Q2).