- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TF²-Steering, a visual-tactile robot manipulation framework that combines tactile forecasting and real-time tactile feedback to refine actions generated by a pretrained flow-matching policy.
The method introduces a tactile world model to predict future tactile representations and a tactile correction expert that learns residual velocity-field corrections during the flow-matching action generation process. An asymmetric attention mechanism and gradient isolation are employed to preserve the pretrained visual action generation capability.
The framework further adopts a dual-rate asynchronous inference mechanism, where low-frequency world models provide cached predictive priors and high-frequency action experts incorporate real-time tactile feedback for action refinement. A three-stage training procedure is used for tactile prediction pretraining, tactile velocity correction pretraining, and task-specific fine-tuning.
Experiments on six simulated and six real-world manipulation tasks demonstrate improvements over several visual and visual-tactile baselines. The method achieves an average simulation success rate of 68.3% and an average real-world progress success rate of 71.5%. Ablation studies investigate the contributions of tactile forecasting, real-time feedback, velocity-field correction, and asynchronous inference.
Critical strengths
- Practical integration of tactile forecasting and real-time feedback.
The paper presents a reasonably well-designed framework that integrates predicted tactile information and real-time tactile feedback into a pretrained robot policy. The asymmetric attention and gradient isolation mechanisms provide a practical way to incorporate tactile corrections while preserving the original visual action generation capability. The asynchronous dual-rate architecture is also meaningful for contact-rich manipulation, where tactile feedback and high-level planning operate at different temporal scales.
- Promising empirical results on diverse manipulation tasks.
The paper evaluates the proposed framework on six simulated and six real-world manipulation tasks, covering a range of contact-rich interactions. The improvements on precision insertion, bottle-cap twisting, zipper closing, and other dexterous tasks are encouraging. The ablation studies also provide useful evidence that real-time tactile feedback and intermediate action-generation refinement contribute to the observed performance improvements.
Critical weaknesses
- Limited methodological novelty of the proposed velocity-field steering mechanism.
The central technical formulation, v_refined = v_base + Δv_tactile, is essentially a learned residual correction to the flow-matching velocity field. This is closely related to existing guidance and velocity-field adaptation techniques in diffusion and flow-matching models. Although it is not mathematically identical to classifier guidance, the underlying idea of modifying the generative vector field using an additional conditional signal is well established.
The tactile expert, asymmetric attention, gradient isolation, and dual-rate inference mechanisms are also closely related to existing architectural and control strategies. Therefore, the paper appears to primarily combine established techniques rather than introduce a fundamentally new learning or generation mechanism.
More importantly, the experimental evaluation does not sufficiently isolate the methodological contribution of velocity-field steering. The comparison against action-space residual correction is useful but insufficient. Stronger baselines, such as standard tactile-conditioned flow matching or a conventional learned velocity-field residual adapter under matched training and computational settings, are needed to demonstrate that the proposed formulation offers advantages beyond existing guidance mechanisms.
- Insufficient evidence for the unique contribution of tactile forecasting and the robustness of the reported improvements.
The paper emphasizes the collaborative use of predicted tactile information and real-time tactile feedback. However, the ablation results indicate that removing predicted tactile information reduces the average progress success rate by only 2.7 percentage points, whereas removing real-time tactile feedback causes a substantially larger decrease of 9.3 percentage points. This raises questions about the actual contribution of the tactile world model relative to its architectural and computational complexity.
Furthermore, the real-world evaluation primarily reports progress success rate rather than full task completion success rate. While progress-based evaluation is useful, it does not directly establish the reliability of successful task execution. The evaluation also uses only 20 trials per task without reporting uncertainty estimates.
Finally, the comparison with T-REX may be influenced by differences in feedback update schedules, since the paper explicitly notes that T-REX uses a fixed refinement interval in the evaluation. Additional controlled comparisons would be helpful to determine whether the improvements arise from the proposed steering formulation, the tactile forecasting component, or differences in inference and feedback scheduling.
Questions for authors
- What is the fundamental methodological difference between the proposed velocity-field steering and existing learned guidance or residual velocity-field adaptation methods?
The formulation appears to be a standard residual correction of the flow-matching vector field. Could the authors provide comparisons with conventional tactile-conditioned flow matching or learned velocity-field residual adapters using the same backbone, tactile inputs, and comparable parameter and training budgets? Such comparisons would help clarify the actual novelty and effectiveness of the proposed steering mechanism.
- How essential is the tactile forecasting component?
According to Table 3, removing predicted tactile information causes a relatively small performance decrease compared with removing real-time feedback. Could the authors clarify under which contact conditions or manipulation scenarios tactile forecasting provides a substantial advantage over reactive tactile feedback alone? In particular, does the forecast improve contact anticipation, reduce correction latency, or enable behaviors that cannot be achieved with a purely reactive tactile policy?
- Can the authors provide more rigorous real-world evaluation and controlled baseline comparisons?
It would be helpful to report full task completion success rates in addition to progress success rates, together with uncertainty estimates across evaluation trials. Also, since the comparison with T-REX uses a fixed tactile refinement interval, could the authors evaluate different feedback update frequencies under comparable computational budgets to better isolate the benefits of the proposed approach?
- What is the actual end-to-end inference latency of the asynchronous steering system?
The paper reports a 240 ms action-chunk generation time and describes a dual-rate system with 5 Hz world-model updates and 30 Hz action execution. Could the authors clarify the measured latency of each tactile-guided action regeneration, including tactile encoding and flow-matching iterations, and the effective correction frequency achieved during real-world execution?
3 comments
Response to Reviewer G9jW (2/2)
[Q3 & W2] Can the evaluation include full-task success rates, uncertainty estimates, and baseline comparisons across feedback frequencies under comparable compute budgets?
Thanks for this suggestion. Table G is computed from the original evaluation records underlying Table 2 in the main paper, with 20 trials per method per task. We additionally report PSR as mean ± standard error (SEM) to quantify uncertainty, alongside full-task success rates.
Table G. PSR (mean ± SEM) / full-task success rate (%).
| Method | Fruit placement | Zipper closing | Board wiping | Bottle-cap twisting | Cup separation | Dropper transfer | Avg. PSR | Avg. SR |
|---|---|---|---|---|---|---|---|---|
| ViTacFormer | 12.5±2.9 / 0.0 | 4.0±1.8 / 0.0 | 2.0±1.4 / 0.0 | 9.0±2.3 / 0.0 | 7.0±2.2 / 0.0 | 2.0±1.4 / 0.0 | 6.1 | 0.0 |
| π0.5 | 60.0±7.1 / 25.0 | 18.0±3.8 / 0.0 | 36.0±7.8 / 20.0 | 26.0±4.1 / 0.0 | 62.0±6.1 / 25.0 | 39.0±7.6 / 5.0 | 40.2 | 12.5 |
| π0.5 + tactile | 57.5±6.6 / 20.0 | 12.0±3.4 / 0.0 | 39.0±8.9 / 25.0 | 22.0±4.1 / 0.0 | 57.0±7.1 / 25.0 | 23.0±6.2 / 10.0 | 35.1 | 13.3 |
| T-Rex | 72.5±6.5 / 40.0 | 56.0±6.4 / 20.0 | 61.0±7.7 / 35.0 | 58.0±8.7 / 35.0 | 87.0±5.9 / 75.0 | 40.0±8.5 / 15.0 | 62.4 | 36.7 |
| FTP-1 | 75.0±6.5 / 45.0 | 36.0±4.5 / 0.0 | 55.0±7.9 / 30.0 | 40.0±6.2 / 10.0 | 64.0±6.4 / 30.0 | 46.0±9.0 / 25.0 | 52.7 | 23.3 |
| TF²-Steering | 70.0±7.6 / 50.0 | 60.0±7.1 / 30.0 | 76.0±7.3 / 60.0 | 67.0±7.9 / 45.0 | 90.0±4.9 / 80.0 | 66.0±8.0 / 35.0 | 71.5 | 50.0 |
TF²-Steering supports adjustable feedback intervals at inference time to accommodate different tasks. Table E evaluates three tasks with updates every 2, 4, 6, or 8 action steps. Bold entries indicate the settings used in Table 2 of the main paper.
Table E. Feedback-update interval ablation. Entries report PSR (mean ± SEM) / full-task success rate (%).
| Update interval k (action steps) | Bottle-cap twisting | Dropper transfer | Cup separation | Avg. PSR | Avg. SR |
|---|---|---|---|---|---|
| 2 | 59.0±8.0 / 35.0 | 63.0±8.2 / 35.0 | 85.0±6.6 / 75.0 | 69.0 | 48.3 |
| 4 | 67.0±7.9 / 45.0 | 66.0±8.0 / 35.0 | 88.0±5.1 / 75.0 | 73.7 | 51.7 |
| 6 | 65.0±7.7 / 40.0 | 65.0±8.6 / 30.0 | 88.0±5.1 / 75.0 | 72.7 | 48.3 |
| 8 | 60.0±8.1 / 35.0 | 64.0±8.4 / 40.0 | 90.0±4.9 / 80.0 | 71.3 | 51.7 |
Updating every 2 steps yields lower PSR than every 4 steps on all three tasks, suggesting that overly frequent action regeneration can be counterproductive, potentially due to accumulated correction errors. An interval of 4 provides consistently good performance across tasks. For cup separation, increasing the interval to 8 achieves the highest PSR and full-task success rate while requiring fewer inference calls, illustrating the benefit of adapting the update interval to the task.
[Q4] What is the latency of tactile-guided action regeneration, and how frequently are action commands sent?
Thanks for raising this question. On a single RTX 4090, each tactile-guided action regeneration takes 72.19 ms on average, including tactile encoding and all 10 flow-matching steps. Tactile encoding takes 12.15 ms and is already included in the total.
For a fast-update interval of action steps, one cycle includes an update taking 72.19 ms and steps at the nominal 30 Hz control rate. The estimated cycle duration and average action-command frequency are:
\qquad f_{\mathrm{cmd}}(k)=\frac{k}{T(k)}\ \text{Hz}.$$ **Table F. Estimated average action-command frequency at different fast-update intervals.** | Fast-update interval $k$ | Cycle duration (ms) | Average action-command frequency (Hz) | |---|---:|---:| | 2 | 105.52 | 18.95 | | 4 | 172.19 | 23.23 | | 6 | 238.86 | 25.12 | | 8 | 305.52 | 26.18 | These estimates assume cached world-model priors and exclude sensor acquisition, communication, scheduling, and chunk-initialization overhead. The **30 Hz shown in the pipeline is the nominal control rate**; we will clarify this in the figure caption.Official comment by authors
Dear Reviewer G9jW, thank you for your time and thoughtful feedback. Since the review period ends today, we would greatly appreciate it if you could take a moment to review our response and update your comments or score if appropriate. Thank you again for your consideration.
Response to Reviewer G9jW (1/2)
[Q1 & W1] Clarify the fundamental methodological difference between the proposed velocity-field steering and existing residual velocity-field adaptation methods, and compare them using the same backbone, matched parameter counts, and comparable training budgets.
Thank you for raising this important question. Classifier-guidance-style methods train a classifier or scoring model and use its gradients with respect to intermediate samples to steer generation toward higher scores. TouchGuide follows this general approach for action generation: it uses contrastive learning to train a scoring model that assigns higher scores to actions paired with matching tactile and visual observations. During flow-matching inference, the score gradient with respect to the candidate actions provides a velocity correction that guides subsequent sampling. TF²-Steering shares the additive velocity-correction form but constructs the correction differently. Rather than deriving it from a score gradient at inference time, our tactile expert directly predicts Δv from tactile inputs and action context, trained end-to-end through imitation learning with a flow-matching loss.
Regarding residual velocity-field adaptation, we agree that TF²-Steering is an instance of this approach for tactile-guided robotic manipulation. Our contribution lies in adapting this formulation to tactile control. We adopt this design because it allows additional conditioning while retaining the base policy’s ability to operate independently, making it well suited to fine-grained action corrections driven by tactile feedback. We discuss the novelty and motivation of TF²-Steering in detail in C2.
In C1, we compare TF²-Steering (E1), learned score-gradient guidance (E5), and tactile-conditioned flow matching (E3) using the same world-action backbone. Table A details their architectures, tactile inputs, and training configurations. E1 and E3 receive the same Stage 2 training budget, and E1 achieves higher PSR on all three tasks, averaging 72.7% versus 69.0%. The gap widens to 11.0 percentage points when touch is disabled without retraining, with E1 achieving 63.3% and E3 achieving 52.3%. This supports the benefit of a separate correction branch in retaining visual-only task performance. In our reproduction, E5 achieves 60.7% average PSR, close to the vision-only E0 baseline at 60.3%, indicating limited benefit from gradient guidance in the tested configuration. E5 uses only real-time touch and a separately trained scoring model, as specified in C1.
[Q2 & W2] When does predicted tactile information improve task performance beyond real-time tactile feedback?
Thanks for raising this question. We believe tactile prediction is particularly useful for improving task efficiency and recovering faster from failed contact attempts. In bottle-cap twisting, the policy sometimes fails to grasp the cap and rotates its fingers without holding it. With predicted touch, we observe quicker adjustments toward a correct grasp; without it, recovery tends to require more time and repeated attempts.
To examine this behavior, we compare the full model (predicted + real-time touch) with the variant without predicted tactile inputs (real-time touch only), using the corresponding ablation training configurations from Table 3 of the main paper. We adopt a stricter success criterion: the robot must complete the task with at most one failed grasp attempt; otherwise, the trial is counted as a failure.
Under this criterion, predicted touch improves SR by 25 percentage points, supporting its contribution to task completion with fewer failed grasp attempts.