acceptodds
Withdrawn from ICLR 2027

TacWM: Unified World Modeling of Vision, Touch, Action, and Object Pose for Dexterous Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

High-performance dexterous manipulation requires accurate modeling of the contact dynamics between the hand and manipulated objects. However, existing methods either overlook the importance of tactile feedback in contact-rich tasks or focus solely on fitting action trajectories without modeling object status, thereby limiting manipulation performance. We introduce TacWM, a unified multimodal world model that jointly models visual observations, tactile feedback, actions, and object poses. Rather than modeling each modality independently, TacWM utilizes a unified diffusion transformer to model their joint distribution under causal temporal conditioning. Specifically, each modality is first encoded by a dedicated encoder, and TacWM is trained to predict the next-step representations of all modalities from multimodal history. This joint prediction allows different modalities to complement and reinforce one another. In particular, predicting future tactile feedback and object poses helps the model anticipate contact changes and object motion, thereby improving action generation for dexterous manipulation. To support large-scale training on heterogeneous datasets, we also propose a cross-embodiment mechanism that progressively aligns action and tactile signals from new embodiments in shared latent spaces. Extensive experiments demonstrate that TacWM achieves state-of-the-art performance in dexterous manipulation, marking a step toward general-purpose multimodal dexterous manipulation. Beyond enabling effective dexterous manipulation, TacWM can also effectively predict future videos, tactile feedback, and object poses.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models, Multimodal Learning, Dexterous Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vCH8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TacWM, a unified multimodal world model for dexterous manipulation that jointly models visual observations, tactile feedback, robot actions, and object poses using a diffusion transformer.

Critical strengths

Unlike action-only policies or vision-and-touch-only models, TacWM feeds future object poses directly into prediction. Modeling object motion, contact response, visual change, and actions together gives a principled way to capture the coupled dynamics of dexterous manipulation.

Its cross-embodiment alignment tackles a real barrier to scaling multimodal manipulation models across platforms. Heterogeneous pretraining data helps, and the full model posts the best average score across tested tasks. Modality ablations further back the multimodal design.

Critical weaknesses

TacWM posts a 36.5% average success rate, just 2.8 points above the strongest baseline at 33.7%. It wins on several tasks, but Sliding reaches only 86.0%, well short of π0.5's 97.0%.

The paper also doesn't report variance across training seeds or confidence intervals, so it's unclear whether the gains are statistically real.

Despite jointly predicting future vision, touch, actions, and object poses, the paper only quantifies tactile prediction, and only against a persistence baseline. It skips systematic checks on future video quality, object-pose accuracy, and cross-modal physical consistency.

Cross-embodiment tests also only cover platforms already seen in training. There's no evidence yet for generalization to unseen embodiments or new tactile sensor setups.

Questions for authors

see weakness

Reviewer_hdQ5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TacWM, a multimodal world model for dexterous manipulation that jointly models vision, tactile feedback, actions, and object poses with a single diffusion transformer. Each modality is tokenized by a dedicated encoder (frozen video VAE for RGB, embodiment-specific codecs for actions/touch, 3D translation + 6D rotation for pose); vision, action, and pose are generated with flow matching under independently sampled noise levels, while future tactile latents are regressed through learned queries. A structured attention mask keeps history causal (KV-cache compatible) and lets future tokens interact bidirectionally. A cross-embodiment mechanism aligns heterogeneous actions into a 78-dim shared layout (arm controls + 32-dim hand latents) and tactile signals into 48 canonical functional regions with 12 temporal positions, enabling joint pretraining on three datasets with different hands and tactile sensors (DataDex, HRDexDB, T-Rex). The backbone is initialized from LingBot-VA, pretrained for 100k steps, and fine-tuned per task on VTDexManip expert trajectories. In closed-loop "predict-four-execute-one" evaluation on five VTDexManip tasks, TacWM achieves the best macro average (36.5% vs. 33.7% for π0.5\pi_{0.5}, 32.1% for ViTacFormer, 32.3% for T-Rex), winning three of five tasks. Future tactile prediction beats a persistence baseline on all three pretraining datasets, and ablations show each modality and the alignment design contribute to the average.

Critical strengths

A clean, well-motivated extension of world-action models to contact-rich manipulation. Representing the observation as (vision, touch, object pose) and jointly generating all modalities plus actions in one flow-matching transformer is a principled formulation: touch and pose are modeled targets rather than fused inputs, and the causal-history / bidirectional-future attention design is compatible with KV caching. This is consequential because it subsumes and unifies two lines of prior work (video-based world-action models and visuo-tactile policies) in a single, reproducible architecture, and the ablations confirm that the full combination gives the best macro average.

The first concrete cross-embodiment alignment for tactile signals, validated on three heterogeneous datasets. Mapping image-based, scalar, and six-axis force-torque signals through source-specific frontends into 48 canonical functional regions while preserving 12 temporal positions per transition is a practical answer to a real bottleneck — tactile data are scarce and sensor-dependent. It is consequential because it makes multi-embodiment tactile pretraining possible without changing the backbone, and it is supported by evidence: consistent gains over persistence on all three datasets' tactile prediction, and +4.7 / +3.0 points over a shared-encoder variant on ReorientUp / BottleCap.

Critical weaknesses

The paper's own results do not consistently support its central claim that the four modalities "complement and reinforce one another." TacWM wins the macro average but loses Sliding to the vision-only π0.5\pi_{0.5} by 11 points (86.0 vs. 97.0). More telling, the modality ablations show the full model is not the best configuration on three of five tasks: removing object pose improves Sliding (86.0 → 91.0) and ReorientDown (29.0 → 30.3), and removing vision improves HandOver (17.0 → 18.0). Per task, additional modalities appear as likely to hurt as to help, and the paper offers no analysis of when or why. Since the headline contribution is the joint four-modality formulation — not any single component — this inconsistency directly undercuts the thesis.

The world-modeling claims are evaluated only weakly. The abstract claims effective prediction of future video, touch, and object pose, yet only tactile prediction is quantified, and only against a persistence baseline — no learned predictive baseline (e.g., ViTacFormer-style future-touch prediction, or the cited visuo-tactile world models) is compared. There are no video- or pose-prediction metrics, no closed-loop inference latency numbers despite latency being a stated motivation against large VLA backbones, and no variance or significance reporting alongside the success rates. Object poses used as supervision are pseudo-labels from ProxyPose, whose noise is never discussed. As a result, the "world model" half of the contribution rests largely on qualitative figures.

The evaluation is limited to a single simulation environment, with no real-world experiments to validate the proposed approach. For a paper motivated by contact richness and tactile feedback, the absence of any real-world experiment (where tactile signals are noisier and distribution shift is real) leaves the practical value of the formulation unverified.

Questions for authors
  • Why does removing object pose improve Sliding (86.0 → 91.0) and ReorientDown, and removing vision improve HandOver? Do you observe negative cross-modal transfer, and can you characterize the conditions under which joint modeling helps versus hurts?

  • Can you report quantitative future-video and object-pose prediction metrics, and compare tactile prediction against a learned baseline rather than persistence only?

  • What closed-loop control frequency does the full four-modality joint generation achieve at inference, and what does it cost relative to an action-only prediction?

  • Could the authors include real-world experiments to strengthen the paper's claims, particularly given that the improvements in simulation are relatively modest?

  • Is the DataDex pretraining dataset public? The anonymized citation makes its scale and availability hard to assess, and pretraining-data dependence matters for interpreting the benchmark gains.

Reviewer_Li6wOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents TacWM, a multimodal world model that jointly predicts vision, tactile feedback, actions, and object poses for dexterous manipulation. The method also introduces a cross-embodiment module to support different robot hands and tactile sensors.

Critical strengths

Combining tactile feedback and world modeling for dexterous manipulation is an interesting direction to explore.

Critical weaknesses
  • Limited improvement. The average success rate is 36.5%, compared with 33.7% for π0.5 and 32.3% for T-Rex. The improvement is relatively small, and it is unclear whether the difference is statistically significant, especially since robotics simulation can be quite noisy.
  • Unclear contribution of each component. Tactile feedback and world modeling are two challenging research problems. The paper combines them without clearly demonstrating the benefit of each. Removing tactile only reduces the average success rate from 36.5% to 33.9%. It is also unclear whether world modeling provides a significant benefit over direct policy learning.
  • No real-world evaluation. Tactile sensing is especially challenging in real-world manipulation. Simulation results alone are not sufficient to demonstrate its practical value.
Questions for authors
  • Can the authors provide a more detailed analysis of the benefits of tactile feedback and world modeling separately, preferably with stronger controlled baselines?
  • Can the authors report results across multiple seeds to show whether the improvements are significant?
  • Can the authors provide real-world experiments to demonstrate the benefits of tactile feedback?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.