acceptodds
Under review as a conference paper at ICLR 2027

ME-Dex 1.0: Unifying Heterogeneous Tactile Sensing into World Action Modeling

Abstract

World Action Models predict future visual states to inform robot action generation. Tactile sensing directly captures contact states and force changes during physical interaction. Existing methods often treat tactile sensing as a conditioning input for action generation, thereby modeling only the current contact state. Instead, tactile signals should be modeled alongside video as future observations of the evolving world state. We present ME-Dex 1.0 (ME-Dex), a unified World Action Tactile Model for joint modeling of visual states, tactile states, and robot actions. ME-Dex adopts a Mixture-of-Transformers comprising a Video Expert, a Tactile Expert, and an Action Expert, all trained with flow matching. Multi-modal joint attention connects the three experts at intermediate layers, allowing action generation to exploit learned visual and tactile dynamics during joint generation. To support heterogeneous tactile inputs across embodiments, ME-Dex maps sensor surfaces to a Canonical Hand Model and encodes the aligned observations into a unified latent space with a Unified Tactile Autoencoder. We augment tactile-free simulation trajectories with synchronized tactile observations for joint visual, tactile, and action learning. Experiments on RoboTwin, DexJoCo, and ManiFeel improve average manipulation success over the strongest baselines by 11.2, 8.9, and 15.0 percentage points. Qualitative rollouts on physical robots further demonstrate the use of visual and tactile observations across grippers and dexterous hands. Code and supplementary materials are available at https://anonymousmedex.github.io/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Tactile Action Model, Heterogeneous Tactile Data, Unified Tactile Representation, Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oojkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ME-Dex 1.0, a World Action Model that jointly predicts future visual observations, tactile signals, and robot actions. To accommodate heterogeneous tactile sensors across different robotic embodiments, the authors introduce a Canonical Hand Model and a Unified Tactile Autoencoder. They also develop a simulation-based pipeline to augment existing robot demonstrations with tactile signals. The proposed framework adopts a three-expert architecture with H-Bridge attention for cross-modal interaction and joint Flow Matching for multimodal generation. Experiments on multiple simulation benchmarks and real robotic platforms demonstrate promising improvements in manipulation success rates.

Critical strengths
  1. The paper is well-motivated and presents a coherent framework for incorporating tactile information into world action modeling. The combination of heterogeneous tactile representation, simulation-based data augmentation, and joint multimodal generation provides a practical approach to learning contact-rich manipulation tasks.

  2. The experimental evaluation is fairly comprehensive, spanning several simulation benchmarks and two physical robot platforms. The paper also includes ablations on tactile conditioning, future tactile prediction, and cross-modal attention, providing useful evidence for several design choices.

Critical weaknesses
  1. The novelty of the core modeling framework is not sufficiently distinguished from recent closely related works. Joint prediction of future vision, tactile signals, and actions has already been explored, and similar three-expert architectures with Flow Matching have also been proposed. While the heterogeneous tactile representation is a potentially interesting contribution, the paper could better clarify its specific methodological advances over existing Visual–Tactile–Action WAMs.

  2. Some important aspects of the proposed framework are not fully evaluated. Although the paper provides ablations showing improvements from joint future tactile prediction, it lacks direct quantitative evaluation of the predicted tactile states and their physical consistency. Furthermore, the benefits of the Canonical Hand Model for generalization across unseen robotic embodiments or sensor configurations remain unclear.

Questions for authors

Q1. How does ME-Dex compare with existing Visual–Tactile–Action WAMs?

The current experiments mainly compare ME-Dex against VLA models, vision-based WAMs, and tactile-conditioned policies. However, recent works such as Dream-Tac, Tactile-WAM, VT-WAM, and N₀-TWAM have already explored similar joint prediction frameworks. A more direct comparison with these closely related methods would help clarify the specific advantages of ME-Dex.

Q2. What are the specific advantages of H-Bridge over other cross-modal attention mechanisms?

The paper compares H-Bridge with Full Joint Attention, but does not evaluate it against alternative designs, such as the Asymmetric Attention in VT-WAM, TAAM in Tactile-WAM, or the shared attention mechanism in N₀-TWAM. It would be helpful to understand whether H-Bridge offers meaningful advantages in performance or computational efficiency under comparable settings.

Q3. How much does the Canonical Hand Model contribute to cross-embodiment tactile learning?

The paper reports that multi-source AE pretraining improves RoboTwin Random success rates from 91.28% to 91.92%, which is encouraging. However, this result does not directly isolate the contribution of canonical spatial alignment or demonstrate generalization to unseen embodiments. An ablation that removes or replaces the Canonical Hand Model, along with comparisons to existing unified tactile representations such as NeoForce, would help better establish its effectiveness.

Q4. How does future tactile prediction contribute to action generation at inference time?

The ablation from Tactile Conditioning to Full Joint Prediction shows a 1.06 percentage point improvement, suggesting that future tactile modeling provides additional benefits. However, the paper does not directly evaluate future tactile prediction accuracy or how the Action Expert utilizes the generated tactile representations. Additionally, zeroing the current tactile input at inference time barely affects RoboTwin performance (91.92% vs. 91.74%). It would be interesting to further investigate the respective roles of current tactile feedback and predicted future tactile states, for example by selectively disabling the information flow from the Tactile Expert to the Action Expert during inference.

6 comments

We extend our sincere thanks for your comprehensive review and positive assessment of our work. We would first like to clarify that a key distinction of ME-Dex lies in its Canonical Hand Model and Unified Tactile AE, which establish a shared 3D force representation across grippers and dexterous hands. We have expanded the related-work discussion and conducted additional canonical alignment ablations and zero-shot transfer evaluations on an unseen real-world sensor without AE fine-tuning. We also quantitatively evaluated future tactile predictions against ground truth and conducted open-loop and closed-loop ablations to examine their contributions to action quality and manipulation performance. Furthermore, we extended the attention comparisons to TAAM-style, Fast-WAM-style, and N0-TWAM-style variants, providing further insights into the strengths and limitations of H-Bridge. We hope these additional results address your concerns and strengthen our paper. We would sincerely appreciate your reconsideration of our paper's rating. Detailed responses follow below.


W1: The novelty of the core modeling framework is not sufficiently distinguished from recent closely related works. Joint prediction of future vision, tactile signals, and actions has already been explored, and similar three-expert architectures with Flow Matching have also been proposed. While the heterogeneous tactile representation is a potentially interesting contribution, the paper could better clarify its specific methodological advances over existing Visual–Tactile–Action WAMs.

Q1: How does ME-Dex compare with existing Visual–Tactile–Action WAMs? The current experiments mainly compare ME-Dex against VLA models, vision-based WAMs, and tactile-conditioned policies. However, recent works such as Dream-Tac, Tactile-WAM, VT-WAM, and N₀-TWAM have already explored similar joint prediction frameworks. A more direct comparison with these closely related methods would help clarify the specific advantages of ME-Dex.

Response: We thank the reviewer for highlighting these closely related works. We have further clarified the methodological contributions of ME-Dex and its distinctions from existing approaches in Sections 2.2–2.3 (p. 2) of the revised manuscript.

Specifically, Dream-Tac, Tactile-WAM, and VT-WAM primarily model tactile images or deformation signals, whereas ME-Dex unifies 3D contact forces across different embodiments through the Canonical Hand Model, Valid Mask, and a shared tactile AE. Although N0-TWAM also incorporates force information, it uses separate pathways for observed force representations and future tactile image prediction. In contrast, ME-Dex uses the same unified tactile AE to encode both current and future 3D contact forces for joint prediction across embodiments. Further technical distinctions are discussed in the revised manuscript.

Additionally, Table R3-1 presents published results on four overlapping ManiFeel tasks, where ME-Dex achieves 70.0%average success compared with 18.5% for Tactile-WAM. As these results come from different studies, we present them as a cross-paper reference rather than a controlled comparison.

Table R3-1. Published ManiFeel Results on Four Overlapping Tasks.

MethodPeg InsertionUSB InsertionPower PlugGear AssemblyAverage ↑
Tactile-WAM16.02.018.038.018.5
ME-Dex58.074.088.060.070.0

Note. Tactile-WAM uses vision-based tactile sensing, while ME-Dex uses 3D contact forces.

W2: Some important aspects of the proposed framework are not fully evaluated.

Q4: How does future tactile prediction contribute to action generation at inference time?


W2.1: Although the paper provides ablations showing improvements from joint future tactile prediction, it lacks direct quantitative evaluation of the predicted tactile states and their physical consistency.

Q4.1: The ablation from Tactile Conditioning to Full Joint Prediction shows a 1.06 percentage point improvement, suggesting that future tactile modeling provides additional benefits. However, the paper does not directly evaluate future tactile prediction accuracy or how the Action Expert utilizes the generated tactile representations.

Response: We thank the reviewer for highlighting future tactile prediction accuracy and its contribution to action generation. To quantitatively compare predictions with ground truth, we conduct open-loop evaluations on 100 RoboTwin and 100 DexJoCo samples, covering 16 future frames and different contact states, including contact onset, release, and dynamic contact.

As shown in Table R3-2, the full model achieves higher future tactile prediction accuracy and lower action errors. Both deteriorate when current tactile inputs are zeroed, and degrade further when cross-modal interactions between Tactile and Video/Action are blocked.

The Action Expert accesses predicted future tactile representations through joint attention. These results show a consistent relationship between future tactile prediction quality and action quality, indicating that cross-modal interactions with predicted tactile representations provide useful information for action generation. We have added the experiments and analysis to Appendix C.6 (pp. 26-27) of the revised manuscript.

Table R3-2. Open-loop Evaluation of Future Tactile Prediction and Action Quality.

SettingContact Force MAE ↓Contact F1 ↑Action MAE ↓
Full tactile observations + future tactile prediction0.15780.75580.04021
Zero current tactile + future tactile prediction0.21720.55080.05053
Block Tactile ↔ Video/Action interactions0.47820.20600.07948

W2.2: Furthermore, the benefits of the Canonical Hand Model for generalization across unseen robotic embodiments or sensor configurations remain unclear.

Q3: How much does the Canonical Hand Model contribute to cross-embodiment tactile learning? The paper reports that multi-source AE pretraining improves RoboTwin Random success rates from 91.28% to 91.92%, which is encouraging. However, this result does not directly isolate the contribution of canonical spatial alignment or demonstrate generalization to unseen embodiments. An ablation that removes or replaces the Canonical Hand Model, along with comparisons to existing unified tactile representations such as NeoForce, would help better establish its effectiveness.

Response: We thank the reviewer for highlighting cross-embodiment learning and generalization with the Canonical Hand Model. To isolate the contribution of spatial alignment, we trained a Non-canonical AE using fixed, embodiment-specific region-to-slot permutations that remove anatomical correspondence across embodiments, while keeping the architecture, masking strategy, training data, and optimization settings unchanged.

As shown in Table R3-3, both models achieve comparable results on RoboTwin Random. On DexJoCo dynamic windows, Canonical Alignment improves Transition F1 by 7.89 percentage points, with an improvement in Contact F1 as well. Although it does not consistently achieve lower force reconstruction errors, these results support its benefit for contact-state recognition in dexterous-hand settings.

To further evaluate generalization, we directly applied the simulation-pretrained Unified AE to an unseen real-world SO-101–Paxini configuration. With all AE weights frozen and only fixed sensor-specific normalization, the model achieves 74.62% Contact F1 on 54 real-world episodes, providing evidence of zero-shot tactile representation transfer across robot and sensor configurations.

Regarding NeoForce in N0-TWAM, it jointly encodes RGB observations and 3D force fields with temporal modeling, whereas our AE focuses on frame-wise 3D force representations and cross-embodiment anatomical alignment. Directly substituting NeoForce would introduce additional visual information and different temporal representations, making it difficult to isolate the contribution of canonical alignment under matched conditions.

We have added the controlled alignment ablation to Appendix C.2, the zero-shot tactile evaluation to Appendix C.5 of the revised manuscript.

Table R3-3. Canonical vs. Non-canonical Tactile Alignment.

Benchmark   Alignment       Contact F1   Force MAE   Transition F1
------------------------------------------------------------------
RoboTwin    Canonical          97.20      0.02128        89.73
            Non-canonical      97.22      0.02194        90.44
DexJoCo     Canonical          92.57      0.04556        77.37
            Non-canonical      90.99      0.04288        69.48

Q4.2: Additionally, zeroing the current tactile input at inference time barely affects RoboTwin performance (91.92% vs. 91.74%). It would be interesting to further investigate the respective roles of current tactile feedback and predicted future tactile states, for example by selectively disabling the information flow from the Tactile Expert to the Action Expert during inference.

Response: We thank the reviewer for suggesting a further investigation of the roles of current tactile feedback and future tactile prediction during inference. We first clarify that Table 1 only zeros current tactile observations, while future tactile prediction and its cross-modal interactions remain enabled. Therefore, the small change in success rate does not imply that the model ignores tactile information.

Following the reviewer's suggestion, we conducted an additional closed-loop information-flow ablation. With current tactile observations already zeroed, further blocking information flow from predicted future tactile representations to Video/Action reduces RoboTwin Random success by 2.52 percentage points. These results indicate that although removing current tactile observations has a limited impact on manipulation performance, future tactile prediction and its cross-modal interactions still provide useful information for action generation.


Q2: What are the specific advantages of H-Bridge over other cross-modal attention mechanisms? The paper compares H-Bridge with Full Joint Attention, but does not evaluate it against alternative designs, such as the Asymmetric Attention in VT-WAM, TAAM in Tactile-WAM, or the shared attention mechanism in N₀-TWAM. It would be helpful to understand whether H-Bridge offers meaningful advantages in performance or computational efficiency under comparable settings.

Response: We thank the reviewer for suggesting further comparisons with alternative cross-modal attention mechanisms. We agree that comparing H-Bridge only with Full Joint Attention is insufficient for a comprehensive evaluation. Before submission, we had also evaluated N₀-TWAM-style Shared Attention under our original training configuration. As shown in Table R3-4, H-Bridge achieves 91.92% success on RoboTwin Random, outperforming Full Joint Attention (90.60%) and N0-TWAM-style (89.82%).

Following the reviewer's suggestion, we conducted an additional small-scale comparison of H-Bridge, TAAM-style, and Fast-WAM-style under matched training configurations. As shown in Table R3-5, TAAM-style achieves higher success than H-Bridge, with lower inference latency. We acknowledge that TAAM-style performs better under this experimental setting. Fast-WAM-style further reduces inference latency to 1.181 s by omitting future tactile generation, but achieves a lower success rate of 83.50%.

While H-Bridge outperforms Full Joint Attention and N0-TWAM-style under our original training setting, the additional experiments also demonstrate the advantages of TAAM-style. Since these experiments use a different training scale and configuration from the main paper, it remains unclear whether this advantage would persist under the original training setting.

Table R3-4. Attention Ablations under the Original Training Setting.

Attention VariantRoboTwin Random Success (%) ↑
ME-Dex (H-Bridge)91.92
ME-Dex (Full Joint Attention)90.60
N0-TWAM-style Attention89.82

Note. Results are obtained under the original ME-Dex training and evaluation setting. The configurations differ from Table R3-5.

Table R3-5. Comparison of Attention Variants under Matched Retraining.

Attention VariantRoboTwin Random Success (%) ↑Inference Latency (s) ↓
TAAM-style86.461.281
ME-Dex (H-Bridge)85.301.348
Fast-WAM-style83.501.181

Note. All three models were retrained using 8 NVIDIA RTX PRO 5000 GPUs, a global batch size of 16, and 50,000 training steps. Fast-WAM-style omits future tactile generation at inference.

Reviewer_mCaDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ME-Dex , a World Action Tactile Model that jointly predicts future video, future tactile latents, and robot actions within a Mixture-of-Transformers architecture using conditional flow matching. To unify heterogeneous tactile sensors across grippers and dexterous hands, the authors introduce a canonical hand model (mapping end-effector surfaces to a human-like template) and a tactile autoencoder that processes 3D force field representations.

Critical strengths
  1. Mapping heterogeneous sensors onto a canonical hand model provides a structured spatial layout for cross-embodiment learning (for parallel grippers and dexterous hands).

  2. Predicting future tactile states allow WMs to comprehend contact dynamics.

  3. Evaluated across three simulation suites (RoboTwin, DexJoCo, ManiFeel) with strong quantitative success gains over baseline architectures.

Critical weaknesses
  1. The real-world validation of the method is limited. It does not provide clear procedure to map to force-fields tactile sensors (unless it also applied for taxel-based). This part is the core joint point between real sensing and the proposed WAM.

  2. In the RoboTwin ablation (see Table 1), setting current tactile inputs to zero during evaluation yields almost identical success rates compared to the full model with observed tactile inputs. This suggests that downstream policy execution relies almost entirely on visual tokens and proprioception, making it unclear how much the WAM actually relies on tactile cues.

  3. For real-world deployment (Appendix C.5), the pretrained unified tactile autoencoder must be fine-tuned separately on task-specific tactile data for each physical task.

  4. While 3D force fields are easily extracted from simulation, real-world tactile sensors emit heterogeneous raw signals (e.g., images, pressure, magnetic field changes, etc) that do not always provide reliable shear information. The paper lacks a clear explanation or calibration pipeline for how real-world raw readings are mapped into these calibrated 3D force grids. Without a clear explanation of this mapping, there is a fundamental disconnect in how applicable the method is to real-world manipulation.

  5. The tactile autoencoder operates frame-by-frame on static force grids. Important physical contact states (such as incipient slip or stick-slip transitions) are defined by temporal changes, which static single-frame representations fail to explicitly encode.

  6. Real-robot experiments evaluate ME-Dex against non-tactile baselines (\pi_0,5, Fast-WAM), but do not evaluate ME-Dex without tactile input on physical hardware. Evaluating a no-tactile ME-Dex variant on hardware would be the strongest probe to verify that: (a) Real-world tactile signals can be faithfully represented as force fields. (b) The tactile autoencoder does not wash out the signal. (c) The WAM actually responds to tactile feedback rather than ignoring the modality during execution.

Questions for authors
  1. Why does zeroing out the tactile input at test time in Table 1 have almost no impact on performance? Does this indicate the action expert is ignoring the tactile expert in favor of visual dynamics?

  2. How are raw readings from real-world sensors (e.g., Paxini or Xynova) converted into the 3D force fields required by the tactile autoencoder? How do you match the force magnitudes between the simulation-trained autoencoder and real measurements? Are the force fields normalized?

  3. Since grippers only map to the thumb and index finger regions, the palm and remaining finger tokens are permanently masked for gripper trajectories. How do you prevent the tactile autoencoder from degrading on underrepresented canonical regions?

(I would be willing to upgrade my review based on the rebuttal)

5 comments

We thank the reviewer for recognizing the contributions of our work and for providing valuable suggestions regarding real-world tactile sensing and model validation. We would first like to clarify three aspects of ME-Dex. Real-world tactile measurements are converted into calibrated 3D force grids with sensor-specific normalization. Frame-wise tactile encoding captures instantaneous contact states, while the MoT models their temporal dynamics. Moreover, zeroing current tactile observations does not disable future tactile prediction. Following your suggestions, we conducted additional real-robot ablations against ME-Dex (VA). We also evaluated zero-shot transfer of the Unified Tactile AE to an unseen real-world sensor, compared unified and embodiment-specific encoders, and supplemented ablations on future tactile prediction. We hope these additional results and clarifications address your concerns and strengthen the evidence for ME-Dex. We would sincerely appreciate your reconsideration of our paper's rating. Detailed responses follow below.


W1: The real-world validation of the method is limited. It does not provide clear procedure to map to force-fields tactile sensors (unless it also applied for taxel-based). This part is the core joint point between real sensing and the proposed WAM.

W4.2: The paper lacks a clear explanation or calibration pipeline for how real-world raw readings are mapped into these calibrated 3D force grids. Without a clear explanation of this mapping, there is a fundamental disconnect in how applicable the method is to real-world manipulation.

Q2.1: How are raw readings from real-world sensors (e.g., Paxini or Xynova) converted into the 3D force fields required by the tactile autoencoder?

Response: We thank the reviewer for highlighting the conversion process from real-world tactile signals to 3D force grids. We first clarify that both Paxini and Xynova sensors used in our work directly provide calibrated, spatially distributed force grids, without requiring ME-Dex to estimate contact forces from raw electrical signals.

Specifically, Paxini (PX6AX-GEN3-IP-M2324-Core) employs Hall-effect array-based tactile sensing. Normal and tangential contact forces deform the flexible sensing structure, causing changes in the internal magnetic field distribution. Its AMSS multidimensional measurement system captures these magnetic responses through a Hall sensor array and converts them into force measurements using the manufacturer's embedded nonlinear force-inversion algorithm. Xynova Flex 2 employs a piezoresistive tactile array, which detects contact-induced material deformation and resistance changes, and estimates contact forces through the manufacturer's calibration and signal-processing methods. Both sensors provide spatially distributed 3D force grids, containing one normal and two tangential force components.

The calibration and conversion from raw sensor signals to physical force grids are performed by the sensor manufacturers. The host system directly retrieves force data in Newtons (N) through the provided interfaces. ME-Dex maps these measurements to the corresponding Canonical Hand regions according to the sensor layouts and uses a Valid Mask to distinguish valid sensing positions from unavailable regions. The underlying sensor calibration and force-inversion procedures are not part of our proposed method.


W4.1: While 3D force fields are easily extracted from simulation, real-world tactile sensors emit heterogeneous raw signals (e.g., images, pressure, magnetic field changes, etc) that do not always provide reliable shear information.

Q2.2: How do you match the force magnitudes between the simulation-trained autoencoder and real measurements? Are the force fields normalized?

Response: We thank the reviewer for raising the issue of force-scale consistency between simulation and real-world measurements. As clarified above, ME-Dex directly uses calibrated force measurements provided by the sensors, without estimating shear forces from raw signals.

Simulated and real-world tactile measurements can differ in their force ranges. To reduce this scale mismatch, we normalize all force fields before feeding them into the AE. As detailed in Appendix A.3 (Eqs. 5–6 and Table A1), we apply smooth dead-zone filtering, followed by channel-wise asinh compression and normalization. Normal forces are mapped to [0, 1], while tangential forces are mapped to [−1, 1]. For real-world sensors, we use the same preprocessing formulation with parameters adapted to their force ranges and kept fixed during each evaluation.

W3: For real-world deployment (Appendix C.5), the pretrained unified tactile autoencoder must be fine-tuned separately on task-specific tactile data for each physical task.

Response: We are particularly grateful to the reviewer for highlighting the task-specific AE fine-tuning in Appendix C.5. This observation prompted us to conduct an important evaluation that was missing from our original submission: zero-shot transfer to an unseen real-world tactile sensor. Our Unified Tactile AE is designed to enable joint pretraining on larger and more diverse tactile datasets across sensors and robot embodiments, with the goal of learning transferable tactile representations. Following the reviewer's observation, we evaluated the original pretrained Unified AE on 54 real-world Paxini episodes (83k frames), keeping all AE weights frozen and applying fixed sensor-specific normalization. The model achieved 74.62% Contact F1, 0.0580 N Contact Force MAE, and 49.30% Transition F1.

Furthermore, we kept the pretrained Unified Tactile AE frozen and performed task-specific post-training of ME-Dex, achieving a 70% success rate on the real-world Upright Medicine Bottle task.

These results show that the pretrained unified AE can retain meaningful contact and transition information on a real-world sensor configuration not seen during AE pretraining, without task-specific fine-tuning. In practice, task-specific fine-tuning is commonly adopted to improve real-world manipulation performance, which motivated its use in our original real-robot experiments. Nevertheless, this additional evaluation provides evidence of zero-shot transfer at the tactile representation level. We have added the quantitative evaluation and implementation details to Appendix C.5 (p. 26) of the revised manuscript.


W2: In the RoboTwin ablation (see Table 1), setting current tactile inputs to zero during evaluation yields almost identical success rates compared to the full model with observed tactile inputs. This suggests that downstream policy execution relies almost entirely on visual tokens and proprioception, making it unclear how much the WAM actually relies on tactile cues.

Q1: Why does zeroing out the tactile input at test time in Table 1 have almost no impact on performance? Does this indicate the action expert is ignoring the tactile expert in favor of visual dynamics?

Response: We thank the reviewer for highlighting the tactile-zeroing results. We first clarify that Table 1 only zeros current tactile observations, while future tactile prediction remains enabled. The Tactile Expert can still predict future contact states from visual and action-related information. Since physical contact persists even without tactile sensors, this experiment aims to evaluate whether ME-Dex can operate without direct tactile input and how well it maintains manipulation performance. Under this setting, its success rate remains 4.84 percentage points higher than ME-Dex (VA), which excludes tactile modeling entirely.

To further examine the contribution of future tactile information, we conducted an additional closed-loop ablation. With current tactile observations already zeroed, blocking information flow from future tactile predictions to Video/Action reduces RoboTwin Random success by 2.52 percentage points. These results indicate that the minimal performance change after zeroing current tactile inputs does not mean the model ignores tactile information; future tactile prediction and its cross-modal interactions still contribute to action generation.

W5: The tactile autoencoder operates frame-by-frame on static force grids. Important physical contact states (such as incipient slip or stick-slip transitions) are defined by temporal changes, which static single-frame representations fail to explicitly encode.

Response: We thank the reviewer for highlighting the importance of temporal contact dynamics. Robot actions directly affect contact states, and tactile forces exhibit abrupt, step-like changes during contact onset, release, and state transitions. The primary objective of our tactile AE is to compress and preserve instantaneous spatial force distributions rather than fuse information across frames. We therefore adopt frame-wise encoding to avoid potentially smoothing transient contact changes through early temporal fusion within the AE, following a similar principle to Video LDM [1] and Gen-1 [2].

Specifically, the AE preserves spatial force distributions at each frame, while an additional Temporal Change Loss (Appendix A.4, Eq. 7) encourages accurate reconstruction of frame-to-frame force changes. The MoT Tactile Expert then models sequences of tactile latents and learns their relationships with visual states and robot actions through joint attention.

As shown in Table 5, the AE achieves good Contact Transition F1 scores across RoboTwin, DexJoCo, and ManiFeel, indicating that frame-wise encoding can still preserve contact onset and release information. We have further clarified this design in Section 3.3.2 and Appendix A.4 (p. 17) of the revised manuscript.

[1] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563–22575, 2023.

[2] Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and Content-Guided Video Synthesis with Diffusion Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7346–7356, 2023.


W6: Real-robot experiments evaluate ME-Dex against non-tactile baselines (\pi_0,5, Fast-WAM), but do not evaluate ME-Dex without tactile input on physical hardware. Evaluating a no-tactile ME-Dex variant on hardware would be the strongest probe to verify that: (a) Real-world tactile signals can be faithfully represented as force fields. (b) The tactile autoencoder does not wash out the signal. (c) The WAM actually responds to tactile feedback rather than ignoring the modality during execution.

Response: We thank the reviewer for suggesting a no-tactile ablation on physical hardware. We additionally trained ME-Dex (VA) without tactile modeling and compared it with the full model on two LeRobot SO-101 PX manipulation tasks, with 20 trials per task.

As shown in Table R2-1, the full ME-Dex consistently outperforms ME-Dex (VA) on both real-robot tasks, demonstrating that incorporating tactile modeling improves real-world manipulation performance.

Together with the calibrated 3D force-grid conversion and frozen Unified Tactile AE evaluations discussed above, these results provide further evidence that real-world tactile signals can be effectively represented and encoded, and that tactile modeling contributes useful information to downstream manipulation. We have added the experiments and analysis to Paper Sec. 4.5 (pp. 9-10, Table 6) of the revised manuscript.

Table R2-1. Real-Robot Ablation of Tactile Modeling on LeRobot SO-101 PX.

MethodUpright Medicine Bottle ↑Relocate Power Plug ↑
ME-Dex (VA)55.0%0.0%
ME-Dex (Ours)80.0%35.0%
Reviewer_RYBNOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

World Action Models (WAM) learn to predict future visual representations to enhance physical understanding, while visual information still lacks object response to operations. This paper introduces ME-Dex, a WAM for jointly modeling vision, action together with tactile states: three modalities are handled with individual experts connected by a Mixture-of-Transformers architecture. Furthermore, to support heterogeneous tactile inputs from different embodiments, the paper proposes a Unified Tactile Encoder that encodes the heterogeneous inputs mapped by a Canonical Hand Model. The paper evaluates the proposes model on RoboTwin, DexJoCo, and ManiFeel and reports qualitative results on both gripper and dexterous hands.

Critical strengths

The paper addresses the meaningful challenge of incorporating tactile observations into world action models and learning unified tactile representations across embodiments. Treating future tactile states as prediction targets, rather than merely conditioning signals, is a meaningful direction for physical learning.

  • The proposed framework combines tactile information by a canonical hand and a shared tactile autoencoder, which effectively enables training from datasets across multiple embodiments. The tactile augmentation pipeline also enables training on simulation datasets that originally lack tactile measurements.

  • ME-Dex reports substantial improvements across RoboTwin, DexJoCo, and ManiFeel, with additional real-robot evaluations covering both grippers and dexterous hands. The results suggest potential practical value beyond an experimantal prototype.

Critical weaknesses

Insufficient analysis of the role of future tactile prediction.** Table 1 shows that zeroing tactile observations at inference barely degrades performance, potentially suggesting that the model implicitly learns contact dynamics from other modalities. However, the paper does not quantitatively evaluate the accuracy of its generated future tactile states against ground truth. Moreover, it does not investigate whether future tactile generation is necessary at inference, rather than primarily serving as a useful training objective, despite the close comparison with Fast-WAM, leaving the interpretation of the proposed modeling objective incomplete.

  • Insufficient ablation of the unified tactile autoencoder. It remains unclear whether a unified encoder outperforms embodiment-specific encoders, or whether canonical hand alignment provides benefits. The generalization to unseen embodiments has not been explicitly evaluated.

  • Lack of comparative evaluation for the unified tactile autoencoder. Table 5 reports reconstruction and contact metrics, but does not compare against other encoder baselines, making it difficult to assess the competitiveness of the representation. Prior work such as OmniVTA provides examples of gripper-based tactile autoencoder comparisons, while dexterous-hand methods exist such as PercepSkin. Comparisons against one or two reasonably adapted encoders would strengthen the contribution of this part.

  • Incomplete comparison against native tactile-involved world models. The main comparisons are primarily against vision-based WAMs and their tactile-conditioned variants. Recent relevant approaches, such as OmniVTA and VT-WAM, already investigate predictive tactile modeling though not using a cross-embodiment tactile encoder. The paper should discuss the technical distinctions and, if possible, include a comparison against native tactile-predictive approaches. In particular, VT-WAM is cited in the bibliography but is not discussed in relative work.

  • Unexplained discrepancies in reproduced baseline performance. The reproduced Fast-WAM results on RoboTwin are lower than the originally reported 91.88% and 91.78%. The discrepancy for pi05 is even larger. The paper should provide sufficient implementation and training details to explain the differences.

Questions for authors

Could the authors report future tactile prediction accuracy against ground truth, preferably across multiple prediction horizons and contact-transition events? How does prediction quality relate to downstream manipulation performance?

  • What happens if future tactile prediction is omitted only during inference, following the motivation of Fast-WAM? How does this affect success rate and inference latency?

  • Could the authors provide controlled comparisons between: (a) unified and embodiment-specific tactile encoders and (b) canonical and non-canonical tactile representations?

  • Could the authors quantitatively compare their method against tactile-native models like VT-WAM and OmniVTA, if feasible? If cross-embodiment comparison is unavailable, could the authors perform a gripper-only or dexterous hand-only comparison to show if the unified representation sacrifices performance?

  • What explains the deviations from the originally reported Fast-WAM and pi05 results? Please clarify pretrained checkpoints, training schedules, evaluation protocols, and other implementation details.

6 comments

Thank you for your thorough reviews, insightful comments, and acknowledgment of our work. We would first like to clarify two key aspects of ME-Dex. First, the Canonical Hand Model and Unified Tactile AE enable grippers and dexterous hands to share a common 3D force representation. Second, zeroing current tactile input does not disable future tactile prediction, which can still infer contact states from visual and action-related information and contribute to action generation. Following your suggestions, we conducted open-loop and closed-loop evaluations of future tactile prediction, along with unified encoder comparisons, Canonical Hand ablations, and zero-shot transfer experiments. We also expanded comparisons with related tactile encoders and world models, and clarified our baseline setting. We hope these additional results address your concerns and strengthen the evidence for ME-Dex. We would sincerely appreciate your reconsideration of our paper's rating. Detailed responses follow below.


W1: Insufficient analysis of the role of future tactile prediction. Table 1 shows that zeroing tactile observations at inference barely degrades performance, potentially suggesting that the model implicitly learns contact dynamics from other modalities.

Response: We appreciate the reviewer's observation regarding the tactile-zeroing results in Table 1. Since physical contact still occurs without tactile sensors, we designed this experiment to evaluate whether ME-Dex can operate without direct tactile input and how well it maintains manipulation performance. As clarified above, future tactile prediction remains enabled.

To further examine its contribution, we conducted an additional closed-loop ablation. With current tactile input already zeroed, blocking information flow from future tactile predictions to Video/Action reduces RoboTwin Random success from 91.74% to 89.22%. In comparison, ME-Dex (VA), which excludes tactile modeling entirely, achieves only 86.90%. These results support the contribution of future tactile prediction and its cross-modal interactions to manipulation performance.


W1.1: However, the paper does not quantitatively evaluate the accuracy of its generated future tactile states against ground truth.

Q1: Could the authors report future tactile prediction accuracy against ground truth, preferably across multiple prediction horizons and contact-transition events? How does prediction quality relate to downstream manipulation performance?

Response: We thank the reviewer for highlighting future tactile prediction accuracy and its relationship with action quality. To quantitatively compare predictions with ground truth, we conduct open-loop evaluations on 100 RoboTwin and 100 DexJoCo samples, covering 16 future frames and different contact states, including no-contact, contact onset, release, dynamic contact, and stable contact.

As shown in Table R1-1, the model achieves the best future tactile prediction accuracy and action quality with full tactile observations. Both deteriorate when current tactile inputs are zeroed, and degrade further when cross-modal interactions between Tactile and Video/Action are blocked.

These results show a consistent relationship between future tactile prediction quality and action quality, with more accurate tactile predictions associated with lower action errors, supporting the contribution of future tactile modeling to action quality. We have added the quantitative comparison and analysis to Appendix C.6 (p. 26, Table C9) of the revised manuscript.

Table R1-1. Open-loop Evaluation of Future Tactile Prediction and Action Quality.

SettingContact Force MAE ↓Contact F1 ↑Action MAE ↓
Full tactile observations + future tactile prediction0.15780.75580.04021
Zero current tactile + future tactile prediction0.21720.55080.05053
Block Tactile ↔ Video/Action interactions0.47820.20600.07948

W1.2: Moreover, it does not investigate whether future tactile generation is necessary at inference, rather than primarily serving as a useful training objective, despite the close comparison with Fast-WAM, leaving the interpretation of the proposed modeling objective incomplete.

Q2: What happens if future tactile prediction is omitted only during inference, following the motivation of Fast-WAM? How does this affect success rate and inference latency?

Response: We thank the reviewer for suggesting a further investigation of the role of future tactile generation during inference. We conducted a small-scale comparison between the original Fast-WAM and a tactile-augmented Fast-WAM-style variant, evaluating the latter with and without future tactile generation during inference.

As shown in Table R1-2, incorporating current tactile observations improves RoboTwin Random success over the original Fast-WAM. Retaining future tactile generation further increases success to 85.36%. Omitting future tactile generation reduces inference latency by approximately 0.16 s, but also decreases manipulation success.

These results indicate that future tactile prediction is not merely a training objective but also contributes positively during inference.

Table R1-2. Effect of Tactile Observations and Future Tactile Generation on RoboTwin Random.

ModelCurrent TactileFuture Tactile at InferenceSuccess (%) ↑Latency (s) ↓
Fast-WAMNoNo82.780.679
Fast-WAM-styleYesNo83.501.181
Fast-WAM-style†YesYes85.361.348

†:In this setting, observed tactile tokens are allowed to attend to future tactile tokens, and their keys and values are recomputed at each denoising step.

W2: Insufficient ablation of the unified tactile autoencoder.

Q3: Could the authors provide controlled comparisons between:


W2.1: It remains unclear whether a unified encoder outperforms embodiment-specific encoders

Q3.1: (a) unified and embodiment-specific tactile encoders

Response: We appreciate the reviewer's suggestion to directly compare unified and embodiment-specific tactile encoders. Our existing downstream ablation (Appendix C.2, Table C3) shows that unified AE pretraining improves RoboTwin Random success from 91.28% to 91.92%.

Following the reviewer's suggestion, we further compared the Unified AE with embodiment-specific AEs on identical test windows from RoboTwin and DexJoCo. As shown in Table R1-3, although embodiment-specific AEs achieve lower force reconstruction errors on their respective datasets, the Unified AE maintains comparable Contact F1 and Transition F1 on both benchmarks using a single shared model. Together with the existing downstream results, this indicates that multi-source tactile learning provides practical benefits without requiring the best reconstruction accuracy on every dataset. We have added the quantitative comparison and analysis to Appendix C.2 (pp. 21–22) of the revised manuscript.

Table R1-3. Comparison of Unified and Embodiment-Specific Tactile Autoencoders.

BenchmarkTactile AEContact F1 (%) ↑Contact Force MAE ↓Transition F1 (%) ↑
RoboTwinRoboTwin-only AE97.680.0142689.77
RoboTwinUnified AE (Ours)97.200.0212889.73
DexJoCoDexJoCo-only AE92.890.0408678.09
DexJoCoUnified AE (Ours)92.570.0455677.37

W2.2: whether canonical hand alignment provides benefits.

Q3.2: (b) canonical and non-canonical tactile representations

Response: We thank the reviewer for emphasizing the need to isolate the contribution of canonical alignment. Our Canonical Hand Model is designed to establish consistent anatomical correspondences across different end effectors, allowing grippers and dexterous hands to share a common spatial representation rather than merely minimizing reconstruction errors within each embodiment.

Following the reviewer's suggestion, we trained a Non-canonical AE using fixed, embodiment-specific region-to-slot permutations that remove cross-embodiment anatomical correspondence. Both models use the same architecture, masking strategy, training data, seeds, and optimization settings.

As shown in Table R1-4, the two models achieve comparable results on RoboTwin Random. On DexJoCo dynamic windows, canonical alignment improves Contact F1 from 90.99% to 92.57% and Transition F1 from 69.48% to 77.37%. Although it does not consistently reduce force reconstruction errors, canonical alignment provides a shared anatomical structure and improves contact-state recognition in the evaluated dexterous-hand setting. We have added the controlled ablation and quantitative analysis to Appendix C.2 (pp. 21–22) of the revised manuscript.

Table R1-4. Canonical vs. Non-canonical Tactile Alignment.

Benchmark   Alignment       Contact F1   Force MAE   Transition F1
------------------------------------------------------------------
RoboTwin    Canonical          97.20      0.02128        89.73
            Non-canonical      97.22      0.02194        90.44
DexJoCo     Canonical          92.57      0.04556        77.37
            Non-canonical      90.99      0.04288        69.48

W2.3: The generalization to unseen embodiments has not been explicitly evaluated.

Response: We thank the reviewer for this valuable suggestion, which prompted us to evaluate the zero-shot transfer capability of our unified tactile AE. Unlike our previous real-robot experiments involving task-specific AE fine-tuning, we directly apply the AE pretrained on simulated tactile data to an unseen real-world SO-101 robot equipped with a Paxini tactile sensor. Without updating any AE parameters and using only fixed sensor-specific normalization, we evaluate 54 episodes (83k frames). The model achieves 74.62% Contact F1 and 0.0580 Contact Force MAE, demonstrating zero-shot tactile reconstruction on a new robot-sensor configuration without AE fine-tuning. Furthermore, keeping the pretrained Unified Tactile AE frozen and performing only task-specific post-training of ME-Dex, we achieved a 70% success rate on the real-world Upright Medicine Bottle task. We have added the quantitative evaluation and implementation details to Appendix C.5 (p. 26) of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.