- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents ME-Dex 1.0, a World Action Model that jointly predicts future visual observations, tactile signals, and robot actions. To accommodate heterogeneous tactile sensors across different robotic embodiments, the authors introduce a Canonical Hand Model and a Unified Tactile Autoencoder. They also develop a simulation-based pipeline to augment existing robot demonstrations with tactile signals. The proposed framework adopts a three-expert architecture with H-Bridge attention for cross-modal interaction and joint Flow Matching for multimodal generation. Experiments on multiple simulation benchmarks and real robotic platforms demonstrate promising improvements in manipulation success rates.
Critical strengths
-
The paper is well-motivated and presents a coherent framework for incorporating tactile information into world action modeling. The combination of heterogeneous tactile representation, simulation-based data augmentation, and joint multimodal generation provides a practical approach to learning contact-rich manipulation tasks.
-
The experimental evaluation is fairly comprehensive, spanning several simulation benchmarks and two physical robot platforms. The paper also includes ablations on tactile conditioning, future tactile prediction, and cross-modal attention, providing useful evidence for several design choices.
Critical weaknesses
-
The novelty of the core modeling framework is not sufficiently distinguished from recent closely related works. Joint prediction of future vision, tactile signals, and actions has already been explored, and similar three-expert architectures with Flow Matching have also been proposed. While the heterogeneous tactile representation is a potentially interesting contribution, the paper could better clarify its specific methodological advances over existing Visual–Tactile–Action WAMs.
-
Some important aspects of the proposed framework are not fully evaluated. Although the paper provides ablations showing improvements from joint future tactile prediction, it lacks direct quantitative evaluation of the predicted tactile states and their physical consistency. Furthermore, the benefits of the Canonical Hand Model for generalization across unseen robotic embodiments or sensor configurations remain unclear.
Questions for authors
Q1. How does ME-Dex compare with existing Visual–Tactile–Action WAMs?
The current experiments mainly compare ME-Dex against VLA models, vision-based WAMs, and tactile-conditioned policies. However, recent works such as Dream-Tac, Tactile-WAM, VT-WAM, and N₀-TWAM have already explored similar joint prediction frameworks. A more direct comparison with these closely related methods would help clarify the specific advantages of ME-Dex.
Q2. What are the specific advantages of H-Bridge over other cross-modal attention mechanisms?
The paper compares H-Bridge with Full Joint Attention, but does not evaluate it against alternative designs, such as the Asymmetric Attention in VT-WAM, TAAM in Tactile-WAM, or the shared attention mechanism in N₀-TWAM. It would be helpful to understand whether H-Bridge offers meaningful advantages in performance or computational efficiency under comparable settings.
Q3. How much does the Canonical Hand Model contribute to cross-embodiment tactile learning?
The paper reports that multi-source AE pretraining improves RoboTwin Random success rates from 91.28% to 91.92%, which is encouraging. However, this result does not directly isolate the contribution of canonical spatial alignment or demonstrate generalization to unseen embodiments. An ablation that removes or replaces the Canonical Hand Model, along with comparisons to existing unified tactile representations such as NeoForce, would help better establish its effectiveness.
Q4. How does future tactile prediction contribute to action generation at inference time?
The ablation from Tactile Conditioning to Full Joint Prediction shows a 1.06 percentage point improvement, suggesting that future tactile modeling provides additional benefits. However, the paper does not directly evaluate future tactile prediction accuracy or how the Action Expert utilizes the generated tactile representations. Additionally, zeroing the current tactile input at inference time barely affects RoboTwin performance (91.92% vs. 91.74%). It would be interesting to further investigate the respective roles of current tactile feedback and predicted future tactile states, for example by selectively disabling the information flow from the Tactile Expert to the Action Expert during inference.
6 comments
W2: Some important aspects of the proposed framework are not fully evaluated.
Q4: How does future tactile prediction contribute to action generation at inference time?
W2.1: Although the paper provides ablations showing improvements from joint future tactile prediction, it lacks direct quantitative evaluation of the predicted tactile states and their physical consistency.
Q4.1: The ablation from Tactile Conditioning to Full Joint Prediction shows a 1.06 percentage point improvement, suggesting that future tactile modeling provides additional benefits. However, the paper does not directly evaluate future tactile prediction accuracy or how the Action Expert utilizes the generated tactile representations.
Response: We thank the reviewer for highlighting future tactile prediction accuracy and its contribution to action generation. To quantitatively compare predictions with ground truth, we conduct open-loop evaluations on 100 RoboTwin and 100 DexJoCo samples, covering 16 future frames and different contact states, including contact onset, release, and dynamic contact.
As shown in Table R3-2, the full model achieves higher future tactile prediction accuracy and lower action errors. Both deteriorate when current tactile inputs are zeroed, and degrade further when cross-modal interactions between Tactile and Video/Action are blocked.
The Action Expert accesses predicted future tactile representations through joint attention. These results show a consistent relationship between future tactile prediction quality and action quality, indicating that cross-modal interactions with predicted tactile representations provide useful information for action generation. We have added the experiments and analysis to Appendix C.6 (pp. 26-27) of the revised manuscript.
Table R3-2. Open-loop Evaluation of Future Tactile Prediction and Action Quality.
| Setting | Contact Force MAE ↓ | Contact F1 ↑ | Action MAE ↓ |
|---|---|---|---|
| Full tactile observations + future tactile prediction | 0.1578 | 0.7558 | 0.04021 |
| Zero current tactile + future tactile prediction | 0.2172 | 0.5508 | 0.05053 |
| Block Tactile ↔ Video/Action interactions | 0.4782 | 0.2060 | 0.07948 |
W2.2: Furthermore, the benefits of the Canonical Hand Model for generalization across unseen robotic embodiments or sensor configurations remain unclear.
Q3: How much does the Canonical Hand Model contribute to cross-embodiment tactile learning? The paper reports that multi-source AE pretraining improves RoboTwin Random success rates from 91.28% to 91.92%, which is encouraging. However, this result does not directly isolate the contribution of canonical spatial alignment or demonstrate generalization to unseen embodiments. An ablation that removes or replaces the Canonical Hand Model, along with comparisons to existing unified tactile representations such as NeoForce, would help better establish its effectiveness.
Response: We thank the reviewer for highlighting cross-embodiment learning and generalization with the Canonical Hand Model. To isolate the contribution of spatial alignment, we trained a Non-canonical AE using fixed, embodiment-specific region-to-slot permutations that remove anatomical correspondence across embodiments, while keeping the architecture, masking strategy, training data, and optimization settings unchanged.
As shown in Table R3-3, both models achieve comparable results on RoboTwin Random. On DexJoCo dynamic windows, Canonical Alignment improves Transition F1 by 7.89 percentage points, with an improvement in Contact F1 as well. Although it does not consistently achieve lower force reconstruction errors, these results support its benefit for contact-state recognition in dexterous-hand settings.
To further evaluate generalization, we directly applied the simulation-pretrained Unified AE to an unseen real-world SO-101–Paxini configuration. With all AE weights frozen and only fixed sensor-specific normalization, the model achieves 74.62% Contact F1 on 54 real-world episodes, providing evidence of zero-shot tactile representation transfer across robot and sensor configurations.
Regarding NeoForce in N0-TWAM, it jointly encodes RGB observations and 3D force fields with temporal modeling, whereas our AE focuses on frame-wise 3D force representations and cross-embodiment anatomical alignment. Directly substituting NeoForce would introduce additional visual information and different temporal representations, making it difficult to isolate the contribution of canonical alignment under matched conditions.
We have added the controlled alignment ablation to Appendix C.2, the zero-shot tactile evaluation to Appendix C.5 of the revised manuscript.
Table R3-3. Canonical vs. Non-canonical Tactile Alignment.
Benchmark Alignment Contact F1 Force MAE Transition F1
------------------------------------------------------------------
RoboTwin Canonical 97.20 0.02128 89.73
Non-canonical 97.22 0.02194 90.44
DexJoCo Canonical 92.57 0.04556 77.37
Non-canonical 90.99 0.04288 69.48
Q4.2: Additionally, zeroing the current tactile input at inference time barely affects RoboTwin performance (91.92% vs. 91.74%). It would be interesting to further investigate the respective roles of current tactile feedback and predicted future tactile states, for example by selectively disabling the information flow from the Tactile Expert to the Action Expert during inference.
Response: We thank the reviewer for suggesting a further investigation of the roles of current tactile feedback and future tactile prediction during inference. We first clarify that Table 1 only zeros current tactile observations, while future tactile prediction and its cross-modal interactions remain enabled. Therefore, the small change in success rate does not imply that the model ignores tactile information.
Following the reviewer's suggestion, we conducted an additional closed-loop information-flow ablation. With current tactile observations already zeroed, further blocking information flow from predicted future tactile representations to Video/Action reduces RoboTwin Random success by 2.52 percentage points. These results indicate that although removing current tactile observations has a limited impact on manipulation performance, future tactile prediction and its cross-modal interactions still provide useful information for action generation.
Q2: What are the specific advantages of H-Bridge over other cross-modal attention mechanisms? The paper compares H-Bridge with Full Joint Attention, but does not evaluate it against alternative designs, such as the Asymmetric Attention in VT-WAM, TAAM in Tactile-WAM, or the shared attention mechanism in N₀-TWAM. It would be helpful to understand whether H-Bridge offers meaningful advantages in performance or computational efficiency under comparable settings.
Response: We thank the reviewer for suggesting further comparisons with alternative cross-modal attention mechanisms. We agree that comparing H-Bridge only with Full Joint Attention is insufficient for a comprehensive evaluation. Before submission, we had also evaluated N₀-TWAM-style Shared Attention under our original training configuration. As shown in Table R3-4, H-Bridge achieves 91.92% success on RoboTwin Random, outperforming Full Joint Attention (90.60%) and N0-TWAM-style (89.82%).
Following the reviewer's suggestion, we conducted an additional small-scale comparison of H-Bridge, TAAM-style, and Fast-WAM-style under matched training configurations. As shown in Table R3-5, TAAM-style achieves higher success than H-Bridge, with lower inference latency. We acknowledge that TAAM-style performs better under this experimental setting. Fast-WAM-style further reduces inference latency to 1.181 s by omitting future tactile generation, but achieves a lower success rate of 83.50%.
While H-Bridge outperforms Full Joint Attention and N0-TWAM-style under our original training setting, the additional experiments also demonstrate the advantages of TAAM-style. Since these experiments use a different training scale and configuration from the main paper, it remains unclear whether this advantage would persist under the original training setting.
Table R3-4. Attention Ablations under the Original Training Setting.
| Attention Variant | RoboTwin Random Success (%) ↑ |
|---|---|
| ME-Dex (H-Bridge) | 91.92 |
| ME-Dex (Full Joint Attention) | 90.60 |
| N0-TWAM-style Attention | 89.82 |
Note. Results are obtained under the original ME-Dex training and evaluation setting. The configurations differ from Table R3-5.
Table R3-5. Comparison of Attention Variants under Matched Retraining.
| Attention Variant | RoboTwin Random Success (%) ↑ | Inference Latency (s) ↓ |
|---|---|---|
| TAAM-style | 86.46 | 1.281 |
| ME-Dex (H-Bridge) | 85.30 | 1.348 |
| Fast-WAM-style | 83.50 | 1.181 |
Note. All three models were retrained using 8 NVIDIA RTX PRO 5000 GPUs, a global batch size of 16, and 50,000 training steps. Fast-WAM-style omits future tactile generation at inference.
We extend our sincere thanks for your comprehensive review and positive assessment of our work. We would first like to clarify that a key distinction of ME-Dex lies in its Canonical Hand Model and Unified Tactile AE, which establish a shared 3D force representation across grippers and dexterous hands. We have expanded the related-work discussion and conducted additional canonical alignment ablations and zero-shot transfer evaluations on an unseen real-world sensor without AE fine-tuning. We also quantitatively evaluated future tactile predictions against ground truth and conducted open-loop and closed-loop ablations to examine their contributions to action quality and manipulation performance. Furthermore, we extended the attention comparisons to TAAM-style, Fast-WAM-style, and N0-TWAM-style variants, providing further insights into the strengths and limitations of H-Bridge. We hope these additional results address your concerns and strengthen our paper. We would sincerely appreciate your reconsideration of our paper's rating. Detailed responses follow below.
Response: We thank the reviewer for highlighting these closely related works. We have further clarified the methodological contributions of ME-Dex and its distinctions from existing approaches in Sections 2.2–2.3 (p. 2) of the revised manuscript.
Specifically, Dream-Tac, Tactile-WAM, and VT-WAM primarily model tactile images or deformation signals, whereas ME-Dex unifies 3D contact forces across different embodiments through the Canonical Hand Model, Valid Mask, and a shared tactile AE. Although N0-TWAM also incorporates force information, it uses separate pathways for observed force representations and future tactile image prediction. In contrast, ME-Dex uses the same unified tactile AE to encode both current and future 3D contact forces for joint prediction across embodiments. Further technical distinctions are discussed in the revised manuscript.
Additionally, Table R3-1 presents published results on four overlapping ManiFeel tasks, where ME-Dex achieves 70.0%average success compared with 18.5% for Tactile-WAM. As these results come from different studies, we present them as a cross-paper reference rather than a controlled comparison.
Table R3-1. Published ManiFeel Results on Four Overlapping Tasks.
Note. Tactile-WAM uses vision-based tactile sensing, while ME-Dex uses 3D contact forces.