acceptodds
Under review as a conference paper at ICLR 2027

Video Generators Can Predict Touch: Tactile Synthesis from Egocentric Video

Abstract

Tactile sensing matters for dexterous manipulation. However, tactile data is expensive and slow to collect at scale. In this work, we investigate how the interaction priors learned by video generators can be transferred to touch prediction from egocentric videos. We find that Wan2.1-Fun-Control can be easily repurposed for tactile pressure prediction via lightweight LoRA fine-tuning with its base weights frozen. On the OpenTouch test set, our model, using only video observations, achieves an MSE of , hard IoU of , and soft IoU of for tactile prediction. This setup also extends naturally to joint prediction of tactile pressure, hand configuration, and hand–object segmentation. We additionally evaluate tactile prediction on the bimanual EgoTouch dataset and qualitatively assess transfer to in-the-wild video clips without further fine-tuning. A randomly initialized generator fails to recover tactile structure in our control experiment, supporting the benefit of generative pretraining.

Primary area:
generative models
Keywords:
video diffusion, tactile sensing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XS73OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates adapting a pretrained video generator for tactile-pressure estimation from egocentric RGB video. It freezes the Wan2.1-Fun-Control backbone and VAE, trains LoRA adapters, and represents tactile measurements as rendered pressure videos. A multi-stream extension spatially stacks target latents to jointly generate pressure, hand configuration, and hand–object segmentation. Experiments on OpenTouch and EgoTouch evaluate tactile-only and joint prediction, supplemented by pretraining and LoRA-rank ablations and qualitative cross-dataset transfer. The results demonstrate the feasibility of this adaptation, while tactile-only models achieve better tactile reconstruction and contact overlap than the joint models.

Critical strengths
  1. A simple adaptation with practical input requirements. The approach requires only RGB video at inference and adapts a frozen backbone through LoRA, without introducing task-specific output heads. This provides a straightforward recipe for exploring vision-to-touch prediction using existing video generators.

  2. Evaluation across single-hand and bimanual settings. Testing on both OpenTouch and EgoTouch provides useful evidence that the formulation can accommodate different tactile layouts and output configurations. The comparison between tactile-only and joint prediction also documents a relevant trade-off: generating auxiliary modalities does not automatically improve tactile accuracy.

Critical weaknesses
  1. The comparative evidence for the added predictive value remains limited. The paper reports quantitative agreement with recorded tactile signals and appropriately acknowledges that RGB observations do not uniquely determine pressure. However, without a matched pressure-prediction baseline, it remains difficult to assess the benefit of the proposed adaptation relative to simpler supervised predictors or closely related approaches. The HACO comparison evaluates contact localization under different training conditions and does not resolve this issue. In particular, the current comparisons do not establish how much the method improves instance-specific pressure prediction beyond common contact patterns associated with the observed interaction.

  2. Several experimental details require clarification to support the conclusions. Appendix A.1 provides a general frozen-backbone, LoRA-training recipe, but its application to the random-initialization control is unclear, as are the corresponding training budget and convergence behavior. The train/test grouping also needs clarification because multiple clips are extracted from individual EgoTouch recordings. Finally, the apparently corresponding OpenTouch joint/rank-64 configurations produce different results in Tables 2 and 6 without explanation. These uncertainties limit the interpretation of the pretraining ablation and the reported generalization results.

Questions for authors
  1. Clarification — random initialization Which components are randomly initialized, and which parameters are trainable in this experiment? In particular, is the randomly initialized backbone frozen with only LoRA optimized, or is the backbone trained? Please specify whether the VAE retains pretrained weights and provide the training budget and convergence information needed to interpret the comparison.

  2. Evaluation protocol Section 4.2 specifies masked MSE and a threshold of 0.1 for hard IoU. Please clarify the quantity to which this threshold is applied, including how generated RGB renderings are converted into pressure or activity values. What are the exact soft-IoU definition and valid-pixel masks used for evaluation? If evaluation in the original sensor representation is available, how does it relate to the reported rendering-space results?

  3. Additional evidence — comparative value Could you provide a matched comparison on at least one dataset against a reasonable supervised pressure-prediction baseline, or identify any existing results that establish the benefit of the proposed adaptation? The comparison should use the same RGB input setting, data split, and evaluation protocol, with any differences in target representation made explicit.

5 comments

Dear Reviewer XS73

Thank you once again for your time and constructive feedback on our paper. We have posted our response and supplementary experiments addressing your points regarding method clarification and comparison to additional baselines. As we approach the end of the discussion period, please let us know if any part of our response requires further clarification or if there are remaining questions we can help resolve.

Thank you, The Authors

Dear Reviewer XS73,

We wanted to briefly follow up on our earlier message, as we have not yet received any further feedback on our rebuttal and additional experiments. With the discussion period ending, we would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns, or if there are any remaining issues we should clarify.

Thank you

Dear Reviewer XS73,

We want to follow up again on our earlier message as we have yet to receive feedback on the additional experiments and clarifications we provided. We are excited about this work and are very eager to address remaining concerns that you have, while there is still time for discussion.

Thank you, Authors

Reviewer_t3nGOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper uses pretrained video diffusion models to infer tactile-pressure videos using RGB data. The proposed approach adapts the pretrained Wan2.1-Fun-Control model using lightweight LoRA fine-tuning, while keeping its backbone frozen. The problem is important for robotics because collecting tactile data is expensive, and the possibility of augmenting visual data with tactile estimates is a promising alternative to collecting pressure data directly. The paper is clearly organized, provides useful qualitative visualizations, and includes ablations demonstrating that generative pretraining substantially improves performance over random initialization.

Critical strengths

I cannot find critical strengths

Critical weaknesses

The main limitation is the lack of strong quantitative comparisons with existing vision-to-tactile methods. Generalization claims are also limited: transfer to Manus and EgoTactile is qualitative because comparable ground-truth tactile signals are unavailable. The held-out test sets are relatively small (101 OpenTouch clips and 230 EgoTouch clips), and the paper does not clarify how clips from the same recording sessions are separated to rule out train–test leakage, particularly for EgoTouch, where multiple clips are extracted from longer recordings.

The evaluation would benefit from uncertainty estimates across repeated runs plus additional measures of physical plausibility. Pixel-level agreement alone cannot establish that predicted pressure magnitudes or contact dynamics are physically meaningful. In addition, the model requires 50 denoising steps with a 1.3-billion-parameter backbone; inference latency, computational cost, and practical suitability for robotics are not sufficiently characterized. The segmentation targets are generated by an automatic pipeline, so errors in those pseudo-labels may also affect the joint-prediction results.

Questions for authors

The paper cites several relevant studies, including PressureVision / PressureVision++, EgoPressure, EgoTactile, and TouchSight. Rather than simply adding citations, the authors should provide direct quantitative comparisons with the closest feasible methods, especially EgoTactile (Zeng et al., 2026), EgoTAC (Zhang et al., 2026), and PressureVision++ (Grady et al., 2024), clearly explaining differences in input, output, training data, and evaluation protocol. The authors should also discuss whether the results demonstrate an advantage of video-generative pretraining over less expensive discriminative or regression-based alternative

1 comment

Dear Reviewer t3nG

Thank you once again for your time and constructive feedback on our paper. We have posted our response and supplementary experiments addressing your points, including comparison to additional baselines and further investigation into the benefits of generative models. As we approach the end of the discussion period, please let us know if any part of our response requires further clarification or if there are remaining questions we can help resolve.

Thank you, The Authors

Reviewer_BBZxOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presented a framework for predicting tactile signals from egocentric RGB video by adapting a frozen Wan2.1-Fun-Control backbone with LoRA. The results shows that pretrained video generators can be repurposed to predict spatially and temporally structured tactile patterns as well as hand configuration and hand-object segmentation.

Critical strengths
  1. The paper clearly motivates tactile prediction from egocentric videos by highlighting the scarcity of tactile data and the abundance of visual interaction data. The potential application to scalable tactile supervision is compelling.

  2. The paper is generally well-written and easy to follow. The problem formulation and methodology are clearly explained, and the architecture diagram clearly illustrates the proposed approach.

  3. The paper provides clear visual comparisons of predicted and ground-truth tactile pressure, including bimanual examples and failure cases, making the results easy to interpret.

Critical weaknesses
  1. The paper lacks quantitative comparisons with established methods, particularly EgoPressureDiff. Simple mean/zero prediction and nearest-neighbor retrieval baselines are also missing, making the reported reconstruction errors difficult to interpret. TouchAnything and HOPE are recently published, which are not necessary to include, but including them would strengthen the paper by a lot.

  2. The only control is random initialization, without specifying whether the random backbone is fully trained or frozen with LoRA. The latter would be a particularly weak baseline. A deterministic regression model using the same pretrained backbone, or a discriminative encoder with a lightweight decoder, would help distinguish the benefit of generative modeling from pretrained visual features.

  3. Metrics are computed on rendered tactile images rather than raw taxels, without reporting VAE reconstruction error. MSE and IoU provide limited evidence of pressure magnitude and dynamics. Per-sequence pressure correlation, peak-pressure error, and contact timing would be useful. Despite the stochastic formulation, no sampling-variance analysis is provided, and cross-dataset evaluation remains qualitative.

  4. Table 6 reports different rank-64 OpenTouch results from Table 2 (MSE 0.0126 vs. 0.0146). The OpenTouch contact-based filtering procedure and its application to the test set are unclear. EgoTouch clips are extracted from longer recordings without specifying whether train/test splits are disjoint by episode, participant, or object, raising concerns about data leakage.

  5. Joint prediction consistently reduces tactile accuracy, especially on EgoTouch (MSE 0.0331 vs. 0.0111). Segmentation is supervised using automatically generated labels, and no downstream benefit of joint prediction is demonstrated. The very low HACO contact IoU (0.0433) also raises questions about the fairness of the contact-localization comparison and alignment between representations.

Questions for authors
  1. Can you add at least EgoPressureDiff as a baseline using comparable taxel-space metrics?

  2. How was the random-initialization baseline trained? Can you compare against deterministic regression using pretrained Wan features?

  3. Are the train/test splits disjoint at the recording or episode level? Does contact-based filtering affect test data? Can you use the official EgoTouch split?

  4. Why do Tables 2 and 6 report different results for the OpenTouch joint rank-64 model?

  5. What is the VAE reconstruction error, and how accurately does the model predict pressure magnitude and temporal contact changes? How sensitive are predictions to sampling seeds?

7 comments

We thank the reviewer for recognizing the potential of our approach for scalable tactile supervision and highlighting several important points regarding our methodology and evaluation. We address the concerns below to clarify and strengthen our findings.

Evaluation in Taxel Space We thank the reviewer for their suggestion to assess our model in taxel space. We have since trained our model to predict the 16×16 taxel grid and observe improved performance across the evaluation metrics. We have updated all tables to reflect the new results.

EgoPressureDiff and Other Models

Following the reviewer's suggestions, we have substantially expanded our comparisons against recent tactile prediction methods, including HOPE and EgoTac. The EgoPressureDiff repository does not provide the pretrained U-Net and MaskNet checkpoints required for inference. Hence, we are unable to assess the model against our dataset or metrics.

As already discussed in Section 2 (Related Work, p. 3), EgoPressureDiff differs from our approach in its use of physical-attribute conditioning, spatial masks, pressure prototypes, and full-backbone adaptation. Unlike our RGB-only approach, it additionally receives object weight and other physical metadata; its own ablation shows that removing text conditioning reduces volumetric IoU from 38.9% to 30.5%.

Moreover, EgoTactile primarily uses controlled green-screen grasping with object/subject-held-out evaluation, rather than the naturalistic OpenTouch/EgoTouch interactions used in our experiments. EgoPressureDiff also employs a fixed 162-sensor representation with anatomical postprocessing and 40k training steps, whereas our framework adapts a frozen 1.3B Wan transformer through LoRA and accommodates different sensor layouts without architectural modifications (Section 3.2; Appendix B).

Instead, we provide direct quantitative comparisons with HOPE and EgoTac, whose models became available following the ICLR deadline. We map all predictions to the same hand-mesh configuration for consistent evaluation. Comparisons against HOPE and EgoTac are reported separately by retraining our Wan model on HOPE's published train–test split to avoid overlap between HOPE's training data and our original test clips.

HOPE's hand-detection pipeline struggled to detect gloved hands, preventing downstream hand reconstruction and pressure prediction. To address this, we supplied our hand-segmentation masks, allowing the pipeline to localize the hands and proceed with inference. We explicitly indicate this adaptation in our results.

Following the reviewer's suggestion, we additionally report two sequence-level metrics: total-pressure correlation, measuring Pearson correlation between predicted and ground-truth total-pressure curves over time, and peak total-pressure error, measuring the relative error in maximum total pressure.

The HACO contact comparison is also explicitly identified as asymmetric in Section 4.4, since HACO is independently trained and not evaluated under the same tactile-supervision setting.

MethodMSE ↓Over ↓Under ↓Hard IoU ↑Soft IoU ↑Flow MSE ↓Total-pressure corr ↑Peak total err ↓
EgoTac0.03690.08160.01970.2690.4090.705−0.0101.694
HOPE + Segmentation0.01400.03660.01500.4190.5760.454−0.0020.529
Ours (tactile-only)0.00840.01780.02020.4150.6380.4230.1310.293

** General Baselines and Seed Sensitivity**

Following the reviewer's feedback, we evaluate our model against the following general baselines:

  • Zero-pressure: Zero tactile activity across all frames.
  • Mean-pressure: Average tactile pressure map computed from the training set, repeated across all frames.
  • Nearest-neighbor: Uses DINOv2 features to retrieve the most visually similar RGB clip from the training set and uses its corresponding tactile sequence as the prediction.

We also thank the reviewer for highlighting the importance of sampling-seed sensitivity. To quantify its effects, we evaluate the OpenTouch tactile-only LoRA-128 model using 10 independent sampling seeds across all 101 test clips.

MethodMSE ↓Over ↓Under ↓Hard IoU ↑Soft IoU ↑Flow MSE ↓Total-pressure corr ↑Peak total err ↓
Zero (blank hand)0.0211n/a0.04660.0000.354n/an/a1.000
Mean pressure0.00700.03880.00760.3340.575n/an/a0.441
RGB NN (taxel video)0.00910.02060.01580.3950.6261.0480.0810.481
Ours (10 seeds)0.0073 ± 0.00030.0147 ± 0.00080.0172 ± 0.00030.401 ± 0.0080.648 ± 0.0060.869 ± 0.0100.197 ± 0.0320.338 ± 0.037

Benefit of Joint Prediction

We introduced joint prediction to investigate whether auxiliary supervision from other modalities can improve tactile prediction. Although quantitative results do not demonstrate improved tactile accuracy, we observe qualitative improvements.

These findings highlight that quantitative metrics capture only certain aspects of generative predictions, motivating further investigation into more comprehensive evaluation measures.

The predicted segmentation relates tactile signals to the interacting hand and object, while hand configuration provides geometric context. Together, these modalities offer contact-aware supervision from RGB video, with potential applications in grasp learning and dexterous manipulation without requiring tactile measurements at inference.

Deterministic Regression

Following the reviewer's suggestion, we additionally train a single-step regression baseline using pretrained Wan features. Regression achieves MSE 0.0116 and Hard IoU 0.397, compared with MSE 0.0073 ± 0.0003 and Hard IoU 0.401 ± 0.008 for our generative model. However, regression exhibits substantially less temporal variation (0.0038 versus 0.0214 in ground truth), with nearly identical predictions across different clips (pairwise correlation 0.986).

In contrast, our generative model better preserves:

  • Temporal variation: 0.0191 ± 0.0007, compared with 0.0214 in ground truth.
  • Cross-clip variation: 0.0310 ± 0.0022, compared with 0.0425 in ground truth and 0.0057 for regression.
  • Cross-clip diversity: Pairwise correlation 0.733 ± 0.016, closer to the ground-truth correlation of 0.64.

These findings suggest that deterministic regression converges toward a mean-like pressure pattern, whereas multi-step generation better preserves the variability of tactile sequences. Qualitative examples are available on our project website.

Random Initialization

We clarify the random-initialization control in Section 4.7 and Appendix C. The randomly initialized DiT is frozen, with only rank-128 LoRA adapters optimized, matching our pretrained configuration.

Both models use the same training data and comparable optimization budgets (2,400 versus 2,519 steps).

This experiment isolates the contribution of pretrained DiT weights under fixed LoRA adaptation, rather than comparing against end-to-end training from scratch.

VAE Reconstruction

To isolate the reconstruction error introduced by the frozen Wan VAE, we directly encode and decode ground-truth OpenTouch tactile videos.

The VAE achieves:

  • MSE: 0.0006
  • Hard IoU: 0.9429
  • Soft IoU: 0.9493

These results indicate that the VAE preserves tactile representations with high fidelity and that reconstruction error alone explains only a small portion of the full model's prediction errors.

Training Split

We clarify that EgoTouch is organized into distinct recording sessions, each corresponding to a specific scene or interaction. The train–test split is defined at the session level rather than the clip level.

Longer recordings are divided into 81-frame clips, with all clips from the same recording retaining their original split assignment. Thus, no overlapping or neighboring clips appear across training and testing.

We verified that all 42 test sessions (231 clips) are disjoint from the 1,087 training sessions.

Our experiments use EgoTouch's official test_unseen sessions for evaluation and the remaining sessions for training.

Table Clarification

We thank the reviewer for identifying the discrepancy between Tables 2 and 6.

Table 2 reported earlier checkpoints for the OpenTouch and EgoTouch models, whereas Table 6 reported final-checkpoint results. We will update both tables to report the final checkpoints in the revised manuscript.

Dear Reviewer BBZx

Thank you once again for your time and constructive feedback on our paper. We have posted our response and supplementary experiments addressing your points, including comparison to additional baselines and clarification of our methodology. As we approach the end of the discussion period, please let us know if any part of our response requires further clarification or if there are remaining questions we can help resolve.

Thank you, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response summary

We thank all reviewers for their constructive feedback and thoughtful suggestions. In response, we have substantially expanded our experimental evaluation and clarified several aspects of our methodology. We have additionally revised our manuscript and provided additional qualitative examples in the project website. We provide a summary below:

Expanded quantitative benchmarks. We introduce direct taxel-space prediction and extend our comparisons to HOPE, EgoTac, and general baselines (zero-pressure, mean-pressure, and RGB nearest-neighbor retrieval). On a shared OpenTouch evaluation split, our model achieves lower MSE, higher Soft IoU, and lower temporal flow error than HOPE and EgoTac. We also investigate additional methods, including EgoPressureDiff, and clarify the limitations preventing direct evaluation where pretrained checkpoints are unavailable.

Isolating the benefits of a generative model. We introduce a deterministic regression baseline using the same pretrained Wan backbone. This model produces nearly identical predictions across different videos with limited temporal variation. We additionally clarify the random-initialization experiment and its scope in evaluating the contribution of generative pretraining.

Expanded evaluation and robustness analysis. We introduce total-pressure correlation and peak-pressure error to further evaluate pressure dynamics beyond spatial reconstruction. We further evaluate predictions across 10 independent sampling seeds on all 101 OpenTouch test clips, demonstrating relatively stable spatial performance. A separate VAE reconstruction experiment confirms that the frozen VAE preserves tactile representations with low reconstruction error.

Methodology clarifications. We clarify the EgoTouch recording-level train–test splits, resolve discrepancies between previously reported checkpoints, detail the evaluation protocol, and acknowledge the limitations of contact-localization and joint-prediction comparisons.

Practical considerations and limitations. We examine the computational cost of iterative generation relative to deterministic regression and clarify our primary objective of generating scalable tactile supervision from RGB videos rather than real-time inference.

Together, these additional experiments and clarifications provide a more comprehensive assessment of our approach, provide stronger baselines comparison, and clarify the contributions and limitations of repurposing pretrained video generators for tactile prediction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.