acceptodds
Under review as a conference paper at ICLR 2027

TacDyn-WAM: Learning Implicit Tactile Dynamics in a Heterogeneous Visuo-Tactile World Action Model

Abstract

World action models improve robotic manipulation by conditioning actions on predicted futures, yet existing tactile variants largely inherit video-generation pipelines that reconstruct future tactile observations through iterative denoising. Such prediction can become unreliable under deployment drift: small changes in contact position or force may substantially alter tactile pixels even when the underlying contact evolution remains predictable. We introduce TacDyn-WAM, a heterogeneous visuo-tactile world action model that predicts implicit tactile dynamics rather than reconstructing future tactile observations. It learns TacRep, a dynamics-aware tactile target space trained through masked spatio-temporal prediction on tactile clips and regularized by relational structure distillation. A visual expert and an Implicit Tactile Dynamics Expert predict future visual and tactile representations in separate target spaces while interacting through joint attention; the tactile expert predicts future representations and their changes at multiple horizons in a single forward pass, and a read-only tactile memory supplies the current tactile state. On UniVTAC, TacDyn-WAM achieves an average success rate of 81.5% using only the provided demonstrations, competitive with large-scale-pretrained models, and ablations confirm the benefits of both tactile pathways and TacRep over pixel-reconstruction and static alternatives. On five real-robot tasks, TacDyn-WAM reaches 71.0% average success, and modest-scale pretraining on about 6,000 real visuo-tactile trajectories raises it to 85.0%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
tactile perception, world models, vision-language-action models, robot learning, contact-rich manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kxm8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TacDyn-WAM extends InternVLA-A1 with a tactile dynamics expert and a read-only tactile understanding memory. It learns TacRep using masked spatiotemporal feature prediction and relational distillation, then predicts future tactile features and feature changes at three horizons. Visual, tactile, and action experts interact through joint attention and staged training. The method reports 81.5% mean success across eight UniVTAC simulation tasks, 71.0% across five real-robot tasks, and 85.0% after additional real-data pretraining. Validation includes pathway removal, alternative tactile target spaces, and GPU inference-latency comparisons.

Critical strengths
  1. The complete system produces substantial improvements in both simulation and real manipulation. It exceeds the reported base model by 25.7 points on UniVTAC and by 31 points in the real-robot evaluation without additional OmniViTac pretraining. Evidence across contact-rich tasks makes the contribution consequential for robotic control rather than only representation prediction.
  2. The separation of future tactile dynamics from current tactile understanding is coherent and empirically useful. Removing the tactile expert or memory reduces UniVTAC mean success from 81.5% to 67.9% or 71.3%, respectively, and the target-space comparisons favor TacRep over Cosmos VAE and DINOv2. These ablations support the main architectural choices.
Critical weaknesses
  1. The ablations do not isolate several mechanisms central to the explanation. Table 2 removes whole pathways or swaps target encoders, but does not separately test future versus delta prediction, single versus multiple horizons, relational distillation, or change-based patch weighting. A capacity-matched tactile expert with its prediction losses disabled would also separate auxiliary forecasting from extra tactile processing. Consequently, the results establish the complete system's usefulness more clearly than the claimed causal importance of implicit dynamics learning.
  2. The evaluation leaves robustness and statistical uncertainty undercharacterized. Real tasks have only 20 trials per policy, with no reported training-seed variability, and initial conditions are sampled within the demonstration workspace (Appendix H.2). UniVTAC uses published baseline results with potentially different execution settings, although the base model and ablations are re-evaluated consistently. These limitations matter for the strength of superiority and deployment-drift claims.
Questions for authors
  1. Under matched tactile-expert capacity and training budget, what happens when the future/delta supervision is removed, or when only one horizon is predicted? These comparisons would distinguish dynamics supervision from additional tactile capacity.

  2. Are the reported success rates from a single trained checkpoint per task? Please provide any existing variation across training runs and explain whether all real-world methods used comparable action-execution schedules. This affects the interpretation of the success-rate gaps.

2 comments
AuthorsOpenReview ↗

Response to Reviewer kxm8 (Part 1/2): Predictive Supervision and Component Ablations

Thank you for your careful review and constructive feedback. We address your concerns point by point below.

1. Contributions of predictive supervision, multiple horizons, and RSD

We have added the two ablations highlighted in your question—retaining the tactile expert without its future/delta prediction supervision, and predicting a single horizon—alongside an RSD ablation. These comparisons examine specific mechanisms beyond removing entire pathways or replacing target encoders.

The table reports success rates (%) on seven UniVTAC tasks. Insert HDMI is omitted from this mechanism-focused evaluation because its demonstrations contain nearly no useful tactile signal; it remains included in the original eight-task benchmark results. All averages below use the same seven tasks.

MethodInsert HoleInsert TubeLift CanPull-out KeyPut BottleLift BottleGrasp ClassifyAvg. (7 tasks)Drop (pp)
TacDyn-WAM (Full, horizons 5/15/25)9197879772979991.4—
Single horizon (15)8789839063899785.46.0
w/o ITDE predictive training8587818759879883.48.0
w/o RSD8791849364919886.94.6

Contribution of predictive supervision. The w/o ITDE predictive training variant retains the expert and its connections for action learning, but removes future/delta prediction supervision. Average success falls by 8.0 points, with the largest decrease on Put Bottle (13 points) and 10-point decreases on Insert Tube, Pull-out Key, and Lift Bottle. Retaining the tactile processing module alone therefore does not recover the full model's performance, supporting the contribution of explicitly training it to predict contact evolution.

Multiple horizons improve control. Predicting only the middle horizon, 15 frames, reduces average success by 6.0 points relative to predicting 5/15/25 frames. The largest decreases occur on Put Bottle (9 points), Insert Tube and Lift Bottle (8 points each), and Pull-out Key (7 points). These tasks require tracking evolving contact during reorientation, alignment, or extraction. The pattern is consistent with the benefit of anticipating contact evolution at multiple temporal scales.

RSD complements temporal prediction. Removing RSD reduces average success by 4.6 points, with the largest drops on Put Bottle (8 points), Insert Tube and Lift Bottle (6 points each). As discussed in the paper, these tasks involve sustained rotation of the contact pattern. RSD preserves local spatial relations while TDP learns temporal evolution, helping the representation capture contact dynamics as tactile pixel patterns change. The ablation supports the contribution of this spatial constraint beyond temporal predictive learning alone.

AuthorsOpenReview ↗

Response to Reviewer kxm8 (Part 2/2): Robustness, Training Variability, and Execution Protocols

2. Evaluation robustness, training variability, and execution protocols

Expanded real-robot evaluation. We have increased evaluation from 20 to 50 trials per task for all five methods, giving 250 trials per method across five tasks. We also use a broader range of randomized initial object placements, increasing task difficulty and testing generalization under more varied contact conditions. These variations make tactile pixel patterns more likely to depart from the demonstration distribution. Success rates and successful-trial counts are:

MethodStack CupsRemove PlugInsert PlugUnscrew Cup LidWipe WhiteboardAvg.
LingBot-VA30% (15/50)8% (4/50)32% (16/50)10% (5/50)40% (20/50)24.0%
InternVLA-A142% (21/50)14% (7/50)36% (18/50)8% (4/50)58% (29/50)31.6%
FTP-152% (26/50)34% (17/50)50% (25/50)14% (7/50)68% (34/50)43.6%
TacDyn-WAM76% (38/50)56% (28/50)78% (39/50)36% (18/50)92% (46/50)67.6%
TacDyn-WAM (modest pretraining)94% (47/50)84% (42/50)90% (45/50)56% (28/50)98% (49/50)84.4%

Under these harder conditions, TacDyn-WAM and its modestly pretrained variant decline by only 3.4 and 0.6 percentage points relative to the original evaluation, versus 8.0–11.4 points for the baselines. TacDyn-WAM retains a 24.0-point advantage over FTP-1, the strongest baseline. The larger evaluation therefore supports the performance advantage and robustness to initial-state variation in closed-loop execution.

Training runs and checkpoints. Each reported task result uses one trained checkpoint, evaluated over multiple trials. We do not currently have results from independent training seeds. The expanded evaluation measures execution variability for these checkpoints; it does not estimate variability across training runs. We will make this distinction explicit in the evaluation description.

Sensitivity to the simulation execution horizon. We additionally evaluated executing all 50 predicted steps on Insert Hole and Lift Can, where the original 32-step setting was chosen for contact stability. Success changes only slightly, from 91% to 89% and 87% to 84%, respectively. With the other six tasks unchanged, the eight-task average is 80.9% versus 81.5% originally. This shows that the main performance gains remain with full 50-step execution. The base-model and pathway-ablation comparisons use consistent execution settings; results quoted from external publications retain their original evaluation protocols.

Real-world execution schedules. Both TacDyn-WAM variants, InternVLA-A1, and FTP-1 predict 50-step action chunks and execute the first 32 steps before replanning. LingBot-VA uses its default deployment configuration because its action chunks are coupled to video-latent frames, with observation feedback and KV-cache updates aligned to those chunks. Adopting a 50-step prediction / 32-step execution schedule would require adapting this temporal alignment, rather than simply changing two horizon parameters. We therefore preserve its native deployment protocol.

Reviewer_vk2tOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TacDyn-WAM adds touch sensing to a robot world action model, a policy that predicts what will happen next before choosing its actions. The central design choice is what the tactile part should predict. Instead of generating future tactile images, the model predicts future tactile features in a learned space called TacRep. A separate tactile expert predicts these features and how they change at three future horizons in one pass, a small read-only memory supplies the current contact state, and training proceeds in four stages. On the UniVTAC simulation benchmark the model reaches 81.5% average success, and on five real tasks 71%, rising to 85% after pretraining on 6,000 real trajectories.

Critical strengths
  • The paper asks a clear question and well motivated: what should a tactile world model predict, and answers it with a controlled test. With everything else fixed, predicting in TacRep gives 81.5% success, a frozen image encoder gives 74%, and a video VAE gives 62%.

  • The gains over the vision-only base model are large, about 26 points in simulation and 31 points on the real robot. The real-robot comparison with the tactile policy FTP-1 is fair, since both are fine-tuned on the same 60 demonstrations per task, and TacDyn-WAM leads by 16 points.

  • The method is described precisely, down to token counts and per-stage hyperparameters, and the paper is honest about the tasks where touch does not help.

Critical weaknesses
  • The core idea is not new. Predicting tactile change in a feature space instead of in pixels already appears in N0-VTLA, TacForeSight and ContactWorld, all cited by the paper, and the new target space is a direct adaptation of V-JEPA 2.1. The novelty lies in the combination.

  • The paper never measures the thing it says it learns. There is no prediction error for the tactile expert, no probe showing that TacRep encodes slip, rotation or force, and no test of how it behaves when tactile input drifts away from the demonstrations. None of TacRep's ingredients is ablated, so the only evidence is final task success.

  • The main table reads better than the numbers. TacDyn-WAM is best on only two of eight simulated tasks, and the two pretrained N0 models beat it on average but are grayed out and excluded from the bold comparison. On Insert HDMI it stays at 12%, the same as its base model.

  • Some protocol choices favor the authors' models. On two tasks they execute only 32 of the 50 predicted steps because this "gives more stable contact", a tweak the published baselines did not get, and the pathway ablations retrain only the last training stage, so the removed pathway still shaped the earlier ones.

Questions for authors

N/A

4 comments
AuthorsOpenReview ↗

Response to Reviewer vk2t (Part 1/3): Novelty

Thank you for your careful review and constructive feedback. We address your concerns point by point below.

1. Novelty of the tactile prediction space

Our novelty is not predicting a latent, but determining what that latent should represent for tactile prediction. The target spaces serve different purposes:

  • VAE: optimized for pixel reconstruction, tying prediction to tactile pixel patterns that can change sharply under small contact deviations.
  • DINOv2 (N0-VTLA): pretrained on static natural images, with neither tactile-domain adaptation of its frozen backbone nor a temporal training objective. Static visual features are not equivalent to features explicitly trained to capture tactile evolution.
  • CNN-based tactile-image latents (TacForeSight, ContactWorld): simply encoding tactile images into CNN features does not give those features an explicit physical meaning for predicting contact evolution. Calling them “latents” does not establish their suitability for temporal dynamics prediction under tactile-image shifts.

Much existing work reuses frozen visual VAEs or established image encoders without explicitly examining whether their representation objectives suit tactile dynamics. Our starting point is an observed property of visuo-tactile interaction: execution drift can substantially alter tactile pixel patterns while the underlying contact evolution remains predictable. TacRep is trained around the resulting joint requirements of temporal contact dynamics and robustness to tactile-image distribution shifts. To our knowledge, we are among the few works that explicitly investigate and validate this choice of prediction space. Our architecture further separates current tactile understanding from future tactile evolution, allowing each representation to serve its intended purpose.

TacRep goes beyond directly adapting V-JEPA 2.1: it adds a local relational structure constraint to temporal predictive learning. Distilling patch relations rather than static feature values supplies a spatial prior without forcing TacRep to reproduce DINOv2’s representation, addressing the two tactile-specific requirements together.

The distinction is measurable: within the same policy framework, TacRep achieves 81.5%, versus 74.0% with DINOv2 and 62.0% with Cosmos VAE, supporting the importance of the prediction-space design.

AuthorsOpenReview ↗

Response to Reviewer vk2t (Part 2/3): Prediction Quality, Ablation, and Robustness

2. Direct prediction measurements, component ablation, and robustness

Direct measurement of tactile prediction. TacRep follows a JEPA-style predictive objective: the tactile expert predicts future representations, with no decoder reconstructing future tactile pixels. The relevant prediction error is therefore measured against encoded future observations in TacRep space. We compare predicted future features with the frozen TacRep encoder's features of the corresponding observed future tactile images, reporting cosine similarity and MSE:

TaskMean Cosine Similarity ↑Mean MSE (×10⁻⁴) ↓
Insert Hole0.9301.82
Insert Tube0.9202.08
Lift Can0.9501.30
Pull-out Key0.9202.08
Put Bottle0.9102.34
Lift Bottle0.9401.56
Grasp Classify0.9700.78
Insert HDMI0.9800.52
Avg.0.9401.56

These results directly quantify future-feature prediction quality beyond policy success. Probing slip, rotation, or force tests whether specific physical quantities can be decoded; the measurements above evaluate TacRep as a target for predicting contact evolution.

Ablating RSD. Removing relational structure distillation reduces success on every task in the seven-task comparison below:

MethodInsert HoleInsert TubeLift CanPull-out KeyPut BottleLift BottleGrasp ClassifyAvg. (7 tasks)
TacDyn-WAM (Full, %)9197879772979991.4
w/o RSD (%)8791849364919886.9
Drop (percentage points)46348614.6

The largest drops occur on Put Bottle (8 points), Insert Tube and Lift Bottle (6 points each), tasks identified in the paper as involving sustained rotation of the contact pattern. On Put Bottle, for example, the tactile imprint rotates as the bottle is reoriented for placement; execution drift changes its position and pixel pattern while preserving the overall rotation trend. This pattern of performance drops is consistent with RSD's intended role: preserving local spatial relations so that TDP can learn contact evolution despite changes in tactile pixels.

Robustness under harder execution conditions. We expand real-robot evaluation from 20 to 50 trials per task for all five methods, using a broader range of randomized initial object placements to increase task difficulty and test generalization. These settings induce more varied contact positions and interaction trajectories, making tactile pixel patterns more likely to depart from the demonstration distribution. Success rates are:

MethodStack CupsRemove PlugInsert PlugUnscrew Cup LidWipe WhiteboardAvg.
LingBot-VA30%8%32%10%40%24.0%
InternVLA-A142%14%36%8%58%31.6%
FTP-152%34%50%14%68%43.6%
TacDyn-WAM76%56%78%36%92%67.6%
TacDyn-WAM (modest pretraining)94%84%90%56%98%84.4%

Compared with the original evaluation, TacDyn-WAM and its modestly pretrained variant decline by only 3.4 and 0.6 percentage points, versus 8.0–11.4 points for the baselines. This supports robustness to initial-state variation in closed-loop execution, complementing the appendix's tactile-drift observations.

AuthorsOpenReview ↗

Response to Reviewer vk2t (Part 3/3): Benchmark Comparisons and Execution Horizon

3. Interpreting the main comparison and Insert HDMI

The key result is competitive performance with substantially less tactile training data. TacDyn-WAM reaches 81.5% using only the benchmark demonstrations for tactile trajectory training, compared with 83.1% for N0-VTLA and 84.5% for N0-TWAM, which benefit from large-scale tactile pretraining on 30,000+ hours of data. The paper explicitly states that these two models achieve higher averages. Gray rows distinguish models with additional visuo-tactile pretraining, while bold entries compare methods without it. We will clarify this convention and distinguish the overall best results from the best within each training-data group.

The number of task wins alone does not capture the performance differences. Compared with N0-VTLA, TacDyn-WAM is only 1–2 points behind on five tasks—Insert Tube, Lift Can, Pull-out Key, Lift Bottle, and Grasp Classify—and 12 points ahead on Put Bottle. It also improves its base model's average by 25.7 points and substantially outperforms the other methods without large-scale tactile pretraining. These results demonstrate broad improvements and a small overall gap to extensively pretrained models, despite the difference in tactile training resources.

Insert HDMI demonstrations in UniVTAC contain nearly no useful tactile signal. These nearly perfect trajectories offer little contact-correction experience, making performance more dependent on the pretrained model's visual capabilities for alignment, with limited contribution from touch. Many methods omit this task from their reported UniVTAC evaluations, but we retain all eight tasks, including the unchanged 12% on Insert HDMI, in our overall average.

4. Execution horizon

We selected the 32-step execution horizon on Insert Hole and Lift Can for more stable contact during execution. We have now additionally evaluated execution of all 50 predicted steps: success decreases only slightly, from 91% to 89% on Insert Hole and 87% to 84% on Lift Can. With the other six tasks unchanged, the eight-task average becomes 80.9% versus 81.5% originally. Thus, the shorter execution horizon has a small effect on aggregate success, and the substantial performance gains remain when executing the full 50-step chunks.

Reviewer_vk2tOpenReview ↗

Reply to authors

I thank the authors for their added experiments and the explanation. My concerns are partially resolved, but two remaining ones are still open: 1. for the table of prediction quality, it is hard to judge cosine similarity of 0.94 without a baseline. Can you add a simple baseline that can predict the future features equal to current feature, so that we can know whether expert predicts the change well. 2. my concern that the pathway ablations were retrained from Stage 4 only was not addressed.

However, given the large amount of added experiments and overall the paper is of better quality, I raise the score from 2 to 3

Reviewer_qUDkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper statements that tactile world models shouldn’t try to regenerate future tactile images, since a slightly shifted contact point or force can produce a totally different image while the physical contact evolution is unchanged. That premise is sound and well argued, and the gel-tear failure case in Fig. 5 makes it concrete. So they build TacRep, a representation trained by masked spatiotemporal prediction with relational distillation from DINOv2, then bolt a visual expert to an “implicit dynamics expert” that predicts future tactile representations (and their deltas) in one pass with no denoising, plus a small read-only tactile memory.

Critical strengths

The framing is the best part, the design holds together (TDP+RSD, the future-plus-delta readout, the read-only memory as a cheap way to inject current state), and the results are stated honestly. The ablations do real work, swapping TacRep for DINOv2 (74%) or Cosmos-VAE (62%) pins down where the gain comes from. The latency number is a genuine deployment argument, and the reproducibility detail is above average.

Critical weaknesses

Most baseline numbers are lifted from the original papers rather than rerun, so the 81.5 vs 84.5 headline isn’t a like-for-like comparison and shouldn’t be read as a real ranking. The real-robot part is weak, 20 trials per task, no seeds or variance, so a few points of difference probably means nothing. The loss weights are off: lambda_tac=500 and lambda_vis=0.01 are extreme, never swept, and it’s odd to down-weight the visual loss when a “heterogeneous” model leans on vision. “Heterogeneous” also means heterogeneous experts, not sensors, only one GelSight-style modality is tested. And a lot of the positioning depends on preprints that aren’t out yet, which makes it hard to judge how much is really new versus N0-VTLA or TacForeSight. Table 1 is also partly broken in the source. I’ve seen far too much homogenized work recently. During the rebuttal, I’d like to hear the authors discuss what truly sets their methods apart.

Currently, I tend to weak rejection but I keep positive once the author address my concerns.

Questions for authors

None

3 comments
AuthorsOpenReview ↗

- Response to Reviewer qUDk (Part 1/2): Evaluation and Real robot experiments

Thank you for your careful review and constructive feedback. We address your concerns point by point below.

1. Baseline comparisons

We tested most available baselines under the same configurations as TacDyn-WAM and obtained results generally similar to or even lower than the published numbers. But we still retained published numbers for consistency with previous benchmark reports. For methods such as TacForcing, whose code was unavailable for our evaluation, we could only report published UniVTAC results and could not fill in the missing Insert HDMI and Grasp Classify results. We included them for broader comparison. On UniVTAC, Grasp Classify is near saturation for several tactile methods, while Insert HDMI demonstrations contain nearly no informative tactile variation. Although some prior reports omit these tasks, we evaluated all eight and re-evaluated RDP across the full benchmark:

MethodInsert HoleInsert TubeLift CanPull-out KeyPut BottleLift BottleGrasp ClassifyInsert HDMIAvg.
RDP (re-evaluated, %)278115204687941147.6

The UniVTAC results show that TacDyn-WAM achieves 81.5% without additional large-scale visuo-tactile trajectory pretraining, substantially outperforming baselines without such pretraining. It also reaches a comparable performance level to large-scale tactile-pretrained models such as N0-VTLA (83.1%) and N0-TWAM (84.5%), highlighting the effectiveness of our approach with limited tactile trajectory data.

2. Expanded real-robot evaluation

We have expanded evaluation from 20 to 50 trials per task for all five methods, with more varied and randomized initial object placements. Success rates and successful-trial counts are:

MethodStack CupsRemove PlugInsert PlugUnscrew Cup LidWipe WhiteboardAvg.
LingBot-VA30% (15/50)8% (4/50)32% (16/50)10% (5/50)40% (20/50)24.0%
InternVLA-A142% (21/50)14% (7/50)36% (18/50)8% (4/50)58% (29/50)31.6%
FTP-152% (26/50)34% (17/50)50% (25/50)14% (7/50)68% (34/50)43.6%
TacDyn-WAM76% (38/50)56% (28/50)78% (39/50)36% (18/50)92% (46/50)67.6%
TacDyn-WAM (modest pretraining)94% (47/50)84% (42/50)90% (45/50)56% (28/50)98% (49/50)84.4%

Under these harder initialization settings, TacDyn-WAM and its modestly pretrained variant retain 67.6% and 84.4% average success, declining by only 3.4 and 0.6 percentage points, versus 8.0–11.4 points for the baselines. The full model exceeds the strongest baseline, FTP-1, by 24.0 points. The expanded evaluation therefore shows a substantial performance advantage and supports robustness to initial-state variation.

AuthorsOpenReview ↗

Response to Reviewer qUDk (Part 2/2): Loss Coefficient Clarification and Novelty

3. Loss coefficients and heterogeneous prediction spaces

The visual coefficient of 0.01 is inherited directly from InternVLA-A1. It weights auxiliary visual prediction; visual representations also condition action generation and receive gradients from the action objective.

The tactile loss uses Smooth L1 averaged over 768 feature dimensions, with unit-normalized future targets and their differences. If we summed over these dimensions instead of averaging, the same objective would require a tactile coefficient of only about 0.65 (500/768), rather than 500. With all other reductions unchanged, this would produce exactly the same loss and gradients. Thus, the seemingly large coefficient reflects the normalization convention; it can be expressed as a coefficient below 1 without changing the optimization.

The full-model Put Bottle run provides quantitative support: mean raw tactile and visual losses are 1.18 × 10⁻⁴ and 0.235, respectively. Gradient-scale calibration before joint fine-tuning gives an equal-norm tactile coefficient of approximately 838; the chosen coefficient of 500 gives a tactile-to-action ITDE gradient ratio of 0.596 at calibration. With λtac=1, the corresponding ratio would be only 0.00119, or 0.12%. Rescaling therefore gives tactile dynamics supervision a meaningful gradient contribution alongside the action objective.

“Heterogeneous” refers to distinct prediction spaces and objectives, implemented by experts with separate parameters that interact through joint attention: vision predicts reconstruction-oriented Cosmos latents, while touch predicts dynamics-aware TacRep features and their evolution. It does not denote integration across heterogeneous tactile sensors, which is outside this work's scope.

4. What distinguishes our contribution

Our novelty is not predicting a latent, but determining what that latent should represent for tactile prediction. The target spaces serve different purposes:

  • VAE: optimized for pixel reconstruction, tying prediction to appearance that can change sharply under small contact deviations.
  • DINOv2 (N0-VTLA): pretrained on static natural images, with neither tactile-domain adaptation of its frozen backbone nor a temporal training objective. Static visual features are not equivalent to features explicitly trained to capture tactile evolution.
  • CNN-based latents (TacForeSight): simply encoding tactile images into CNN features does not give those features an explicit physical meaning for predicting contact evolution. Calling them “latents” does not establish their suitability for temporal dynamics prediction under tactile-image shifts.

Much existing work reuses frozen visual VAEs or established image encoders without explicitly examining whether their representation objectives suit tactile dynamics. Our starting point is an observed property of visuo-tactile interaction: execution drift can substantially alter tactile appearance while the underlying contact evolution remains predictable. TacRep is trained around the resulting joint requirements of temporal contact dynamics and robustness to tactile-image distribution shifts. To our knowledge, we are among the few works that explicitly investigate and validate this choice of prediction space. Our architecture further separates current tactile understanding from future tactile evolution, allowing each representation to serve its intended purpose.

The distinction is measurable: within the same policy framework, TacRep achieves 81.5%, versus 74.0% with DINOv2 and 62.0% with Cosmos VAE. Without additional large-scale visuo-tactile trajectory pretraining, it remains competitive with the N0 models, including N0-TWAM with 30,000+ hours of pretraining data. This provides concrete empirical evidence of competitive performance with substantially less tactile trajectory training data, highlighting the data efficiency of our approach.

I thank the authors for their detailed rebuttal and experiments, but core concerns regarding unfair baseline comparisons, lack of statistical variance in real-robot trials, and unswept loss weights remain unresolved. Therefore, I maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.