acceptodds
Withdrawn from ICLR 2027

ACTS: Learning Visuo-Tactile World Models from Human Interaction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Action-conditioned world models for contact-rich manipulation require data that captures how contact forms, slides, and releases under varying force. Such data are slow to collect through robot teleoperation, which often lacks force feedback and is constrained by gripper geometry. We ask whether these dynamics can instead be learned from human physical interaction and transferred to a robot. We present ACTS, a visuo-tactile world model trained only on human interaction data. Operators manipulate handheld GelSight sensors while we record third-person and wrist video, tactile images, 6-DoF sensor motion, and tactile-derived normal force, yielding React, a 4.9-hour bimanual dataset across three objects. We represent actions at the tactile sensor as motion augmented by force change, providing a shared action space for human collection and robot execution. ACTS jointly predicts future scene, wrist, and tactile observations. On held-out human data, ACTS improves scene-view prediction by 0.7–2.0 dB PSNR over two visuo-tactile world models trained from scratch on React. Without fine-tuning, predicted tactile imprints on robot teleoperation still appear and disappear with recorded contact. Visual transfer is less robust: camera predictions match or exceed frame copying on two of three objects, while wrist views can blur or hallucinate structure. These results suggest that contact dynamics transfer more reliably than visual appearance from human to robot. Project website and supplementary videos: https://acts-anon.github.io/

Primary area:
applications to robotics, autonomy, planning
Keywords:
Visuo-tactile world model, tactile sensing, contact-rich manipulation, multimodal learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6wvgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes ACTS, a visual-tactile world model, fine-tuned from Cosmos3. The paper collects data, React with a handheld GelSight sensors on three non-prehensile tasks. ACTS outperform two baselines trained from scratch on 4.9 hours of React data.

Critical strengths
  1. The paper is clearly written and easy to follow.
Critical weaknesses
  1. The problem set-up needs a lot more justification. In the intro, the paper argues that comparable robot-free data collection solution, such as TacUMI "retain a fixed contact geometry" and "actions are tied to the collection device". And their solution "define motion and contact at the tactile sensor". But nowhere in the paper did they demonstrate this point in any way. They demonstrate "zero-shot transfer" to robot teleoperation; but the wrist views look the same as with human or robot teleoperation.

  2. The experimental results are not very convincing. Their model is initialized with Cosmos3 weights, while the baselines are trained from scratch.

Questions for authors
  1. What's the value of predicting tactile information for non-prehensile tasks?

  2. PSNR, SSIM, LPIPS are commonly used for visual world models. But, are they correct metrics to use for tactile information?

1 comment

Thank you for the review and for finding the paper easy to follow. We respond to each point below.

For the weakness: Problem setup

The introduction makes two claims, and we address them separately. Contact geometry. We conducted an extra comparison. We measured where contact lands on the sensing surface in React and in two gripper-based tactile datasets: exUMI, a handheld UMI gripper with two fingertip visuotactile sensors (7 tasks, 45 episodes), and ToucHD-Mani, a robot gripper with markerless GelSight Mini (39 tasks, 113 episodes). TacUMI's public repository contains no tactile stream, so we could not include it. For each tactile stream, we sampled frames at 5 Hz, estimated a per-stream quiet reference, thresholded deformation at six times the reference noise, and tracked the contact centroid and footprint. React contacts move over a wider area of the sensor:

MetricReactexUMIToucHD-Mani
RMS centroid displacement (normalized by sensor diagonal)0.19 [0.18, 0.20]0.07 [0.06, 0.08]0.07 [0.07, 0.08]
Fraction of sensing surface touched0.520.320.26
Normalized contact-location entropy (4×4 grid)0.720.220.26
Mean contact patch area (fraction of surface)0.050.170.10

All values are task-equal means, with 95% confidence intervals for centroid displacement obtained by episode bootstrap within task.

Gripper contacts are larger and more centered, as expected when an object is pinched between opposed pads. The two React sensors also move relative to each other: the per-task 90th-percentile pairwise angular distance of their relative orientation is 59°–72°, whereas both fingertips of a gripper sit on one rigid body.

A bare sensor, free of gripper fingers, covers a broader range of contact locations and relative configurations than a handheld gripper.

This comparison is descriptive. The datasets differ in objects and tasks; exUMI uses a different sensor, so we compare only normalized deformation masks; and the statistics measure where contact occurs, not force or information content. We will add the method and table to the appendix.

Actions tied to the device. The robot evaluation (Sec. 6.4, App. C) shows that with actions defined at the sensor, a model trained only on human data predicts robot interaction without fine-tuning. The robot carries the same device, so this result alone does not separate our interface from a gripper-based one, and the wrist views look alike for the same reason. The transfer still fails in places: on Motherboard, ACTS falls below frame copying on every camera stream (Sec. 6.4).

Pretrained backbone

We agree the comparison does not isolate pretraining. VT-WM-style receives the same five streams, eight-frame history, VAE latents, and actions as ACTS (App. G), so the two differ in the predictor and its initialization. The force ablation in App. H holds the backbone fixed.

For the questions:

  1. Why predict touch in non-prehensile tasks?

Pushing and sliding depend on whether, where, and how hard the sensor touches the object, and under occlusion the cameras often cannot see this. Pick and placing tasks depend less on contact dynamics (it only matters when making grasping/releasing). Fig. 6 shows an example: with and without the force term, the wrist-camera predictions are nearly identical, and only the tactile stream shows whether contact forms.

  1. Are PSNR, SSIM, and LPIPS the right metrics for tactile images?

Not on their own, and we discussed this in limitation section. Full-frame GelSight scores are dominated by static gel and do not show whether contact is made or broken at the right time, which we assessed only qualitatively. Contact-specific measures would be better, for example onset and release timing from a force estimator run on predicted frames.

Reviewer_tdYrOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ACTS, a visuo-tactile world model trained on human interaction data to predict future visual and tactile observations. The authors introduce the React dataset and a sensor-centric action representation combining relative motion and estimated contact forces to support human-to-robot transfer. Experiments evaluate prediction performance on human demonstrations and zero-shot transfer to robot teleoperation trajectories, showing improvements in scene-view prediction and qualitative evidence of contact-transition prediction.

Critical strengths
  1. Sensor-centric action representation. Defining actions at the tactile sensor rather than through embodiment-specific joint coordinates provides a reasonable interface for modeling human and robot interactions. Although its components build on existing methods, their integration offers a concrete approach to investigating human-to-robot transfer of visuo-tactile prediction.

  2. Real-world human-to-robot evaluation. The authors test a human-trained world model on separately collected robot teleoperation trajectories without robot-specific fine-tuning. The work includes quantitative prediction evaluations on human data and qualitative observations of contact transitions on robot recordings.

Critical weaknesses
  1. Future force conditioning complicates the interpretation of contact prediction. The action representation incorporates ft+1f_{t+1}, estimated from future recorded tactile observations. Since this provides information about the contact outcome being predicted, the reported performance does not clearly establish how accurately ACTS can predict contact dynamics from sensor motions alone.

  2. Insufficient isolation of methodological contributions. ACTS uses a large pretrained Cosmos3 backbone, whereas the compared world-model baselines are trained from scratch. Differences in model initialization, capacity, and conditioning context make it difficult to attribute the reported prediction improvements to the proposed action representation and multimodal modeling design.

Questions for authors
  1. Could the authors provide contact-specific metrics, such as contact-state F1, contact-region IoU, and contact onset/release timing errors, to quantitatively evaluate the model's ability to predict contact dynamics?

  2. Could the authors compare visual-only, tactile-only, and joint visuo-tactile conditioning under a controlled setting to clarify whether cross-modal information improves prediction performance?

  3. Could the authors provide a quantitative ablation without ft+1f_{t+1}, using only sensor-motion conditioning, to determine how much of the reported contact prediction performance depends on future force information?

  4. Could the authors provide more controlled ablations of the proposed action representation and conditioning components, ideally with comparable pretrained backbones, to distinguish their contributions from the benefits of Cosmos3 pretraining?

  5. Could the authors further analyze the object- and view-dependent performance degradation in robot evaluation, particularly on Motherboard, and identify the main factors limiting transfer from human to robot interactions?

1 comment

Thank you for the review, and for recognizing the sensor-level action interface and the zero-shot robot evaluation.

  1. The paper includes none of the contact-specific metrics, and Sec. 7 says that full-frame tactile scores cannot show whether contact is made or broken at the right time. We agree that contact-state F1, imprint IoU, and onset/release timing might serve as better metrics.

  2. We have one partial comparison, not in the submission: with the same backbone, a scene-only model matches the five-stream model on 8-frame scene PSNR (within 0.26 dB on all three objects) and is better in long rollouts. The wrist and tactile streams do not improve scene prediction. Our tactile-only model performs much worse (can not tell from the visual metrics though), so vision does help tactile prediction.

  3. (weakness 1) The motion-only ablation is in App. H (Tab. 6): the same model without the force term, trained and tested identically. Camera PSNR changes by at most 0.3 dB, never significantly. With the force term, tactile PSNR rises significantly on both PushT sensors and SSIM/LPIPS improve on the left Motherboard sensor, while PSNR drops by 0.3 dB on the right Rope sensor. In Fig. 6, only the force-conditioned model forms the imprint when the sensor reaches the block. Contact onset in that window depends on the force term, and we do not claim that ACTS predicts contact from sensor motion alone. The term is defined as the target of a compliance controller (App. E, following DexForce), but our robot episodes did not log commands, so the paper does not test whether it can be set in advance.

  4. We agree that pretraining is not isolated. One clarification on conditioning context: VT-WM-style receives the same five streams, eight-frame history, VAE latents, and actions as ACTS (App. G); only ContactWorld runs in its native single-frame, two-stream setting (App. I). The force ablation is controlled on the same backbone; the paper has no view-action ablation and no pretrained baseline.

  5. Sec. 6.4 gives a qualitative analysis. Slow robot motion makes frame copying a strong baseline. Motherboard has many fine edges from connectors, slots, and heat sinks, and the robot wrist cameras record it darker than the human data, even after color and tone mapping (App. B). Across objects, the wrist views degrade most, while tactile imprints still follow recorded contact (App. C). We did not separate these factors quantitatively.

Reviewer_2zuuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents ACTS, a model that predicts future scene, wrist-camera, and tactile videos. It takes past videos and a recorded action sequence as input. The actions describe sensor motion, estimated force changes, and sensor geometry in the scene view. The authors also collect React, a dataset with 4.9 hours of human interaction with three objects. The model is trained on human interaction data and tested on both held-out human recordings and robot recordings. It improves visual prediction on human data and shows mixed results on robot data. Qualitative examples suggest that some tactile contact changes transfer from human to robot interaction.

Critical strengths

The paper raises an important and ambitious problem. Using human interaction to train robot world models could reduce the dependence on robot-collected data. The paper takes a concrete step toward this goal through a synchronized dataset and a human-to-robot prediction evaluation.

The sensor-level action representation is a thoughtful design choice. Describing motion and loading at the sensor provides a common interface for human and robot recordings. This avoids requiring a direct correspondence between human joints and robot joints.

Critical weaknesses
  1. Future force information makes it hard to tell what results benefit from. The model receives force changes estimated from ground-truth tactile images within the future prediction window. This information is used during both training and evaluation. It already provides clues about whether and when contact occurs. Therefore, the results do not clearly establish that ACTS can infer future contact from past observations and planned motion alone. Conditioning on a known loading trajectory can be useful. However, the paper does not explain how the required future force values would be obtained before robot execution. Actual force estimates from recorded tactile images are not necessarily equivalent to force commands that a robot can specify in advance.
  2. The tactile evaluation does not clearly show whether contact changes are predicted correctly. Tactile videos may contain many no-contact frames or frames with little change. Even during contact, much of the image can remain unchanged. The paper includes a frame-copying baseline, which is a useful control. However, robot tactile results remain difficult to interpret: copying achieves higher PSNR on five of the six streams, while ACTS achieves better LPIPS on five. It will help a lot if the authors could categorize the tactile sequences into no contact, stable contact, and contact transitions, and report the proportion of frames or duration in each category. Reporting prediction performance separately for these categories would help clarify whether ACTS captures meaningful changes.
  3. The comparisons do not clearly explain why ACTS performs better. ACTS uses a large pretrained video backbone, while the comparison models are trained from scratch on React. ContactWorld also receives fewer observation streams and a shorter history. These differences make it difficult to attribute the gains to the proposed design. The paper already includes a force ablation. However, this ablation should also be evaluated with contact-specific metrics. Additional ablations should remove the view-aligned actions and compare joint prediction with separate visual and tactile prediction. These comparisons should use the same pretrained backbone and matched training settings.
  4. The claimed advantage of human data collection is not directly demonstrated. I am not yet convinced that human collection is cheaper or easier in the controlled tabletop settings studied here. Some interactions may also be collected through repeated robot execution. At the same time, the human setup still requires external motion tracking and several sensing devices.
  5. The limited object coverage leaves generalization unclear. The dataset contains only three objects, and a separate model is trained for each object. Robot evaluation uses the same objects and sensing devices. There is no evaluation on unseen objects. It is difficult to tell whether the model learns general contact dynamics or relies heavily on object-specific patterns.
  6. (Minor) Many frames in Figure 3 look very similar at the displayed scale, making it difficult to follow the motion trajectory and compare the predicted changes with the ground truth. Larger crops, motion overlays, or highlighted contact regions would make it easier to follow.
Questions for authors
  1. Are all future force conditions estimated from the corresponding ground-truth tactile images during evaluation? How would these conditions be obtained before robot execution, and can the model operate using only planned motion and information available from the observation history?
  2. What proportions of the video data contain no contact, stable contact, and contact transitions? Can you compare ACTS, frame copying, and the pose-only ablation using contact-specific metrics?
  3. Can you provide controlled ablations of view-aligned actions and joint multimodal prediction using the same pretrained backbone and matched training settings?
  4. What advantages does handheld human collection offer over robot teleoperation in the studied tabletop settings? Could you provide evidence that it reduces collection time or effort, or covers a wider range of contact positions, orientations, and forces under a comparable collection budget?
  5. What motivated the choice of three objects and separate models for each object? Can you provide any evidence of generalization to unseen objects?
1 comment

Thank you for the careful review. We answer in order and say plainly where the submission falls short.

  1. Test actions are computed exactly as training actions, so at evaluation the force term comes from the normal force estimated on the recorded tactile frames. The motion term likewise uses the recorded future pose, as is usual when evaluating action-conditioned world models.

We should have been clearer about what the force term stands for. Following DexForce, App. E (Eq. 8) defines a virtual target p=p+kffRns, where kf=0.5 mm/N is an assumed controller compliance. The action is the step of this target, i.e., the command a compliance controller with this stiffness would track, producing force f once in contact. A planner would choose kff rather than read it from future tactile frames. You are right, though, that we did not demonstrate this in paper. Our GELLO teleoperation episodes did not log commanded targets, so the robot actions are also reconstructed from recorded pose and estimated force.

On whether the model can work from planned motion and history alone, the answer is yes. We have tested action following at inference time by conditioning visuo-tactile generation on actions specifying different levels of contact force (pressing harder or lighter against objects or the table). Even when initialized from scenes with no contact, the model can correctly predict contact making in response to the specified actions (although tactile imprints might be off for some texture-rich objects, like motherboard).

  1. We agree, and Sec. 7 ("Evaluation metrics") says the same: full-frame GelSight scores are dominated by static gel and cannot show whether contact is made or broken at the right time, which we assessed only qualitatively. Sec. 6.4 reports the robot result you cite in those terms; slow robot motion makes frame copying a strong baseline.

The split you propose into no contact, stable contact, and contact transitions, with per-category scores for ACTS, Copy, and the pose-only model, is the analysis this evaluation lacks. Our force estimator could supply the labels and, on predicted frames, onset and release timing.

  1. One clarification: fewer streams and a shorter history apply only to ContactWorld, which we ran in its native setting of one visual and one tactile stream with single-frame context (App. I). VT-WM-style is matched to ACTS on inputs, with the same five streams, eight-frame history, frozen Wan2.2 latents, and action vectors (App. G). What differs is the predictor, a 116.6M-parameter transformer trained from scratch versus the pretrained Cosmos3-Nano generator adapted with LoRA. We agree this does not separate pretraining from the rest of the design.

For joint versus separate prediction, one controlled comparison with the same backbone and training, not in the submission, pits a scene-only model against the five-stream model. Over 8 frames, scene PSNR is:

ObjectScene-onlyFive-stream
Rope28.74 dB28.48 dB
PushT28.96 dB29.14 dB
Motherboard27.80 dB27.56 dB

In long rollouts, the scene-only model is better on all three objects, e.g., 23.58 vs. 22.99 dB on Rope. Joint prediction therefore does not improve the scene view; what it adds is that one model also predicts the wrist and tactile streams. We have no ablation of the view-aligned actions and no tactile-only model.

  1. We did not measure collection time, effort, or coverage against a matched robot budget, and our 15 minutes of robot data are an evaluation set, so the paper does not show that human collection is cheaper. The introduction's sentence on teleoperation speed is motivation from prior work, not our own measurement.

The paper shows something narrower: with actions defined at the sensor, hand-collected data can predict robot interaction with the same sensors, and operators can vary contact location, orientation, and loading without a gripper in the way. Sec. 7 ("Motion capture") lists the tracking requirement as a limitation.

  1. The three objects cover different contact conditions: a rigid block pushed and pivoted on the table (PushT), a deformable rope, and a rigid motherboard with dense fine geometry.

We trained one model per object and did not evaluate on unseen objects, so our results speak to these three objects only.

  1. For Figure 3, larger crops around the contact region, with the imprint location marked, would make the comparison easier to follow.

Thank you again, especially for the contact-category suggestion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.