- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes ACTS, a visual-tactile world model, fine-tuned from Cosmos3. The paper collects data, React with a handheld GelSight sensors on three non-prehensile tasks. ACTS outperform two baselines trained from scratch on 4.9 hours of React data.
Critical strengths
- The paper is clearly written and easy to follow.
Critical weaknesses
-
The problem set-up needs a lot more justification. In the intro, the paper argues that comparable robot-free data collection solution, such as TacUMI "retain a fixed contact geometry" and "actions are tied to the collection device". And their solution "define motion and contact at the tactile sensor". But nowhere in the paper did they demonstrate this point in any way. They demonstrate "zero-shot transfer" to robot teleoperation; but the wrist views look the same as with human or robot teleoperation.
-
The experimental results are not very convincing. Their model is initialized with Cosmos3 weights, while the baselines are trained from scratch.
Questions for authors
-
What's the value of predicting tactile information for non-prehensile tasks?
-
PSNR, SSIM, LPIPS are commonly used for visual world models. But, are they correct metrics to use for tactile information?
Thank you for the review and for finding the paper easy to follow. We respond to each point below.
For the weakness: Problem setup
The introduction makes two claims, and we address them separately. Contact geometry. We conducted an extra comparison. We measured where contact lands on the sensing surface in React and in two gripper-based tactile datasets: exUMI, a handheld UMI gripper with two fingertip visuotactile sensors (7 tasks, 45 episodes), and ToucHD-Mani, a robot gripper with markerless GelSight Mini (39 tasks, 113 episodes). TacUMI's public repository contains no tactile stream, so we could not include it. For each tactile stream, we sampled frames at 5 Hz, estimated a per-stream quiet reference, thresholded deformation at six times the reference noise, and tracked the contact centroid and footprint. React contacts move over a wider area of the sensor:
All values are task-equal means, with 95% confidence intervals for centroid displacement obtained by episode bootstrap within task.
Gripper contacts are larger and more centered, as expected when an object is pinched between opposed pads. The two React sensors also move relative to each other: the per-task 90th-percentile pairwise angular distance of their relative orientation is 59°–72°, whereas both fingertips of a gripper sit on one rigid body.
A bare sensor, free of gripper fingers, covers a broader range of contact locations and relative configurations than a handheld gripper.
This comparison is descriptive. The datasets differ in objects and tasks; exUMI uses a different sensor, so we compare only normalized deformation masks; and the statistics measure where contact occurs, not force or information content. We will add the method and table to the appendix.
Actions tied to the device. The robot evaluation (Sec. 6.4, App. C) shows that with actions defined at the sensor, a model trained only on human data predicts robot interaction without fine-tuning. The robot carries the same device, so this result alone does not separate our interface from a gripper-based one, and the wrist views look alike for the same reason. The transfer still fails in places: on Motherboard, ACTS falls below frame copying on every camera stream (Sec. 6.4).
Pretrained backbone
We agree the comparison does not isolate pretraining. VT-WM-style receives the same five streams, eight-frame history, VAE latents, and actions as ACTS (App. G), so the two differ in the predictor and its initialization. The force ablation in App. H holds the backbone fixed.
For the questions:
Pushing and sliding depend on whether, where, and how hard the sensor touches the object, and under occlusion the cameras often cannot see this. Pick and placing tasks depend less on contact dynamics (it only matters when making grasping/releasing). Fig. 6 shows an example: with and without the force term, the wrist-camera predictions are nearly identical, and only the tactile stream shows whether contact forms.
Not on their own, and we discussed this in limitation section. Full-frame GelSight scores are dominated by static gel and do not show whether contact is made or broken at the right time, which we assessed only qualitatively. Contact-specific measures would be better, for example onset and release timing from a force estimator run on predicted frames.