SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation

Robot learning typically evaluates manipulation by task outcome, yet task success alone cannot distinguish a stable grasp from one that completes the task by overcompressing a deformable object. Evaluating both requires a record that existing datasets generally lack: tactile observations available to the policy and physical state available only to the evaluator. We introduce SoftVTBench, a simulated visuo-tactile dataset and benchmark for deformable-object manipulation, instantiated through complete pick-and-place tasks. The dataset contains 4,000 demonstrations across matched deformable objects and rigid twins, pairing policy-visible vision, touch, proprioception, language, and actions with evaluator-only finite-element states. The benchmark introduces the Deformation-aware Success Rate (DSR), which credits a rollout only if it completes the task without exceeding an object-specific deformation tolerance calibrated before policy training. Across three policy families, this evaluation reveals behavior that task success alone obscures: every in-distribution configuration contains successful rollouts that exceed the tolerance; configurations with nearly identical task success can differ substantially in deformation compliance; and, in a controlled ${\pi }_{0.5}$ ablation, finer gripper control can reproduce an apparent modality gain. Visuo-tactile variants achieve higher absolute out-of-distribution task success in all six evaluated comparisons, while their in-distribution effects are mixed. Within this simulated setting, these results show that task completion and deformation compliance capture distinct aspects of policy behavior: higher task success does not necessarily imply better deformation compliance.