acceptodds
Withdrawn from ICLR 2027

SoftVTBench: A Deformation-Aware Visuo-Tactile Dataset and Benchmark for Deformable-Object Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Robot learning typically evaluates manipulation by task outcome, yet task success alone cannot distinguish a stable grasp from one that completes the task by overcompressing a deformable object. Evaluating both requires a record that existing datasets generally lack: tactile observations available to the policy and physical state available only to the evaluator. We introduce SoftVTBench, a simulated visuo-tactile dataset and benchmark for deformable-object manipulation, instantiated through complete pick-and-place tasks. The dataset contains 4,000 demonstrations across matched deformable objects and rigid twins, pairing policy-visible vision, touch, proprioception, language, and actions with evaluator-only finite-element states. The benchmark introduces the Deformation-aware Success Rate (DSR), which credits a rollout only if it completes the task without exceeding an object-specific deformation tolerance calibrated before policy training. Across three policy families, this evaluation reveals behavior that task success alone obscures: every in-distribution configuration contains successful rollouts that exceed the tolerance; configurations with nearly identical task success can differ substantially in deformation compliance; and, in a controlled ablation, finer gripper control can reproduce an apparent modality gain. Visuo-tactile variants achieve higher absolute out-of-distribution task success in all six evaluated comparisons, while their in-distribution effects are mixed. Within this simulated setting, these results show that task completion and deformation compliance capture distinct aspects of policy behavior: higher task success does not necessarily imply better deformation compliance.

Primary area:
applications to robotics, autonomy, planning
Keywords:
visuo-tactile learning, deformable object manipulation, robot manipulation, tactile sensing, physical interaction, deformation-aware evaluation, dataset, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_8A4kOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces SoftVTBench, a fully simulated visuo-tactile dataset and benchmark for deformable-object manipulation. The benchmark focuses on deformation-aware pick-and-place tasks and contains 4,000 demonstrations across deformable objects and matched rigid twins, with synchronized RGB observations, tactile sensing, proprioception, language instructions, and robot actions. In addition to policy-visible observations, the simulator also records evaluator-only FEM states that are used to measure object deformation during execution.

Critical strengths

The paper identifies a meaningful gap in current manipulation benchmarks: standard task success metrics do not distinguish between successful but overly aggressive manipulation and genuinely deformation-aware handling. The proposed DSR metric is simple, easy to interpret, and provides additional diagnostic information beyond conventional success rates. This is an important contribution because many existing manipulation benchmarks focus almost exclusively on task completion.

Critical weaknesses
  1. The entire benchmark is simulation-only, including both the tactile sensing pipeline and the deformable-object dynamics, which substantially limits the strength of the paper’s conclusions. The tactile observations are generated through TacEx/Taxim/FOTS rendering, and the deformable objects rely on a simplified FEM model with corotational linear elasticity. However, the paper does not provide any validation that these simulated tactile signals or deformation measurements correspond well to real visuo-tactile interaction on the benchmark assets. Since the main claims of the paper concern deformation-aware manipulation behavior, it is unclear to what extent the reported findings reflect meaningful physical phenomena rather than artifacts of the simulator stack.

2.For deformable-object manipulation, the fidelity of tactile simulation and soft-body simulation is arguably the central challenge. In this work, the benchmark conclusions are built entirely on top of these simulated components, yet both the tactile rendering and deformation modeling appear relatively simplified and are not validated against real-world measurements. As a result, the practical significance of the benchmark is difficult to assess: if the underlying contact and deformation dynamics are inaccurate, then metrics such as DSR may mostly reflect compliance with the simulator’s internal assumptions rather than meaningful real-world manipulation quality. This concern is especially important because the paper positions itself as a benchmark for studying visuo-tactile deformable manipulation rather than as a purely simulation-engineering study.

Questions for authors
  • The benchmark relies entirely on simulated tactile sensing and simulated deformable-object dynamics. Do the authors have any evidence that the tactile signals and deformation measurements used in SoftVTBench correlate with real-world behavior on similar objects or sensors? Even a limited qualitative or quantitative comparison would help assess whether the benchmark captures meaningful visuo-tactile manipulation phenomena rather than simulator-specific artifacts.

  • The deformation tolerances are calibrated from simulator rollouts using a fixed percentile heuristic. How sensitive are the main conclusions to this calibration choice? For example, would the relative rankings or TSR–DSR gaps change substantially under different tolerance definitions?

  • The paper argues that DSR captures “gentle” or deformation-aware manipulation behavior. However, the underlying deformation signal is derived from a simplified FEM model with linear elasticity. Could the authors clarify what types of real-world deformation behaviors are expected to be faithfully represented by this simulator setup, and which are likely outside its validity range?

Reviewer_fkcwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SoftVTBench is a simulation‑only visuo‑tactile benchmark for deformable‑object pick‑and‑place manipluation task. It contains 4000 expert demonstrations for deformable objects and their rigid twins.

Critical strengths
  1. The experimental setup has clean controls. The authors use rigid‑deformable object twins, test binary / continuous gripper control, and systematically shift mass, stiffness and lighting. These careful confounder analyses would be extremely hard to run on real‑human collected data.
  2. DSR seems interesting. Most benchmarks only check whether a task finishes. DSR additionally enforces safe deformation limits for soft objects.
  3. With simulation, closed-loop evaluations are conducted.
Critical weaknesses
  1. All demonstrations and tactile observations are synthetic. How large will it be sim-2-real gap?
  2. Limited manipulation skill (restricted to pick‑and‑place). Complex functional tool‑use tasks are not supported.
  3. No high‑level semantic‑cognition QA benchmark. Will it be useful for training VLM?
Questions for authors

See the weakness.

Reviewer_tEH9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes SoftVTBench, a benchmark based on data collected in Isaac Sim/PhysX, together with a closed-loop evaluation protocol. It provides over 4,000 pick-and-place demonstrations involving 10 soft objects and their rigid counterparts. The inputs include vision, tactile, and proprioceptive observations, while FEM states are available only to the evaluator. The experiments use Diffusion Policy, π0.5, and FastWAM and report the relevant metrics.

The paper research question of the paper is interesting. However, the current work demonstrates this phenomenon only under a specific simulation setup. The evidence is insufficient to support its positioning as a general-purpose visuo-tactile benchmark or to substantiate some of its conclusions about the benefits of tactile sensing and generalisability.

Critical strengths

This paper is motivated by an interesting observation: task success cannot distinguish a stable grasp from one that completes the task by overcompressing a deformable object.

Critical weaknesses
  1. The role of tactile feedback is not fully clear from the results. The paper compares visuo-tactile policies with policies using vision and proprioception, but adding tactile input does not consistently improve DSR. It would help to show which contact situations benefit from tactile feedback.

  2. The practical meaning of the deformation threshold is unclear. The paper defines the threshold as the 90th percentile of deformation observed during scripted stable-grasp calibration and describes it as an operational criterion. However, it does not explain why the 90th percentile was chosen, show how DSR changes with other thresholds, or link the threshold to an acceptable level of deformation in practice. A sensitivity analysis would help clarify how the threshold affects DSR.

  3. The benchmark would benefit from further validation. In my experience, Isaac Sim/PhysX contact behavior can differ from real-world interactions, even for rigid bodies. This makes validation particularly relevant for the soft-body and tactile simulations used here. A sim-to-sim or sim-to-real comparison would help establish the benchmark’s reliability and relevance to physical manipulation.

  4. Figures 1 and 2 are crowded, and some labels are small. The object images in Figure 1 give me the impression that they may have been generated or substantially edited, although I may be mistaken. Since the visual and tactile appearance is important to this work, could the authors clarify what “figure polishing” involved and whether these images faithfully show the original simulation outputs?

  5. The tactile comparisons are based on a single training run per configuration. The results therefore provide limited evidence about whether the observed differences are consistent across runs. Results from multiple training seeds would help assess the effect of tactile input.

  6. The OOD evaluation covers a limited set of shifts. It tests single-factor changes in illumination, mass, and stiffness, while combined shifts and changes to objects, contact conditions, sensors, or task structure are not evaluated. The generalisation claims should be interpreted within the specific shifts tested.

  7. More detail on the material parameters would improve reproducibility. The paper would benefit from reporting each object’s nominal Young’s modulus (with units) and explaining how these values were obtained. It would also help to clarify how the ×0.5, ×0.8, and ×2.0 OOD values were selected and whether they are based on physical measurements.

  8. The object and task diversity is limited. The ten deformable objects consist mainly of bakery-style assets and simple geometric primitives, and all four task suites use variations of pick-and-place. This limits the extent to which the results support conclusions about deformable-object manipulation more broadly.

Questions for authors
  1. Which specific failure modes in these pick-and-place tasks are addressed by tactile input beyond vision and proprioception?
  2. What is the rationale for using the 90th percentile as the deformation threshold? Do the main DSR results and policy rankings remain consistent with alternative thresholds, such as the 80th or 95th percentile?
  3. Could the authors provide a sim-to-sim or sim-to-real validation of the soft-body deformation and tactile observations? Are the bakery-style objects based on physical counterparts?
  4. Do the reported tactile effects persist across multiple training seeds?
  5. What are the nominal Young’s modulus values for each object, how were they obtained, and how were the ×0.5, ×0.8, and ×2.0 OOD values selected?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.