- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces SoftVTBench, a fully simulated visuo-tactile dataset and benchmark for deformable-object manipulation. The benchmark focuses on deformation-aware pick-and-place tasks and contains 4,000 demonstrations across deformable objects and matched rigid twins, with synchronized RGB observations, tactile sensing, proprioception, language instructions, and robot actions. In addition to policy-visible observations, the simulator also records evaluator-only FEM states that are used to measure object deformation during execution.
Critical strengths
The paper identifies a meaningful gap in current manipulation benchmarks: standard task success metrics do not distinguish between successful but overly aggressive manipulation and genuinely deformation-aware handling. The proposed DSR metric is simple, easy to interpret, and provides additional diagnostic information beyond conventional success rates. This is an important contribution because many existing manipulation benchmarks focus almost exclusively on task completion.
Critical weaknesses
- The entire benchmark is simulation-only, including both the tactile sensing pipeline and the deformable-object dynamics, which substantially limits the strength of the paper’s conclusions. The tactile observations are generated through TacEx/Taxim/FOTS rendering, and the deformable objects rely on a simplified FEM model with corotational linear elasticity. However, the paper does not provide any validation that these simulated tactile signals or deformation measurements correspond well to real visuo-tactile interaction on the benchmark assets. Since the main claims of the paper concern deformation-aware manipulation behavior, it is unclear to what extent the reported findings reflect meaningful physical phenomena rather than artifacts of the simulator stack.
2.For deformable-object manipulation, the fidelity of tactile simulation and soft-body simulation is arguably the central challenge. In this work, the benchmark conclusions are built entirely on top of these simulated components, yet both the tactile rendering and deformation modeling appear relatively simplified and are not validated against real-world measurements. As a result, the practical significance of the benchmark is difficult to assess: if the underlying contact and deformation dynamics are inaccurate, then metrics such as DSR may mostly reflect compliance with the simulator’s internal assumptions rather than meaningful real-world manipulation quality. This concern is especially important because the paper positions itself as a benchmark for studying visuo-tactile deformable manipulation rather than as a purely simulation-engineering study.
Questions for authors
-
The benchmark relies entirely on simulated tactile sensing and simulated deformable-object dynamics. Do the authors have any evidence that the tactile signals and deformation measurements used in SoftVTBench correlate with real-world behavior on similar objects or sensors? Even a limited qualitative or quantitative comparison would help assess whether the benchmark captures meaningful visuo-tactile manipulation phenomena rather than simulator-specific artifacts.
-
The deformation tolerances are calibrated from simulator rollouts using a fixed percentile heuristic. How sensitive are the main conclusions to this calibration choice? For example, would the relative rankings or TSR–DSR gaps change substantially under different tolerance definitions?
-
The paper argues that DSR captures “gentle” or deformation-aware manipulation behavior. However, the underlying deformation signal is derived from a simplified FEM model with linear elasticity. Could the authors clarify what types of real-world deformation behaviors are expected to be faithfully represented by this simulator setup, and which are likely outside its validity range?