- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
RoboTactile benchmarks vision-tactile manipulation under 14 types of tactile observation failures. It builds on UniVTAC and evaluates four models under faults such as missing frames, delayed signals, and distorted observations. The results show different failure patterns across tasks and models.
Critical strengths
I like the distinction between missing touch and touch that is present but misleading. The task-level breakdown is also more useful than a single robustness score. For example, Table 3 reports 8% success for N0-TWAM on Lift Bottle under Local Nonresponsive Patch, compared with 52% under Stream Absence. This is an interesting contrast, even though the fault settings need better control.
Critical weaknesses
The experiments are not documented well enough. Appendix 1.8 says the available records lack trial counts, seeds, checkpoint versions, and the actual fault parameters and schedules. The operator definitions are detailed, but they do not tell us how the reported experiments were run. This makes the numbers hard to assess or reproduce. The clean scores for both N0 models also come from other papers, and the vision-only references are not verified matched controls. The authors acknowledge this, but it still leaves the clean-to-fault gaps and tactile gain retention without a controlled comparison.
The most interesting parts of the benchmark are not evaluated. Contact-phase sensitivity, recovery after touch returns, and offline prediction are all outside the current experiments. These could provide useful insight beyond showing that input corruption hurts performance. Without them, the demonstrated contribution is much narrower than the proposed framework. The fault examples are also synthetic interventions, not calibrated observations of physical sensor failures. A full real-robot study is not necessarily required, but some evidence supporting the chosen fault strengths and timing would make the benchmark more convincing.
Questions for authors
- How were the reported fault scores generated? Please provide the trial counts, checkpoints, fault parameters, exposure windows, and affected sensors.
- Are matched clean/fault evaluations and matched vision-only results available for any model?
- Table A.2 reports 99% success for ACT on Grasp Classify under both Clean and Stream Absence. For this specific result, please report which sensors were interrupted, the interruption onset and duration, and whether it covered the contact observations needed for classification. How does ACT’s adapter handle interrupted delivery, including any retained tactile images or feature history?