acceptodds
Withdrawn from ICLR 2027

RoboTactile: Benchmarking the Robustness of Visuo-Tactile Manipulation under Tactile Observation Failures

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tactile perception plays a vital role in precise robotic manipulation. Despite their strong performance, existing vision-tactile models may suffer from tactile observation failures caused by contact-induced wear, sensor noise, and signal delays, raising concerns about the reliability of robotic systems. However, existing benchmarks primarily focus on performance under ideal tactile conditions, leaving robustness to tactile failures insufficiently explored. To systematically evaluate this robustness, we introduce RoboTactile, a benchmark built on UniVTAC and its Isaac Sim environment. Grounded in sensing and deployment failure mechanisms, we design 14 controlled tactile observation faults spanning four dimensions: Observation Availability, Signal Fidelity, Temporal Integrity, and Contextual Consistency. The benchmark records contact-phase annotations, with restored-input evaluation defined as an extension. RoboTactile establishes track-specific protocols for offline prediction and closed-loop manipulation across world models, world–action models, policies, and vision–language–action models. Experiments on four models reveal substantial task- and fault-dependent variation. We report absolute success, performance gaps, and descriptive tactile gain retention relative to available visual references, while distinguishing these from verified matched-control estimates. Matched visual controls, phase-conditioned evaluation, and post-restoration evaluation remain future work. We hope RoboTactile will support the development and evaluation of robust vision-tactile models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
TODO.

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_svWbOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

RoboTactile benchmarks vision-tactile manipulation under 14 types of tactile observation failures. It builds on UniVTAC and evaluates four models under faults such as missing frames, delayed signals, and distorted observations. The results show different failure patterns across tasks and models.

Critical strengths

I like the distinction between missing touch and touch that is present but misleading. The task-level breakdown is also more useful than a single robustness score. For example, Table 3 reports 8% success for N0-TWAM on Lift Bottle under Local Nonresponsive Patch, compared with 52% under Stream Absence. This is an interesting contrast, even though the fault settings need better control.

Critical weaknesses

The experiments are not documented well enough. Appendix 1.8 says the available records lack trial counts, seeds, checkpoint versions, and the actual fault parameters and schedules. The operator definitions are detailed, but they do not tell us how the reported experiments were run. This makes the numbers hard to assess or reproduce. The clean scores for both N0 models also come from other papers, and the vision-only references are not verified matched controls. The authors acknowledge this, but it still leaves the clean-to-fault gaps and tactile gain retention without a controlled comparison.

The most interesting parts of the benchmark are not evaluated. Contact-phase sensitivity, recovery after touch returns, and offline prediction are all outside the current experiments. These could provide useful insight beyond showing that input corruption hurts performance. Without them, the demonstrated contribution is much narrower than the proposed framework. The fault examples are also synthetic interventions, not calibrated observations of physical sensor failures. A full real-robot study is not necessarily required, but some evidence supporting the chosen fault strengths and timing would make the benchmark more convincing.

Questions for authors
  • How were the reported fault scores generated? Please provide the trial counts, checkpoints, fault parameters, exposure windows, and affected sensors.
  • Are matched clean/fault evaluations and matched vision-only results available for any model?
  • Table A.2 reports 99% success for ACT on Grasp Classify under both Clean and Stream Absence. For this specific result, please report which sensors were interrupted, the interruption onset and duration, and whether it covered the contact observations needed for classification. How does ACT’s adapter handle interrupted delivery, including any retained tactile images or feature history?
Reviewer_sZaTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

ROBOTACTILE presents a benchmarking method to measure the robustness of policies under various tactile conditions. The paper presents 14 tactile observation faults and compares various. methods using these.

Critical strengths

The motivation and idea are sound and relevant. They are also timely with where the literature in tactile and visuo-tactile policies is going.

Critical weaknesses

Overall, the paper is relevant and useful with a good motivation. However, I have some reservations because the paper reads as an unfinished draft due to several gaps in the experiments. Some of the major ones are as follows:

  1. Line 108-109 says that there are separate tracks for world-models, but no world model is actually tested and offline prediction protocol seems to have never been run.
  2. The N0-TWAM and N0-VTLA numbers seem to come from the original papers, not from running them in RoboTactile. So the delta Q for those could be the effect of different seeds, episode.
  3. It seems that the Contact Phases are recorded but never used in analysis.
  4. Matched vision-only controls are not given. But Table 1 still gives RoboTactile marks for contact phase, vision-only, and recovery, but this oversells what is actually seen in the experiments.

A few minor comments but would be good to polish the paper

Keywords are missing on the ICLR portal. Table 1 says “N/R = not reported”, but the table shows “-“. I am guessing “-“ means “N/R”. It would be good to be consistent throughout with the symbols. The AI use statement says “This statement will be updated to reflect the complete research workflow before submission.” Does this mean the current statement is incomplete and not relevant to the submission?

Questions for authors
  1. Could the authors clarify whether the world-model experiment was done and how? If this was not done, please remove it from the claims. Table 1 gives a tick mark for the world-model track but details are missing on this.
  2. Could the authors clarify if the N0-TWAM and N0-VTLA models were trained by the authors?
  3. How were the ACT and FTP-1 vision-only and clean models trained, and are they matched?
Reviewer_ifeoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a benchmark for evaluating the performance of different visual-tactile trained policies under different kinds of sensor flaws. It defines 14 fault operators across availability, signal fidelity, temporal integrity, and contextual consistency, and reports closed-loop results for four models on UniVTAC tasks.

Critical strengths
  1. This paper discusses a big problem for vision-based tactile sensing: policies that benefit from clean tactile inputs may become unreliable when those inputs are corrupted.

  2. The taxonomy covers useful failure modes beyond generic noise, including delayed observations, frozen streams, and sensor identity swaps.

  3. The task–operator breakdown reveals vulnerabilities that aggregate success rates can obscure, particularly cases where corrupted touch performs worse than interrupted touch.

Critical weaknesses

All those tactile flaws are simulated in a simulator with UniVTAC, but it might not simulate the real tactile challenges:

a. Different visual-tactile sensors have different lighting conditions, different gels' physical properties, and friction, which result in different rendering, shadows, and deformations. It is hard for the simulator to simulate realistic differences.

b. The sensor breaking mainly comes from gels being broken or cracked, which are not simulated.

c. The simulated marker and image will be influenced by shear force, with corresponding deformation (Taxim [1], HydroShear [2]). Using only depth is not enough.

Then, my major concern is that such a method cannot simulate realistic flaws of the sensor, and the simulator cannot simulate realistic interactions like real sensors. As a result, this benchmark might not be able to reflect the real performance.

I believe the authors should conduct at least some real-world experiments to validate whether the performance trends observed in simulation are consistent with those in reality. In particular, demonstrating that methods performing better in simulation also perform better under corresponding real-world sensor imperfections would significantly strengthen the validity of the proposed benchmark.

[1]. Z. Si et al., Taxim: An Example-based Simulation Model for GelSight Tactile Sensors, RA-L 2021

[2]. Dang et al., HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning, RSS 2026

Questions for authors

All questions are shown in weakness

Reviewer_Go3hOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

RoboTactile proposes 14 tactile observation failures across 4 dimensions on the 8 UniVTAC manipulation tasks in Isaac Sim. It specifies offline and closed-loop evaluation protocols and reports close-loop success for ACT, FTP-1, N0-TWAM and N0-VTLA. It reports performance gaps and descriptive tacitle gain retention relative to visual references. But it doesn't include offline prediction, phase-conditioned evaluation, post-restoration recovery and verified matched visual controls in its experiments.

Critical strengths
  1. The problem is important to the tactile-related research. Clean-input tactile benefits do not establish reliability when tactile observations become misleading. Studying these failures in closed-loop manipulation addresses an important deployment concern.
  2. The fault taxonomy offers a useful diagnostic starting point. It covers missing, degraded, stale, and misassigned observations beyond generic noise.
Critical weaknesses
  1. The reported experiments cannot be reconstructed from the submission. The main paper's Reproducibility Statement (p. 10) and Supplement Sections. 1.2, 1.6, and 1.8 explicitly identify missing run-specific fault parameters/schedules, model and environment configurations, trial counts, and seeds. No executable code or configuration download link is provided. Operator definitions and qualitative replays do not identify the settings producing the success matrices. This undermines the central reusable-benchmark contribution.
  2. The simulation-only evaluation leaves the benchmark's real-world relevance unvalidated. The paper provides no real-robot tests or empirical analysis of real sensor failures to validate the synthetic faults.
  3. The comparisons provide limited evidence of diagnostic and comparative utility. The N0 clean scores are external references, ACT/FTP-1 visual controls are not verified as matched, and task coverage differs across models. Thus, Table 2 does not establish controlled degradation or tactile benefit, or support ranking models by its overall averages. The broader offline, phase-conditioned, and recovery capabilities remain unvalidated.
Questions for authors

Please see above weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.