acceptodds
Under review as a conference paper at ICLR 2027

OpenViTac: Learning and Benchmarking Visuo-Tactile Policies in a Unified Sim-and-Real Framework

Abstract

Tactile feedback provides embodied agents with physical information beyond visual observations, enabling more reliable interaction with the real world. However, despite the rapid progress of vision-tactile-language-action (VTLA) policies, there remains a lack of unified benchmarks for evaluating tactile-enabled robot manipulation across simulation and the real world. To address this gap, we introduce OpenViTac, a visuo-tactile manipulation benchmark for evaluating robot policies across simulation and the real world. OpenViTac organizes contact-rich manipulation into four tactile-relevant capability dimensions and provides paired simulation–real-world settings for consistent evaluation of VLA, WAM, and VTLA policies. Building upon this benchmark, we investigate how different tactile representations and integration strategies affect the performance of pretrained VLA models. Correspondingly, we introduce OpenVTLA, a tactile augmentation framework that combines the best-performing representation and integration strategy. Furthermore, we leverage the paired benchmark setting to study sim–real co-training and analyze factors affecting cross-domain policy learning. Together, OpenViTac provides a unified platform for evaluating and advancing visuo-tactile robot manipulation.

Primary area:
datasets and benchmarks
Keywords:
Embodied AI, Visuo-Tactile, Simulation and Real

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bRwXOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces OpenViTac, a visuo-tactile manipulation benchmark with paired simulation–real-world settings for consistent evaluation of VLA, WAM, and VTLA policies. It also introduces OpenVTLA, a tactile augmentation framework that combines the best-performing tactile representation and integration strategy identified in the study. The benchmark provides a comprehensive evaluation of representative policies and tactile augmentation approaches, offering useful insights and a valuable foundation for future research in tactile manipulation.

Critical strengths

Strengths

  1. The paper provides a comprehensive overview of representative simulated manipulation benchmarks and clearly compares their capabilities with OpenViTac. The benchmark covers diverse capabilities, ranging from precision manipulation to physical-property perception, addressing gaps in existing benchmarks. The inclusion of recent baselines also makes the evaluation timely and relevant.

  2. The experimental evaluation is extensive, covering 11 simulation tasks and eight paired real-world tasks for systematic comparison of VLA, WAM, and VTLA policies. The paper also investigates how tactile information can be incorporated into the π0.5 VLA model.

  3. For real-to-sim alignment, the paper introduces an agent-assisted pipeline to reconstruct simulation assets and task settings from real-world references. It also supports scene diversification through variations in backgrounds and surrounding context.

  4. The paper investigates two interesting questions: which tactile representations are effective, and how tactile information should be integrated into existing VLA models such as π0.5. The additional sim–real co-training study further broadens the practical value of the work.

Overall, the paper presents a comprehensive study of recent tactile representations and integration strategies, evaluates representative policy families, and explores the benefits of sim–real co-training.

Critical weaknesses
  1. Qualitative analysis of task performance. More qualitative analysis would help explain how tactile feedback contributes to different capability categories, particularly property perception and fragility-aware manipulation. For example, which failure modes are resolved by tactile feedback, and which remain challenging? OpenVTLA improves the simulation average over the strongest baseline by approximately seven percentage points, but the practical significance of this advantage could be explained more clearly. Reporting task completion time and the number of execution steps for each method would also help assess whether tactile feedback facilitates more efficient decision-making.

  2. Number of real-world evaluation trials. Most real-world results are based on only 20 trials per task, which may be insufficient to reliably distinguish methods with similar performance. OpenVTLA achieves an average success rate of 54.6%, compared with FTP-1’s 51.9%, so the advantage is relatively small. Increasing the number of trials—for example, to 50 per task—and reporting confidence intervals would help determine whether this difference reflects a consistent benefit or occasional successes and failures. Qualitative examples could further clarify whether richer tactile information improves decision-making.

Questions for authors
  1. Task abbreviations in Table 2. It would improve readability to include the full names of abbreviations such as WC and HC in the caption or a table note, allowing readers to interpret the results without referring to earlier sections.

  2. Explanation of sim–real consistency metrics. Briefly explaining Pearson correlation and MMRV, including how their values should be interpreted, would make the sim–real consistency analysis easier to follow.

  3. Additional current-frame tactile baseline. In the tactile representation study, it would be helpful to include a baseline explicitly designed and trained to use only the current tactile frame, in addition to repeating the current frame across the history window. Repeated identical frames may interact differently with a history-based encoder, so this comparison would help distinguish the benefit of temporal information from the effects of the input format or model capacity.

  4. Evaluation details for Table 4. Please explicitly state the number of evaluation trials used for each policy in Table 4. Given that OpenVTLA improves over the strongest reported baseline by approximately seven percentage points, reporting uncertainty would also help establish the reliability of the comparison.

  5. Tactile sensor configurations. Please clarify which tactile sensor and gripper configurations were used in each experiment, including the main comparisons, ablations, and co-training study. Although the appendix describes support for multiple sensors, it is unclear whether the reported results use GelSight exclusively or also include Xense and NeoteAI.

Reviewer_P89POpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces OpenViTac, a visuo-tactile manipulation benchmark designed to evaluate robot policies in both simulation and the real world. The benchmark organizes manipulation tasks into four tactile-relevant capability dimensions: physical-property perception, fragility-aware manipulation, contact-rich manipulation, and precision manipulation. It provides corresponding simulation and real-world task settings and evaluates representative VLA, WAM, and VTLA policies under a common protocol.

Beyond benchmarking, the paper studies how tactile observations can be incorporated into pretrained VLA models. It compares several tactile representations and integration strategies, and introduces OpenVTLA by combining a temporally aware tactile representation with token-level fusion into the VLM input sequence. The paper further investigates sim-real co-training, showing that additional simulated demonstrations can improve real-world performance under a fixed real-data budget. Overall, the work aims to provide a unified platform for analyzing tactile representation, tactile integration, and simulation-to-real transfer in visuo-tactile manipulation.

Critical strengths
  1. The paper clearly explains the importance of tactile feedback for contact-sensitive manipulation and presents the overall framework in an accessible manner.
  2. Well-structured tactile benchmark with broad policy evaluation. OpenViTac organizes tasks around tactile-relevant capabilities and evaluates a relatively broad set of VLA, WAM, and VTLA policies in both simulation and the real world.
  3. The comparison of tactile representations and integration strategies provides useful insight into how tactile observations can be incorporated into pretrained VLA models.
Critical weaknesses
  1. The paper does not provide enough details about the construction of simulation assets, tactile rendering, sensor and camera calibration, physical parameters, scene randomization, or task success criteria. It also lacks quantitative measurements of the gap between simulated and real visual-tactile observations.
  2. The novelty relative to existing platforms is not sufficiently clear. OpenViTac relies on simulation components such as Isaac and TacEx that are also used in related work, including UniVTAC. The paper should more clearly identify which parts are genuinely new. For example, the paired sim-real benchmark, task design, policy evaluation protocol, or tactile-integration study, and provide direct comparisons with existing benchmarks.
  3. The paper should clarify whether it will release the Isaac/TacEx environment, simulation assets, task configurations, data-generation scripts, evaluation code, and trained checkpoints.
Questions for authors
  1. Can the authors provide quantitative measurements of the sim-real gap for visual and tactile observations, rather than only comparing policy success rates or rankings?
  2. What exact steps are automated by the agent-assisted real-to-sim pipeline, and which steps require manual intervention? How are the generated assets and task configurations validated?
  3. Which components of OpenViTac are currently open-sourced or planned for release, including simulation assets, task configurations, data-generation scripts, evaluation code, and trained checkpoints?
  4. How does OpenViTac differ from existing Isaac/TacEx-based benchmarks under a more detailed comparison?
Reviewer_LNLYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces OpenViTac, a unified visuo-tactile manipulation benchmark with 11 simulation tasks and 8 corresponding real-world tasks. It evaluates representative VLA, WAM, and VTLA policies, investigates tactile representation and integration strategies, and explores sim–real co-training.

Critical strengths
  • Comprehensive benchmark: OpenViTac covers diverse tactile-relevant manipulation capabilities, with corresponding simulation and real-world environments, multiple tactile sensors, and gripper configurations.

  • Broad empirical evaluation: The evaluation of various VLA, WAM, and VTLA models provides useful comparisons and insights into their capabilities on contact-sensitive manipulation tasks.

  • Practical findings: The controlled ablations on tactile pretraining and temporal context provide useful evidence for adapting pretrained VLAs to tactile observations. The real-world experiments further strengthen the practical relevance of the benchmark.

Critical weaknesses

W1. Limited distinctiveness and methodological contributions. While the benchmark itself is a useful contribution, its distinctive advantages over existing visuo-tactile benchmarks appear relatively limited. ManiFeel (Luu et al., 2025) already investigates tactile policy learning and representations with real-world validation, while UniVTAC (Chen et al., 2026) provides a multi-sensor tactile simulation and evaluation platform. Furthermore, OpenVTLA mainly combines an existing tactile encoder (Feng et al., 2026) with token-level concatenation, while the sim–real co-training study follows an established paradigm (Maddukuri et al., 2025). Thus, the independent methodological contributions are relatively limited.

W2. Incompletely controlled integration comparisons. Table 4 compares five tactile integration strategies, but not all comparisons use the same tactile representation and policy architecture. While concatenation, action conditioning, and refinement are compared under a shared AnyTouch2-based setup, modulation uses Sparsh, and the expert-based approach uses FTP-1 with T3. These differences confound the broader comparison across integration strategies. Additional controlled ablations would make the conclusions more convincing.

W3. Task-level sim–real consistency. The reported Pearson correlation of 0.913 is computed from task-averaged scores across models. However, the consistency varies substantially across individual tasks. For example, the task-level Pearson correlation for Gear Assembly is only approximately 0.06 based on Tables 2–3. A more detailed task-level analysis is needed to substantiate the benchmark's reliability as a simulation proxy.

W4. Limited statistical evidence for real-world improvements. OpenVTLA achieves 54.6% average success in the real world, only 2.7 percentage points above FTP-1 (51.9%). Given the relatively small number of evaluation trials (20–30 per task), the absence of confidence intervals or multi-run statistics makes it difficult to assess the robustness of this improvement.

[1] Luu, Q. K., et al. (2025). ManiFeel: Benchmarking and Understanding Visuotactile Manipulation Policy Learning. arXiv:2505.18472.

[2] Chen, B., et al. (2026). UniVTAC: A Unified Simulation Platform for Visuo-Tactile Manipulation Data Generation, Learning, and Benchmarking. arXiv:2602.10093.

[3] Feng, R., et al. (2026). AnyTouch 2: General Optical Tactile Representation Learning for Dynamic Tactile Perception. arXiv:2602.09617.

[4] Maddukuri, A., et al. (2025). Sim-and-Real Co-Training: A Simple Recipe for Vision-Based Robotic Manipulation. Robotics: Science and Systems (RSS).

Questions for authors

see weakness

If these concerns are well addressed, I am happy to raise the score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair for their time and guidance, and Reviewers LNLY, P89P, and bRwX for their constructive feedback. Notably, Reviewers P89P and bRwX have confirmed that our responses and revisions addressed their questions and maintained their positive assessments (Weak Acceptance and Clear Acceptance, respectively).

While we have not yet received a follow-up response from Reviewer LNLY before the discussion closes, we believe our newly added controlled experiments and quantitative analyses thoroughly resolve all four points raised in their review (W1–W4). Below is a concise summary of the acknowledged strengths and major revisions made in our manuscript:


1. Reviewer Recognition & Core Strengths

  • Comprehensive & Well-Structured Benchmark (LNLY, P89P, bRwX): Acknowledged as a comprehensive visuo-tactile manipulation benchmark covering diverse capabilities, paired simulation–real-world environments, and a broad evaluation of VLA, WAM, and VTLA policy families.
  • Valuable Insights on Tactile Adaptation (LNLY, P89P, bRwX): Praised for providing useful insights and systematic ablations on tactile pretraining, temporal context, and integration strategies for pretrained VLAs.
  • Practical Relevance & Sim-to-Real Value (LNLY, bRwX): Recognized for strong practical relevance through real-world experiments, agent-assisted sim-to-real pipelines, and demonstrated benefits of sim-real co-training.

2. Summary of Major Revisions & Responses

(1) Benchmark Distinctiveness & Contributions (Addressing LNLY [W1], P89P [Q4])

  • Unique Paired Sim–Real Alignment & Multi-Sensor Support: Clarified that unlike existing platforms (e.g., UniVTAC and ManiFeel), OpenViTac establishes systematic sim–real alignment across four capability categories, supports three distinct tactile sensors (GelSight Mini, Xense, and NeoteAI), and introduces modular task and policy interfaces (Appendix B.3).
  • Systematic Tactile Adaptation Study: Highlighted our controlled joint investigation of four tactile representations and five integration strategies on a shared pretrained VLA backbone, alongside applying sim–real co-training to visuo-tactile policies.

(2) Fully Controlled Tactile Integration Comparisons (Addressing LNLY [W2])

  • Complete AnyTouch2-Based Setup: Directly resolved the confounding factor concern by adding AnyTouch2-based modulation and tactile-expert variants. All five integration strategies (Token-level Concatenation, Feature-level Modulation, Action-level Conditioning, Tactile-driven Expert, and Tactile-guided Refinement) are now evaluated under the exact same AnyTouch2 representation and pretrained π0.5\pi_{0.5} backbone (Table 4, Appendix Table S7), confirming the clear advantage of token-level concatenation (68.7% vs. 55.1% for the next best).

(3) Fine-Grained Sim–Real Consistency & Gap Analysis (Addressing LNLY [W3], P89P [Q1])

  • Task-Level Consistency Metrics: Added fine-grained task-level Pearson correlations and MMRV results across all eight tasks for the nine shared methods (Appendix E.1), explicitly analyzing lower agreement on complex contact-rich tasks such as Gear Assembly.
  • Qualitative Representation Gaps: Added qualitative t-SNE visualizations of visual and tactile representations (Appendix E.2) to examine remaining observation-level sim–real gaps.

(4) Multi-Round Real-World Evaluations & Qualitative Insights (Addressing LNLY [W4], bRwX [W1, W2])

  • Three-Round Real-World Validation: Conducted two additional full evaluation rounds across all eight tasks (Appendix D.1). OpenVTLA consistently achieves the highest overall success rate across all rounds (54.0 ± 1.3% vs. 51.0 ± 0.9% for FTP-1 and 38.0 ± 2.7% for π0.5\pi_{0.5}), supported by 95% bootstrap confidence intervals and paired t-tests (p=0.0215p = 0.0215).
  • Qualitative Case Studies: Added failure-mode and rollout analyses (Appendix D) illustrating how tactile feedback resolves visually ambiguous physical properties and stabilizes delicate grasps.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.