- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces OpenViTac, a visuo-tactile manipulation benchmark with paired simulation–real-world settings for consistent evaluation of VLA, WAM, and VTLA policies. It also introduces OpenVTLA, a tactile augmentation framework that combines the best-performing tactile representation and integration strategy identified in the study. The benchmark provides a comprehensive evaluation of representative policies and tactile augmentation approaches, offering useful insights and a valuable foundation for future research in tactile manipulation.
Critical strengths
Strengths
-
The paper provides a comprehensive overview of representative simulated manipulation benchmarks and clearly compares their capabilities with OpenViTac. The benchmark covers diverse capabilities, ranging from precision manipulation to physical-property perception, addressing gaps in existing benchmarks. The inclusion of recent baselines also makes the evaluation timely and relevant.
-
The experimental evaluation is extensive, covering 11 simulation tasks and eight paired real-world tasks for systematic comparison of VLA, WAM, and VTLA policies. The paper also investigates how tactile information can be incorporated into the π0.5 VLA model.
-
For real-to-sim alignment, the paper introduces an agent-assisted pipeline to reconstruct simulation assets and task settings from real-world references. It also supports scene diversification through variations in backgrounds and surrounding context.
-
The paper investigates two interesting questions: which tactile representations are effective, and how tactile information should be integrated into existing VLA models such as π0.5. The additional sim–real co-training study further broadens the practical value of the work.
Overall, the paper presents a comprehensive study of recent tactile representations and integration strategies, evaluates representative policy families, and explores the benefits of sim–real co-training.
Critical weaknesses
-
Qualitative analysis of task performance. More qualitative analysis would help explain how tactile feedback contributes to different capability categories, particularly property perception and fragility-aware manipulation. For example, which failure modes are resolved by tactile feedback, and which remain challenging? OpenVTLA improves the simulation average over the strongest baseline by approximately seven percentage points, but the practical significance of this advantage could be explained more clearly. Reporting task completion time and the number of execution steps for each method would also help assess whether tactile feedback facilitates more efficient decision-making.
-
Number of real-world evaluation trials. Most real-world results are based on only 20 trials per task, which may be insufficient to reliably distinguish methods with similar performance. OpenVTLA achieves an average success rate of 54.6%, compared with FTP-1’s 51.9%, so the advantage is relatively small. Increasing the number of trials—for example, to 50 per task—and reporting confidence intervals would help determine whether this difference reflects a consistent benefit or occasional successes and failures. Qualitative examples could further clarify whether richer tactile information improves decision-making.
Questions for authors
-
Task abbreviations in Table 2. It would improve readability to include the full names of abbreviations such as WC and HC in the caption or a table note, allowing readers to interpret the results without referring to earlier sections.
-
Explanation of sim–real consistency metrics. Briefly explaining Pearson correlation and MMRV, including how their values should be interpreted, would make the sim–real consistency analysis easier to follow.
-
Additional current-frame tactile baseline. In the tactile representation study, it would be helpful to include a baseline explicitly designed and trained to use only the current tactile frame, in addition to repeating the current frame across the history window. Repeated identical frames may interact differently with a history-based encoder, so this comparison would help distinguish the benefit of temporal information from the effects of the input format or model capacity.
-
Evaluation details for Table 4. Please explicitly state the number of evaluation trials used for each policy in Table 4. Given that OpenVTLA improves over the strongest reported baseline by approximately seven percentage points, reporting uncertainty would also help establish the reliability of the comparison.
-
Tactile sensor configurations. Please clarify which tactile sensor and gripper configurations were used in each experiment, including the main comparisons, ablations, and co-training study. Although the appendix describes support for multiple sensors, it is unclear whether the reported results use GelSight exclusively or also include Xense and NeoteAI.
Rebuttal Summary
Dear Area Chair and Reviewers,
We sincerely thank the Area Chair for their time and guidance, and Reviewers LNLY, P89P, and bRwX for their constructive feedback. Notably, Reviewers P89P and bRwX have confirmed that our responses and revisions addressed their questions and maintained their positive assessments (Weak Acceptance and Clear Acceptance, respectively).
While we have not yet received a follow-up response from Reviewer LNLY before the discussion closes, we believe our newly added controlled experiments and quantitative analyses thoroughly resolve all four points raised in their review (W1–W4). Below is a concise summary of the acknowledged strengths and major revisions made in our manuscript:
(1) Benchmark Distinctiveness & Contributions (Addressing LNLY [W1], P89P [Q4])
Appendix B.3).(2) Fully Controlled Tactile Integration Comparisons (Addressing LNLY [W2])
Table 4,Appendix Table S7), confirming the clear advantage of token-level concatenation (68.7% vs. 55.1% for the next best).(3) Fine-Grained Sim–Real Consistency & Gap Analysis (Addressing LNLY [W3], P89P [Q1])
Appendix E.1), explicitly analyzing lower agreement on complex contact-rich tasks such as Gear Assembly.Appendix E.2) to examine remaining observation-level sim–real gaps.(4) Multi-Round Real-World Evaluations & Qualitative Insights (Addressing LNLY [W4], bRwX [W1, W2])
Appendix D.1). OpenVTLA consistently achieves the highest overall success rate across all rounds (54.0 ± 1.3% vs. 51.0 ± 0.9% for FTP-1 and 38.0 ± 2.7% for π0.5), supported by 95% bootstrap confidence intervals and paired t-tests (p=0.0215).Appendix D) illustrating how tactile feedback resolves visually ambiguous physical properties and stabilizes delicate grasps.