acceptodds
Withdrawn from ICLR 2027

TacMind: Benchmarking and Modeling Tactile Interaction States for Robotic Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Tactile sensing provides robots with direct observations of contact physics, local geometry, and interaction dynamics, making it a vital source of feedback for contact-rich manipulation. However, existing tactile manipulation studies are often developed and evaluated within specific task settings, making it difficult to systematically compare how tactile information contributes under different interaction conditions. In this work, we develop a benchmark in simulation comprising 128 tasks organized around three complementary evaluation regimes: Precise, Recognition, and Dynamic. The benchmark further includes controlled static and dynamic task variants and 12 perturbation settings for evaluating robustness under changes in visual and physical conditions. These settings motivate studying tactile information as a temporally evolving interaction representation rather than only as an instantaneous observation. We therefore propose an action-conditioned latent predictive world model that integrates tactile history and learns a tactile state through prediction of future tactile latents. Building on this representation, we design a model that conditions action generation on the learned tactile state and uses predicted future tactile information to refine candidate actions through predictive tactile reflection. Experiments on the proposed benchmark show consistent gains from incorporating tactile information across the evaluated regimes.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2XM1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work studies the contribution of tactile information across different manipulation tasks in simulation environments. Specifically, it first introduces an Isaac Sim-based tactile manipulation benchmark, which organizes 128 tactile manipulation tasks into three categories: Precise (fine-grained contact manipulation), Recognition (recognition under visual ambiguity), and Dynamic (manipulation involving moving targets or source objects). Furthermore, this work proposes an action-conditioned latent space predictive model, Tac-LeWM, and a vision-tactile-language-action (VTLA) model, TacMind-0.

Critical strengths
  1. The organization dimension of the benchmark is novel, and the considered task scope goes beyond existing simulated tactile benchmarks.
  2. The model incorporates modality-specific designs tailored for tactile sensing.
Critical weaknesses

The key weaknesses of this work lie in (1) serious fairness issues that undermine the reliability of the evaluation, and (2) insufficient validation and analysis of the proposed benchmark and methodology.

Specifically:

  1. At the beginning of the paper, the authors state that the benchmark consists of 28 standard tasks and 100 additional tasks, forming a total of 128 tasks. However, in Section 3, the authors explicitly state that the Extra suite only “extends these tasks with greater diversity in textures and language descriptions.” Therefore, these extensions cannot be considered as new tasks. Claiming them as additional tasks is misleading, as the increased diversity in textures and language descriptions does not necessarily constitute the introduction of new task definitions or manipulation capabilities. This is a serious issue, as it raises concerns about the credibility of the paper’s stated contributions.
  2. Another serious potential data leakage issue arises from the use of TacMind-Bench. In Section 5.1, the authors explicitly state that “To ensure a fair comparison, TacMind-Bench is excluded from both the pretraining and fine-tuning stages of all models.” However, in Appendix B, the authors twice mention that “we construct a heterogeneous visuo-tactile corpus from TacMind-Bench, ......” (Line 862), and “the final training mixture emphasizes TacMind-Bench together with RDP, VLA-Touch, and VisuoTactile-QINGLOONG. TacMind-Bench receives the largest sampling mass because it defines the downstream task distribution ......” (Line 1009–1012). This creates a serious inconsistency between the experimental protocol described in the main paper and the training data composition reported in the appendix. If TacMind-Bench data (in particular, it currently appears that all 28 standard tasks are included in the pre-training data) are indeed included during pretraining or fine-tuning, the reported benchmark results may suffer from data leakage, making the evaluation unreliable. The authors should clarify whether any TacMind-Bench data were used during model pre-training, and provide a detailed breakdown of the training data sources to ensure a fair comparison.
  3. The checkpoint selection strategy is inappropriate. In Line 1133, the authors state that “For each method, we evaluate checkpoints at 20,000, 30,000, and 40,000 training steps and report the best result among these checkpoints.” However, the checkpoint should be selected fully based on the validation set and fixed before evaluation. Selecting the best checkpoint directly on the test set is not an appropriate setting. You should determine a single checkpoint using validation performance and then reporting its performance on the test set.
  4. The paper claims that the proposed method supports “long interaction histories,” but this claim is not sufficiently validated by experiments. Figure 4 only reports the inference time under larger inference batch sizes and the reconstruction performance with a maximum cache length of (k=8). However, neither of these experiments provides convincing evidence for long-horizon tactile history modeling. In real-world robotic deployment, the key challenge is maintaining longer temporal memory length rather than increasing the inference batch size, since online robot inference typically operates with batchsize = 1. Moreover, a cache length of (k=8) only corresponds to a very short temporal window and cannot demonstrate the ability to capture long interaction histories. In addition, the reconstruction performance cannot replace the wanted manipulation performance. This further raises serious concerns about the validity of the paper claim and its actual contribution.
  5. In the current experiments, the performance on the Dynamic task setting is generally below 10%, and some settings in the Precise category show similar issues. Such extremely low performance suggests that the difficulty design of the benchmark tasks may be problematic. When the tasks are excessively challenging and most models fail to achieve meaningful performance, comparisons between different methods become less informative and may not accurately reflect the actual capability differences among models. Additionally, ACT+Tactile achieves substantially worse performance than DP on the Dynamic setting. This result weakens the paper’s claim regarding the importance of tactile information for dynamic manipulation. A more comprehensive and fair comparison should include ACT, ACT+Tactile, DP, and DP+Tactile.
  6. The ablation study on tactile reflection is not sufficient. Since the proposed “reflection” mechanism performs two forward passes during both training and inference, the authors should consider the following additional ablations to isolate the actual contribution of predictive tactile reflection: (1) Performing two consecutive forward passes with vision observation. (2) Replacing the future tactile latent in Tactile Reflection with zero vectors or random noise. (3) Replacing the predicted future tactile latent with the current tactile observation. These ablations are necessary to validate that the performance improvements come from the proposed predictive tactile reflection mechanism, rather than from additional inference steps or extra latent inputs.
  7. This method is closely related to numerous tactile world models and WAM-based approaches. As a work that positions the benchmark as one of its key contributions, it requires more comprehensive comparisons with existing approaches to thoroughly evaluate the effectiveness and significance of the proposed method.

Overall, I appreciate the authors’ efforts in constructing the benchmark and making it available to the community. However, the experimental evaluation and fairness considerations require substantial revisions before the work can be considered reliable.

Questions for authors

See weaknesses.

Reviewer_zLtLOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TacMind-Bench, a simulation benchmark with 128 tactile manipulation tasks covering precise manipulation, object recognition, and dynamic interaction. The authors also designed a data collection pipeline for their tasks that supports their model.

On the fusion side, they propose Tac-LeWM, which learns tactile representations by predicting future tactile features from previous observations and actions. The model uses marker deformation information, a tactile-specific encoding, and a streaming encoder to maintain interaction history. They then introduce TacMind-0, a VTLA policy that uses these representations for action generation. An additional tactile reflection step predicts the outcome of proposed actions and uses it to refine them.

The authors evaluate their approach against several baselines in simulation and report improvements mainly under perturbations. They also test it on six real-world manipulation tasks, including insertion, gear assembly, and weight discrimination.

Critical strengths
  • The paper represents a substantial amount of engineering work, both in building the simulation benchmark and developing the tactile world model and policy. Supporting different manipulation tasks, tactile sensors, and perturbations, along with the data collection pipeline, is a considerable effort.

  • Modeling tactile history through a streaming encoder is well motivated and pretty realistic, especially since a single tactile frame may not capture how contact evolves.

  • It's good to see real-world experiments beyond simulation. The results on gear assembly and weight discrimination are encouraging.

  • I like that the benchmark covers different aspects of tactile manipulation, especially recognition under visual ambiguity and dynamic interactions, which are less commonly evaluated together.

Critical weaknesses
  • The results don't always support the claims. For example, ACT+Tactile performs better than TacMind-0 on in-domain tasks (34.4% vs. 31.4% with GelSight Mini). Although the authors mention that single-task and multi-task policies are not directly comparable, a fairer comparison would be useful.

  • I understand tasks with moving targets are hard but the Dynamic tasks are almost unsolved. TacMind-0 only achieves 4.1% success with GelSight Mini, which makes it hard to tell whether the proposed method actually helps with changing contact conditions.

  • I'm not convinced that the predictive tactile representation is necessary. Why not use a simpler temporal tactile encoder? It would be useful to compare against or discuss other tactile representation methods such as T3 [1], AnyTouch 2 [2], and SkillFormer [3]. There are also closely related tactile world models, such as VT-WM [4] and TouchWorld [5], that are discussed but not experimentally compared.

  • Tactile reflection seems computationally expensive. It requires reversing and rerunning the flow model, along with predicting future tactile features. I wonder how much this slows down the policy. The authors should report the actual inference frequency with and without reflection.

  • The real-world results are interesting, but they don't really show sim-to-real transfer. The authors collect 50 real-world demonstrations per task, so it's unclear how much the simulated tactile data helps. An experiment with and without simulation pretraining would make this clearer.

References [1] Zhao et al., Transferable Tactile Transformers for Representation Learning Across Diverse Sensors and Tasks. 2024. [2] Feng et al., AnyTouch 2: General Optical Tactile Representation Learning for Dynamic Tactile Perception. 2026. [3] Shahidzadeh et al., What to Attend, What to Keep: Skill-Conditioned Visuotactile Representation with Progress-Guided Event Memory. [4] Higuera et al., Visuo-Tactile World Models. 2026. [5] Zhou et al., TouchWorld: A Predictive and Reactive Tactile Foundation Model for Dexterous Manipulation. 2026.

Questions for authors

Please address the weaknesses.

Reviewer_yQqwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a systematic benchmark to study tactile sensing on the three types of tasks: precise manipulation, recognition, and dynamic interaction. Additionally, authors introduce TacMind-0 as a vision-language-tactile-action model. On top of TacMind-0, authors introduce Tac-LeWM as an additional world model to select better actions from the policy.

Critical strengths
  • Dynamic tasks are not very well considered in previous tactile benchmarks and tasks. This paper addresses this blind spot well.
  • The paper is well written and easy to follow.
Critical weaknesses
  • To build an impactful benchmark in tactile community, it might be important to include different types of the tactile sensors as the community has not converged to a single hardware choice. However, currently only a GelSight-like sensor is considered, which could limit the applicability of this benchmark.
  • While one of the contribution of this paper is proposing a different vision-tactile-language-model, its performance is somewhat mixed. In Recognition type of task, its performance is worse than simple ACT + Tactile. This might also indicate that this benchmark is not very well built.
  • The task suite needs more detailed evidence. One of the major advancement claimed by this paper is that this benchmark consists of a lot more tasks (128 tasks) than prior works. Among these 128 tasks, 100 tasks belong to Extra suite. I tried to find a more detailed information regarding these 100 tasks and how different they are from these 28 tasks, but I cannot find such information. Additionally, such task diversity is very important for a benchmark paper. For example, if another task is simply changing the texture or setup a bit, the benchmark might not give us enough information regarding the policy quality.
  • While TacMind-Bench are TacMind-0 two major contributions in this paper, their novelty, difference from the prior works, and effectiveness are not very clear.
  • This paper lacks qualitative analysis and visualization for Tac-LeWM.
Questions for authors
  • OpenPI0.5 performance is very bad on precision tasks, which is a bit unexpected. Does the OpenPI0.5 take in the tactile images as well? Is OpenPI0.5 traied properly?
  • In L409, authors mention "For ACT-Tactile and Diffusion Policy, we train a separate model for each task, whereas the VLA and VTLA models are jointly trained across all tasks." However, In L416, authors mention "we do not recommend directly comparing the performance of single-task policies with that of multi-task policies, as they are trained under substantially different learning settings." This seems to be conflicting with each other. Why is that?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.