- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work studies the contribution of tactile information across different manipulation tasks in simulation environments. Specifically, it first introduces an Isaac Sim-based tactile manipulation benchmark, which organizes 128 tactile manipulation tasks into three categories: Precise (fine-grained contact manipulation), Recognition (recognition under visual ambiguity), and Dynamic (manipulation involving moving targets or source objects). Furthermore, this work proposes an action-conditioned latent space predictive model, Tac-LeWM, and a vision-tactile-language-action (VTLA) model, TacMind-0.
Critical strengths
- The organization dimension of the benchmark is novel, and the considered task scope goes beyond existing simulated tactile benchmarks.
- The model incorporates modality-specific designs tailored for tactile sensing.
Critical weaknesses
The key weaknesses of this work lie in (1) serious fairness issues that undermine the reliability of the evaluation, and (2) insufficient validation and analysis of the proposed benchmark and methodology.
Specifically:
- At the beginning of the paper, the authors state that the benchmark consists of 28 standard tasks and 100 additional tasks, forming a total of 128 tasks. However, in Section 3, the authors explicitly state that the Extra suite only “extends these tasks with greater diversity in textures and language descriptions.” Therefore, these extensions cannot be considered as new tasks. Claiming them as additional tasks is misleading, as the increased diversity in textures and language descriptions does not necessarily constitute the introduction of new task definitions or manipulation capabilities. This is a serious issue, as it raises concerns about the credibility of the paper’s stated contributions.
- Another serious potential data leakage issue arises from the use of TacMind-Bench. In Section 5.1, the authors explicitly state that “To ensure a fair comparison, TacMind-Bench is excluded from both the pretraining and fine-tuning stages of all models.” However, in Appendix B, the authors twice mention that “we construct a heterogeneous visuo-tactile corpus from TacMind-Bench, ......” (Line 862), and “the final training mixture emphasizes TacMind-Bench together with RDP, VLA-Touch, and VisuoTactile-QINGLOONG. TacMind-Bench receives the largest sampling mass because it defines the downstream task distribution ......” (Line 1009–1012). This creates a serious inconsistency between the experimental protocol described in the main paper and the training data composition reported in the appendix. If TacMind-Bench data (in particular, it currently appears that all 28 standard tasks are included in the pre-training data) are indeed included during pretraining or fine-tuning, the reported benchmark results may suffer from data leakage, making the evaluation unreliable. The authors should clarify whether any TacMind-Bench data were used during model pre-training, and provide a detailed breakdown of the training data sources to ensure a fair comparison.
- The checkpoint selection strategy is inappropriate. In Line 1133, the authors state that “For each method, we evaluate checkpoints at 20,000, 30,000, and 40,000 training steps and report the best result among these checkpoints.” However, the checkpoint should be selected fully based on the validation set and fixed before evaluation. Selecting the best checkpoint directly on the test set is not an appropriate setting. You should determine a single checkpoint using validation performance and then reporting its performance on the test set.
- The paper claims that the proposed method supports “long interaction histories,” but this claim is not sufficiently validated by experiments. Figure 4 only reports the inference time under larger inference batch sizes and the reconstruction performance with a maximum cache length of (k=8). However, neither of these experiments provides convincing evidence for long-horizon tactile history modeling. In real-world robotic deployment, the key challenge is maintaining longer temporal memory length rather than increasing the inference batch size, since online robot inference typically operates with batchsize = 1. Moreover, a cache length of (k=8) only corresponds to a very short temporal window and cannot demonstrate the ability to capture long interaction histories. In addition, the reconstruction performance cannot replace the wanted manipulation performance. This further raises serious concerns about the validity of the paper claim and its actual contribution.
- In the current experiments, the performance on the Dynamic task setting is generally below 10%, and some settings in the Precise category show similar issues. Such extremely low performance suggests that the difficulty design of the benchmark tasks may be problematic. When the tasks are excessively challenging and most models fail to achieve meaningful performance, comparisons between different methods become less informative and may not accurately reflect the actual capability differences among models. Additionally, ACT+Tactile achieves substantially worse performance than DP on the Dynamic setting. This result weakens the paper’s claim regarding the importance of tactile information for dynamic manipulation. A more comprehensive and fair comparison should include ACT, ACT+Tactile, DP, and DP+Tactile.
- The ablation study on tactile reflection is not sufficient. Since the proposed “reflection” mechanism performs two forward passes during both training and inference, the authors should consider the following additional ablations to isolate the actual contribution of predictive tactile reflection: (1) Performing two consecutive forward passes with vision observation. (2) Replacing the future tactile latent in Tactile Reflection with zero vectors or random noise. (3) Replacing the predicted future tactile latent with the current tactile observation. These ablations are necessary to validate that the performance improvements come from the proposed predictive tactile reflection mechanism, rather than from additional inference steps or extra latent inputs.
- This method is closely related to numerous tactile world models and WAM-based approaches. As a work that positions the benchmark as one of its key contributions, it requires more comprehensive comparisons with existing approaches to thoroughly evaluate the effectiveness and significance of the proposed method.
Overall, I appreciate the authors’ efforts in constructing the benchmark and making it available to the community. However, the experimental evaluation and fairness considerations require substantial revisions before the work can be considered reliable.
Questions for authors
See weaknesses.