acceptodds
Under review as a conference paper at ICLR 2027

TouchAgent: Benchmarking and Advancing Real-World Tactile Intelligence

Abstract

Recent advances in multimodal large language models (MLLMs) have expanded multimodal intelligence. However, progress in the tactile domain remains constrained by several distinctive challenges, including heterogeneous sensor modalities, diverse domain knowledge, and a fragmented task landscape. To address this gap, we introduce TouchBench, a comprehensive benchmark for real-world tactile interaction with **10K** questions and approximately **383K** observation frames, covering **6** real-world tactile task categories ranging from perception to reasoning and decision-making, thereby enabling broader and more systematic evaluation than prior benchmarks. Using TouchBench, we evaluate 14 open-source, proprietary, and tactile-language models, revealing systematic deficiencies in perceptual grounding, domain knowledge, and reasoning, all of which are essential for real-world tactile intelligence. Beyond evaluation, we propose TouchAgent, an agent framework for tactile intelligence that strategically integrates knowledge retrieval, perception, and reasoning through domain-specific tactile models and tools. To further improve tool planning and evidence integration, we construct TouchAgent-Instruct with **40K** multi-round interaction trajectories and optimize TouchAgent through supervised fine-tuning and reinforcement learning. Extensive experiments show that TouchAgent consistently outperforms existing MLLMs and tactile-specific models across tactile benchmarks, highlighting the effectiveness of tool-augmented agents in advancing real-world tactile intelligence. Our code and benchmark will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Tactile Intelligence, Multimodal Large Language Models, Benchmark, Tool-Augmented Agents, Multi-Step Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_GRDSOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TouchBench, a benchmark containing 10K questions across six tactile task categories, and TouchAgent, a framework combining tactile perception tools, knowledge retrieval, and numerical reasoning. The authors additionally construct 40K interaction trajectories and train the agent through supervised fine-tuning and evidence-grounded reinforcement learning. The results show substantial improvements over the evaluated general-purpose and tactile-language models.

Critical strengths
  1. TouchBench represents a substantial curation effort, integrating observations from multiple datasets and sensor types with expert annotation and verification.
  2. The integration of tactile perception, external knowledge, and computation is sensible. The training-stage and tool-component ablations provide useful evidence for the proposed system.
Critical weaknesses
  1. A particularly relevant omission is Octopi-1.5, which supports tactile observations from multiple object parts and incorporates retrieval-augmented generation. The paper only compares against the original Octopi, leaving its advantage over an existing retrieval-enhanced tactile-language system unclear. Although Figure 2 lists TactileBench, the associated SToLa framework, which addresses tactile interaction perception and open-ended commonsense reasoning, is not substantively discussed. The distinction between these reasoning capabilities and those evaluated by TouchBench deserves a clearer explanation. Furthermore, Touch-R1 is cited but not evaluated, despite its closely related emphasis on tactile-grounded reinforcement learning. These omissions weaken the assessment of the paper’s contribution relative to existing tactile reasoning approaches. Empirical comparisons should be included where implementations and input compatibility permit, with explicit explanations for exclusions.
  2. Several evaluated models, including InternVL2.5-8B, LLaVA-OneVision-Qwen2-7B, and MiniCPM-V-2.6, represent older generations, while the open-source and tactile-language comparisons are concentrated around 7–8B parameters. Although Qwen3-VL-8B and several strong proprietary models are included, the paper does not examine scaling within an open model family. Larger alternatives, such as Qwen3-VL-32B or Qwen3-VL-235B-A22B, and newer InternVL generations such as InternVL3.5, would provide a more informative evaluation. TouchAgent itself is tested only with a Qwen2.5-7B controller, leaving its dependence on controller capacity and generation unclear.
  3. The main results compare a specialized tool-augmented system against models evaluated through direct multiple-choice prompting. TouchAgent additionally benefits from domain-specific perception models, curated knowledge, and trajectory training. The base TouchAgent and component ablations are useful, but they do not establish an advantage over a strong general-purpose agent given the same resources. This is particularly relevant because KnowledgeSearch can itself invoke GPT-5.2 during inference, as described in Appendix A.3.3. A comparison with a GPT-5.2- or Claude-based agent using the same tools, knowledge base, and interaction budget would better establish the contribution of the proposed planning and training approach.
Questions for authors
  1. How does TouchBench differ from SToLa’s TactileBench in the reasoning capabilities it evaluates, beyond scale and tool-trajectory annotations? Could you include comparisons with Octopi-1.5 using retrieval and with Touch-R1 on compatible tasks, or explain the constraints preventing these comparisons?
  2. Could you extend the evaluation to larger open-source models and newer model generations, and test TouchAgent with a newer or larger controller while keeping the tool environment fixed? Do the reported capability gaps and training gains persist as model capacity increases?
  3. How does TouchAgent compare with a strong general-purpose agent using the identical toolkit, knowledge base, and tool-call budget? Please also clarify the frequency and computational cost of GPT-5.2 calls within KnowledgeSearch.
7 comments
AuthorsOpenReview ↗

Response to Reviewer GRDS

We sincerely thank the reviewer for the constructive feedback and positive assessment of our paper’s presentation, informativeness, and soundness. We have carefully considered all comments and revised the manuscript accordingly.

W1 & Q1: Benchmark Differences and Comparisons with Related Methods.

We sincerely thank the reviewer for the constructive comments. We further clarify the benchmark differences and add comparisons with related methods.

(1) Differences from SToLa and TactileBench. SToLa primarily improves tactile understanding and question answering through joint modeling of touch and language. Its corresponding benchmark, TactileBench, takes tactile observations and predefined question templates as input, without additional modalities, and evaluates basic physical property understanding and commonsense reasoning [1]. Its evaluation paradigm primarily follows an end-to-end, single-turn understanding paradigm, similar to existing tactile evaluation studies [2–5], without systematically assessing knowledge-enhanced reasoning, multi-source evidence integration, and multi-step reasoning. In contrast, while covering these foundational capabilities, TouchBench further evaluates heterogeneous information integration, domain knowledge utilization, cross-observation relational inference, and cross-modal evidence verification through multi-level task design, emphasizing multi-source evidence integration and multi-step reasoning to better assess the complex reasoning capabilities required for real-world tactile intelligence.

(2) Comparisons with Octopi-1.5 using retrieval and Touch-R1. We have additionally evaluated both methods on TouchBench, with each method following a unified evaluation protocol on test samples with compatible inputs. As shown in Table 1, TouchAgent-7B achieves an average accuracy of 76.1%, with improvements of 24.1 and 22.2 percentage points over Octopi-1.5[6] and Touch-R1[5], respectively. These results further demonstrate its advantages across multiple tactile understanding and reasoning tasks.

We have further clarified the distinctions from related work in Section 2 and added the corresponding experimental results in Table 1 of the revised manuscript (highlighted in red).

Table 1: Comparison with related tactile reasoning methods.

ModelAttr.Match.Inter.Scene.Dyn.Cross.Avg.Δ (TouchAgent − Baseline)
Octopi-1.5-7B65.260.151.458.436.440.352.0+24.1
Touch-R1-7B64.357.947.855.951.246.253.9+22.2
TouchAgent-7B78.674.677.973.876.375.476.1—

Reference

[1] STOLA: Self-Adaptive Touch-Language Framework for Tactile Commonsense Reasoning in Open-Ended Scenarios. AAAI 2026.

[2] Octopi: Object property reasoning with large tactile-language models. RSS 2024.

[3] Universal visuo-tactile video understanding for embodied interaction. NeurIPS 2025.

[4] Touchthinker: Scaling tactile commonsense reasoning to the open world with large-scale data and action-aware representation. EMNLP 2026.

[5] Touch-R1: Reinforcing Touch Reasoning in MLLMs. NeurIPS 2026.

[6] Demonstrating the Octopi-1.5 Visual-Tactile-Language Model. RSS 2025.

AuthorsOpenReview ↗

Response to Reviewer GRDS

W2 & Q2: Model Scale and Controller Scalability.

We agree with this suggestion and have added experiments on both expanded baselines and alternative controllers to further validate the effectiveness and scalability of our method.

(1) Comparisons with newer and larger baseline models. Following the reviewer's suggestion, we have included newer general-purpose MLLMs, namely InternVL3.5-8B, LLaVA-OneVision-2-8B-Instruct, and MiniCPM-V-4.5 (8B), as well as larger tactile-language models, namely Octopi-13B, VTV-LLM-14B, and TouchThinker-14B, under the same TouchBench evaluation protocol. In addition, we have included Qwen3-VL-32B and Qwen3-VL-235B-A22B alongside the previously evaluated Qwen3-VL-8B to compare different scales within the same model family. As shown in Table 2, TouchAgent with a 7B controller remains highly competitive against newer general-purpose MLLMs and larger tactile-language models, demonstrating the effectiveness of our method.

Table 2: TouchBench evaluation results with expanded baselines.

ModelAttr.Match.Inter.Scene.Dyn.Cross.Avg.
Open-source MLLMs
InternVL3.5-8B16.231.120.736.226.531.727.1
LLaVA-OneVision-2-8B-Instruct22.928.324.232.734.832.829.3
MiniCPM-V-4.5 (8B)16.227.920.736.236.822.726.8
Qwen3-VL-8B24.334.823.728.534.131.529.5
Qwen3-VL-32B26.132.424.527.836.934.730.4
Qwen3-VL-235B-A22B26.433.729.337.539.738.434.2
Tactile-Language Models
Octopi-13B57.652.138.645.439.539.145.4
VTV-LLM-14B61.263.841.447.343.840.149.6
TouchThinker-14B63.860.549.254.650.942.853.6
TouchAgent-7B78.674.677.973.876.375.476.1

(2) Comparisons across controller scales. To assess the dependence of our method on controller capacity and model generation, we extended the evaluation to six controller configurations: Qwen2.5 (7B, 14B, and 32B) and Qwen3 (8B, 14B, and 32B), each evaluated at the Base, SFT, and SFT+RL stages. All settings use the same tool environment, knowledge base, training data, and evaluation protocol. The Base models retain access to the tools and knowledge base but receive no training. As shown in Table 3, SFT improves macro-average accuracy by at least 5 percentage points (pp) across all six controllers, while RL provides a further gain of at least 3 pp over SFT. The average gains from the two stages are 5.7 pp and 4.1 pp, respectively. These results demonstrate that TouchAgent's training benefits extend across controller scales and model generations.

Table 3: Comparison of training stages across controllers of different capacities and model generations. All values are macro-averaged accuracies (%) across the six TouchBench task categories.

ControllerBaseSFTΔSFTSFT+RLΔRL
Qwen2.5-7B66.271.9+5.776.1+4.2
Qwen2.5-14B65.772.1+6.475.5+3.4
Qwen2.5-32B68.573.6+5.178.3+4.7
Qwen3-8B67.173.3+6.277.2+3.9
Qwen3-14B67.973.1+5.276.9+3.8
Qwen3-32B69.274.8+5.679.4+4.6

We have added the above experimental results to Table 1 in the main text and Appendix A.7 of the revised manuscript (highlighted in red).

AuthorsOpenReview ↗

Response to Reviewer GRDS

W3 & Q3: Comparisons with Strong General-Purpose Agents under the Same Resources and Additional API Costs

We thank the reviewer for these helpful suggestions and have conducted additional experiments and provided clarifications to address them.

(1) Comparisons with strong general-purpose agents under the same resources. We replaced the original Qwen2.5-7B with GPT-5.2 and Claude-Opus-4.6 as Agent controllers, while keeping the toolset, knowledge base, and interaction budget identical. Neither general-purpose model receives additional training. As shown in Table 4, both general-purpose agents outperform the untrained Qwen2.5-7B Base (66.2%), indicating that controller capability contributes to tool-augmented reasoning. However, trajectory-supervised SFT raises TouchAgent-7B's accuracy to 71.9%, and subsequent RL further improves it to 76.1%, exceeding the strongest general-purpose agent, Claude Opus 4.6 (69.9%), by 2.0 pp and 6.2 pp, respectively. These results demonstrate that specialized interaction-trajectory training and reinforcement learning provide additional benefits for tool planning and evidence integration under identical domain-specific resources. We have added the above experimental results to Table 8 in Appendix A.7.1 of the revised manuscript (highlighted in red).

Table 4: Comparison of agents using the same toolset, knowledge base, and interaction budget.

ModelAttr.Match.Inter.Scene.Dyn.Cross.Avg.Δ (vs. TouchAgent (SFT+RL))
GPT-5.271.367.669.165.468.967.368.3+7.8
Claude Opus 4.672.868.170.269.270.368.569.9+6.2
TouchAgent (Base)69.264.767.464.266.165.566.2+9.9
TouchAgent (SFT)74.871.573.170.471.670.171.9+4.2
TouchAgent (SFT+RL, Ours)78.674.677.973.876.375.476.1—

(2) Frequency and cost of GPT-5.2 calls within KnowledgeSearch. As described in Appendix A.4.3, KnowledgeSearch prioritizes exact matching and semantic retrieval to obtain knowledge, calling GPT-5.2 only when the retrieved results are incomplete or additional physical explanations are needed. We recorded the calls made by TouchAgent-7B on the full 10K test set: 1,349 questions triggered GPT-5.2 calls within KnowledgeSearch, accounting for 13.49% of the test set. These calls incurred an additional computational cost of approximately USD 34.2.

Reviewer_kVrhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TouchBench, a tactile understanding benchmark comprising 10,000 four-option multiple-choice questions across six task categories and 16 subtask types, constructed from 12 public datasets containing approximately 383,000 observation frames. It also presents TouchAgent, a ReAct-style framework that integrates a Qwen2.5-7B controller with eight tools for tactile perception, visual analysis, knowledge retrieval, and numerical and semantic reasoning. The controller is trained on approximately 40,000 synthesized interaction trajectories through supervised fine-tuning and reinforcement learning, with rewards for answer correctness, trajectory adherence, output format, and evidence grounding. Evaluation against 14 baselines shows that TouchAgent achieves 76.1% macro-averaged accuracy on TouchBench, compared with 52.3% for TouchThinker and 34.7% for the strongest evaluated general-purpose MLLM. Training-stage and tool-component ablations, alongside qualitative analyses of successful and failed trajectories, further characterize the system’s performance.

Critical strengths
  1. Broad task coverage with substantial curation effort. TouchBench integrates heterogeneous tactile data into a unified evaluation framework spanning perception, reasoning, and cross-modal understanding. Expert annotation and multi-stage question review strengthen its potential value as a benchmark for diverse tactile capabilities.

  2. Consistent system-level gains from controller training. Accuracy improves from 66.2% to 76.1% through SFT and RL, with gains across all six task categories. Tool ablations and analyses of both successful and failed trajectories provide useful insight into the system’s capabilities and limitations.

Critical weaknesses

Update after author response: The authors clarified training/evaluation separation and numerical reference targets, corrected dataset documentation, and added a fixed-pipeline comparison and an independent expert evaluation. Reference-trajectory length may still overstate the reasoning needed to distinguish options, and canonical-route rewards favor predefined paths over valid alternatives. However, most of my concerns have been addressed, and I am therefore raising my rating from 2 to 3.

  1. Evaluation independence and reasoning requirements remain insufficiently established. Although TouchBench preferentially samples official source-dataset test splits, the manuscript does not clearly establish how its observations are separated from instruction-training instances and the training data of its constituent tools (Apps. A.1, A.3–A.5). This limits confidence in evaluation independence without establishing that contamination occurred. Moreover, an illustrated coordinate-transformation question can be answered using normal force alone, suggesting that reference-trajectory length does not necessarily reflect the reasoning required to solve a question (Fig. 7, p.17).

  2. The benefits of adaptive planning and evidence grounding remain insufficiently established. The base/SFT/RL comparisons support the benefits of controller training, but do not establish whether adaptive planning outperforms a fixed pipeline using the same controller, tools, knowledge access, and comparable inference budget. Evidence-grounding improvements lack independent held-out validation, while canonical-route rewards may penalize valid alternatives and should be distinguished from flexible, evidence-supported planning.

Questions for authors

Update after author response: The authors’ response and additional evaluations have addressed my main questions. I have no further questions at this stage.

  1. Training/evaluation separation. What are the sources of the 32K SFT and 8K RL instances, and how are their observations and interactions separated from TouchBench? Please clarify tool-training splits, relevant overlaps, and whether the results demonstrate generalization to unseen interactions, objects, or sensors.

  2. Dataset and sensor documentation. Figure 6 labels FeelSight as “GelSight17 var.2” and HaTT as “GelSight var.2,” conflicting with Appendix A.1’s descriptions of DIGIT sensors and non-image signals, respectively. Which descriptions are correct, and how are HaTT signals represented as inputs and counted as “frames”?

  3. Controlled attribution and shortcuts. Beyond the base/SFT/RL comparisons, how does adaptive planning compare with a fixed pipeline using the same controller, tools, knowledge access, and comparable inference budget? How common are questions answerable from a single discriminating quantity, as in Figure 7’s rotation example?

  4. Definition of numerical ground truth. Figure 9 uses predicted interval midpoints to obtain 3.81 mm and a component ratio of 0.43. Are these midpoint estimates the defined benchmark targets, or should answers describe measured physical motion? How is uncertainty handled when the intervals span different categorical outcomes?

10 comments
AuthorsOpenReview ↗

Response to Reviewer kVrh

We sincerely thank the reviewer for the constructive feedback. We have carefully considered all comments and revised the manuscript accordingly.

W1: Evaluation independence and reasoning requirements remain insufficiently established.

We appreciate these concerns and have provided clarifications and conducted additional evaluations to address them.

(1) Evaluation independence. We would like to clarify our data splits in detail. We collected a large volume of data from multiple sources and constructed TouchBench by sampling from test splits defined by the source datasets or our own splitting protocols (please see Appendix A.1 for details on data sources and test-set sampling). All training samples, including instruction-tuning data and tool-training data, were strictly sampled from outside these test splits. Therefore, the training and evaluation samples from these sources do not overlap. We have clarified the data sources and split protocols for the instruction-tuning data and each tool's training data in Appendix A.2 of the revised manuscript (highlighted in red).

(2) Actual reasoning requirements. We would like to clarify that, although the example in Fig. 7 allows the correct answer to be identified using normal force alone because of differences between the options, the solution process still requires understanding the task requirements and coordinate-frame relationships, estimating three-axis contact forces from tactile observations, and applying the relevant physical knowledge to reason about and compute (Fx′,Fy′,Fz′)(F_x^{\prime}, F_y^{\prime}, F_z^{\prime}). Therefore, simplifying some numerical computations does not imply that answering the question requires little reasoning. Moreover, the reference-trajectory length reflects the complete, verifiable process of evidence acquisition and reasoning for the task. Errors in earlier perception or intermediate reasoning can affect the final prediction, reflecting the practical requirements for multi-stage evidence integration and reasoning reliability in real-world tactile question answering. We understand your concern regarding how the options can be distinguished, and we further quantify the prevalence of such cases in TouchBench, as reported in Table 1. Only 57 questions in the dynamic understanding category exhibit similar cases, accounting for 2.74% of that category and 0.57% of the entire benchmark. No such cases were found in the other five categories, indicating that this phenomenon is not prevalent in TouchBench.

Table 1. Audit of questions answerable using a single discriminative cue across TouchBench task categories.

TaskAudited QuestionsSingle-cue Shortcut QuestionsProportion (%)
Attr.1,87200.00
Match.2,10200.00
Inter.1,00100.00
Scene1,16100.00
Dyn.2,082572.74
Cross.1,78200.00
Overall10,000570.57
AuthorsOpenReview ↗

Response to Reviewer kVrh

W2: The benefits of adaptive planning and evidence grounding remain insufficiently established.

We thank the reviewer for this suggestion. We have added experiments to evaluate the benefits of adaptive planning and evidence grounding. We also further clarify the reference-route reward below.

(1) Practical benefits of adaptive planning. We added a Fixed Pipeline comparison to evaluate the independent contribution of adaptive planning. To ensure a fair comparison, both settings use the same controller, tool library, and knowledge access, with comparable inference budgets, differing only in tool scheduling and termination strategies. Specifically, the fixed pipeline executes a fixed tool sequence based on a predefined task category or format, without adjusting tool selection, invocation order, or termination timing in response to intermediate observations. Adaptive planning instead selects tools dynamically based on the task goal and accumulated observations and determines whether further evidence is needed. In both settings, the same controller integrates the tool observations and generates the final answer. As shown in Table 2, adaptive planning outperforms the fixed pipeline in accuracy across all six TouchBench task categories. These results show that it can dynamically adjust tool selection, invocation order, and termination timing to the requirements of individual instances, thereby improving task adaptability and overall reasoning accuracy.

Table 2. Comparison between the fixed pipeline and adaptive planning on TouchBench. Accuracy is reported in %. Avg. denotes the mean accuracy across the six task categories.

MethodAttr.Match.Inter.SceneDyn.Cross.Avg.
Fixed Pipeline78.172.773.871.372.972.573.6
Adaptive TouchAgent78.674.677.973.876.375.476.1

(2) Independent Held-out Validation of evidence grounding. To independently validate the model's evidence-grounding capability, we constructed a held-out evaluation set by drawing a stratified sample of 600 questions across the six TouchBench task categories, ensuring that none of these questions was used in SFT, GRPO, or reward tuning. We then collected the complete interaction trajectories generated by TouchAgent for these questions. Two tactile-domain experts independently conducted blind evaluations without access to the ground-truth answers or GPT-5.2 reward scores, with a third expert adjudicating any disagreements. The evaluation was based on the actual tool observations and the extent to which they supported the model's reasoning and final conclusions. Specifically, it assessed the completeness of key evidence acquisition, consistency in interpreting tool observations, and the degree of support for final conclusions. The results in Table 3 provide independent validation of the model's evidence acquisition, integration, and reasoning reliability.

Table 3. Independent evaluation of evidence grounding on the held-out set. Task accuracy is reported as a percentage. Each of the three evidence-grounding metrics is rated on a scale of 1–3, where 3 indicates complete evidence acquisition, consistent observation interpretation, or sufficient conclusion support; 2 indicates partial deficiencies; and 1 indicates substantial deficiencies. Scores are averaged over the same 600 questions.

MethodTask Accuracy (%) ↑Evidence Completeness (1–3) ↑Observation Consistency (1–3) ↑Conclusion Support (1–3) ↑
TouchAgent (Base)67.52.172.352.13
TouchAgent (SFT)70.82.512.562.65
TouchAgent (SFT+RL)76.82.842.892.87

(3) Reference-route reward. We acknowledge that strict canonical-route matching may prevent valid alternative routes from receiving the trajectory reward. However, the reference route is intended to guide tool use rather than prescribe the only valid solution path. We also introduce complementary rewards to independently evaluate answer correctness, interaction format, and evidence faithfulness (Appendix A.5.2). In particular, the evidence-faithfulness reward (RfaithR_{\mathrm{faith}}) assesses evidence integration and conclusion support based on actual tool observations, without requiring a match to the reference route. Therefore, valid alternative routes can still receive the corresponding non-trajectory rewards, provided that the final answer is correct, the interaction is valid, and the conclusion is sufficiently supported by the acquired evidence.

AuthorsOpenReview ↗

Response to Reviewer kVrh

Q1: Training/evaluation separation.

We appreciate the reviewer’s concern and clarify it below.

(1) Separation of training and evaluation data. (i) The 32K SFT and 8K RL samples were drawn from a single pool of 40K TouchAgent-Instruct multi-turn interaction trajectories, generated by GPT-5.2 based on training instances and their task-specific reference trajectories. All underlying instances were sampled exclusively from the training splits of the original data sources. TouchBench is constructed primarily from the corresponding official test splits; for datasets without official splits, we adopt independent data-splitting protocols (see Appendix A.1 for details). Thus, the underlying observations and interaction records used for Agent training do not overlap with TouchBench evaluation data. (ii) Regarding tool training, the CLIP-based attribute model used by AnalyzeAttributes is trained on tactile attribute annotations from the training splits of VTV-150K and TouchThinker. InterpretTactile uses a pretrained TouchThinker model (see Tables 4 and 5 in the paper), whose pretraining follows its object-level training split. EstimateState employs Sparsh-based downstream predictors, trained on the corresponding training split following the original work's train/test protocol [1]. The task-specific training samples of all three tools are non-overlapping with the TouchBench evaluation samples.

(2) Scope of the generalization evaluation. TouchBench test samples and TouchAgent training data are completely separated at the object-instance level. In addition, when constructing the training data, we excluded some of the sensor configurations and interaction types used for evaluation. TouchBench therefore includes object instances unseen during training, as well as some unseen interaction types and sensor configurations, allowing us to evaluate generalization in real-world tactile scenarios. Thus, the experimental results in Table 1 of the original manuscript demonstrate that the model can generalize to unseen interactions, objects, or sensors.

We have clarified the separation between training and evaluation data in Appendix A.2 of the revised manuscript (highlighted in red).

Q2: Dataset and sensor documentation.

We thank the reviewer for the careful review and corrections. FeelSight uses DIGIT sensors, as correctly described in Appendix A.1. We have corrected the corresponding sensor label in Figure 6. For HaTT, TouchBench uses the supplementary GelSight visuotactile pressing videos collected by Heravi et al. (2020) [2] using the original HaTT materials, rather than the vibration, force, position, and other time-series signals in the original HaTT dataset. Accordingly, the model receives tactile image frames extracted from the GelSight videos. The 7,362 frames refer to the GelSight tactile image frames included in TouchBench, rather than samples of the original physical signals. We have corrected the relevant data descriptions in the revised manuscript.

Q3: Controlled attribution and shortcuts.

We sincerely thank the reviewer for the constructive feedback and have conducted controlled comparisons and a full-dataset statistical analysis.

(1) Independent contribution of adaptive planning. We added a fixed-pipeline comparison. Both settings use the same controller, tool library, and knowledge access, with comparable inference budgets, and differ only in tool scheduling and termination strategies. As shown in Table 2, adaptive planning outperforms the fixed pipeline in accuracy across all six TouchBench task categories, demonstrating its practical benefits.

(2) Prevalence of questions answerable using a single physical quantity. We systematically audited all 10,000 questions in TouchBench. As shown in Table 1, only 57 questions in the dynamic understanding category can potentially be answered using a single physical quantity, accounting for 2.74% of that category and 0.57% of the entire benchmark. No such cases were found in the other five categories, indicating that this phenomenon is not prevalent in TouchBench.

Q4: Definition of numerical ground truth.

We thank the reviewer for the careful reading. The underlying displacement GT annotations for these samples are provided as intervals. The values of 3.81 mm and 0.43 in Figure 9 are derived benchmark reference targets, computed by applying predefined formulas to the midpoints of the corresponding GT intervals, rather than directly measured physical motion quantities. When a predicted metric interval crosses a categorical threshold, the agent invokes relevant tools for additional evidence. If uncertainty remains, it reports an uncertain result, counted as incorrect under the benchmark protocol.

Reference

[1] Sparsh: Self-supervised touch representations for vision-based tactile sensing. CoRL 2024.

[2] Learning an Action-Conditional Model for Haptic Texture Generation. ICRA 2020.

Reviewer_79eVOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces TouchBench, a tactile reasoning benchmark with 10K questions across six tasks, and TouchAgent, a tool-augmented framework achieving 76.1% accuracy.

Critical strengths

1.Large-scale benchmark covering diverse sensors and tasks.​

  1. Meaningful exploration of tactile perception and reasoning.​

  2. Strong empirical results with reasonable ablations.

Critical weaknesses
  1. Limited novelty: The method mainly combines existing tool-calling, SFT, and GRPO techniques.​

  2. Baseline fairness: Competing models lack access to the same tactile tools.

​3. Potential data leakage: Overlap between benchmark and tool training data is unclear.​

  1. Limited generalization: Insufficient evaluation on unseen sensors, objects, and datasets.​

  2. Limited realism: Multiple-choice QA does not demonstrate closed-loop robotic capabilities.​

  3. Trajectory bias: Rewarding reference tool sequences may penalize alternative valid strategies.

Questions for authors

1.Can the authors provide baselines with identical tool access, including a simple LLM given all tool outputs?​

  1. Is there any overlap between evaluation and tool training data?

​3. Can the authors evaluate generalization to unseen sensors and objects?​

  1. Why reward reference tool trajectories rather than task success alone?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 79eV

We sincerely thank the reviewer for the thoughtful feedback and are encouraged by their recognition of our work. We have carefully considered all comments and revised the manuscript accordingly.

W1: Clarification of the Novelty

We understand the reviewer's concern and would like to further clarify our motivation. Our contribution lies in moving beyond conventional end-to-end tactile understanding based on predefined observations by developing a tool-augmented reasoning framework and its associated training paradigm for real-world tactile tasks. Rather than simply combining existing components, we design two complementary training stages. First, Trajectory-Supervised SFT is not conventional answer-level fine-tuning [1-3]; instead, it leverages multi-turn interaction trajectories containing tool calls, intermediate observations, and evidence integration to teach task-aware evidence acquisition and tool-planning strategies. Second, Evidence-Grounded RL goes beyond rewarding final-answer correctness by jointly optimizing tool-use trajectories, answer accuracy, interaction format, and evidence faithfulness, encouraging predictions grounded in actual tool observations. More importantly, the two stages are explicitly coupled: SFT establishes structured tool-use and evidence-integration capabilities, while RL further refines tool planning and evidence-grounded reasoning. The ablation results in Table 2 and qualitative results in Figure 5 of the main text further demonstrate that SFT alone or RL with partial reward signals is less effective than the complete two-stage framework.

W2 & Q1: Baseline Comparisons with the Same Tool Access

We thank the reviewer for the constructive suggestions and have added two types of baselines. (a) Stronger general-purpose controllers. We replaced the original Qwen2.5-7B with GPT-5.2 and Claude Opus 4.6 as Agent controllers, while keeping the tools, knowledge base, and Interaction Budget identical, to evaluate the planning and reasoning capabilities of strong general-purpose models with identical resources and tool access. (b) LLM with all tool outputs. We provide an LLM without task-specific training (Qwen2.5-7B Base) with all tool observations produced in the complete interaction trajectory for each question, allowing it to directly integrate the evidence and generate an answer without further tool calls. This evaluates its ability to integrate evidence from all tool outputs.

As shown in Table 1, TouchAgent achieves a macro-average accuracy of 76.1%, outperforming GPT-5.2, Claude Opus 4.6, and the All-Tool-Outputs baseline by 7.8, 6.2, and 6.0 pp, respectively. These results indicate that neither a stronger general-purpose controller with identical tool access nor direct access to all tool observations alone is sufficient to match TouchAgent's performance, highlighting the benefits of interaction-trajectory SFT and RL for autonomous tool planning and multi-source evidence integration.

Table 1: Comparison with general-purpose agents under matched tool resources and an All-Tool-Outputs baseline. All values are accuracies (%). Δ denotes the absolute improvement of TouchAgent over each baseline, measured in percentage points (pp).

ModelAttr.Match.Inter.Scene.Dyn.Cross.Avg.Δ (TouchAgent − Baseline)
General-Purpose Controllers
GPT-5.271.367.669.165.468.967.368.3+7.8
Claude Opus 4.672.868.170.269.270.368.569.9+6.2
All-Tool-Outputs Baseline
Qwen2.5-7B Base + All Tool Outputs71.968.572.668.269.769.570.1+6.0
Ours
TouchAgent-7B (SFT+RL)78.674.677.973.876.375.476.1—

Reference

[1] Octopi: Object property reasoning with large tactile-language models. RSS 2024.

[2] Universal visuo-tactile video understanding for embodied interaction. NeurIPS 2025.

[3] Touchthinker: Scaling tactile commonsense reasoning to the open world with large-scale data and action-aware representation. EMNLP 2026.

AuthorsOpenReview ↗

Response to Reviewer 79eV

W3 & Q2: Separation of Tool Training and Evaluation Data

We thank the reviewer for the careful reading. We further clarify the sources of the tool training data and how these data are separated from TouchBench. The tactile tools involving training include AnalyzeAttributes, InterpretTactile, and EstimateState. AnalyzeAttributes uses a CLIP-based attribute model trained on tactile attribute annotations from the VTV-150K [1] and TouchThinker training splits. InterpretTactile uses the pretrained TouchThinker model and follows its object-level training/test split [2]. EstimateState uses Sparsh-based downstream predictors trained according to the task-specific data splits of the original work [3]. The corresponding TouchBench evaluation samples are drawn from the respective test splits and do not overlap with the training samples of these tools. We have clarified the training data sources and data-splitting protocols for each tool in Appendix A.2 of the revised manuscript (highlighted in red).

W4 & Q3: Generalization to Unseen Sensors and Objects

TouchBench test samples are completely disjoint from both TouchAgent training data and the training data of the relevant tactile tools at the object-instance level. We excluded certain sensor configurations used in the evaluation from the training data. As a result, TouchBench contains object instances unseen during training, as well as some unseen interaction types and sensor configurations. Thus, our existing evaluation already covers cross-object and cross-sensor generalization, with the overall performance reported in Table 1 of the main text.

W5: Closed-Loop Capabilities on Real Robots

We agree that multiple-choice question answering is insufficient to establish closed-loop robotic control capabilities. However, we would like to clarify that the core aim of this work is to study perception, understanding, and evidence-supported multi-step reasoning over real tactile observations, rather than directly implementing closed-loop robotic control, as discussed in Appendix A.9 (Limitations & Future Work). In the future, we plan to integrate TouchAgent with tactile VLA models and use it as a high-level reasoning and decision-making module to guide robot actions, thereby enabling closed-loop tactile interaction.

W6 & Q4: Reference Trajectory Reward

We acknowledge that rewarding only reference tool-call sequences may prevent valid alternative paths from receiving the trajectory reward. We introduce this reward because rewards based solely on task success provide limited supervision for intermediate evidence acquisition and tool use. To avoid excessive dependence on the reference trajectory, we introduce two complementary rewards: the answer-correctness reward (RresultR_{\mathrm{result}}) and the evidence-faithfulness reward (RfaithR_{\mathrm{faith}}), which evaluate the correctness of the final answer and the extent to which it is supported by actual tool observations, respectively. Neither reward requires matching the reference trajectory (see Appendix A.5.2). Therefore, valid alternative tool-use strategies can still receive rewards, provided that the final answer is correct and sufficiently supported by actual tool observations. This complementary reward design promotes evidence-grounded reasoning while preserving flexibility in valid tool-use strategies.

Reference

[1] Universal visuo-tactile video understanding for embodied interaction. NeurIPS 2025.

[2] Touchthinker: Scaling tactile commonsense reasoning to the open world with large-scale data and action-aware representation. EMNLP 2026.

[3] Sparsh: Self-supervised touch representations for vision-based tactile sensing. CoRL 2024.

Dear Reviewer,

Thank you again for your constructive feedback. We have revised the manuscript in response to the reviewers' comments and highlighted the main textual additions and revisions in red. The changes most relevant to your concerns are:

1. Comparisons under the same tool access. We have added comparisons with GPT-5.2 and Claude Opus 4.6 as controllers using the same toolset, knowledge base, and interaction budget, alongside the Base, SFT, and SFT+RL configurations of TouchAgent (Appendix A.7.1, Table 8).

2. Training and evaluation data separation. We have added a dedicated section documenting the sources and split protocols of the agent-training data and the task-specific training data for AnalyzeAttributes, InterpretTactile, and EstimateState, including their separation from TouchBench evaluation samples (Appendix A.2).

Please let us know if you have any further questions or concerns. We would be pleased to address them.

Best regards,

The Authors

Comments on the paper

1 comment

Dear Area Chair and reviewers,

Thank you and the reviewers for the careful evaluation and constructive feedback. We appreciate the recognition of the significance of our work (Reviewer #79eV), the comprehensiveness of TouchBench and the substantial effort involved in its construction (all reviewers), the significant performance improvements (all reviewers), and the soundness of our design, as supported by comprehensive ablation studies (all reviewers). We have revised the manuscript, with additions and substantive changes highlighted in red. We summarize the main updates below; detailed evidence and responses are provided in our individual replies.

1. Training/Evaluation Split Protocol.

We have further clarified that all agent-training and tool-training samples are strictly separated from the TouchBench evaluation samples. Accordingly, we have added a dedicated section detailing the data sources and split protocols for the 32K SFT and 8K RL training instances, as well as the task-specific training data for the AnalyzeAttributes, InterpretTactile, and EstimateState tools, together with the protocols ensuring this separation (Appendix A.2).

2. Expanded baseline comparisons.

We added comparisons with newer general-purpose MLLMs and additional tactile-language models, including larger models, under a unified TouchBench evaluation protocol. TouchAgent-7B achieves a macro-average accuracy of 76.1%, outperforming the compared baselines and further supporting its effectiveness in tactile understanding and multi-step reasoning tasks (Table 1 of the revised manuscript, with added results highlighted in red).

3. Comparisons with strong general-purpose agents under matched resources.

We added comparisons with GPT-5.2 and Claude Opus 4.6 using the same toolset, knowledge base, and interaction budget. After SFT and subsequent RL, TouchAgent outperforms the strongest general-purpose agent by 2.0 and 6.2 percentage points, respectively, supporting the additional benefits of specialized training under matched resources (Appendix A.7.1, Table 8).

4. Controller scalability.

We added evaluations of six Qwen2.5 and Qwen3 controllers across different scales and model generations at the Base, SFT, and SFT+RL stages. SFT and subsequent RL yield average accuracy gains of 5.7 and 4.1 percentage points, respectively, demonstrating that the training benefits extend across controller scales and model generations (Appendix A.7.2, Table 9).

We hope these revisions and additional experimental results have adequately addressed the reviewers’ concerns and helped support a favorable assessment of TouchAgent. With less than three days remaining in the discussion period, we would sincerely welcome any further comments or questions from the reviewers and the Area Chair. We would be pleased to provide any additional clarification if needed. Thank you again for your time and careful consideration throughout the review process.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.