- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces PalmDex, a robot-free multimodal dataset for dexterous tool use, combining ego-/exocentric videos, full-hand tactile signals, hand kinematics, and procedural annotations. It proposes four benchmarks covering tool-use cognition, interaction knowledge, cross-modal alignment, and motion prediction. Experiments evaluate 26 MLLMs and several task-specific baselines, reporting complementary visual–tactile information and an average 12.7 point improvement on EgoTouch after fine-tuning.
Critical strengths
PalmDex combines synchronized vision, full-hand touch, hand kinematics, and procedural annotations for functional tool use. This is a useful combination for studying physical interactions beyond conventional grasping.
Critical weaknesses
W1. Dataset statistics and annotation definitions lack clarity.
The reported episode counts (2,696/3,071/3,112), step counts (12,399/12,774), tool/action counts, and recording duration are inconsistent. The 31-hour claim is also unclear given 1.8M frames per view at 30 Hz. The paper lacks participant statistics, and clear mappings among task families, actions, steps, and stages. The seven grasp classes, success-label granularity, and annotation quality-control procedures are insufficiently described. Some labels are derived from tool mappings or tactile thresholds rather than independent human annotations. Action boundaries, background intervals, and transitions between grasping, positioning, and functional contact are also unclear. Moreover, Step Completion appears to measure segment termination rather than actual task success. These details are important for judging the dataset's reliability and novelty.
W2. Benchmark validity and the distinctive contribution need stronger evidence.
Several tasks resemble standard recognition, classification, retrieval, or forecasting benchmarks. The paper does not sufficiently justify its task definitions or show how the additional tactile, kinematic, and multiview data provide unique benefits. The evaluation relies heavily on general-purpose MLLMs. A few specialized or simple baselines would help establish task difficulty and rule out shortcuts, especially for tool recognition, grasp classification, and force-related prediction. Some protocols are unclear: open-ended answer scoring, inconsistencies between Figure 5 and the appendix, different video-frame budgets, and tactile heatmap encoding for VLMs. The analysis also overlooks substantial performance drops from multimodal fusion and lacks test-set sizes, uncertainty estimates, and detailed failure analysis. The EgoTouch transfer results are promising, but possible source overlap and improvements due merely to fine-tuning or learning answer formats need to be ruled out. Retrieval and motion-prediction protocols also require clarification. Finally, the dataset mainly measures hand–tool contact rather than tool–target interaction, limiting some claims about functional tool use.
Questions for authors
Q1. The numbers of episodes, steps, tools/actions, and recording hours differ across the paper. Could the authors clarify which numbers are correct? Please also provide the number of participants, and the data distribution across tasks.
Q2. What are the seven grasp types, and how are the high-level stages defined? Is task success labeled at the episode or step level? Which annotations were checked by humans, and which were generated automatically? How were action boundaries, transitions, and background periods handled? Was annotation consistency checked?
Q3. Were the 416 EgoTouch episodes excluded from training and model selection? How were the tactile signals from the two datasets processed to ensure consistent inputs? Could some of the improvement simply come from learning to interpret tactile heatmaps or follow the required answer format?
Q4. How are free-form answers scored? Figure 5 shows multiple-choice examples for tasks described as open-ended in the appendix. Which setting was actually used? Please also clarify the frame budgets, model versions, and masking procedures. Some tasks, such as tool and action recognition, would benefit from comparisons with standard recognition models or simple baselines.
Q5. How are raw tactile signals converted into heatmaps for the VLMs? Table 3(a) shows that adding touch sometimes reduces accuracy. Could the authors explain these cases? Please also report test-set sizes, confidence intervals, and the temporal-shuffling results for Force Trend.
Q6. How large is the retrieval gallery, and how are negative samples selected? What task information is given to the motion-prediction models? Are CVAE and DDPM evaluated using single predictions or multiple samples? Also, why are joint-position and joint-angle errors used inconsistently?
Q7. What does PalmDex offer that existing visual–tactile datasets do not? In particular, how do the annotations distinguish actual tool use from simply holding a tool? Which experiments show that the additional tactile, hand-pose, and multiview data are useful?
The related work could also be more complete. VTDexManip, DexViTac, IMPACT, TVL, and UniTouch seem particularly relevant and deserve discussion. Also, the star ratings for tool-use diversity in Table 1 are subjective without clear criteria. A comparison using actual dataset statistics would be more convincing.