acceptodds
Under review as a conference paper at ICLR 2027

EgoFact: Estimating Dense Full-Hand Tactile Fields from Egocentric Human Videos

Abstract

Understanding human dexterity from egocentric videos requires the ability to recover not only the kinematics, but also the physical tactile signals underlying the hand-object interface. However, estimating such tactile signals from egocentric videos remains a challenging task due to severe occlusions and heterogeneous tactile representations across sensing devices and datasets. To bridge this gap, we introduce EgoFact, a unified framework for estimating dense, full-hand tactile fields of both hands directly from egocentric videos. Specifically, 1) we map heterogeneous tactile measurements onto a canonical 3D hand surface with spatial decay and propagation around sensor centers, yielding a unified full-hand tactile representation for cross-device training and evaluation; 2) we develop a tracker-free vision-to-tactile model with force-guided learning objective to effectively infer tactile fields directly from RGB inputs; and 3) we introduce EgoFact-Data, a cross-device egocentric-tactile dataset with 100 hours of recordings, 800 clips, three glove types, six scenes, and 200 manipulation tasks, together with scene- and task-held-out protocols for evaluating generalization. Extensive experiments across five egocentric–tactile benchmarks demonstrate consistent improvements over existing baselines, achieving up to around 25% higher contact accuracy, 30% higher field IoU, and 40% lower force error. Notably, EgoFact further improves 3D hand–object interaction reconstruction and supports dexterous skill imitation, presenting the broader value of vision-to-tactile estimation for physically grounded egocentric intelligence.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Full-hand tactile estimation, egocentric vision, embodied Ai

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_tYBbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a framework to predict whole-hand tactile from visual inputs. The network is trained from glove tactile data collected by the authors (100 hours, 3 glove types, 6 scenes and 200 tasks). The proposed network outperforms baselines such as egotactile, touchanything.

Critical strengths

The tactile sensor template mapping is reasonable choice. The proposed network architecture looks solid and the efforts of dataset building is respectable.

Critical weaknesses

Despite the efforts, I do not find the technical depth and novelty of this paper for a venue like ICLR. This paper seems to include lots of engineering about mapping glove tactiles to a normalized hand, and some standard transformer details.

The predicted tactiles are heatmap instead of real force. The heatmap are not verified with any downstream applications. Most of the hand object interactions in the attached videos are object holding behavior without complicated interactions. So the tactile pattern mostly look similar. Also the proposed method does not work with robotics tactile (much more difficult to normalize into a standard template) or shear force. In sum, I do not learn any technical insight after reading this paper, maybe ICLR is not the right venue for it.

Questions for authors

See weakness.

3 comments
AuthorsOpenReview ↗

Response to Reviewer tYBb (Part 1/2)

Thank you for recognizing that our tactile template mapping is a reasonable choice, our network architecture is solid, and our 100-hour dataset collection effort is respectable. Although no specific questions or constructive suggestions are given, we hope the following explanations can help you to better understand our work.

Technical contribution of our work

Unifying tactile data from heterogeneous hardware into a transferable learning space is challenging and has not been solved well by prior methods. Also, our vision-to-tactile model, with a simple yet effective architecture, tackles the challenge of predicting full-hand tactile without 3D hand pose trackers. Rather than ad-hoc engineering or generic Transformer stacking, our study establishes two key representation insights:

  • Peak-Preserving Geometric Unification: Direct Mapping introduces sharp boundary discontinuities and overlap peak dilution, while unmapped raw regression locks models to a single glove's channel count. By contrast, our geodesic Gaussian upper-envelope representation preserves physical sensor peaks—reducing Peak Err. from 7.13 to 1.80 on EgoTouch in Table 4—improves native-sensor accuracy over Direct Mapping and Mean aggregation in Reviewer v4gG Q2, outperforms direct raw-signal regression in Common Q2, and unlocks cross-dataset pre-training gains in Reviewer 7YcA Q4.

  • Tracker-Free Multi-Level Fusion and Force-Guided Supervision over Generic Video Transformers: Pairing multi-level DINOv3 spatial fusion with our universal force-guided objective in Eq. 5 and Table 5 captures fine-grained contact cues under egocentric occlusion—whereas naively adding standard cross-frame spatiotemporal self-attention actually degrades V-IoU and force MAE by over-smoothing rapid contact transitions, as shown in Reviewer 7YcA Q10.

Misunderstanding of heatmap visualization and real physical force prediction.

The presented heatmap in our figures is just a visualization of our results, not our predictions and learning targets. All the ground-truth tactile signals are carefully calibrated and normalized for training, and the prediction of our model will be de-normalized back to real physical force in Newtons. Our experiments are also quantitatively evaluated as physical forces in Newtons—for example, reducing physical force MAE from 0.3800 N to 0.2229 N on EgoTactile and achieving 0.1442 N on OpenTouch (please refer to Table 2).

AuthorsOpenReview ↗

Response to Reviewer tYBb (Part 2/2)

Two downstream applications.

In the initial submitted version, we have already conducted two downstream applications to validate the effectiveness of our tactile representation and vision-to-tactile prediction model, which is clearly presented in Section 4.4, Figures 5–6, and Appendix Sec. D.1–D.2:

  • 3D Hand-Object Interaction Reconstruction: As shown in Fig. 5, and Appendix Table 2, incorporating our predicted tactile fields as additional guidance improves the reconstruction of the RGB-only baseline. Quantitatively, we evaluate on the OakInk benchmark and demonstrate that our tactile prediction helps to reduce hand MPJPE from 11.24 mm to 10.31 mm, improve Contact IoU from 13.25% to 24.06%, and avoid hand-object interpenetration from 22.11% to 3.18%.

  • Dexterous Robot Skill Imitation: In Section. 4.4, Fig. 6, and Appendix Sec. D.2, we also clearly demonstrate that our predicted tactile fields can be leveraged to guide dexterous imitation learning in the simulator, consistently improving task success rate and accelerating the convergence process.

The presented “holding” interactions are important and not simple

Object holding is a fundamental form of human-object interaction and an essential component of most everyday manipulation actions. Across different object geometries, weights, and grasp poses, holding naturally involves distinct contact regions and physical force distributions across the hand, providing key tactile cues for downstream interaction modeling. Our experiment shows that, even for the state-of-the-art RGB-only baseline, the 3D HOI reconstruction results of “holding” still suffer from severe penetrations and inaccuracy, which can be effectively improved by introducing our tactile predictions.

Robotics tactile and shear forces are out of the scope.

As stated in our title, abstract, and introduction, the research scope of this work is human full-hand tactile learning from human egocentric videos, rather than standard tactile patterns of grippers, etc. Also, our application of tactile-guided dexterous imitation learning, to some extent, has verified the practical value of transferring predicted human full-hand tactile to isomorphic robots such as dexterous hands. Also, shear forces are out of scope since the research target of our work and all the related works (e.g., OpenTouch, TouchAnything, EgoTouch, EgoTactile, HumanTouch, HOPE, EgoTac, etc.) is normal forces. And, to the best of our knowledge, there does not exist a publicly available dataset capturing human full-hand tactile with shear forces, which we agree is a promising research direction in the future, but not related to this work.

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. We have tried our best to provide responses to all of your questions and we hope our responses would address your concerns.

As the discussion period nears its conclusion, we would be grateful for any final feedback you may have. Please don't hesitate to let us know if further clarifications would be helpful - we remain ready to provide additional details as needed.

Thank you again for your valuable insights and constructive feedback throughout this process.

Reviewer_7YcAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EgoFact, a unified framework for estimating full-hand dense tactile fields from egocentric videos. The paper claims three contributions: (1) It presents a canonicalized sensor-to-hand mapping (using geodesic Gaussian smoothing) method to preserve the spatial force peaks and spatial force smoothness, supported by the ablation study. (2) With the canonicalized representation, the paper introduces an egocentric image-to-tactile prediction framework that integrates multi-level DINOv3 features to obtain tactile features, supported by the network backbone comparison. (3) The paper also provides a 100-hour egocentric video-to-tactile dataset (EgoFact-Data) across 3 devices. It also presents two applications: enabling accurate HOI reconstruction and tactile-informed imitation learning from egocentric videos.

Critique and summary recommendation: The paper is mostly valuable for the practical use of image-to-tactile generation tasks with the proposed canonicalized representations, with two promising applications. The proposed geodesic mapping is reasonable, and the 100 hours of collected data is valuable. However, I remain somewhat hesitant about the usability of the dataset. The paper lacks systematic analysis of sensor calibration, drifting overtime, and cross-device consistency, which cannot be resolved by a canonical mapping alone. The paper also doesn't sufficiently show if this large-scale dataset with canonicalized representation improves downstream performances in cross-dataset or cross-device generalization.

Overall, I find the image-to-tactile prediction results promising to support a weak acceptance, but I think the contributions of the canonicalized representation and dataset are not as convincing as claimed.

Critical strengths

(1) The proposed image-to-tactile backbone enables egocentric videos to empower tactile-related tasks, such as HOI reconstruction and force-guided imitation learning.

(2) Using geodesic Gaussian smoothing preserves more spatial granularity of the source sensor readings.

(3) The training weight guided by force magnitude is practically useful.

(4) The EgoFact dataset would be valuable if open-sourced to the community, as it is a cross-device tactile dataset.

Critical weaknesses

(1) Though the paper claims that the proposed representation unifies heterogeneous tactile data, it doesn't actually resolve sensor placement variation across subjects, as the sensor placement on the hand depends on a manually designed placement. The proposed method is more like a "glove-surface canonicalization".

(2) The proposed dataset, though collected across devices, does not show any analysis of cross-device consistency, which is supposed to be an advantage of a cross-device dataset.

(3) Though the mapped canonical representation appears more spatially smoothed and preserves peak values, it doesn't show how the proposed new representation helps with downstream tasks. For example, does the canonicalized representation help with cross-dataset/cross-device generalization? Table 3 here tries to present a transfer comparison, but the methods are all evaluated on the EgoFact or HumanTouch dataset. What is the performance like if trained on EgoFact but tested on other datasets? Also, what if the image-to-tactile generation generates the raw signal directly rather than the canonicalized signal?

Questions for authors

(1) Minor: L180, We -> we

(2) Will EgoFact-Data be released?

(3) The paper doesn't state if the proposed dataset reduces the self-deformation from raw glove readings and if it is calibrated across devices. From Figure 5, clearly self-deformation matters more than contact. It would be interesting to have this cross-device dataset to inspect consistent self-deformation patterns.

(4) Why does Figure 6 have a different contact pattern from the other images?

(5) yi = maxk=1,...,K[W(sk, vi)rk]

This suggests that the force peak is highest only when there is a sensor at that placement. I suppose the proposed method only works for full-hand dense sensors, but not for sparse fingertip sensors or with partially broken sensors.

(6) The force-guided loss seems like a per-glove design choice. Doesn't no mapping perform the best? The σ-5mm also seems like an heuristic, would it be beneficial if it's designed by sensor resolution?

(7) For 100 hours of data, how many hours have a hand in the frame and involve manipulation?

(8) As the current proposed image-to-tactile prediction is frame-wise, I wonder how is the spatial and temporal consistency of the predicted tactile reading across frames?

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7YcA (Part 1/2)

Thank you for recognizing the strengths of our image-to-tactile prediction backbone, the spatial granularity preserved by our geodesic Gaussian smoothing, the practical utility of our force-guided training weight, and the value of our 100-hour cross-device dataset (EgoFact-Data) for the community. We hope the responses below can address your concerns.

Q1: How is temporal sensor drift controlled during data acquisition?

Prior to data collection, we consulted directly with technical engineers from the glove manufacturers and incorporated their operational engineering experience to define our acquisition protocol. Specifically, our protocol bounds each continuous recording session to 3–15 minutes across 800 clips and re-estimates a fresh per-channel no-contact baseline at the start of every clip in Appendix Sec. C.3, effectively controlling slow thermal and mechanical sensor drift across recordings.

Q2: How is cross-device consistency achieved, and how does the mapping handle cross-subject sensor shifts?

  • Force-Magnitude and Spatial Consistency Across Devices: Cross-device consistency comprises both pressure-magnitude consistency and spatial consistency. Pressure-magnitude consistency is guaranteed by the careful per-clip baseline calibration, Newton conversion, and shared reference normalization detailed in Common Q1, whereas spatial consistency is achieved by propagating device-specific sensing layouts onto the shared 7,190-vertex surface via our proposed representation, as validated by the cross-dataset transfer gains in Q4.

  • Cross-Subject Consistency: Because tactile glove hardware comes in a fixed size and fixed sensor layout, we map this fixed hardware layout onto the statistical MANO hand model to maintain maximally consistent spatial alignment across subjects wearing the same glove. Experimental results on the EgoTactile cross-subject test set in Table 2 confirm the effectiveness of this design, raising C-IoU from 10.18% to 40.58%, V-IoU from 26.77% to 38.42%, and reducing force MAE from 0.3800 N to 0.2229 N.

Q3: How does the canonical representation benefit downstream tasks compared to Direct Mapping?

Following your constructive suggestion, we evaluate the performance improvement caused by tactile guidance in different representations on OakInk 3D HOI reconstruction benchmark. Results demonstrate that tactile guidance in our representation brings more improvement to RGB-only baseline, compared to piecewise-constant Direct Mapping:

Downstream 3D HOI (OakInk)MPJPE (mm) ↓Contact IoU (%) ↑Intersection (%) ↓
RGB-Only Baseline11.2413.2522.11
+ Tactile (Direct Mapping)10.7818.648.45
+ Tactile (Our Canonical Field)10.3124.063.18

Q4: How well do models trained on EgoFact-Data generalize to external benchmarks (expanding Table 3)?

Evaluating models trained on EgoFact-Data across representative external benchmarks with distinct sensor layouts confirms effective zero-shot transfer and consistent improvements over single-dataset in-domain training when fine-tuned:

Training Setting (V-IoU [%] ↑ / MAE [N] ↓)OpenTouchEgoTouch (Unseen)EgoTactile (Subject)
Zero-Shot (EgoFact-Data Only)31.42 / 0.185633.85 / 2.684027.94 / 0.3412
In-Domain Only (Table 2)48.09 / 0.144244.01 / 2.373638.42 / 0.2229
EgoFact-Data Pre-train + Fine-Tune50.64 / 0.135846.38 / 2.241541.15 / 0.2064
AuthorsOpenReview ↗

Response to Reviewer 7YcA (Part 2/2)

Q5: How much of the 100-hour dataset involves active manipulation, and will EgoFact-Data be publicly released?

Stratified sampling across the 100 hours shows that hands are visible in the egocentric frame for ~80 hours (80%) and involve active manipulation for ~70 hours (70%), with the remaining intervals capturing natural pre-grasp reaching, object releasing, and inter-task transitions. EgoFact-Data will be released in full upon publication, including all 100 hours of synchronized egocentric and exocentric videos, raw and calibrated tactile recordings across all three glove types, 7,190-vertex canonical mesh labels, sensor-to-MANO mapping files, and our training and evaluation codebase.

Q6: How are glove self-deformation responses during finger flexion handled in EgoFact-Data?

To ensure our data as clean and cross-device consistent as possible, we apply cross-device Newton calibration, per-clip baseline subtraction to remove initial static wearing offsets from each recording. While dynamic fabric-bending responses caused by strong finger flexion during manipulation are extremely hard to avoid, so they will be retained in our recordings. We agree that both hardware-level and software-level analysis and processing of signals from self-deformation is a very important research direction in the future.

Q7: Why does Figure 6 use a different contact visualization style from Figures 1–5?

The tactile predictions are visualized using lightweight Matplotlib in Fig.6, rather than blender engine in Figure 1-5, primarily for visualization efficiency. We will unify the presentation style in our final version.

Q8: Does upper-envelope aggregation in Eq. 3 generalize to sparse fingertip sensors and partially broken sensors?

In Eq. 3, tactile signals indeed peak at active sensor centers and decay geodesically with distance. Regarding the two scenarios:

  • Sparse Sensor Layouts: For sparse fingertip sensors, decaying weights toward zero in uninstrumented regions such as the palm avoids hallucinating spurious forces where no sensors exist while locally forming smooth fingerpad patches—as reflected on sparser regional layouts like OpenTouch (Table 2), where V-IoU improves from 28.33% to 48.09%.

  • Partially Broken Sensors: Online fault detection or signal restoration for defective hardware channels is outside the scope of this work; our representation is designed to map valid measurements from functional sensors, while hardware channel integrity during dataset collection is ensured via routine pre-session inspection.

Q9: Is the force-guided loss weight h(y)h(y) a per-glove heuristic or universal across devices?

Our force-guided loss weight h(y)h(y) is universal across all glove types and all five benchmarks, grounded in the shared distribution in Appendix Fig. 1.

Q10: How spatially and temporally consistent are the frame-wise predictions across consecutive video frames?

Spatially, predicting onto the geodesic-smoothed 7,190-vertex mesh inherently enforces within-frame surface smoothness, as verified in Table 4. Temporally, on continuous EgoTouch test clips, we evaluate cross-frame smoothness via 1D temporal Laplacian Temp. Lap. alongside prediction accuracy, and compare our frame-wise model against a temporal variant using spatial and temporal attention following the architecture of HOPE:

Method on EgoTouch (Seen / Unseen)V-IoU (%) ↑MAE (N) ↓Temp. Lap. ↓
Ours + Spatiotemporal Attention48.08 / 42.341.7809 / 2.62552.36 / 2.54
Ours (Frame-wise)52.56 / 44.011.6186 / 2.37363.84 / 4.12

Supported by stable SAM3 crops and smooth multi-level DINOv3 features across adjacent frames, our frame-wise predictions already maintain strong temporal smoothness close to the 3.65 ground-truth Temp. Lap. While adding the spatiotemporal module further lowers Temp. Lap. across frames, it incurs a noticeable drop in per-frame V-IoU and MAE due to its over-smoothing on high-frequency tactile signals and contact state transitions.

Minor Revision (Line 180): Thank you for pointing it out. We have corrected "We" to "we" in Line 180.

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. We have tried our best to provide responses to all of your questions and we hope our responses would address your concerns.

As the discussion period nears its conclusion, we would be grateful for any final feedback you may have. Please don't hesitate to let us know if further clarifications would be helpful - we remain ready to provide additional details as needed.

Thank you again for your valuable insights and constructive feedback throughout this process.

Reviewer_v4gGOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

EgoFact studies dense full-hand tactile estimation from egocentric RGB video. It maps heterogeneous glove measurements to a canonical palmar MANO surface through geodesic Gaussian kernels and upper-envelope aggregation, then predicts the constructed fields using multilevel visual features without an upstream hand-pose tracker. The paper also introduces EgoFact-Data, reported to contain 100 hours, 800 clips, three glove types, six scenes, and 200 manipulation tasks. Evaluations cover several tactile datasets and applications to interaction reconstruction and simulated imitation. On the methodology side, the novelty is not particularly substantial.

  1. Is Equation 5 implemented literally, or is a target-dependent non-contact mask omitted from the printed expression? Please provide the exact objective and the complete weighting profile.
  2. How accurately does the proposed field predict independently withheld sensor measurements, and how sensitive are the results to kernel width and aggregation?
  3. How do you comment on the possibly different trustworthiness of data from different tactile gloves? (Given that some gloves are very poor in accuracy)
Critical strengths

The cross-device representation addresses a practical interoperability problem. A shared geometric domain is useful for training interfaces, comparison, and downstream integration. The new collection could be a valuable resource if its calibration and evaluation protocols are reproducible.

Critical weaknesses

Dense targets are constructed from measured sensor readings using the proposed interpolation rule. Evaluation on the same constructed domain measures agreement with that rule, not independently measured accuracy at unsensed locations. Smoothness and peak preservation in Table 4 are relevant representation properties, but neither establishes that the spatial decay and maximum aggregation recover the physical field. Native-sensor evaluation, held-out-sensor reconstruction, or independent higher-density sensing would strengthen the central interpretation.

Questions for authors
  1. Is Equation 5 implemented literally, or is a target-dependent non-contact mask omitted from the printed expression? Please provide the exact objective and the complete weighting profile.
  2. How accurately does the proposed field predict independently withheld sensor measurements, and how sensitive are the results to kernel width and aggregation?
2 comments
AuthorsOpenReview ↗

Response to Reviewer v4gG

Thank you for recognizing the practical value of our cross-device representation in addressing hardware interoperability, providing a shared geometric domain for training and comparison, and supporting downstream integration, as well as the potential of EgoFact-Data as a community resource. We hope the responses below can address your concerns.

Q1: Is a non-contact mask applied in Equation 5? What is the exact implemented objective and weighting profile?

Thank you for pointing it out. A target-dependent non-contact mask I[yi<ε]I[y _ i \lt \varepsilon] is applied in our implementation and was condensed from the printed expression on Line 262 alongside its warm-up schedule λnc(t)\lambda _ {nc}(t) and the base uniform weight of 1. The exact objective executed in our code is:

L=1∣Vtac∣∑i=0∣Vtac∣−1[(1+h(yi))(y^i−yi)2+λnc(t)I[yi<ε]max⁡(y^i−ε,0)2]L = \frac{1}{\lvert V _ {tac} \rvert} \sum _ {i=0}^{\lvert V _ {tac} \rvert - 1} \left[(1 + h(y _ i))(\hat{y} _ i - y _ i)^2 + \lambda _ {nc}(t)I[y _ i \lt \varepsilon]\max(\hat{y} _ i - \varepsilon, 0)^2\right]

  • Complete Weighting Profile: The base weight of 1 maintains supervision across the full normalized range, including high-force peaks yi≥0.30y _ i \ge 0.30, while h(y)h(y) in Appendix Eq. 1 adds piecewise-linear emphasis over [0.05,0.30][0.05, 0.30]: (y−0.05)/0.05(y - 0.05)/0.05 on [0.05,0.10][0.05, 0.10], (0.30−y)/0.20(0.30 - y)/0.20 on (0.10,0.30)(0.10, 0.30), and 00 elsewhere, yielding total regression weights in [1.0,2.0][1.0, 2.0].

  • Non-Contact Regularization: This regularization term is active strictly on non-contact targets via I[yi<ε]I[y _ i \lt \varepsilon] with ε=0.05\varepsilon = 0.05, where λnc(t)\lambda _ {nc}(t) warms up linearly from 0.2 to 1.0 over the first 5 epochs and remains 1.0 thereafter.

We have clarified this in Eq. 5 and Appendix Sec. A in our revision.

Q2: How accurately does the proposed field recover independently withheld sensor measurements across aggregation rules?

We agree that smoothness and peak preservation in Table 4 characterize geometric regularity and in-situ peak retention rather than unsensed physical accuracy. To directly test physical field recovery at unsensed locations, we randomly withheld 20% of active sensors on EgoTouch during field construction and measured reconstruction error against true hardware readings at the withheld sites. Geodesic decay with upper-envelope (Max) aggregation at σ=5 mm\sigma = 5\text{ mm} recovers withheld physical measurements substantially more accurately than Direct Mapping and Mean aggregation, which dilutes local contact peaks:

Field Construction Rule (EgoTouch, 20% Withheld)Held-Out MAE (N) ↓Peak Err. ↓
Direct Mapping (Region Avg)2.547.13
Geodesic Gaussian + Mean (σ = 5 mm)1.875.64
Ours: Geodesic Gaussian + Max (σ = 5 mm)1.291.80

We sincerely appreciate the time and effort you have dedicated to reviewing our manuscript. We have tried our best to provide responses to all of your questions and we hope our responses would address your concerns.

As the discussion period nears its conclusion, we would be grateful for any final feedback you may have. Please don't hesitate to let us know if further clarifications would be helpful - we remain ready to provide additional details as needed.

Thank you again for your valuable insights and constructive feedback throughout this process.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to Common Questions (Part 2/2)

Common Q3: Why choose a fixed geodesic kernel width σ=5 mm\sigma = 5\text{ mm}, and how sensitive is tactile prediction to σ\sigma and sensor resolution? (v4gG, 7YcA)

  • Physical Locality and Empirical Sensitivity: Physically, single-site contact deforms soft skin tissue within a bounded local neighborhood rather than diffusing indefinitely across the hand, so resolution-adaptive kernel width is not suitable for current commercial gloves, which are typically sensor-sparse with only 162–549 discrete sensing sites per hand across our three gloves versus 7,190 canonical mesh vertices. Therefore, we train models on EgoTouch using targets built with representative narrow, default, and wide geodesic kernels with Max aggregation and evaluate predictions on native sensors across seen and unseen objects, confirming that σ=5 mm\sigma = 5\text{ mm} effectively balances local pressure locality and inter-sensor gap coverage:
Geodesic Kernel Width (EgoTouch)V-IoU (%) ↑MAE (N) ↓
σ = 2 mm (Narrow)51.46 / 43.081.7024 / 2.4580
σ = 5 mm (Our Default)52.91 / 44.361.6415 / 2.4018
σ = 10 mm (Wide)50.72 / 42.451.7385 / 2.4912
  • Future Dense-Coverage Gloves: Looking ahead, as future tactile gloves evolve toward substantially denser or quasi-continuous taxel distributions with negligible inter-sensor gaps, incorporating a finer, density-aware and resolution-adaptive kernel width could yield an even more expressive representation than a single fixed σ\sigma.
AuthorsOpenReview ↗

Response to Common Questions (Part 1/2)

We thank all the reviewers for your time, constructive feedback, and insightful comments. We hope the responses below can address your common concerns.

Common Q1: How are heterogeneous tactile gloves calibrated onto a unified physical force scale while ensuring cross-device data trustworthiness? (v4gG, 7YcA)

Wearable tactile gloves vary in native precision across sensing mechanisms, and spatial mapping alone cannot resolve hardware-level discrepancies. We ensure cross-device physical consistency and dataset trustworthiness through commercial-grade hardware selection and standardized per-clip force calibration:

  • Quantified Commercial Hardware Specifications: All three devices in EgoFact-Data are calibrated commercial systems with verified specifications:

    • Wuji-Glove: ±6% repeatability, 0.1 N force resolution.
    • JQ-Glove: ±8% repeatability and consistency, 0.01 N force sensitivity.
    • Matrix-Glove: ±10% full-scale (FS) repeatability and consistency, ±15% FS typical measurement error.
  • Standardized Cross-Device Force Calibration: Prior to spatial mapping, every clip undergoes per-channel no-contact baseline subtraction to remove initial wearing offsets, device-specific conversion into physical normal forces in Newtons strictly within its calibrated range to prevent out-of-range non-linearities, and normalization under a shared reference force scale so identical physical forces map to consistent normalized values across devices. We have expanded Appendix Sec. C.1 and C.3 with more details in our revision. All calibration parameters and evaluation scripts will be publicly released.

Common Q2: How does mesh-space evaluation relate to physical sensor accuracy, and how about the performance of training and evaluating directly on native sensor readings? (v4gG, 7YcA)

  • Physical Validity of Mesh-Space Evaluation and Interpolation Choice: Because ground-truth fields are anchored in calibrated physical normal forces (see response to Common Q1), mesh-space evaluation directly evaluates physical tactile accuracy, while different interpolation rules simply represent alternative ways to lift discrete native sensor readings into a continuous and dense space. Among the compared mapping rules, our geodesic Gaussian kernel with upper-envelope Max aggregation is specifically designed to preserve native physical sensor measurements at sensing sites while maintaining realistic surface continuity. In Table 4, compared with piecewise-constant Direct Mapping, it reduces in-situ peak error from 7.13 to 1.80 on EgoTouch and improves spatial smoothness from 36.19 to 7.86 on EgoTouch and from 37.79 to 4.22 on OpenTouch.

  • Native-Sensor Evaluation: Following your suggestion, we also train and evaluate a variant directly on native sensor readings, denoted as "Ours (Raw)", and compare it with "Ours (Mesh-space)". Results demonstrate that our method achieves consistent performance, which indicates that our representation and evaluation in mesh-space indeed assess the physical accuracy, rather than interpolation-rule agreement.

Method on EgoTouch (Seen / Unseen)V-IoU (%) ↑MAE (N) ↓
Best Prior Baseline45.82 / 38.641.8942 / 2.6415
Ours (Raw)51.84 / 43.421.6854 / 2.4310
Ours (Mesh-space)52.91 / 44.361.6415 / 2.4018
  • Unifying Heterogeneous Hardware: Crucially, our primary goal in adopting the 7,190-vertex canonical representation is cross-device unification. Regressing native sensor signals locks the output head to a single glove's channel count, such as 162(juqiao-glove), 368(matrix-glove), or 549(wuji-glove) channels, precluding cross-device joint training and cross-dataset transfer, whereas our mesh-space representation facilitates downstream tasks such as 3D HOI reconstruction. Our canonical field unifies disjoint sensor layouts into a shared geometric domain while preserving and slightly improving single-device native accuracy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.